Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis
Authors: Deepak Sridhar
Abhishek Peri
Rohith Reddy Rachala
Nuno Vasconcelos
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Synthesis; Controllable 2D/3D Synthesis; Diffusion
Primary area: diffusion_based_models
TL;DR: We introduce a new framework for modeling the joint distribution of images and conditioning variables by adapting Stable Diffusion to enhance prompt compliance, controllability and editing of images.

Abstract: Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text in images, and meeting constraints like object locations and pose. For fine-grained editing and manipulation, they also require fine-grained semantic or instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions at inference, this is time consuming and does not scale to complex scenes. 
  To overcome these limitations, this work introduces a new family of  $\textit{Factor Graph Diffusion Models}$ (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, semi-automated fine-grained editing, explainability at intermediate levels, ability to produce labeled datasets for the training of downstream models such as segmentation or depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose an implementation of FG-DMs by adapting a pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image. We also show that training FG-DMs from scratch on MM-CelebA-HQ, Cityscapes, ADE20K, and COCO produce images of high quality (FID) and diversity (LPIPS).

Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis
Deepak Sridhar
Abhishek Peri
Rohith Rachala
Nuno Vasconcelos
Department of Electrical and Computer Engineering University of California, San Diego
dderiidha, aperi, rrachala, nvasconcelos) LacilosssSuuucss @ucsd. edu
Abstract
Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text images, and meeting constraints like object locations and pose. For ane-grrined editing and manipulation. they also require fine-grained semantic instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions inference, this time consuming and does not scale to complex scenes. To overcome these limitations, this work introduces new family of Factor Graph Diffusion Models (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, remi-automated fine-grained editing, text-based editing of conditions with noise inversion, explainability intermediate levels, ability to produce labeled datasets for the training downstream models such as segmentation depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose implementation of FG-DMs by adapting pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image. We also show that training FG-DMs from scratch on MM-CelebA-HH Cityscapes, ADE20K, and COCO produce images of high quality (FID) and diversity (LPIPS). Project Page: FG-DM
Introduction
Diffusion models (DMs) 44 15 10:
41 have recently shown great promise for image synthesis and popularized text-to-image (T2I) synthesis, where image is generated in response to text prompt. However, T2I synthesis offers limited control over image details. Even models trained at scale, such Stable Diffusion (SD) (41 or DALL-E (37), have significant prompt compliance problems, such
as difficulty in generating multiple objects 7 34 difficulty in generating text in images (26) 46). or to
consistently produce images under certain spatial constraints, like object locations and poses (56) [].
These limitations have been addressed through two main lines of research. One possibility is to use inference-base operate on the cross-attention maps between image and prompt tokens to improve prompt compliance
(IBPC) methods 7 34 40), which use loss functions that
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Object Reed
masks
Existing
Ply 3O
Promer
Image
Seprentation Mole
Select Mac Objeet Facal
Creativve
Prtthes
Control Fast
eg. Stable Offiaton
<g Sog hume
Soaciyy Gigens/enaatee Orjer Carsse win LM
Faster
joytieeel Pig "a''y'>> y'
Proposed
Law rastitin Syrihhub
V Crestive
DEI
V Control
PCP" yP)
MORD of a
V Fast
Specly OkjectuGaveetee Siged Cassss with
Sebet Mac Objen ferell
Jicke dling
Avocaola
with
High resolation Image Syrtheik
Okrect Recal
budigtteaa
enary
ca. Controliee
Figure I: Comparison of FG-DM (bottom) against Stable Diffusion (top) for sampling images with high object
recall by modeling the joint distribution of images and conditioning variables. FG-DM supports creative, controllable, interpretable and faster (4x) image synthesis than Stable Diffusion to achieve the desired object recall. Note that the conditions y y can be null due classifier-free guidance training.
inference. While these methods are effective for prompts involving small number of objects, they tend to underperform for prompts involving complex scenes, Furthermore, because their complexity grows linearly with the number of scene objects, they tend to be prohibitively time-consuming for such scenes. A second possibility is to rely on DMs that support visual conditioning, in the form of sketches (33), bounding boxes (9), scene graphs (53). reference images (52), etc. Visually conditioned DMs (VC-DMs) are usually extensions of T2I-DMs trained at scale. For example, ControlNet (58), T21-Adapter (29) and Uni-ControlNet (60) use learnable branch to modulate the features of a pre-trained SD model according to visual condition. Despite their success, VC-DMs have important limitations, inherent to models of the conditional distribution P(x](y') of image x given conditions y": the need for user supplied conditions. The manual specification of visual conditions, like segmentation masks or normal maps, requires users with considerable time and skill. While, as illustrated in Figure I (top), conditions y can be extracted from existing images, this requires additional vision models (e.g. segmentation or edge detection), which is time-consuming.
In this work, we consider an alternative framework that attempts to mitigate all these problems using a simple but unexplored prompt compliance scheme that we denote as sampling- -aased prompt compliance (SBPC). The idea, illustrated in the top of Figure i to sample a batch of N images using different DM seeds, relying on an external model (e-g- segmentation) to measure prompt compliance (e.g. by measuring object recall) and choosing the image that best complies with the prompt. While this strategy is frequently successful, generating multiple high resolution images significantly increases the inference time, rendering the approach impractical even for small values of N as we will show in section 4 Furthermore, it does not address the need for specification of the conditions y' required by the VC-DM. We address these problems by introducing new family of Factor Graph-DMs (FG-DMs). As illustrated in the bottom of Figure [ a FG-DM is a modular implementation of the joint distribution P(x. {y'}) by decomposing the image synthesis into two or more factors, that are implemented by jointly trained VC-DMs. The figure shows an example decomposition of the distribution P(x, {y*} ) of image x, pose y2, and segmentation y2, given prompt y3, into three factors: P(y y for the synthesis of segmentation given prompt, Kyy' } for pose y synthesis conditioned on both, and P(x|[y }²_,) for image synthesis given all conditions.
The FG-DM framework has several advantages. First, prompt compliance can usually be measured (e.g. by computing object recall) by inspecting the conditions y; (e.g. segmentation map). The gain is that these can be generated with less diffusion steps and resolution than the final image For example, we have observed no loss of image quality by sampling segmentation maps of quarter resolution. This increases the speed of SBPC by 4x, making it practical prompt compliance scheme. We show that sampling with = 10 different seeds and choosing the image of maximum recall increases prompt compliance (object recall) by 15% compared to sampling with one seed. For complex scenes, it is also much faster and more effective than using IBPC methods (see Table ( Second, as illustrated editing capabilities. New objects can be added by synthesizing them separately while existing objects can be resized and/or moved the desired spatial location. In Figure (also in Figure with more
the bottom of Figure [] the modular nature of the FG-DM offers image

---

rod sling
buikdig Beady"
sying
Gereratdd
Wa Lee FIED DM
HELLD FG DM
Figure 2: PG-DM-based controllable image generation via editing segmentation, depth and sketch maps. Top: generated conditions and images. Bottom: edited ones. Note that only the segmentation map is edited, pose and images are conditionally generated given edited map.
"a white chimp pasing"
"a realistic picture a parcupina"
"Two littlc piris uniform"
train travelling through rustic countryside"
noalistic picture boy with lon's hairstyle"
"a faced horse"
Figure 3: Synthesized semmotationdpppissccchooonmn maps and corresponding images by an FG-DM adapted from SD using COCO. The FG-DM generalizes prompts beyond this dataset such as porcupine, chimp and other creative prompts shown.
detail), an airplane is added to the background while the person is resized, moved to the left of the image, and pose flipped. We introduce a simple image editing tool for performing these edits. Figure shows other examples of fine-grained image editing with FG-DM for semantic, depth, and sketch maps factors. In the center, the dog is placed behind the sandcastle (and some objects are added to the foreground) by manipulation of depth map, and in the right the desired text "Hello FG-DM" is scribbled on the sketch map produced by the model. Third, the FG-DM can reuse factors in the literature. For example, the ControlNet is used to implement the image synthesis factor P(x {y 13 Fourth, the FG-DM can produce labeled datasets for the training of downstream systems (e.g. image segmentation), and naturally supports continual learning schemes, where image synthesis and manipulation gradually enhanced by the progressive addition of new VC-DMs to the factor graph, with limited or no retraining of existing ones.
)i=1)
in Figures and 2
Since FG-DM models the joint distribution P(x, {y*)), it is variant of joint DMs (JDMs) and training a FG-DM from scratch requires large scale datasets of (condition, image) pairs, which are expensive to obtain. However, we show that this difficulty can be overcome by adapting existing foundation VC-DMs, such as SD, to implement each factor of the FG-DM. We propose a joint prompting scheme to implement this adaptation and introduce an attention distillation loss that distills the attention maps from a pre-trained SD model to implement the condition synthesis factors (( by minimizing the KI. divergence between the two and show that this improves the fidelity of the generated conditions. This greatly reduces the training costs and enables much greater generalization ability than would be possible by training on existing (condition,image datasets alone. Figure 3 shows that FG-DM exhibits robust generalization by synthesizing depth, normals and their corresponding images for novel objects not present in the training data. This approach also facilitates cross moosl information transfer, enhancing explainability and showcasing the FG-DM's versatility in complex synthesis tasks. In summary, this paper makes the following contributions
We propose a new framework for T2I, the FG-DM, which supports the modeling of the joint distribution of images and conditioning variables, while maintaining access to all their conditional relationships
We show that FG-DMs enable practical SBPC, by leveraging fast condition synthesis and filtering by object recall, and allow both fine-grained image editing with minimal effort and data augmentation.
3

---

We show that FG-DMs can be implemented by adapting pre-trained T2I models (e.g. SD) using efficient prompt-basee baspd adaptation on relatively small datasets (e.g. COCO) while exhibiting interesting generalization, e.g. generalizing to prompts involving concepts not covered by these datasets.
We introduce an attention distillation loss that improves the fidelity of the synthesized conditions and enables transfer of information from SD to all factors of the FG-DM.
We show that FG-DMs trained from scratch on domain-specifie datasets such as MM- CelebA-HQ, ADE20K, Cityscapes and COCO consistently obtains high quality images (lower FID scores) with higher image diversity, i.c. higher LPIPS scores, than standard JDMs.
2 Related Work
Text-to-Image (T21) Diffusson models[16] 30: 37 38 learn to synthesize images from noise conditioned by a text encoder, usually CLIP (36). Latent DMs (LDMs) (4I) implement DMs in the latent space learned by an autoencoder trained very large image dataset, to reduce inference and training costs. T2I models typically employ classifier
guidance (I7) to balance prompt compliance with sample diversity. However, they often struggle with complex prompts, which led to the development of conditional DMs. They model the distribution P(x }) of image x given a set of K conditions y². VC-DMs, such as ControlNet (58), T2I-Adapter (29), and HumanSD (19), use adapters to condition image generation pose, or normal maps. However, these methods are limited to conditions y generated from existing images by manual sketching, which can be hard to obtain, especially for K > (e.g. simultaneous segmentation and surface normals), and may be inconsistent. Instead, the proposed FG-DM enables the automated joint generation of all conditions while still allowing users the ability to edit them.
visual variables y", such as depth, semantic,
Joint Models model the distribution P(x, {y }), frequently by concatenating all variables during image generation. For example, SemanticGAN (2) and GCDP (32) use a single model to generate pairs of images X and semantic maps y. while Hyper Human (24) uses a single model to generate depth maps, normal maps, and images from human pose skeletons. These models lack access to the conditional distribution P(xly), which is critical for fine-grained image editing. 4M (27) trains an autoregressive generalist model using all conditioning variables jointly. However, it is not modular, requires large paired datasets to train and does not support continual learning of new classes. FG-DMs more closely related to methods like Semantic Bottleneck GAN (2, Make-a-Scene (12) and Semantic Palette (21), which model the joint distribution as the composition of a semantic map distribution P and a conditional model P(xly) for generating image x given synthesized layout y. "Make a Scene" (12) learns a VAE from segmentation maps and samples segmentation tokens from its latent space. A transformer then combines those with tokens derived from text, to synthesize an image. All joint models above have important limitations: they are trained from scratch, only consider semantic (discrete) conditioning variables and do not scale the generation of images of complex natural scenes, involving a large number of semantic classes (32), such as those in COCO without access to large-scale datasets.
attempt to improve prompt compliance by optimizing the noise latent at each diffusion iteration with loss that maximizes attention to each noun or the binding between prompt attributes and nouns. However, these methods time-consuming. require careful hyperparameter fine-tuning and do not work well for multiple object scenes. The proposed FG-DMs build on the power of VC-DMs and rely on SBPC, which samples various images and selects the one most compliant with the prompt. Since this only requires the synthesis of conditions, it can be done efficiently even for complex scenes.
Inference-Based Prompt Compliance (IBPC) methods 34 40)
3
The Factor-Graph Diffusion Model
3.1
Diffusion Models
probabilistic models based Markov chains. In the forward direction, white Gaussian noise is recursively added to image x, according to
DMs: DMs 44 15)
Z; Vaizo VI ane,, & (0,,1)
(I)

---

Cemo on beriggmed
Shared Eoander
Adudes
Control Net
Sogmeraator
Imags
[bpticel)
Stared Decodar
Frozon
256 256
Training
512 512
Resible Inference
Figure 4: Left:: Training of FG-DM for distribution P(x. y²ly²) image x, segmentation mask y² and pose map y', given text prompt y². Each factor (conditional probability written at top of each figure) implemented by adapting pretrained SD) model generate visual condition. The SD model is frozen and only small adapter is learned per factor. The final (image generation) factor adaptation. The encoder-decoder pair and SD backbone are shared among factors, reducing total number of parameters. Conditional generation chains are trained at lower resolution for better inference throughput. Right: The FG-DM offers ficxible inference framework due to classifier free guidance training, where only a desired subset of the factors run, shown the highlighted green area.
ControlNet without
where 2a / is the normal distribution, I the identity matrix, 04 IT 3x), and Br pre-specified variance. In the reverse process, a neural network co (z,, t) recurrently denoises z, to recover This network is trained to predict noise ct. by minimizing the risk defined by the loss = ||e, - (2, t)l Samples are generated with z = f(z,, Co(Z,, ((2z;,)) t)) where
3e _ + to5. v1 00
f(z,, co)
Zt
(2)
va
- (NO.)) NOO, The network eg (zc. t) is usually a U-Net (42 with attention (47
3.2 The FG-DM model
The FG-DM is a conceptually simple generalization the DM to support conditioning variables y'. Rather than the conditional distribution P(x((y*) it models the joint P(x, {y')). Prior joint models (32) (24) use single joint denoising U-Net in the pixel or latent space to jointly synthesize and { J. This limits scalability multiple conditions, increases the difficulty of editing the
synthesized x, and requires retraining the entire model to add new conditions y. The FG-DM instead leverages the decomposition of the joint into a factor graph (II composed by sequence of conditional distributions, or factors, according to
P(x, (y')) = P(x (y') K P(y
P(y*))
(3)
where there are usually conditional independence relations that simplify the terms the right hand side. In any case, (3) enables the implementation of the joint DM as a modular composition of conditional DMs. This is illustrated in Figure which shows the FG-DM discussed in Figure !I
Synthesis conditioning variables. We convert conditioning variables 3-channel inputs and use the pre-trained E-D pair from the SD model to map them into the latent codes for efficient training. See appendix section I.I.I for more details on this process.
Sampling: The FG-DM samples y and x as follows. Let eœ= be a DM for P(x(yy')), and egi DM for P'y"ly .......................... ). In the forward direction, 2 and zi. the noisy versions of and y respectively, are generated by using (I) with z x and zi -y" In the reverse process, each denoising step is implemented with
z
K
f(zl' cax (zt .t))
(4)
z f(z), eos (z) (z)..... ... K t)), vi K
(5)
2-1 = f(z,, (2) ........ K
(6)
where f(.) is the recursion of 2 All conditions sampled at each denoising step.
5

---

3.3 Adaptation of pretrained DM
Architecture: To adapt a pretrained SD model into a FG-DM factor, we modify the T2I-Adapter 29 to be conditioned on the current timestep t and use encoded latent features of the condition(s) from previous factors as imput to the adapter of the current factor. Figure Siguresshowe shows how the conditioning of 64 is implemented: noisy latent feature zi is fed to the first adapter, and the denoised latents z of each VC-DM are fed to the adapters of the subsequent VC-DMs (denoising of z k < i). The adapter consists four feature extraction blocks with one convolution layer and two timestep-residual blocks for each scale. The encoder features Fir at the output of U-Net block are modulated as 2, 3, 3.4.. where FT, are the features produced by an adapter branch
fre = f.t. + P E 2,, associated with F,t at timestep The SD model is kept frozen, only the adapter branches learned per factor model.
Training with Classifier-Free Guidance (CFG): Given training example with all the conditioning variables (x,.y..... y)....,yf ), we randomly select ... K - conditioning variables as null condition for CFG, 20% of the training time. This facilitates unconditional training of each factor model which supports flexible inference. As a result, only a desired subset of the conditions of the FG-DM are run, as illustrated in the right of Figure (highlighted in green).
Attention Distillation. The synthesis of conditions like semantic or normal maps requires learning to precisely associate spatial regions with object identities geometry, which is difficult task. To encourage the binding these properties across conditions and image, we ground them on the SD attention maps, which are known encode prompt semanties (33). The intuition is that the attention dot dot-products between word tokens and image regions should remain approximately constant for all factor models. Ideally, given text prompt word should elicit similar region of attention in the VC-DM that synthesizes each condition. To encourage this, we introduce an attention distillation loss, based the KI divergence between the self and cross attention maps of the pretrained SI) model and those of the adapted models. This can be viewed as using the pretrained DM as a teacher that distills the knowledge contained in the attention maps the adapted student. This distillation also helps the adapted model retain the generalization to unseen text prompts, such as the examples of Fig. 1 Formally, the attention distillation loss is defined
ES 9. (F QQ KC) ES (P KC) (7)
(x) > KL (F ()) (1(F5(99) (3) Se(0.))
> Σ au(FL K)) SE(0.2)
where superscript (s) denotes SD-teacher (SD-student), g implements bilinear interpolation needed to upscale all maps to a common size, F is the softmax of the products between query noise feature matrix (Q) and key CLIP text feature (K), Lj number of attention layers, and index j 0, 1 denotes self and cross attention layers respectively. The overall loss is the sum of distillation losses between teacher t (pre-trained SD model) and all students LKL = E (KL(f", f''), where s' is the student model adapted to condition y". For multi-condition FG-DMs, the distillation loss is only required for the synthesis of the first condition, which is conditioned by text alone. For the subsequent factors, which are already conditioned by visual condition (e.g. pose conditioned by segmentation Figure 4 the attention distillation loss not needed.
Loss: Since the noise introduced in the different factors is independent, the networks are optimized to minimize the risk defined by the loss
K Σ le cb(
LPG =l|e" - (z] 2) ......... 12
, + AKICKL
(8)
Training from scratch: The FG-DM can also be trained from scratch by simply concatenating the latent representation of the previous condition(s) and noise to generate the next condition instead of using adapters. Please refer to the appendix section 1.1.2 for detailed discussion.
Experimental Results
Datasets and models: We consider four conditioning variables in this paper: segmentation, depth, normal and sketch maps. The pretrained SD v1.4 model adapted using the COCo-WholeBody dataset 23 18), with 256 input resolution, to train all condition factors. Groundtruth (GT) is as

---

'An of a small rhino"
'An Image of a giant squirrel
"A eating hotdog"
"A picture of a volcanic erustion"
"A picture of a surgod"
'A pengui n riding bicycle"
man with a hat and glasses
"A bird with headphooes singing in front microphane"
"An older woman sttting a window a train."
with fireplace and a television"
Figure 5: More qualitative results of FG-DM to synthesize segmentation, depth, normal and sketch maps and their corresponding images. See appendix Figure. 14 the higher resolution version.
follows: COCO GT segmentations, HED soft edge (5I) for sketch maps, and off-the-shelf MIDAS (39 B) detector for depth and normal maps. We also present results for an FG-DM trained from scratch on LMM-ClobA.MassHQ (22), and for other datasets in appendix, where implementation details are also given.
Evaluation: Visual quality is evaluated with Frechet Inception Distance (FID) (4), image diversity with LPIPS (59). We also report the Precision/Recit Values of (20) and evaluate prompt alignment with CLIP score (CLIP-s) (36). All speeds are reported using a NVIDIA A-1 GPU.
Qualitative and Quantitative results: Figure 5 shows additional qualitative results of synthesized segmentation (columns 1-3), depth (columns 4-6), normal (columns 7-8), sketch (columns 9-10) maps and their corresponding images for the prompts shown below each image. The FG-DM leverages the generalization of the pre-trained SD model synthesize segmentation maps for object classes such as rhino and squirrel, beyond the training set (COCO). The semantic maps are colored with unique colors, allowing the easy extraction of both object masks and class labels. This shows the potential of the FG-DM for open-set segmentation, e.g the synthesis of training data for segmentation models. Note that these results demonstrate a waauble-genrrllzzttion never trained squirrels or rhinos, SD was never trained to produce segmentation masks. However, FG-DM adapted from SD produces segmentations for squirrels and rhinos. The fourth column shows the depth map and image synthesized for the prompt "A picture of volcanic eruption". In the fifth column the same caption is used to create the depth map, while the image is created with the prompt "A picture of sungod". This shows that even when there is mismatch between the prompts used for different factors, the FG-DM is able to produce meaningful images. This is a benefit of the FG-DM modularity. 5 and four FG-DM models. Despite using a bigger model and training from scratch, 4M-XL model generates lower quality images than SD and FG-DM as seen from the higher FID score. The FG-DM has higher image quality than SD for the segmentation and normal map conditions and higher clip score for sketch condition showing the effectiveness of adaptation.
ability. While the FG-DM was
the ext-to-image synthesis quality of SD, 4M-XL (28)
We conducted a human evaluation to compare the qualitative performance of the FG- DM (adapted from SD) with N = 1 to the conventional combination of SD+CEM, where CEM is an external condition extraction model (CEM). for both segmentation and depth conditions. We collected 51 unique prompts, composed by random subset of COCO validation prompts and subset of creative examples. We sampled 51 dimage,condition pairs 35 pairs of (image,depth map), 16 pairs of (image,segmentation map)- using the FG-DM. For SD+CEM, images were sampled with SD for the same prompts, and fed to CEM implemented with MIDAS B for depth and OpenSeed (57) for segmentation. The study was performed on Amazon Mechanical Turk, using 10 unique expert human evaluators per image. These were asked to compare the quality of the pairs produced by the two approaches and vote for the best result in terms of prompt alignment and visual appeal. Table T shoows that evaluators found FG-DM generated images (masks) to have higher quality 61. 61.77% (63.13% 13%) ) and better prompt alignment 57. 37.68% (60.98%) of the time. These results show that the FG-DM produces images and masks that have higher quality and better prompt alignment.
User Study:
Qualitative Image Editing Results: Figure shows additional examples of synthetic image editing using the FG-DM. Diffusion models have well known difficulties to perform operations such as switching people's locations (5) or synthesizing images with text (26) 46). The first four columns show that the FG-DM is a viable solution to these problems. The first column shows the image
7

---

Table I: User study on the qualitative preference of images/condition pairs generated by the FG-DM and SD+CEM, using 10 unique human evaluators. A. de- notes (prompt) Adherence and Q. denotes Quality.
Table 2: Ablation of attention distillation loss for T21
synthesis on COCO validation for FG-DM. FID re- ported for Images/Conditions
[Dsiil|]
FID
| |R||//
|
Model
|mng A.t/Cond A. [lng Q-7 [Cund Q.t
Q.7[Cond
Seg
20.9/164.8
0.544
28.5
No clear winner|
4.11
4.70
2.90
3,33
Seg
19.6/19922
0.56
28.5
SD&CEM
37,84
34.31
35,49
33.52
Normal
66667
0.56 0.49
28.6
FG-DM
57.84
60.98
61.37
63.13
Normal
<
19,30/12399 0..59 0.55
28.7
Table 3: Object recall statisties for sampling FG-DM with different seeds and timesteps on the ADE20K validation prompts.
Table 4: Quantitative comparison of Ohject Recall for different models and configurations on the ADE20K validation set prompts.
Samples/ Ratch
Arg. Min.
Avg. Max. f Avg. Med. 1 Avg. # Imgs Time
Model
Clip S1 Avy. Recall I Time (s) +
Recall (%)
Recall
Recall (%)
0.75 0.9
SD (N=1,=200)
0,301
59.8
3.25
1-10,=11
69
0.1
(N=1,1=20)
63.6
1=20NN=1
68
68
68
0.9
0.4
0.1
0.81
FG-DM (N- 1,-2)
0.295
67.8
4.81
110.N=5
60
74
70
4.5
19
0.6
1.10
SD (N=10.220)
0.301
75.1
23.0
=10.N=10
55
70
9.0
1.2
1.75
FG-DM (N=10,=10)
0.294
75.1
5.75 (4x.)
=02
55
75
70
8.9
3.6
1.1
3.3
PG-DM (N=10,=20)
0.296
75.0
7.33 (3x.))
Generated
Edited
Generated
Edited
Edited
Edited
Generated
Generated
"Two men shaking
"Two men facing away hands" from one other"
"a dog wearing hoodle saying DIFF
"An image of a giant squirrel beside small <hino
Stablc Diffusion v1.4
Stable
Diffusion
v1.5
Figure 6: Examples of images generated by FG-DM after editing and comparison with popular text-to- image models. Editing is shown for flipping persons (columns 1-2). writing the desired text (columns 3-4) or realizing a difficult prompt (columns 5-6). Images generated by Stable Diffusion v1.4 and v1.5 for the same prompt are shown in the last two columns.
Table 5: Quantitative results of Text-to- Texaeeo-Image synthe- sis COCO for FG-DM with segmentation, depth, sketch and normal conditions.
Model
FID
4 P R f CLLIPss
SD
20.5
0.50
0.69
28.5
4M-XI. Sey
39.1
0.49
0.30
28.8
Figure 7: Attribute recall verification with FG-DM on MM-CelebA-HQ, Left: Semantic Attribute Recall. Right: Histogram of the number of trials needed to reach the specified recall.
FG-DM (G-DM Seg
19.6
PG DM Depth
27.0
0.45 0.52
27.6
FG-DM Sketch
25.3
0.39
0.61
30.3
FG DM Normal
19.3
0.59
(0.5
28.7
generated by FG-DM for two shaking hands while the second shows the edited version where the two men are flipped, so as to face away from each other, and combined with a different background. The 3rd and 4th columns show example where the user scribbles "DIFF" in the synthesized sketch map, which is then propagated to the image. The last four columns show examples of a difficult prompt, "An image of a giant squirrel beside a small rhino", ". unlikely to be found on the web, for which existing T21 Models (SD) v1.4/1.5) fail (columns 7-8). The FG-DM generates meaningful images (5th and 6th column) by simply editing the masks sampled by the segmentation factor, as discussed in Figure I In this example, the animal regions shown in Figure 5 (columns 1-2) were resized, flipped and pasted onto a common segmentation. Note how FG-DM allows precise control over object positions and orientations, which differ from those of Figure 5
SBPC (Object Recall): Table 3, ablates the influence of noise seeds (batch size N) and number of sampling timesteps t on the prompt adherence (measured by object recall) of images synthesized by the FG-DM for ADE20K validation prompts (8.5 objects per prompt on average). Avg.
8

---

Max Recall is the average, over the 2000 prompts, of the maximum per image recall in each batch. Similar definitions hold for Avg.Min and Avg. Median. The top part of the table (columns 2-4) shows that for N = 1 recall saturates at 69% for /. = 10 timesteps. The bottom part shows that increasing N maintains the Avg. Median recall at this value but produces images with a significant variation of recall values. The Avg. Max recall is 6 points higher (75%) and fairly stable across configurations of N and t. The fifth column shows the number of images in a batch that satisfy the object recall thresholds of 0.5,0.75 and 0.9, averaged over 2000 prompts. While this number decreases for higher thresholds, a batch size of N = 10 can produce at least one image with even higher prompt compliance, on average.
The FG-DM is particularly well suited to implement SBPC, because recall can be computed as soon as segmentations are sampled. Since the segmentation factor can be run with a smaller number of timesteps and at lower resolution, this is much faster than sampling the images themselves. The image synthesis factor is run only once, for the segmentation mask of highest recall. On the contrary, an SDD-bssed implementation of SBPC requires synthesizing N images and then feeding them to an external segmentation model (we used Segformer-B4 (50) in our experients) to compute recall. Table 4 commpree the object recall of different sampling configurations of SD and FG-DM for ADE20K validation prompts. The objects in the groundtruth masks are considered to compute recall. In appendix section A.2.1 we show that the object classes can be automatically extracted from the prompt using an LLM. The top part of the table compares single run (N 1) implementations of SBPC by SD and the FG-DM to the popular IBPC Attend & Excite (A-E) method (7). The table shows that the FG-DM images have 8% and 4% higher recall than those of SD and A-E respectively, even though SD has higher clip score. This illustrates the deficiencies of clip score to evaluate prompt adherence. The IBPC method underperforms the FG-DM by 4 points while drastically increasing inference time to 36 seconds per prompt. This is because A-E computation scales linearly with the number of objects and it fails in multiple object scenarios. While SBPC achieves good results for N N=1 1, the bottom half of the table shows that its true potential is unlocked by larger batch sizes. Both the SD and FG-DM implementations of SBPC achieved the much higher average recall of 75%. However. for SD, the sampling of a batch of N = 10 high resolution images requires an amount of computation (23 seconds per prompt) prohibitive for most applications. The FG-DM is 4x (3x) faster when using 10 (20) DDIM steps for the segmentation mask and 20 steps for image synthesis.
Besides object recall, the FG-DM can also improve the recall of semantic attributes. We illustrate this with an experiment in face synthesis for FG-DM trained from scratch on MM-CelebA-HQ(4)) We compute the recall of semantic attributes such as bald, necklace, and pale skin etc. using the generated segmentation masks on validation prompts. Fig. 7 (left) compares the attribute recall of FG-DM to those of prior methods for N 1. The FG-DM has average recall of 75% over all semantic attributes, outperforming competing methods (32) Figure 7 (right) shows an histogram of the number of trials (N) required by the FG-DM achieve 70% and 80% recal!" The FG-DM generates x 90% of its samples with the specified 70% recall in five trials.
Attention distillation loss: Table 2 ablates the adaptation of SD with and without attention distilla-
tion loss. The FG-DM with attention distillation improves the image quality by 1.29/1.3 points for segmentationnnorrall map conditions respectively. The effect is more pronounced when comparing the fidelity of the conditions which improves the quality by 5.6/2.8 points for segmentation/normal synthesis showing the effectiveness of the loss in adapting SD to different conditions. Appendix Figure 9 shows the qualitative comparison of ablating the attention distillation loss.
Real Image Editing with FG-DM: We show the results of editing of both real images and their segmentation masks with FG-DM. The top of Figure & refers to inversion of the segmentation mask. We use aff-the-shelf OpenSEED (57) model to extract the segmentation map of real image (shown on the bottom left of the figure) and apply the FG-DM segmentation factor model for inversion and editing using LEDITS+ (4) recent method for text based image editing. We apply LEDITS+ to the segmentation factor to 1) replace the mask of the woman by that of a chimp (third image of the top row) and 2) to delete the mask of the umbrella (fifth image). New images (fourth and sixth) then generated by the image synthesis factor conditioned on the edited segmentation masks. We have found that the inversion and editing of segmentation masks is quite robust. The synthesized masks usually reflect the desired edits. However, because the final image synthesis is only conditioned on these masks, the synthesized image does not maintain the background of the original image. The
'10 runs includes samples that required 10 more trials.

---

synthesized image is replica of the original image at the semantic level (similar objects and layout) but not at the pixel level. From our experiments. this method has high robustness and quality for semantic-level editing.
We next investigated pixel level inversion and editing, which is harder. The bottom part of Figure 8 shows the comparison of LEDITS+ editing with inversion by SD and by the image synthesis factor of the FG-DM. For the latter, we apply inversion to the ControlNet image generation factor using the real image and the segmentation mask extracted from it. Then we perform the LEDITS++ edit using the edited mask from the top part of Figure (inverted with the FG-DM segmentation factor) to produce the edited image shown in columns 4 and 5. This pixel-level inversion and editing tends maintain the background of the original image but is much less robust than mask- editing in terms of editing quality. This can be seen from the images in columns 2 and 3, which show the inversion using SD, which fails to produce a realistic chimp and turns the woman into a stone sculpture. The FG-DM produces much more meaningful edits, shown in columns 4 and 5. The last column of the bottom part of the Figure [ shows an added advantage of FG-DM where the chimp generated in the top portion can be pasted to the original image due to availability of the segmentation mask. In this example the pasting rough around the object edges since we have made no attempts to beautify it. It can be improved by denoising the generated image with one forward pass of SD at higher timestep.
Image
chimp
ching umbrolla umbretla
woman swimsutt
Inventon with Stable Diffuston
Inversion with RG-DM DM
Real Image
'+' Edit chimp
Edit: umbrella
' Edit chimp
*/ umbrelll
Paste edit with mask
Figure 8: Top: Inverting segmentation masks with FG-DM segmentation factor using the LEDITS+ method. Edits to replace the woman by chimp or eliminate the umbrella. The FG-DM enbables text-based edits to modify delete objects in given mask. The image generated with the edited mask as condition is shown to the right of each edited masks. Bottom: Original image, LEDITS+ edited image for stable diffusion and the image synthesis factor of the FG-DM. Please Zoom in for details.
5 Limitations, Future Work and Conclusion
Although, the FG-DM uses low resolution synthesis for the conditions, runtime increases for chains with more than two factors. Since, the attention maps generated per factor must be consistent according to the joint model, sharing them across different factors and timesteps direction for further reducing the runtime. Furthermore, while the FG-DM allows easier control over generated images for operations like deleting, moving, or flipping objects, fine-grained manipulations (e.g. changing the branches of tree) can still require considerable user effort. Automating the pipeline by using an LLM or methods like Instructpix2pix (5) to instruct the edits of the synthesized conditions is another interesting research direction. See A.3 for discussion broader impact.
promising
In this work, we proposed the FG-DM framework for efficiently adapting SD for improved prompt compliance and controllable image synthesis. We showed that an FG-DM trained with relatively small datasets generalizes to prompts beyond these datasets, supports fine- fine-grained image editing, enables improved prompt compliance by SBPC, allows adding new conditions without having to retrain existing ones, and supports data augmentation for training downstream models. was also shown that the FG-DM enables faster and creative image synthesis, which can be tedious or impossible with existing conditional image synthesis models. Due to this, we believe that the FG-DM is a highly flexible, modular and useful framework for various image synthesis applications.
10

---

Acknowledgements
This work was partially funded by the NSF award 118-2303153. We also acknowledge and thank the use of the Nautilus platform for some of the experiments discussed above.
References
Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., Yin, X.: Spatext: Spatio-textual representation for controllable image generation, Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 18370-18380 (June 2023)
[2] Azadi, S., Tschannen, M., Tzeng, E., Gelly, S., Darrell, T., Lucic, M.: Semantic bottleneck seene generation. arXiv preprint 19XXv:1911.113555 (2019)
[3] Birkl, R. Wofk, D., Müller, M.: Midas v3.1 model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307. 14460 (2023)
[4] Brack, M., Friedrich, Kormeeier, K., Tsaban, L., Schramowski, P., Kersting, K., Passos, A.: Ledits++: Limitless image editing using text-to-image models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
[5] Brooks. T.. Holynski, A., Elros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023)
Casanova, A., Careil, M., Romero-Soriano, A., Pal, C.J., Verbeek, J., Drozdzal, M.: Controllable image generation via collage representations. arXiv preprint arXiv:12304.1377 (2023)
[7] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, I.., Cohen-Or, D.: Attend-andeexcit:: Attention- -hased semantic guidance for text- text-to-image imme diffusion models. SIGGRAPH (2023)
[8] Chen, L.C., Papandroou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40(4), 834-848 (2017)
Cheng, J., Liang, X., Shi, X., He, T., Xiao, T., Li, M.: Layoutdiffuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv22022008000 (2023)
[10] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances Neural Information
Processing Systems 34, 8780-8794 (2021)
[11] Forney G.D.; Codes on graphs: Normal realizations. IEEE Transactions on Information Theory 47(2). 520-548 (2001)
[12] Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D.. Taigman, Y.: Make-a-scene: Scene-based text-to- image generation with human priors. In: Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XV. p. 19-106. Springer- Verlag, Berlin, Hcidelberg (2022), http:: wwg/o0.//110 1000/0700303000 0-01---33-3301-33333- - 19784-0_6
[13] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen -r., D.: Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01 626 (2022)
[14] Heusel. M., Ramsauer, H., Unterthiner, T.. Nessler, B., Hochreiter, S.: Gans trained by time-scale update rule converge to local nash equilibrium. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), //p.ceede// neur cc/paper..........pppo ffmerr_fill files/ieeppeell /appe////////////e////eeeaeaa//e
[15] Ho. J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 6840-6851. Curran Associates, Inc. http00.
wwwpiinnnnnn.uureeeeeeeeeeeeeeee nwwrinu..ccccom hcuripp.cccpaper. cccppppprrffllssss__eeeeeeesssee files/ pdf
[16] Ho, J., Jain, A., Abbcel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840-6851 (2020)
[17] Ho, J.. Salimans, T.: Classifier-free diffusion guidance (2022)

---

[18] Jin. S.. Xu, L.. Xu. J., Wang, C., Liu. W., Qian, C., Ouyang. W., Luo, P:: Whole-body human pose estimation in the wild. In: Proceedings of the European Conference Computer Vision (ECCV) (2020)
[19] Ju. X., Zeng. A., Zhao, C.. Wang. J., Zhang, L., Xu, Q. HumanSD: A native sicleton-guides diffusion model for human image generation. arXiv preprint a1Xiv:1904.06533. (2023)
[20] Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved precision and recall metric for assessing generative models. CoRR abs/1904.06991 (2019)
[21] Moing, G., Vu, T.H., Jain, H., Pérez, P., Cord, M.: Semantic palette: Guiding scene generation with class proportions. In: CVPR (2021)
[22] Lee. C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: Towards diverse and interactive facial image manipulation (2020)
[23] Lin, T., Maire, M.. Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Doll'a r. P.. Zitmick. C.L.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014),
http://arxia: /ttp://aavvvv... wwtp://..rg/bbba//eeee 1405. 0312
[24] Liu, X., Ren, Siarohin, A., Skorokhodov, L,, Li, Y., Lin, D., Liu, X., Liu, Z., Tulyakov, S.: Hyperhuman: Hyper-realistic human generation with latent structural diffusion. arXiv preprint arXiv:2310.08579 (2023)
[25] Luo, X., Goebel, M., Barshan, E., Yang, F:: Leca: A learned approach for efficient cover-agnostic watermarking (2022)
[26] Mirjalili, S.: if-ai- inrgeggaaagggnnoattoor are -re---a--------t-- so-s--o-- -he------------sugllleee wortige-----tit-addddeed write- (2023), https
//theconversation. www.coooo///... al 1aaa--maae---- gengeratrner are so smart smart-why-do do- stres-d-othh--uugggglll
[27] Mizrahi, D., Bachmann, R., Kar, O.F., Yeo, T., Gao, M., Dehghan, A., Zamir, A.: 4m: Massively
multimodal masked modeling. In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https:// http:/ppwerereeeeeeeeee////rrra/ www.te/eoorrurrdddooollllllllll
[28] Mizrahi, D., Bachmann, R., Kar, O.F., Yeo, T., Gao, M., Dehghan, A., Zamir, A.: 4M: Massively multimodal masked modeling. Advances in Neural Information Processing Systems (2023)
[29] Mou, C., Wang, X., Xie, L.. Wu, Y., Zhang, J., Qi. Z., Shan, Y., Qie, X.: T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08455 (2023)
[30] Nichol, A.Q., Dhariwal, P:: Improved denoising diffusion probabilistic models. In: International Conference Machine Learning. pp. 8 8162-8171. PMLR (2021)
[31] Obukhov, A.. Seitzer, M., Wu, P.W.. Zhydenko, S., Kyl. J. Lin, E.Y.J: High-fidelig performance metrics for generative models in pytorch (2020).
wtt.://rg/...2/1/eeoooo...990 495.:m3......7777...... http::ttttt///tttt com/ foshas/torch-fidellry version: 0.3.0, DOI: 10.5281/2encd0049957733
[32] Park, M., Yun, Choi, S., Choo, J.: Leamning to generate semantic layouts for higher text-image correspondence in text-to-inne toaattg--imeeeeee synthesis. ICCV (2023)
[33] Peng, Y., Zhao, C., Xic, H., Fukusato, T., Miyata, K.: Difffacesketch: High-fidelity face image synthesis with sketch-guided latent diffusion model. arXiv preprint arXiv:2302.06900 (2023)
[34] Phung, Q., Ge, S., Huang, J.B.: Grounded text-to-image synthesis with attention refocusing, In: CVPR (2024)
[35] Poole, B., Jain, A., Barron. J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffussion. arXiv (2022)
[36] Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.. et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763 (2021)
[37] Ramesh, A., Dhariwal, Nichol, A., Chu, C., Chen, M.: Hicrarchical text-conditional image generation with clip latents. ArXiv abs/2204.06125 (2022). httppp///pp..eenattttttttttrrrrr 248097655
[38] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, L: Zero-shot eot-to-image generation. In: International Conference on Machine Learning. pp. 8821-883 PMLR (2021)
12

---

[39] Ranftl, R., Lasinger, K.. Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions Pattern Analysis and Machine Intelligence 44(3) (2022)
[40] Rassin, R., Hirsch, E., Glickman, D., Ravfogel. S.. Goldberg, Y., Chechik, G.: Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems 36 (2024)
[41] Rombach, R. Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684-10695 (2022)
[42] Ronneberger, O., Fischer, P., Brox, T:: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assister Intervention-MM -MICCCI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234-241. Springer (2015)
[43] Sagonas, C., Tzimiropoulos, G., 7afciriou, S., Pantic, M.: 300 faces in-the-wild challenge: The first facial landmark localization challenge. In: 2013 IEEE International Conference on Computer Vision Workshops. pp. 397-403 (2013). https://doi.org//000 109/CCVW.2013.59
[44] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium dhermodynammies. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings Machine Learning Research, vol. 37, pp. 2256-2265 PMLR, Lille, France (07-09 Jul 2015)
[45] Song, J., Meng, C., Ermon, S.: Denoising difftusion implicit models. In: International Conference on Learning
[46] Truly, A.:
(2023) (2022222ats33 http://wt..pp/p//////ataaattaaaa wigitaltrendds com/ computing/5 Somputann/5-thing---aa htaaaa-mmmneeeeeerrttrr-----t---
[47] Vaswani, A., Shazeer. N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A.N., Kaiser, L.,, Polosukhin, Attention is all you need. Advances in neural information processing systems 30 (2017)
[48] Wang. J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Liu, D., Mu, Y., Tan, M., Wang. X., Liu, W.,
Xiao, B.: Deep tigh-rosollttin reprosentation learning for visual recognition. TPAMI (2019)
[49] Xia, W., Yang, Y., Xue, J.H., Wu, B.: Tedigan: Text-guided diverse face image generation and manipulation.
In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[50] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P:: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Neural Information Processing Systems (NeurIPS) (2021)
[51] "Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of IEEE International Conference on Computer Vision (2015)
[52] Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., Wen, F: Paint by example: Exemplar- hased image editing with diffusion models, arXiv preprint arXiv:2211.13227 (2022)
[53] Yang, L., Huang, Z., Song, Y., Hong, Li, G., Zhang, W., Cui, B., Ghanem, B., Yang, M.H.: Diffusion-based scene graph to image generation with masked contrastive pre-training. arXiv preprint 3XXx:2211.111333 (2022)
[54] Yu, C., Wang, J., Peng, C., Gao. C., Yu, G., Sang, N.: Bisenet: Bilateral segmentation network for real- time semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
[55] Yu. F., Koltun, V. Funkhouser, T.: Dilated residual networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
[56] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang. H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to- image generation. Transactions Machine Learning Research (2022), https://openreviiee wwwp:/ewwwww...weeeeeeeeeeeeee AFF.//rr featured Certification
[57] Zhang, H., Li, F., Zou, X., Liu, S., Li, C., Gao, J., Yang, J., Zhang, L.: A simple framework for open- vocabulary segmentation and detection. arXiv preprint ar XXxx:2303.0813 (2023)
[58] Zhang, L., Agrawala, M.: Adding conditional control text-to-image diffusion models. ICCV (2023)
13

---

[59] Zhang, R., Isola, P. Efros, A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features perceptual metric. CVPR (01 2018)
[60] Zhao, S., Chen, D., Chen, Y.C., Bao, J., Hao, S., Yuan, L.,, Wong, K.Y.K: Uni-controlnet: All-in-one control to text-to-image difffusion models. Advances in Neural Information Processing Systems (2023)
[61] Zhou, B., Zhao, H., Puig. X., Fidler, S., Barriuso, A., Torralba, A.; Scene parsing through ade20k dataset. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (2017)
14

---

A
Appendix
"A man is in kitchen making pizzas."
"a person with a shopping cart on city street"
"Two zebras standing in a rocky area in the wild."
4M-UL [compatiion)
Withoutt distilation
With dienilition
Figure 9: Comparison of FG-DM with attention distill loss (bottom) against FG-DM without attention distill loss (center) and the recent 4M-XL model (top) for prompts shown at top the figure. The two versions FG-DM use the same seed. Both versions of the FG-DM produce images higher quality than 4M-XI. Attention distillation helps improve the quality of the generated segmentations. For example, the model without distillation has inaccurate masks/missing cart/less realistic zebras from left to right.
Image hyena"
picture teddy bear wearing poliic
realistic picture pack of wolves"
realistic picture of recroon reading a
"A picture of bird singing microphone"
"A picture surfing"
Figure 10: Qualitative results of Depth map/Image pairs synthesized by FG-DM.
A.1
Experimental Results
A.1.1 FG-DMs Adapted from Stable Diffusion (SD)
We start by the discussion of additional results for FG-DMs obtained by adaptation of Stable Diffusion model, shown in Figure of the paper.
Qualitative comparison of attention distillation loss
Figure 9 shows some qualitative results of the ablation for the impact of the attention distillation loss. There clear qualitative benefit in introducing this loss. Without the model generates less accurate masks, leading to an unrealistic pizza making depiction/ cart-person relationship/ zebra pair from left to right. This confirms the qualitative ablation showing the benefits of the attention distillation loss in Table but provides a stronger illustration of the advantages of the loss, which
15

---

picture point hear"
"A vingin many holding an orb of light"
"Apicture of blue elephant"
"A counter and rable chairs."
"Closeup of plate with chickan, mushrooms and broccoll."
large airport rurwayy"
Figure II: Qualitative results of Normal map/image and Sketch map/image pairs synthesized by FG-DM. FG- DM generalizes well across conditions and able to generate condition image pairs that not seen during training.
"a standing kitchen"
"a person and white chimp posing"
"A plate waffle, somr fruit and ice cream it"
"A bost the with lighthouse background"
bride and the beckdrop mountains"
"A small bathroom with a sink and toilet"
Figure 12: Qualitative results of Segmentation map/image pairs synthesized by FG-DM. The second column shows the benefit of explainability with FG-DM which allows verifying intermediate conditions understand the hallucinations (mixup of chimp and person) of SD which are opaque otherwise. Here, FG-DM correctly generates the chimp and the person mask with different colors while ControlNet confuses between the two showing that the ControlNct needs to be corrected.
tends to produce more "plausible" scenes. Such plausibility is difficult to measure with qualitative metrics. For example, the CLIP score is not sensitive the fact that the cart and the person interacting in a normal way, or that the pizza making activity is unrealistic.
We also compared with the recent 4M model (28), autoregressive model trained from scratch both discriminative and generative tasks. In this case, we use the largest model (4X-ML) released by the authors. Figure shows qualitative comparison between FG-DM and 4M. It can be seen that 4M generates images of weaker quality (distorted hands, missing person's 's head, deformed zebra bodies) as compared to FG-DM with/without the attention distillation loss.
Additional Qualitative Results for Segmentation, Depth, Normal and Sketch conditions
Figure I0 shows qualitative results of synthesized depth maps and images for the creative prompts shown on top/bottom of each image. The FG-DM framework is able to generate high quality images and normal maps for prompts that are not seen in the training setleveraging the generalization of SD.
Figure TT (first three columns) shows qualitative results of synthesized normal maps and images for the creative prompts shown on top/bottom of each image. Figure [T (last three columns) shows qualitative results of synthesized sketch maps and images for the creative prompts shown on top/bottom of each image.
16

---

Inputs
MAS
SpaText
FG-DM
FG-DM
mug and
white plate
with cookies on table
black
elephant near lake
bathroom
with artificial light, a mirror, white sink and with red flowers
Figure 13: Qualitative comparison of FG-DM with prior works such as Make-a-Scene and SpaText for the prompts shown on the left. Note that FG-DM generates both the map and the image while for MAS and SpaText, the condition manually sketched and fed the model.
Figure 12 shows qualitative results of synthesized segmentation maps and images for the creative prompts shown on top/bottom of each image. As shown in the main paper, the FG-DM is able to synthesize segmentation maps for object classes beyond the training set and the semantic maps are colored with unique colors, allowing the easy extraction of both object masks and class labels. This shows the potential of the FG-DM for open-set segmentation, e.g the synthesis of training sets to generate training data for segmentation models. Further, a number of interesting generalization properties emerge. Although the FG-DM only trained associate persons with black semantic maps segments, it also assigns the chimp of Figure [ class that it was not trained on, to that color. This shows that FG-DM can integrate the prior knowledge by SD that "chimps and persons are similar" into the segmentation task, which receives supervision from COCO alone. Conversely, the similarity between chimps and people might induce SD to synthesize chimp in response to a prompt for people, vice-versa. This is shown in the bottom left of Figure 12 where the FG-DM correctly synthesizes different colors for the chimp and the person, but the ControlNct fails. While the black box nature of SD makes these errors opaque, the FG-DM allows inspection of the intermediate conditions to understand these hallucinations and make corrections accordingly. For example, the above example ControlNet has be corrected by either finetuning using inference optimization methods like A-E (7). This illustrates its benefits in terms of explainability.
Comparison with prior works
Figure 13 shows the qualitative comparison of FG-DM with prior works such as Make-a-Scene (12) or SpaText (I) in addition 4M model compared Figure 9 Note that FG-DM generates both the segmentation and the image while for the other methods, it is manually sketched and fed to them. It seen that FG-DM generates high quality images that adhere well to the prompts as compared the prior works.
Comparison of generated conditions by FG-DM to conditions recovered by off-the-shelf models
Figure 13 shows qualitative comparison of the conditions synthesized by FG-DMs to those recov- ered from the synthesized image using off-the-shelf pretrained models for segmentation and depth estimation. The qualitative results corroborate with the user study the generated conditions better than extracted ones for depth while they are similar for segmentation conditions.
Synthesis Conditioning Variables with SD Autoencoder
17

---

"An Image of small rhino
"An image of a giant squirrel
"A man cating hatdag
"A picture of volcanic eruption"
"A picture of sungod
"A penguin riding bicycle"
man with a hat and glasses"
"A bird with headphones singing front of microphone"
"An older sitting window of train."
"A with fireplace and television"
Figure 14: Zoomed version of Figure Qualitative results of FG-DM to synthesize segmentation, depth, normal and sketch maps and their corresponding images.
High-quality image synthesizes requires DMs trained from large datasets. A common solution is to adopt the LDM (41) architecture, where an encoder (E)-deooder (D) pair used map images into lower dimensional latent space. Using pre-trained E-D pair, e.g. from the SD model, guarantees that latent codes map into high quality images, making possible to train high-fidelity DMs with relatively small datasets. However, it is unclear that this approach will work for the synthesis of conditioning variables, such as segmentation maps, which SD is not trained on. example, (21 (32 explicitly address training DMs produce the discrete outputs required by many conditioning variables. Somewhat surprisingly, our experiments show that off-the-shelf foundation DMs are quite effective at synthesizing visual conditioning variables. All our results use the following procedure: 1) visual conditioning variables are converted to 3 channel inputs. For discrete variables, a different color is simply assigned each variable value (e.g. each semantic label of segmentation map). 2) All VC-DMs in the FG-DM re-use the pre-trained
recoder-decoded pair of SD (41), as illustrated in Figure 4 3) At the decoder output, discrete values recovered with hashmap that matches image colors to their color prototypes. This is done by simple thresholding, with a margin threshold empirically
28 for each pixel. To test this procedure, reconstruction error of the segmentation maps from 2,000 validation images of the ADE20K dataset. This was 0,0053 (normalized) highly effective in reconstructing discrete maps. Figure [6 compares the auto-encoder reconstructed maps against the groundtruth maps for different conditioning variables such as segmentation, depth, sketch and normal. This shows that simply representing the conditioning variables as 3-channel
measured the mean squared pixel
1.34 pixel error showing that the pretrained SD autoencoder is
18

---

inputs allows the faithful reconstruction of all conditions. Further, we also trained a lightweight segmentation model to recover the labels from the RGB semantic maps but observed no improvement over this simple heuristic.
( Segmentatiin maps. (Middle: Generated, Top: Extracted)
(b) Depth maps. (Middle: Generated, Top: Extracted)
Figure 15: Qualitative comparison of generated conditions for FG-DM vs extracted conditions using SD (Stable Diffusion) + CEM (Condition Extraction Model) for segmentation and depth maps. The generated conditions for depth maps are superior to the extracted ones.
Computation-Performmane Tradeoff:
The image generation chain (final factor) of FG-DM has the same computation as existing VC-DMs since it re-uses existing conditional models such as ControlNet. The only additional computation comes from the condition synthesis factor and faithful comparison of complexity must include the condition synthesis step. For existing DMs, generating/cditing conditions for segmentation

---

Segmentation mups. (Left:: Groundiruth, Right: Resonstructed)
(b) Depth maps. (Left: Groundruthh, Right: Reconstracted)
(c) Sketch maps. (Left: Groundtruth, Right: Reconstructed)
(d) Noemal maps. (Left: Groundruth,, Right: Reconstructed) Figure 16: Visualization groundtruth (left) and reconstructed (right) maps by applying the pretrained stable diffusion autoencoder to segmentation, depth, sketch and normal maps.
Table 6:
Comparison of segmentation mask quality and throughput of FG-DM trained ADE20K dataset against state-offthe-art conventional segmentation models of similar size. The FG-DM samples higher quality masks with only 10 DDIM steps, comparable
throughput of Segformer-Bi but with superior quality as
shown in the FID, Precision and Recall metrics.
Model (ADE20K)
Params (M)
FID.
LPIPST
Precision?
Recal
fimgs/sf
Segirormer-B5 B5 (50
85
112.6
0.781
0.61
0.04
6,4
FG. Seg (10 DDIM steps)
PG-DM-Se Seg (20 DDIM steps)
53
$4.0
0.776
0.72
0.05
3.7
FG-DM-Seg (200 DDIM steps)
53
83.6
0.768
masks requires the following steps: (1) Generate image with DM. (2) Segment with off-the-shell segmentation model. (3) Edit the segmentation. (4) Generate the image conditional on the manipulated segmentation. The FG-DM eliminates step 1. replaces step with the segmentation synthesis step. This is much more efficient than running an image generation DM and segmentation model as illustrated in Table Tsinc FG-DM samples segmentations at lower resolution. For example, a 20- step image generation, FG-DM (with ControlNet as the final factor) takes only 4.5s (1.7x speedup) as compared to 7.5s for the standard pipeline (image generation with SD, Segmentation with SegFormer (CEM), and conditional image generation with ControlNet).
20

---

Table Ablation study on Image Synthesis by FG-DM trained separately and jointly with segmentation factor. FG-DM results presented as Images/Semantic maps.
Model
HP (MD FID
MM-CCllbA
ScnpSs [LPIPS
ADE-20K
COCO-Stuff
LPIPS t jixt [Bet [ID
|Prt
[Ra7 [PI L
[LPIPS
|Pr f|Re 1 FID
[LPIPS Pr 1/
FG-DM (Separate truining)
1:0 8 50.57/0.54 29 51.7/61.8 560.51
0.71 140 21.3/20.3 0.5800.44 (0.81 (..10444441444 (034 47.6/618 (.59.0..7 0.69/..31 29.6/836 (0.79/0.77 0.72 (33.1/57,4 (.8300. 0.69 00.43
wtt.a.................. (0.79/0.77 176600.7733.0000231
2. iss 8.8.0.
FG-DM (Joint training)
Table 8: Ablation study on data augmentation with synthetic data generated by the FG-DM for facial part segmentation on MM-ColebA-HQ(2) dataset.
Data
#Samples
mlolit
F.W. mloUt
Fl-score""
Orig
24993
51.1
85.6
61.5
Orig+Syn
Orig+Sya
+2000
55.0
87.1
65.0
Table 9: Ablation study on data augmentation with synthetic data generated by the FG-DM for face landmark estimation on 340W(43) dataset.
NME
#Samples
Common
Full
Challenge
Orig
3000
3.21
3.64
5.81
Orig+Syn +1000
3.12
3.54
5.80
Orig+Syn
+2000
3.18
3.61
5.81
Next, we evaluate the generated mask quality from FG-DM. Table 6 compares the segmentation mask quality vs. throughput, for FG-DM and SegFormer (CEM) on the ADE20K dataset. In these experiments, SegFormer are applied to the validation dataset images and FG-DM masks are obtained with the validation prompts. Throughput is calculated on single NVIDIA-Titane
GPU with batch size averaged over 2,000 runs. Performance is reported in terms of FID, LPIPS, Precision, and Recall metrics of the masks. For the FG-DM, results are presented at different sampling steps. With only 10 timesteps, the FG-DM produces segmentations of quality superior to those of the CEM segmentation model. This is achieved with a throughput comparable to that of the bigger SegFormer-B B5 model. Beyond 10 steps there are negligible improvements in all performance metrics, showing that conditions like segmentation masks can be generated much faster than natural images, due to their lower frequency content. The FG-DM merges the capability to generate segmentation masks of superior quality compared to current segmentation models, crucial for creative tasks with unseen images, with a throughput comparable to the latter, which can only segment existing images.
A.1.2 Models trained from scratch: Ablation Studies
We next discuss some results for FG-DMs trained from scratch where the adapter in Fig. ii removed and the intermediate conditions are concatenated to be fed to the subsequent factors.
Joint Synthesis: Table 7 compares end-to-end training of FG-DM to separate training of VC-DM factors. Jointly training the FG-DM improves the image quality (lower FID) and diversity (higher LPIPS) on all four datasets.
Table TO shows the comparison of FG-DM with joint modeling by concatenation, where conditioning variable(s) are concatenated in the latent space and denoised jointly with the image using a single DM. The table clearly shows that FG-DM outperforms the concatenation approach by 8 points on the FID metric despite being smaller in size. Note that joint denoising by concatenation requires larger model and forfeits many advantages of the FG-DM, such as higher object recall, image editing, and computational efficiency.
Order of conditions: Table IT ablates the order of the conditioning variables for the model with semantic (S) and pose (P) condition on CelebA -HQ and COCO datasets which shows that the order of the chain affects final image synthesis quality (see FID). This primarily stems from the misalignment of the pose map with semantic map and image, where the generated pose maps are inferior for the alternate order (pose- sens-ssmantic-magg)) It shows that the unconditional generation of sparse pose maps is much more difficult than segmentations, due to the incomplete poses (occlusion) and missing context in the pose images. Further, Table [1 shows that the extra pose conditioning improves the image synthesis quality, reducing FID by 3.7/2.33 points on CelebA-HQ/CCCOO datasets respectively. This shows that adding more conditions is helpful for improving the image quality.
Data Augmentation
additional benefit of the FG-DM that it can be used as a data augmentation technique, synthesiz- ing data to train models for segmentation, pose estimation, etc. By sampling data from the FG-DM,
21

---

Table 10: Comparison of FG-DM con- Table 11: Ablation on the order of generated semantic map ditioning vs concatenation approach for (S) and pose (P) conditions on CelebA HQ (Top) and COCO joint synthesis on CelebA-HQ. U-LDM re- ported for reference.
(Bottom). I- Image.
Model
FID 4
|.PPIS
1|
P1
R
(M)
LPIPS
(P-+S-11)
23.43
0.616
[0.616
0.466
U-LDM (41
24.3
0.586
P->P-
17.61
0.594
0.754
0.403
Joint (Concat)
248
29.4
0.618
(P +--5-11
31,43
0.855
0.547
0.343
PG. VG.DM (Ours)
140
21.3
0.578
(S+P-+1)
30.77
0.857
0.564
0.322
Table 12: Ablation study on Image Synthesis by FG-DM for sequential and joint inference. For the same segmentation maps, joint inference superior to sequential inference showing the henefit of joint modeling.
Model
RP
MMI-CelehA
Closccpes
ADL-30K
COCO
(M)
FID
[LPIPS T [Pvt [Ret FID
[LPIPS
|Pr Ret
[ID
LIPIPS
P:
FID
[LPIP
1
|Rc
(Sequential Inference)
34.5/20.3
10200
4
0,52008
0067
FG-DM (Joint Inference)
21.3/0.3
0.500.54
(881 (1.4 47.6/618 0.59HL57
(160 0.31 29,68336
0.79.0.77
0.72 0.33 33.0/7.44
0,33/L 0.09
is possible to produce labeled datasets of virtually unlimited size. These could be, in principle, useful to train downstream models for various vision applications. To investigate this, we start by training both the FG-DM and the downstream model on a labeled dataset A. We then use the FG-DM to synthesize an additional dataset B of images and labels and retrain the downstream model on A U B. We finally compare the performance of the two downstream models. We performed this experiment for two downstream tasks: part segmentation with the (5Sc network on MM-CelebAA-HQ and pose estimation with the HRNetV2-W W1 (48) network on the 300W (43) dataset. In all cases, the downstream model is initialized with ImageNet preitre weights. Tables X and show that the addition of synthetic data always improves downstream model performance. The gains are larger for the more challenging segmentation task, where 1000 samples of synthetic data significantly improve the baseline mIOU and FI-score by 4 %. For pose estimation, the addition of synthetic data reduces the already low normalized mean error (NME) of keypoint location by an additional 0. %. Performance saturates or decreases slightly when the number of synthetic images is increased to 2000 as compared to using only 1000 images. We conjecture that this occurs from the inclusion of redundant or noisy images in the enlarged dataset, which are less likely to contribute meaningfully to the model's training. Nevertheless, the performance still shows improvement compared to the baseline model that did not use any synthetic data augmentation. These results suggest that the FG-DM can be used for data augmentation, enabling significant performance gains by automated data augmentation.
Figure 17: Unconditional generated semantic, pose masks, and corresponding images using an FG-DM trained from scratch on MM-CelebA-HQ (256 x 256).
Sequential vs Joint Inference Table 12 shows the comparison of two modes of operation of FG-DM during inference. Joint inference is the standard mode of operation that samples the condition(s) and image jointly at each timestep (equations (2)-(4) in the paper). Sequential inference the alternative mode of operation where the condition synthesis chain is run fully before feeding it to the conditional image generation factor. It is observed that the standard mode of joint inference sampling produces images of higher quality (lower FID) than the alternative mode.
Qualitative results: Unconditional and Conditional Image Generation
22

---

IQQ samples
samples
COCO 256. 256 samples Figure 18: Segmentation mask/image pairs synthesized by FG-DMs trained from seratch (53 M parame- ters) on MM-CelebA-HHQ. ADE-20K, Cityscapes and COCO datasets.
(c) Cityscapes 256 512 samples
(a) MMM-CeehAA HQ
(b) ADE-20K
(c) Citycapes
(d) COCO
Figure 19: Semantic Guided conditional image synthesis by FG-DM models trained from scratch (53 M parameters). For conditional synthesis, only the image- synthesis VC-DM factor is used. When conditioned by the validation dataset segmentation maps shown the top, this factor synthesizes the images shown at the bottom.
shows qualitative results of unconditional synthesis on HM-CelebA-HQ for a model with factors for semantic map. pose and image synthesis. This improves the overall image quality as shown in Table [T] It observed that the generated pose maps are accurate even for hard examples such side-view faces.
Fig. 17
Figure shows qualitative results of image/segmentation mask pairs generated by FG-DM models trained from scratch on four popular semantic segmentation datasets. The FG-DM can generate good quality samples even on complex datasets such as ADE-20K COCO using only a small model (53M parameters for each factor).
Figure shows qualitative results of conditional image synthesis with groundtruth maps of validation set on the four datasets using FG-DMs trained from scratch. For conditional synthesis. only the image-synthesis factor the FG-DM used. In these experiments, the model conditioned by the
23

---

Table 13: Ablation study on Image Alignment with the segmentation masks by FG-DM trained separately and jointly.
Model
WP
MM-CelebA
Cityscapes
ADE-20K
COCO
(M)
mloU f/... loU
mloU f f.w. loU f mloU f.t... loU
mloU
f/E... loU
FG-DM (Separate training)
140
70.26
85.77
45.38 52.73
81.84 86.72
18:07
48.65
23.29 23.76
37.33
FG-DM (Joint training)
140
70.57
187.79
22.22
52.76
38.64
ground-truth validation segmentation masks of the datasets. It is seen that FG-DM can generate high quality samples that align with the latter.
Segmentation-Imagg Alignment
We quantitatively evaluate the alignment of the generated images with the corresponding segmentation mask using off- off-the-shelff pretrained models described in Sec. A.2.5 We compare the image alignment of FG-DM when training separately and jointly. Table 13 shows the results on four datasets where the mloU score computed using the groundtruth validation samples for each of the dataset. Once again, FG-DM trained jointly outperforms the FG-DM trained separately indicating the advantage of joint modeling in following the segmentation conditions accurately.
A.2
Implementation details
A.2.1
Extracting COCO Object classes from the prompt using LLM
We proposed FG-DM for faster sampling of images with high object recall and validated it by by using the groundtruth segmentation maps from the ADE20K validation dataset. For practical use, object classes can either be manually specified or automatically extracted from the captions which is useful for images involving cluttered scenes with multiple objects. Here, we show that the object classes can be extracted from the caption using LLM (c.g., chatGPT-3.5).
Specifically, we use the following prompt to elicit responses from chatGPT-3.5.
Prompt
You are ObjectGPT. You will list all possible objects in scene from the caption description using the set of available classes. The available classes are as follows. {COCO Classes inserted here as dictionary mapping the class id to class name.)
Figure 20a|show shows example outputs for the two prompts shown in the image. Note that this is zero-shot output where the model is not provided with any example pairs of prompt and corresponding object classes. The accuracy of the task can be further improved by using few-shot in-context examples as shown in the in-context lcarning (ICL) literature. We show example of one-shot ICL with one prompt and its corresponding object classes from the COCO validation dataset.
1-shot ICL
"A man is in a kitchen making pizzas"
{Object classes = "pprrson", "bottle", "cup", "knife", "spoon", "bowl", "broccoli", "carrot". "dining table", "oven", "sink", "branch", "cabinet", "floor-other", "floor-stone", "food-other", "furnitureoother", "leaves". "light", "metal", "table", "textile-other", "wall-other", "wall- stone"]]
Fig. 20b shows the result for the prompt "The dining table near the kitchen has bowl of fruit on it." using one-shot ICL shown above.
A.2.2 Segmented Image Editor
We introduce a tool, developed using PyQT to edit segmentation masks synthesized by the FG-DM, enabling an array of options. As shown in Figure 21 this app has a user-friendly interface for loading and manipulating objects across two distinct segmented maps, which enables very flexible image synthesis. Key features include the ability to add. move, resize remove objects, flip them, or
24

---

bovi
bok I60
famwrter
Fummll Temiaardte
wile
1240030
Extracting object prompt using chatGPT.
Extracting object ICL with cha1GPT-3.5
prompt using
3.5 zero-shot
Figure 20: Snapshots of using chatGPTT3.5 to extract the object classes from the input prompt.
replace backgrounds with ease. unique drawing tool, augmented by color palette representing 183 objects from the COCO dataset, allows for precise and detailed customization. Furthermore, the app's s pointer size adjustment slider for drawing and resizing ensures users achieve the exact level of size, detail and boldness needed for image editing. Currently, the complexity of these edits is limited by the simplicity the image editing tool developed. More complex images will likely be possible with further editing tool development.
(a) Loading the gemerated segmentation masks into the tool
(b) Resizing the person und moving them to the left of the image
(c) Final edited mask after resizing and moving the airplane
Figure 21: Snapshots of editing capabilities using our segmented image editing tool.
Table 14: Hyperparameter Settings for the FG-DMs trained from scratch on MM-CelebA-HO, ADE-20K,
Cityscapes and COCO datasets. We image resolution of 256 512 for Cityscapes and 256 256 for the others.
MM-CeeebA-HQ
ADE-20K
Cityacupes
COCO
4
2-shape
64 x
64 x x
x 128 x
64 64
[Z]
8192
8192
8192
8192
Diffusion steps
1000
1000
1000
1000
Optimizer
Adam linear
Adamn
Adamm
Noise Schedule
Tinear
linear
linear
Nparams
86M
86M
Channels
128
128
128
128
Depth
Channel Multiplier
1,4,8
1.4,8
1,4,8
1,4,8
Atlention resolutions
32,
16,
32, 16, 8
32, 16,
32, 16,
Channels
Batch Size
12
12
12
12
Iterations
632k
93k
632k
Leaming Rate
le-6
le-6
1.0e44
le-6
A.2.3 Hyperparameter Settings
Table 14 summarizes the detailed hyperparameter settings of the FG-DMs trained from scratch reported in the main paper. For FG-DMs adapted from Stable Diffusion, we use the same settings as Stable Diffusion (41) and train only the adapters for 100 epochs with learning rate of 1e-6 using AdamW optimizer.
25

---

A.2.4 Training Details
We train all models using 2-4 NVIDIA-A4I GPUs or 2 NVIDIA-A 100 GPUs based on the availability. For adapting Stable Diffusion, since we reuse existing conditional model such as ControlNet, we first pretrain the model for 100 epochs to synthesize the conditions (e.g., segmentation, depth, normal or sketch). We then jointly finetune the condition (e.g., segmentation) factor with the conditional image synthesis factor (e.g., ControlNet) for an additional 100 epochs by only updating subset of parameters of the ControlNet adapter denoted by the input-hint-ble the rest frozen. Note that the pretrained ControlNet can still be used in FG-DM without finetuning which results in a slightly lower image quality (also validated in Table 12). For models trained from scratch, we train all the parameters of the U-Net model from scratch.
in the model while keeping
A.2.5 Evaluation Details
This section provides additional details on evaluation for the experiments of Sec. 4. We follow common practice and estimate the statistics for calculating the FID values (14 shown in Table 7 are based on 10k samples between FG-DM generated samples and the entire training set of each of the datasets. For calculating FID scores practice, we pre-process all the images by resizing to 256 x 256 for MM-CelebA-HO,, ADE-20K and COCO datasets and 256 512 for Cityscapes dataset for calculating the metries. Samples generated with 200 DDIM 445) steps and rj 1.0. For the measuring the semantic alignment, we use off-the-shelf networks evaluate the alignment of generated results. We use DRN-D-110 (55) for Cityscapes, ResNet50Dilated- PPM (61 for ADE-20K, Unet (22) 42) for MM-CelebA-HQ and DeepLab (8) for COCO dataset. The generated images are fed to these segmentation models to obtain a pseudo-mask which is compared against the mask which was used to generate the image. We use mean ineerecction-overurr between the generated images and the semantic masks. We calculate the mloU by upsampling the generated images to the same resolution as default input resolution of the off the-shh segmentation models. For computing the metrics, we use the validation segmentation maps for each dataset (3000 images for MM-CelebA-HQ. 2000 images for ADF-20K, 500 images for Cityscapes and 5000 images for COCO) is used. However, it should be noted that this pseudo metric highly depends on the capability of the off-the-shelf network. A strong segmentation network measures the semantic alignment of the generated images more accurately. For all text-to-image image synthesis results reported in the paper, we use classifier free guidance with scale 7.5, 7J = 0.0 and 20 DDIM steps unless otherwise stated.
the torch-fidelity package (3I). Following standard
(mloU) and frequency weighted mloU measure the overlap
A.3 Broader Impact
We introduce a new framework for controlling diffusion models that offers creative image synthesis with higher recall, greater flexibility, modularity and explainability. While it offers the benefits of revealing hidden harmful biases in existing image generative models and offers better interpretability. it can also be potentially misused to propagate harmful, unlawful unethical information with harmful edits. Since, the framework is modular, any harmful edits can be identified before the image generation step where the segmentation or pose map factor can be filtered (automatically or manually) before proceeding to the image generation factor. Additionally, recent advancements in image watermarking (25) can help to identify generated image contents to protect against these risks.
A.4
Future Work
For future work, FG-DM can be extended for Novel View Synthesis by adding a factor for Novel Views. Further, the Novel View FG-DM with depth/normal factors can be used as strong prior for controllable Text-to-3D generation with SDS technique (35). The modular nature FG-DM also allows potential extension for audio/video generation making FG-DM framework to be strong candidate for multi-modal content generation.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The major claims made by the paper regarding the object recall, image editing capabilities and data augmentation are justified by experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The conclusion section discusses the limitations of the current approach and possible future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
There are no theorems or proofs introduced in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The hyperparameter settings, GPU configurations and auxiliary models em- ployed by the work are detailed in the appendix which is sufficient to reproduce the results presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code and trained models will released on the project page. FG-DM
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nips.co)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /nips..c/public/uuiiee /CodeSubmiodmeebm ssi wwwssnooiiiiieeeeeeeoo Poylicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix lists down the detailed hyperparameter configurations for all models trained for this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following the literature, evaluations are reported consistent with prior works where the error bars not shown.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed compute resources and the GPUs used for training islisted in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics h https/////pp/tttitcccieieeeeeeee
Answer: [Yes]
Justification:
The paper followsthe code of ethics listed in the guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Limitations and Broader impacts in Appendix discuss this
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Image generators will be released with the safety filters
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All prior works and prior models used in this work are cited in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, thcor com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subject is involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "uAJvWVnFNc", "reviewer": "Reviewer_F3xm", "summary": "This paper introduces a Factor Graph Diffusion Models (FG-DMs) to address limitations in current generative models. FG-DMs model the joint distribution of images and conditioning variables using a factor graph decomposition, offering advantages like efficient sampling for better prompt compliance, fine-grained editing, and explainability. The method's effectiveness is validated on several datasets.", "questions": "What's the training cost of your model? Is there a comparison with other methods?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces a method to model the joint distribution of images and conditioning variables, enhancing prompt compliance and control over image synthesis.\n\nThe Attention Distillation Loss is proposed to enhance the consistency and quality of generated images and conditions.", "weaknesses": "The proposed method seems promising, however, it's very important to compare with current SOTA works to justify its effectiveness. The author only put Stable Diffusion in Table 5, which is not fair. Please include  more recent works especially sine 2023 such as  StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis,  Make a Scene.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and useful feedback on the paper.\n\nQ1. The proposed method seems promising, however, it's very important to compare with current SOTA works to justify its effectiveness. The author only put Stable Diffusion in Table 5, which is not fair. Please include more recent works especially sine 2023 such as StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis, Make a Scene.\n\nUnfortunately, the StyleGAN-T work, mentioned by the reviewer, only has open-source code. The trained checkpoints are not available. This makes it impossible for us to compare, since we are an academic group and do not have the resources to train the model from scratch. Make-a-Scene does not have either code or checkpoint released, which makes us unable to perform quantitative comparisons on COCO validation dataset. Instead, we compared the qualitative results of FG-DM with Make-a-Scene and SpaText in Figure 13 of the Appendix (figures taken from the respective papers). Note that the proposed FG-DM synthesizes both segmentation maps and images, while the results of these methods were obtained with manual sketching of the segmentation, which is cumbersome and requires some skill. Nevertheless, the FG-DM generates high quality images that adhere well to the prompts as compared to these prior works.\n\nIn any case, the reviewer has a good point in requesting quantitative comparisons to more recent methods. For this, we compare with the recent 4M model [1], an autoregressive model trained from scratch on both discriminative and generative tasks. In this case, we use the largest model (4X-ML) released by the authors. Table 1 in the rebuttal pdf presents the updated comparison. Figure 1 in the rebuttal pdf shows a qualitative comparison between FG-DM and 4M. It can be seen that 4M generates images of weaker quality  (distorted hands, missing person's head, deformed zebra bodies) as compared to FG-DM with/without the attention distillation loss. \n\n\nQ2. What's the training cost of your model? Is there a comparison with other methods?\n\nRegarding training time, one of the significant benefits of the proposed FG-DM is the short training time, since it only requires training adapters that are added to the pretrained SD model. Unfortunately, detailed comparisons are hard to perform since, as mentioned above, we do not even have the resources needed to train many of the models that have appeared in the literature. For the FG-DM factors, we simply train the adapters for 100 epochs, please see section A.5.2 in the Appendix of the original paper for a more detailed discussion. The training time per factor is about 1.5 days on 2 A100 GPUs or 72 A100 GPU hours. Finetuning the ControlNet takes about 2 additional days on 2 A100 GPUs. We find it quite interesting that these simple adaptations work quite well even for the synthesis of conditions, which the SD model was not originally trained to synthesize. For certain conditions the training even converges faster, e.g for depth maps it only takes 10 epochs and we do not observe any improvement after that. \n\nFor reference, training the original SD model requires about 150,000 A100 GPU hours, which is $\\approx 2,000$ times larger. The FG-DM is also much more efficient than 4M, which requires 1.5 days training on 64 A100 GPUs for their smallest model (4M-B) and 8 days on 128 A100 GPUs for their largest model (4M-XL) which produces lower quality images as discussed above. \n\n[1] Mizrahi, D., Bachmann, R., Kar, O.F., Yeo, T., Gao, M., Dehghan, A., Zamir, A.: 4M: Massively\nmultimodal masked modeling. In: Thirty-seventh Conference on Neural Information Processing Systems\n(2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGZPFbN6iB", "reviewer": "Reviewer_cyg3", "summary": "The paper proposes unified generation framework of simultaneously generate image, segmap. depthmap, etc.", "questions": "See weakness", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of simultaneous generation of images with various conditioning maps is very interesting.", "weaknesses": "1. Although the method shows good performance in image editing for generated images, it does not include any discussion of whether the framework can be applied to real image editing case. Does it can be applied to inverted real images? Does the framework can still predict the segmentation map and keypoints of given real images? If that is possible, there would be great impact and future usage.\nIf it is still applicable to real images, then please show some results which are similar to ControlNet.\n\n2. The manuscript does not contain any qualitative comparison of ablation study. Please show some visual results on ablation study.\n\n3. Please show the generated image results using handcrafted conditions, not the edited condition from generated conditions. For example, give the hand-drawn sketches (user-identified one) to the model and show the generated outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and useful feedback on the paper.\n\nQ1. Does it can be applied to inverted real images? Does the framework can still predict the segmentation map and keypoints of given real images? If it is still applicable to real images, then please show some results which are similar to ControlNet.\n\nThe editing question is quite interesting and motivated us to consider aspects that we had not thought about. Since the framework is based on Stable-Diffusion, it can be applied for inversion. Figure 2 of the rebuttal pdf provides an answer to our understanding of the question \"Can it be applied to inverted real images? Does the framework can still predict the segmentation map and keypoints of given real images?\" \n\nWe have experimented with editing of both real images and their segmentation masks. The top of Figure 2 refers to inversion of the segmentation mask. We use an off-the-shelf OpenSEED [2] model to extract the segmentation map of a real image (shown on the bottom left of the figure) and apply the FG-DM segmentation factor model for inversion and editing using LEDITS++ [3], a recent method for text based image editing. We apply LEDITS++ to the segmentation factor to 1) replace the mask of the woman by that of a chimp (third image of the top row) and 2) to delete the mask of the umbrella (fifth image). New images (fourth and sixth) are then generated by the image synthesis factor conditioned on the edited segmentation masks. We have found that the inversion and editing of segmentation masks is quite robust. The synthesized masks usually reflect the desired edits. However, because the final image synthesis is only conditioned on these masks, the synthesized image does not maintain the background of the original image. The synthesized image is a replica of the original image at the semantic level (similar objects and layout) but not at the pixel level. From our experiments, this method has high robustness and quality for semantic-level editing. It can also be used for pixel-level editing using copy and paste as discussed below.\n\nWe next investigated pixel level inversion and editing, which is harder.  The bottom part of Figure 2 shows the comparison of LEDITS++ editing with inversion by SD and by the image synthesis factor of the FG-DM. For the latter, we apply inversion to the ControlNet image generation factor using the real image and the segmentation mask extracted from it. Then we perform the LEDITS++ edit using the edited mask from the top part of Figure 2 (inverted with the FG-DM segmentation factor) to produce the edited image as shown in columns 4 and 5. This pixel-level inversion and editing tends to maintain the background of the original image but is much less robust than mask-level editing in terms of editing quality. This can be seen from the images in columns 2 and 3, which show the inversion using SD, which fails to produce a realistic chimp and turns the woman into a stone sculpture. The FG-DM produces more meaningful edits but which still has room for improvement, as shown in columns 4 and 5. The last column of the bottom part of the Figure 2 shows an added advantage of FG-DM where the chimp generated in the top portion can be pasted to the original image due to availability of the segmentation mask. In this example the pasting is rough around the object edges since we have made no attempts to beautify it. We believe that this copy and paste technique can produce high quality images with a bit more of work. Nevertheless, these examples show that real images can be edited directly or used as semantic guidance to generate new images following the specified semantic layouts. The pixel-level inversion can also be improved with more work but not within the time frame of the rebuttal. It will require some additional technical contributions and we leave it, as well as the optimization of the copy and past technique, as a topic for a subsequent paper on the use of FG-DMs for text-based image editing. \n\nQ2. The manuscript does not contain any qualitative comparison of ablation study. Please show some visual results on ablation study.\n\nGreat point about ablations. Figure 1 of the rebuttal pdf shows some qualitative results of the ablation for the impact of the attention distillation loss.  There is a clear qualitative benefit in introducing this loss. Without it, the model generates less accurate masks, leading to an unrealistic pizza making depiction/ cart-person relationship/ zebra pair from left to right. This confirms the qualitative ablation showing the benefits of the attention distillation loss in Table 2 of the main paper but provides a stronger illustration of the advantages of the loss, which tends to produce more \"plausible\" scenes. Such plausibility is difficult to measure with qualitative metrics. For example, the CLIP score is not sensitive to the fact that the cart and the person are not interacting in a normal way, or that the pizza making activity is unrealistic.\n\nQ3. For example, give the hand-drawn sketches (user-identified one) to the model and show the generated outputs.\n\nWe note that if hand-drawn sketches are provided the model reduces to a standard conditional DM. In this case, it inherits all the capabilities of the ControlNet, or whatever DM is used to implement the final factor. Figure 3 of the rebuttal pdf shows some qualitative results on using hand drawn sketches to synthesize images. These are as good the ones produced with the ControlNet. \n\n[2] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, Lei Zhang. A Simple Framework for Open-Vocabulary Segmentation and Detection, ICCV 2023\n\n[3] Manuel Brack, Felix Friedrich, Katharina Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, Apolinário Passos. LEDITS++: Limitless Image Editing using Text-to-Image Models, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5FC5ky9Ys", "reviewer": "Reviewer_Xxm8", "summary": "This paper proposes a method for sequentially generating images using a frozen SD. Starting from a text prompt, the process iteratively generates several visual conditions and images, with each step depending on the previous ones. The model uses the VAE in SD to encode and decode visual conditions and employs an condition-specific adapter to introduce information into the SD for generating these conditions. This approach makes the image generation process more controllable, flexible, and easier to edit.", "questions": "1. The assumption behind attention distillation suggests that word tokens and visual regions remain constant, even if the visual regions are non-semantic, like poses. Why does this happen? It seems odd that the attention score of a general word like \"person\" can help generate the poses of a person.\n2. Why do you use independent noise for different factors?\n3. The experimental details are unclear. Does the model run with a single visual condition for each result, or does it generate all conditions sequentially? It would help to clearly present the generation process for each experiment, including Tables 1-5 and Figures 5-7.\n4. How do you implement null conditions for different variables?\n5. You claim that after choosing the best segmentation factor, the image synthesis only needs to run once. Why is this? Does the segmentation factor determine the quality of the generated images, or is your goal just to ensure the object appears in the generated image?", "limitations": "The authors have  adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The iterative generation method using a stable diffusion model is impressive. It provides strong control over the output while maintaining flexibility, aided by classifier-free guidance training.\n2. The new method allows for flexible editing by modifying visual conditions and letting the model generate realistic content based on these changes.", "weaknesses": "1. The newly introduced iterative process adds extra loop to each generation step.\n2. Some components and experimental details are not well-explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and useful feedback on the paper.\n\nQ1. The newly introduced iterative process adds extra loop to each generation step.\n\nAlthough the proposed FG-DM adds an extra loop in the generation process, in Table 4 we show that using lower resolution and fewer timesteps for conditional generation has no impact on overall FG-DM performance. Hence, the increase in inference time over vanilla SD is small (4.45 secs for FG-DM vs 3.25 secs for vanilla SD). However, this is not a fair comparison since the object recall of the FG-DM (67.8) is much higher than that of SD (59.8). Note that the whole point is to improve prompt compliance. For reference, the popular Attend-Excite method [4] has a much lower improvement in recall (63.6) for a much larger inference time (36 secs). On the other hand, the table shows that the FG-DM is 4 times faster than SD under a fair comparison, where the two methods have equal recall. Please see the reply to Reviewer t2kD for a more detailed discussion. \n\nQ2. Some components and experimental details are not well-explained.\n\nGood point about missing details. Table 1-5 and Figures 5-7 use a single visual condition for the image generation.  The results presented in Figure 2 and Figure 19 follow the chain segmentation--pose--image (image conditioned on both). In general, the FG-DM can be trained with a single or multiple conditions. However the complexity (training, inference, and model size) increases with the number of conditions. Hence, the best model configuration is a trade-off between meeting application requirements and complexity. The critical condition for improved prompt compliance is segmentation, since it allows the measurement of object recall. It is also important for editing, since it allows moving and resizing objects. In the rebuttal pdf we show that it can even be combined with editing methods like LEDITS++ to enable text-based object editing operations. The other conditions are important for some image editing applications, e.g. changing object depths, people pose, or sketching, and can also contribute image quality improvements as shown in Figures 2-3. Overall, the FG-DM is a very flexible framework, allowing to mix and match different factors so as to best meet application needs. We will add all details to the final revision. \n\nQ3. It seems odd that the attention score of a general word like \"person\" can help generate the poses of a person.\n\nThe reviewer has a good point that it does not make a lot of sense to use attention distillation for pose. We do not use it for pose factor and only use it for other conditions when these are the only conditioning factors. Note that the pose factor is conditioned by the segmentation, which is itself subject to the attention distillation loss. For multi-condition FG-DMs, the distillation loss is only required for the synthesis of the first condition, which is conditioned by text alone (see Figure 4 of main paper). For the subsequent factors, which are already conditioned by a visual condition (e.g. pose conditioned by segmentation), the attention loss is not needed. We will clarify this in the revised version.\n\nQ4. Why do you use independent noise for different factors?\n\nThe question about independent noise is an interesting one. Independent noise is a natural assumption (after all, the noise is independent even across steps of the same diffusion model factor) and enables a simple training and inference process. It is also not clear what a natural model for the dependencies between the noise of a segmentation mask and an image would be, for example. Maybe one could consider the use of a common noise in all diffusion chains. This would be trivial to implement in the forward chain (just sample noise once per step instead of per step and per factor) but less trivial for the reverse denoising step, as some regularization would be needed to guarantee that all the chains produced the same noise. Maintaining the consistency of the noise across factors in the denoising process would likely be difficult. So, while we do not rule out that dependent noise may be an interesting possibility, we do not see an obvious way to implement it and we are not clear that it would be beneficial. We leave this question as a topic for future work. \n\nQ5. How do you implement null conditions for different variables?\n\nWe use a null condition to support classifier free guidance during inference when generating conditions, even for the single condition FG-DMs. The null condition is implemented by using an image filled with zero values as the conditional input for the final factor of FG-DM (ControlNet). For each visual condition model (seg, depth, sketch, normal), we use the empty prompt (\"\") as the null condition while training the models.\n\nQ6. is your goal just to ensure the object appears in the generated image?\n\nThe image synthesis only needs to run once because we are only attempting to improve prompt compliance. As the reviewer notes, the idea is to sample segmentations until we can ensure that the objects appear in the image.  We verify the object recall with the segmentation factor for the results presented in Tables 3 and 4 and Figure 7. It is true that we could also sample multiple images given segmentation but this would not improve prompt compliance, maybe only image quality given the segmentation. We do not do this because 1) the existing models already produce high quality images (it is not clear that the extra computation would be justified), 2) the existing metrics of image quality do not support an evaluation of image quality fine-grained enough to even know if this would make a big difference in terms of the human judgements of quality for the resulting images, and 3) it can be done for any diffusion model (there is no special advantage of the FG-DM here).\n\n[4] Chefer, H., et. al.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. SIGGRAPH (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h355ImxUM6", "reviewer": "Reviewer_t2kD", "summary": "This paper propose a new family of diffusion model: FG-DM for better prompt compliance. Unlike traditional diffusion model, FG-DM models the joint distribution of images and conditioning variables like semantic, sketch, depth or normal maps via a factor graph. These extra factors largely boost prompt compliance compared to use text only. Users can also easily edit target image by modifying intermediate conditions. This work shows better results compared to vanilla diffusion.", "questions": "1) How much more time do we need for the conditional generation chains? (for example, how much more time do we need when we use one or two conditions compared to original SD?)\n2) If we use an off-the-shelf condition generator e.g. a text-to-depth map generator, and we fed the generated depth map to controlnet for generation, what is the advantage of our method?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "By decomposing the text into different factors such as seg maps, depth maps, etc., and using these factors as conditions for generation, this method achieves better prompt compliance. Additionally, the introduction of intermediate factors makes this method easier to edit.", "weaknesses": "Since intermediate factors are generated sequentially, these method need more time for generation. The novelty of this method is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and useful feedback on the paper.\n\nQ1. How much more time do we need for the conditional generation chains? (for example, how much more time do we need when we use one or two conditions compared to original SD?)\n\nRegarding time, we make several considerations. First, the fact that the conditions can be synthesized with lower resolution makes their synthesis much faster than vanilla SD. Typically SD requires 3+ second per image while each condition typically requires less than 1 second. Hence, the  increase in overall synthesis time is small. This is shown in Table 4 of the main paper, where generating an image with SD requires 3.25 seconds while generating an image with one condition using FG-DM takes 4.45 seconds for 10 timesteps and 4.81 seconds for 20 timesteps. For two conditions, these numbers become 4.9/5.26 seconds for t=10/t=20 timesteps. Second, a comparison to vanilla SD is somewhat unfair because the whole point is to increase the prompt compliance of vanilla SD. A fairer comparison is to popular methods to achieve this goal, such as A-E, which the table shows to require 36 seconds per image! Third, while the implementation of SBPC with the FG-DM achieves good results for single image synthesis (N = 1), the bottom half of the table shows that its true potential is unlocked by larger batch sizes. Both the SD and FG-DM implementations of SBPC achieved the much higher average object recall in this setting. However, for SD, the sampling of a batch of $N = 10$ high resolution images requires 23s per prompt, while the FG-DM is 4x faster (when using 10 DDIM steps for the segmentation mask and the same 20 steps for image synthesis) and achieves the same object recall. \n\nQ2. If we use an off-the-shelf condition generator e.g. a text-to-depth map generator, and we fed the generated depth map to controlnet for generation, what is the advantage of our method?\n\nTable 8 and Table 13 in the Appendix compare to the sequential implementation trained with segmentation condition, where the models are trained independently and used as the reviewer suggests. They show that joint training and joint inference of image and factor synthesis improves the quality of the generated images as compared to sequential operation. \n\nQ3.  The novelty of this method is limited.\n\nRegarding novelty, we would note the following facts. \n- First, the paper is the first to propose Sampling based Prompt Compliance (SBPC), which is shown to be much more effective than the existing Inference based prompt compliance (IBPC) approach. We show that IBPC methods are slower and underperform the proposed implementation of SBPC with the FG-DM. \n- Second, we do introduce the attention distillation loss, which is shown to make a difference, both in the quantitative ablations of Table 2 of the main paper and the qualitative examples that we now show in the rebuttal pdf file (also see reply to Reviewer cyg3). \n- Third, the concept of the FG-DM is novel and could have many other applications beyond improved prompt compliance. For example, its modularity enables more effective continual learning schemes, where factors are added or updated for specific conditions without the need to fully retrain the model. These type of benefits are not easy to measure/quantify without devoting a complete paper to the formulation of the continual learning problem, introducing experimental protocols, baselines, etc. We intend to consider the issue in future work. \n- Finally, the paper shows some results that we consider somewhat surprising (and thus \"novel\"), e.g. 1) that SD can be prompted, without even changing the auto encoder, to produce segmentation maps or other conditions and that this even maintains the generalization ability of SD, or 2) that, as is now shown in the rebuttal pdf, model inversion and editing methods work as well for conditions as for image synthesis (see reply to Reviewer cyg3).   Overall, we believe that all these contributions make the paper quite novel, and the FG-DM an interesting new direction for further research by the text-to-image synthesis community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and comments. The rebuttal for individual reviews are posted below each review.  Here we list the summary of the responses.

- In the rebuttal pdf, we added the qualitative results of ablation study with and without attention distillation loss (Figure 1). We also added quantitative and qualitative comparisons to the recent 4M-XL model (Figure 1 and Table 1). We show additional qualitative results using hand-drawn sketches (Figure 3).

- We show that FG-DM is also applicable for inversion and editing of real images and their extracted segmentation maps in Figure 2 of the rebuttal pdf.

- We clarify concerns regarding novelty, computational cost and the inference time for FG-DM.

- We apologize for missing the details on the experiments and clarified them in the rebuttal. We will add the details in the revised version.

Meta-review: This paper proposes FG-DM for better prompt compliance and controllable generation. By using the iterative generation process with generating several visual conditions and images, FG-DM shows high controllability and enabling prompt compliance.

Although this paper has a borderline score (an average score of 5.25), all reviewers reached a positive consensus that the rebuttal addressed their concerns. As I have no significant concerns regarding the paper, I recommend acceptance.
Decision: Accept (poster)