Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis
Authors: Jonghyun Lee
Hansam Cho
YoungJoon Yoo
Seoung Bum Kim
Yonghyun Jeong
Venue: ICLR 2024 poster (2024)
Keywords: Generative Models
Diffusion Models
Image Editing
Image Composition
Primary area: generative models

Abstract: Addressing the limitations of text as a source of accurate layout representation in text-conditional diffusion models, many works incorporate additional signals to condition certain attributes within a generated image. Although successful, previous works do not account for the specific localization of said attributes extended into the three dimensional plane. In this context, we present a conditional diffusion model that integrates control over three-dimensional object placement with disentangled representations of global stylistic semantics from multiple exemplar images. Specifically, we first introduce depth disentanglement training to leverage the relative depth of objects as an estimator, allowing the model to identify the absolute positions of unseen objects through the use of synthetic image triplets. We also introduce soft guidance, a method for imposing global semantics onto targeted regions without the use of any additional localization cues. Our integrated framework, Compose and Conquer (CnC), unifies these techniques to localize multiple conditions in a disentangled manner. We demonstrate that our approach allows perception of objects at varying depths while offering a versatile framework for composing localized objects with different global semantics.

Published as a conference paper at ICLR 2024
COMPOSE AND CONQUER: DIFFUSION-BASED 3D DEPTH AWARE COMPOSABLE IMAGE S SYNTHESIS
Jonghyun Leel,2, Hansam Cho Korea University, "NAVER Cloud otomtom1 03, chosam95, abk ml @korea ac.kr kr
Youngjoon Yoo?, Seoung Bum Kim Yonghyun Jeong"t
[youngjoon, yoo, yonghyun. J com
(a) Gureground/Backgeenn object ordering. Samples are generated via two depth maps (top left, bottom left) of foreground and backgruund objects, with object faithfully occluding each other.
(b) Localized global semantic injection. Given two exemplar images (top lefi, bottom left), semantics of each image are injected onto localized area in disentangled manner.
(c) Composition with Forreground/Backgeee Backgroudd depth maps and global semanties. By mix mutching depth maps exemplar images, CnC generates samples with localized global conditions.
Figure 1: Compose and Conquer is able localize both local and global conditions in 3D depth aware manner. For details on the figure, see Section
ABSTRACT
Addressing the limitations of text as a source of accurate layout representation in text-conditional diffusion models, many works incorporate additional signals to condition certain attributes within generated image. Although successful,
previous works do account for the specific localization of said attributes tended into the three dimensional plane. In this context, we present a conditional diffusion model that integrates control over three-dimensional object placement with disentangled representations of global stylistic semantics from multiple ex- emplar images. Specifically, first introduce depth disentanglement training leverage the relative depth of objects an estimator, allowing the model to iden- tify the absolute positions unseen objects through the use of synthetic image triplets. We also introduce soft guidance, a method for imposing global semantics onto targeted regions without the use of any additional localization cues. Our in- tegrated framework. COMPOSE AND CONQUER (CNC), unifies these techniques to localize multiple conditions in a disentangled manner. We demonstrate that our approach allows perception of objects at varying depths while offering versatile framework for composing localized objects with different global semantics.
First Author. Work done during an internship at NAVER Cloud. Corresponding Co-Authors.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Following the recent progress in text-conditional diffusion models (Rombach et al. 2022 Ramesh et al.) 2022; Saharia et al. 2022 Nichol ct al.) 2021), many subsequent studies have emerged to address their inherent limitation in accurately representing the global layout of generated images. These follow-up works enrich the text-based conditioning capabilities of diffusion models by in- corporating additional conditions such as segmentation maps (Zeng et al.; [023 Goel et al. [023), depth maps (Zhang & Agrawala, 2023 Mou et al. 2023), bounding boxes (Li al. (023), and in- painting masks (Yang et al.] These modifications effectively retain the extensive knowledge encapsulated in the pretrained priors.
Despite these advancements, two primary challenges persist in the current literature. Firstly, while existing models are efficient in generating an object under locally constrained conditions like depth maps and bounding boxes, which inherently capture structural attributes, they confine the genera- tive space to a wwo-dimensional plane. This limitation makes them less adept at handling object placement within a three-dimensional (3D) or z-axis (depth) perspective, and hence vulnerable to generating images without properly reflecting the depth-aware placement of multiple objects. Sec- ondly, the issue of applying global conditions, such as style and semantics, from multiple image sources to specific regions of the target image in a controlled manner has yet to be resolved.
To address the existing limitations local and global conditions and also enhance the capabilities of image generation models, we introduce COMPOSE AND CONQUER (CnC). Our proposed CnC consists of two building blocks: local fuser and global fuser designed tackle each problem. First, the local fuser operates with a new training paradigm called depth disentanglement training (DDT) let our model understand how multiple objects should be placed in relation to each other in a 3D space. DDT distills information about the relative placement of salient objects to the local fuser by extracting depth maps from synthetic image triplets. originally introduced in the field of image composition. Second, the global fuser employs a method termed soft guidance, which aids our model in localizing global conditions without any explicit structural signals. Soft guidance selectively masks out regions of the similarity matrix of cross-attention layers that attend to the specific regions of each salient object.
Figure I demonstrates the main capabilities of our model trained on DDT and soft guidance. In Figure Ta), DDT lets our model infer relative depth associations of multiple objects within one image, and generates objects that are placed in different depths of the z-axis with foreground objects effectively occluding other objects. In Figure [b). we show that by applying soft guidance, our model can localize global semantics in disentangled manner. By utilizing the local and global fuser simultaneously as demonstrated in Figure [c), our model gives users the ability to compose multiple localized objects with different global semantics injected into each localized area, providing a vast degree of creative freedom.
We quantitatively evaluate our model against other baseline models and gauge the fidelity of sam- ples and robustness to multiple input conditions, and demonstrate that our model substantially out- performs other models various metrics. We also evaluate our model in terms of reconstruction ability, and the ability to order objects into different relative depths. We shed light onto the use of DDT, where we demonstrate that DDT dissipates the need to provide additional viewpoints of scene to infer the relative depth placement of objects. Furthermore, we show that soft guidance not only enables our model inject global semantics onto localized areas, but also prevents different semantics from bleeding into other regions.
Our contributions are summarized as follows:
We propose depth disentanglement training (DDT), new training paradigm that facilitates model's 'ss understanding of the 3D relative positioning of multiple objects.
We introduce soft guidance, a technique that allows for the localization of global condi- tions without requiring explicit structural cues, thereby providing a unique mechanism for imposing global semantics onto specific image regions.
By combining these two propositions, we present COMPOSE AND CONQUER (CnC), a framework that augments text-conditional diffusion models with enhanced control over tiree-dimensional object placement and injection of global semantics onto localized re- gions.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
2015 Ho et al., Conditional Diffusion Models. Diffusion models (DMs) (Sohl-Dicksteinn et al.
2020) are generative latent variable models that are trained reverse forward process that grad- ually transforms a target data distribution into known prior. Proving highly effective in its ability to generate samples in an unconditional manner, many following works (Dhariwal & Nichol 2021
Ho et al. 2022 Nichol et al. [2021] Rombach et al., [2022) Ramesh et al. 2022) formulate the diffu- sion process take in specific condition to generate corresponding images. Out of said models,
Rombach et al. (2022) proposes LDM, latent text-conditional DM that utilizes an autoencoder, ef-
fectively reducing the computational complexity of generation while achieving high-fidelity results. LDMs, more commonly known as Stable Diffusion, is one of the most potent diffusson models open to the research community. LDMs utilize a twofold approach, where an encoder maps x to its latent representation 2, and proceeds denoising in much lower, memory-frienndle dimension. Once fully denoised, a decoder maps z to the original image dimension, effectively generating a sample.
Beyond Text Conditions. While text-conditional DMs enable creatives to use free-form prompts, text as the sole condition has limitations. Namely, text-conditional DMs struggles with localiz- ing objects and certain semantic concepts with text alone, because text prompts of large web-scale datasets used to train said models (Schuhmann et al. (202T) do not provide explicit localized de- scriptions and/or semantic information. Addressing this limitation, many works have introduced methods to incorporate additional conditional signals to the models while preserving its powerful prior, e.g. freezing the model while training an additional module. Among these models, ControlNet (Zhang & Agrawala| 2023) and T2I-Adapter (Mou et al., 2023) train additional modules that incor- porate modalities such as depth maps and canny edge images to aid generation of localized objects. However, these models only support single condition, lacking the ability to condition multiple signals or objects. Taking inspiration from ControlNet, Uni-ControlNet (Zhao et al.] (023) extends its framework accept multiple local conditions and a single global condition at once. Whereas the works detailed above all leverage Stable Diffusion as a source of their priors, Composer (Huang al. 2023) operates in the pixel-space. Although being able process multiple conditions once, both Composer and Uni- ControlNee struggle in processing incompatible conditions, or conditions that overlap with each other. They also do not provide methods localize global semantics onto a localized region. In contrast, our approach directly addresses these challenges by proposing two novel methods, depth disentanglement training and soft guidance, which enables the composition of multiple local/global conditions onto localized regions.
3 METHODOLOGY: COMPOSE AND CONQUER
The architecture illustrated in Figure shows the overall framework of our proposed method. CnC consists of a local fuser, a global fuser, and components of a pretrained text-conditional DM. Our local fuser captures the relative z-axis placements of images through depth maps, and our global fuser imposes global semantics from CLIP image embeddings (Radford et al. 2021) on specified regions. Explanations of the local and global fuser are detailed below.
3.1
GENERATIVE PRIOR UTILIZATION
In line with earlier studies that incorporate additional condition signals (Li et al. 2023 Mou al.
2023 Zhang & Agrawala 2023 Zhao et al. 2023). we use the LDM variant known as Stable Diffusion (SD) our source of prior. Specifically, ly, SD utilizes UNet (Ronneberger et al. 2015) like structure, where noisy latent features consecutively pass through 12 spatial downsampling blocks, one center block C. and 12 spatial upsampling blocks. Each block consists of either a ResNet (He et al., 2016) block or Transformer (Vaswani et al. (017) block, and for brevity, we respectively refer to each group of the 12 blocks as the encoder F. and decoder D.. Inspired by Control ZZang & Agrawala (2023) and Uni-ConnrolNNe/2 /hao et al. (2023), the Stable Diffusion architecture is utilized twofold in our model, where we first freeze the full model and clone a trainable copy of the encoder and center block, denoted E" and C'. We initialize the weights of the full model from SD, and the weights of E' and C" from Cnii-onntollNe. The cloned encoder E' and center block C" receives localized signals from our local fuser, which acts the starting point of our model. We detail our model architecture, methodologies of the two building blocks and corresponding training paradigms in the section below.
3

---

Published as a conference paper at ICLR 2024
Local Finer
Extracted
Pg. Stream
Depth
Extraction
8' = !QK" /v3;
V3.00 M
Layer
Convolution
Stream
D
Self- Afff-tteiioon
(M)
((M)
Cross-Attention
Pg.
Of Stream
Imauge Embeekiings
Bg
Stecam
Q
1
0
Global Fisee
1
concod
Vidi
Ytoc
Y4a
Ybx
Figure 2: Model Architecture. Our model consists of local fuser, global fuser, and the cloned encoder/center block [E",C' The input depth maps are fed into the local fuser, producing four latent representations of different spatial resolutions, which are incorporated into E'. The CLIP image embeddings are fed into the global fuser, producing 2 extra tokens to be concatenated with the text token embeddings. Masks M are flattened and repeated to produce M' concat (M)). up( (M)),, which serves as a source of soft guidance of the cross- moss-attention layers.
3.2 LOCAL FUSER
We first provide details on local fuser, which incorporates depth maps extracted form a pretrained monocular depth estimation network (Ranftl et al.) 2020) as our local condition. Specifically, our local fuser serves source of localized signals that incorporated into the frozen SD blocks. We also provide details on depth disentanglemt training, and how synthetic image triplets are leveraged as sources of relative depth placements of objects.
Synthetic Image Triplets. For a model to be able to represent overlapping objects with varied scopes of depth during inference, the model needs to learn recognize different elements obscured by objects during training. Although straightforward a 3D world, informing model about objects that occluded by another in 2D image is non-trivial, due the fact that once the image is captured, any spatial information about objects behind it is forever lost. To overcome this limitation, first adopt a process utilized in image composition (Fang et al. 2019 generate synthetic image triplets, which serves as training samples for our depth disentanglement training (DDT), detailed in the next section. The synthetic image triplets {Iʃ, Tb. .) are derived from single source image I, #xxx33 and is composed of the foreground image I E H the background image Is xxxx and binary foreground object mask M {0, HX The foreground image 1 is derived using the Hadamard product of L = I, @ M, leaving just the salient object of I,. To generate Ib, we utilize Stable Diffusion's inpainting module (Rombach et al. 2022). This is achieved by inpainting the result I, & (1 - M), where M is the binary dilated M. Conceptually, this process can be thought of as inpainting the depiction of 7, without its salient object, effectively letting our model see behind it. For details on this choice, see appendix A.2
Depth Disentanglement Training. Once the synthetic image triplets {L,, 4b, M) are prepared, we proceed to extract the depth maps of T and J, train our local fuser, which we refer to as depth disentanglement training (DDT). Our local fuser incorporates these two depth maps, which passes through its own individual stream consisting of ResNet blocks, and are concatenated along its channel dimension. Apart from previous works that directly fuse different local conditions before entering network, DDT first process each depth map of I and I, in their own independent layers. Reminiscent of early and late fusion methodologies of salient object detection (Zhou et al.) (021), we consider DDT a variant late fusion, where the network first distinguishes each representation in disentangled manner. Once concatenated, features containing spatial information about objects
4

---

Published as a conference paper at ICLR 2024
(o) benuriful dessent waiting be shared by two perple
(b) Diamond Mineeruft black Dacing Bencwards backgronnd
(c) Cus knight, portaii, Cacly desailed annoe, intricate devign, silver, silk, cinematic lighting
Depth disentanglement training. Our model trained on DDT (Left) successfully rec- ognizes that objects portrayed by the foreground depth map (Top left) should be placed closer that the background depth map (Bottom left), and fully occludes objects that are larger. On the other hand, when trained on just the depth maps La (Right), our model struggles to disentangle the depth maps, resulting in either objects becoming fused (samples (a), (c)) or completely ignoring the foreground object (sample (b)).
Figure 3:
in varying depths are extracted along different resolutions by an extraction layer. These features are then incorporated into the cloned and frozen SD blocks, which we provide additional details in our appendix A.2 We formulate DDT to train our model because to represent overlapping objects with varied depths during inference, the model needs to recognize the elements obscured by the salient object. By providing our model with an explicit depth representation of what lies behind the salient object-albed albeit synthetic- our model is able to effectively distinguish the relative depths of multiple objects. Figure demonstrates the effect of training model with DDT compared to training model on depth maps of Z.. as done in previous works. Even though that our local fuser was trained on depth maps that convey only the relative depth associations between the foreground salient object and background depth maps, we find that our model extends to localizing different salient objects.
A.22
3.3 GLOBAL FUSER
While our local fuser incorporates depth maps as a source of relative object placements, our global fuser leverages saft guidance to localize global semantics onto specific regions. We use image em- beddings derived from the CLIP image encoder (Radford et al. 2021) as our global semantic condi- tion. This choice informed by the training methodology of SD, which is designed incorporate text embeddings from its contrastively trained counterpart, the CLIP text encoder. The text em- beddings Yieu are integrated into the intermediate SD blocks through cross atssettttnnioon mechanism. In this setup, the text embeddings serve as the context for keys and values, while the intermediate noised latents act as the queries. Although prior works (Nichol et al. 2021 Ramesh et al.) [022] Huang et al. 2023) have merged CLIP image embeddings with text embeddings within the cross- attention layers of the DM, this approach lacks the provision of spatial grounding information. As a result, the global semantics
conditioned on the entire generated image, lacking precise localiza- tion capabilities. To overcome this limitation, our method leverages the binary foreground mask M used to extract I in synthetic image triplet.
Soft Guidance. In detail, we first project the image embeddings of I, and I, using our global fuser, which consists of stacked feedforward layers. The global fuser consists of separate fore- ground/background streams, where each image embedding is projected and reshaped to N global tokens each, resulting in Yig and Ybg. Unlike our local fuser module, our global fuser doesn't fuse each stream within the feedforward layers. We instead choose to concatenate Yig and Yog directly to Ylexs where the extended context yiul concat(yusu: Aig.ig.AAb..
is utilized in the cross- attention layers of our cloned and frozen modules. Xrg and Aog denotes scalar hyperparameters that control the weight of each token, which are set to I during training. In the cross-attention layers, the similarity matrix S is given by S = (QKT) /Vd) with Q Wq 2, and K = WK Yiull where z is noised variant of z with the diffusion forward process applied steps.
Once is calculated, we apply soft guidance by first creating a Boolean matrix M', which has the same dimensionality of S. Given that S E Rixj, M' is defined as M' = concat(J, (MM), (MM),, where J E 1'xj-2N denotes an all ones matrix, ((M) E BixN denotes the reshaped, flattened and repeated boolean mask M. and 1 - (MM) denotes the complement of (MM) of the same shape. By overriding with the Hadamard product S' = S 8 M', the attention operation softmax(S") V is completed. Intuitively, soft guidance can be thought of masking out parts of
5

---

Published as a conference paper at ICLR 2024
S where z; should not be attending to, e.g. forcing the cross-attention computations of Yig and Yog to be performed only on its corresponding flattened values of z₁ We find that by letting the tokens of Yieu attend to the whole latent and restricting the extra tokens, generated samples are able to stay true to their text conditions while also reflecting the conditioned global semantics in their localized areas, even though that spatial information of M is forfeited.
3.4 TRAINING
LDMs are optimized in noise-prediction manner, utilizing& variation of the reweighted variational lower bound on g (Xn)). first proposed by Ho et al. (020). We extend this formulation and optimize our model with Eq. to learn the conditional distribution of p(zly). where y denotes the set of our conditions of depthmaps and image embeddings accompanied with CLIP text embeddings. As mentioned above. we freeze the initial SD model while jointly optimizing the weights of the cloned encoder F", center block C², and the local/global fuser modules, denoted as 0'.
[||e E(0,0)) (z,, (e,t,y) v)
min = N(,.). 0'
(1)
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. Our synthetic image triplets 1, Ib. . generated from two distinct datasets: COCO- Stuff (Caesar et al.] 2018) and Pick-a-Pic (Kirstain et al. (023). We refer the readers to Section 4.3 for our reasoning behind this choice. The COCO-Stuff dataset, with 164K images, has pixel-wise annotations classifying objects into "things" (well-defined shapes) and "stuff" (background regions). be seen the salient object, and create
We leverage the fact that objects in the things category M by setting each pixel of the indexed mask to I if it belongs to either one of the 80 things classes, and 0 otherwise. Text prompts are randomly chosen from five available for each image. The Pick- a-Pic dataset contains 584K synthetic image-text pairs generated by SD and its variants, and was collected as an effort to train a preference scoring model. Each text prompt is paired with two generated images, and holds a label denoting the preferred image in terms of fidelity and semantic alignment to the given prompt. By only keep the preferred image and filtering out inappropriate content, we end up with 138K image-text pairs. Because Pick-a-Pic doesn't hold ground truth labels for the salient object unlike COCO-Stuff, we utilize salient object detection module (Qin et al.) 2020) to generate M. Combining these two datasets, generate 302K synthetic image triplets [Ty, Jr, M) through the process detailed in Section 3.2
Implementation Details. For depth map representations, we utilize a monocular depth estimation network (Ranful et al. 2020). and the CLIP image encoder (Radford et al. 2021] for global semantic conditions. Although formulated global fuser independently, and finetuning the combined weights lead to faster convergence. During training, images are resized and center cropped to a resolution of 512 x 512. We train our local fuser with the cloned E' and C" for 28 epochs, our global fuser for 24 epochs, and finetune the full model for 9 epochs, all with a batch size of 32 across 8 NVIDIA V100s. During training, we set an independent dropout probability for each condition to ensure that our model learns to generalize various combinations. For our evaluation, we employ DDIM (Song et al., 2020) sampling with 50 steps, and CFG (Ho & Salimans 2021) scale of 7 to generate images of 768 768.
single model, we empirically find that training the local and
4.2 EVALUATION
Qualitative Evaluation. We demonstrate the results of our method compared to other bascline models that incorporate either depth maps as local condition. CLIP image embeddings as a global condition, or both. The baseline models are listed in Table D GLIGEN (Li et al.) 2023) and Con- trolNet Zhang & Agrawala 2023) are trained to accept a single depth map as a local condition. Uni-ControlNet (Zhao et al. 2023) and T2I-Adapter (Mou al. 2023) are trained to accept a depth map and an exemplar image as a source of CLIP image embeddings. In Figure 4 we show quali- tative results of our model compared to models that accept both depth maps and exemplar images as a source of global semantics. Since our model accepts two depth maps and two exemplar im- ages, note that condition the same images for each modality in Figure 4 While other models

---

Published as a conference paper at ICLR 2024
Input Depth Semaniic
Uni- Coutollett
Adspter
TAMA
CaC (Ours)
Pronet
Garman home by the lake, trees, boats, glass windows
Bioluminescent mushrooms
A green glass bottl
Beautiful dancing unicoms
Animal made of semp metal, auto
Coastal village
Figure 4: Samples compared to other baseline models. Compared to others, CnC strikes a balance between the given depth maps, exemplar images, and text prompts.
Depth
Semanties
Depth+Semantics
Model/Condition
CLIP
CLIP
CLIP Score (1)
FID (L)
IS (t)
FID (L) IS (t)
FID (L) IS(T)
Score (t)
Score (1)
GLIGEN
18.887
29.602
25.815
ControlNet
17.303
31.652
25.741
Uni ControlNet
19.277
31.287
25.620
23.632
28.364
24.096
18.945
28.218
24.839
T21-Adapter
20.949
31.485
26.736
35.812
23.254
23.666
30.611
23.938
24.579
CnC
19.804
27.555
25.211
35.178
21.932
22.161
21.318
25.421
24.659
CnC Finetuned
22.257
27.981
26.870
17.254
32.131
25.940
18.191
29.304
25.880
Table 1: Evaluation metrics the COCO-Stuff val-set. We omit the results of semantics and
depth+semantics on GLIGEN and ControlNet due to the models not supporting these conditions. Best results are in bold.
do grasp the ideas of each condition, it can be that model exceeds in finding balance between the structural information provided by the depth maps and semantic information provided by exemplar images and text prompts. Taking the first column an example, Uni-ControlNet of- ten fails to incorporate the global semantics, while T2I-Adapter often overconditions the semantics from the exemplar image, ignoring textual cues such "lake" "boats". Our approach adeptly interweaves these aspects, accurately reflecting global semantics while also emphasizing text-driven details and structural information provided by depth maps. For additional qualitative results, we refer the readers to Figure
Quantitative Evaluation. As our baseline metrics, we utilize FID (Heusel et al.) 2017) and Incep- tion Score (Salimans al., 2016) evaluate the quality of generated images, and CLIPScore (Hessell et al. 202T) to evaluate the semantic alignment the generated images to its text prompts. Table reports the results evaluated on 5K images of the COCO-Stule validation set. It can be seen that while our model with the local and global fuser trained independently and joined during inference (CnC, for brevity) shows adequate performance, our finetuned model excels in most metrics, except for FID and IS of our depth-only experiment. This may be attributed to the fact that while baseline models only take additional depth map with inpainted regions that may not reflect the prior distribution of the dataset. The fact that our model has to process depth maps of conflicting localization information is another possible explanation. Though all models report similar CLIPScores, given their shared generative prior of SD, our finetuned model excels when generating from just an exemplar image, thanks to the integration of soft guidance. We conclude that our finetuned model achieves substantial performance
single depth map extracted from the validation image, our model takes in an

---

Published as a conference paper at ICLR 2024
A stntue a barzai dog
A mythical mannequin dancing in the forest
Sailbeal, oil painting
plate of pasin
purtrail of'a man
Dramatic lurtics flying
A statuc adurable cal
A Pholo of raven
Groeo
giiani pavilion
magician bunny
Figure 5: Qualitative Results. Foregrounddhackgrrund conditions are on the left of each sample.
Ground Truth
CaC (Ours)
Uni-ControINet
T21-Adapter
Ground Truth CnC (Ours)
Uni-ControlNet
121-Adapier
COCO-Suff
Pick-a-Pic
Figure 6: Qualitative reconstruction comparison. Samples are generated using conditions ex- tracted from validation samples of COCO-Stuff (Left) and Pick-a-Pic (Right).
due its additional finetuning phase, which enables the fusers to better adapt and understand each other's conditioning processes. See Section A.3 for results on the Pick-a-Pic validation set.
Reconstruction. We additionally report quantitative reconstruction metrics evaluated on COCO- Stuff and Pick-a-Pic validation sets. listed Table 2 For reconstruction, our model utilizes the depth maps and CLIP image embeddings extracted from the image triplets of ground truth validation images, and baseline models utilize the depth maps and CLIPP image embeddings extracted from than condition per modality. We
the ground truth images, since they do not support adopt LPIPS (Zhang et al. 2018) as our metric of perceptual similarity and SSIM (Wang et al.) 2004) our metric of structural similarity. We also report the MAE of ground truth depth maps and the depth maps extracted from its generated counterpart as an additional measure of structural similarity, extended the z-axis. Apart from the SSIM value of COCO SSufff we find that our model outperforms other models by large margin. As seen in Figure 6 we find that our model is able to faithfully recreate objects in different localized areas while preserving its depth of field. While other baseline models succeed in localizing objects, they struggle in synthesizing the depth perspective, resulting in images looking relatively flat.
Ablation Study. The phenomenon known as concept bleeding Podell et al. 2023) leads to dif- ferent semanties overlap with each other, resulting in unintentional results. Soft guidance allows
8

---

Published as a conference paper at ICLR 2024
Model/Condition
COCO-Stuff
Pick-a-Pic
SSIM(1)
LPIPS(J)
MАE())
SSIM(()
LPIPS(L)
MAE())
Uni-ControlNet
0.2362
0.6539
0.1061
0.2506
0.6504
0.1111
T21-Adapier
0.1907
0.6806
0.1201
0.2238
0.6724
0.1270
CnC
0.2345
0.6621
0.1061
0.2436
0.6431
0.1080
CnC Finetuned
0.2248
0.6509
0.0990
0.2690
0.6216
0.1027
Table Quantitative reconstruction metrics evaluated on COCO-Stuff and Pick-a-Pic val-sets.
1.0
1.0
1,0
0,5
1.0
2,0
Ang
Figure 7: The effect of soft guidance with conflicting semantics. We condition the same depth map for each stream the local fuser, and generate each sample with the prompt "An Igloo". By fixing Aig and increasing Ang. the effect of the background global semantics substantially increases. Soft guidance prevents the two global semantics from bleeding into each other, egg. concept bleeding, effectively maintaining the semanties of the igloo.
global semantics to be conditioned onto localized regions while preventing this undesirable effect. Figure demonstrates this capability, where two contradicting semantics are localized. By fixing Arg to and steadily increasing Ang. the effects of the background semantics are amplified. However, due soft guidance, even the background semantics intensify, the contradicting semantic of the foreground object stays intact. Even though spatial information of M is lost during soft guidance, we find that it faithfully creates a barrier for any semantics from bleeding in. For additional ablations, see Section A.4
4.3 DISCUSSIONS
Dataset Choices. As mentioned in Section[4.1] the two datasets we employ to train our model are very different from another. Namely, images in COCO-Stuff include everyday scenes while images in Pick-a-Pic are fundamentally synthetic, being generated by variants of SD from prompts that transcend any description of real life scenarios. This design choice is intentional: we first point to the fact that most of our baseline models are trained on variants of MS-COCO (Lin et al. 2014). These models show that training only on real images as a method introduce new conditions are adequate, but Kirstain et al. (2023) and Podell et al. (2023) report that COCO zero-shot FID is negatively correlated with human preferences and visual aesthetics of generated images. Although images from COCO and its variants do serve its purpose on introducing new conditions, argue that leveraging another dataset that aligns with the learnt prior of pretrained DMs provides a safety net from prior drifting. By harnessing the detailed ground truth pixel-wise annotations of COCO- Stuff and letting our model Iearn additional representations from its original prior provided by Pick- a-Pi Pic, take advantage of the best of both worlds; providing robust posterior of conditions while staying true to the user- preferred prior of DMs.
5 CONCLUSION & LIMITATIONS
We presented Compose and Conquer (CnC), a novel text-conditional diffusion model addressing two main challenges in the field: siree-dimensionll placement multiple objects and region-specific lo- calization of global semantics from multiple sources. CnC employs two main components: the local and global fuser, which respectively leverages the new Depth Disentanglement Training (DDT) and soft guidance techniques. We show that DDT infers the absolute depth placement of objects, and soft guidance is able to incorporate semantics on to localized regions. Evaluations on the COCO-stuff and Pick-a-Pic datasets illustrates CnC's proficiency in addressing these challenges, as demonstrated through extensive experimental results. Since the current framework limits the number of available conditions and the disentangled spatial grounds to the foreground and background, further decomposition of images into depth portraying primitives and the middle ground to leverage for future work.
leave the

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the Image Vision team of NAVER Cloud for their thoughtful advice and discussions, Train- ing and experiments were done the Naver Smart Machine Learning (NSML) platform (Kim al. 2018). This study was supported by BK21 FOUR.
ETHICS STATEMENT
Diffusion models, type of generative model, have the potential to generate synthetic content that could be used in both beneficial and potentially harmful ways. While our work aims to advance the understanding and capabilities of these models, we acknowledge the importance of their responsi- ble use. We encourage practitioners to consider the broader socictal implications when deploying such models and to implement safeguards against malicious applications. Specifically. the diffusion model we utilize as a source of prior in our work is trained on the LAION (Schuhmann et al. 2021) dataset, a web-scraped collection. Despite the best intentions of the dataset's creators to filter out inappropriate data, LAION includes content that may be inappropriate for models to internalize, such racial stereotypes, violence, and pornography. Recognizing these challenges, underscore the necessity of rigorous scrutiny in using such models to prevent the perpetuation of harmful biases and misinformation.
REPRODUCIBILITY STATEMENT
The source code and pretrained models can be found at httpss//gittub.coo//ommmmm// http:////ooooooooo//oooo/ooommmm 103/compose- and-conquer
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting instance segmentation via probability map guided copy-pasting, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 682-691, 2019.
Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with centuure-nd-apppaare paired diffusion models. arXiv preprint arXiv: /7546, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A feference-free evaluation metric for image captioning. arXiv preprint arXhv:2104.08718, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6835 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-22) 222222222122,,2,,2-22222 2022.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778 2023.
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. Nsml: Meet the mlaas platform with real-world case study. arXiv preprint 1810.0999 2018.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01565 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded ext-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-5221,,,, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:21 2021.
Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, and Liqing Zhang. Mak- ing images real again: A comprehensive survey on deep image composition. arXiv preprint arXiv:2106. 14490, 2021.
Taesung Park, Ming-Yo Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2337-2346, 2019.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.00995: 2023.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern recognition, 107:407 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 1(2):3, 2022.
René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datase transfer. IEEE transac- fions on pattern analysis and machine intelligence, 1233)11623-1677 2020.

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assised Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part 18, pp. 234-241, Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479 -9494,, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint 211.::21111..11114.. 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price. Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Generative object compositing. arXiv preprint 12X00222/..00932, 2022.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference upplications of computer vision, pp. 2149-2159, 2022.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Pro- ceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 18391, 2023.
Yu Zeng, Zhe Lin, Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and Vishal M Patel. Scenecomposer: Any-level semantic image synthesis. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition, pp. 22468-22478. 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to ext-to-imaae diffusion models. arXiv preprint 230::2302.005433 2023.
12

---

Published as a conference paper at ICLR 2024
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 2023.
Tao Zhou, Deng-Ping Fan, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. Rgb-d salient object detection: A survey, Computational Visual Media, 7:37-69, 2021.
13

---

where i == i+ 13. where 2 2 12, i j = 13. Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
EXTENDED RELATED WORK
Image Composition. Image composition (Niu et al. 2021) involves the task of blending given foreground with a background produce unified composite image. Traditional methods typically follow sequential pipeline comprising of object placement, image blending/harmonization and shadow generation. These steps aim to minimize the visual discrepancies between the two image components. With the recent advances in generative models, notably GGANsGoooffellow et al. (2014) and DMs Ho et al. (2020), the image composition challenge has been reframed as a generative task. While GAN-based models have led in terms of the number of research contributions, iiffusion-basee models, as exemplified by works such as ObjectStitch Song et al. (022) and Paint By Example Yang et al. (2023), showcase the potential of DMs one-shot solution for image composition, offering departure from the multi-step traditional methods. However, it is essential to note that our approach diverges from typical image composition. Rather than aiming preserve the distinct identity of the foreground and background, our model utilizes them as localized representations for text and global semantics to fill. Although our work aims solve an inherently different task, we draw parallels to image compositioning in the way we leverage synthetic image triplets and handle the target image be generated.
A.2 DETAILS ON CNC
Details on the Local Fuser. Depth disentanglement training (DDT) leverages synthetic image triplets (T Ib, M) in order to train the local fuser. DDT first incorporates the depth maps of I and I, in their own foreground/backgrounn are concatenated along their channel dimension, and features that incorporate both spatial features about I and Ia are extracted in four spatial resolutions. Each extracted feature subsequently passes through a zero convolution layer, and are finally incorporated into E' through feature denormaliza- tion layers (Park et al., 2019), done in Zhao et al. (2023). The frozen SD receives the localized signals from E' and C at its decoder D, integrated by residual skip connections. Denoting outputs of the i-th blocks of E, D, and C e₁, d, and respectively, and the corresponding outputs of E' and C' as c';, and c', the integration is captured as:
streams, as shown in Figure The features from the streams
{
concat (c+ c', Bj + +e') concat (d,-1, to))
(2)
Details on the Global Fuser. We provide an algorithm outlining the process of incorporating soft guidance to the cross-attention layers to train the global fuser below.
Algorithm Soft guidance for a single training timestep f
Require: Ts. Th. M, Yuex. Aig 1, Aby
1, Zt, CLIP IMAGE ENCODER, GLOBAL FUSER,
(E, C, D)(Frozen layers of SD)
(E,,E,) < CLIP IMAGE ENCODER I,, Ib)
2: (Vig, Vig. + GLOBAL FUSER E., Es)
3: Viall &- concat(yuuss Apy Ang
4: for all cross-attention layers in E, C, do
5:
(Q, K,V) <- (Wo Zt, WK Yiul, Wy Yta))
6:
S + (QKT/Va)
S E Rixj
7:
J 1xx)-2N)
> Initialize J as an all ones matrix (M(M E B
8:
p(M) Reshape, Flatten, Repeat(M)
9:
M' + concat(J, (a, (M), (MM))
S" S- M
11: Z: t softmax(S"). V
end for
14

---

Published as a conference paper at ICLR 2024
Masking
Masking
Mask Esuraciinn
Dilziio
Eapointing
Figure 8: The process of generating our synthetic image triplets.
M
LaMa I
LaMa & Depth
SD 7
SD Is Depch
Figure 9: Comparison of LaMa and SD for inpainting, and its corresponding depth maps. Although images inpainted by LaMa seem to have their salient objects removed, their corresponding depth maps contain artifacts of the salient object.
Inpainting of 7, & .). To generate I, in { M). we utilize variant of SD specifically trained for inpainting, setting the target prompt as "empty scenery, highly detailed, no people". We also test LaMa (Suvorov al. 2022). widely adopted inpainting model, and gauge its suitability as our inpainting module, focusing on the quality of the depth map of Is. Figure we observe that the Is generated from LaMa exhibits certain artifacts that may not align well with the requirements of our pipeline. A notable characteristic of LaMa is that the depth maps of In often retain the shape of the salient object, which could impact the information relayed the local fuser. On the other hand, the SD inpainting module proves adept for the generation of T.. Focusing on To of the first row of Figure can be seen that certain objects that weren't 't once present L, has been generated. This attribute of SD's inpainting module deems attractive to leverage in depth disentanglement training: distill information about the relative placement of salient objects, critical for our model to effectively see objects occluded by the salient object during training. For visualization on the synthetic image triplets generation pipeline, see Figure
A.3 ADDITIONAL RESULTS
More Quantitative Results. We provide additional quantitative results on the Pick-a-Pic valida- tion Table 3 Following the trend on the COCO-Stuff validation set, our finetuned model excels over other models in all metrics with the exception of the FID and IS values from our depth- only experiment. Interestingly, when comparing results from the COCO-Stuff validation shown in Table we observe that the performance rankings of each model remain largely consistent. How- ever, the specific values for FID and metrics deteriorate significantly, while the CLIP Scores see notable improvement. One potential reason for this trend relates to the underlying nature of the pre- trained models, such as Inception- V3 (Szegedy et al. 2016). used in these metrics. While both sets of images being compared in this experiment are synthetic, these models are trained real- world images, inherently capturing real-world image features and patterns. The synthetic nature of the Pick-a-Pic images might diverge considerably from these real-world expectations, thereby influenc-
15

---

Published as a conference paper at ICLR 2024
Model/Condition
Depth
Semantics
Depth+Semantics
CLIP
CLIP Score (T)
CLIP
FID (L)
IS (1)
FID (L)
IS (1)
FID (L)
IS(T)
Score (t)
Score (t)
GLIGEN
22.540
12.733
28.227
ControlNet
21.183
13.685
28.112
24.561
13.260
28.053
28.964
12.809
25.245
22.808
12,006
26.722
121-Adapter
26.262
13.309
28.017
47.996
11.408
25.033
34.698
10.745
26,583
CnC
28.192
11.460
27.347
36.272
10.353
24.301
25,524
11.131
27.109
CnC Finetuned
32.155
11.512
28.274
26.042
12.838
27.681
22.484
12.602
28.094
Table 3: Evaluation metrics on the Pick-a-Pic val-set. We omit the results of semantics and
depth+semantics on GLIGEN and ControlNet due to the models not supporting these conditions. Best results are in bold.
CnC (Ours)
Oni-CentrolNet -ketch-Depch
Uni-ControlNett Canny- Depth
Uni-Controlet Canns+Skettch
Castiie
Bedgmeed
Pangmeed
Ceediiss
Prompt: Golden retriever puppies
Figure 10: Comparing the ordering ability of localized conditions against Uni-ControlNet. Uni- ControlNet reports that the combinations listed the most effective, against other combinations of 7 local conditions.
ing the FID scores. Moreover, even if both datasets under comparison are synthetic, the variance and distribution of features in the synthetic Pick-a-Pic dataset could be distinct enough from typical real-world datasets to lead to the observed differences in FID and IS scores. This highlights the nu- ances associated with evaluating models on synthetic versus real datasets and emphasizes the need for careful consideration when drawing conclusions from such evaluations.
Ordering of localized objects. In Figure 10 we compare our model's 's ability to place objects in front of another through the use of local conditions against Uni-ControlNet Uni-ControlNet is able to take in 7 local conditions, and report that the local conditions pair listed in Figure[TOJan the most robust in handling conflicting conditions, i.e. overlapping objects. Although some samples do show the foreground local condition being placed in front of its counterpart, Uni-ControlNet often fails in conveying a sense of depth its samples, resulting in the two objects be generated on the same z-axis. On the other hand, even if the two depth maps conditioned have relatively same depth values, our model is able consistently occlude overlapping parts of the background depth map.
Additional details on Reconstruction. In Figure we provide additional samples of the depth maps extracted from the reconstruction experiment detailed in Section 4.2 Although depth maps produced by MiDaS does not reflect the true metric depth of an object, comparing depth maps of the ground truth images and the reconstructed images shed light onto how well models reflects the images and depth maps exposed to them during training. While the reconstructed depth maps of baseline models hold the overall shape of object, it can be seen that our model succeeds in capturing the relative depth of regions relative to the salient object. Additionally, we elucidate our choice in selecting MAE as metric for gauging the quality of reconstruction. Although depth maps produced by MiDaS do not predict metric depth as mentioned above, our model and baseline models were trained to generate images based image-depth map pairs. Comparing depth maps
16

---

Published as conference paper at ICLR 2024
Ground Truth
CnC (Ours)
Ami-ControlNet
T21-Adapter
Ground Truth
CnC (Ours)
Uni- ControlNet
T21- Add-Aee
COCO-Stuff
Pick-a-Pic
Figure 11: Qualitative comparison of depth maps extracted from reconstructed images.
COCO-Stuff
Pick-a-Pic
Model/Condition
(L)
(1)
CLIPScore (1)
FID (L)
IS (f)
CLIPScore (1)
Uni-ControlNet (Canny Edge)
17.119
30.440
25.726
21.955
12,469
28.517
T21 Adapter (Canny Edge)
20.051
28.449
25.850
30.547
12.230
28.412
CnC (Canny Edge, Ours)
17.745
29.809
26.283
20.501
12.215
28.786
Table 4: Canny edge evaluation metrics on the COCO-Stufi and Pick-a-Pic val-set. Best results are in bold.
can be thought of how well a model "predicts" depth maps given ground truth depth maps, which in turn gauges how well model has learned the relationship between images and depth maps. In Figure 12 we visualize the relationship between how similar the ground truth depth maps are to the reconstructed depth maps in terms of MAE. Each set randomly chosen from the top 50 pairs with the lowest/highest MAE values. It can be seen that the pairs with the lowest depth map MAE scores directly result in the quality of reconstruction, with the reconstructed images faithfully portraying the relative depth present in the ground truth images. On the other hand, the pairs with the highest MAE scores result in sub-par reconstructed images. Taking the second row of Figure b2) as an example, it can be seen that the reconstructed image fails in capturing the relative depth of the tree and person present in the ground truth image.
A.4 ABLATION STUDY
Ablation on different spatial conditions. The ability of our model to effectively order objects into spatial regions stems from depth disentanglement training, where the spatial information of, and what lies behind the salient object is distilled into their respective streams of the local fuser. To this end, it can be that our model can be trained different types of local conditions, given that the condition holds spatial information of an image. We explore the capabilities of our local fuser, and show the effects of training on canny edges compared to depth maps in Figure [3] Canny edges hold properties and biases different to that of depth maps, in the way that canny edges values are binary, and trade off the ability represent depth with more fine grained details. Because of these properties, it can be seen that while DDT does learn the relative placement of objects even with canny edges, using canny edges has its own pros and cons. Figure 13 (a) and (c) report cases where using depth maps are preferred, while (b) and (d) report the opposite. We find that canny edges often fail generating a sense of depth, as seen case (c) where the apples look relatively flat. However, this property can be preferred when leveraging base images that are flat to begin with. Figure 13 b) and (d) demonstrates such cases, where depth maps of flat base images (such posters and vector graphics) fail to capture spatial information, resulting in sub-par images. We find that DDT able to effectively leverage the inductive biases of given representation, whether be canny edges depth maps, and special cases might call for variants of model trained
17

---

Published as a conference paper at ICLR 2024
GT.
Recon
GT. Depth
Recon. Depth
GT.
Recon
GT. Depth
Recon. Depth
WALL
WELES EAMY
STCAR SOC
OUR
COCO-Stuff Lowest Depth MAE
SOCo Highest Depth MAE
Pick-a-Pic Lowest Depth MAE (c)
Pick-a-Pic Highest Depth MAE (d)
Figure 12: Qualitative comparison of depth map pairs with the lowest/highest MAE values from each dataset. Each set was randomly chosen from the top 50 pairs of the lowest/highest MAE values from COCO-Stuff and Pick-a-Pic aicc, respectively.
18

---

Published as a conference paper at ICLR 2024
(a) A hello kitty mecha in a dark underground facility
(b) of cheese front of propaganda poster
(c) An apple (in front offbehind) another apple
(d) neon sign of dragon (in front ofbehind] a bunny in a cup
Figure 13: Ablation study on canny edge as representation for our Local Fuser. (a) and (c) report cases where depth maps yield better images, while (b) and (d) report cases where canny edges might be preferred.
19

---

Published as a conference paper at ICLR 2024
Inpul Depth
0.2
0.4
0.6
0.8
1.0
Ingut Semantics
1.6
1.8
Atg. Aog
Prompt:
Figure 14: Conflicting text and semantics. We condition the same depth map and exemplar image to each foreground/background stream, and increase Arg, Xbg with the same increments.
different representations. We additionally report the quantitative results of our local fuser trained on canny edges, compared to other baseline models the COCO-Stuff and Pick-a-Pic validation sets Table 4 We follow the same experimental setup of our main quantitative experiments detailed in Section 4.2 We find the variant of model trained canny edges are comparable to other baseline models, resulting in the best CLIPScore for COCO-Stuff, and the best Inception Score and CLIPScore for Pick-a-Pic.
Conflicting Text & Semantics. We explore the use of conflicting text prompts and semantics, and the effect of hyperparameters Àg and Ang used determine the influence of each global semantic. In Figure [4] same depth maps and exemplar images are conditioned to each foreground/backgrounm stream to generate each sample. We fix the text prompt as semantic contradiction to the exemplar image of an empty icy tundra. The effects of steadily increasing the hyperparameter values can be seen clearly, where the value of 1 strikes an adequate balance between the effect of the text prompt and the global semantic. Although the effect of the text prompt is nullified when the hyperparameters are set to larger values, it can be seen that the shape of the original object stays intact due to the combination of the depth map and soft guidance.
castle in the desert", which serves
Additional results conflicting semantics. Figure shows the full effect of the hyperparame- ters Ai and Ang when conditioning two contradicting global semanties. The exemplar image for foreground global semantic is an image of flowing lava, and the background global semantic is an image a snowy field. By fixing the prompt with "A volcano" and increasing each hyperparameter A, the effect of each semantic is steadily amplified. Empirically, and Ang to yields results that are visually harmonious, which aligns with our training configura- tion where each hyperparameter is set to 1. We also provide metric to gauge our model's ability to localize global semantics. Given a foreground and background exemplar image Iig.sm. Ingsem- the corresponding generated image Igen and the mask utilized for soft guidance M, leverage the CLIP image encoder to measure how well each semantic is applied to 1 gen- The mask is uti- lized to create two images, Igen @ M and Igen (1 M). and fed into the CLIP image encoder to create two image embeddings. These embeddings are then compared with the embeddings of the original exemplar images Jigsem and Jey via cosine similarity, and the average cosine similarity can be utilized as metric of how well each semantic applied into each region of /gen- A high average cosine similarity implies that each semantic has been faithfully applied to each region via soft guidance. This characteristic can be seen in Figure 16 where two contradicting semantics of a hot air balloon and a coral reef, are localized. The average cosine similarity steadily increases as Abg increases, which reflects how much the effect of the coral reef increases can be seen in the generated images. The average cosine similarity drops sharply Xbp = 2.0, due to the model over saturating the effect of the coral reef, resulting in low fidelity sample.
observe that setting both Are
20

---

Published as a conference paper at ICLR 2024
0.0
0.5
1.0
1.5
2.0 AEE
0.0
0.5
1.
1.5
2.0
Abg
Prompt: volcano
Conflicting semantics. The foreground global semantic image an image of flowing lava, and the background global semantic image is an image of snowy field. We fix the text prompt to "A volcano" and demonstrate the effects of hyperparameters Arg and Ang-
Figure 15:
21

---

Published as a conference paper at ICLR 2024
CosineSim.
CLIP
CLIP
Averuee
+
/2
Cosine Similarity
CosinaSim.
CLIP
CLIP
-A
Cosine Similarity
Additional results the effect of soft guidance with conflicting semantics, and its spective average cosine similarity scores. We condition the same depth image for each stream in the local fuser, and generate samples without conditioning a prompt.
Figure 16:
M
Versin
Virsion 2 (Ours)
Versien
Foreground Image
Background Image
Figure 17: Depth map Ablations. We find that our model generalizes well to different versions of depth maps. Version refers depth maps extracted from I,. Version refers to depth maps extracted from /y. Version 3 refers to depth maps extracted from M 8 depthnapp(T)).
22

---

Published as a conference paper at ICLR 2024
Depth map Ablations. The local fuser incorporates depth maps extracted from the synthetic im- age triplets Ig,A,. during DDT. However, we find that our model is also able to generate samples conditioned on different versions of depth maps, in which Version refers to conditioning the foreground stream the local fuser on the depth map extracted from s. instead of f.- Version 2 refers to conditioning via how the local fuser was originally trained with DDT, or conditioning the depth map of f. Version refers to conditioning the local fuser on M & depthmmpp(Zy). or masked version of the depth map of f. Pedagogically, the difference between version 2 and can be thought of as whether masking with M is done before after the extraction of the depth map of I,. Although our model is trained on depth maps of version 2, the sample generated by version 1 shows that our model has learned the relative positioning of objects through its interaction with the background depth map. Because the foreground depth map of ver- sion retains additional depth cues of the ground, the generated sample retains this information and places parts of the drum set over the ground. Interestingly, we also find that version also generates high fidelity samples, even though this form of depth maps are never found in the wild. This also can be attributed to the relative depth estimation capabilities of our model paired with the background depth map.
demonstrate in Figure T7
23


Reviews:
[{"review_id": "bWwlBMZynd", "reviewer": "Reviewer_taqR", "summary": "The authors propose the Compose and Conquer (CnC) network that achieves 3D object placement and successfully integrates global styles and local conditions. To this end, the authors first propose depth disentanglement training (DDT) which disentangles the foreground and background depth and processes them with independent layers before fusing them together. Moreover, the paper also involves a novel soft guidance block that efficiently combines global and local conditions. Thorough qualitative and quantitative evaluations demonstrate the design and the effectiveness of the proposed method.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the proposed paper can be summarized as:\n1. The paper is well-written and easy to follow\n2. The proposed DDT and soft guidance modules are novel and effective, demonstrated by both qualitative and quantitative results\n3. Evaluations are comprehensive and showcase better results than existing SOTA methods", "weaknesses": "The weaknesses of the proposed paper can be summarized as:\n1. Type of conditions. (1) Is the model capable of applying different types of conditions? (2) Is the model capable of applying two different conditions simultaneously while recognizing the 3D relations? I am rather interested in these different situations especially considering that the authors only employ depth in the submitted paper. More examples or scenarios would be appreciated.\n2. Image triplets. There is no visualization of the prepared image triplets for training. I am curious regarding the quality of foreground image, background image and foreground mask. It's also especially important to analyze the inpainted background image and how it would negatively affect the training and final outcomes.\n3. Qualitative results. (1) What are the prompts for examples in Figure 3? (2) Through the visualization in Figure 3, 4 and 5, it's interesting to see that the final generated images do not fully reflect the foreground depth condition. Meanwhile, the background depth map is often ignored through the qualitative results.\n4. No limitations and societal impacts are discussed in the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXlL6m7tJ3", "reviewer": "Reviewer_iDhm", "summary": "This paper presents a method for controllable text-to-image generation. The method learns auxiliary modules on top of a pre-trained Stable diffusion model, and introduces a novel training scheme to facilitate compositional image synthesis given two depth images that represent the foreground and background. Further, foreground and background styles are controlled by separate images thanks to a localized cross-attention mechanism. The qualitative and quantitative experiments demonstrate that the proposed method outperforms several baselines in terms of image quality, image-text alignment and foreground-background disentanglement.", "questions": "- The illustration of soft guidance in Figure 2 is confusing. I personally prefer color coding of the attention maps to highlight the regions influenced by different tokens.\n\n- Details about the reconstruction experiments (Figure 6) are lacking. It is unclear from the text what is the exact evaluation procedure. Also the MAE values reported in Table 2 is not meaningful, again because MiDaS does not predict metric depth. Please include qualitative comparison between the input and reconstructed depth maps.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper studies the composition of control signals in controllable text-to-image diffusion. Unlike previous approach which takes a single control image, the proposed method allows the conditioning on two depth images. This provides a means to separately control foreground and background image content. The method also enables localized control of image styles using exemplar images. To the best of my knowledge, compositional image generation remains a challenging problem, and this paper demonstrates one feasible solution to the problem by engineering pre-trained diffusion models.\n\n- The paper presents a novel training scheme to instill depth awareness into the diffusion model. Training of the method relies on RGB images and their foreground / background depth maps which are not readily available. To this end, the paper introduces a simple strategy to create synthetic training data from single-view image datasets. This is key to the success of the proposed method and may be of interest to the image synthesis community in a broader context.\n\n- The method allows localized control of image styles using exemplar images. The key idea is to limit the extent of cross-attention so that tokens representing an exemplar image only contribute to a local region in the image. Many works have used attention maps to localize objects or control their shapes and appearance. This paper for the first time uses attention maps to control (local) image styles.\n\n- The experiments demonstrate superior qualitative and quantitative results. The proposed method outperforms several strong baselines in image quality and image-text alignment while supporting broader applications.", "weaknesses": "- Calling the model \"depth-aware\" is misleading. I would rather say it learns to compose two spatial layouts and generate a coherent image. Using the teaser figure as an example, the cat can appear either in front of or behind the cake given the same depth maps, and similarly, the castle can either occlude or be occluded by the mountain. In other words, the exact ordering of objects is not induced by the depth maps. This phenomenon is likely because the depth produced by MiDaS is scale and shift invariant (i.e., it is not metric depth, and background can appear closer than foreground).\n\n- Since all that matters is generating a coherent image, I would imagine that other types of spatial conditions (e.g., segmentation masks for both foreground and background) can work equally well if used for training. I encourage the authors to test this hypothesis, and design additional ablation experiments to fully reveal the behavior of their model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JUozWVTxI0", "reviewer": "Reviewer_FnjA", "summary": "This article proposes two methods to develop a diffusion model that merges the capability to manipulate the three-dimensional positioning of objects with the application of disentangled global stylistic semantics from exemplar images onto the objects.\n\nSpecifically, the paper introduces depth disentanglement training, which makes the model realize the  3D relative positioning of multiple objects by disentangling the salient object depth and the background object depth for the fusion of the condition during training.  In the meantime, this work presents a technique called soft guidance, which imposes the mask information into cross-attention mechanism to facilitate apply global semantics onto targeted regions without specific local localization cues.", "questions": "1. In my understanding, the size of S is $i\\times j$, where i is the number of queries, and j is the number of keys; the size of $W_K \\dot y_{full}$ is $j \\times C$ and the size of $W_Q \\dot z_t$ is $ i \\times C$. However, I am puzzled as to why $j$ needs to be greater than $2N$.\n\n2. Additionally, I am uncertain about whether the mask should be applied along the dimension of $C$. It perplexes me that the mask is utilized on the calculated similarity rather than during the actual computation of similarity.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written in a clear and coherent style, presenting ideas in a manner that is easily comprehensible. Additionally, most figures in the paper effectively visualize and reinforce the concepts discussed.\n\n2. As stated in the paper, this work is the first to leverage the disentaglement of images to salient object depth and impainted (unseen) background depth map for training the relative depth aware diffusion model.  The soft guidance technique is also novel for applying the global semantics to specific localizations.\n\n3. The adequate experiment supports the effectiveness of the model. Both the qualitative and quantitative comparisons demonstrate that the model can control the relative placement of the objects and the effectively prevent the concept bleeding.", "weaknesses": "1. Correct me if I have misunderstood. I'm confused about the details of the soft guidance. I understand that the work wants to leverage the mask map to selectively impose the foreground embedding and background embedding for the cross-attention mechanism. However, I'm uncertain as to whether this process should take place during the computation of similarity or subsequent to it. I would appreciate it if the author could elucidate the specific dimensions and computational details related to both the cross-attention mechanism and the soft guidance technique, to enhance reader comprehension. Please refer to the questions for additional context on my confusion.\n\n2. Apart from the standard metrics used for evaluating generative models, I wonder if there exist specific metrics that can accurately assess the model's capability to control the three-dimensional placement of objects and localize global semantics, as these are the primary objectives of this study. While the Mean Absolute Error (MAE) between the ground truth depth map and the depth maps derived from the generated images may offer some insight into the model’s proficiency in 3D object placement, I am curious about how we might effectively gauge its ability to localize global semantics. Could there be other metrics or methods of evaluation that address this second capability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)