Generating compositional scenes via Text-to-image RGBA Instance Generation
Authors: Alessandro Fontanella
Petru-Daniel Tudosiu
Yongxin Yang
Shifeng Zhang
Sarah Parisot
Venue: NeurIPS 2024 poster (2024)
Keywords: RGBA generation
scene composition
diffusion models
Primary area: diffusion_based_models
TL;DR: We present a multi-layer approach for text-to-image diffusion models that improves fine-grained control over object attributes and layout by generating isolated RGBA images and blending them into detailed composite scenes.

Abstract: Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. Controllability can be improved by introducing layout conditioning, however existing methods lack layout editing ability and fine-grained control over object attributes. The concept of multi-layer generation holds great potential to address these limitations, however generating image instances concurrently to scene composition limits control over fine-grained object attributes, relative positioning in 3D space and scene manipulation abilities. In this work, we propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. To ensure control over instance attributes, we devise a novel training paradigm to adapt a diffusion model to generate isolated scene components as RGBA images with transparency information. To build complex images, we employ these pre-generated instances and introduce a multi-layer composite generation process that smoothly assembles components in realistic scenes. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes. Through multi-layer composition, we demonstrate that our approach allows to build and manipulate images from highly complex prompts with fine-grained control over object appearance and location, granting a higher degree of control than competing methods.

Generating Compositional Scenes via Text-to-image RGBA Instance Generation
Alessandro Fontanella" University of Edinburgh
Petru-Daniel Tudosiu Huawei Noah's Ark Lab
Yongxin Yang Huawei Noah's Ark Lab
Shifeng Zhang Huawei Noah's Ark Lab
Sarah Parisot" Microsoft Research
Abstract
text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering, Controllability can be improved by introducing layout conditioning, however existing methods lack layout editing ability and fine- grained control over object attributes. The concept of multi-layer generation holds great potential to address these limitations, however generating image instances concurrently to relative positioning in 3D space and scene manipulation abilities. In this work, we propose novel multi-stage generation paradigm that designed for fine-grained control, flexibility and interactivity. To ensure control over instance attributes, we devise novel training paradigm adapt diffusion model generate isolated scene components as RGBA images with transparency information. To build complex images, employ these pre-generated instances and introduce a multi- layer composite generation process that smoothly assembles components in realistic scenes. Our experiments show that our RGBA diffusion model capable of generating diverse and high quality instances with precise control over object attributes. Through multi-layer composition, we demonstrate that our approach allows to build and manipulate images from highly complex prompts with fine- grained control over object appearance and location, granting a higher degree of control than competing methods.
composition limits control over fine aie-grained object attributes,
Introduction
The development of text-to-image generative diffusion models has allowed the generation of high- quality synthetic images guided by textual prompts. However, achieving desired image quality and properties inherently requires tedious and meticulous prompt engineering [35]. As image content solely described using textual description, prompt content has be carefully crafted, notably taking into account the models' struggles to understand subtleties of language (e.g. counting, object attributes, negation or spatial relationships)
[16]. Additionally, minor prompt modifications can lead to substantial modifications of generated image content, further increasing prompt crafting tediousness.
An intuitive way of improving controllability generated content is by providing alternative image descriptors, e.g. using image layouts which describe the location of specific objects in image as bounding boxes. Several approaches have been proposed to inject layout information in the generative process. Training based methods (24] 54] adapt introduce new model weights to introduce object coordinates as conditioning, while 8 5] leverage cross-attention manipulation
Work done while at Huawei Noah's Ark Lab
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and inference time optimisation. While substantially increasing control over image structure, these solutions still generate all components as a single denoised image. This limits the user's ability to finely control object attributes, and layout adjustments requires re-generation with limited content preservation guarantees. These issues can potentially be addressed with image editing techniques. This, however, can require expensive data construction and training [4) while the displacement or resizing of objects (essential for layout manipulation) remains a very challenging type of edit that few have explored [9 31. This challenge can be attributed to the fact that objects and backgrounds have to be regenerated, while at the same time ensuring appearance preservation with the original image.
The emerging layer-based generation paradigm 18 limitations. The idea is to represent images as multi layer stacks, where different image components are generated on separate layers. This has two key benefits: 1) instances can be generated more precisely as separate images with individual prompts, 2) separating image components simplifies image manipulation tasks, as only the relevant layers will be modified. Nonetheless, layer-based methods typically suffer from the fact that layers are not fully generated, as they are collapsed to a single image in the later stages of diffusion process. Generating all components jointly can affect the ability to achieve fine-graines control of layers attributes, control their relative positionnnn in 3D space (as layers are collapse with no hierarchical structure), ensure a smooth composition of all image component, and achieve layout manipulation abilities with strong content preservation (as layers and composite image influence each other).
49 40]
has shown promise in addressing these
In this work, we seek address current limitations through a multi-stage generation process that is designed for fine-graines control, flexibility and interactivity. We propose to build complex scenes by first generating individual instances as RGBA images, then iteratively integrating them in multi-layer composite image according to specific layout. The ability to generate individual instances allows to finely control instance appearance and attributes (e.g. colours, patterns and pose), while their layered composition allows to easily control and modify positioning, scale and ordering. In particular, our two-stage approach provides intrinsic layout and attribute manipulation abilities with strong content preservation. To achieve our goal, we firstly train a diffusion model capable of generating RGBA images, i.c. images with alpha transparency information. Directly generating isolated instances, c.f. generating and extracting them with a segmentation model, ensures better transparency masks and more fine-grained control of instance attributes. Our RGBA generator is obtained by fine-tuning latent diffusion model (LDM) using RGBA instance data from the recently released MuL.An dataset (47). In order to incorporate transparency information in the generative process, we devise transparency-awee contrast with the contemporary transparent generation method of (57). which implicitly encodes transparency information in the null space of the VAE carry out standard LDM fine-tuning, we explicitly integrate transparency in our generation and training process. Our VAE is trained so as to disentangle RGB and alpha channels, ensuring colour and detail preservation in RGB reconstruction. Our LDM is then fine-tuned with a novel training paradigm leveraging our disentangled latent space, that allows a conditional sampling-driven inference where alpha and RGB latents are sequentially denoised with mutual conditioning. Finally, we leverage our RGBA generator to build composite images with fine-grained control over object attributes and scene layout. We build scenes via a multi-layer noise blending solution, where each instance is associated to a specific image layer. Crucially, each instance is integrated in the scene one layer at a time, yielding increasingly complex image layers so ensure scene coherence and accurate relative positioning. This contrasts with pre-existing multi-layer works which assemble all instances at once 38 49 and is uniquely afforded by our RGBA generation process. By manipulating latent representations in early stages of the denoising process, we are able to achieve high degrees of precision and control, while at the same time generating smooth and realistic scenes. An overview of our complete methodological process is shown in Fig. [
training procedure for both VAE and diffusion model. In
We provide a thorough evaluation of our RGBA generator's is capabilities, showing that we are able to generate highly diverse objects and precisely control their attributes and style, outperforming alternative solutions. In addition, our scene composition experiments show that we consistently outperform state of the art layout controlled methods, while highlighting our unique ability to generate and manipulate complex scenes with strongly overlapping objects. In summary, our main contributions are the following:
2

---

We propose a multi-layer generation framework designed for fine- fine-graing grained controllable generation that grants user control over instance appearance and location, with intrinsic scene manipulation capabilities.
We introduce novel fine-tuning paradigm for pre-trained diffusion models to generate indi- vidual objects as RGBA images with transparency information. We propose a disentangled training strategy that relies on mutual conditioning of RGB and alpha channel latents.
We develop a multi-layer scene compositing strategy based on noise blending that allows to build and manipulate complex scenes from multiple instance images. By combining RGBA generation with scene compositing, we are able to generate images with accurate attribute assignment and relative object positions.
2
Related Work
Text-to-image and transparent generation. GLIDE [32] first proposed text-to-image generation with diffusion models by adopting classifier-free guidance through text. While GLIDE trains the text encoder jointly with the diffusion prior using paired image and text data, Imagen 4 employs
a frozen large language model as the text encoder. More recently, Latent Diffusion Models (LDM) trained in latent space have increased in popularity, given their lower computational requirements [T]. Stable Diffusion incorporates an adversarial objective to learn the latent representation and enhance the authenticity of generated images and introduces rross-attention for text conditioning. While the U-Net 42) architecture was originally the most popular for latent diffusion models, recent works are converging towards the use of Diffusion Transformers (DiT) [36), which operate on latent patches. For example, Pixart-a [6] has demonstrated remarkable image generation capabilities, while reducing the computational requirements to 10.8% of Stable Diffusion-vl. 1.5's training time. Leveraging these models to generate instances with transparency information has been rarely explored. Text2Layer (60) generates foreground and background images separately, defining foreground as the ensemble of all salient objects and predicting the foreground alpha mask separately from RGB pixels. Developed concurrently, LiyerDiff (18) generates image components as separate images, providing the transparency masks as input. This technique tends to generate partial instances, occluded areas are not generated. Also contemporary, LiferDiffusion generates transparent images by encoding transparency in the null space of the pre-trained VAE, allowing to fine-tune diffusion model using standard techniques. While able to generate high quality instances, this implicit transparent modelling can lead to inconsistent results, without guarantees that transparency will be achieved. In contrast, explicit disentangled latent space allows for more control and guarantees transparent images outputs.
Image editing and scene controllability The increasing popularity of diffusion models has high- lighted the limitations of working solely with text based control. We identify two main areas of research seeking to increase generative controllability: image editing and layout controlled genera- tion. Image editing can be achieved by fine-tuning on dedicated dataset [4), constrained sampling
with CLIP (39 losses 20 21
45 17
50 mask guided inpainting [30 59 II 52 training- free
cross attention manipulation 12
48 and inference-time optimisation (.. Despite achieving
impressive results for replacement and local modification tasks, image editing often struggles with more complex manipulations such as moving, rescaling and removing, especially on non isolated instances. By modifying pre-generated content, editing methods additionally have limited layout control capabilities. Alternatively, precise layout control during generation has been explored by introducing additional conditioning. GLIGEN 24. ReCo 54 and Boxdiff (5T) achieve image generation from layouts given as bounding boxes specified by the user. ControlNet (58) allows to incorporate several forms of image based conditioning, such as sketches, depth maps, or canny edges. These techniques afford high generative controllability. but lack layout editing capability. Composite 2 and based generation techniques (38] 49 40] offer a promising avenue, where individual scene components are associated with unique prompts and assembled in scene with bounding boxes or instance specific masks, leveraging the diffusion model's intrinsic priors to build coherent scenes. However, generating scenes and all individual components jointly requires compromising between instance quality, attribute accuracy and scene coherence, reducing flexibility and control. We build from layered approaches, and use multi-layer paradigms to iteratively integrate pre-generated RGBA instances in increasingly complex images, affording us precise control over layout, instance attributes and allowing us to focus scene composition quality solely during the composition step.
3

---

RGB reconstruction
MRGB RGG
Decode
YRGB
YRGB
RGB
t-
Me 2.
aittuaton
Diflusion
Sample
ya
-Ya
[-1
Y/
A reconstrucion loss
(a) VAE latent representation and training
(b) Diffusion model inference with mutual condi- tioning
SOTA layer-wise methods
Ours
Generate
Instances
Flatten layers
Diffusion model
Oitfusion model
Diffusion model
Inversion
Pre-generated RGBA Instances
(c) Our scene composition process compared to state the art multi llyter methods
Figure I: Overview of key components of our proposed methodology.
Methods
3.1 Preliminaries
Diffusion models. Diffusion models learn to reverse a forward diffusion process, where images are iteratively converted to near isotropic Gaussian noise over T timesteps as y/: v(1 B,)ut-11 + 3te, where e ~ (0, 7) and 3 describes a noise schedule. Diffusion models are trained to reverse the process by learning predict the noise Eø that added at a specific timestep:
C Ey.,,t ||e eo ev C)Il
Where C' is conditioning variable, typically text prompt embedding. Starting from isotropic Gaussian noise yr at inference time, diffusion models generate images through the reverse iterative denoising process, where Yo is the final denoised output. At given timestep of this backwards diffussoon process, the pre-trained model predicts the noise eg that updates current noisy image ye to move to the next timestep - 1. In this work, we consider latent diffusion models (LDMs), where diffusion process is carried out in the latent space of a Variational AutoEncoder (VAE)
Alternatively, the pre-trained model can be leveraged to convert an image to Gaussian noise by iteratively adding predicted noise eø (2,,, according to the same noise schedule (44) This inversion process provides an estimation of the input noise 2T that allows to obtain via the backwards diffusion process. This technique particularly relevant to image editing tasks and noise blending methods like ours. To differentiate between both processes, we use ye to refer to generated latents and 24 to refer to inverted latents in the remainder of this paper.
RGBA representation. An RGBA instance image is modelled as a four channel image / E R Ru, where the additional Alpha channel describes the level of transparency of RGB pixels. The alpha channel takes values in (0, 1]. where 0 correspond to full transparency. Transparent pixels for which we not have RGB information are set black (RGB (0, 0, 0)).
3.2 RGBA Instance Generation
A simple method for creating instances with transparency involves using a text-to-image model to generate an initial image, then applying image matting techniques (55) to extract the transparency alpha mask. This reliance on automated matting can yield inaccurate masks, particularly when dealing with complex object structures and image backgrounds. In addition, this approach reduces controllability. object attributes can bleed in the background content rather than the instance itself

---

Alternatively, we propose to natively generate RGBA instances using a generative diffusion model. To this end, we developed transparency aware training procedure for a pre-trained LDM that focuses on the interaction between RGB and alpha channels. This section will first introduce our training dataset, then detail our fine-tuning approach for the L.M's VAE and diffusion model.
16
Training data. We employed 87989 instances from the MuLAn dataset [47]. a novel dataset consisting of automatically generated RGBA decompositions with a diverse array of scenes, styles and object categories, and 15791 instances extracted from a variety of image matting datasets with high quality masks, More details about the datasets are provided in Appendix A.I
RGBA VAE. We train our RGBA VAE following the training procedure of [4]] comprising a L1 reconstruction loss Lrecon (I, ((E(I))), a perceptual loss CLPIPS (()())). discriminator based adversarial loss Lado (D((())))))) and a Kullback Liebler (KL.) loss between the estimated latent distribution and normal distribution W(x : 0, 1). However, we make key modelling and training adjustments in order to accommodate for the new alpha transparency information.
Firstly, the additional channel is simply taken into account by replacing and retraining the input and output layers of the model. Secondly, we observed that learning a joint RGBA latent space leads to entanglement of RGB and alpha channels, affecting generation capability of diffusion models trained in this latent space. We address this challenge by disentangling representations in the latent space: our VAE predicts two separate distributions, N(x JRRGB ERGB) and M(x Mas Ea), each associated with a separate KL loss. While we do not explicitly model the separate distributions to encode RGB and alpha channels respectively, it is natural disentanglement of the data.
Lastly, the VAE in [1 is trained with very small KI. regularisation (e.g. weight factor of WKL = 10-6 on the KL. loss), so as to focus on reconstruction quality. In our setting, we observed that this deviation from standard VAE training was harmful to generative ability, yielding dark and highly contrasted images. In contrast, training our VAE with large regularisation (WKL = 1) noticeably improved image quality, while at the same time enforcing disentanglement more strongly (see Appendix AA.3 for visual comparisons). The perceptual loss is also computed separately for RGB and alpha channels and the two components LLPIPS (IRGB, ((())) RGB) and LLPIps(Ia, are then averaged.
RGBA Diffusion Model. In the second stage, we fine-tune the LDM on our instances datasets. Our VAE fine-tuning keeps the dimension of the original LDM latent space (4 channels and 64 x 64 spatial dimension), allowing us to directly fine-tune without architecture adaptation. However, in our case, the latent space also encodes information on the transparency layer. When sampling from model, we seek to exploit the mutual dependency between RGB and alpha channels. In particular, given noised latents fG and yf for RGB and alpha channels respectively at timestep 6, information contained in y RGB could be employed to inform the update from yf to yf-1 and vice-versa versa. In order to train the network to leverage this conditional information, we modify the training procedure of the LDM. Given RGB and yo we sample two different Gaussian noise vectors eCch and e" and use them to compute y; RGB Vt-1
RGB ./f Vf-1 with the forward process of the diffusion model and t randomly sampled in [o, 7]. with T number of training steps. Our network is trained to predict (CRGB ea) jointly given one of the following input: RGB ufr), RGB 1) or RGB vft).
This training procedure unlocks the ability to perform conditional sampling. Given (yRGB ). we can alternate between updating the alpha component yf-1 and then use (yz nGB yf-1) to update the RGB component, obtaining (y 1/- Alternatively, we can update the RGB component first and use it to condition the alpha update, but observed that the former approach worked best in practice. When sampling from diffusion models, it is common to use fewer sampling steps than at training time for faster image generation (53) Therefore, in order make our training regime more flexible and applicable to a variety of sampling strategies, we additionally use pairs (y RGB >rr),, ) and (yf RGB yf) as conditioning input in the second half of training iterations, with & randomly sampled in 10, 1).
3.3 Multi-Layer Noise Blending for Scene Composition
We consider that we have an image layout available, and generated K instances (Tx, M& using our RGBA generator, where J3 refers to RGB values and M₁ is the transparency alpha mask. We represent a layout as a collection of bounding boxes, where each box is associated with a specific instance: L = {[CZK, Cyk, Wk, hx], for & E ..... K}. We design our scene composition approach
5

---

as multi-layer noise blending process, where instances are sequentially integrated in intermediate layered representations. As a result, we concurrently generate K 1 images (sorted as background. and K composite images with increasing number of instances). While more costly, we observed that generating layered images affords more flexibility, better control over relative positions of instances and yields more natural compositions, An algorithmic overview is provided in Appendix B
We consider the latent representation =f of instance image I5, and compute its noisy representation Fy at all diffusion timesteps t [1. using DDIM inversion 44] To generate our layered composite image, we first initialise our K + 1 images with the same noise vector yr. Inspired from compositing ( and layered (38 image generation methods, we combine noisy image representations for the first n timesteps of the diffusion denoising process. After carrying out the denoising update at time I, and before moving on the next time step, we sequentially update noisy images as follows:
y/ 1/2 k-1 -1-110 -1) + x mg for k E []
where my is the instance alpha mask Mz downsampled to the latent space and yf is the background image generated concurrently. Equation Saaatiio@nbiile builds layered images by iteratively injecting new instances in the noisy images at the desired locations. Carrying this process only the first n generative timesteps enables to precisely control scene composition and instance appearance, while at the same time allowing for a smooth and realistic composition. The smaller is, the more freedom is given to the generative model to adapt scene content. We further detail two optional additional mechanisms.
Background blending, In the noise blending procedure described above, the background is generated independently from the rest of the image. That can reduce blending quality as generated background elements can be incompatible with instance locations. We propose address this limitation by introducing background blending. We inject appearance information from the composite image to the background as:
1
y - ve m + 2 70( + +v )
(3)
where m² is the union of all instances alpha masks. The background area of the composite image gets adjusted throughout the generation timesteps to accommodate instances neighbourhood. Equation 3 has two benefits: 1) it ensures stronger consistency between the background layer and final composite image, and 2) allows the background image to be generated with stronger awareness of instances. This blending process can be carried out for the first b steps of the composition process.
Increasing cross-layer consistency. In situation where intermediate layers are needed, and n is small, consistency can be enforced on the composite image by applying Eq. 2 to yK for nis subsequent timesteps: yt"h y.¹ (1 m²) + - my for k [1, -K--- 1].
Scene editing. Our design intrinsically allows scene manipulation and editing easily. Once instances are pre-generated, scene content can easily altered locally by replacing instances, or modifying instance location by providing new bounding boxes. The modified image can easily be generated following our scene composition process with fixed seed, without any additional constraints.
4 Experiments
4.1
RGBA generation
Our RGBA generator is fine-tuned from pre-trained PixArt-a model 66. As baselines, we compare quantitatively and qualitatively to Stable Diffusion 1.5 (SD-1.5) and PixArt-ax adding 'on a black background" to the captions to replicate instance generation. We additionally compare to both models combined with the Matte Anything (MA) matting algorithm (55), our our reimplementation of Text2Layer 160), and LayerDiffusion (57] In all approaches we used 100 steps. Our RGBA generator leverages the conditional sampling approach described in Sec. 3.2
In Fig. and we show qualitative examples of instances obtained with our RGBA generator. Fig. 2 shows that we are able to generate instances across different styles and to follow fine-grained attributes in prompts. Fig. 3 provides visual comparisons to our transparent baselines. We can observe that our approach is able to generate realistic instances following the instructions given. Text2Layer shows lower image quality and excessive transparency, while LayerDiffusion struggles to follow prompt details, such as image style. Combining SD with Matting allows to achieve reasonable segmentation of the instances generated, while when applied to PixArt-o it can sometimes struggle

---

Figure 2: Our model generalise different styles and to follow detailed instructions. Top row: 'a cartoon style frog', 'a digital artwork of an anime-style character with long, flowing white hair and large and expressive purple eyes in a white attire', 'a stylised character with traditional Asian hat, with a red and green pattern', 'a man with a contemplative expression and a neatly trimmed beard', Bottom row: 'a woman with classic, vintage style, curly hair, red lipstick, fair skin in a dark attire', 'a bird mid-flight with brown and white feathers and orange head', hand-painted ceramic vase in blue and yellow colours and with floral pattern', 'a woman with short, blonde hair, vivid green eyes, white blouse, with a gold necklace featuring pendant with gemstone'.
(a) PixArt-o (b) Pixart-a++ (c) SD-v1.5 (d) SD-v1.5 (e) Text2Layer (f) LayerDiffu- sion
(g) Ours
MA
MA
Figure 3: Instances generated with the captions: 'a majestic brown bear with dark brown fur, its head slightly tilted to the left and its mouth slightly open', 'an Impressionist portrait of a woman', portrait of a young man, depicted in a blend of blue and red tones'.
to correctly identify and segment the main object of the image, especially when dealing with artwork content. On top of attributes bleeding in the background, this highlights how unreliable matting can be for instance generation purposes.
We quantitatively evaluate the quality of the generated instances with the Kernel Inception Distance (KID) 3 a metric that was proposed to reduce bias the Fréchet Inception Distance, especially when evaluated on a low number of samples. We compute it using features from the last convolutional layer of the Inception v3 model and considering only the RGB channels of the generated images. In order to evaluate the alpha mask generated by the RGBA generator, following (60), employ masks

---

Table 1: Quantitative evaluation of our RGBA generator. We measure KID for instance generation quality, IoU (Jac- card) between the alpha masks and ICON segmentation, and CLIP Score for the caption/image similarity. t: our reimplementation. best results high- lighted in bold.
J
IeU1
SD-v1.5
0.0839
N.A.
18.33
PicArt-n
0.0447
N.A.
18.10
SD-v1.5
00711
18.40
PioAitt
18.03
Texe2
0.0500
0.526
18.21
(a) Standard rrain-bb Standard sam- pling
(c) Ours
LayerDiffasion
0.0772
0.320
18.39
ing
Ours
0.0150
0.892
18.49
Ablation experiments
Figure Our proposed training and sampling approaches
No paired training
0.0181
0.814
18.41
(c) improve results obtained with standard training (a) and standard sampling (b).
No conditional sampling
0.0152
0.887
18.49
obtained with ICON with PVT backbone [6]] ground truth, and compute the Intersection- -verr-- Union (IOU) with the predicted alpha masks. Lastly, to evaluate the correlation between the captions and the content of the generated instances, we employ the CLIP Score [13), which computes the cosine similarity between image and text features using the CLIP model (ViT-B/16 variant) [39]. The results obtained are shown in Table [ We can see that our approach obtains the best results for all the metrics, with KID of 0.0150, IoU of 0.892 and CLIP score of 1 18.49. PixArt-a has the second-best KID of 0.0447. However, the result is worsened when applying MA. Pixart MA also has the second best IoU of 0.8111. SD-v1.5 has KID of 0.0839, result which is improved to 0.0711 I by adding MA. The same model also achieves IoU of 0.649. Text2Layer has KID of 0.0500 and loU 0.326, while LayerDiffusion obtains KID of 0.0772 and IoU of 0.320. The latter's lower performance could be attributed to unreliable transparent generation, as instances can be generated without transparency due to the modelling strategy, affecting average performance. With regards to CLIP scores, they relatively close to each other, with SD-v1.5 + MA obtaining the second-best score of 18.40.
Ablation studies. In order to explore the impact of our novel training procedure, we train two RGBA LDMs 1) without our conditioned training procedure (i.e. predicting noise without conditioning from different timesteps) and 2) without our conditional sampling inference where RGB and alpha are mutually conditioned for generation. Performance of these models is reported in Table [ bottom rows. We can see that the largest gains in performance are obtained with our novel training procedure, while conditional generation results more subtle improvement. shown in Fig. our conditional sampling allows to correct small details and critical areas of the alpha masks.
4.2 Scene Compositing results
Generation details. We build composite scenes three steps: RGBA instance generation, 2) layout building: we draw one bounding box per generated instance and rescale instances accordingly, 3) noise blending generated instances according to a global prompt. We keep parameters consistent across all scene compositions unless specified otherwise: guidance scale 2.5 (RGBA) and 4.5 (Blending), guidance rescale 0.25, noise blending steps n = 30, background blending b = 20 steps and consistency regularisation n, 10 steps. Both instances and composite image are generated over 50 steps. We use the Pixart-a model for compositing.
Baselines. We compare our approach to state of the art scene composition methods focusing on layout control and interactivity. Our first baseline Pixart-a [6] to provide intuition into the complexity of requested prompts and limitation of relying solely on text controls. Next, we compare to GLIGEN [24], which learns additional cross attention layer to integrate bounding boxes and bounding box text descriptors as generation conditioning. Finally, we compare our approach to multi-layer methods MultiDiffusion 2 and contemporary approach Instance Diffusion (49] for completeness. Similarly to ours, these approaches rely on noise blending techniques build composite scenes. Besides

---

Layout
Pixart-o
GLIGEN
MultiDiffusion
Inst. Diffusion
Ours
(a) apple next yellow tea mug with a smiley face on a kitchen table.
(b) A white dog lying purple couch next to yellow cushion with flower pattern, with swan painting in the back.
(c) a blue and yellow car in front of a red bus covered with white flower pattern on the road with white airplane with orange stripes in the sky.
(d) fox front of golden unicorn and blonde woman a long green dress walking away in meadow.
Figure 5: Visual examples of scene composition results. RGBA instances are highlighted in bold.
RGBA instance pre- mre-generation, crucial difference, in contrast to our multi-layer approach, is the use of noise averaging for overlapping objects. All methods use the same global prompt, GLIGEN and Instance diffusion leverage our bounding box layout and RGBA instance prompts as descriptors, while DiftiDiffusion uses our instances alpha masks and captions as well.
Due to the inherently interactive nature of our and competing approach (layout design, instance ordering), we provide visual results in Fig. 5 comparing all methods across different scene com- positions. For fair comparison, we generate competing methods with multiple seeds and select the best result. We provide results for simple scenes with unusual attributes (a) and complex scenes with overlapping objects (b,c.d) (Additional results and ablations are available in Appendix E.2 and F.2). While GLIGEN is capable of accurately reproducing the desired layout, it often fails to assign the right attributes to objects (e.g. blue apple, orange plane, swan painting) and struggles with highly overlapping objects (fox and unicorn). In contrast, Multidiffusion more accurate in terms of attribute assignments, but struggles to handle overlapping objects. This can be attributed to the noise averaging process, which fails to integrate a notion of instance ordering like our multi-layer approach. Instance diffusion achieves performance closest patterns, attributes and relative positioning (swan painting behind the couch, white bus with red flower pattern, woman walking away). With RGBA instance generation and multi-layer noise blending, we are able to accurately assign object attributes and follow the required layout, while successfully building smooth and realistic scenes.
but still struggles with complex
Scene manipulation results. Finally, we evaluate our method's potential for scene manipulation. To optimise cross scene consistency, we set b = 0 (remove background blending) and set common generation seed across all versions of the same scene. The manipulations we consider here are: attribute modification, instance replacement, and layout adjustment. We note that the first two tasks

---

Ouis
Enstaacc Dillission sion
Original image
A smiley face sad face
Mug + a green bowl with sur- rescale apple prised face
Move
and
Move and
Remove mug
rescale apple
Qurs
Instunce Dithusin
Original image
overalls red shirt
green yarm blue yarn
and
and
Remove bear
rescale cat
rescale cat
and yarn
Figure 6: Visual examples of scene manipulations compared to Instance Diffusion. Our layer-based approach allows to replace instances or modify their positions.
require RGBA generation of instances. We highlight that we do introduce any new explicit scene preservation or image editing technology in this experiment, therefore evaluating our method's potential for scene manipulation and controllability. We compare our results contemporary method Instance Diffusion, which has been highlighted to possess similar editing capabilities 49 Results are reported Fig. G showing that we are able to control and modify image content easily while maintaining strong consistency across different versions of the scene, without explicitly enforcing content preservation. This highlights the strong potential of multi-layer approaches facilitate the development of image editing methods. We can see that we achieve substantially stronger scene preservation compared instance diffusion, which can generate entirely different images and instances when modifications are too strong.
5 Conclusion
In this work, we introduced a novel multi-layer strategy to scene composition, that focuses on interactivity and fine-grained control. To achieve this, proposed adapts diffusion models to generate transparent images, through channel disentanglement and condi- tional sampling. To build composite scenes from RGBA instances, further present multi-layer compositing strategy that concurrently generates increasingly complex scenes through cross-layer noise blending. Extensive experiments show that we able generate diverse, fine-grained RGBA instances, and assemble them in complex scenes, achieving precise control and high flexibility over scene structure. One key limitation of our approach is the independent generation instances. in- creasing the challenge of assembling them in a coherent scene. Future work will explore conditioned RGBA generation intrinsically generate coherent scenes, as well RGBA editing methods to further improve fine-grained control over scene content.
training paradigm that
10

---

References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.1422 14211, 2024.
[2] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[3] Mikolaj Binkkwski,, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans, arXiv preprint 1rXiv:1801.000 2018.
[4] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprini wrXiv:2211.0900000 1.09000, 2022.
[5] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560- 22570, 2023.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:23 10.0042, 2023.
[7] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level obrjetleee image customization. arXiv preprint arXiv:2307.09481, 2023.
[8] Guillaume Couairon. Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2174-2183 2023.
[9] Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36: 16:162222-62233 2023.
[10] Xiaonan Fang, Song- Zhang, Tao Chen, Xian Wu, Ariel Shamir, and Shi-Min Hu. User- guided deep human image matting using arbitrary trimaps. IEEE Transactions Image Processing, 31:2040-2052, 2022.
[11] Alessandro Fontanella, Grant Mair, Joanna Wardlaw, Emanuele Trucco, and Amos Storkey, Diffusion models for counterfactual generation and anomaly detection brain images. arXiv preprint arXiv:2308.002065 02062, 2023.
[12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 01626, 2022.
[13] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.088 08,08, 2021.
[14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598 2022.
arXiv preprint
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00666 106.09685, 2021.
[16] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open- compositional text-to-image generation. Advances Neural Information Processing Systems, 36:78723-78747, 2023.

---

[17] Nisha Huang, Yuxin Zhang, Fan Tang, Chongyang Ma, Haibin Huang. Weiming Dong, and Changsheng Xu. Diffstyler: Controllable dual diffusion for text-driven image stylization. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[18] Runhui Huang, Kaixin Cai, Jianhua Han, Xiaodan Liang, Renjing Pei, Guansong Lu, Songcen Xu, Wei Zhang, and Hang Xu. Layerdiff: Exploring exxt-guided multi-layered composable image synthesis via layer-collaborative diffusion model, 2024.
[19] Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson WH Lau. Modnet: Real-time trimap-free portrait matting via objective decomposition. Proceedings tfe AAAI Conference on Artificial Intelligence, volume 36, pages 1140-1147, 2022.
[20] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
[21] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. arXiv preprint arXiv:2210. 10960, 2022.
[22] Jizhizi Li, Jing Zhang, Stephen J Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. International Journal of Computer Vision, 130(2):246- 266, 2022.
[23] Jizhizi Li, Jing Zhang, and Dacheng Tao. Deep automatic natural image matting. arXiv preprint arXiv:2107.07235 2021.
[24] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511-22522 2023.
[25] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5404-5411, 2024.
[26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[27] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778. 2022.
[28] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.0510,, 2017.
[29] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[30] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[31] Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusiin: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307 02200.02222,,.. 2023.
[32] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 12. 10741, 2021.
[33] Nicholas Guttenberg. Diffusion with offset noise, 2023. https: htwps://www.... www.:wwwwwwsaabb.commmom .rgg/ blog/ diffusionnnwii witoonwwwttttttte ott.fusioooottttt-offffsetooeeee
12

---

[34] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zeoo-shot paaggett-immge translation. ACM SIGGRAPH 2023 Conference Proceedings. pages 1-11, 2023.
[35] Nikita Pavlichenko and Dmitry Ustalov. Best prompts for text-to-image models and how to find them. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2067-2071, 2023.
[36] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 41 4195-4205, 2023.
[37] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.0/952. 2023.
[38] Zipeng Qi, Guoxi Huang, Zebin Huang, Qin Guo. Jinwen Chen, Junyu Han, Jian Wang, Gang Zhang, Lufei Liu, Errui Ding, et al. Layered rendering diffusion model for zero-shot guided image synthesis. arXiv preprint arXiv:2311. 18435, 2023.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748 8763. PMLR, 2021.
[40] Jiawei Ren, Mengmeng Xu, Jui-Chieh Wu, Ziwei Liu, Tao Xiang, and Antoine Toisoul. Move anything with layered scene diffusion, 2024.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assiste intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 11.36479-36499 2022.
[44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02552, 2020.
[45] Nikita Starodubcev, Dmitry Baranchuk, Valentin Khrulkov, and Artem Babenko. Towards real-time text-driven image manipulation with unconditional diffusion models. arXiv preprint arXiv:2304.044 043444, 2023.
[46] Yanan Sun, Chi-Keung Tang, and Yu-Wing Tai. Human instance matting via mutual guidance and multi-instance refinement. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2647-2656, 2022.
[47] Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Fei Chen, Steven McDonagh, Gerasimos Lampouras, Ignacio lacobacci, and Sarah Parisot. Mulan: A multi layer annotated dataset for controllable text-to-image generation. arXiv preprint arXiv:2404.0277 200.029940 2024.
[48] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930. 2023.
[49] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra, In- stancediffusion: Instance-level control for image generation, 2024.
13

---

[50] Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7677-7689, 2023.
[51] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-fre free box-constrained diffu- sion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7452-7461, 2023.
[52] Shaoan Xie, Yang Zhao, Zhisheng Xiao, Kelvin CK Chan, Yandong Li, Yanwu Xu, Kun Zhang, and Tingbo Hou. Dreaminpainter: Text-guided subject-driven image inpainting with diffusion models. arXiv preprini arXiv:2312.0377) 2023.
[53] Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. arXiv preprint 17X17:2402.17376. 2024.
[54] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14255,14255,.00 2023.
[55] Jingfeng Yao, Xinggang Wang. Lang Ye, and Wenyu Liu. Matte anything: Interactive natural image matting with segment anything model. Image and Vision Computing, page 105067, 2024.
[56] Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang. Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1154-1163, 2021.
[57] Lymin Zhang and Maneesh Agrawala. Transparent image layer diffusion using latent trans- parency. arXiv preprint arXiv:24022 17113, 2024.
[58] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[59] Xin Zhang, Jiaxian Guo, Paul Yoo, Yutaka Matsuo, and Yusuke Iwasawa. Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model. arXiv preprint arXiv:2306.07556 2023.
[60] Xinyang Zhang, Wentian Zhao, Xin Lu, and Jeff Chien. Text2layer: Layered image generation using latent diffusion model. arXiv preprint arXiv:2307.09781, 2023.
[61] Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, and Ling Shao. Salient object detection via integrity learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3738-3752, 2022.
14

---

A RGBA training details and more experiments
A.1 Datasets
The MuLAn 44 dataset consists of RGBA decompositions for over 44.000 RGB images from the LAION and COCO datasets. Developed through a training- pipeline, MuLAn decomposes monocular RGB images into series of RGBA layers, comprising of background and isolated instances. Our training RGBA images are instances that were automatically extracted from natural images, making this noisy but diverse dataset excellent solution for first fine-tuning step. In order to assemble the matting dataset, we employed data from AIM-500 [23], HIM-2k [46], Interactive
Human Matting [10], RWP-636 (56). PPM-100 [9] AM-2k 22
A.2 Training details
Images were resized with bilinear interpolation then centre cropped to obtain a 512 512 image.
We observed that this particular transformation normally preserved the majority of the content of the instances. The VAE was trained for a total of 23 epochs (with each epoch comprising of 42908 steps performed with a batch size of 2) employing the Adam optimiser with starting learning rate of 4.5e -6 3 = 0.5 and 322 = 0.9. The discriminator adversarial loss was introduced after 50k steps with weight of 0.5.
For the LDM fine-tuning we followed similar pre-processing, but additionally performing horizontal flipping with = 0.5. To train the model for classifier-free guidance, we dropped the caption conditioning with p 0.1. We first perform first stage of fine-tuning on MuLAn for 200k iterations and a second stage employing 50% of the data from MuLAn and 50% from the dataset obtained combining different matting datasets for 80k steps. This is because MuL.A is larger and offers a wider variety of subjects, while the matting dataset, primarily comprising humans and animals, offers finer, -anaatand alpha masks. In particular, we employed the Adam W optimiser (28] with a starting learning rate of le and cosine scheduler. and trained in mixed precision with a batch size of 12. The VAE and LDM were initialised with the weights from PixArt-a, while the text encoder (4.3B F-an-T5-XXI. large language model) was kept frozen.
To maximise model expressiveness, detailed captions were automatically generated for instances using the Llava-NeXt model (26) with the following instruction: "Write detailed caption of image. ". Since the captioning model operates only in the RGB space, transparent pixels were included in the captions as descriptions of black backgrounds. We employed Phi-3 [ to remove all such references by using the following instruction "You are a talented ghostwriter given a text remove any mentions of the black background while keeping the text as close as possible to the original.".
Resizing and cropping training data can cause the generative model also produce truncated instances. To limit this behaviour, inspired by [37], we store crop coordinates, indicating the pixels cropped from the top-left corner along the height Grop and width Cleft- These coordinates are then mapped with a sinusoidal embedding, similar to timesteps embeddings (6], which is followed by two-layer MLP featuring SiLU activations. At inference time, we set (Ctop: feftt) (0, 0). In addition, we employed Offset Noise, which improves the contrast in images created by the model 33.. At inference time, we observed improvements employing timestep trailing (25,, classifier free guidance [14] with the rescaling proposed by [25]. In particular, we observed the best results with guidance scale 2.5 and rescaling parameter 0.25 as shown in Sec. F.
For Text2Layer, we trained the 1-A autoencoder and U-Net LDM following 660 on the MuLAn dataset, as the data used in [60] is not available. When sampling from Pit and our model,
we employed DPM-Solver (29). while for SD and Texr2Layer we employed PNDM (7), default
recommended parameters were used for LayerDiffusion.
A.3 VAE Ablation
In Fig. we visualise the impact of different training paradigms for our RGBA VAE on generated samples. We visualise images generated with a diffusion model fine-tuned on different VAE latent spaces. We can see that failing to disentangle the alpha and RGB channels (with a joint KI. loss, or low regularisation weight) yields poor quality images with high contrast.
15

---

(b)
Figure 7: Images generated with our LDM fine-tuned in the latent space of VAEs that were trained with single KL loss with weight 10e 6 (a), separate KL losses each with weight 10c (b), and 2 KL. losses with weight (c).
A.4 Further Experiments
When fine-tuning the VAE, we also experimented with increasing the latent channels from four to eight in order to enhance the expressiveness of the latent space. However, we observed that while this modification improves the VAE reconstruction capabilities, it makes fine-tuning the LDM significantly more challenging and costly. Therefore, we decided to preserve the original latent dimension and encode both RGB and transparency information with disentangled approach.
For the LDM,, we experimented with parameter-efficient fine-tuning techniques such LoRA [5). but observing an excessive decrease in image quality, we finally opted for full fine-tuning of the network.
B Noise Blending Algorithm
To aid reader comprehension, we provide an algorithmic description of in Algorithm
our multi-layer noise blending
C Broader and Societal Impact
Our work contributes positively to several domains. Artists, designers, and content creators benefit from more precise control over their generated images, enhancing creativity and productivity. Additionally, the technology about design, art, and computer graphics. Improved text-to-image generation can also make graphic design more accessible to individuals without extensive training in the field.
be used in educational tools to help students and professionals learn
However, as with any generative model, there is a risk that the technology could be used create disinformation, generate fake profiles, produce harmful content. There is also risk that generated images could be used infringe on personal privacy, particularly the model is misused create realistic but fake images of individuals. Furthermore, the generated images could be used in ways that compromise security, such as creating counterfeit documents or misleading imagery for fraud.
To mitigate these risks, we propose several safeguarding measures. Releasing the model in a controlled manner can ensure that only verified and responsible users have access to it. Conducting thorough audits to ensure that the model does not perpetuate amplify existing biases is also important. Maintaining transparency about the capabilities and limitations of the model, and establishing accountability protocols for its use, will help manage its impact. Providing resources and guidelines for ethical use, along with training for users on the potential societal impacts of the technology, can further promote responsible usage. Finally, developing features that can detect and prevent the creation of malicious content, such as watermarking techniques and usage restrictions, will enhance security measures.
16

---

Algorithm 1: Multi-layer scene composition
Inputs: K pre-generated RGBA instances (Tt, Ms).
A bounding box based scene layout & = {[er,, cyj,, wk: ha), for h E EI, K).
latent diffusion model (VAE &, diffusion model D)
T generation timesteps,
Random noise 7 E (O I)
Number of blending timesteps 71, background blending timesteps b, cross-layer consistency timesteps n,
Output: K multi-layer images (background, K images with increasing number of instances)
Generation set-up
Rescale (Tg, M& according to [CTR, Cyk, Wk, hg]]
2g < E(T²)
me < Downsample Mr to latent space dimension
Compute σ at all diffusion timesteps & E [1, using DDIM inversion
Initialise all images with same Gaussian noise: y e
Composite scene generation For = T to 0:
Loop timesteps
> Predict noise update with diffusion model 2 &- Update according to noise scheduler and Eo (y/ 1)
co (v. t) + D
If t b:
-Backrround blending optional step
y t 36-1 m + [ + v.= (1- a1-m" m²)
If
Scene composition
For = 0 to K:
8 +- Vi-1
mm) xt-1 m.
Elif t 2 n + ns:
cross-layer consistency optional step
For
K:
v t V (1 mx) + y mk
D Limitations
RGBA generation. Our RGBA generator is trained on relatively small dataset, and doesn't ' leverage training data with intrinsic transparency information. We would expect substantial image quality improvement by increasing training data volume and quality. As our approach requires full fine-tuning of the diffusion model (including the VAE), interaction with pre-traines diffusion models is more challenging, as models operate in different latent spaces. Finally, our mutual conditioning sampling increases computation complexity inference time. As our model was trained handle different sampling strategies, we leave it up to the user to determine whether the fine- ine-grained improvements observed are needed for generation tasks.
Scene compositing. By combining image inversion and multi-layer composition, we achieve much greater flexibility and control the cost of increased computational and framework complexity. Future work will seek to simplify and automatise the generation pipeline, notably through automated instance prompt and layout generation. Generating instances independently can make scene composition challenging and require prompt engineering to obtain the right instance. Combining RGBA generation with techniques like ControlNer, or conditioning RGBA generation with background images has the potential to address these limitations.
17

---

E
Additional Visual results
E.1
RGBA Samples
In Fig. g we present additional examples of generated RGBA instances. Our model is able to successfully cover broad range of subjects, including realistic portraits, stylised figures, inanimate objects, and fantasy characters.
E.2 Scene Composition Results
In Fig. we provide additional examples of scene compositions, highlighting our model's ability to successfully build complex scenes with fine-grained attributes and instance ordering. Limitations of competing methods are observed here well, with multi-diffusion struggling with overlapping instances and relative positioning (couch image, flower vase image) and GLIGEN struggling to assign the right attributes to instances (turtle image, marble table in couch image). Contemporary method Instance Diffusion shows similar performance to ours here, as attributes and patterns are simpler.
For completeness, we additionally provide experiments with LayerDiffusion [57], aiming to generate similar scenes. However, the model does not allow to use layout to guide the generation process, with instances being generated based on background content only. Our experiments, using publicly available code, have shown that instances tend to be generated in the middle of the image, making complex composition difficult. Results are provided in Fig. 14
F Parameter analysis
F.1 RGBA generation: classifier-free guidance parameters
In order to find the best parameters for the classifier- free Guidance Scale (GS) and Guidance Rescaling (GR), we perform grid search on validation data, measuring KID to evaluate image quality. As can be observed in Fig. I0] we obtain the best result (lowest KID), with GS = 2.5, GR = 0.25
F.2 Scene composition parameters
We provide visual analysis of the impact of our key noise blending parameters the generated image output. In Fig. 11 we visualise different images when modifying n (number of timesteps where inverted instances noise is injected) and (number of steps where background noise is blended with the composite image). We can see that n is the most crucial parameter: when n is too low, no control over image content is achieved and the scene is solely controlled by the prompt input. Increasing increases faithfulness to the original RGBA instance appearance, at the cost of reduced blending quality. We observed that setting instance attributes and positioning while at the same time adjusting appearance to fit the scene composition better.
30 yields the smoothest blending, maintaining
Parameter b has a subtler impact, adjusting the generated background image to blend with the scene layout more accurately. We can see notably in Fig. 11 that the blanket under the cat is adjusted and looks more realistic when b > 0. However, setting too high risks introducing instance information in the background area, strongly affecting the overall generated scene.
Next, we visualise the impact of parameter nt which enforces consistency between the composite scene and individual layers' instance areas in Fig. T2 For this experiment, we visualise intermediate layers as well, to highlight cross-layer consistency. We can see that setting n, >> 0 substantially improves consistency of instance appearance between individual instance layers and the final compos- ite scene. While we considered introducing a similar consistency regularisation between individual layers, observed reduced composition performance with this kind of strategy.
Finally, in Fig. 13 we visualise our method's 's consistency across multiple random seeds compared to competing methods Multidiffusion (2). GLIGEN 24] and Instance Diffusion 49). By first generating instances and compositing them, we achieve strong scene consistency when modifying the random seed during our blending procedure. In contrast, GLIGEN and Instance Diffusion friggle to maintain attributes (including object types), while Multidiffusion often struggles with overlapping instances and their relative positioning.
18

---

Figure 8: Sample RGBA instances. We are able generate a wide variety of subjects.
19

---

layout
Pixart-cr
GLIGEN
MultiDiffusion
Inst. Diffusion
Ours
(a) a green ball of yarn in front of ginger cat sleeping next a teddy bear wearing overalls sitting on purple blanket.
(b) red leather armehair next to vase of tulips top of round marble coffee table in front of black leather couch in grey carpeted living room.
(c) A white beachball in front of tortoise beach with red sailboat.
(d) book with drawing of sunflower the cover in front of bouquet red roses in blue
Figure 9: Additional scene composition results. RGBA instances and their attributes are bolded in the prompt.
Kernel Inception Distance
G5 2.5
G5 4.5
G5 7.5
0.040
0.035
0.025
0.020
0.6
Guidance Rescaling
Figure 10: KID obtained with different combination of the guidance scale and guidance rescaling parameters. Setting GS = 2.5, GR = 0.25, we achieve the best results on validation data.
20

---

n=0
n=10
n=20
n=30
n=40
n=50
(a) Impact of adjusting instance blending parameter n on scene composition. For this experiment, b 20 and
0.
b=0
b=10
b=20
b=30
b=40
(b) Impact of adjusting background blending parameter b scene composition. For this experiment, 30 and n, 10.
Figure 11: Influence of scene composition parameters generated scene content.
n. = 0, background layer
ng O, toddy bear layer
n, = 0, cat layer
n, 0, final layer
10, background
layer
ny 10, teddy bear layer
n₂ 10, cat layer
n 10, final layer
Rs
20, background
layer
20, teddy hear layer
20, cat layer
20, final layer
Figure 12: Influence of layer consistency parameter n, over all generated layers. For this experiment, set 30 and b 20.
21

---

(a) GLIGEN
(b) Multidiffusion
(c) Instance Diffusion
(d) Ours
Figure 13: Impact of changing the random seed scene composition consistency. Image caption: blue and yellow car in front of a red bus on the road with an orange airplane in the sky.
22

---

(a) apple next yellow tea mug with smiley face on a kitchen table.
(b) white dog lying on a purple couch next to yellow cushion with flower pattern, with swan painting the back.
(c) and yellow car front red bus on with an orange airplane in sky.
(d) fox front of golden unicorn and blonde woman long green dress walking away in meadow.
Figure 14: Visual examples of LayerDiffusion scene composition results. RGBA instances are highlighted in bold. From left right: Layout image, LayerDiffusion images layer per layer, and our composite image.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We present the experiments justifying our claims in the Experiments Section and in the Appendix
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss methods limitations in the conclusion of the main paper, and in dedicated section the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[NA]
Justification:
We do not present theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all the details needed to reproduce experiments in the Methods, Experiments sections and Appendix. All datasets used in this work are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
dataset used is available the following url: watere/aaaaae//maaaaaaateeeeerre Open sourcing of our code will depend on internal approval.
Justification:
The
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Poolliiicciiooioiiiioioocooiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are discussed in the Experiments Section and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification: We not able to perform multiple runs due compute limitations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In the Experiments and Appendix Sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We made sure to follow the Ethics Guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This is discussed in a dedicated broader and societal impact section in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Potential safeguards are discussed in the broader and societal impact section in the Appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the sources of our data and pre-trained models throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No released new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "ecyl2FoW9P", "reviewer": "Reviewer_GbJP", "summary": "This paper describes a diffusion model based method for layered text-to-image generation with RGBA masks (transparency information). This is a useful approach when aiming to generate complex images with many objects. To that end, the base VAE as well as the latent diffusion model are adapted to handle another channel and fine-tune to a corresponding dataset. The method is evaluated against base models and visual results show convincing results.", "questions": "-", "limitations": "-", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The method is sound and well-engineered for this problem using sota building blocks.\n- The paper is well-written and structured.\n- It is easy to follow the paper.\n- The visual results are convincing that the method works.", "weaknesses": "My main concern is regarding evaluation. While the method demonstrates that it works (visually), and quantitative evaluation shows that it performs better than off-the-shelf text-to-image models, there is a complete lack of comparison against methods that are more relevant to that particular task. The paper mentions many related works, but evaluation is only against base models, and comparisons to alternative methods are missing. \n  - Naive Text2Layer extension to multiple layers instead of just two\n  - No editing / inpainting methods (discussed in related work) are used for comparison\n  - LayerDiff https://arxiv.org/abs/2403.11929\n  - No comparison to controllability methods such as ReCo, BoxDiff etc.\n  - No comparison to Layered rendering diffusion model for zero-shot guided image synthesis\nBecause of this, it is difficult to judge the technical contributions and novelty of the proposed method. Mainly because it seems like a straightforward adaptation of an existing pipeline through fine-tuning. More justification and experimental evidence are required to demonstrate and ablate the necessary design choices to go from an existing backbone to the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments that increase the quality of our work. We are encouraged\nby the positive comments that our work is useful, our methodology is sound, and our visual results\nare convincing. We now address the main limitations raised: 1) limited contribution and novelty,\nand 2) experimental comparisons.\n\n**Novelty:** we have aimed to design a novel controllable generation method with fine-grained control\nand interactivity in mind. To this end, we propose to first generate RGBA assets, and then assemble\nthem in a composite scene. Generating images with transparency information is far from trivial (as\nmentioned by RwSp1) as the concept of transparency is foreign to diffusion models. We introduce\nnovel modelling and training methodology for this purpose. In contrast with LayerDiffusion [1],\nwhich injects transparency information in the null space of the VAE to facilitate fine-tuning, we\nexplicitly model transparency through disentanglement of RGB and alpha channels. Explicitly modelling transparency allows for more precise and nuanced generation of RGBA images, and guarantees\nthat we generate images with a transparent background, which is not the case with LayerDiffusion.\nText2Layer (reference [55]) is a substantially different approach as well, requiring triplets of background, foreground and saliency maps for training; and predicting all components together. Crucially,\nthis approach doesn’t generate individual instances, but a foreground comprising all instances.\nOur scene composition leverages these RGBA assets in a novel multi-layer approach. In contrast\nwith existing multi-layer methods which generate layers in parallel before merging them, we build\na multi-layer composition process where each layer is dependent on the lower image layer, by\nintegrating new instances in an increasingly complex image one layer at a time. This process allows\nto smoothly integrate each component individually (rather than trying to blend all instances together\nat once) and is uniquely afforded by our RGBA asset pre-generation. This further allows scene\nmanipulations very easily, as pre-generated RGBA instances can easily be moved, replaced, and\nhave their appearance frozen.\n\n**Evaluation:** We have made efforts to compare to highly relevant, widely used, state of the art works,\nfor both RGBA generation and scene composition, at the time of submission. For the former, we\nreimplemented Text2Layer as the closest published work capable of RGBA generation. We highlight\nthat extension of this work to multiple layers is far from straightforward, as disentanglement of\nforeground instances through their saliency segmentation approach is non trivial, and is beyond the\nscope of this work. We agree with the reviewer that nearly contemporaneous work LayerDiffusion\nis an important comparison, and provide quantitative and qualitative comparisons in this rebuttal.\nExperimental details, results and discussion are available in the global rebuttal.\n\n\nRegarding scene composition evaluation, we have made efforts to compare to open-source, widely\nused state of the art works designed for compositional generation. As discussed in section 4.2 (Baselines), we have compared to layout-based generation method GLIGEN, which in our experiments, more consistently outperformed boxdiff and ReCO (references [47] and [50]) and therefore\nconstituted our best baseline bounding box based method. Similarly, for multi-layer scene generation,\nwe have chosen multidiffusion, due to its strong performance and available implementation.\nUnfortunately, we were not able to reimplement Layered rendering diffusion model for zero-shot\nguided image synthesis (LRDiff, reference [36] in our manuscript) for comparison, due to missing\nimplementation details and no public code. We highlight that LayerDiff[2] is contemporaneous work\nwithout available opensource code. Due to the expensive data construction and training costs, we are\nunable to provide comparisons to this work. While our focus is on generating all image components,\nincluding occluded areas, LayerDiff only generates visible pixels. This in turns, strongly limit scene\nmanipulation abilities, which is one of the key benefits of our representation.\nFinally, while inpainting and editing techniques are relevant related work, we highlight that the focus\nof our work is not editing, but interactive and controllable scene generation. Our scene manipulation\nexperiments highlight intrinsic benefits of using RGBA assets to easily manipulate scene content\nwith strong content preservation. However, we do not aim to directly compete with editing methods,\nas we have not introduced any explicit editing mechanisms.\n\n[1] Zhang, Lvmin, and Maneesh Agrawala. \"Transparent image layer diffusion using latent\ntransparency.\" arXiv preprint arXiv:2402.17113 (2024).\n\n[2] Huang, Runhui, et al. \"LayerDiff: Exploring Text-guided Multi-layered Composable Image\nSynthesis via Layer-Collaborative Diffusion Model.\" arXiv preprint arXiv:2403.11929 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JK1b3il6fp", "reviewer": "Reviewer_wSp1", "summary": "This paper proposes to use a diffusion model to generate separate objects and then apply multi-layer noise blending to build a composite scene. A RGBA generator is finetuned from a latent diffusion model to generate alpha transparency for objects in addition to RGB. A transparency-aware training procedure is designed for both VAE and diffusion models. Then the generated objects are iteratively injected into a scene with a background according to a specific layout.", "questions": "A more detailed algorithm for 3.3 might be necessary to clarify how all the mechanisms work together, since the diffusion processes in the first n steps and b steps are different.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tTo adapt the VAE and diffusion model for RGBA transpancy generation, they make several key changes in training and model, such as using disentangling latents. These changes are nontrivial.\n2.\tProposed a novel background blending in addition to multi-layer composition process, which ensures the consistency between objects and the background.\n3.\tExtensive experiments have been conducted on state-of-the-art models and have shown good results.", "weaknesses": "The multi-layer scene composition method requires generating K + 1 images for composition, which is inefficient. Since RGBA objects have already been generated, there might be more efficient ways to fuse them in one step or reduce the cost of each iteration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments that increase the quality of our work and are encouraged\nby the positive feedback. As discussed in our limitation section in supp. D, we agree that our\nmore expensive generation is a limitation of our work. However, we expect this cost to be heavily\namortised once a critical mass of RGBA assets has been generated. The same principle is also the\ncurrent practice for 3D digital assets. We further highlight that it is a common limitation of layered\nrepresentations, where layers require separate generation processes, and consider that the added\nflexibility and blending quality can outweigh additional generation cost. Nonetheless, we intend\nto explore more efficient solutions in future works, notably by exploring ways to combine instance\ngeneration and blending while maintaining fine-grained control.\n\nWe thank the reviewer for highlighting clarity issues in section 3.3. We will update our manuscript\nto include the following algorithmic description of the section in supplementary materials to facilitate\nunderstanding.\n\nMulti-layer scene composition algorithm:\n> **Inputs**: $K$ pre-generated RGBA instances $\\{I_k,M_k\\}$,  \nA bounding box based scene layout $\\mathcal{L}=\\{[cx_k, cy_k, w_k, h_k ], \\text{for } k \\in 1,\\cdots,K\\}$,  \nA latent diffusion model (VAE $\\mathcal{E}$, diffusion model $\\mathcal{D}$)  \nT generation timesteps  \nRandom noise $\\eta \\in \\mathcal{N}(0,I)$  \nNumber of blending timesteps $n$,  \nbackground blending timesteps $b$,  \ncross-layer consistency timesteps $n_s$  \n**Output**: $K+1$ multi-layer images (background, $K$ images with increasing number of instances)  \n  \n> *Generation set-up*\n        >>Rescale $\\{I_k,M_k\\}$ according to $[cx_k, cy_k, w_k, h_k ]$  \n        $x^0_k \\leftarrow \\mathcal{E}(I_k)$  \n        $m_k \\leftarrow$ Downsample $M_k$ to latent space dimension  \n        Compute $x^k_t$ at all diffusion timesteps $t \\in [1,T]$ using DDIM inversion  \n         Initialise all images with same Gaussian noise: $y^k_T \\leftarrow \\eta$  \n\n>*Composite scene generation*\n       >>**For** $t=T$ **to** 0:  *Loop over timesteps*    \n        >>> *Predict noise update with diffusion mode*l: $\\epsilon_{\\theta}(y^k_t,t) \\leftarrow \\mathcal{D}(y^k_t,t)$  \n       $y^k_{t-1} \\leftarrow$ Update according to noise scheduler and $\\epsilon_{\\theta}(y^k_t,t)$  \n       **If** $t \\geq b$: *Background blending optional step*\n       >>>> $y^0_{t-1} \\leftarrow$ $y^0_{t-1} \\cdot m^*+\\frac{1}{2} (y^0_{t-1}+ y^K_{t-1})\\cdot (1-m^*)$  \n\n >>>**If** $t \\geq n$: *Scene composition*\n >>>>**For** $k=1$ **to** K:\n>>>>> $y^k_{t-1} \\leftarrow$  $y_{t-1}^{k-1} \\cdot(1-m_k)+x^k_{t-1}\\cdot m_k$\n        \n>>>**Elif** $t \\geq n+n_s$: *cross-layer consistency optional step*\n>>>>**For** $k=0$ **to** K:\n>>>>> $ y^K_{t-1} \\leftarrow y^K_{t-1} \\cdot(1-m_k)+y_{t-1}^{k-1} \\cdot m_k$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GAcdIOJxf7", "reviewer": "Reviewer_S8j5", "summary": "This paper introduces a novel method for generating complex images from textual descriptions with a high degree of control over object attributes and scene composition. They introduce a new training paradigm for adapting a diffusion model to generate isolated objects with transparency, using a disentangled representation for RGB and alpha channels. Moreover, a multi-layer scene compositing strategy based on noise blending is developed, enabling the generation and manipulation of complex scenes from multiple instance images. Overall, the task is interesting and the proposed method achieves fantastic generation ability.", "questions": "None", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a framework for fine-grained controllable generation, providing user control over instance appearance and location with intrinsic scene manipulation capabilities. The results show superior controllability of their approach.", "weaknesses": "1. I think this paper ignores a similar and pioneer work: LayerDiffusion [1], which achieves the RGBA generation by finetuning the VAE and diffusion models. It is necessary that this paper should compare with this prior work. \n2. It seems that the Multi-Layer Noise Blending would cause unnatural image generation. I hope you can provide more discussion about the composition of foreground and background.\n3. A deep and clear literature review of prior works is very important for an excellent work. More discussions about the LayerDiffusion[1] should be included in this paper.\n\n[1] Zhang L, Agrawala M. Transparent image layer diffusion using latent transparency[J]. arXiv preprint arXiv:2402.17113, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments that improve the quality of our work. We are grateful\nfor the positive remarks (interesting work, fantastic generation ability, superior controllability), and\naddress the reviewer’s main concerns: comparison to LayerDiffusion and the impact of multi-layer\nnoise blending on realistic generation.\n\n**LayerDiffusion:** Firstly, we highlight that our work was developed in parallel to LayerDiffusion [1],\nwhich is nearly contemporaneous to ours, and our methodological decisions were not inspired from\nthis work. This can be evidenced by our entirely different methodological decision : we explicitly\nmodel the transparent channel, and disentangle the latent space; while LayerDiffusion implicitly\nincorporates transparent information. However, due to the very similar objective of transparent\ngeneration, we agree with the reviewer that comparisons are important, and extend our evaluation\nto include a comparison to LayerDiffusion. Experimental details, results and discussion are available\nin the global rebuttal.\n\nBesides transparent generation, LayerDiffusion also offers multi-layer generation properties by\ngenerating instances conditioned on a background image. We point out here that our blending strategy\naffords substantially better controllability, as LayerDiffusion does not allow to specify the location\nof these instances (see Figure 2 of this rebuttal). We will update our manuscript to incorporate\ndiscussions on LayerDiffusion in related work, and add experiments provided in this rebuttal.\n\n**Multi-layer noise blending** can lead to unnatural generation, as evidenced in our supplementary\nexperiments (see Supp. F.2 and Figure 12), when noise injection constraints are too strong. Our\nmethod is designed with this challenge in mind, with two important components to ensure natural\nlooking images: 1) iterative integration of instances in the global scene, and 2) imposing constraints\nin early timesteps only. The latter ensures naturalness of generated scenes by leveraging the intrinsic\nbiases of diffusion models. Imposing constraints only at early timesteps ensures instance locations\nand key attributes are integrated, while giving enough freedom to the model to generate realistic\nimages. The impact of increasing or decreasing constraints is discussed in more details in our\nsupplementary section F.2. The former component, unique to our multi-layer approach, iteratively\nintegrates new instances in separate layers. Generating separate images allows to smoothly integrate\neach instance in the scene individually, simplifying the task in comparison to methods that aim to\nblend all instances in the scene at once.\n\n[1] Zhang, Lvmin, and Maneesh Agrawala. \"Transparent image layer diffusion using latent\ntransparency.\" arXiv preprint arXiv:2402.17113 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TpzEuLjKfF", "reviewer": "Reviewer_tSA3", "summary": "This study enhances text-based image generation diffusion models by introducing multi-layer noise blending and transparency-aware training procedures, enhancing control over the generated image content. This technique allows for finer control over the elements composing the image and improves the quality and consistency of the generated images through conditional sampling.", "questions": "How does the model manage unexpected object interactions or complex overlapping in scene compositing? Providing in-the-wild results that showcase complex object interactions and instance occlusions, such as scenarios depicting a person riding a motorcycle and a person wielding a wand, would help clarify the model’s capabilities in  intricate scene compositions.", "limitations": "The authors mention limitations briefly at the end of the conclusion.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The approach allows for detailed control of object appearance and placement within images.\n2. Images are produced with greater detail and consistency through the use of conditional sampling and transparency handling.\n3. The method is straightforward and easy to understand.", "weaknesses": "1. Novelty: The field of layered-based image generation, which includes both depth-based and instance-based layering, has been thoroughly investigated. It remains unclear what theoretical or methodological innovations the author's approach offers compared to existing instance-layering methodologies. The capability for fine-grained control, as mentioned, seems achievable by most contemporary methods.\n\n2. Evaluation Concerns: Recent developments in instance-based layering methods appear to have been neglected. An analytical discussion or empirical benchmarking against these newer methods is warranted. Pertinent literature includes:\n\n Huang, Runhui, et al. \"LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model.\" arXiv preprint arXiv:2403.11929 (2024).\n\n Zhang, Lvmin, and Maneesh Agrawala. \"Transparent image layer diffusion using latent transparency.\" arXiv preprint arXiv:2402.17113 (2024).\n\n3. Experimental Assessment: There appears to be a lack of quantitative evaluations for GLIGEN and MultiDiffusion methodologies. It would be helpful if the authors could clarify why quantitative comparisons with these methods were not included. Additionally, presenting visual results for entire scenes from non-layer-based methods, such as the SD series models, would be valuable in assessing whether layout-independent methods enable more coherent or effective inter-instance interactions.\n\n4. The independent generation of instances introduces significant complexity and poses challenges in assembling them into a coherent scene, as noted by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments that increase the quality of our work, and address main\nconcerns below.\n\n**Novelty:** while we agree with the reviewer that layered-based representation are receiving increasing\nattention, we argue that our proposed methodology has several unique components which paves\nthe way for multi-layer innovation. Firstly, transparent image generation is an emerging topic, with\nvery few related works dedicated to the task. To tackle this task, we propose a novel modelling\nstrategy with a disentangled latent space and mutual conditioning. In addition, we provide several\ntraining insights to achieve better results with this type of generation task. Secondly, we disagree\nwith the reviewer that the level of fine-grained control we are providing can be achieved with\ncontemporary method, as evidenced by our experiments. While common layered methods allow\ncontrol over location and appearance of objects, they often struggle with patterns and fine-grained\ninstance details, as all components are generated at the same time. We have investigated competing\nmethods with publicly available implementations, and consistently observed difficulties in achieving\nthe right level of details, and/or accurate instance positioning for the type of complex prompts we\nare showing in our manuscript. We highlight that we are working with substantially more complex\nimage descriptions than e.g. T2I Compbench (reference [16]). In contrast, by generating instances\nseparately, we can control fine-grained attributes and focus on scene coherence at blending time.\nLastly, our multi-layer scene composition is also different from pre-existing methodology, due to our\npre-generated instances. Here, instead of focusing on generating the right instance prior to a global\nmerger, we focus on instance integration within the scene, through iterative instance addition layer\nby layer, at each timestep. This allows more natural integration of individual instances in the scene.\n\n**Related works:** While we focused our evaluation on published work, we agree with the reviewer\nthat LayerDiffusion [1] is highly relevant recent related work, and update our manuscript to add\ncomparisons to this work, both in related works and experiments. We highlight that this work\nwas developed in parallel to ours, and our methodology and handling of transparency is entirely\ndifferent. Furthermore, we highlight that the multi-layer scene generation process of LayerDiffusion\nis substantially more restricted, as instance location cannot be specified and is solely controlled\nby background appearance. We provide quantitative and qualitative comparisons in this rebuttal.\nExperimental details, and discussion are available in the global rebuttal.\nLayerDiff [2] is contemporaneous work to ours, and is closer to a multi-layer extension to Text2Layer\nthan our work. We note an important limitation of this work, which is that occluded components are\nnot generated. This makes scene manipulation tasks more challenging, and substantially reduces flexibility. Unfortunately, due to the complex dataset building required and lack of publicly available code,\nwe are not able to provide comparisons to this work, but will integrate it in our related works section.\n\n**Experiments:** We agree with the reviewer that quantitative experiments would be a great addition.\nUnfortunately, as discussed in sec 4.2 (Baselines), quantitative evaluation is highly challenging due\nto the interactive nature of our methodology, as well as the highly complex nature of image prompts\nwe are working with. This would require the development of a novel benchmark, which we leave\nto future work as quantitative evaluation remains an open problem for controllable generation tasks.\nWe have made efforts to provide many examples highlighting different properties and benefits of\nour work, and to thoroughly analyse our method’s behaviour in our supplementary experiments. We\nnote that we do provide a non-layered baseline with the PixArt-alpha model (reference [6]) and are\nhappy to additionally provide a Stable Diffusion baseline as well if needed.\n\n**Instance interactions:** We agree with the reviewer that independent instance generation is a\nlimitation of our work, and an important focus of our future work. Our method in its current state can\nmitigate these issues with precise instance prompts, and stronger blending freedom at composition\ntime. Highly intricate interactions can be achieved without altering our main methodology, by\nfine-tuning our RGBA generator with higher quality, fine-grained training data, or making it capable\nof generating grouped instances.\n\n[1] Zhang, Lvmin, and Maneesh Agrawala. \"Transparent image layer diffusion using latent\ntransparency.\" arXiv preprint arXiv:2402.17113 (2024).\n\n[2] Huang, Runhui, et al. \"LayerDiff: Exploring Text-guided Multi-layered Composable Image\nSynthesis via Layer-Collaborative Diffusion Model.\" arXiv preprint arXiv:2403.11929 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed reviews, insights and comments that improve the quality of
our work. We are encouraged by the positive comments: the soundness of our method (RGbJP), the
quality of our results (RGbJP, RwSp1, RS8j5), the non triviality of our work (RwSp1), and detailed
control afforded by our method (RtSA3). We have made efforts to address all reviewers concerns in
individual rebuttals, through additional experiments and clarifications. The reviewers main concerns
were comparison of our method with LayerDiffusion [1] and novelty of our work. We restate our
key innovations below, and discuss our experimental details and results on LayerDiffusion.

**Novelty:** We designed our approach with interactivity and controllability in mind. Our key objective
is the development of a controllable generation solution were the user can easily build complex
scenes, control instances appearances, and adjust scene layout and content. In particular, we
are considering scenes with complex layouts, where multiple instances have unique patterns and
attributes, going well beyond compositional benchmarks such as T2I-Compbench (reference [16]).
Our key idea is the use of pre-generated RGBA assets, and their composition through a multi-layer
process. This allows precise attribute control, ensures that the blending process focuses on scene
coherence, and substantially facilitates scene manipulation tasks. To achieve this, we introduce 1) a novel modelling and training approach for generation of transparent images, and 2) a novel multi-layer scene composition method, that blends pre-generated assets into a scene. Integrating instances one by one in the scene allows for smoother scenes where relative position of instances in 3D space (A in front of B) is clearly established and easily controlled. Our experiments have shown that our approach allows a degree of fine-grained control that competing methods cannot achieve.

**LayerDiffusion:** While we focused our original manuscript on comparison with published, state
of the art work, we agree that the recent LayerDiffusion is highly relevant work, and now provide
comparisons with this related work in our rebuttal which will be added to our manuscript. We
ran experiments using the official implementation and recommended parameters [3], we used the
Juggernaut XL V6 SD XL model with 20 Steps, DPM++ 2M SDE sampler and a CFG scale of
5. For RGBA evaluation, instances were generated using the Foreground Model model with the
same captions as for the other models. For the scene composition visualisation we have used the
Background to blended image model where we iteratively add instances to the previously blended
image, following the protocol in [1] ( Figure 9). Images were generated across multiple seeds, and the
best results was selected. Results are available in the attached PDF document. Figure 1 shows visual
comparisons of generated instances, showing that while LayerDiffusion is able to generate high
quality instances, it does not necessarily respect prompt details (for example, the impressionist style).
Quantitative results in Table 1 show that we outperform LayerDiffusion on all metrics. The lower
IoU performance could notably be attributed to the implicit transparency modelling, which does
not guarantee that accurate transparency layers are generated. Lastly scene composition experiments
in Figure 2 show that the model struggles with iterative integration of instances, with a strong bias
towards the centre of the image, and no ability to control where instances are added.
We restate key differences between our work and LayerDiffusion: while both train a model capable
of generating RGBA images, our RGBA modelling approach is entirely different, where we build
a disentangled latent space and explicitly model interactions between RGB and Alpha channels.
In contrast, LayerDiffusion encodes transparency within the null space of the VAE, so as to train
a diffusion model following standard practice. We further highlight that our scene composition is
substantially more flexible, as LayerDiffusion only generates instances conditioned on a background
image, with no control over instance positioning. As shown in Figure 2 of our attached results, this
can lead to poorly positioned instances, with a bias towards the centre of the image.

**Baselines:** The field of controllable generation is very popular, and there exist many competing works
aiming for similar objectives. Considering the interactive nature of our work, providing a reliable
quantitative evaluation of our results is highly challenging and an open problem. For qualitative
evaluations, we have made efforts to select the strongest open-source baselines, covering standard
image generation (PixArt-alpha (reference [6])), layout based generation (GLIGEN (reference [23])),
and multi-layer (multi-diffusion (reference [2])), and attempted to reproduce alternative closed source
methods. We now additionally add LayerDiffusion as a highly relevant baseline. We will additionally
enrich our related work to discuss contemporaneous, closed source, work LayerDiff[2]. This work
focuses on predicting different image regions separately with dedicated prompts, and in contrast
to ours, doesn’t predict occluded areas. The methodology, closer to Text2Layer (reference [55]),
requires expensive dataset building through instance segmentation.

[1] Zhang, Lvmin, and Maneesh Agrawala. "Transparent image layer diffusion using latent
transparency." arXiv preprint arXiv:2402.17113 (2024).

[2] Huang, Runhui, et al. "LayerDiff: Exploring Text-guided Multi-layered Composable Image
Synthesis via Layer-Collaborative Diffusion Model." arXiv preprint arXiv:2403.11929 (2024).

[3] https://github.com/lllyasviel/sd-forge-layerdiffuse

Meta-review: This paper proposes a text-to-image (T2I) diffusion model (DM) to generate separate objects (with RGBA masks to represent transparency information). The proposed method can generate complex images with a high degree of controllability (e.g, instance appearance and location).

The reviewers have mixed opinions. The significant concern of the reviewers is the lack of novelty of the proposed method. More specifically, three reviewers pointed out the lack of comparison with LayerDiff [1] and LayerDiffusion [2]. Although this paper and the previous works [1,2] have a high similarity, I can't entirely agree with the reviewers' opinion that these works should be compared with the proposed method at the submission time. Both of them were uploaded to ArXiv this year (Feb and Mar), and the submission deadline for NeurIPS was May. Also, only [2] is published at TOG in this July. In my opinion, they are more like concurrent works rather than previous works.

- [1] LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model
- [2] Transparent Image Layer Diffusion using Latent Transparency

In the rebuttal document and comments, the authors have provided the comparison results with LayerDiffusion qualitatively and quantitatively. I think that the additional experimental results provided by the authors support that the proposed method and LayerDiffusion have distinct properties and there are some scenarios when the proposed method works better (e.g., when we need more controllability by text, or layout image).

In terms of evaluation, I partially agree with Reviewer GbJP that this paper can include more baselines, but at the same time, I partially agree with the rebuttal, except LayerDiffusion, they chose the best methods that were available at the submission time. I personally recommend the authors to add more baseline methods despite they are not current state-of-the-art methods, to emphasize the contribution of the proposed method.

Except for the novelty and the evaluation issue, I think the remaining weaknesses are not significant issues.

Considering the advantage of the proposed method (high controllability, enabling more controllable factors compared to the existing state-of-the-arts -- including LayerDiffusion) (wSp1, S8j5, tSA3), and the novelty issue seems to be partially resolved by the rebuttal, I recommend acceptance of this paper.
Decision: Accept (poster)