Generating compositional scenes via Text-to-image RGBA Instance Generation

Authors: alessandro  fontanella
petru-daniel  tudosiu
yongxin  yang
shifeng  zhang
sarah  parisot
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:10:13

Abstract: Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. Controllability can be improved by introducing layout conditioning, however existing methods lack layout editing ability and fine-grained control over object attributes. The concept of multi-layer generation holds great potential to address these limitations, however generating image instances concurrently to scene composition limits control over fine-grained object attributes, relative positioning in 3D space and scene manipulation abilities. In this work, we propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. To ensure control over instance attributes, we devise a novel training paradigm to adapt a diffusion model to generate isolated scene components as RGBA images with transparency information. To build complex images, we employ these pre-generated instances and introduce a multi-layer composite generation process that smoothly assembles components in realistic scenes. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes. Through multi-layer composition, we demonstrate that our approach allows to build and manipulate images from highly complex prompts with fine-grained control over object appearance and location, granting a higher degree of control than competing methods.

Main paper: 
 introduction: 
 The development of text-to-image generative diffusion models has allowed the generation of highquality synthetic images guided by textual prompts. However, achieving desired image quality and properties inherently requires tedious and meticulous prompt engineering [35] . As image content is solely described using a textual description, prompt content has to be carefully crafted, notably taking into account the models' struggles to understand subtleties of language (e.g. counting, object attributes, negation or spatial relationships) [16] . Additionally, minor prompt modifications can lead to substantial modifications of generated image content, further increasing prompt crafting tediousness.An intuitive way of improving controllability of generated content is by providing alternative image descriptors, e.g. using image layouts which describe the location of specific objects in an image as bounding boxes. Several approaches have been proposed to inject layout information in the generative process. Training based methods [24, 54] adapt or introduce new model weights to introduce object coordinates as conditioning, while [8, 51] leverage cross-attention manipulation and inference time optimisation. While substantially increasing control over image structure, these solutions still generate all components as a single denoised image. This limits the user's ability to finely control object attributes, and layout adjustments requires re-generation with limited content preservation guarantees. These issues can potentially be addressed with image editing techniques. This, however, can require expensive data construction and training [4] , while the displacement or resizing of objects (essential for layout manipulation) remains a very challenging type of edit that few have explored [9, 31] . This challenge can be attributed to the fact that objects and backgrounds have to be regenerated, while at the same time ensuring appearance preservation with the original image.The emerging layer-based generation paradigm [38, 49, 40] has shown promise in addressing these limitations. The idea is to represent images as multi-layer stacks, where different image components are generated on separate layers. This has two key benefits: 1) instances can be generated more precisely as separate images with individual prompts, 2) separating image components simplifies image manipulation tasks, as only the relevant layers will be modified. Nonetheless, layer-based methods typically suffer from the fact that layers are not fully generated, as they are collapsed to a single image in the later stages of a diffusion process. Generating all components jointly can affect the ability to achieve fine-grained control of layers attributes, control their relative positioning in 3D space (as layers are collapse with no hierarchical structure), ensure a smooth composition of all image component, and achieve layout manipulation abilities with strong content preservation (as layers and composite image influence each other).In this work, we seek to address current limitations through a multi-stage generation process that is designed for fine-grained control, flexibility and interactivity. We propose to build complex scenes by first generating individual instances as RGBA images, then iteratively integrating them in a multi-layer composite image according to a specific layout. The ability to generate individual instances allows to finely control instance appearance and attributes (e.g. colours, patterns and pose), while their layered composition allows to easily control and modify positioning, scale and ordering. In particular, our two-stage approach provides intrinsic layout and attribute manipulation abilities with strong content preservation. To achieve our goal, we firstly train a diffusion model capable of generating RGBA images, i.e. images with alpha transparency information. Directly generating isolated instances, c.f. generating and extracting them with a segmentation model, ensures better transparency masks and more fine-grained control of instance attributes. Our RGBA generator is obtained by fine-tuning a latent diffusion model (LDM) using RGBA instance data from the recently released MuLAn dataset [47] . In order to incorporate transparency information in the generative process, we devise a transparency-aware training procedure for both VAE and diffusion model. In contrast with the contemporary transparent generation method of [57] , which implicitly encodes transparency information in the null space of the VAE to carry out standard LDM fine-tuning, we explicitly integrate transparency in our generation and training process. Our VAE is trained so as to disentangle RGB and alpha channels, ensuring colour and detail preservation in RGB reconstruction. Our LDM is then fine-tuned with a novel training paradigm leveraging our disentangled latent space, that allows a conditional sampling-driven inference where alpha and RGB latents are sequentially denoised with mutual conditioning. Finally, we leverage our RGBA generator to build composite images with fine-grained control over object attributes and scene layout. We build multi-instance scenes via a multi-layer noise blending solution, where each instance is associated to a specific image layer. Crucially, each instance is integrated in the scene one layer at a time, yielding increasingly complex image layers so as to ensure scene coherence and accurate relative positioning. This contrasts with pre-existing multi-layer works which assemble all instances at once [38, 49] , and is uniquely afforded by our RGBA generation process. By manipulating latent representations in early stages of the denoising process, we are able to achieve high degrees of precision and control, while at the same time generating smooth and realistic scenes. An overview of our complete methodological process is shown in Fig. 1 .We provide a thorough evaluation of our RGBA generator's capabilities, showing that we are able to generate highly diverse objects and precisely control their attributes and style, outperforming alternative solutions. In addition, our scene composition experiments show that we consistently outperform state of the art layout controlled methods, while highlighting our unique ability to generate and manipulate complex scenes with strongly overlapping objects. In summary, our main contributions are the following:• We propose a multi-layer generation framework designed for fine-grained controllable generation that grants user control over instance appearance and location, with intrinsic scene manipulation capabilities.• We introduce a novel fine-tuning paradigm for pre-trained diffusion models to generate individual objects as RGBA images with transparency information. We propose a disentangled training strategy that relies on mutual conditioning of RGB and alpha channel latents.• We develop a multi-layer scene compositing strategy based on noise blending that allows to build and manipulate complex scenes from multiple instance images. By combining RGBA generation with scene compositing, we are able to generate images with accurate attribute assignment and relative object positions. 
related work: 
 Text-to-image and transparent generation. GLIDE [32] first proposed text-to-image generation with diffusion models by adopting classifier-free guidance through text. While GLIDE trains the text encoder jointly with the diffusion prior using paired image and text data, Imagen [43] employs a frozen large language model as the text encoder. More recently, Latent Diffusion Models (LDM) trained in latent space have increased in popularity, given their lower computational requirements [41] .Stable Diffusion incorporates an adversarial objective to learn the latent representation and enhance the authenticity of generated images and introduces cross-attention for text conditioning. While the U-Net [42] architecture was originally the most popular for latent diffusion models, recent works are converging towards the use of Diffusion Transformers (DiT) [36] , which operate on latent patches. For example, Pixart-α [6] has demonstrated remarkable image generation capabilities, while reducing the computational requirements to 10.8% of Stable Diffusion-v1.5's training time.Leveraging these models to generate instances with transparency information has been rarely explored.Text2Layer [60] generates foreground and background images separately, defining foreground as the ensemble of all salient objects and predicting the foreground alpha mask separately from RGB pixels. Developed concurrently, LayerDiff [18] generates image components as separate images, providing the transparency masks as input. This technique tends to generate partial instances, as occluded areas are not generated. Also contemporary, LayerDiffusion generates transparent images by encoding transparency in the null space of the pre-trained VAE, allowing to fine-tune a diffusion model using standard techniques. While able to generate high quality instances, this implicit transparent modelling can lead to inconsistent results, without guarantees that transparency will be achieved. In contrast, our explicit disentangled latent space allows for more control and guarantees transparent images outputs.Image editing and scene controllability The increasing popularity of diffusion models has highlighted the limitations of working solely with text based control. We identify two main areas of research seeking to increase generative controllability: image editing and layout controlled generation. Image editing can be achieved by fine-tuning on dedicated dataset [4] , constrained sampling with CLIP [39] losses [20, 21, 45, 17, 50] , mask guided inpainting [30, 59, 11, 52, 7] , training-free cross attention manipulation [12, 34, 5, 48] , and inference-time optimisation [9] . Despite achieving impressive results for replacement and local modification tasks, image editing often struggles with more complex manipulations such as moving, rescaling and removing, especially on non isolated instances. By modifying pre-generated content, editing methods additionally have limited layout control capabilities. Alternatively, precise layout control during generation has been explored by introducing additional conditioning. GLIGEN [24] , ReCo [54] and Boxdiff [51] achieve image generation from layouts given as bounding boxes specified by the user. ControlNet [58] allows to incorporate several forms of image based conditioning, such as sketches, depth maps, or canny edges. These techniques afford high generative controllability, but lack layout editing capability. Composite [2] and layer-based generation techniques [38, 49, 40] offer a promising avenue, where individual scene components are associated with unique prompts and assembled in a scene with bounding boxes or instance specific masks, leveraging the diffusion model's intrinsic priors to build coherent scenes. However, generating scenes and all individual components jointly requires compromising between instance quality, attribute accuracy and scene coherence, reducing flexibility and control. We build from layered approaches, and use multi-layer paradigms to iteratively integrate pre-generated RGBA instances in increasingly complex images, affording us precise control over layout, instance attributes and allowing us to focus on scene composition quality solely during the composition step. 3 MethodsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss methods limitations in the conclusion of the main paper, and in a dedicated section in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Diffusion models. Diffusion models learn to reverse a forward diffusion process, where images are iteratively converted to near isotropic Gaussian noise over T timesteps asy t = (1 -β t )y t-1 + β t ϵ,where ϵ ∼ N (0, I) and β describes a noise schedule. Diffusion models are trained to reverse the process by learning to predict the noise ϵ θ that was added at a specific timestep:EQUATIONWhere C is a conditioning variable, typically a text prompt embedding. Starting from isotropic Gaussian noise y T at inference time, diffusion models generate images through the reverse iterative denoising process, where y 0 is the final denoised output. At a given timestep t of this backwards diffusion process, the pre-trained model predicts the noise ϵ θ that updates current noisy image y t to move on to the next timestep t -1. In this work, we consider latent diffusion models (LDMs), where the diffusion process is carried out in the latent space of a Variational AutoEncoder (VAE) [41] .Alternatively, the pre-trained model can be leveraged to convert an image x 0 to Gaussian noise by iteratively adding predicted noise ϵ θ (x t , t) according to the same noise schedule [44] . This inversion process provides an estimation of the input noise x T that allows to obtain x 0 via the backwards diffusion process. This technique is particularly relevant to image editing tasks and noise blending methods like ours. To differentiate between both processes, we use y t to refer to generated latents and x t to refer to inverted latents in the remainder of this paper.RGBA representation. An RGBA instance image is modelled as a four channel image I ∈ R w,h,4 , where the additional Alpha channel describes the level of transparency of RGB pixels. The alpha channel takes values in [0, 1], where 0 correspond to full transparency. Transparent pixels for which we do not have RGB information are set to black (RGB = (0, 0, 0)).A simple method for creating instances with transparency involves using a text-to-image model to generate an initial image, then applying image matting techniques [55] to extract the transparency alpha mask. This reliance on automated matting can yield inaccurate masks, particularly when dealing with complex object structures and image backgrounds. In addition, this approach reduces controllability, as object attributes can bleed in the background content rather than the instance itself [16] . Alternatively, we propose to natively generate RGBA instances using a generative diffusion model. To this end, we developed a transparency aware training procedure for a pre-trained LDM that focuses on the interaction between RGB and alpha channels. This section will first introduce our training dataset, then detail our fine-tuning approach for the LDM's VAE and diffusion model.Training data. We employed 87989 instances from the MuLAn dataset [47] , a novel dataset consisting of automatically generated RGBA decompositions with a diverse array of scenes, styles and object categories, and 15791 instances extracted from a variety of image matting datasets with high quality masks. More details about the datasets are provided in Appendix A.1.RGBA VAE. We train our RGBA VAE following the training procedure of [41] , comprising a L 1 reconstruction loss L recon (I, D(E(I))), a perceptual loss L LP IP S (I, D(E(I))), a discriminator based adversarial loss L adv (D(E(I)))), and a Kullback Liebler (KL) loss between the estimated latent distribution and a normal distribution N (x : 0, 1). However, we make key modelling and training adjustments in order to accommodate for the new alpha transparency information.Firstly, the additional channel is simply taken into account by replacing and retraining the input and output layers of the model. Secondly, we observed that learning a joint RGBA latent space leads to entanglement of RGB and alpha channels, affecting generation capability of diffusion models trained in this latent space. We address this challenge by disentangling representations in the latent space: our VAE predicts two separate distributions N (x : µ RGB , Σ RGB ) and N (x : µ α , Σ α ), each associated with a separate KL loss. While we do not explicitly model the separate distributions to encode RGB and alpha channels respectively, it is a natural disentanglement of the data.Lastly, the VAE in [41] is trained with very small KL regularisation (e.g. a weight factor of w KL = 10 -6 on the KL loss), so as to focus on reconstruction quality. In our setting, we observed that this deviation from standard VAE training was harmful to generative ability, yielding dark and highly contrasted images. In contrast, training our VAE with large regularisation (w KL = 1) noticeably improved image quality, while at the same time enforcing disentanglement more strongly (see Appendix A.3 for visual comparisons). The perceptual loss is also computed separately for RGB and alpha channels and the two components L LP IP S (I RGB , D(E(I)) RGB ) and L LP IP S (I α , D(E(I)) α ) are then averaged.In the second stage, we fine-tune the LDM on our instances datasets. Our VAE fine-tuning keeps the dimension of the original LDM latent space (4 channels and 64 × 64 spatial dimension), allowing us to directly fine-tune without architecture adaptation. However, in our case, the latent space also encodes information on the transparency layer. When sampling from our model, we seek to exploit the mutual dependency between RGB and alpha channels. In particular, given noised latents y RGB t and y α t for RGB and alpha channels respectively at timestep t, information contained in y RGB t-1 could be employed to inform the update from y α t to y α t-1 and vice-versa. In order to train the network to leverage this conditional information, we modify the training procedure of the LDM. Given y RGB 0 and y α 0 , we sample two different Gaussian noise vectors ϵ RGB and ϵ α and use them to compute y RGB t , y RGB t-1 ,y α t , y α t-1 with the forward process of the diffusion model and t randomly sampled in [0, T ], with T number of training steps. Our network is trained to predict (ϵ RGB , ϵ α ) jointly given one of the following as input: (y RGB t , y α t ), (y RGB t , y α t-1 ), or (y RGB t-1 , y α t ). This training procedure unlocks the ability to perform conditional sampling. Given (y RGB t , y α t ), we can alternate between updating the alpha component y α t-1 and then use (y RGB t , y α t-1 ) to update the RGB component, obtaining (y RGB t-1 , y α t-1 ). Alternatively, we can update the RGB component first and use it to condition the alpha update, but observed that the former approach worked best in practice. When sampling from diffusion models, it is common to use fewer sampling steps than at training time for faster image generation [53] . Therefore, in order to make our training regime more flexible and applicable to a variety of sampling strategies, we additionally use pairs (y RGB t , y α k ), and (y RGB k , y α t ) as conditioning input in the second half of training iterations, with k randomly sampled in [0, t -1].We consider that we have an image layout available, and generated K instances {I k , M k } using our RGBA generator, where I k refers to RGB values and M k is the transparency alpha mask. We represent a layout as a collection of bounding boxes, where each box is associated with a specific instance:L = {[cx k , cy k , w k , h k ], for k ∈ 1, • • • , K}.We design our scene composition approach as a multi-layer noise blending process, where instances are sequentially integrated in intermediate layered representations. As a result, we concurrently generate K + 1 images (sorted as background, and K composite images with increasing number of instances). While more costly, we observed that generating layered images affords more flexibility, better control over relative positions of instances and yields more natural compositions. An algorithmic overview is provided in Appendix B.We consider the latent representation x k 0 of instance image I k , and compute its noisy representation x k t at all diffusion timesteps t ∈ [1, T ] using DDIM inversion [44] . To generate our layered composite image, we first initialise our K + 1 images with the same noise vector y T . Inspired from compositing [2] and layered [38] image generation methods, we combine noisy image representations for the first n timesteps of the diffusion denoising process. After carrying out the denoising update at time t, and before moving on to the next time step, we sequentially update noisy images as follows:EQUATIONwhere m k is the instance alpha mask M k downsampled to the latent space and y 0 t is the background image generated concurrently. Equation 2 builds layered images by iteratively injecting new instances in the noisy images at the desired locations. Carrying this process only at the first n generative timesteps enables to precisely control scene composition and instance appearance, while at the same time allowing for a smooth and realistic composition. The smaller n is, the more freedom is given to the generative model to adapt scene content. We further detail two optional additional mechanisms.Background blending. In the noise blending procedure described above, the background is generated independently from the rest of the image. That can reduce blending quality as generated background elements can be incompatible with instance locations. We propose to address this limitation by introducing background blending. We inject appearance information from the composite image to the background as:EQUATIONwhere m * is the union of all instances alpha masks. The background area of the composite image gets adjusted throughout the generation timesteps to accommodate instances neighbourhood. Equation 3has two benefits: 1) it ensures stronger consistency between the background layer and final composite image, and 2) allows the background image to be generated with stronger awareness of instances. This blending process can be carried out for the first b steps of the composition process.Increasing cross-layer consistency. In situation where intermediate layers are needed, and n is small, consistency can be enforced on the composite image by applying Eq. 2 to y K for n s subsequent timesteps:y K t = y K t • (1 -m k ) + y k-1 t • m k for k ∈ [1, K -1].Scene editing. Our design intrinsically allows scene manipulation and editing easily. Once instances are pre-generated, scene content can easily altered locally by replacing instances, or modifying instance location by providing new bounding boxes. The modified image can easily be generated following our scene composition process with a fixed seed, without any additional constraints.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [NA] Justification:The dataset used is available at the following url: https://huggingface.co/datasets/mulan-dataset/v1.0. Open sourcing of our code will depend on internal approval. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We made sure to follow the Ethics Guidelines.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
rgba generation: 
 Our RGBA generator is fine-tuned from a pre-trained PixArt-α model [6] . As baselines, we compare quantitatively and qualitatively to Stable Diffusion 1.5 (SD-1.5) and PixArt-α, adding 'on a black background' to the captions to replicate instance generation. We additionally compare to both models combined with the Matte Anything (MA) matting algorithm [55] , our reimplementation of Text2Layer [60] , and LayerDiffusion [57] . In all approaches we used 100 steps. Our RGBA generator leverages the conditional sampling approach described in Sec. 3.2.In Fig. 2 and 3 we show qualitative examples of instances obtained with our RGBA generator. Fig. 2 shows that we are able to generate instances across different styles and to follow fine-grained attributes in prompts. Fig. 3 provides visual comparisons to our transparent baselines. We can observe that our approach is able to generate realistic instances following the instructions given. Text2Layer shows lower image quality and excessive transparency, while LayerDiffusion struggles to follow prompt details, such as image style. Combining SD with Matting allows to achieve reasonable segmentation of the instances generated, while when applied to PixArt-α it can sometimes struggle to correctly identify and segment the main object of the image, especially when dealing with artwork content. On top of attributes bleeding in the background, this highlights how unreliable matting can be for instance generation purposes.We quantitatively evaluate the quality of the generated instances with the Kernel Inception Distance (KID) [3] , a metric that was proposed to reduce the bias of the Fréchet Inception Distance, especially when evaluated on a low number of samples. We compute it using features from the last convolutional layer of the Inception v3 model and considering only the RGB channels of the generated images. In order to evaluate the alpha mask generated by the RGBA generator, following [60] , we employ masks obtained with ICON with PVT backbone [61] as ground truth, and compute the Intersection-Over-Union (IOU) with the predicted alpha masks. Lastly, to evaluate the correlation between the captions and the content of the generated instances, we employ the CLIP Score [13] , which computes the cosine similarity between image and text features using the CLIP model (ViT-B/16 variant) [39] . The results obtained are shown in Table 1 . We can see that our approach obtains the best results for all the metrics, with KID of 0.0150, IoU of 0. Ablation studies. In order to explore the impact of our novel training procedure, we train two RGBA LDMs 1) without our conditioned training procedure (i.e. predicting noise ϵ without conditioning from different timesteps) and 2) without our conditional sampling inference where RGB and alpha are mutually conditioned for generation. Performance of these models is reported in Table 1 , bottom rows. We can see that the largest gains in performance are obtained with our novel training procedure, while conditional generation results in more subtle improvement. As shown in Fig. 4 , our conditional sampling allows to correct small details and critical areas of the alpha masks.Generation details. We build composite scenes in three steps: 1) RGBA instance generation, 2) layout building: we draw one bounding box per generated instance and rescale instances accordingly, 3) noise blending of generated instances according to a global prompt. We keep parameters consistent across all scene compositions unless specified otherwise: guidance scale 2.5 (RGBA) and 4.5 Baselines. We compare our approach to state of the art scene composition methods focusing on layout control and interactivity. Our first baseline is Pixart-α [6] to provide intuition into the complexity of requested prompts and limitation of relying solely on text controls. Next, we compare to GLIGEN [24] , which learns additional cross attention layer to integrate bounding boxes and bounding box text descriptors as generation conditioning. Finally, we compare our approach to multi-layer methods MultiDiffusion [2] and contemporary approach Instance Diffusion [49] for completeness. Similarly to ours, these approaches rely on noise blending techniques to build composite scenes. Besides our (b) A white dog lying on a purple couch next to a yellow cushion with a flower pattern, with a swan painting in the back.(c) a blue and yellow car in front of a red bus covered with a white flower pattern on the road with a white airplane with orange stripes in the sky.(d) A red fox in front of a golden unicorn and a blonde woman in a long green dress walking away in a meadow. RGBA instance pre-generation, a crucial difference, in contrast to our multi-layer approach, is the use of noise averaging for overlapping objects. All methods use the same global prompt, GLIGEN and Instance diffusion leverage our bounding box layout and RGBA instance prompts as descriptors, while MultiDiffusion uses our instances alpha masks and captions as well.Due to the inherently interactive nature of our and competing approach (layout design, instance ordering), we provide visual results in Fig. 5 , comparing all methods across different scene compositions. For a fair comparison, we generate competing methods with multiple seeds and select the best result. We provide results for simple scenes with unusual attributes (a) and complex scenes with overlapping objects (b,c,d) (Additional results and ablations are available in Appendix E.2 and F.2). While GLIGEN is capable of accurately reproducing the desired layout, it often fails to assign the right attributes to objects (e.g. blue apple, orange plane, swan painting) and struggles with highly overlapping objects (fox and unicorn). In contrast, Multidiffusion is more accurate in terms of attribute assignments, but struggles to handle overlapping objects. This can be attributed to the noise averaging process, which fails to integrate a notion of instance ordering like our multi-layer approach. Instance diffusion achieves performance closest to ours, but still struggles with complex patterns, attributes and relative positioning (swan painting behind the couch, white bus with a red flower pattern, woman walking away). With our RGBA instance generation and multi-layer noise blending, we are able to accurately assign object attributes and follow the required layout, while successfully building smooth and realistic scenes.Scene manipulation results. Finally, we evaluate our method's potential for scene manipulation. To optimise cross scene consistency, we set b = 0 (remove background blending) and set a common generation seed across all versions of the same scene. The manipulations we consider here are: attribute modification, instance replacement, and layout adjustment. We note that the first two tasks require RGBA generation of new instances. We highlight that we do not introduce any new explicit scene preservation or image editing technology in this experiment, therefore evaluating our method's potential for scene manipulation and controllability. We compare our results to contemporary method Instance Diffusion, which has been highlighted to possess similar editing capabilities [49] . Results are reported in Fig. 6 , showing that we are able to control and modify image content easily while maintaining strong consistency across different versions of the scene, without explicitly enforcing content preservation. This highlights the strong potential of multi-layer approaches to facilitate the development of image editing methods. We can see that we achieve substantially stronger scene preservation compared to instance diffusion, which can generate entirely different images and instances when modifications are too strong. 
conclusion: 
 In this work, we introduced a novel multi-layer strategy to scene composition, that focuses on interactivity and fine-grained control. To achieve this, we proposed a new training paradigm that adapts diffusion models to generate transparent images, through channel disentanglement and conditional sampling. To build composite scenes from RGBA instances, we further present a multi-layer compositing strategy that concurrently generates increasingly complex scenes through cross-layer noise blending. Extensive experiments show that we are able to generate diverse, fine-grained RGBA instances, and assemble them in complex scenes, achieving precise control and high flexibility over scene structure. One key limitation of our approach is the independent generation of instances, increasing the challenge of assembling them in a coherent scene. Future work will explore conditioned RGBA generation to intrinsically generate coherent scenes, as well as RGBA editing methods to further improve fine-grained control over scene content. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This is discussed in a dedicated broader and societal impact section in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: Potential safeguards are discussed in the broader and societal impact section in the Appendix.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the sources of our data and pre-trained models throughout the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No released new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
 
 Appendix: The MuLAn [47] dataset consists of RGBA decompositions for over 44,000 RGB images from the LAION and COCO datasets. Developed through a training-free pipeline, MuLAn decomposes monocular RGB images into a series of RGBA layers, comprising of background and isolated instances. Our training RGBA images are instances that were automatically extracted from natural images, making this noisy but diverse dataset an excellent solution for a first fine-tuning step. In order to assemble the matting dataset, we employed data from AIM-500 [23] , HIM-2k [46] , Interactive Human Matting [10] , RWP-636 [56] , PPM-100 [19] , AM-2k [22] .Images were resized with bilinear interpolation then centre cropped to obtain a 512 × 512 image. We observed that this particular transformation normally preserved the majority of the content of the instances. The VAE was trained for a total of 23 epochs (with each epoch comprising of 42908 steps performed with a batch size of 2) employing the Adam optimiser with starting learning rate of 4.5e -6 , β 1 = 0.5 and β 2 = 0.9. The discriminator adversarial loss was introduced after 50k steps with a weight of 0.5.For the LDM fine-tuning we followed a similar pre-processing, but additionally performing horizontal flipping with p = 0.5. To train the model for classifier-free guidance, we dropped the caption conditioning with p = 0.1. We first perform a first stage of fine-tuning on MuLAn for 200k iterations and a second stage employing 50% of the data from MuLAn and 50% from the dataset obtained combining different matting datasets for 80k steps. This is because MuLAn is larger and offers a wider variety of subjects, while the matting dataset, primarily comprising humans and animals, offers finer, human-annotated alpha masks. In particular, we employed the AdamW optimiser [28] with a starting learning rate of 1e -5 and cosine scheduler. and trained in mixed precision with a batch size of 12. The VAE and LDM were initialised with the weights from PixArt-α, while the text encoder (4.3B Flan-T5-XXL large language model) was kept frozen.To maximise model expressiveness, detailed captions were automatically generated for instances using the Llava-NeXt model [26] with the following instruction: "Write a detailed caption of image.". Since the captioning model operates only in the RGB space, transparent pixels were included in the captions as descriptions of black backgrounds. We employed Phi-3 [1] to remove all such references by using the following instruction "You are a talented ghostwriter given a text remove any mentions of the a black background while keeping the text as close as possible to the original.".Resizing and cropping training data can cause the generative model to also produce truncated instances. To limit this behaviour, inspired by [37] , we store crop coordinates, indicating the pixels cropped from the top-left corner along the height c top and width c lef t . These coordinates are then mapped with a sinusoidal embedding, similar to timesteps embeddings [6] , which is followed by a two-layer MLP featuring SiLU activations. At inference time, we set (c top , c lef t ) = (0, 0). In addition, we employed Offset Noise, which improves the contrast in images created by the model [33] . At inference time, we observed improvements employing timestep trailing [25] , classifier-free guidance [14] with the rescaling proposed by [25] . In particular, we observed the best results with guidance scale 2.5 and rescaling parameter 0.25 as shown in Sec. F.1.For Text2Layer, we trained the CaT 2 I-AE autoencoder and U-Net LDM following [60] on the MuLAn dataset, as the data used in [60] is not available. When sampling from PixArt-α and our model, we employed DPM-Solver [29] , while for SD and Text2Layer we employed PNDM [27] , default recommended parameters were used for LayerDiffusion.In Fig. 7 , we visualise the impact of different training paradigms for our RGBA VAE on generated samples. We visualise images generated with a diffusion model fine-tuned on different VAE latent spaces. We can see that failing to disentangle the alpha and RGB channels (with a joint KL loss, or low regularisation weight) yields poor quality images with high contrast. When fine-tuning the VAE, we also experimented with increasing the latent channels from four to eight in order to enhance the expressiveness of the latent space. However, we observed that while this modification improves the VAE reconstruction capabilities, it makes fine-tuning the LDM significantly more challenging and costly. Therefore, we decided to preserve the original latent dimension and encode both RGB and transparency information with our disentangled approach.For the LDM, we experimented with parameter-efficient fine-tuning techniques such as LoRA [15] , but observing an excessive decrease in image quality, we finally opted for full fine-tuning of the network.To aid reader comprehension, we provide an algorithmic description of our multi-layer noise blending in Algorithm 1.Our work contributes positively to several domains. Artists, designers, and content creators can benefit from more precise control over their generated images, enhancing creativity and productivity. Additionally, the technology can be used in educational tools to help students and professionals learn about design, art, and computer graphics. Improved text-to-image generation can also make graphic design more accessible to individuals without extensive training in the field.However, as with any generative model, there is a risk that the technology could be used to create disinformation, generate fake profiles, or produce harmful content. There is also a risk that generated images could be used to infringe on personal privacy, particularly if the model is misused to create realistic but fake images of individuals. Furthermore, the generated images could be used in ways that compromise security, such as creating counterfeit documents or misleading imagery for fraud.To mitigate these risks, we propose several safeguarding measures. Releasing the model in a controlled manner can ensure that only verified and responsible users have access to it. Conducting thorough audits to ensure that the model does not perpetuate or amplify existing biases is also important. Maintaining transparency about the capabilities and limitations of the model, and establishing accountability protocols for its use, will help manage its impact. Providing resources and guidelines for ethical use, along with training for users on the potential societal impacts of the technology, can further promote responsible usage. Finally, developing features that can detect and prevent the creation of malicious content, such as watermarking techniques and usage restrictions, will enhance security measures.Algorithm 1: Multi-layer scene compositionInputs: K pre-generated RGBA instances {I k , M k }, A bounding box based scene layout L = {[cx k , cy k , w k , h k ], for k ∈ 1, • • • , K}, A latent diffusion model (VAE E, diffusion model D)T generation timesteps, Random noise η ∈ N (0, I) Number of blending timesteps n, background blending timesteps b, cross-layer consistency timesteps n s Output: K + 1 multi-layer images (background, K images with increasing number of instances) Generation set-up Rescale {I k , M k } according to [cx k , cy k , w k , h k ] x 0 k ← E(I k ) m k ← Downsample M k to latenty 0 t-1 ← y 0 t-1 • m * + 1 2 (y 0 t-1 + y K t-1 ) • (1 -m * ) If t ≥ n: ▷ Scene composition For k = 0 to K: y k t-1 ← y k-1 t-1 • (1 -m k ) + x k t-1 • m k Elif t ≥ n + n s :▷ cross-layer consistency optional step For k = 0 to K:y K t-1 ← y K t-1 • (1 -m k ) + y k-1 t-1 • m k D LimitationsRGBA generation. Our RGBA generator is trained on a relatively small dataset, and doesn't leverage training data with intrinsic transparency information. We would expect substantial image quality improvement by increasing training data volume and quality. As our approach requires full fine-tuning of the diffusion model (including the VAE), interaction with pre-trained diffusion models is more challenging, as models operate in different latent spaces. Finally, our mutual conditioning sampling increases computation complexity at inference time. As our model was trained to handle different sampling strategies, we leave it up to the user to determine whether the fine-grained improvements observed are needed for generation tasks.Scene compositing. By combining image inversion and multi-layer composition, we achieve much greater flexibility and control at the cost of increased computational and framework complexity. Future work will seek to simplify and automatise the generation pipeline, notably through automated instance prompt and layout generation. Generating instances independently can make scene composition challenging and require prompt engineering to obtain the right instance. Combining RGBA generation with techniques like ControlNet, or conditioning RGBA generation with background images has the potential to address these limitations.In Fig. 8 we present additional examples of generated RGBA instances. Our model is able to successfully cover a broad range of subjects, including realistic portraits, stylised figures, inanimate objects, and fantasy characters.In Fig. 9 , we provide additional examples of scene compositions, highlighting our model's ability to successfully build complex scenes with fine-grained attributes and instance ordering. Limitations of competing methods are observed here as well, with multi-diffusion struggling with overlapping instances and relative positioning (couch image, flower vase image) and GLIGEN struggling to assign the right attributes to instances (turtle image, marble table in couch image). Contemporary method Instance Diffusion shows similar performance to ours here, as attributes and patterns are simpler.For completeness, we additionally provide experiments with LayerDiffusion [57] , aiming to generate similar scenes. However, the model does not allow to use layout to guide the generation process, with instances being generated based on background content only. Our experiments, using publicly available code, have shown that instances tend to be generated in the middle of the image, making complex composition difficult. Results are provided in Fig. 14 .In order to find the best parameters for the classifier-free Guidance Scale (GS) and Guidance Rescaling (GR), we perform a grid search on validation data, measuring KID to evaluate image quality. As it can be observed in Fig. 10 , we obtain the best result (lowest KID), with GS = 2.5, GR = 0.25We provide a visual analysis of the impact of our key noise blending parameters on the generated image output. In Fig. 11 , we visualise different images when modifying n (number of timesteps where inverted instances noise is injected) and b (number of steps where background noise is blended with the composite image). We can see that n is the most crucial parameter: when n is too low, no control over image content is achieved and the scene is solely controlled by the prompt input.Increasing n increases faithfulness to the original RGBA instance appearance, at the cost of reduced blending quality. We observed that setting n = 30 yields the smoothest blending, maintaining instance attributes and positioning while at the same time adjusting appearance to fit the scene composition better.Parameter b has a subtler impact, adjusting the generated background image to blend with the scene layout more accurately. We can see notably in Fig. 11 that the blanket under the cat is adjusted and looks more realistic when b > 0. However, setting b too high risks introducing instance information in the background area, strongly affecting the overall generated scene.Next, we visualise the impact of parameter n s , which enforces consistency between the composite scene and individual layers' instance areas in Fig. 12 . For this experiment, we visualise intermediate layers as well, to highlight cross-layer consistency. We can see that setting n s > 0 substantially improves consistency of instance appearance between individual instance layers and the final composite scene. While we considered introducing a similar consistency regularisation between individual layers, we observed reduced composition performance with this kind of strategy.Finally, in Fig. 13 we visualise our method's consistency across multiple random seeds compared to competing methods Multidiffusion [2] , GLIGEN [24] and Instance Diffusion [49] . By first generating instances and compositing them, we achieve strong scene consistency when modifying the random seed during our blending procedure. In contrast, GLIGEN and Instance Diffusion struggle to maintain attributes (including object types), while Multidiffusion often struggles with overlapping instances and their relative positioning. (b) a red leather armchair next to a vase of tulips on top of a round marble coffee table in front of a black leather couch in a grey carpeted living room.(c) A white beachball in front of a canoe next to a tortoise on a beach with a red sailboat.(d) a book with a drawing of a sunflower on the cover in front of a bouquet of red roses in a blue vase. • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes a text-to-image (T2I) diffusion model (DM) to generate separate objects (with RGBA masks to represent transparency information). The proposed method can generate complex images with a high degree of controllability (e.g, instance appearance and location).

The reviewers have mixed opinions. The significant concern of the reviewers is the lack of novelty of the proposed method. More specifically, three reviewers pointed out the lack of comparison with LayerDiff [1] and LayerDiffusion [2]. Although this paper and the previous works [1,2] have a high similarity, I can't entirely agree with the reviewers' opinion that these works should be compared with the proposed method at the submission time. Both of them were uploaded to ArXiv this year (Feb and Mar), and the submission deadline for NeurIPS was May. Also, only [2] is published at TOG in this July. In my opinion, they are more like concurrent works rather than previous works.

- [1] LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model
- [2] Transparent Image Layer Diffusion using Latent Transparency

In the rebuttal document and comments, the authors have provided the comparison results with LayerDiffusion qualitatively and quantitatively. I think that the additional experimental results provided by the authors support that the proposed method and LayerDiffusion have distinct properties and there are some scenarios when the proposed method works better (e.g., when we need more controllability by text, or layout image).

In terms of evaluation, I partially agree with Reviewer GbJP that this paper can include more baselines, but at the same time, I partially agree with the rebuttal, except LayerDiffusion, they chose the best methods that were available at the submission time. I personally recommend the authors to add more baseline methods despite they are not current state-of-the-art methods, to emphasize the contribution of the proposed method.

Except for the novelty and the evaluation issue, I think the remaining weaknesses are not significant issues.

Considering the advantage of the proposed method (high controllability, enabling more controllable factors compared to the existing state-of-the-arts -- including LayerDiffusion) (wSp1, S8j5, tSA3), and the novelty issue seems to be partially resolved by the rebuttal, I recommend acceptance of this paper.

Reviews: review: paper_summary: paper_summary: This paper describes a diffusion model based method for layered text-to-image generation with RGBA masks (transparency information). This is a useful approach when aiming to generate complex images with many objects. To that end, the base VAE as well as the latent diffusion model are adapted to handle another channel and fine-tune to a corresponding dataset. The method is evaluated against base models and visual results show convincing results.
strength_weakness: strength_weakness: My main concern is regarding evaluation. While the method demonstrates that it works (visually), and quantitative evaluation shows that it performs better than off-the-shelf text-to-image models, there is a complete lack of comparison against methods that are more relevant to that particular task. The paper mentions many related works, but evaluation is only against base models, and comparisons to alternative methods are missing. 
  - Naive Text2Layer extension to multiple layers instead of just two
  - No editing / inpainting methods (discussed in related work) are used for comparison
  - LayerDiff https://arxiv.org/abs/2403.11929
  - No comparison to controllability methods such as ReCo, BoxDiff etc.
  - No comparison to Layered rendering diffusion model for zero-shot guided image synthesis
Because of this, it is difficult to judge the technical contributions and novelty of the proposed method. Mainly because it seems like a straightforward adaptation of an existing pipeline through fine-tuning. More justification and experimental evidence are required to demonstrate and ablate the necessary design choices to go from an existing backbone to the proposed method.
questions: questions: -
limitations: limitations: -
paper_summary: paper_summary: This paper proposes to use a diffusion model to generate separate objects and then apply multi-layer noise blending to build a composite scene. A RGBA generator is finetuned from a latent diffusion model to generate alpha transparency for objects in addition to RGB. A transparency-aware training procedure is designed for both VAE and diffusion models. Then the generated objects are iteratively injected into a scene with a background according to a specific layout.
strength_weakness: strength_weakness: The multi-layer scene composition method requires generating K + 1 images for composition, which is inefficient. Since RGBA objects have already been generated, there might be more efficient ways to fuse them in one step or reduce the cost of each iteration.
questions: questions: A more detailed algorithm for 3.3 might be necessary to clarify how all the mechanisms work together, since the diffusion processes in the first n steps and b steps are different.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces a novel method for generating complex images from textual descriptions with a high degree of control over object attributes and scene composition. They introduce a new training paradigm for adapting a diffusion model to generate isolated objects with transparency, using a disentangled representation for RGB and alpha channels. Moreover, a multi-layer scene compositing strategy based on noise blending is developed, enabling the generation and manipulation of complex scenes from multiple instance images. Overall, the task is interesting and the proposed method achieves fantastic generation ability.
strength_weakness: strength_weakness: 1. I think this paper ignores a similar and pioneer work: LayerDiffusion [1], which achieves the RGBA generation by finetuning the VAE and diffusion models. It is necessary that this paper should compare with this prior work. 
2. It seems that the Multi-Layer Noise Blending would cause unnatural image generation. I hope you can provide more discussion about the composition of foreground and background.
3. A deep and clear literature review of prior works is very important for an excellent work. More discussions about the LayerDiffusion[1] should be included in this paper.

[1] Zhang L, Agrawala M. Transparent image layer diffusion using latent transparency[J]. arXiv preprint arXiv:2402.17113, 2024.
questions: questions: None
limitations: limitations: None
paper_summary: paper_summary: This study enhances text-based image generation diffusion models by introducing multi-layer noise blending and transparency-aware training procedures, enhancing control over the generated image content. This technique allows for finer control over the elements composing the image and improves the quality and consistency of the generated images through conditional sampling.
strength_weakness: strength_weakness: 1. Novelty: The field of layered-based image generation, which includes both depth-based and instance-based layering, has been thoroughly investigated. It remains unclear what theoretical or methodological innovations the author's approach offers compared to existing instance-layering methodologies. The capability for fine-grained control, as mentioned, seems achievable by most contemporary methods.

2. Evaluation Concerns: Recent developments in instance-based layering methods appear to have been neglected. An analytical discussion or empirical benchmarking against these newer methods is warranted. Pertinent literature includes:

 Huang, Runhui, et al. "LayerDiff: Exploring Text-guided Multi-layered Composable Image Synthesis via Layer-Collaborative Diffusion Model." arXiv preprint arXiv:2403.11929 (2024).

 Zhang, Lvmin, and Maneesh Agrawala. "Transparent image layer diffusion using latent transparency." arXiv preprint arXiv:2402.17113 (2024).

3. Experimental Assessment: There appears to be a lack of quantitative evaluations for GLIGEN and MultiDiffusion methodologies. It would be helpful if the authors could clarify why quantitative comparisons with these methods were not included. Additionally, presenting visual results for entire scenes from non-layer-based methods, such as the SD series models, would be valuable in assessing whether layout-independent methods enable more coherent or effective inter-instance interactions.

4. The independent generation of instances introduces significant complexity and poses challenges in assembling them into a coherent scene, as noted by the authors.
questions: questions: How does the model manage unexpected object interactions or complex overlapping in scene compositing? Providing in-the-wild results that showcase complex object interactions and instance occlusions, such as scenarios depicting a person riding a motorcycle and a person wielding a wand, would help clarify the model’s capabilities in  intricate scene compositions.
limitations: limitations: The authors mention limitations briefly at the end of the conclusion.

Comments: comment: Thanks for the response. I think this is an interesting work. I will keep the current rating.
Dear reviewer, we thank you for upgrading your score. We have provided comparisons with all relevant state of the art open-source works, including now LayerDiffusion which we outperform, and discussed all modern closed source works in our related works. Our adaptation to transparent generation is entirely novel with no alternative works proposing disentangled latent spaces and mutual conditioning. Our second contribution, our multi-layer blending method, proposes novel ideas such as our iterative layer-wise instance integration process.  Please let us know what aspects of our work remain unsatisfactory so we can address them.
Dear reviewer, we thank you for your quick response. We have made efforts to address all your comments: we have provided a comparison to LayerDiffusion where we clearly outperform the method quantitatively with markedly different methodology, and have thoroughly discussed the limitations of multi-layer noise blending and our key innovations to address them. We are not aware of alternative works proposing to pre-generate RGBA instances for multi-layer scene composition with fine-grained control. Please let us know what aspects of our work remain unsatisfactory so we can address them.
Thanks for the respones. I have updated my score to BR because I still think that the technical contributions are limited and the experimental comparisons are not convincing enough to lean towards acceptance.
Thanks for the response!

I still think that the novelty is limited and the experimental results are not very convincing. I will keep my score.