Generating Compositional Scenes via Text-to-image RGBA Instance
  Generation
Authors: name: Alessandro Fontanella; name: Petru-Daniel Tudosiu; name: Yongxin Yang; name: Shifeng Zhang; name: Sarah Parisot
Published: 2024-11-16 23:44:14

Abstract: Text-to-image diffusion generative models can generate high quality images at
the cost of tedious prompt engineering. Controllability can be improved by
introducing layout conditioning, however existing methods lack layout editing
ability and fine-grained control over object attributes. The concept of
multi-layer generation holds great potential to address these limitations,
however generating image instances concurrently to scene composition limits
control over fine-grained object attributes, relative positioning in 3D space
and scene manipulation abilities. In this work, we propose a novel multi-stage
generation paradigm that is designed for fine-grained control, flexibility and
interactivity. To ensure control over instance attributes, we devise a novel
training paradigm to adapt a diffusion model to generate isolated scene
components as RGBA images with transparency information. To build complex
images, we employ these pre-generated instances and introduce a multi-layer
composite generation process that smoothly assembles components in realistic
scenes. Our experiments show that our RGBA diffusion model is capable of
generating diverse and high quality instances with precise control over object
attributes. Through multi-layer composition, we demonstrate that our approach
allows to build and manipulate images from highly complex prompts with
fine-grained control over object appearance and location, granting a higher
degree of control than competing methods.