Title: MixDiffusion: Mixing Diffusion-based Uni-condition Text-to-Image Generation Models for Multi-conditions Image Synthesis
Primary area: generative models
Keywords: Text-to-Image Sythesis, Diffusion Model, CoT

Abstract: Recently, there has been a notable surge in the field of text-to-image (T2I) generation thanks to the powerful generation ability of diffusion models. To achieve controllable image generation, researchers explore to leverage additional control conditions besides text to guide image generation with diffusion models. Despite their advancements, most existing diffusion-based T2I generation models are primarily focused on one type of condition (uni-condition) controllable image generation. However, to better cater to users’ increasing demands for more detailed and creative outputs, there is a growing need for models to simultaneously support multiple control conditions (multi-conditions) for controllable image generation. In this work, we seek to expand the control conditions for diffusion-based T2I generation models and propose MixDiffusion, a training-free diffusion-based image generation framework supporting an unlimited number of control conditions, e.g., bounding boxes, keypoints, sketches, etc., besides text by incorporating multiple pre-trained diffusion based uni-condition image generation models to collaboratively synthesize images that satisfy all specified control conditions. The key insight of the proposed approach is to derive the predicted noise distribution in each denoising step of the diffusion-based multi-conditions image generation model MixDiffusion from the predicted noise distributions of multiple diffusion-based uni-condition models with a derived integration formula, which is supported by rigorous theory proof. The training-free manner makes the proposed MixDiffusion highly convenient for deployment and for adding new type of control conditions as needed.