DiffusionGPT: LLM-Driven Text-to-Image Generation System
Authors: name: Jie Qin; name: Jie Wu; name: Weifeng Chen; name: Yuxi Ren; name: Huixia Li; name: Hefeng Wu; name: Xuefeng Xiao; name: Rui Wang; name: Shilei Wen
Published: 2024-01-18 15:30:58

Abstract: Diffusion models have opened up new avenues for the field of image
generation, resulting in the proliferation of high-quality models shared on
open-source platforms. However, a major challenge persists in current
text-to-image systems are often unable to handle diverse inputs, or are limited
to single model results. Current unified attempts often fall into two
orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate
expert model to output. To combine the best of both worlds, we propose
DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified
generation system capable of seamlessly accommodating various types of prompts
and integrating domain-expert models. DiffusionGPT constructs domain-specific
Trees for various generative models based on prior knowledge. When provided
with an input, the LLM parses the prompt and employs the Trees-of-Thought to
guide the selection of an appropriate model, thereby relaxing input constraints
and ensuring exceptional performance across diverse domains. Moreover, we
introduce Advantage Databases, where the Tree-of-Thought is enriched with human
feedback, aligning the model selection process with human preferences. Through
extensive experiments and comparisons, we demonstrate the effectiveness of
DiffusionGPT, showcasing its potential for pushing the boundaries of image
synthesis in diverse domains.