EvolveDirector: Approaching Advanced Text-to-Image Generation with Large
  Vision-Language Models
Authors: name: Rui Zhao; name: Hangjie Yuan; name: Yujie Wei; name: Shiwei Zhang; name: Yuchao Gu; name: Lingmin Ran; name: Xiang Wang; name: Zhangjie Wu; name: Junhao Zhang; name: Yingya Zhang; name: Mike Zheng Shou
Published: 2024-10-09 17:52:28

Abstract: Recent advancements in generation models have showcased remarkable
capabilities in generating fantastic content. However, most of them are trained
on proprietary high-quality data, and some models withhold their parameters and
only provide accessible application programming interfaces (APIs), limiting
their benefits for downstream tasks. To explore the feasibility of training a
text-to-image generation model comparable to advanced models using publicly
available resources, we introduce EvolveDirector. This framework interacts with
advanced models through their public APIs to obtain text-image data pairs to
train a base model. Our experiments with extensive data indicate that the model
trained on generated data of the advanced model can approximate its generation
capability. However, it requires large-scale samples of 10 million or more.
This incurs significant expenses in time, computational resources, and
especially the costs associated with calling fee-based APIs. To address this
problem, we leverage pre-trained large vision-language models (VLMs) to guide
the evolution of the base model. VLM continuously evaluates the base model
during training and dynamically updates and refines the training dataset by the
discrimination, expansion, deletion, and mutation operations. Experimental
results show that this paradigm significantly reduces the required data volume.
Furthermore, when approaching multiple advanced models, EvolveDirector can
select the best samples generated by them to learn powerful and balanced
abilities. The final trained model Edgen is demonstrated to outperform these
advanced models. The code and model weights are available at
https://github.com/showlab/EvolveDirector.