Instruct-Imagen: Image Generation with Multi-modal Instruction
Authors: name: Hexiang Hu; name: Kelvin C. K. Chan; name: Yu-Chuan Su; name: Wenhu Chen; name: Yandong Li; name: Kihyuk Sohn; name: Yang Zhao; name: Xue Ben; name: Boqing Gong; name: William Cohen; name: Ming-Wei Chang; name: Xuhui Jia
Published: 2024-01-03 19:31:58

Abstract: This paper presents instruct-imagen, a model that tackles heterogeneous image
generation tasks and generalizes across unseen tasks. We introduce *multi-modal
instruction* for image generation, a task representation articulating a range
of generation intents with precision. It uses natural language to amalgamate
disparate modalities (e.g., text, edge, style, subject, etc.), such that
abundant generation intents can be standardized in a uniform format.
  We then build instruct-imagen by fine-tuning a pre-trained text-to-image
diffusion model with a two-stage framework. First, we adapt the model using the
retrieval-augmented training, to enhance model's capabilities to ground its
generation on external multimodal context. Subsequently, we fine-tune the
adapted model on diverse image generation tasks that requires vision-language
understanding (e.g., subject-driven generation, etc.), each paired with a
multi-modal instruction encapsulating the task's essence. Human evaluation on
various image generation datasets reveals that instruct-imagen matches or
surpasses prior task-specific models in-domain and demonstrates promising
generalization to unseen and more complex tasks.