Multimodal Representation Alignment for Image Generation: Text-Image
  Interleaved Control Is Easier Than You Think
Authors: name: Liang Chen, name: Shuai Bai, name: Wenhao Chai, name: Weichu Xie, name: Haozhe Zhao, name: Leon Vinci, name: Junyang Lin, name: Baobao Chang

The field of advanced text-to-image generation is witnessing the emergence of
unified frameworks that integrate powerful text encoders, such as CLIP and T5,
with Diffusion Transformer backbones. Although there have been efforts to
control output images with additional conditions, like canny and depth map, a
comprehensive framework for arbitrary text-image interleaved control is still
lacking. This gap is especially evident when attempting to merge concepts or
visual elements from multiple images in the generation process. To mitigate the
gap, we conducted preliminary experiments showing that large multimodal models
(LMMs) offer an effective shared representation space, where image and text can
be well-aligned to serve as a condition for external diffusion models. Based on
this discovery, we propose Dream Engine, an efficient and unified framework
designed for arbitrary text-image interleaved control in image generation
models. Building on powerful text-to-image models like SD3.5, we replace the
original text-only encoders by incorporating versatile multimodal information
encoders such as QwenVL. Our approach utilizes a two-stage training paradigm,
consisting of joint text-image alignment and multimodal interleaved instruction
tuning. Our experiments demonstrate that this training method is effective,
achieving a 0.69 overall score on the GenEval benchmark, and matching the
performance of state-of-the-art text-to-image models like SD3.5 and FLUX.