Exploring the Deep Fusion of Large Language Models and Diffusion
  Transformers for Text-to-Image Synthesis
Authors: name: Bingda Tang; name: Boyang Zheng; name: Xichen Pan; name: Sayak Paul; name: Saining Xie
Published: 2025-05-15 07:43:23

Abstract: This paper does not describe a new method; instead, it provides a thorough
exploration of an important yet understudied design space related to recent
advances in text-to-image synthesis -- specifically, the deep fusion of large
language models (LLMs) and diffusion transformers (DiTs) for multi-modal
generation. Previous studies mainly focused on overall system performance
rather than detailed comparisons with alternative methods, and key design
details and training recipes were often left undisclosed. These gaps create
uncertainty about the real potential of this approach. To fill these gaps, we
conduct an empirical study on text-to-image generation, performing controlled
comparisons with established baselines, analyzing important design choices, and
providing a clear, reproducible recipe for training at scale. We hope this work
offers meaningful data points and practical guidelines for future research in
multi-modal generation.