Unified Multimodal Understanding and Generation Models: Advances,
  Challenges, and Opportunities
Authors: name: Xinjie Zhang; name: Jintao Guo; name: Shanshan Zhao; name: Minghao Fu; name: Lunhao Duan; name: Guo-Hua Wang; name: Qing-Guo Chen; name: Zhao Xu; name: Weihua Luo; name: Kaifu Zhang
Published: 2025-05-05 11:18:03

Abstract: Recent years have seen remarkable progress in both multimodal understanding
models and image generation models. Despite their respective successes, these
two domains have evolved independently, leading to distinct architectural
paradigms: While autoregressive-based architectures have dominated multimodal
understanding, diffusion-based models have become the cornerstone of image
generation. Recently, there has been growing interest in developing unified
frameworks that integrate these tasks. The emergence of GPT-4o's new
capabilities exemplifies this trend, highlighting the potential for
unification. However, the architectural differences between the two domains
pose significant challenges. To provide a clear overview of current efforts
toward unification, we present a comprehensive survey aimed at guiding future
research. First, we introduce the foundational concepts and recent advancements
in multimodal understanding and text-to-image generation models. Next, we
review existing unified models, categorizing them into three main architectural
paradigms: diffusion-based, autoregressive-based, and hybrid approaches that
fuse autoregressive and diffusion mechanisms. For each category, we analyze the
structural designs and innovations introduced by related works. Additionally,
we compile datasets and benchmarks tailored for unified models, offering
resources for future exploration. Finally, we discuss the key challenges facing
this nascent field, including tokenization strategy, cross-modal attention, and
data. As this area is still in its early stages, we anticipate rapid
advancements and will regularly update this survey. Our goal is to inspire
further research and provide a valuable reference for the community. The
references associated with this survey are available on GitHub
(https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).