SynerGen-VL: Towards Synergistic Image Understanding and Generation with
  Vision Experts and Token Folding
Authors: name: Hao Li; name: Changyao Tian; name: Jie Shao; name: Xizhou Zhu; name: Zhaokai Wang; name: Jinguo Zhu; name: Wenhan Dou; name: Xiaogang Wang; name: Hongsheng Li; name: Lewei Lu; name: Jifeng Dai
Published: 2024-12-12 18:59:26

Abstract: The remarkable success of Large Language Models (LLMs) has extended to the
multimodal domain, achieving outstanding performance in image understanding and
generation. Recent efforts to develop unified Multimodal Large Language Models
(MLLMs) that integrate these capabilities have shown promising results.
However, existing approaches often involve complex designs in model
architecture or training pipeline, increasing the difficulty of model training
and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful
encoder-free MLLM capable of both image understanding and generation. To
address challenges identified in existing encoder-free unified MLLMs, we
introduce the token folding mechanism and the vision-expert-based progressive
alignment pretraining strategy, which effectively support high-resolution image
understanding while reducing training complexity. After being trained on
large-scale mixed image-text data with a unified next-token prediction
objective, SynerGen-VL achieves or surpasses the performance of existing
encoder-free unified MLLMs with comparable or smaller parameter sizes, and
narrows the gap with task-specific state-of-the-art models, highlighting a
promising path toward future unified MLLMs. Our code and models shall be
released.