SEED-X: Multimodal Models with Unified Multi-granularity Comprehension
  and Generation
Authors: name: Yuying Ge; name: Sijie Zhao; name: Jinguo Zhu; name: Yixiao Ge; name: Kun Yi; name: Lin Song; name: Chen Li; name: Xiaohan Ding; name: Ying Shan
Published: 2024-04-22 17:56:09

Abstract: The rapid evolution of multimodal foundation model has demonstrated
significant progresses in vision-language understanding and generation, e.g.,
our previous work SEED-LLaMA. However, there remains a gap between its
capability and the real-world applicability, primarily due to the model's
limited capacity to effectively respond to various user instructions and
interact with diverse visual data. In this work, we focus on bridging this gap
through integrating two enhanced features: (1) comprehending images of
arbitrary sizes and ratios, and (2) enabling multi-granularity image
generation. We present a unified and versatile foundation model, namely,
SEED-X, which is able to model multi-granularity visual semantics for
comprehension and generation tasks. Besides the competitive results on public
benchmarks, SEED-X demonstrates its effectiveness in handling real-world
applications across various domains after instruction tuning. We hope that our
work will inspire future research into what can be achieved by versatile
multimodal foundation models in real-world applications. The models, codes, and
datasets will be released in https://github.com/AILab-CVC/SEED-X.