Show-o: One Single Transformer to Unify Multimodal Understanding and
  Generation
Authors: name: Jinheng Xie; name: Weijia Mao; name: Zechen Bai; name: David Junhao Zhang; name: Weihao Wang; name: Kevin Qinghong Lin; name: Yuchao Gu; name: Zhijie Chen; name: Zhenheng Yang; name: Mike Zheng Shou
Published: 2024-08-22 16:32:32

Abstract: We present a unified transformer, i.e., Show-o, that unifies multimodal
understanding and generation. Unlike fully autoregressive models, Show-o
unifies autoregressive and (discrete) diffusion modeling to adaptively handle
inputs and outputs of various and mixed modalities. The unified model flexibly
supports a wide range of vision-language tasks including visual
question-answering, text-to-image generation, text-guided
inpainting/extrapolation, and mixed-modality generation. Across various
benchmarks, it demonstrates comparable or superior performance to existing
individual models with an equivalent or larger number of parameters tailored
for understanding or generation. This significantly highlights its potential as
a next-generation foundation model. Code and models are released at
https://github.com/showlab/Show-o.