mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with
  Modality Collaboration
Authors: name: Qinghao Ye; name: Haiyang Xu; name: Jiabo Ye; name: Ming Yan; name: Haowei Liu; name: Qi Qian; name: Ji Zhang; name: Fei Huang; name: Jingren Zhou; name: Anwen HU
Published: 2023-11-07 14:21:29

Abstract: Multi-modal Large Language Models (MLLMs) have demonstrated impressive
instruction abilities across various open-ended tasks. However, previous
methods primarily focus on enhancing multi-modal capabilities. In this work, we
introduce a versatile multi-modal large language model, mPLUG-Owl2, which
effectively leverages modality collaboration to improve performance in both
text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
with the language decoder acting as a universal interface for managing
different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
modules to facilitate modality collaboration and introduces a modality-adaptive
module that preserves modality-specific features. Extensive experiments reveal
that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
tasks and achieving state-of-the-art performances with a single generic model.
Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
pioneering path in the development of future multi-modal foundation models.