MM-LLMs: Recent Advances in MultiModal Large Language Models
Authors: name: Duzhen Zhang; name: Yahan Yu; name: Chenxing Li; name: Jiahua Dong; name: Dan Su; name: Chenhui Chu; name: Dong Yu
Published: 2024-01-24 17:10:45

Abstract: In the past year, MultiModal Large Language Models (MM-LLMs) have undergone
substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or
outputs via cost-effective training strategies. The resulting models not only
preserve the inherent reasoning and decision-making capabilities of LLMs but
also empower a diverse range of MM tasks. In this paper, we provide a
comprehensive survey aimed at facilitating further research of MM-LLMs.
Specifically, we first outline general design formulations for model
architecture and training pipeline. Subsequently, we provide brief
introductions of 26 existing MM-LLMs, each characterized by its specific
formulations. Additionally, we review the performance of MM-LLMs on mainstream
benchmarks and summarize key training recipes to enhance the potency of
MM-LLMs. Lastly, we explore promising directions for MM-LLMs while concurrently
maintaining a real-time tracking website for the latest developments in the
field. We hope that this survey contributes to the ongoing advancement of the
MM-LLMs domain.