The Synergy between Data and Multi-Modal Large Language Models: A Survey
  from Co-Development Perspective
Authors: name: Zhen Qin; name: Daoyuan Chen; name: Wenhao Zhang; name: Liuyi Yao; name: Yilun Huang; name: Bolin Ding; name: Yaliang Li; name: Shuiguang Deng
Published: 2024-07-11 15:08:11

Abstract: The rapid development of large language models (LLMs) has been witnessed in
recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the
modality from text to a broader spectrum of domains, attracting widespread
attention due to the broader range of application scenarios. As LLMs and MLLMs
rely on vast amounts of model parameters and data to achieve emergent
capabilities, the importance of data is receiving increasingly widespread
attention and recognition. Tracing and analyzing recent data-oriented works for
MLLMs, we find that the development of models and data is not two separate
paths but rather interconnected. On the one hand, vaster and higher-quality
data contribute to better performance of MLLMs, on the other hand, MLLMs can
facilitate the development of data. The co-development of multi-modal data and
MLLMs requires a clear view of 1) at which development stage of MLLMs can
specific data-centric approaches be employed to enhance which capabilities, and
2) by utilizing which capabilities and acting as which roles can models
contribute to multi-modal data. To promote the data-model co-development for
MLLM community, we systematically review existing works related to MLLMs from
the data-model co-development perspective. A regularly maintained project
associated with this survey is accessible at
https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.