Can Large Language Models Help Multimodal Language Analysis? MMLA: A
  Comprehensive Benchmark
Authors: name: Hanlei Zhang, name: Zhuohang Li, name: Yeshuang Zhu, name: Hua Xu, name: Peiwu Wang, name: Haige Zhu, name: Jie Zhou, name: Jinchao Zhang

Multimodal language analysis is a rapidly evolving field that leverages
multiple modalities to enhance the understanding of high-level semantics
underlying human conversational utterances. Despite its significance, little
research has investigated the capability of multimodal large language models
(MLLMs) to comprehend cognitive-level semantics. In this paper, we introduce
MMLA, a comprehensive benchmark specifically designed to address this gap. MMLA
comprises over 61K multimodal utterances drawn from both staged and real-world
scenarios, covering six core dimensions of multimodal semantics: intent,
emotion, dialogue act, sentiment, speaking style, and communication behavior.
We evaluate eight mainstream branches of LLMs and MLLMs using three methods:
zero-shot inference, supervised fine-tuning, and instruction tuning. Extensive
experiments reveal that even fine-tuned models achieve only about 60%~70%
accuracy, underscoring the limitations of current MLLMs in understanding
complex human language. We believe that MMLA will serve as a solid foundation
for exploring the potential of large language models in multimodal language
analysis and provide valuable resources to advance this field. The datasets and
code are open-sourced at https://github.com/thuiar/MMLA.