Can MLLMs Understand the Deep Implication Behind Chinese Images?
Authors: name: Chenhao Zhang; name: Xi Feng; name: Yuelin Bai; name: Xinrun Du; name: Jinchang Hou; name: Kaixin Deng; name: Guangzeng Han; name: Qinrui Li; name: Bingli Wang; name: Jiaheng Liu; name: Xingwei Qu; name: Yifei Zhang; name: Qixuan Zhao; name: Yiming Liang; name: Ziqiang Liu; name: Feiteng Fang; name: Min Yang; name: Wenhao Huang; name: Chenghua Lin; name: Ge Zhang; name: Shiwen Ni
Published: 2024-10-17 17:59:24

Abstract: As the capabilities of Multimodal Large Language Models (MLLMs) continue to
improve, the need for higher-order capability evaluation of MLLMs is
increasing. However, there is a lack of work evaluating MLLM for higher-order
perception and understanding of Chinese visual content. To fill the gap, we
introduce the **C**hinese **I**mage **I**mplication understanding
**Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
out in several ways compared to existing benchmarks. Firstly, to ensure the
authenticity of the Chinese context, images in CII-Bench are sourced from the
Chinese Internet and manually reviewed, with corresponding answers also
manually crafted. Additionally, CII-Bench incorporates images that represent
Chinese traditional culture, such as famous Chinese traditional paintings,
which can deeply reflect the model's understanding of Chinese traditional
culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
have made significant findings. Initially, a substantial gap is observed
between the performance of MLLMs and humans on CII-Bench. The highest accuracy
of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
culture images, suggesting limitations in their ability to understand
high-level semantics and lack a deep knowledge base of Chinese traditional
culture. Finally, it is observed that most models exhibit enhanced accuracy
when image emotion hints are incorporated into the prompts. We believe that
CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
and Chinese-specific images, advancing the journey towards expert artificial
general intelligence (AGI). Our project is publicly available at
https://cii-bench.github.io/.