Can MLLMs Understand the Deep Implication Behind Chinese Images?
Authors: name: Chenhao Zhang, name: Xi Feng, name: Yuelin Bai, name: Xinrun Du, name: Jinchang Hou, name: Kaixin Deng, name: Guangzeng Han, name: Qinrui Li, name: Bingli Wang, name: Jiaheng Liu, name: Xingwei Qu, name: Yifei Zhang, name: Qixuan Zhao, name: Yiming Liang, name: Ziqiang Liu, name: Feiteng Fang, name: Min Yang, name: Wenhao Huang, name: Chenghua Lin, name: Ge Zhang, name: Shiwen Ni

As the capabilities of Multimodal Large Language Models (MLLMs) continue to
improve, the need for higher-order capability evaluation of MLLMs is
increasing. However, there is a lack of work evaluating MLLM for higher-order
perception and understanding of Chinese visual content. To fill the gap, we
introduce the **C**hinese **I**mage **I**mplication understanding
**Bench**mark, **CII-Bench**, which aims to assess the higher-order perception
and understanding capabilities of MLLMs for Chinese images. CII-Bench stands
out in several ways compared to existing benchmarks. Firstly, to ensure the
authenticity of the Chinese context, images in CII-Bench are sourced from the
Chinese Internet and manually reviewed, with corresponding answers also
manually crafted. Additionally, CII-Bench incorporates images that represent
Chinese traditional culture, such as famous Chinese traditional paintings,
which can deeply reflect the model's understanding of Chinese traditional
culture. Through extensive experiments on CII-Bench across multiple MLLMs, we
have made significant findings. Initially, a substantial gap is observed
between the performance of MLLMs and humans on CII-Bench. The highest accuracy
of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an
impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional
culture images, suggesting limitations in their ability to understand
high-level semantics and lack a deep knowledge base of Chinese traditional
culture. Finally, it is observed that most models exhibit enhanced accuracy
when image emotion hints are incorporated into the prompts. We believe that
CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics
and Chinese-specific images, advancing the journey towards expert artificial
general intelligence (AGI). Our project is publicly available at
https://cii-bench.github.io/.