Can MLLMs Understand the Deep Implication Behind Chinese Images?
Authors: Chenhao Zhang, Xi Feng, yuelin bai, Xeron Du, Jinchang Hou, Kaixin Deng, Guangzeng Han, Qinrui Li, Bingli Wang, Jiaheng Liu, Xingwei Qu, Yifei Zhang, Qixuan Zhao, Yiming Liang, Ziqiang Liu, Feiteng Fang, Min Yang, Stephen Huang, Chenghua Lin, Ge Zhang, Shiwen Ni
Accepted at: acl25

Can MLLMs Understand the Deep Implication Behind Chinese Images?

Chenhao Zhang1,2,* Xi Feng2,3* Yuelin Bai2* Xinrun Du4,5*
Jinchang Hou2,3 Kaixin Deng6 Guangzeng Han7 Qinrui Li8 Bingli Wang9
Jiaheng Liu4 Xingwei Qu10 Yifei Zhang11 Qixuan Zhao2,3 Yiming Liang12
Ziqiang Liu2 Feiteng Fang2,3 Min Yang2,13 Wenhao Huang5 Chenghua Lin10
Ge Zhang4,5† Shiwen Ni2‡

1Huazhong University of Science and Technology
2Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences 3USTC 4M-A-P 501.ai 6Hokudai 7University of Memphis 8University of California, Santa Barbara 9SICAU 10University of Manchester 11SWU 12UCAS 13Shenzhen University of Advanced Technology

Abstract

As the capabilities of Multimodal Large Language Models (MLLMs) improve, the need for higher-order evaluation of them is increasing. However, there is a lack of work evaluating MLLM for higher-order perception and understanding of Chinese visual content. To address this, we introduce the CII-Bench, which aims to assess MLLMs’ such capabilities for Chinese images. To ensure the authenticity of the Chinese context, images in CII-Bench are sourced from the Chinese Internet and manually reviewed, with corresponding answers also manually crafted. Additionally, CII-Bench incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, which can deeply reflect the model’s understanding of Chinese traditional culture. Through experiments on multiple MLLMs using CII-Bench, significant findings emerged. There is a large gap between MLLMs and humans in performance. The highest MLLM accuracy is 64.4%, while the human average is 78.2% and the peak is 81.0%. MLLMs perform poorly on traditional culture images, indicating limitations in understanding high-level semantics and lacking a deep knowledge base of Chinese traditional culture. Moreover, most models have higher accuracy when image emotion hints are added to the prompts. We believe CII-Bench will help MLLMs better understand Chinese semantics and specific images, and move forward the development of expert artificial general intelligence (AGI). Our project is publicly available at https://cii-bench.github.io/.

1 Introduction

With the rapid advancement of artificial intelligence, Multimodal Large Language Models (MLLMs) (Liu et al., 2023b; Li et al., 2023c; Ye et al., 2023; Tong et al., 2024) have demonstrated exceptional performance across various domains, including natural language processing (Chowdhary and Chowdhary, 2020; Luo et al., 2024; Zhang et al., 2024a) and computer vision (Lu et al., 2022; Li et al., 2023b,a; Xu et al., 2023; Fu et al., 2023; Cai et al., 2023; Zhang et al., 2023; Chen et al., 2024b; Jin et al., 2024). These models are not only capable of processing and generating text but also excel at integrating and interpreting information across multiple modalities, such as images, sound, and video. However, despite the significant progress made in tasks like image recognition and generation, a crucial research question remains: Can these models truly understand and interpret images that have deep implications? Based on the experiments by (Liu et al., 2024b) on the English image implication understanding benchmark II-Bench, despite MLLMs like Claude-3.5-Sonnet achieving an impressive 80.9% accuracy, approaching the human average of 90.3%, there remains a significant gap in their higher-order perception abilities, particularly in nuanced emotional understand-

![Comparison of Chinese and English image implications. Chinese images often embody richer scenes and deeper implications with Chinese traditional culture compared with the straightforward and explicit symbolism in English images.](page_1012_370_388_246.png)

Figure 1: Comparison of Chinese and English image implications. Chinese images often embody richer scenes and deeper implications with Chinese traditional culture compared with the straightforward and explicit symbolism in English images.
ing and profound meaning extraction, emphasizing the need for more challenging benchmarks with richer scenes and deeper implications.

In contrast to English images, Chinese images often embody richer scenes (Xu, 2023) and deeper implications as Figure 1 shows. For instance, Chinese traditional landscape paintings not only depict natural scenery but also convey profound philosophical concepts, such as the harmony between humans and nature, through artistic techniques like the interplay of void and solid, the use of negative space, and the brushwork. As the famous Chinese poet Su Shi noted, “Poetry and painting share the same essence, embodying both craftsmanship and purity”. The depth of Chinese images lies not only in their aesthetic appeal but also in the underlying spirit and philosophy they express. Similarly, New Year paintings, as a significant carrier of Chinese traditional culture, typically use symbolism and implication to convey wishes for good fortune, prosperity, and peace. Unlike the directness often found in English imagery, Chinese images emphasize the creation of atmosphere and subtle expression, requiring viewers to possess certain cultural knowledge to accurately grasp their meanings. This cultural disparity leads to significant differences in the modes of expression and meaning conveyed between Chinese and English images, highlighting the need to consider cultural context when evaluating the capability of MLLMs to understand the deep implications of images.

To address this gap, we develop CII-Bench, a benchmark designed to comprehensively test the higher-order perception, reasoning, and understanding abilities of models within a Chinese context. This benchmark allows us to gain a clearer understanding of these models’ interpretive capacities, offering new insights into their application in cross-cultural environments, and thus advancing the research and development of MLLMs. As illustrated in Figure 2, CII-Bench comprises 698 images and 800 multiple-choice questions spanning six domains: Life, Art, Society, Politics, Environment, and Chinese Traditional Culture. Moreover, to ensure diversity, CII-Bench includes six types of images: Illustration, Meme, Poster, Single-panel Comic, Multi-panel Comic, and Painting. By employing images of various types and from different domains, the benchmark provides a more robust evaluation of models’ comprehension and reasoning abilities. We conduct extensive experiments to evaluate MLLMs on CII-Bench and deeply evaluate the model’s grasp of Chinese traditional culture. Our key contributions are listed as follows:

• We introduce CII-Bench, the first benchmark designed to assess the understanding of implications in Chinese images, which poses a significant challenge to current MLLMs.

• We design a comprehensive evaluation metric based on GPT-4o to evaluate Chinese traditional culture. This metric aligns more closely with human annotations and is better suited for evaluating Chinese traditional painting.

• Our experimental findings are as follows: (1) There is a notable performance gap between MLLMs and humans. Models demonstrate the highest accuracy of 64.4%, while human accuracy averages at 78.2% and best at 81.0%. (2) Closed-source models generally outperform open-source models, but the best-performing open-source model surpasses the top closed-source model, with a difference of more than 3%. (3) Models perform significantly worse in Chinese traditional culture compared to other domains, indicating that current models still lack sufficient understanding of Chinese culture. Further analysis shows GPT-4o only observes the surface-level information, it’s difficult to deeply interpret the complex cultural elements contained in Chinese traditional painting. (4) Incorporating image emotion hints into prompts generally improves model scores, indicating that models struggle with emotional understanding, leading to misinterpretation of the implicit meanings in the images.

2 Related Work

2.1 Multimodal Large Language Models

With the rapid development of large language models (LLMs) (Aakanksha et al., 2022; Won et al., 2022; Chiang et al., 2023; Touvron et al., 2023; OpenAI, 2023a,b; Team, 2024; Cai et al., 2024),
Multimodal Large Language Models (MLLMs) have made significant improvements. Many works incorporate additional module inputs on LLMs, effectively bridging the gap between visual and language. BLIP-2 (Li et al., 2023c) encodes images using ViT (Dosovitskiy et al., 2020) and employs a Q-Former to map visual features into the language space. LLaVA (Liu et al., 2023b,a, 2024a; Li et al., 2024a) utilizes an MLP as the connector between the visual encoder and the LLM backbone. Similarly, mPLUG-Owl2 (Ye et al., 2023) employs a modality-adaptive module to facilitate the collaboration between visual and language modalities by mapping them into a unified representation space. Subsequent works (Wang et al., 2023; Lu et al., 2024; Chen et al., 2024c; Young et al., 2024; Laurençon et al., 2024; GLM et al., 2024; Yao et al., 2024; Anthropic, 2024; Wang et al., 2024) further enhance MLLMs by designing novel modules for more sufficient modality alignment.

2.2 Image Implication Understanding

Image implication understanding represents a more complex and challenging task than conventional image understanding. This advanced cognitive process necessitates multi-hop reasoning ability and sophisticated theory of mind (ToM), capabilities that are intrinsic to human cognition (Desai et al., 2022; Hessel et al., 2023; Yang et al., 2024; Zhong et al., 2024; Strachan et al., 2024; Street et al., 2024; Horvitz et al., 2024). II-Bench (Liu et al., 2024b) is the first benchmark specifically designed to evaluate MLLMs in both image understanding and reasoning through English image implication.

3 The CII-Bench

3.1 Overview of CII-Bench

We present the Chinese Image Implication Understanding Benchmark (CII-Bench), a novel benchmark designed to assess the perceptual, reasoning, and comprehension abilities of MLLMs in the context of Chinese imagery. This benchmark includes a diverse range of visual content such as traditional Chinese traditional artworks, comics, and Chinese Internet memes, all rich in visual information and cultural significance. The main goal of CII-Bench is to evaluate whether current MLLMs can leverage their understanding and knowledge of Chinese culture to accurately interpret the deeper implications and abstract information within these images.

CII-Bench comprises 698 images across various categories, with detailed classification and domain statistics provided in Appendix A. These images are manually collected and annotated by 30 undergraduate students from different disciplines and institutions, sourced from several well-known image websites. Each image is paired with 1 to 3 multiple-choice questions, each offering six options with only one correct answer. One fixed question asks, “What is the implication in this image?” Additional questions for the same image probe different levels of understanding, such as overarching interpretation and nuanced details. The benchmark includes 800 multiple-choice questions, with 765 for the test set and 35 for developing and validating few-shot tasks. Figure 3 provides representative examples from CII-Bench.

3.2 Data Curation Process

3.2.1 Data Collection

We collect 17,695 raw images from various renowned illustration websites, ensuring a sufficiently extensive raw dataset. Our collectors are well instructed to adhere to copyright and license regulations, avoiding data from sites prohibiting copy and redistribution. For detailed information on the specific websites from which we collect images, please refer to Appendix C.

3.2.2 Data Filtration

After collecting the raw images, we meticulously design a three-stage data filtering process: In the first stage, we focus on image deduplication. We utilize image similarity algorithms for pixel-level comparison to eliminate duplicates and preserve dataset uniqueness; In the second stage, we regulate text prevalence in images. Optical Character Recognition (OCR) technology identifies textual areas and disqualifies images exceeding set textarea ratios, maintaining a visual-centric dataset; In the third stage, images undergo rigorous visual inspection, discarding those without metaphorical depth based on strict criteria. This process refines the dataset, rejecting over 95% of initial images and securing under 1,000 high-quality ones.

3.2.3 Data Annotation

The annotation process for the benchmark was meticulously designed through several steps to ensure rigor and precision as following. The detailed annotation protocol can be found in Appendix C.

Preparation and Consistency Check: Before formal annotation, annotators first acquaint them-
<table>
  <tr>
    <th>Life</th>
    <th>Art</th>
    <th>Society</th>
  </tr>
  <tr>
    <td>
      <b>Question:</b> What metaphor does this image contain?<br>
      <b>Option:</b><br>
      (A) Some people pursue certain things considering their own circumstances.<br>
      (B) Extinguishing a candle with a feather is an imaginative idea, although it failed, this kind of innovation is commendable.<br>
      (C) Everyone has the right to pursue fashion.
    </td>
    <td>
      <b>Question:</b> What does the distant island suggest?<br>
      <b>Option:</b><br>
      (A) The distant island is depicted as a symbol of spiritual sustenance, representing a place where people seek shelter from the world’s chaos.<br>
      (B) The distant island, together with the distant horizon, suggests a sense of harmony with nature, suggesting a harmonious coexistence between man and nature in the world beyond.<br>
      (C) The distant island symbolizes hope and a goal, which is worth seeking.
    </td>
    <td>
      <b>Question:</b> What metaphor does this image contain?<br>
      <b>Option:</b><br>
      (A) Perseverance is an important virtue.<br>
      (B) Parents’ behavior and habits determine the future of their children.<br>
      (C) The failure of education is due to parents not providing enough opportunities.<br>
      (D) Piano learning should start at an early age and be persisted with, as this is the path to success.<br>
      (E) Some parents blame the failure of education on their children, while neglecting their own responsibility.<br>
      (F) If parents do not lead by example and become role models, their children’s education will inevitably fail.
    </td>
  </tr>
  <tr>
    <td>
      <b>Image Type:</b> Multi-panel Comic<br>
      <b>Rhetoric:</b> Metaphor<br>
      <b>Emotion:</b> Negative<br>
      <b>Difficulty Level:</b> Easy
    </td>
    <td>
      <b>Image Type:</b> Illustration<br>
      <b>Rhetoric:</b> Symbolism<br>
      <b>Emotion:</b> Positive<br>
      <b>Difficulty Level:</b> Easy
    </td>
    <td>
      <b>Image Type:</b> Single-panel Comic<br>
      <b>Rhetoric:</b> Contrast<br>
      <b>Emotion:</b> Negative<br>
      <b>Difficulty Level:</b> Easy
    </td>
  </tr>
</table>

<table>
  <tr>
    <th>Chinese Traditional Culture</th>
    <th>Environment</th>
    <th>Politics</th>
  </tr>
  <tr>
    <td>
      <b>Question:</b> What metaphor does this image contain?<br>
      <b>Option:</b><br>
      (A) The desolate winter scene hints at the character’s longing for the warmth and comfort of home, the source of all things.<br>
      (B) Viewing the snowy landscape reflects the character’s serene and detached attitude towards worldly matters.<br>
      (C) The bleak winter scene implies the character’s inner loneliness and the sorrow of finding it hard to meet a kindred spirit.<br>
      (D) The act of looking up suggests that the character is in deep thought about life and the world.<br>
      (E) Being alone hints at the character’s longing for family and homeland.<br>
      (F) The bleak winter scene hints at the character’s inner sadness.
    </td>
    <td>
      <b>Question:</b> What metaphor does this image contain?<br>
      <b>Option:</b><br>
      (A) Symbols that create in the natural environment severely impacted by human activities, even facing the threat of environmental collapse.<br>
      (B) This image represents the rapid development of industrial technology, implying that future life will be more convenient and prosperous.<br>
      (C) The image shows the impact of modern industrial technologies, depicting a harmonious coexistence between industry and nature.<br>
      (D) Suggests that people have the ability to mitigate damage to the natural environment through environmental protection policies, making rational use of resources and energy.<br>
      (E) Expresses humanity’s complex conquest of nature by altering the Earth’s surface through technology.<br>
      (F) The image warns viewers about environmental pollution and ecological destruction, reminding viewers not to neglect the protection and preservation of the natural environment while pursuing industrial development.
    </td>
    <td>
      <b>Question:</b> What metaphor does this image contain?<br>
      <b>Option:</b><br>
      (A) The individual faces a moral dilemma and has been confronted with a group or higher authority.<br>
      (B) The conflict between the image of the angel and the devil suggests the tension between faith and reality, and how an individual can maintain their beliefs when faced with harsh realities.<br>
      (C) The image expresses the complexity of religious faith, expressing a longing for the spiritual world.<br>
      (D) The image may satisfy those who use war to intervene in international conflicts, but also expresses a desire for peace and concern about the consequences of war.<br>
      (E) An individual’s fate is influenced both by external forces and by their own choices.<br>
      (F) Even in times of peace, the threat of war can be ever-present; and even in war, people may still hold onto their desire for peace.
    </td>
  </tr>
  <tr>
    <td>
      <b>Image Type:</b> Painting<br>
      <b>Rhetoric:</b> Metaphor<br>
      <b>Emotion:</b> Positive<br>
      <b>Difficulty Level:</b> Hard
    </td>
    <td>
      <b>Image Type:</b> Poster<br>
      <b>Rhetoric:</b> Contrast<br>
      <b>Emotion:</b> Negative<br>
      <b>Difficulty Level:</b> Medium
    </td>
    <td>
      <b>Image Type:</b> Illustration<br>
      <b>Rhetoric:</b> Metaphor, Contrast<br>
      <b>Emotion:</b> Negative<br>
      <b>Difficulty Level:</b> Hard
    </td>
  </tr>
</table>

Figure 3: CII-Bench examples sampled from each domain. The pictures include life, art, society, Chinese traditional culture, environment and politics. Understanding these images and completing the corresponding questions requires strong comprehension. The original Chinese version in Appendix B.

selves with standard templates and guidelines. A pre-annotation round on a shared image batch ensures uniform standard understanding, with discrepancies resolved through discussion.

Multiple Rounds of Annotation and Cross-Validation: To reduce bias, each image receives annotations from two different annotators. Cross-validation follows, with a third-party review for significant discrepancies, guaranteeing accuracy.

Refinement of Annotation Content: Annotators annotate each image’s difficulty, type, emotional label, domain, and rhetorical devices based on specific criteria, ensuring consistency and comparability. They also craft 1 to 3 refined questions per image, each with one correct answer among five distractor options, including the default question, “What is the implication in this image?”

Context Analysis: During the annotation process, annotators assess the image’s cultural and background significance, especially for implications and rhetorical devices, consulting relevant materials for accuracy.

Post-Annotation Review: Upon completion, annotations undergo a thorough quality review for any oversight, errors, or inconsistencies. Based on the evaluation results, feedback is provided to the annotators, with re-annotations as necessary to maintain data quality.

3.3 Dataset Statistics

CII-Bench comprises 698 images, each accompanied by 1 to 3 multiple-choice questions, totaling 800 questions. We randomly select 35 of these questions to construct a few-shot development set and validation set. On average, each question is approximately 11 characters long, while each option has an average length of 28 characters. Additionally, each image is supplemented with a manually written description by the annotators, which provides a detailed explanation of the image’s content, nuances, and the human interpretation of its deep implication.

CII-Bench covers images across six distinct domains: Life, Art, Society, Politics, Environment, and Chinese Traditional Culture. The types of images are diverse, including Illustration, Meme, Poster, Single-panel Comic, Multi-panel Comic,
and Painting. Based on human understanding, these images are categorized into three levels of difficulty: Easy, Medium, and Hard. Moreover, the images are classified according to the emotional information they convey: Positive, Neutral, or Negative. Each image is also manually annotated with the rhetorical devices employed, including Metaphor, Exaggeration, Symbolism, Visual Dislocation, Antithesis, Analogy, Personification, and Contrast. Detailed statistical information is provided in Appendix A.

4 Experiment

We conduct systematic experiments on both open-source and closed-source MLLMs using CII-Bench. For each model, we employ eight different configurations: None (zero-shot), 1-shot, 2-shot, 3-shot, CoT, Domain, Emotion, and Rhetoric. “None” represents the use of a standard prompt without any additional information. “Emotion” indicates the inclusion of information related to the emotional polarity of the image (e.g., positive, negative) in the prompt. “Domain” involves adding information about the image’s domain (e.g., life, art), and “Rhetoric” refers to including details about the rhetorical devices used in the image (e.g., metaphor, contrast) in the prompt. Additionally, to verify the necessity of images in problem-solving, we select a portion of LLMs to complete tasks without image input. For consistency across all MLLMs and LLMs, we use identical prompts and experiment setup, with specific details available in Appendix D.

4.1 Baselines

MLLMs. To comprehensively evaluate CII-Bench, we carefully select a diverse range of MLLMs, encompassing both open-source and closed-source models, with the aim of covering a wide spectrum of model characteristics and scales. These models span parameter sizes from 7B to 100B, ensuring that models of varying complexity and capability are thoroughly assessed. In selecting the models, we focus on the following key aspects: 1) model diversity, 2) Open-Source vs. Closed-Source models, and 3) model parameter scaling law.

LLMs. To verify the critical role of images in answering questions, we specifically design an experiment in which some LLMs participate in the task without any image input. The purpose of this experiment is to assess whether these models can accurately understand the questions and make correct choices in the absence of image information, thereby further demonstrating the importance of images in the comprehension and problem-solving process. We select DeepSeek-67B, LLaMA-3-8B, and Qwen2-7b as the LLMs used in this experiment.

Evaluation. We use accuracy as the primary evaluation metric, multi-choice format questions and answer extraction method, which are widely used in previous benchmarks such as Helleswag (Zellers et al., 2019), MMMU (Yue et al., 2023), CMMMU (Zhang et al., 2024b), MMLU (Li et al., 2024b) and so on. Since CII-Bench is entirely composed of multiple-choice questions, the evaluation process only requires extracting the selected option from the model’s response, which simplifies the complexity of rule design. It is important to note that when models use chain-of-thought (CoT) prompts, the responses may include intermediate steps. Therefore, the evaluation rules must be sufficiently robust, or the model’s output must follow a fixed format. If the selected option cannot be extracted from the model’s response, the model is considered to have answered the question incorrectly. For the detailed statistics of the model output, please see Appendix F. For reference, we also select three Chinese PhD students to evaluate human performance on CII-Bench.

4.2 Main Results

In this section, we conduct a comprehensive comparison of the performance of various MLLMs, LLMs, and humans on CII-Bench. Detailed results across different domains and emotional dimensions are presented in Table 1, while different image types, difficulty levels, and rhetoric can be found in Appendix E. The main experimental results and findings are summarized as follows:

4.2.1 Natural Challenges of CII-Bench

This benchmark presents a significant challenge for current models. Notably, despite GPT-4o being an advanced model, its accuracy is only 54.1%, indicating substantial room for improvement. This reflects the rigorous and demanding nature of the benchmark. Further analysis reveals that most models perform worst in the domain of Chinese traditional culture, highlighting a significant deficiency in their understanding of Chinese cultural nuances. It is also noteworthy that human performance in this domain is not ideal, as questions related to Chinese traditional culture often require deep cul-
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">Overall (800)</th>
    <th colspan="2">Life (216)</th>
    <th colspan="2">Art (123)</th>
    <th colspan="2">Society (157)</th>
    <th colspan="2">Politics (21)</th>
    <th colspan="2">Env. (51)</th>
    <th colspan="2">CTC (130)</th>
    <th colspan="2">Positive (220)</th>
    <th colspan="2">Negative (247)</th>
    <th colspan="2">Neutral (231)</th>
  </tr>
  <tr>
    <th colspan="22">Open-source Models</th>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>34.3</td><td>27.9</td><td>34.7</td><td>32.5</td><td>45.8</td><td>55.2</td><td>36.5</td><td>34.0</td><td>35.1</td><td>33.6</td>
  </tr>
  <tr>
    <td>idefics2-8b</td><td>36.3</td><td>25.0</td><td>46.3</td><td>38.1</td><td>41.7</td><td>56.9</td><td>32.9</td><td>32.8</td><td>39.1</td><td>36.4</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-2.5</td><td>40.4</td><td>36.3</td><td>45.6</td><td>37.1</td><td>50.0</td><td>51.7</td><td>40.2</td><td>43.2</td><td>37.0</td><td>41.3</td>
  </tr>
  <tr>
    <td>CogVLM2-Llama3-Chinese-Chat</td><td>43.4</td><td>37.1</td><td>48.3</td><td>42.3</td><td>54.2</td><td>63.8</td><td>40.2</td><td>40.3</td><td>45.7</td><td>43.8</td>
  </tr>
  <tr>
    <td>MiniCPM-v2.6</td><td>45.0</td><td>37.5</td><td>47.6</td><td>49.5</td><td>58.3</td><td>55.2</td><td>42.3</td><td>45.6</td><td>44.6</td><td>44.9</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-34B</td><td>46.0</td><td>40.8</td><td>55.1</td><td>42.8</td><td>45.8</td><td>62.1</td><td>43.1</td><td>44.4</td><td>48.2</td><td>45.2</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-72B</td><td>48.0</td><td>43.8</td><td>48.3</td><td>49.5</td><td>70.8</td><td>60.3</td><td>43.8</td><td>41.5</td><td>52.5</td><td>49.2</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td><td>49.6</td><td>42.5</td><td>51.7</td><td>54.1</td><td>62.5</td><td>65.5</td><td>44.5</td><td>50.2</td><td>47.5</td><td>51.2</td>
  </tr>
  <tr>
    <td>GLM-4V-9b</td><td>50.3</td><td>46.7</td><td>48.3</td><td>53.6</td><td>54.2</td><td>62.1</td><td>48.2</td><td>51.9</td><td>52.9</td><td>46.3</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td><td>52.9</td><td>50.8</td><td>53.7</td><td>51.0</td><td>58.3</td><td>67.2</td><td>51.1</td><td>54.8</td><td>51.8</td><td>52.3</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td><td>53.1</td><td>49.2</td><td>53.1</td><td>55.7</td><td>62.5</td><td>63.8</td><td>50.4</td><td>50.6</td><td>53.3</td><td>55.1</td>
  </tr>
  <tr>
    <td>InternVL2-40B</td><td>57.9</td><td>55.8</td><td>55.1</td><td>61.9</td><td>62.5</td><td>70.7</td><td>52.6</td><td>54.4</td><td>58.0</td><td>60.8</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td><td><b>64.4</b></td><td><b>61.7</b></td><td><b>61.2</b></td><td><b>68.0</b></td><td><b>79.2</b></td><td><b>75.9</b></td><td><b>59.9</b></td><td><b>62.7</b></td><td><b>63.8</b></td><td><b>66.4</b></td>
  </tr>
  <tr>
    <th colspan="22">Closed-source Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>54.1</td><td>54.1</td><td>55.8</td><td>52.1</td><td>50.0</td><td>63.8</td><td>51.8</td><td>51.9</td><td>56.2</td><td>54.1</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td><td>54.1</td><td>52.1</td><td>61.9</td><td>52.6</td><td>62.5</td><td>46.6</td><td>53.3</td><td>52.7</td><td>56.5</td><td>53.0</td>
  </tr>
  <tr>
    <td>Qwen-VL-MAX</td><td>56.9</td><td>53.3</td><td>59.2</td><td>58.8</td><td>62.5</td><td>67.2</td><td>52.6</td><td>53.9</td><td>58.3</td><td>58.0</td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td><td>60.1</td><td><b>60.0</b></td><td><b>63.3</b></td><td><b>62.4</b></td><td><b>70.8</b></td><td>62.1</td><td>51.1</td><td>54.8</td><td>65.6</td><td>59.4</td>
  </tr>
  <tr>
    <td>GLM-4V</td><td>60.9</td><td>55.0</td><td>59.9</td><td>66.5</td><td>66.7</td><td>79.3</td><td>55.5</td><td>58.5</td><td>64.5</td><td>59.4</td>
  </tr>
  <tr>
    <th colspan="22">Text-Only Models</th>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct</td><td>21.7</td><td>22.2</td><td>26.9</td><td>18.6</td><td>25.0</td><td>27.8</td><td>20.4</td><td>21.2</td><td>24.4</td><td>19.5</td>
  </tr>
  <tr>
    <td>DeepSeek-67B-Chat</td><td>27.1</td><td>26.6</td><td>32.7</td><td>30.9</td><td>20.0</td><td>35.2</td><td>18.2</td><td>25.7</td><td>22.2</td><td>33.2</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td><td><b>32.5</b></td><td><b>33.2</b></td><td><b>34.6</b></td><td><b>30.9</b></td><td><b>35.0</b></td><td><b>40.7</b></td><td><b>28.5</b></td><td><b>33.6</b></td><td><b>30.4</b></td><td><b>33.6</b></td>
  </tr>
  <tr>
    <th colspan="22">Humans</th>
  </tr>
  <tr>
    <td>Human_avg</td><td>78.2</td><td>81.0</td><td>67.7</td><td>82.7</td><td>87.7</td><td>84.0</td><td>65.9</td><td>77.9</td><td>75.2</td><td>81.6</td>
  </tr>
  <tr>
    <td>Human_best</td><td><b>81.0</b></td><td><b>83.2</b></td><td><b>73.6</b></td><td><b>87.2</b></td><td><b>89.5</b></td><td><b>86.0</b></td><td><b>66.7</b></td><td><b>78.2</b></td><td><b>78.8</b></td><td><b>83.3</b></td>
  </tr>
</table>

Table 1: Overall results of different MLLMs, LLMs and humans on different domains and emotions. The best-performing model in each category is in-bold, and the second best is underlined.

tural knowledge. The lack of this knowledge base poses difficulties for both models and humans when dealing with Chinese cultural content. In addition, text-only models like DeepSeek-67B-Chat only get 27.1% accuracy, which shows that most of the questions in CII-Bench require image information to be answered correctly, proving that CII-Bench is highly dependent on visual content (Chen et al., 2024a).

4.2.2 Gap between Humans and MLLMs

The results indicate a significant gap between human performance and multimodal large models (MLLMs) on CII-Bench. Human participants achieved an average accuracy of 78.2%, with the highest accuracy reaching 81.0%. In contrast, the best-performing closed-source model, GLM-4V, achieved an accuracy of 60.9%, while the top open-source model, Qwen2-VL-72B, scored 64.4%. These findings highlight the substantial disparity between human abilities and even the most advanced models in understanding image implications. The highest accuracy achieved by the models is considerably lower than the average human score, indicating that multimodal large models still face significant challenges in this domain.

4.2.3 Model Performance across Different Domains and Emotions

In terms of domain performance, our results in Table 1 indicate that the models generally perform better in the Environment and Politics domains, achieving higher accuracy. Conversely, the accuracy is lower in the Life and Society domains, proving that everyday metaphors are generally more difficult in the Chinese context. The lowest score for the Chinese Traditional Culture and Art domains, which shows that while the models generalize well in common domains, they struggle with the more abstract and logically demanding information found in Chinese Traditional Culture and Art.

From an emotional perspective, the models tend to exhibit higher accuracy when the image implications convey negative emotions, while accuracy is the lowest for images with positive emotions. This
<table>
  <tr>
    <th>Model</th>
    <th>None</th>
    <th>CoT</th>
    <th>Domain</th>
    <th>Emotion</th>
    <th>Rhetoric</th>
  </tr>
  <tr>
    <th colspan="6">Open-source Models</th>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td>
    <td>34.3</td>
    <td>34.0</td>
    <td>32.1</td>
    <td>35.0</td>
    <td>33.4</td>
  </tr>
  <tr>
    <td>idefcs2-8b</td>
    <td>36.3</td>
    <td>33.3</td>
    <td>37.5</td>
    <td>38.6</td>
    <td>37.4</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-2.5</td>
    <td>40.4</td>
    <td>35.8</td>
    <td>41.1</td>
    <td>39.0</td>
    <td>34.8</td>
  </tr>
  <tr>
    <td>CogVLM2-Llama3-Chinese-Chat</td>
    <td>43.4</td>
    <td>42.6</td>
    <td>43.5</td>
    <td>44.0</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>MiniCPM-v2.6</td>
    <td>45.0</td>
    <td>38.9</td>
    <td>44.4</td>
    <td>45.4</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-34B</td>
    <td>46.0</td>
    <td>44.5</td>
    <td>46.4</td>
    <td>47.1</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-72B</td>
    <td>48.0</td>
    <td>45.3</td>
    <td>47.3</td>
    <td>48.6</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>49.6</td>
    <td>50.0</td>
    <td>51.0</td>
    <td>50.8</td>
    <td>49.3</td>
  </tr>
  <tr>
    <td>GLM-4V-9b</td>
    <td>50.3</td>
    <td>49.1</td>
    <td>49.9</td>
    <td>51.1</td>
    <td>49.5</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td>
    <td>52.9</td>
    <td>52.6</td>
    <td>54.1</td>
    <td>52.8</td>
    <td>53.5</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>53.1</td>
    <td>47.9</td>
    <td>53.5</td>
    <td>56.3</td>
    <td>53.8</td>
  </tr>
  <tr>
    <td>InternVL2-40B</td>
    <td><u>57.9</u></td>
    <td><u>57.6</u></td>
    <td><u>57.1</u></td>
    <td><u>60.0</u></td>
    <td><u>57.9</u></td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td><b>64.4</b></td>
    <td><b>62.1</b></td>
    <td><b>66.0</b></td>
    <td><b>64.3</b></td>
    <td><b>63.0</b></td>
  </tr>
  <tr>
    <th colspan="6">Closed-source Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>54.1</td>
    <td><b>54.9</b></td>
    <td>55.4</td>
    <td>54.9</td>
    <td>51.9</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>54.1</td>
    <td>51.6</td>
    <td>56.4</td>
    <td>53.5</td>
    <td>54.9</td>
  </tr>
  <tr>
    <td>Qwen-VL-MAX</td>
    <td>56.9</td>
    <td>54.0</td>
    <td><b>59.1</b></td>
    <td><b>59.9</b></td>
    <td>54.8</td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td>
    <td><b>60.1</b></td>
    <td><b>54.1</b></td>
    <td>59.0</td>
    <td>57.9</td>
    <td><b>55.6</b></td>
  </tr>
  <tr>
    <td>GLM-4V</td>
    <td><b>60.9</b></td>
    <td>48.8</td>
    <td><b>60.4</b></td>
    <td><b>60.6</b></td>
    <td><b>58.8</b></td>
  </tr>
</table>

Table 2: Overall results of different prompts on CII-Bench. The label (Emotion, Domain, Rhetoric) means providing corresponding information for the images in the prompt. The best-performing model in each category is in-bold, and the second best is underlined.

discrepancy highlights that the models’ preferences do not align with those of humans, as humans are significantly more sensitive to positive implications. The performance of the model is opposite to the conclusion shown in II-Bench (Liu et al., 2024b), reflecting the obvious difference in emotional expression in the Chinese and English contexts.

4.2.4 Analysis on different prompt skills

Analysis of Chain-of-Thought (CoT). In Table 2, we evaluate the impact of Chain-of-Thought (CoT) prompting on model performance. The results indicate that CoT does not significantly improve the accuracy of the models. In some cases, particularly with smaller open-source models, the accuracy even declined when CoT was used. For example, MiniCPM-v2.6 scores 45.0% without CoT, but this drops to 38.9% with CoT; similarly, LLaVA-1.6-72B scores decrease from 48.0% to 45.3%.

Upon analyzing the models’ responses, we find that those models showing a decrease in accuracy with CoT often suffer from overinterpretation, where questions that were initially answered correctly are misinterpreted after CoT is applied. Additionally, for questions that were originally answered incorrectly, CoT does not lead to significant improvements and sometimes even causes confusion, such as selecting multiple options. However, for most models, the probability of failing to extract an answer option from the response decreases after using CoT, which explains why some models show improved accuracy with CoT.

Analysis of Different Types and Domains. To evaluate the impact of different label information on model accuracy, we conduct an ablation study by providing relevant label information (such as emotion, domain, and rhetoric) in the prompts. The results in Table 2 show that emotion labels significantly improve model accuracy, followed by domain and rhetoric labels, both of which exhibit similar effectiveness.

This result aligns with human intuition. The answer options typically include negative, positive, and neutral choices. When the model receives emotional information, it can eliminate some irrelevant options, naturally leading to higher accuracy. In contrast, domain and rhetoric information generally do not effectively help the model eliminate options, resulting in more limited improvements. Additionally, from a model training perspective,
<table>
  <tr>
    <th>Model</th>
    <th>None</th>
    <th>1-shot</th>
    <th>2-shot</th>
    <th>3-shot</th>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>49.6</td>
    <td>44.1</td>
    <td>39.3</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>54.1</td>
    <td>51.8</td>
    <td>49.5</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>54.1</td>
    <td>55.4</td>
    <td>55.3</td>
    <td>55.4</td>
  </tr>
  <tr>
    <td>InternVL2-40B</td>
    <td>57.9</td>
    <td>53.0</td>
    <td>47.1</td>
    <td>41.9</td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td>
    <td>60.1</td>
    <td>57.4</td>
    <td>55.8</td>
    <td>55.4</td>
  </tr>
</table>

Table 3: Few-shot results of different models on the CII-Bench.

<table>
  <tr>
    <th>Model</th>
    <th>Overall</th>
    <th>Easy</th>
    <th>Middle</th>
    <th>Difficult</th>
    <th>Positive</th>
    <th>Negative</th>
    <th>Neutral</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>2.71</td>
    <td>3.0</td>
    <td>3.2</td>
    <td>2.35</td>
    <td>2.63</td>
    <td>3.0</td>
    <td>2.82</td>
  </tr>
</table>

Table 4: Overall result of Chinese traditional painting.

models tend to have a more mature understanding of emotions, while specific nouns in rhetoric and domain labels are often custom-defined. During pre-training, the model may not have encountered a large number of descriptions for such specific nouns, making these labels less helpful in improving accuracy.

Analysis of Few-shot Examples. The results in Table 3 indicate that few-shot examples do not improve the models’ accuracy. Specifically, performance declines as the number of examples increases. This decline can be attributed to the models’ inferior capabilities in handling multiple images compared to single images, leading to a decrease in accuracy with a higher number of shots. Furthermore, as the number of shots increases, the input length also extends, and the models’ ability to process long texts is inadequate, resulting in suboptimal performance with long contexts.

4.3 Evaluation of Chinese Traditional Culture

The Chinese traditional culture category is a distinctive feature of the CII-Bench dataset, where MLLMs consistently score the lowest. Therefore, we need a deeper evaluation of this field to analyze the extent to which MLLM understands Chinese traditional culture. We chose to deeply analyze MLLM’s understanding of Chinese traditional culture by evaluating Chinese traditional paintings.

4.3.1 Evaluation Metric

Chinese traditional painting, a cornerstone of Chinese traditional culture, encompasses a rich tapestry of styles and techniques developed over millennia. These paintings are typically categorized based on their subject matter (e.g., landscape paintings, flower-and-bird paintings, figure paintings, and New Year paintings) or their stylistic and skill (e.g., court paintings, meticulous brush paintings, freehand brush paintings, and color-and-ink paintings). Each category embodies unique characteristics that reflect China’s artistic evolution and philosophical underpinnings.

To comprehensively assess MLLMs’ understanding of Chinese traditional paintings, we develop a multifaceted evaluation metric. This metric is designed to probe both the surface-level information readily apparent in the artwork and the deeper culture and history that informs its creation and interpretation. Our evaluation metric encompasses five key perspectives: Surface-level Information, Aesthetic Characteristics, Brush and Ink Skills, Culture and History, and Deep Implications. For each perspective, we give its detailed description in Figure 12 in Appendix G.

4.3.2 LLM-based Chinese Traditional Painting Automatic Evaluation

To evaluate Chinese traditional painting comprehension in MLLMs, we develop an LLM-based evaluation standard based on evaluation metrics, as illustrated in Appendix G. Our experiment utilize the CTC domain data from CII-Bench, comprising 130 Chinese traditional paintings. We employ human-written descriptions and implication interpretations as ground truth. We choose GPT-4o to generate descriptions for these images, which are subsequently scored using GPT-4o and our evaluation standard. Please see the evaluation prompt in Appendix D. To validate the model’s scoring efficacy, we enlist three PhD students well-versed in Chinese metaphorical imagery to independently score the 130 paintings.

The model-human scoring consistency reached
98%, affirming the method’s validity for assessing Chinese traditional painting comprehension. Table 4 presents the detailed model scores. Analysis of these results, in conjunction with our evaluation standard, reveals insights across three dimensions: overall performance, difficulty levels, and emotions. The overall score of 2.71 indicates that while MLLM is able to observe the surface-level information of paintings, it has a large gap with humans in deeply interpreting the complex cultural elements contained in Chinese traditional art. In terms of difficulty evaluation, the model is consistent with human cognition, while in terms of emotion, the model has a higher negative score, indicating that the model can identify negative implications in paintings, such as using the past to satirize the present, and not appreciating talents.

4.4 Error Analysis

To conduct a comprehensive error analysis of GPT-4o’s performance (under CoT setting) on CII-Bench, we randomly select a total of 100 erroneous samples from various domains, distributed according to their proportions in the dataset. These samples are subjected to in-depth analysis by expert annotators. As illustrated in Figure 4, GPT-4o’s errors can be categorized into the following types: Information Neglect, Misunderstanding of Visual Information, Over-Inference, Superficial Reasoning, and Lack of Cultural Background Knowledge. For detailed analysis of cases, please see the Appendix J.

![Pie chart showing error response distribution: Information Neglect (36%), Over-Inference (25%), Lack of Cultural Background... (16%), Superficial Reasoning (12%), Misunderstanding of Visual Information (11%)](page_180_1042_1087_377.png)

Figure 4: GPT-4o error responses distribution.

Information Neglect (36%):
Complex images contain both visual and textual elements. Sole reliance on visual information makes accurate interpretation challenging due to diversity in meaning. Incorporating textual information clarifies the author’s emotional intent, aiding accurate interpretation. Unfortunately, GPT-4o frequently overlooks key visual (13%) and textual (23%) information. When directly asked about these elements, we find that GPT-4o can often answer correctly, indicating two main issues: 1) Insufficient image recognition abilities, and 2) Significant shortcomings in multimodal fusion, leading to underutilization of acquired information.

Over-Inference (25%):
During answer construction, distractors are included at surface and deep levels. GPT-4o often selects more exaggerated, deep-level incorrect options, ignoring narrower but correct ones, especially in Chinese memes. This suggests that GPT-4o has a preference for selecting abstract options.

Other Errors: The remaining errors are Lack of Cultural Background Knowledge (16%), Superficial Reasoning (12%), Misunderstanding of Visual Information (11%). The description of these errors can be found in Appendix H.

5 Conclusion

The development of CII-Bench marks a significant step forward in evaluating the capabilities of multimodal large models (MLLMs) and brings us closer to achieving expert artificial general intelligence (AGI). This benchmark promotes a deeper exploration of the higher-order theory of mind in MLLMs. Experimental results indicate that current MLLMs still exhibit a significant gap compared to humans in understanding the implications of images within a Chinese context. We found that most MLLMs lack a deep knowledge base of Chinese traditional culture, leading to a superficial understanding of this cultural content. Finally, the experiments showed that incorporating image emotion hints into prompts often improves model performance, suggesting that models still struggle with emotional understanding, which in turn leads to misinterpretation of implications. We believe that CII-Bench will inspire the academic community to further develop the next generation of multimodal foundational models that move toward expert AGI.

Limitations

We acknowledge several limitations in our study. Although CII-Bench is comprehensive, subjective elements can result in varying interpretations, impacting result consistency. In addition, in order to ensure high quality and practicability, our benchmark is not particularly large. The evaluation metrics may not fully capture the advanced understand-
ing and reasoning capabilities of AI systems. These limitations underscore the necessity for continuous refinement and expansion of our benchmarks. Future work will focus on developing and incorporating more stringent and objective test sets to enhance the reliability and validity of our benchmark.

Ethics Statement

In developing CII-Bench, we strictly adhere to ethical guidelines and legal regulations, ensuring fairness, transparency, inclusivity and respect for all stakeholders. We stress the importance of safeguarding privacy and intellectual property rights, underscoring our commitment to responsible and lawful data management. We have taken steps to anonymize any personal data to protect privacy and have made every effort to minimize harmful or biased content. However, we recognize that biases can inadvertently arise and some information may be potentially offensive. We are committed to continuous monitoring and improvement to mitigate such biases. Furthermore, we encourage users of our dataset to employ it responsibly and to consider the ethical implications of their work, particularly in applications that may impact individuals or communities.

Acknowledgments

This work was partially supported by National Key Research and Development Program of China (2022YFF0902100), National Natural Science Foundation of China (Grant No. 62376262), the Natural Science Foundation of Guangdong Province of China (2025B1515020032, 2024A1515030166), GuangDong Basic and Applied Basic Research Foundation (2023A1515110718 and 2024A1515012003), China Postdoctoral Science Foundation (2024M753398), Postdoctoral Fellowship Program of CPSF (GZC20232873).

References

Sharan Aakanksha, Maarten Jacob, Adam Gaurav, Chung Paul, Sebastian Charles, Kensen Parker, Joshua Sasha, et al. 2022. Palm: Scaling language modeling with pathways.

Anthropic. 2024. Claude 3.5 sonnet model card addendum.

Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, and Alex Kot. 2023. BenchLmm: Benchmarking cross-style visual capability of large multimodal models. arXiv preprint arXiv:2312.02896.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, et al. 2024. Internlm2 technical report.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024a. Are we on the right way for evaluating large vision-language models?

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, et al. 2024b. Are we on the right way for evaluating large vision-language models?

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024c. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023).

KR1442 Chowdhary and KR Chowdhary. 2020. Natural language processing. Fundamentals of artificial intelligence, pages 603–649.

Poorav Desai, Tanmoy Chakraborty, and Md Shad Akhtar. 2022. Nice perfume, how long did you marinate in it? multimodal sarcasm explanation. AAAI.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools.

Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest. In ACL.

Zachary Horvitz, Jingru Chen, Rahul Aditya, Harshvardhan Srivastava, Robert West, Zhou Yu, and Kathleen McKeown. 2024. Getting serious about humor:
Crafting humor datasets with unfunny large language models. In ACL.

Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiheng Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. 2024. MMTom-QA: Multimodal theory of mind question answering. In ACL.

Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, et al. 2024a. Llava-onevision: Easy visual task transfer.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2023a. Seed-bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, et al. 2024b. Cmmlu: Measuring massive multitask language understanding in chinese.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023c. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning.

Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, et al. 2024b. Ii-bench: An image implication understanding benchmark for multimodal large language models.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, et al. 2024. Deepseek-vl: Towards real-world vision-language understanding.

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS.

Zihan Luo, Xiran Song, Hong Huang, Jianxun Lian, Chenhao Zhang, Jinqi Jiang, and Xing Xie. 2024. Graphinstruct: Empowering large language models with graph understanding and reasoning capability.

OpenAI. 2023a. Chatgpt. https://chat.openai.com/.

OpenAI. 2023b. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufio, et al. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour.

Winnie Street, John Oliver Siy, Geoff Keeling, Adrien Baranes, Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Robin IM Dunbar, et al. 2024. Llms achieve adult human performance on higher-order theory of mind tasks. arXiv preprint arXiv:2405.18870.

Gemini Team. 2024. Gemini: A family of highly capable multimodal models.

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, , et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079.

Chung Won, Hou Le, Longpre Shayne, Zoph Barret, Tay Yi, Fedus William, Li Yunxuan, et al. 2022. Scaling instruction-finetuned language models.

Peng Xu, Wensi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023. Lyvm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265.

Qingshu Xu. 2023. Comparing covid-19 metaphors in chinese and english social media with critical metaphor analysis. Frontiers in Psychology.
Yixin Yang, Zheng Li, Qingxiu Dong, Heming Xia, and Zhifang Sui. 2024. Can large multimodal models uncover deep semantics behind images? In ACL.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mlm on your phone. arXiv preprint arXiv:2408.01800.

Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, et al. 2023. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence?

Chenhao Zhang, Renhao Li, Minghuan Tan, Min Yang, Jingwei Zhu, Di Yang, Jiahao Zhao, Guancheng Ye, Chengming Li, and Xiping Hu. 2024a. CPsyCoun: A report-based multi-turn dialogue reconstruction and evaluation framework for Chinese psychological counseling. In ACL.

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, et al. 2024b. Cmmmtu: A chinese massive multi-discipline multimodal understanding benchmark.

Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. arXiv preprint arXiv:2306.05179.

Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. 2024. Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation.
A  Statistics of CII-Bench

Please see the general statistics of CII-Bench in Table 5 and specific image type and domain statistics in Figure 5.

<table>
  <tr>
    <th colspan="2">Statistics</th>
  </tr>
  <tr>
    <td>Total Questions</td>
    <td>800</td>
  </tr>
  <tr>
    <td>Total Images</td>
    <td>698</td>
  </tr>
  <tr>
    <td>Dev : Validation : Test</td>
    <td>15 : 20 : 765</td>
  </tr>
  <tr>
    <td>Easy : Medium : Hard</td>
    <td>305 : 282 : 111</td>
  </tr>
  <tr>
    <td>Average Question Length</td>
    <td>10.54</td>
  </tr>
  <tr>
    <td>Average Option Length</td>
    <td>28.31</td>
  </tr>
  <tr>
    <td>Average Explanation Length</td>
    <td>121.06</td>
  </tr>
  <tr>
    <td>Metaphor</td>
    <td>562</td>
  </tr>
  <tr>
    <td>Exaggerate</td>
    <td>121</td>
  </tr>
  <tr>
    <td>Symbolism</td>
    <td>236</td>
  </tr>
  <tr>
    <td>Visual Dislocation</td>
    <td>42</td>
  </tr>
  <tr>
    <td>Antithesis</td>
    <td>13</td>
  </tr>
  <tr>
    <td>Analogy</td>
    <td>19</td>
  </tr>
  <tr>
    <td>Personification</td>
    <td>73</td>
  </tr>
  <tr>
    <td>Contrast</td>
    <td>87</td>
  </tr>
  <tr>
    <th colspan="2">Statistics</th>
  </tr>
  <tr>
    <td>Life</td>
    <td>216 (30.95%)</td>
  </tr>
  <tr>
    <td>Art</td>
    <td>123 (17.62%)</td>
  </tr>
  <tr>
    <td>Society</td>
    <td>157 (22.49%)</td>
  </tr>
  <tr>
    <td>Environment</td>
    <td>51 (7.31%)</td>
  </tr>
  <tr>
    <td>Politics</td>
    <td>21 (3.01%)</td>
  </tr>
  <tr>
    <td>Chinese Traditional Culture</td>
    <td>130 (18.62%)</td>
  </tr>
  <tr>
    <td>Positive</td>
    <td>220 (31.52%)</td>
  </tr>
  <tr>
    <td>Neutral</td>
    <td>247 (35.39%)</td>
  </tr>
  <tr>
    <td>Negative</td>
    <td>231 (33.09%)</td>
  </tr>
  <tr>
    <td>Illustration</td>
    <td>178 (25.50%)</td>
  </tr>
  <tr>
    <td>Meme</td>
    <td>145 (20.77%)</td>
  </tr>
  <tr>
    <td>Poster</td>
    <td>87 (12.46%)</td>
  </tr>
  <tr>
    <td>Multi-panel Comic</td>
    <td>34 (4.87%)</td>
  </tr>
  <tr>
    <td>Single-panel Comic</td>
    <td>143 (20.49%)</td>
  </tr>
  <tr>
    <td>Painting</td>
    <td>119 (17.05%)</td>
  </tr>
</table>

Table 5: General statistics of CII-Bench.

B  Original Chinese Version of CII-Bench Examples

Please see the original Chinese version examples of CII-Bench in Figure 6.

C  Data Annotation Protocol

This document outlines a comprehensive protocol for annotating a dataset consisting of questions that explore the metaphorical implications of images.

C.1  Data Collection

Some websites from which we collect data are as follows:

• https://fabiaoqing.com/biaoqing/lists/page.html
• https://www.sohu.com/a/282205200_439969
• https://www.sohu.com/a/300233985_616741
• https://www.zcool.com.cn/u/746800
• https://www.shencaitang.com/news/1940
• https://www.dpm.org.cn/collection/paints.html
• https://www.zuomeme.com/wangyou/all

C.2  General Guidelines

General Principles:

• Annotations should be accurate and consistent.

• All questions, options and explanations should be written in Chinese.

• Any images without metaphorical implications should be discarded.

Specific Instructions:

• Each image needs to be categorized as one of the following image types: single-panel comic, multi-panel comic, poster, meme, illustration or painting.

• Each image needs to be categorized as one of the following difficulty levels from a human understanding perspective: easy, middle, or hard.

• Each image needs to be categorized as one of the following domains: life, art, society, politics, environment or Chinese traditional culture.

• Each image needs to be categorized as one of the following emotions: positive, neutral or negative.

• Each image needs to be categorized as one or more of the following rhetoric: metaphor, exaggerate, symbolism, contrast, visual dislocation, antithesis, analogy, personification or others.

• Each image needs a human explanation and implication description.

• Each image needs 1-3 questions about the fine-grained metaphorical implications of the image, each with one correct answer and five distractor options.
<table>
  <tr>
    <th>Illustration</th>
    <th>Meme</th>
    <th>Poster</th>
  </tr>
  <tr>
    <td>Life (19, 10.68%)</td>
    <td>Life (138, 95.17%)</td>
    <td>Life (8, 9.20%)</td>
  </tr>
  <tr>
    <td>Art (79, 44.38%)</td>
    <td>Art (0, 0%)</td>
    <td>Art (33, 37.93%)</td>
  </tr>
  <tr>
    <td>Society (58, 32.58%)</td>
    <td>Society (4, 2.76%)</td>
    <td>Society (3, 3.45%)</td>
  </tr>
  <tr>
    <td>CTC (0, 0%)</td>
    <td>CTC (0, 0%)</td>
    <td>CTC (7, 8.05%)</td>
  </tr>
  <tr>
    <td>Environment (12, 6.74%)</td>
    <td>Environment (3, 2.07%)</td>
    <td>Environment (36,41.37%)</td>
  </tr>
  <tr>
    <td>Politics (10, 5.62%)</td>
    <td>Politics (0, 0%)</td>
    <td>Politics (0, 0%)</td>
  </tr>
  <tr>
    <th colspan="3">Multi-panel Comic</th>
  </tr>
  <tr>
    <td>Life (25, 73.53%)</td>
    <td>Life (27, 18.88%)</td>
    <td>Life (0, 0%)</td>
  </tr>
  <tr>
    <td>Art (0, 0%)</td>
    <td>Art (11, 7.69%)</td>
    <td>Art (0, 0%)</td>
  </tr>
  <tr>
    <td>Society (8, 23.53%)</td>
    <td>Society (88, 61.54%)</td>
    <td>Society (0, 0%)</td>
  </tr>
  <tr>
    <td>CTC (0, 0%)</td>
    <td>CTC (4, 2.80%)</td>
    <td>CTC (119, 100%)</td>
  </tr>
  <tr>
    <td>Environment (0, 0%)</td>
    <td>Environment (2, 1.40%)</td>
    <td>Environment (0, 0%)</td>
  </tr>
  <tr>
    <td>Politics (1, 2.94%)</td>
    <td>Politics (11, 7.69%)</td>
    <td>Politics (0, 0%)</td>
  </tr>
  <tr>
    <th colspan="3">Single-panel Comic</th>
  </tr>
  <tr>
    <td>Life (27, 18.88%)</td>
    <td>Art (11, 7.69%)</td>
    <td>Art (0, 0%)</td>
  </tr>
  <tr>
    <td>Society (88, 61.54%)</td>
    <td>Society (0, 0%)</td>
    <td>Society (0, 0%)</td>
  </tr>
  <tr>
    <td>CTC (4, 2.80%)</td>
    <td>CTC (119, 100%)</td>
    <td>CTC (0, 0%)</td>
  </tr>
  <tr>
    <td>Environment (2, 1.40%)</td>
    <td>Environment (0, 0%)</td>
    <td>Environment (0, 0%)</td>
  </tr>
  <tr>
    <td>Politics (11, 7.69%)</td>
    <td>Politics (0, 0%)</td>
    <td>Politics (0, 0%)</td>
  </tr>
  <tr>
    <th colspan="3">Painting</th>
  </tr>
  <tr>
    <td>Life (0, 0%)</td>
    <td>Art (0, 0%)</td>
    <td>Society (0, 0%)</td>
  </tr>
  <tr>
    <td>CTC (119, 100%)</td>
    <td>Environment (0, 0%)</td>
    <td>Politics (0, 0%)</td>
  </tr>
</table>

Figure 5: CII-Bench specific image type and domain statistics.

<table>
  <tr>
    <th>生活 Life</th>
    <th>艺术 Art</th>
    <th>社会 Society</th>
  </tr>
  <tr>
    <td>Question: 这张图片有什么趣味？<br>Option:<br>(A) 有人在做不寻常的事情<br>(B) 有自然景观<br>(C) 有动物<br>(D) 有建筑物<br>(E) 有食物<br>(F) 有其他有趣的事物<br>Image Type: 图片(Chinese Traditional Culture)<br>Rhetoric: 新闻<br>Emotion: 惊讶<br>Difficulty Level: 高难</td>
    <td>Question: 这张图片的小人做了什么？<br>Option:<br>(A) 小人正在做不寻常的事情<br>(B) 小人正在做自然景观<br>(C) 小人正在做动物<br>(D) 小人正在做建筑物<br>(E) 小人正在做食物<br>(F) 小人正在做其他有趣的事物<br>Image Type: 图片(Illustration)<br>Rhetoric: 新闻<br>Emotion: 惊讶<br>Difficulty Level: 高难</td>
    <td>Question: 这张图片有什么趣味？<br>Option:<br>(A) 有人在做不寻常的事情<br>(B) 有人正在做自然景观<br>(C) 有人正在做动物<br>(D) 有人正在做建筑物<br>(E) 有人正在做食物<br>(F) 有人正在做其他有趣的事物<br>Image Type: 图片(Poster)<br>Rhetoric: 新闻<br>Emotion: 惊讶<br>Difficulty Level: 高难</td>
  </tr>
  <tr>
    <th>中华传统文化 Chinese Traditional Culture</th>
    <th>环境 Environment</th>
    <th>政治 Politics</th>
  </tr>
  <tr>
    <td>Question: 这张图片有什么趣味？<br>Option:<br>(A) 画中的人或物展示了人物内心的活动和情感表达<br>(B) 画中的人或物展示了人物的思考<br>(C) 画中的人或物展示了人物的外貌和身份<br>(D) 画中的人或物展示了人物的环境<br>Image Type: 图片(Painting)<br>Rhetoric: 新闻<br>Emotion: 惊讶<br>Difficulty Level: 高难</td>
    <td>Question: 这张图片有什么趣味？<br>Option:<br>(A) 画中的人或物展示了人物内心的活动和情感表达<br>(B) 画中的人或物展示了人物的思考<br>(C) 画中的人或物展示了人物的外貌和身份<br>(D) 画中的人或物展示了人物的环境<br>Image Type: 图片(Poster)<br>Rhetoric: 新闻<br>Emotion: 惊讶<br>Difficulty Level: 高难</td>
    <td>Question: 这张图片有什么趣味？<br>Option:<br>(A) 画中的人或物展示了人物内心的活动和情感表达<br>(B) 画中的人或物展示了人物的思考<br>(C) 画中的人或物展示了人物的外貌和身份<br>(D) 画中的人或物展示了人物的环境<br>Image Type: 图片(Illustration)<br>Rhetoric: 新闻<br>Emotion: 惊讶<br>Difficulty Level: 高难</td>
  </tr>
</table>

Figure 6: CII-Bench examples sampled from each domain. The pictures include life, art, society, Chinese traditional culture, environment and politics. Understanding these images and completing the corresponding questions requires strong comprehension.
C.3  Data Quality Assurance
To further ensure the quality and reliability of the data, the annotated datasets were double-checked and cross-validated. Each question was manually validated by at least five annotators. Any inconsistencies or misinterpretations found were thoroughly examined and resolved by consensus of the annotation team, thus improving the reliability of the dataset while ensuring consistency of the annotations. In total, we conducted five rounds of data quality checks to ensure data quality and ultimately obtain CII-Bench.

C.4  Ethical Considerations
Copyright and Licensing. It is essential to strictly follow all copyright and licensing regulations. Data from sources that do not permit copying or redistribution will be explicitly excluded.

Data Privacy. Adherence to privacy laws and ethical standards in data handling is crucial. Annotators must avoid collecting questions that contain any personal information.

D  Experiment Setup
In experiments, we set the model temperature as 0, and all experiments are conducted on Nvidia A800 GPUs. The prompts of different settings are as follows Figure 7 to Figure 10. Particularly, the evaluation prompt of Chinese traditional painting is Figure 11.
<table>
  <tr>
    <th>Evaluation Prompt: Direct</th>
  </tr>
  <tr>
    <td>请根据提供的图片尝试回答下面的单选题。直接回答正确选项，不要包含额外的解释。请使用以下格式：“答案：$LETTER”，其中$LETTER是你认为正确答案的字母。<br><br>{question}<br>{options}<br><br>答案:</td>
  </tr>
</table>

Figure 7: The prompt used in direct output setting.

<table>
  <tr>
    <th>Evaluation Prompt: Keywords</th>
  </tr>
  <tr>
    <td>请根据提供的图片尝试回答下面的单选题。请使用以下格式：“答案：$LETTER”，其中$LETTER是你认为正确答案的字母。<br><br>关键词：{key_words}<br><br>{question}<br>{options}<br><br>答案:</td>
  </tr>
</table>

Figure 8: The prompt used in keyword setting.

<table>
  <tr>
    <th>Evaluation Prompt: CoT</th>
  </tr>
  <tr>
    <td>请尝试根据提供的图片回答以下单选题。让我们逐一思考每个选项，逐步分析。你回答的最后一行应该用以下格式：“答案：$LETTER”，其中$LETTER是你认为正确答案的字母。<br><br>{question}<br>{options}</td>
  </tr>
</table>

Figure 9: The prompt used in CoT setting.
<table>
  <tr>
    <th colspan="2">Evaluation Prompt: Few-Shot</th>
  </tr>
  <tr>
    <td colspan="2">
      请根据提供的图片尝试回答下面的单选题。直接回答正确选项，不要包含额外的解释。请使用以下格式：“答案：$LETTER”，其中$LETTER是你认为正确答案的字母。<br><br>
      {question 1}<br>
      {options 1}<br><br>
      答案：{answer 1}<br><br>
      (one or two more examples)<br><br>
      请根据提供的图片尝试回答下面的单选题。直接回答正确选项，不要包含额外的解释。请使用以下格式：“答案：$LETTER”，其中$LETTER是你认为正确答案的字母。<br><br>
      {question}<br>
      {options}<br><br>
      答案：
    </td>
  </tr>
</table>

Figure 10: The prompt used in Few-Shot setting.

# Role
You are an impartial judge, familiar with Chinese traditional culture and traditional paintings.

## Attention
You are responsible for evaluating the quality of the descriptions provided by the model for traditional Chinese paintings. Your evaluation should refer to the human answer and score based on the Evaluation Standard.

## Evaluation Standard
- [1 point]:
The description of the picture is incomplete and does not include any background information. It only mentions the most obvious elements in the picture, lacking recognition and understanding of traditional cultural elements.
- [2 points]:
The description of the image is relatively complete, but the background information is limited. Some traditional cultural elements can be identified, but their meaning is not explained in depth. There is a lack of insight into the aesthetic characteristics or skills of the image.
- [3 points]:
The description of the image is comprehensive and includes some background information. The traditional cultural elements can be accurately identified and briefly explained. There is a preliminary understanding of the aesthetic characteristics or skills of the picture, but there is a lack of in-depth analysis. Some historical or cultural background is mentioned, but it is not fully expanded.
- [4 points]:
The description of the image is detailed and contains rich background information. It is able to explain the significance of traditional cultural elements in depth. It provides a relatively in-depth analysis of the aesthetic characteristics and skills of the image. It provides relevant historical and cultural background and attempts to explore the implication of the image, but the implication is misunderstood.
- [5 points]:
The description of the image is comprehensive and accurate, and the background information is in-depth and comprehensive. It can not only accurately interpret traditional cultural elements, but also reveal their deep meaning. It has unique insights into the aesthetic characteristics and skills of the image. It deeply expounds on the relevant historical and cultural background, and can see through the philosophical thoughts, humanistic spirit or social values contained in the image, showing a deep understanding of Chinese traditional culture.

## Constraints
- Avoid any position biases and be as objective as possible
- Do not allow the length of the descriptions to influence your evaluation

## Workflow
Output your final verdict by strictly following this format: "[ratings]".

Figure 11: The prompt of Chinese traditional painting evaluation.
E   Results on Different Types, Difficulties and Rhetoric

In this section, we report the performance of different MLLMs and humans on different types of images, levels of difficulty, and rhetoric types. Please see the results in Table 6, Table 7 and Table 8.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="1">Overall</th>
    <th colspan="1">Illus.</th>
    <th colspan="1">Paint.</th>
    <th colspan="1">Poster</th>
    <th colspan="1">Single-C.</th>
    <th colspan="1">Multi-C.</th>
    <th colspan="1">Meme</th>
  </tr>
  <tr>
    <th colspan="8">Open-source Models</th>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>34.3</td><td>33.5</td><td>36.8</td><td>45.1</td><td>35.2</td><td>23.7</td><td>27.5</td>
  </tr>
  <tr>
    <td>idefics2-8b</td><td>36.3</td><td>44.0</td><td>32.8</td><td>45.1</td><td>35.2</td><td>23.7</td><td>24.8</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-2.5</td><td>40.4</td><td>39.5</td><td>38.4</td><td>49.0</td><td>42.6</td><td>34.2</td><td>37.3</td>
  </tr>
  <tr>
    <td>CogVLM2-Llama3-Chinese-Chat</td><td>43.4</td><td>45.0</td><td>39.2</td><td>52.9</td><td>45.5</td><td>23.7</td><td>39.2</td>
  </tr>
  <tr>
    <td>MiniCPM-v2.6</td><td>45.0</td><td>44.0</td><td>40.8</td><td>53.9</td><td>51.1</td><td>36.8</td><td>39.2</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-34B</td><td>46.0</td><td>50.0</td><td>44.0</td><td>48.0</td><td>47.7</td><td>29.0</td><td>42.5</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-72B</td><td>48.0</td><td>50.9</td><td>44.0</td><td>43.1</td><td>56.8</td><td>39.5</td><td>43.1</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td><td>49.6</td><td>47.7</td><td>43.2</td><td>0.8</td><td>58.0</td><td>31.6</td><td>46.4</td>
  </tr>
  <tr>
    <td>GLM-4V-9b</td><td>50.3</td><td>46.8</td><td>47.2</td><td>55.9</td><td>59.7</td><td>42.1</td><td>47.1</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td><td>52.9</td><td>48.2</td><td>50.4</td><td>59.8</td><td>62.5</td><td>39.5</td><td>49.7</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td><td>53.1</td><td>48.2</td><td>48.0</td><td>56.9</td><td>64.8</td><td>52.6</td><td>51.0</td>
  </tr>
  <tr>
    <td>InternVL2-40B</td><td>57.9</td><td>53.7</td><td>51.2</td><td>56.9</td><td>68.2</td><td>50.0</td><td>59.5</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td><td><b>64.4</b></td><td><b>61.5</b></td><td><b>59.2</b></td><td><b>68.6</b></td><td><b>70.5</b></td><td>47.4</td><td><b>67.3</b></td>
  </tr>
  <tr>
    <th colspan="8">Closed-source Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>54.1</td><td>54.1</td><td>50.4</td><td>56.9</td><td>54.6</td><td>47.4</td><td>57.5</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td><td>54.1</td><td>55.1</td><td><b>54.4</b></td><td>47.1</td><td>55.1</td><td>50.0</td><td>57.5</td>
  </tr>
  <tr>
    <td>Qwen-VL-MAX</td><td>56.9</td><td>57.3</td><td>51.2</td><td>60.8</td><td>62.5</td><td>39.5</td><td>56.2</td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td><td><b>60.1</b></td><td><b>64.7</b></td><td>50.4</td><td>52.0</td><td>66.5</td><td><b>52.6</b></td><td><b>62.1</b></td>
  </tr>
  <tr>
    <td>GLM-4V</td><td><b>60.9</b></td><td><b>59.6</b></td><td><b>54.4</b></td><td><b>67.7</b></td><td><b>70.5</b></td><td>44.7</td><td>57.5</td>
  </tr>
  <tr>
    <th colspan="8">Humans</th>
  </tr>
  <tr>
    <td>Human_avg</td><td>78.2</td><td>71.5</td><td>65.6</td><td>75.2</td><td>79.8</td><td>74.5</td><td>83.6</td>
  </tr>
  <tr>
    <td>Human_best</td><td><b>81.0</b></td><td><b>76.9</b></td><td><b>66.1</b></td><td><b>78.6</b></td><td><b>81.7</b></td><td><b>78.4</b></td><td><b>85.0</b></td>
  </tr>
</table>

Table 6: Overall results of different MLLMs on different image types. The best-performing model in each category is in-bold, and the second best is underlined. For brevity, Illus. refers to Illustration, Paint. refers to Painting, Single-C. refers to Single-panel Comic, Multi-C. refers to Multi-panel Comic.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="1">Overall</th>
    <th colspan="1">Easy</th>
    <th colspan="1">Medium</th>
    <th colspan="1">Hard</th>
  </tr>
  <tr>
    <th colspan="4">Open-source Models</th>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td><td>34.3</td><td>36.3</td><td>33.5</td><td>30.3</td>
  </tr>
  <tr>
    <td>idefics2-8b</td><td>36.3</td><td>35.4</td><td>39.3</td><td>30.3</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-2.5</td><td>40.4</td><td>43.1</td><td>39.3</td><td>35.3</td>
  </tr>
  <tr>
    <td>CogVLM2-Llama3-Chinese-Chat</td><td>43.4</td><td>46.3</td><td>39.9</td><td>44.3</td>
  </tr>
  <tr>
    <td>MiniCPM-v2.6</td><td>45.0</td><td>47.1</td><td>44.2</td><td>41.0</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-34B</td><td>46.0</td><td>44.9</td><td>47.0</td><td>46.7</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-72B</td><td>48.0</td><td>50.0</td><td>47.0</td><td>45.1</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td><td>49.6</td><td>52.6</td><td>47.9</td><td>45.9</td>
  </tr>
  <tr>
    <td>GLM-4V-9b</td><td>50.3</td><td>52.6</td><td>49.1</td><td>46.7</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td><td>52.9</td><td>57.4</td><td>49.7</td><td>48.4</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td><td>53.1</td><td>57.7</td><td>49.4</td><td>50.0</td>
  </tr>
  <tr>
    <td>InternVL2-40B</td><td>57.9</td><td>62.3</td><td>55.5</td><td>51.6</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td><td><b>64.4</b></td><td><b>68.9</b></td><td><b>63.1</b></td><td><b>54.9</b></td>
  </tr>
  <tr>
    <th colspan="5">Closed-source Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>54.1</td><td>56.0</td><td>54.9</td><td>46.7</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td><td>54.1</td><td>55.1</td><td>52.4</td><td><u>55.7</u></td>
  </tr>
  <tr>
    <td>Qwen-VL-MAX</td><td>56.9</td><td>57.4</td><td>56.7</td><td><u>55.7</u></td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td><td>60.1</td><td>61.1</td><td><u>61.3</u></td><td>54.1</td>
  </tr>
  <tr>
    <td>GLM-4V</td><td><b>60.9</b></td><td><b>62.9</b></td><td><b>59.2</b></td><td><b>59.8</b></td>
  </tr>
  <tr>
    <th colspan="5">Humans</th>
  </tr>
  <tr>
    <td>Human_avg</td><td>78.2</td><td>82.5</td><td>76.1</td><td>70.9</td>
  </tr>
  <tr>
    <td>Human_best</td><td><b>81.0</b></td><td><b>84.0</b></td><td><b>78.9</b></td><td><b>71.8</b></td>
  </tr>
</table>

Table 7: Overall results of different MLLMs on various difficulty levels. The best-performing model in each category is in-bold, and the second best is underlined. The numbers in parentheses indicate the number of samples in each category.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="1">Overall</th>
    <th colspan="8">Meta. Exag. Symb. Contrast VisD. Pers. Anal. Anti.</th>
  </tr>
  <tr>
    <th colspan="8">Open-source Models</th>
  </tr>
  <tr>
    <td>Qwen-VL-Chat</td>
    <td>34.3</td>
    <td>31.8</td>
    <td>38.9</td>
    <td>38.4</td>
    <td>41.0</td>
    <td>37.0</td>
    <td>34.2</td>
    <td>28.6</td>
    <td>30.8</td>
  </tr>
  <tr>
    <td>idefics2-8b</td>
    <td>36.3</td>
    <td>35.2</td>
    <td>32.6</td>
    <td>35.6</td>
    <td>41.9</td>
    <td>30.4</td>
    <td>26.6</td>
    <td>23.8</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>MiniICPM-Llama3-2.5</td>
    <td>40.4</td>
    <td>38.5</td>
    <td>42.4</td>
    <td>40.2</td>
    <td>38.1</td>
    <td>34.8</td>
    <td>44.3</td>
    <td>33.3</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>CogVLM2-Llama3-Chinese-Chat</td>
    <td>43.4</td>
    <td>42.2</td>
    <td>46.5</td>
    <td>42.7</td>
    <td>44.8</td>
    <td>50.0</td>
    <td>44.3</td>
    <td>52.4</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>MiniICPM-v2.6</td>
    <td>45.0</td>
    <td>41.7</td>
    <td>48.6</td>
    <td>43.4</td>
    <td>41.0</td>
    <td>45.7</td>
    <td>45.6</td>
    <td>38.1</td>
    <td>53.9</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-34B</td>
    <td>46.0</td>
    <td>45.1</td>
    <td>47.9</td>
    <td>45.9</td>
    <td>41.0</td>
    <td>45.7</td>
    <td>44.3</td>
    <td>42.9</td>
    <td>30.8</td>
  </tr>
  <tr>
    <td>LLaVA-1.6-72B</td>
    <td>48.0</td>
    <td>46.1</td>
    <td>54.2</td>
    <td>48.0</td>
    <td>49.5</td>
    <td>47.8</td>
    <td>46.8</td>
    <td>47.6</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>49.6</td>
    <td>47.6</td>
    <td>52.1</td>
    <td>48.4</td>
    <td>49.5</td>
    <td><u>56.5</u></td>
    <td>51.9</td>
    <td>47.6</td>
    <td><b>53.9</b></td>
  </tr>
  <tr>
    <td>GLM-4V-9b</td>
    <td>50.3</td>
    <td>48.7</td>
    <td>56.3</td>
    <td>51.3</td>
    <td>52.4</td>
    <td>50.0</td>
    <td>50.6</td>
    <td>57.1</td>
    <td>30.8</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td>
    <td>52.9</td>
    <td>51.5</td>
    <td>59.7</td>
    <td>51.3</td>
    <td>51.4</td>
    <td>52.2</td>
    <td>55.7</td>
    <td>52.4</td>
    <td>46.2</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>53.1</td>
    <td>51.0</td>
    <td>54.9</td>
    <td>55.2</td>
    <td>47.6</td>
    <td>54.4</td>
    <td>57.0</td>
    <td>47.6</td>
    <td>46.2</td>
  </tr>
  <tr>
    <td>InternVL2-40B</td>
    <td>57.9</td>
    <td>55.8</td>
    <td>63.2</td>
    <td>56.6</td>
    <td>55.2</td>
    <td>54.4</td>
    <td><b>69.6</b></td>
    <td><b>71.4</b></td>
    <td>46.2</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td><b>64.4</b></td>
    <td><b>62.5</b></td>
    <td><b>70.1</b></td>
    <td><b>65.8</b></td>
    <td><b>63.8</b></td>
    <td><b>73.9</b></td>
    <td><b>67.1</b></td>
    <td><b>66.7</b></td>
    <td><b>53.9</b></td>
  </tr>
  <tr>
    <th colspan="10">Closed-source Models</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>54.1</td>
    <td>52.6</td>
    <td>54.9</td>
    <td>51.6</td>
    <td>51.4</td>
    <td>60.9</td>
    <td><u>55.7</u></td>
    <td>52.4</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>54.1</td>
    <td>52.1</td>
    <td>54.9</td>
    <td>56.6</td>
    <td>47.6</td>
    <td>50.0</td>
    <td>54.4</td>
    <td>57.1</td>
    <td>38.5</td>
  </tr>
  <tr>
    <td>Qwen-VL-MAX</td>
    <td>56.9</td>
    <td>54.7</td>
    <td>60.4</td>
    <td>58.7</td>
    <td>52.4</td>
    <td>58.7</td>
    <td><u>55.7</u></td>
    <td>57.1</td>
    <td><b>46.2</b></td>
  </tr>
  <tr>
    <td>Gemini-1.5 Pro</td>
    <td><b>60.1</b></td>
    <td><b>59.5</b></td>
    <td><b>64.6</b></td>
    <td><b>60.1</b></td>
    <td><b>61.9</b></td>
    <td>47.8</td>
    <td><u>55.7</u></td>
    <td><b>81.0</b></td>
    <td><b>53.9</b></td>
  </tr>
  <tr>
    <td>GLM-4V</td>
    <td><b>60.9</b></td>
    <td><b>60.2</b></td>
    <td><b>65.3</b></td>
    <td><b>63.4</b></td>
    <td>57.1</td>
    <td><b>65.2</b></td>
    <td><b>60.8</b></td>
    <td><b>66.7</b></td>
    <td>46.2</td>
  </tr>
  <tr>
    <th colspan="10">Humans</th>
  </tr>
  <tr>
    <td>Human_avg</td>
    <td>78.2</td>
    <td>76.0</td>
    <td>82.8</td>
    <td>74.1</td>
    <td>70.4</td>
    <td>73.9</td>
    <td>72.9</td>
    <td>90.0</td>
    <td>52.8</td>
  </tr>
  <tr>
    <td>Human_best</td>
    <td><b>81.0</b></td>
    <td><b>77.0</b></td>
    <td><b>85.2</b></td>
    <td><b>76.5</b></td>
    <td><b>75.7</b></td>
    <td><b>75.6</b></td>
    <td><b>74.7</b></td>
    <td><b>95.0</b></td>
    <td><b>66.7</b></td>
  </tr>
</table>

Table 8: Overall results of different MLLMs and humans on different rhetoric. The best-performing model in each category is <b>in-bold</b>, and the second best is <u>underlined</u>. For brevity, Meta. refers to Metaphor, Exag. refers to Exaggerate, Symb. refers to Symbolism, VisD. refers to Visual Dislocation, Anti. refers to Antithesis, Anal. refers to Analogy, Pers. refers to Personification
F  Additional Details of Results

We do detailed statistics of the model output. The results are shown in Table 9 to 12. Miss is mainly caused by two situations, one is that the model does not give an answer, and the other is the regex is not matched. The Miss rate of most models is controlled below an acceptable ratio. In the CoT setting, some models do not follow instructions well and do not provide the expected letters as answer, which cannot be matched and will be considered a Miss.
<table>
  <tr>
    <th>Mode</th>
    <th>Metric</th>
    <th>InternVL2-40B</th>
    <th>InternVL2-8B</th>
    <th>InternVL2-Llama3-76B</th>
    <th>MiniCPM-Llama3-2.5</th>
    <th>MiniCPM-v2.6</th>
  </tr>
  <tr>
    <td rowspan="3">CoT</td>
    <td>Acc</td>
    <td>57.6</td>
    <td>47.9</td>
    <td>52.6</td>
    <td>35.8</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>8.1</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Domain</td>
    <td>Acc</td>
    <td>57.1</td>
    <td>53.5</td>
    <td>54.1</td>
    <td>41.1</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>5.9</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Emotion</td>
    <td>Acc</td>
    <td>60.0</td>
    <td>56.3</td>
    <td>52.8</td>
    <td>39.0</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>8.4</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">None</td>
    <td>Acc</td>
    <td>57.9</td>
    <td>53.1</td>
    <td>52.9</td>
    <td>40.4</td>
    <td>45.0</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.4</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Rhetoric</td>
    <td>Acc</td>
    <td>57.9</td>
    <td>53.8</td>
    <td>53.5</td>
    <td>34.8</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>10.4</td>
    <td>0.0</td>
  </tr>
</table>

Table 9: Accuracy, Error and Miss rate of different models under different settings.(1/4)

<table>
  <tr>
    <th>Mode</th>
    <th>Metric</th>
    <th>Qwen-VL-Chat</th>
    <th>Qwen2-VL-72B</th>
    <th>Qwen2-VL-7B</th>
    <th>CogVLM2-Llama3-Chinese-Chat</th>
  </tr>
  <tr>
    <td rowspan="3">CoT</td>
    <td>Acc</td>
    <td>34.0</td>
    <td>62.1</td>
    <td>50.0</td>
    <td>43.0</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.3</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Domain</td>
    <td>Acc</td>
    <td>32.1</td>
    <td>66.0</td>
    <td>51.0</td>
    <td>43.5</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Emotion</td>
    <td>Acc</td>
    <td>35.0</td>
    <td>64.3</td>
    <td>50.8</td>
    <td>44.0</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.5</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">None</td>
    <td>Acc</td>
    <td>34.3</td>
    <td>64.4</td>
    <td>49.6</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.5</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.4</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Rhetoric</td>
    <td>Acc</td>
    <td>33.4</td>
    <td>63.0</td>
    <td>49.3</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.3</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
</table>

Table 10: Accuracy, Error and Miss rate of different models under different settings.(2/4)

<table>
  <tr>
    <th>Mode</th>
    <th>Metric</th>
    <th>GLM-4V-9b</th>
    <th>LLaVA-1.6-72B</th>
    <th>LLaVA-1.6-34B</th>
    <th>idefics2-8b</th>
  </tr>
  <tr>
    <td rowspan="3">CoT</td>
    <td>Acc</td>
    <td>49.1</td>
    <td>45.3</td>
    <td>44.5</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Domain</td>
    <td>Acc</td>
    <td>49.9</td>
    <td>47.3</td>
    <td>46.4</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Emotion</td>
    <td>Acc</td>
    <td>51.1</td>
    <td>48.6</td>
    <td>47.1</td>
    <td>38.6</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td rowspan="3">None</td>
    <td>Acc</td>
    <td>50.3</td>
    <td>48.0</td>
    <td>46.0</td>
    <td>36.3</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Rhetoric</td>
    <td>Acc</td>
    <td>49.5</td>
    <td>45.4</td>
    <td>45.4</td>
    <td>37.4</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
</table>

Table 11: Accuracy, Error and Miss rate of different models under different settings.(3/4)
<table>
  <tr>
    <th>Mode</th>
    <th>Metric</th>
    <th>Gemini-1.5 Pro</th>
    <th>GLM-4V</th>
    <th>GPT-4o</th>
    <th>Claude-3-5-Sonnet</th>
    <th>Qwen-VL-MAX</th>
  </tr>
  <tr>
    <td rowspan="3">CoT</td>
    <td>Acc</td>
    <td>54.1</td>
    <td>49.9</td>
    <td>54.9</td>
    <td>51.6</td>
    <td>54.8</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>3.4</td>
    <td>0.0</td>
    <td>1.8</td>
    <td>1.1</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>1.8</td>
    <td>2.4</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Domain</td>
    <td>Acc</td>
    <td>59.0</td>
    <td>60.4</td>
    <td>55.4</td>
    <td>56.4</td>
    <td>59.1</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>1.6</td>
    <td>0.0</td>
    <td>2.5</td>
    <td>1.5</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>1.4</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td rowspan="3">Emotion</td>
    <td>Acc</td>
    <td>58.0</td>
    <td>60.6</td>
    <td>54.9</td>
    <td>53.5</td>
    <td>59.9</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>3.4</td>
    <td>0.0</td>
    <td>2.5</td>
    <td>1.1</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>1.8</td>
    <td>0.0</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">None</td>
    <td>Acc</td>
    <td>60.1</td>
    <td>60.9</td>
    <td>54.1</td>
    <td>54.1</td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>3.3</td>
    <td>1.9</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
    <td>0.9</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td rowspan="3">Rhetoric</td>
    <td>Acc</td>
    <td>55.6</td>
    <td>58.8</td>
    <td>51.9</td>
    <td>54.9</td>
    <td>54.8</td>
  </tr>
  <tr>
    <td>Error</td>
    <td>0.3</td>
    <td>2.1</td>
    <td>0.0</td>
    <td>1.9</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>Miss</td>
    <td>0.9</td>
    <td>0.0</td>
    <td>0.1</td>
    <td>0.0</td>
    <td>0.0</td>
  </tr>
</table>

Table 12: Accuracy, Error and Miss rate of different models under different settings.(4/4)
G   Evaluation metric of Chinese traditional painting

Please see the evaluation metric and standard in Figure 12.
<table>
  <tr>
    <th>Evaluation Metric</th>
    <th>Evaluation Standard</th>
  </tr>
  <tr>
    <td><b>1. Surface-level Information:</b><br>
      • Identification of primary entities within the painting<br>
      • Analysis of color composition and application<br>
      • Recognition of intricate details and their significance
    </td>
    <td>[1 point]:<br>
      The description of the picture is incomplete and does not include any background information. It only mentions the most obvious elements in the picture, lacking recognition and understanding of traditional cultural elements.
    </td>
  </tr>
  <tr>
    <td><b>2. Aesthetic Characteristics:</b><br>
      • Comprehension of compositional techniques<br>
      • Recognition of the painting’s artistic school or movement<br>
      • Interpretation of emotional expression through visual elements
    </td>
    <td>[2 points]:<br>
      The description of the image is relatively complete, but the background information is limited. Some traditional cultural elements can be identified, but their meaning is not explained in depth. There is a lack of insight into the aesthetic characteristics or skills of the image.
    </td>
  </tr>
  <tr>
    <td><b>3. Brush and Ink Skills:</b><br>
      • Identification of specific painting skills<br>
      • Analysis of expressive methods employed by the artist
    </td>
    <td>[3 points]:<br>
      The description of the image is comprehensive and includes some background information. The traditional cultural elements can be accurately identified and briefly explained. There is a preliminary understanding of the aesthetic characteristics or skills of the picture, but there is a lack of in-depth analysis. Some historical or cultural background is mentioned, but it is not fully expanded.
    </td>
  </tr>
  <tr>
    <td><b>4. Culture and History:</b><br>
      • Understanding of the painting’s historical background<br>
      • Interpretation of the artist’s intended message or philosophy<br>
      • Placement of the artwork within its appropriate historical period
    </td>
    <td>[4 points]:<br>
      The description of the image is detailed and contains rich background information. It is able to explore the significance of traditional cultural elements in depth. It provides a relatively in-depth analysis of the aesthetic characteristics and skills of the image. It provides relevant historical and cultural background and attempts to explore the implication of the image, but the implication is misunderstood.
    </td>
  </tr>
  <tr>
    <td><b>5. Deep Implications:</b><br>
      • Recognition of traditional cultural elements embedded in the painting<br>
      • Interpretation of philosophical concepts, humanistic spirit, or social values conveyed<br>
      • Demonstration of a deep understanding of Chinese traditional culture as reflected in the artwork
    </td>
    <td>[5 points]:<br>
      The description of the image is comprehensive and accurate, and the background information is in-depth and comprehensive. It can not only accurately interpret traditional cultural elements, but also reveal their deep meaning. It has unique insights into the aesthetic characteristics and skills of the image. It deeply expounds on the relevant historical and cultural background, and can see through the philosophical thoughts, humanistic spirit or social values contained in the image, showing a deep understanding of Chinese traditional culture.
    </td>
  </tr>
</table>

Figure 12: Evaluation metric and evaluation standard of Chinese traditional painting.
H  Other Errors

Lack of Cultural Background Knowledge (16%):
    CII-Bench requires a model’s deep understanding of Chinese traditional culture. Lacking knowledge of traditional symbols, historical figures, and classical allusions, GPT-4o struggles with interpreting deeper implications within images. Despite reasonable Chinese language handling, the model’s cultural deficiency affects its reasoning and performance.

Superficial Reasoning (12%):
    Understanding extended meanings within images is crucial. However, GPT-4o often only focus on surface-level elements, neglecting the deep implications and deeper cultural connotations behind them. This superficial reasoning hinders the model from fully grasping profound messages that the artist or designer intends to convey.

Misunderstanding of Visual Information (11%):
    Accurate identification of visual information is vital. We find that GPT-4o sometimes misidentifies visual elements within images, particularly when dealing with abstract images. The abstract nature of such images often stems from the inclusion of exaggerated imaginative elements, sometimes even defying physical laws. Therefore, correctly identifying these abstract elements requires the model to have a deep understanding of the essence of objects, a capability that current models clearly do not yet possess.
1 Discussion

1.1 Interpretability Analysis of Chinese Image Implications

The essence of Chinese image implications is deeply rooted in deep cultural heritage and complex contextual associations, which enables them to convey profound messages through nuanced expressions. For example, in traditional Chinese art forms such as landscape and New Year paintings, the imagery transcends mere depiction of nature or daily occurrences. Instead, it embodies emotions, philosophical insights, and societal norms through metaphorical and highly symbolic expressions. These symbols, like the pine tree, plum blossom, and crane, are not superficial meaning but are steeped in centuries of cultural tradition, representing resilience, purity, and longevity.

However, deciphering these complex messages can be challenging, particularly for those unfamiliar with the cultural and historical narratives tied to these symbols. This contrasts with English image implications, which often convey messages through more straightforward and explicit symbolism. As a result, the interpretability of Chinese image implications depends to some extent on reconstructing and resonating with the cultural context, which is what makes them unique: their meaning is not only visual but also culturally resonant, bridging across time and space.

Moreover, the interpretability of Chinese image implications has new changed in the modern era. Globalization and the surge of internet culture have intertwined foreign elements with traditional Chinese culture, birthing new symbols and implications. This intersection introduces additional layers of meaning, complicating the interpretation of traditional symbols.

1.2 Why Choose Chinese Traditional Paintings to Evaluate Chinese Traditional Culture?

The imagery associated with Chinese traditional culture often embodies complex implications, encompassing customs, historical anecdotes, and legendary tales, making direct evaluation particularly challenging. Chinese traditional painting, intrinsically intertwined with Chinese traditional culture, offers a viable proxy for this assessment. The unique value of Chinese traditional painting lies in its embodiment of Chinese cultural connotations, aesthetic implications, and distinctive artistic expression. The core philosophical concepts of Confucianism, Taoism, and Buddhism, along with their humanistic essence, have consistently permeated the entire trajectory of Chinese painting history. Consequently, we have chosen to evaluate MLLMs’ comprehension of Chinese traditional culture through an in-depth analysis of their understanding of Chinese traditional paintings.
J   Case Study

The appendix is our sample analysis of GPT-4o, including an analysis of six error examples.

List of Case Study Figures

1   Error 1: Textual Information Neglect    . .    29
2   Error 2: Visual Information Neglect    . .    30
3   Error 3: Over-Inference    . . . . . . . .    31
4   Error 4: Lack of Cultural Background Knowledge    . . . . . . . . . . . . . .    32
5   Error 5: Superficial Reasoning    . . . .    33
6   Error 6: Misunderstanding of Visual Information    . . . . . . . . . . . . . .    34
Textual Information Neglect

Question: 这张图片有什么隐喻？<image 1>
Option:
(A) 禁止饲养宠物
(B) 禁止拍摄鸟类
(C) 禁止飞行
(D) 禁止发出大的声音
(E) 禁止饲养珍惜动物
(F) 禁止传播色情内容

![禁止色禽的图片](page_324_370_800_600.png)

GPT-4o: 让我们逐一分析每个选项：

(A) 禁止饲养宠物 - 该图片显示一个被禁止符号（红色圆圈和斜线）覆盖在一只鸟的图像上，并且无标识任何宠物特定的信息。
(B) 禁止拍摄鸟类 - 图片并没有显示任何与拍摄相关的符号或装置。
(C) 禁止飞行 - 虽然鸟可以飞，但图片没有特别强调飞行的元素。
(D) 禁止发出大的声音 - 图片没有显示与声音相关的符号。
(E) 禁止饲养珍惜动物 - 该选项与禁止饲养宠物类似，但没有特别标识珍惜动物的信息。
(F) 禁止传播色情内容 - 这完全与图像无关。

综合来看，这张图片最直接的解释是禁止养鸟。

答案：A

Error Reason: 图片中展示了一只鸟，一个禁止符号，以及非常显眼的四个大字“禁止色禽”。这里的色禽谐音了色情，也指图片中颜色花哨的鸟，因此图片想要传达的是禁止色情。而GPT-4o却因为忽略了图片中“禁止色禽”，导致其对图片的深意理解错误。

Ground Truth: (F) 禁止传播色情内容

Figure G1: A sample error case of Information Neglect.
Back to List of figures
Visual Information Neglect

Question: 这张图片有什么隐喻？<image 1>
Option:
(A) 这张图片反映了家庭内部女性受到了男性的欺凌，呼吁社会关注家庭暴力和男女平等。
(B) 红色领带可以被视为外界对个人身份的期望或定义，而女性手中散落的文件则象征着她内在的真实自我。
(C) 这张图片批判职场中女性受到权力的不平等的待遇。
(D) 这张图片展示了女性与巨大红色领带的互动，象征着个人在面对社会传统习俗时的挣扎。红色领带代表传统的束缚，而女性散落的文件则表现出她试图突破这些束缚的努力。
(E) 这张图片批判了职场中男性表现出的虚伪，因为蛇往往象征着虚伪。
(F) 在这张图片中，一个身穿黑白衣服的女性站在巨大的红色领带前，领带如同一条巨蛇，暗示着人类与未知力量的斗争。

![A woman in black and white clothing standing in front of a large red ribbon, which resembles a snake.](page_246_370_563_246.png)

GPT-4o: 为逐一分析每个选项：
(A) 这张图片反映了家庭内部女性受到男性的欺凌，呼吁社会关注家庭暴力和男女平等。
- 这个选项略显偏颇，因为图片似乎并没有直接传达家庭暴力或家庭内部性别不平等的具体细节。
(B) 红色领带可以被视为外界对个人身份的期望或定义，而女性手中散落的文件则象征着她内在的真实自我。
- 确实可能与图片的一部分思想契合，即红色领带作为一种外界期望的象征，而散落的文件可能暗示女性内心的挣扎和真实自我。
(C) 这张图片批判职场中女性受到权力的不平等的待遇。
- 虽然这确实可能是一部分涵义，但图片中的元素（红色领带、散落的文件）更多地暗示社会期望和传统束缚的紧张关系。
(D) 这张图片展示了女性与巨大红色领带的互动，象征着个人在面对社会传统习俗时的挣扎。红色领带代表传统的束缚，而女性散落的文件则表现出她试图突破这些束缚的努力。
- 这个选项很切题，领带代表着传统和期望，而散落的文件体现了突破束缚的努力，符合图片的整体氛围和隐喻。
(E) 这张图片批判了职场中男性表现出的虚伪，因为蛇往往象征着虚伪。
- 把领带比作蛇或虚伪有些勉强，图片中的领带更多的是象征某种外界期望或传统束缚。
(F) 在这张图片中，一个身穿黑白衣服的女性站在巨大的红色领带前，领带如同一条巨蛇，暗示着人类与未知力量的斗争。
- 提到“巨蛇”和“未知力量”有些离题，更多的是在描绘传统束缚的隐喻。
答案：D

Error Reason: 图片中红色领带呈现出了蛇的形象，暗示了职场中女性受到来自男性的权利威压，批判职场中女性受到权力的不平等的待遇。GPT-4o没有认出领带的形状，认为领带代表传统的束缚，这显然不合理，散落的文件也并非代表着努力。总之，GPT-4o由于忽略了视觉信息，导致了对图片的主题理解错误。

Ground Truth: (C) 这张图片批判职场中女性受到权力的不平等的待遇。
Over-Inference

Question: 这张图片有什么隐喻？<image 1>
Option:
(A) 这张图片表达了对纯真和好奇心的赞美。
(B) 这张图片表达了对艺术创作的热爱。
(C) 该作品反映了环境保护的主题，绿色的地毯代表自然，提醒我们要保护地球。
(D) 这张图片强调了环保的重要性。
(E) 图片中的情节暗示了一种教育方法，即通过体验学习来教授孩子纪律和规矩的重要性。
(F) 图画强调了年龄和经验之间的代沟，以及两代人之间沟通的挑战。

GPT-4o:让我们逐一分析每个选项：
(A) 这张图片表达了对纯真和好奇心的赞美。
这与图片内容没有直接的联系。图片演示了某些行为，但并没有特别强调纯真和好奇心。
(B) 这张图片表达了对艺术创作的热爱。
虽然图片中的小孩在绘画，但整体情节似乎更多围绕反应及处理孩子的行为，这也不太符合。
(C) 该作品反映了环境保护的主题，绿色的地毯代表自然，提醒我们要保护地球。
图片中的绿色地毯并未显著成为环保的象征，情节更聚焦在对孩子行为的反应上。
(D) 这张图片强调了环保的重要性。
同样，图片重点不在于环保，地毯色彩也非显著性的环保标志。
(E) 图片中的情节暗示了一种教育方法，即通过体验学习来教授孩子纪律和规矩的重要性。
通过图片中老人对孩子绘画的接收、模仿及重新影响这一过程，确实可以看出一种通过体验进行教育的意图。
(F) 图画强调了年龄和经验之间的代沟，以及两代人之间沟通的挑战。
这图片没有过多出现沟通和代沟的明显象征，而是更专注于行为与反应教育。
答案：E

Error Reason: 图片是经典漫画《父与子》的一小节，这幅漫画主要描写了父亲与儿子之间的有趣互动，充满乐趣。图片中儿子打翻了墨水弄脏了地毯，却巧妙地利用墨水作画掩盖瑕疵，甚至最终父亲也参与作画，乐在其中。作者可能想要表达孩子的纯真、好奇，也可能想要表达错误也能通过努力不断弥补。GPT-4o选择的选项E，提到了纪律和规矩，显然不符合主题，推理过度。

Ground Truth: (A) 这张图片表达了对纯真和好奇心的赞美。

Figure G3: A sample error case of Over-Inference.
Back to List of figures
Lack of Cultural Background Knowledge

Question: 这张图片有什么隐喻？<image 1>
Option:
(A) 老人可能代表着即将放手的母亲，年轻的女性则是即将成为家庭一部分的新成员。中间的男性正处在人生的重要转折点上，需要学会独立。
(B) 这张图片描绘了一个普通的购物场景。
(C) 这张图片展示了家庭外出旅游的场景。
(D) 老人手中的篮子象征着她对年轻一代的教导和支持；年轻女性的时尚装扮则代表了新思想的引入。中间的男性则体现了作为桥梁的作用，连接着过去与未来。
(E) 这张图片形象展示了对婆媳之间关系的调侃，表面和平而实际很分裂。
(F) 这张图片生动地展现了婆媳之间的和谐共处与相互支持的主题。画面中的三人手牵手，象征着家庭成员间的团结与和睦。

<image 1>

GPT-4o:让我们逐一思考每个选项。
(A) 老人可能代表着即将放手的母亲，年轻的女性则是即将成为家庭一部分的新成员。中间的男性正处在人生的重要转折点上，需要学会独立。
- 梳理选项中的描述，老人可能代表一代人，年轻女性代表新一代，男性则是连接两代人的桥梁，中间的男性正处于转折点，这与图片中的人物关系和手牵手的方式似乎匹配。
(B) 这张图片描绘了一个普通的购物场景。
- 从图片来看，没有明显的购物元素。
(C) 这张图片展示了家庭外出旅游的场景。
- 图片中没有明示旅游的细节或装饰，没有明显的展现旅游场景。
(D) 老人手中的篮子象征着她对年轻一代的教导和支持；年轻女性的时尚装扮则代表了新思想的引入。中间的男性则体现了作为桥梁的作用，连接着过去与未来。
- 分享了明确的分析和隐喻，老人手中的篮子象征支持，年轻女性代表新思想，桥梁则表示连接。这与图像所传达的有较好的一致性。
(E) 这张图片形象展示了对婆媳之间关系的调侃，表面和平而实际很分裂。
- 图片没有表现出明显的分裂和矛盾。
(F) 这张图片生动地展现了婆媳之间的和谐共处与相互支持的主题。画面中的三人手牵手，象征着家庭成员间的团结与和睦。
- 表述与图像中的人物关系较为吻合，表达了团结与和睦。
根据以上分析，更合适的选项为 (D)，较为全面地解释了图片中的隐喻。
答案：D

Error Reason: 图片中展示了3个人物手拉手，居中的人是丈夫，右边是妻子，左边则是婆婆。婆婆呈现红色，妻子呈现绿色，而丈夫则是红绿相间。婆婆和妻子两种截然不同的颜色，暗示了她们之间的冲突，讽刺了当代社会中婆媳之间的冲突。丈夫被冲突所波及，左右为难。GPT-4o缺乏对中国文化的理解，因此无法选出正确的答案。

Ground Truth: (E) 这张图片形象展示了对婆媳之间关系的调侃，表面和平而实际很分裂。

Figure G4: A sample error case of Lack of Cultural Background Knowledge.
Back to List of figures
Superficial Reasoning

Question: 这张图片有什么隐喻？<image 1>
Option:
(A) 两者共同拉起横幅，意味着传统与现代技术在对抗浪费方面的合作。人群象征着团结的力量。
(B) 在这张图片中，直升机和起重机代表着不同形式的力量汇聚在一起，共同为反对浪费的目标努力。人群展示了大众对于这一事业的热情和支持。
(C) 图片反映了现代社会信息传播的困境：信息爆炸的同时，有效信息的获取和理解却变得越来越困难。
(D) 直升机代表着政府的支持，而起重机则象征着社会基层的努力。横幅上的“反铺张浪费万人大签名”表明人们希望通过大型活动来提高对节约资源意识的关注。
(E) 这张图片对讽刺了目前因为反对铺张浪费现象而产生的不合理行为。
(F) 图片赞扬了广大人民群众为了共同的目标而努力的合作精神。

![A group of people holding a large banner with the words "Anti-waste" in Chinese](page_246_370_388_246.png)

<image 1>

GPT-4o:从这张图片中，可以看到直升机和起重机共同举起了一个标有“反铺张浪费万人大签名”的横幅，同时下面有大量的人群集结。各个选项的分析如下：
(A) 两者共同拉起横幅，意味着传统与现代技术在对抗浪费方面的合作。人群象征着团结的力量。
- 图片中直升机和起重机确实共同举起了横幅，人群也显示了团结的力量。这种描述贴近图片内容。
(B) 在这张图片中，直升机和起重机代表着不同形式的力量汇聚在一起，共同为反对浪费的目标努力。人群展示了大众对于这一事业的热情和支持。
- 这种说法也与图片内容相符，但没有明确现代与传统技术分别指代什么形式的力量。
(C) 图片反映了现代社会信息传播的困境：信息爆炸的同时，有效信息的获取和理解却变得越来越困难。
- 此选项与图片所展示的内容不符，主要不涉及信息传播的困境。
(D) 直升机代表着政府的支持，而起重机则象征着社会基层的努力。横幅上的“反铺张浪费万人大签名”表明人们希望通过大型活动来提高对节约资源意识的关注。
- 图片中的直升机和起重机虽然显示了支持，但没有明显证据表明直升机代表政府、起重机代表基层。
(E) 这张图片讽刺了目前因为反对铺张浪费现象而产生的不合理行为。
- 从图片来看，没有表现出对不合理行为的讽刺倾向，主要是为了展示反对铺张浪费的行动。
(F) 图片赞扬了广大人民群众为了共同的目标而努力的合作精神。
- 人群众多，似乎确实展示了为了共同目标而合作的精神。
答案分析：综合来看，选项 (A) 和 (B) 都比较贴合图片内容，但 (A) 更为具体地说明了技术与团结的隐喻。因此选 (A)。
答案：A

Error Reason: 图片夸张的展示了反对浪费的万人签名活动，密密麻麻的人群、拉起签名纸的直升机和起重机，的确体现了人民的团结精神，但是这并非作者想要传达的思想。如此浮夸、耗费资源和人力的签名活动，正是一种浪费的体现，可笑的是活动本身却是反对浪费的。GPT-4o只意识到了最肤浅的图片含义，没有充分结合图片信息，挖掘出深意。

Ground Truth: (E) 这张图片对讽刺了目前因为反对铺张浪费现象而产生的不合理行为。

Figure G5: A sample error case of Superficial Reasoning.
Back to List of figures
Misunderstanding of Visual Information

Question: 这张图片有什么隐喻？<image 1>
Option:
(A) 爱情让人们失去了理性判断。
(B) 爱让我们变得更加聪明和睿智。
(C) 爱是一种需要持续投入的投资。
(D) 爱情会导致人们的孤立和疏远。
(E) 过于充盈的爱，会让我们失去独处的思考。
(F) 爱情会让人更加关注自己的内心世界。

![A painting showing two people kissing, with one person's head having a control panel and a small figure inside.](page_324_370_803_324.png)

GPT-4o:这幅图片展示了一对年轻人亲吻的场景，其中一个人的头脑中有一个操作杆的人，这个细节表明隐喻的含义。逐个选项分析：
(A) 爱情让人们失去了理性判断。
- 操作杆的人可能暗示爱情在操控一个人的理智，显示出理性被放在一个小角落来观察，可以一定程度上支持这个观点。
(B) 爱让我们变得更加聪明和睿智。
- 从图片来看，这种解释不太符合，因为图中的操控体现了某种控制，而不是增强智慧。
(C) 爱是一种需要持续投入的投资。
- 没有直接暗示投入或投资的行为，所以这个选项不太符合。
(D) 爱情会导致人们的孤立和疏远。
- 图片中两个人亲密接触，并没有表现出孤立和疏远的感觉。
(E) 过于充盈的爱，会让我们失去独处的思考。
- 这也不符合图片中的暗示，图中没有表现出爱太多导致失去独处思考的情境。
(F) 爱情会让人更加关注自己的内心世界。
- 操作杆的人是在脑中，可能象征着对内心世界的观察和控制，但更接近的是对于理性判断的控制，而不是单纯的内心世界关注。
综上所述：

答案：(A)

Error Reason: 图片展示了一对男女接吻，奇怪的是男子的后脑有一个小阳台，阳台上有个小人正在抽烟。抽烟往往暗示着思考，图片可能想要暗示充斥着的爱让我们没有独立思考的时间与空间。GPT-4o认为图中男子后脑有一个操纵杆，这显然是错误的。正是由于错误的识别为操纵杆，才导致它的分析全部错误。

Ground Truth: (E) 过于充盈的爱，会让我们失去独处的思考。

Figure G6: A sample error case of Misunderstanding of Visual Information.
Back to List of figures