Express 💬 What You See 👀: Can Multimodal LLMs Decode Visual Ciphers with Intuitive Semiosis Comprehension?
Authors: Jiayi Kuang, Yinghui Li, Chen Wang, Haohao Luo, Ying Shen, Wenhao Jiang
Accepted at: acl25

Express 😊 What You See 🖼️: Can Multimodal LLMs Decode Visual Ciphers with Intuitive Semiosis Comprehension?

Jiayi Kuang1*, Yinghui Li2*, Chen Wang3, Haohao Luo1, Ying Shen1, 5†, Wenhao Jiang4

1Sun Yat-sen University, 2Tsinghua University, 3University of Pennsylvania,
4Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ),
5Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology

*These authors contributed equally. †Corresponding author: sheny76@mail.sysu.edu.cn

Abstract

Bridging the gap between visual and language remains a pivotal challenge for the multimodal community. Traditional VQA benchmarks encounter a modality gap and over-reliance on language priors, whereas human cognition excels at intuitive semiosis, associating abstract visual symbols to linguistic semantics. Inspired by this neurocognitive mechanism, we focus on emojis, the visual cipher conveying abstract textual semantics. Specifically, we propose a novel task of generating abstract linguistics from emoji sequence images, where such reasoning underpins critical applications in cryptography, thus challenging MLLMs’ reasoning of decoding complex semantics of visual ciphers. We introduce eWe-bench (Express What you SeE), assessing MLLMs’ capability of intuitive semiosis like humans. Our data construction framework ensures high visual sensitivity and data quality, which can be extended to future data enhancement. Evaluation results on advanced MLLMs highlight critical deficiencies in visual intuitive symbolic reasoning. We believe our interesting insights for advancing visual semiosis in MLLMs will pave the way for cryptographic analysis and high-level intuitive cognition intelligence of MLLMs 1.

1 Introduction

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in recent years (Liu et al., 2022; Dong et al., 2023; Xu et al., 2024; Yin et al., 2023; Wu et al., 2023; Cui et al., 2024; Kuang et al., 2024; Li et al., 2025a), particularly in their ability to integrate visual and linguistic information for more natural human-computer interaction (Li et al., 2022c; Zhang et al., 2024a; Luo et al., 2024; Li et al., 2024e). Despite these advances, a central challenge remains: how can MLLMs truly bridge the gap between vision and language to emulate human-like perception? (Koh et al., 2024; Peng et al., 2023; Wang et al., 2024). Traditional Visual Question Answering (VQA) based MLLM benchmarks have driven progress by evaluating from various dimensions (Li et al., 2024b, 2023a) and different domains (Lu et al., 2024b; Yue et al., 2024; Zhang et al., 2024b). However, their inherent designs of decoupling images and questions have fragmented visual-language representations, which introduces critical limitations in various language scenarios: (1) over-reliance on language priors instead of visual understanding, and (2) question-guided reasoning that masks spontaneous visual intuition (Peng et al., 2023; Wang et al., 2024).

Human cognition presents a different paradigm. Neuroscientific studies reveal that our brains reflect intuitive semiosis capability, which spontaneously maps abstract visual symbols to semantic representations through rapid thalamocortical interactions (Schulze Buschoff et al., 2025; Islam and Bouwman, 2016; Hobson and Pace-Schott, 2002), with symbol recognition occurring 200-300 milliseconds faster than object categorization. This visual symbol intuitive semiosis enables instantaneous understanding of abstract symbols that convey complex meanings through shared cultural codes, including traffic signs, mathematical notations, and emojis (Du et al., 2023). Such visual symbol reasoning underpins critical applications in cryptography and information analysis (Ate-niese et al., 1996; Moulin and O’Sullivan, 2003), where successful decoding requires direct symbol-to-semantic mapping without explanatory prompts.

Motivated by this mechanism, we propose a transformative benchmark to evaluate MLLMs’ visual intuitive semiosis comprehension in multilingual scenarios. Departing from the VQA paradigm that uses textual questions as cognitive scaffolds, our approach employs emojis as visual ciphers, challenging models to decode abstract visual sym-

1All our data are available in eWe-bench.
(a) Previous VQA-based benchmark
(b) Our Intuitive Semiosis Benchmark
seperate vision and language
visual symbol cipher
Intuitive thinking like human
decode with Intuitive Semiosis Comprehension:
that associate visual ciphers with abstract linguistic semantics
(c) Challenge and interesting discoveries
Harmonized Word Reasoning
What is the Chinese idiom the emojis represented in this image?
Ground Truth: 虎落平阳
MLLM: 虎虎生风
What is the English word the emojis represented in this image?
Ground Truth: Apple.
MLLM: Apple.
What is the English idiom the emojis represented in this image?
Ground Truth: Well, I hope so.
MLLM: Whale, I hope so.
Many-to-one or one-to-many mapping
What is the Chinese idiom the emojis represented in this image?
Ground Truth: 中河失舟，一臺千金
MLLM: 依依虎威
What is the English word the emojis represented in this image?
Ground Truth: grape.
MLLM: Kong.
What is the English idiom the emojis represented in this image?
Ground Truth: Blow off steam.
MLLM: blow hot and cold.

Figure 1: This figure illustrates the advantages and challenges of eWe-bench, with examples of ground truth, and MLLMs response. VQA-based benchmark example is from Li et al. (2023a).

bols into linguistic semantics while minimizing language prior interference. As shown in Figure 1, this task demands three-layer reasoning: (1) visual pattern recognition of symbolic elements, (2) combinatorial creativity to reason homophonic mappings and many-to-one correspondences, and (3) cultural interpretation of contextual relationships.

To realize this vision, we introduce eWe-bench, a multilingual benchmark of emoji-idiom pairs spanning Chinese and English cultural contexts. Idioms are selected from historical allusions, mythology, and conventional expressions to ensure broad linguistic and cultural coverage. We propose a data engineering framework comprising (1) retrieval of real-world emoji-text pairs, (2) text-to-emoji generation to mitigate bias, and (3) rigorous machine filtering with human validation to ensure visual sensitivity and ethical compliance.

We further design a fine-grained evaluation strategy combining automatic and human assessments. Results reveal that state-of-the-art MLLMs achieve only 3.3% compared to 67% human performance, exposing critical deficiencies in intuitive semiosis reasoning, particularly in homophonic relationships and multi-to-one mappings. A case study identifies key failures and offers insights for future improvements. By aligning MLLM evaluation with the neurocognitive of visual semiosis, we open new pathways toward human-like multimodal intelligence. Our contributions are as follows:

1. We propose a novel evaluation paradigm assessing visual intuitive semiosis of cryptographical decoding, thereby evaluating the human-like, unified visual-language comprehension of MLLMs.

2. We design a data engineering framework and construct a high-quality benchmark, eWe-bench, which minimizes the modality gap and mitigates the over-reliance on language priors.

3. Empirical evaluation reveals MLLMs’ limitations in intuitive semiosis, with actionable insights applicable to cryptographical analysis and human-like intelligence.

2 Related Work

Language Model Based Cryptic Understanding  Emoji can be represented by UTF-8 (Abel, 2019), and many treat emoji as text and encode them as vectors (Eisner et al., 2016). Leveraging the emoji Unicode library, numerous studies have explored emoji-text translation, including translation text into emoji (Monti et al., 2016; Leonard, 2022; Klein et al., 2024), and bidirectional translation(Danesi, 2022). Beyond this, emoji-based sentiment analysis has become a significant area
Figure 2: This figure illustrates the pipeline of eWe-bench, which is divided into raw data collection, data filtering, and evaluation.

of emoji research (Gibson et al., 2018; Chen et al., 2019, 2018; Liu et al., 2021). However, to the best of our knowledge, our eWe-bench is the first to apply the visual intuitive semiosis of emojis.

MLLMs Benchmark  Earlier unified MLLMs benchmarks collect a substantial number of images and generate corresponding QA pairs to evaluate MLLMs (Fu et al., 2023a; Li et al., 2024c; Yu et al., 2024), with a focus on uniformity and objectivity, as seen in SEEDBENCH (Li et al., 2024b) and SEEDBENCH-2 (Li et al., 2023a). Recent benchmarks have started to assess different capabilities from different dimensions, including visual comprehension (Li et al., 2022b; Fu et al., 2023b; Li et al., 2024a; Tong et al., 2024; Cai et al., 2023), reasoning ability (Li et al., 2024f; Zhang et al., 2024c; Roberts et al., 2023; Huang et al., 2024), in-context learning capability (Shukor et al., 2023; Liu et al., 2023; Li et al., 2024d), hallucination challenge (Cui et al., 2023; Liu et al., 2023; Xu et al., 2025; Li et al., 2025b), and multiple domains (math, physics, music, medical, etc.)(Li et al., 2022a; Ma et al., 2022; Ye et al., 2023; Huang et al., 2023; Zhang et al., 2023; Lu et al., 2024b; Li et al., 2023c,d; Yue et al., 2024; Zhang et al., 2024b; Ye et al., 2024; Li et al., 2025c). However, most benchmarks are based on the VQA annotations and natural scenario image, rather than directly associating an abstract image with its linguistics.

3 The eWe-bench Benchmark

3.1 Task Definition
Our eWe-bench propose a benchmark that images correspond to texts with specific formats and semantics, representing a Chinese idiom word, an English word, or an English idiom sentence. Given an image of a sequence of emoji \( I_i^{\text{emoji}} = \{\text{emoji}_1, \text{emoji}_2, \cdots, \text{emoji}_n\} \), eWe-bench task aims to translate emojis in images to corresponding idiom text by model \( F \):

\[
\text{Text} = F(I_i^{\text{emoji}})
\]

It requires not only understanding the direct corresponding text of a single emoji, but also inferring complex linguistic meaning based on intuitive semiosis capability, with detailed challenges provided in Section 3.3.

3.2 Benchmark Construction
Our data construction process involves retrieving emoji-idiom pairs from the Internet, ensuring diverse cultural representation and real-world usage. To mitigate bias, we supplement underrepresented idioms using an idiom-to-emoji generation method. A rigorous filtering pipeline, combining automatic and human review, guarantees data quality and reliability. All online data are used with proper authorization.

Raw Data Collection  As shown in the Figure 2, we collect raw data through two automatic generation methods: Retrieve from the Internet. There are a large number of expressions of idioms based on emojis on the internet, so we retrieve the relevant web pages such as lovelyemoji 2 to get the original emoji images and the corresponding answers. Generate Emoji Based on Text. The quality of internet retrieval is not very high, due to 1) repetitive emoji-text pairs and recurring emoji-character mapping and 2) a relatively low number of English idioms. We utilize an existing emoji translator capable of bidirectional translation between text and

2https://www.lovelyemoji.com/emojiicaichengyu/
Table 1: The statistics and examples of eWe-bench, where img and txt are short names of image and text.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th colspan="3">Raw data</th>
    <th colspan="2">Auto-filter</th>
    <th colspan="2">Human-filter</th>
    <th colspan="2">Example</th>
  </tr>
  <tr>
    <th>Img-txt</th>
    <th>Txt</th>
    <th>Img</th>
    <th>Img-txt</th>
    <th>Img</th>
    <th>Img-txt</th>
    <th>Img</th>
    <th>Txt</th>
  </tr>
  <tr>
    <td>Four-character idiom</td>
    <td>2,338</td>
    <td>2,379</td>
    <td>2,362</td>
    <td>2,252</td>
    <td></td>
    <td>1,876</td>
    <td></td>
    <td>闷闷不乐</td>
    <td></td>
  </tr>
  <tr>
    <td>Multi-character idiom</td>
    <td>622</td>
    <td>641</td>
    <td>629</td>
    <td>576</td>
    <td></td>
    <td>334</td>
    <td></td>
    <td>不问三七二十一</td>
    <td></td>
  </tr>
  <tr>
    <td>English Word</td>
    <td>1,261</td>
    <td>1,289</td>
    <td>1,263</td>
    <td>1,076</td>
    <td></td>
    <td>842</td>
    <td></td>
    <td>starfish</td>
    <td></td>
  </tr>
  <tr>
    <td>English Idiom</td>
    <td>1,237</td>
    <td>1,254</td>
    <td>1,244</td>
    <td>1,182</td>
    <td></td>
    <td>783</td>
    <td></td>
    <td>Health is wealth.</td>
    <td></td>
  </tr>
</table>

emojis, such as emojiJiall\footnote{https://www.emojiJiall.com/} and additionally generate the corresponding emoji sequences, which greatly mitigates the data bias.

**Automatic Filtering** We employ machines and LLMs to ensure data quality. First, default values are deleted by removing incomplete emoji-idiom pairs, ensuring each emoji corresponds to a unique idiom. Duplicate pairs with identical emoji sequences are removed while retaining those with distinct emoji representations to enhance diversity. Image quality is assessed using GPT-4o to eliminate blur. Additionally, ethics checks are performed by GPT-4o, which removes emoji-idiom pairs involving violence, discrimination, and abuse. More details about automatic filtering can be found in the Appendix A.1.

**Human Filtering** Human experts further refine the data that complement the automatic filtering. They assess emoji-text relevance and delete the ones that have low relevance, eliminating those with overly complex reasoning tasks. Idiom standardization is conducted to ensure alignment with human usage habits, including proper linguistic format, semantic meaning, and cultural significance. To mitigate data bias, repetitive harmonic word mappings are either replaced with alternatives or removed. Finally, safety and ethics checks are conducted to confirm the absence of inappropriate content, such as violence, sexism, stereotypes or discrimination, ensuring the dataset aligns with ethical standards. Additional information and detailed guidelines to eliminate subjectivity for human experts are provided in the Appendix A.2.

3.3 Data Statistics and Challenges

**Statistics** We give the statistics of eWe-bench in Table 1. After filtering, we collect high-quality data of 1,876, 334, 842, 783 emoji-texts of Chinese four-character idioms, multi-character idioms, English word and English idiom respectively (Additional statistics like word cloud in Appendix B).

**Linguistic phenomena and challenges** In our eWe-bench, we observe several interesting linguistic phenomena, which raise great challenges and encourage the exploration of the unified vision-language, with additional details in Appendix C.

*Word Split.* In English word, it is common for multiple emoji to represent one word. The word “Panda” can be split into “Pan-” and “-da,” where “Pan-” corresponds to 🐼. Beyond understanding the meaning of individual emojis, the MLLMs must also remove unnecessary letters and combine the parts to infer a completely new word, raising challenge of multi-to-one mapping reasoning.

*Harmonic Characters.* Since it is sometimes difficult to find directly related emoji to represent, harmonic characters with similar pronunciations are often chosen to replace them. For example, “To be loaded”, “To” harmonizes with “Two” ２, and “be” harmonizes with “bee” 蜂. In the Chinese idiom “难舍难离”, “舍” (pronounced as “she”) harmonizes with “蛇 (snake)” (pronounced as “she”) of emoji 蛇, “离” harmonizes with “梨 (pear)” (pronounced as “li”) of emoji 梨.

*Abstract visual Cipher Understanding.* In addition to referring to the direct meanings of the emoji, it is often necessary to deeply infer the semantics of the emoji. In 🌟❤️🍏🍌“同心协力 (pull together with the same goal) ”, 🍏 is an arm, but it does not mean “arm” in idioms. Instead, it is a very strong arm, which corresponds to “力 (power)”.

**Discussions of Emoji-Character Ambiguity and Ground Truth Validity** Mapping emojis to homophonic characters increases task complexity and raises ground truth validity concerns. Our analysis reveals the mapping: (1) extracting the emoji’s inherent meaning (e.g., 鲸 (whale)), (2) identifying potential homophones 精 (excellent), 惊 (surprise), and (3) determining the correct character based on contextual and idiomatic cues (e.g., distinguishing 惊 in “大失色” (greatly surprised) from 精 in “才绝艳”(exceptionally talented)). While an emoji may correspond to multiple charac-
Table 2: Evaluation results on Chinese idiom task. The Word, Chr-2 and Chr-1 denote the accuracy of guessing the whole word, two or more words, and one or more words correctly.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Idiom with Four words</th>
    <th colspan="6">Idiom with Multi-words</th>
  </tr>
  <tr>
    <th colspan="3">Word-level</th>
    <th colspan="3">Character-level</th>
    <th colspan="3">Word-level</th>
    <th colspan="3">Character-level</th>
  </tr>
  <tr>
    <th></th>
    <th>Word</th>
    <th>Chr-2</th>
    <th>Chr-1</th>
    <th>Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
    <th>Word</th>
    <th>Chr-2</th>
    <th>Chr-1</th>
    <th>Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
  </tr>
  <tr>
    <td>LLaVa-1.5-7B</td>
    <td>0.6</td>
    <td>3.8</td>
    <td>32.2</td>
    <td>10.5</td>
    <td>10.5</td>
    <td>10.5</td>
    <td>2.8</td>
    <td>7.9</td>
    <td>29.9</td>
    <td>9.0</td>
    <td>17.3</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>CogAgent-18B</td>
    <td>0.6</td>
    <td>4.4</td>
    <td>34.7</td>
    <td>11.6</td>
    <td>11.6</td>
    <td>11.6</td>
    <td>3.6</td>
    <td>8.2</td>
    <td>30.4</td>
    <td>8.7</td>
    <td>14.5</td>
    <td>10.9</td>
  </tr>
  <tr>
    <td>DeepSeek-VL-7B</td>
    <td>0.4</td>
    <td>2.3</td>
    <td>25.6</td>
    <td>6.4</td>
    <td>6.4</td>
    <td>6.4</td>
    <td>1.1</td>
    <td>4.9</td>
    <td>29.3</td>
    <td>8.7</td>
    <td>10</td>
    <td>9.3</td>
  </tr>
  <tr>
    <td>Qwen-VL-7B</td>
    <td>0.5</td>
    <td>4.7</td>
    <td>30.2</td>
    <td>8.7</td>
    <td>8.7</td>
    <td>8.7</td>
    <td>2.4</td>
    <td>9.8</td>
    <td>31.7</td>
    <td>9.1</td>
    <td>16.9</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td>
    <td>1.8</td>
    <td>8.6</td>
    <td><b>40.7</b></td>
    <td><b>12.3</b></td>
    <td><b>12.3</b></td>
    <td><b>12.3</b></td>
    <td>5.6</td>
    <td>11.2</td>
    <td><b>34.8</b></td>
    <td><b>11.3</b></td>
    <td><b>19.2</b></td>
    <td><b>14.2</b></td>
  </tr>
  <tr>
    <td>InternVL-2-8B</td>
    <td>0.8</td>
    <td>6.3</td>
    <td>37.8</td>
    <td>9.1</td>
    <td>9.1</td>
    <td>9.1</td>
    <td>3.4</td>
    <td>8.3</td>
    <td>29.4</td>
    <td>8.9</td>
    <td>15.6</td>
    <td>11.3</td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td>1.3</td>
    <td>6.7</td>
    <td>23.3</td>
    <td>8.0</td>
    <td>8.0</td>
    <td>8.0</td>
    <td>1.4</td>
    <td>2.9</td>
    <td>7.1</td>
    <td>6.0</td>
    <td>9.7</td>
    <td>7.4</td>
  </tr>
  <tr>
    <td>GPT-4v</td>
    <td>0.7</td>
    <td>1.3</td>
    <td>22.1</td>
    <td>5.8</td>
    <td>5.8</td>
    <td>5.8</td>
    <td>1.1</td>
    <td>6.8</td>
    <td>28.4</td>
    <td>3.7</td>
    <td>9.1</td>
    <td>5.3</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>3.3</b></td>
    <td><b>8.7</b></td>
    <td>27.5</td>
    <td>10.7</td>
    <td>10.7</td>
    <td>10.7</td>
    <td><b>9.1</b></td>
    <td><b>13.6</b></td>
    <td>27.3</td>
    <td><b>7.5</b></td>
    <td><b>18.1</b></td>
    <td>10.6</td>
  </tr>
</table>

ters, the final mapping remains contextually well-defined. In our eWe-bench, we model this process by encoding emoji sequences with their structure, ensuring a unique and unambiguous ground truth. More details about the emoji sequence and structures ensure the validity of ground truth in Appendix C.4.

3.4 Evaluation Metrics

The researcher can gain details of metric computation and insights that the MLLMs capabilities revealed by each metric in Appendix D. For automatic evaluation (Appendix D.2): 1) For the whole idiom, we compute the Precision, Recall, and F-1 value from the word/sentence-level that exactly matches the ground truth, which is the most direct metric to show whether MLLMs understands the abstract linguistic semantics of the image. 2) For characters in the idiom, we compute the Precision, Recall, F-1. and BLEU value that matches the ground truth from the character/word-level without considering the structural correspondence, which is aligned with fine-grained abstract linguistic semantics of image understanding. 3) In addition to the direct matching, we calculate the semantic similarity between the predicted answers and ground truth using GPT-4o. We also perform human evaluation of the model outputs (Appendix D.3) and discussion of future metrics (Appendix D.4).

4 Experiment Results

4.1 Baselines and Implementation

We select commercial Claude-3.5-sonnet-20241022, gpt-4-vision-preview and GPT-4o-20240513 to evaluate the eWe-bench benchmark. For a richer evaluation, we select a series of open-source MLLMs for testing. These include: 1) Qwen-VL-7B and Qwen2.5-VL-7B (Bai et al., 2023), DeepSeek-VL-7B (Lu et al., 2024a), which have good Chinese language support; 2) LLaVa-1.5-7B (Li et al., 2023b), CogAgent-18B (Hong et al., 2023), InternVL-2-8B which have good visual comprehension capabilities. We provide details of the baselines, implementation details, and the prompt in Appendix E.

4.2 Automatic Evaluation Results

Emoji to Chinese Idiom We evaluate four-character and multi-character idioms shown in Table 2 (error bar in Appendix F.1). We observe that all the MLLMs perform poorly: The latest model, GPT-4o, achieves accuracy scores of 3.3 and 5.0 at the word level for both tasks. However, even the strongest open source model Qwen2.5-VL still has a big gap with it, which proves that the different strengths of the models can still be observed on our more difficult benchmark. The accuracy at the Chr-1 is significantly higher, indicating that MLLMs are equipped with the basic translations of text corresponding to individual emojis, but have limited visual intuitive semiosis capability to further infer the corresponding linguistic meanings based on the relevant emoji context, especially for the harmonization reasoning with detailed analysis in Sec. 4.5. Thus, our eWe-bench is challenging for MLLMs to decode the visual ciphers.

Emoji to English Word and English Idiom MLLMs’s performance is higher compared to the two Chinese tasks ((error bar in Appendix F.1)). In Table 3, GPT-4o achieves impressive F-1 of 55.8
Table 3: Evaluation on English word and idiom task. B-1 and B-2 denote the BLEU-1 and BLEU-2 respectively.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Word-level</th>
    <th colspan="3">Character-level</th>
    <th colspan="3">Sentence-level</th>
    <th colspan="4">Word-level</th>
  </tr>
  <tr>
    <th>Pre.</th><th>Rec.</th><th>F-1</th>
    <th>Pre.</th><th>Rec.</th><th>F-1</th>
    <th>Pre.</th><th>Rec.</th><th>F-1</th>
    <th>Pre.</th><th>Rec.</th><th>F-1</th><th>B-1</th><th>B-2</th>
  </tr>
  <tr>
    <td>LLaVA-1.5-7B</td><td>30.1</td><td>30.1</td><td>30.1</td><td>54.6</td><td>55.7</td><td>55.1</td><td>14.4</td><td>14.4</td><td>14.4</td><td>19.7</td><td>21.3</td><td>20.5</td><td>19.7</td><td>16.4</td>
  </tr>
  <tr>
    <td>CogAgent-18B</td><td>29.8</td><td>29.8</td><td>29.8</td><td>52.8</td><td>51.9</td><td>52.3</td><td>13.2</td><td>13.2</td><td>13.2</td><td>18.3</td><td>19.5</td><td>18.9</td><td>18.3</td><td>15.2</td>
  </tr>
  <tr>
    <td>Deepseek-VL-7B</td><td>23.2</td><td>26.3</td><td>24.7</td><td>46.2</td><td>47.5</td><td>46.8</td><td>11.9</td><td>11.9</td><td>11.9</td><td>15.1</td><td>14.6</td><td>14.8</td><td>15.1</td><td>11</td>
  </tr>
  <tr>
    <td>Qwen-VL-7B</td><td>28.6</td><td>29.1</td><td>28.8</td><td>51.2</td><td>50.4</td><td>50.8</td><td>12.1</td><td>12.1</td><td>12.1</td><td>17.1</td><td>12.5</td><td>14.4</td><td>17.1</td><td>11.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td><td>33.7</td><td>33.7</td><td>33.7</td><td>57.6</td><td>58.9</td><td>58.2</td><td>18.6</td><td>18.6</td><td>18.6</td><td>23.5</td><td>21.6</td><td>22.5</td><td>20.7</td><td>19.2</td>
  </tr>
  <tr>
    <td>Intern-VL-2-8B</td><td>31.1</td><td>31.1</td><td>31.1</td><td>56.6</td><td>57.2</td><td>56.9</td><td>15.3</td><td>15.3</td><td>15.3</td><td>19.3</td><td>22.1</td><td>20.6</td><td>18.4</td><td>16.1</td>
  </tr>
  <tr>
    <td>Claude-3.5</td><td>42.3</td><td>42.3</td><td>42.3</td><td>63.9</td><td>73.8</td><td>68.5</td><td>29.8</td><td>29.8</td><td>29.8</td><td>48.0</td><td>42.7</td><td>45.2</td><td>42.3</td><td>39.7</td>
  </tr>
  <tr>
    <td>GPT-4v</td><td>38.5</td><td>38.5</td><td>38.5</td><td>60.3</td><td>69.2</td><td>64.4</td><td>26.4</td><td>26.4</td><td>26.4</td><td>41.1</td><td>43.1</td><td>42.1</td><td>39.4</td><td>37.5</td>
  </tr>
  <tr>
    <td>GPT-4o</td><td><b>55.8</b></td><td><b>55.8</b></td><td><b>55.8</b></td><td><b>68.5</b></td><td><b>77.5</b></td><td><b>72.7</b></td><td><b>35.2</b></td><td><b>35.2</b></td><td><b>35.2</b></td><td><b>46.8</b></td><td><b>47.3</b></td><td><b>47.0</b></td><td><b>45.0</b></td><td><b>41.6</b></td>
  </tr>
</table>

![The results of semantic similarity scores and distribution of Intern-VL-2 and GPT-4o.](page_1012_1042_388_246.png)

Figure 3: The results of semantic similarity scores and distribution of Intern-VL-2 and GPT-4o.

and 35.2 at the word and sentence levels, in English word and idiom respectively. This is likely because the model has encountered more similar English texts during training, making it more adept at reasoning about English words. However, MLLMs always suffer from hallucination problems when decoding visual cipher emojis, with detailed analysis in Sec. 4.5. When they catch a linguistic meaning of a single emoji, they quickly focus on the word or idiom related to this emoji from the inner knowledge they have, and ignore the relevant context of the emojis. Based on our eWe-bench, the community can explore the hallucination problem and improve the inference ability.

Evaluation of the semantic similarity We further compute the semantic similarity between the responses and the ground truth, applying LLM to score from 1 to 5. As shown in Figure 3, the average scores are low, with the English task significantly higher than those on the Chinese task. When carefully observing the distribution, we observe that 1)for the Chinese task, most of the scores are concentrated in 1 and 2, indicating the poor performance of MLLMs; 2)while for the English task, most of the scores are concentrated in 1 and 5, demonstrating that the MLLMs can either predict the answer correctly, or get irrelevant answers.

4.3 Further Exploration

Exploration with In-context Learning In addition to evaluating the direct inference abilities of MLLMs, we further explore their performance using in-context learning. We select the open-source Qwen-VL and the closed-source GPT-4o, evaluating each task with 3, 5, and 7 context examples, as shown in Table 4 and Table 5. MLLMs improve across various tasks with the addition of contextual examples, indicating the high quality of our Emoji2Idiom that the randomly chosen examples can improve the performance a lot. However, in the Chinese task, performance decreases when using too many samples (7 in-context examples). This decline indicates that MLLMs learn incorrect mappings in this complex task and suffer from hallucination issues.

Exploration with Chain-of-Thought We investigate the enhancement of CoT, prompting the MLLMs to think the fundamental meaning and reason homophones like humans, with the details in Appendix E.4.2. In Figure 4, by mimicking human reasoning, the CoT design enables the MLLMs to
Table 4: Exploration on in-context learning in Chinese idiom tasks.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Idiom with Four words</th>
    <th colspan="4">Idiom with Multi-words</th>
  </tr>
  <tr>
    <th>Word-level Word</th>
    <th>Chr-2</th>
    <th>Chr-1</th>
    <th>Character-level Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
    <th>Word-level Word</th>
    <th>Chr-2</th>
    <th>Chr-1</th>
    <th>Character-level Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
  </tr>
  <tr>
    <td>Qwen-VL-7B</td>
    <td>0.5</td>
    <td>4.7</td>
    <td>30.2</td>
    <td>8.7</td>
    <td>8.7</td>
    <td>8.7</td>
    <td>2.4</td>
    <td>9.8</td>
    <td>31.7</td>
    <td>9.1</td>
    <td>16.9</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>+3 in-context example</td>
    <td>0.5</td>
    <td>5.1</td>
    <td>31.3</td>
    <td>9.3</td>
    <td>9.3</td>
    <td>9.3</td>
    <td>2.2</td>
    <td>10.1</td>
    <td>28.6</td>
    <td>10.4</td>
    <td>13.1</td>
    <td>11.6</td>
  </tr>
  <tr>
    <td>+5 in-context example</td>
    <td>0.6</td>
    <td>5.3</td>
    <td>31.6</td>
    <td>9.4</td>
    <td>9.4</td>
    <td>9.4</td>
    <td>3.3</td>
    <td>12.3</td>
    <td>32.1</td>
    <td>11.7</td>
    <td>16.9</td>
    <td>13.8</td>
  </tr>
  <tr>
    <td>+7 in-context example</td>
    <td>0.5</td>
    <td>4.9</td>
    <td>32.1</td>
    <td>9.4</td>
    <td>9.4</td>
    <td>9.4</td>
    <td>2.8</td>
    <td>10.7</td>
    <td>31.4</td>
    <td>11.4</td>
    <td>15.4</td>
    <td>13.1</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>3.3</td>
    <td>8.7</td>
    <td>27.5</td>
    <td>10.7</td>
    <td>10.7</td>
    <td>10.7</td>
    <td>9.1</td>
    <td>13.6</td>
    <td>27.3</td>
    <td>7.5</td>
    <td>18.1</td>
    <td>10.6</td>
  </tr>
  <tr>
    <td>+3 in-context example</td>
    <td>2.6</td>
    <td>11.3</td>
    <td>33.9</td>
    <td>12.6</td>
    <td>12.6</td>
    <td>12.6</td>
    <td>9.5</td>
    <td>23.8</td>
    <td>36.9</td>
    <td>17.0</td>
    <td>21.9</td>
    <td>19.1</td>
  </tr>
  <tr>
    <td>+5 in-context example</td>
    <td>3.5</td>
    <td>12.2</td>
    <td>35.7</td>
    <td>13.7</td>
    <td>13.7</td>
    <td>13.7</td>
    <td>13.1</td>
    <td>27.4</td>
    <td>42.9</td>
    <td>20.7</td>
    <td>29.1</td>
    <td>24.2</td>
  </tr>
  <tr>
    <td>+7 in-context example</td>
    <td>3.5</td>
    <td>8.7</td>
    <td>31.3</td>
    <td>12.0</td>
    <td>12.0</td>
    <td>12.0</td>
    <td>10.7</td>
    <td>19.0</td>
    <td>34.5</td>
    <td>16.2</td>
    <td>23.1</td>
    <td>19.0</td>
  </tr>
</table>

Table 5: Exploration on in-context learning in English tasks.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">English Words</th>
    <th colspan="4">English Idiom</th>
  </tr>
  <tr>
    <th>Word-level Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
    <th>Character-level Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
    <th>Sentence-level Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
    <th>Word-level Pre.</th>
    <th>Rec.</th>
    <th>F-1</th>
  </tr>
  <tr>
    <td>Qwen-VL</td>
    <td>28.6</td>
    <td>29.1</td>
    <td>28.8</td>
    <td>51.2</td>
    <td>50.4</td>
    <td>50.8</td>
    <td>12.1</td>
    <td>12.1</td>
    <td>12.1</td>
    <td>17.1</td>
    <td>12.5</td>
    <td>14.4</td>
  </tr>
  <tr>
    <td>+3 in-context example</td>
    <td>29.3</td>
    <td>29.3</td>
    <td>29.3</td>
    <td>53.6</td>
    <td>52.1</td>
    <td>52.8</td>
    <td>12.0</td>
    <td>12.0</td>
    <td>12.0</td>
    <td>17.6</td>
    <td>17.9</td>
    <td>17.7</td>
  </tr>
  <tr>
    <td>+5 in-context example</td>
    <td>30.6</td>
    <td>30.6</td>
    <td>30.6</td>
    <td>55.9</td>
    <td>54.2</td>
    <td>55.0</td>
    <td>13.0</td>
    <td>13.0</td>
    <td>13.0</td>
    <td>18.9</td>
    <td>18.4</td>
    <td>18.6</td>
  </tr>
  <tr>
    <td>+7 in-context example</td>
    <td>32.5</td>
    <td>32.5</td>
    <td>32.5</td>
    <td>57.8</td>
    <td>55.7</td>
    <td>56.7</td>
    <td>15.2</td>
    <td>15.2</td>
    <td>15.2</td>
    <td>19.7</td>
    <td>20.2</td>
    <td>19.9</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>55.8</td>
    <td>55.8</td>
    <td>55.8</td>
    <td>68.5</td>
    <td>77.5</td>
    <td>72.7</td>
    <td>35.2</td>
    <td>35.2</td>
    <td>35.2</td>
    <td>46.8</td>
    <td>47.3</td>
    <td>47.0</td>
  </tr>
  <tr>
    <td>+3 in-context example</td>
    <td>57.6</td>
    <td>57.6</td>
    <td>57.6</td>
    <td>72.3</td>
    <td>75.0</td>
    <td>73.6</td>
    <td>36.3</td>
    <td>36.3</td>
    <td>36.3</td>
    <td>47.6</td>
    <td>47.3</td>
    <td>47.4</td>
  </tr>
  <tr>
    <td>+5 in-context example</td>
    <td>54.5</td>
    <td>54.5</td>
    <td>54.5</td>
    <td>77.5</td>
    <td>79.0</td>
    <td>78.2</td>
    <td>37.4</td>
    <td>37.4</td>
    <td>37.4</td>
    <td>48.2</td>
    <td>50.0</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>+7 in-context example</td>
    <td>60.6</td>
    <td>60.6</td>
    <td>60.6</td>
    <td>79.4</td>
    <td>73.9</td>
    <td>76.5</td>
    <td>38.5</td>
    <td>38.5</td>
    <td>38.5</td>
    <td>49.5</td>
    <td>50.5</td>
    <td>50.0</td>
  </tr>
</table>

produce better answers without additional training, improving accuracy at both character and word levels while improving semantic and visual similarity. This demonstrates the method’s effectiveness and the high quality of our data. We provide some further experimental results and insights of this harmonic symbol chain od thought reasoning in Appendix H.1.

Effect of Input Length and Emoji Representation We explore the impact of emoji sequence length. The balanced lengths of emoji sequence of Chinese four-character idioms and English words lead to minimal impact from resizing images. However, Chinese multi-character English idioms have longer sequences (averaging 7.48 and 5.32), tending to suffer from distortion due to resizing and performance degradation. Additionally, we investigate emoji representation differences across platforms (Windows, Mac, Android) and confirm that our unified rendering strategy on Windows does not significantly affect evaluation results. For a detailed discussion and experimental findings, please refer to the Appendix F.3.

4.4 Human Evaluation

Human Performance We invite human experts to participate and assess the task’s difficulty, thereby determining the upper limit of machine performance on this benchmark. The same evaluation metrics tests humans, and task complexity is rated on a scale from one (very easy) to five (very difficult), with more details on the evaluation and detailed results are provided in Appendix D.3 and F.4. Figure 5 show that MLLMs still has significant room for improvement, and our eWe-bench presents significant challenges.

Human Evaluation on MLLMs We conduct a human evaluation, assessing normality, semantic and emotional similarity to the ground truth, visual resemblance to emojis, and text fluency (details in Appendix F.5). As shown in Figure 6, Chinese performance falls behind English, aligning with automatic evaluation. While all MLLMs generate standardized idioms well, lower semantic and visual similarity highlight challenges in semiosis reasoning. Notably, GPT models excel in visual un-
Figure 4: The results of the base, ICL, and CoT approach, with the reasoning process of MLLMs.

Figure 5: Human performance on Chinese idiom task, where we map the score to the 1-100 interval.

Figure 6: Human evaluation. The std., sem, emo., vis., and flu. denote the standardization, semantic and emotional similarity to ground truth, visual similarity to the image, and fluency.

derstanding, while open-source models, especially Qwen2.5-VL, perform better in text semantics, offering insights for future research.

4.5 Case Study

We present a case study highlighting challenges (more examples in Appendix G) and insights for future intuitive semiosis MLLMs in Appendix H.2.

Harmonization Problem  MLLMs often fail on harmonization problems. As shown in Figure 7,

- “蛇 (snake, sound like “she”)” homonym to “舍 (leave, sound like “she”)” but the MLLMs fails to recognize. In the English idiom, - “whale” homonym to “well”. Our eWe-bench includes many harmonic character phenomena. Current MLLMs are not yet capable of effectively capturing visual context and reasoning with harmonic words, and struggle with our challenging eWe-bench.

Hallucination Problem  In Figure 7, the model recognizes and immediately outputs “horsing around”, without considering the surrounding emojis. Another example shows 3 directly associated with the idiom “颠三倒四 (disorderly)”, both containing the number 3. That is due to the hallucination problem. MLLMs often think narrowly, focusing only on words or idioms directly related to a single emoji. Our eWe-bench looks forward to further exploration of the poor performance of MLLMs that we have discovered.

Multi-emoji to One Character Mapping.  eWe-bench presents a huge challenge on this mapping issue, where MLLMs fail to perform a multi-to-one or one-to-multi mapping. For example, 1 0 0 0 are four emojis, but the model does not successfully combine them into one character “千 (one thousand)”, which greatly inspire the application of cryptography.

Abstract Visual Understanding  MLLMs struggle to align emoji semantics with intricate meanings when it comes to deep comprehension. For example, in “receive a kickback”, the model simply captures the meaning of “box”, and interprets it as “out of the box”, but does not combine the
<table>
  <tr>
    <th>Problems</th>
    <th>Chinese Idiom with four character</th>
    <th>Chinese Idiom with Multi-characters</th>
    <th>English Word</th>
    <th>English Idiom</th>
  </tr>
  <tr>
    <td>Couldn’t identity homophonic characters <span style="color:red">(Red color denotes the homophonic characters)</span></td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:<br>Ground truth: Well, I hope so<br>GPT-4o: Whale, I hope so</td>
  </tr>
  <tr>
    <td>Suffer hallucination problem</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:<br>Ground truth: To pony up<br>GPT-4o: horsing around</td>
  </tr>
  <tr>
    <td>Multi-emoji to one character mapping</td>
    <td>Emoji:<br>Ground truth: 万壑千岩<br>GPT-4o: 差强人意</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:<br>Ground truth: Blow off steam<br>GPT-4o: blow hot and cold</td>
  </tr>
  <tr>
    <td>Abstract Visual understanding of the emoji symbol</td>
    <td>Emoji:</td>
    <td>Emoji:<br>Ground truth: 知无不言，言无不听<br>GPT-4o: 不求不得人</td>
    <td>Emoji:</td>
    <td>Emoji:<br>Ground truth: Receive a kickback<br>GPT-4o: out of the box</td>
  </tr>
</table>

Figure 7: Four typical challenges the GPT-4o suffer in our eWe-bench.

attributes of “receiving something” with the hint of money to generate the correct answer. Our eWe-bench highly focuses on this deeper understanding, exploring the capabilities of MLLMs.

5 Conclusion

We propose the eWe-bench benchmark, containing emoji visual ciphers decoding tasks including Chinese idioms, English words, and English idioms, which provides a novel way to measure the visual intuitive semiosis ability of MLLMs to directly associate the image with its abstract linguistic semantics. We design a data engineering framework that performs data collection, data filtering, and evaluation, contributing to validating the unified high-level vision-language understanding and reasoning like human intuitive cognition. We evaluate advanced open-source and closed-source MLLMs with our eWe-bench, analyze the performance, conduct several explorations, and highlight future research directions with further case study.

Limitations

In our proposed eWe-bench, we select the two most commonly used emoji-to-idiom text types: Chinese and English, highlighting the challenges of our benchmark. Additionally, more languages, such as Japanese, Korean, French, German, and Arabic, can be incorporated to further evaluate MLLMs’ ability to understand the correlation between images and linguistic semantics.

Ethics Statement

We introduce a novel benchmark, eWe-bench, incorporating a thorough description of data collection, annotation, and filtration processes. We emphasize that the dataset’s creation adheres strictly to ethical guidelines. Great care has been taken to uphold ethical standards in the dataset, employing anonymization, desensitization, and data cleaning. The samples pose no risk to public welfare. For all data sourced from these websites, we contact the site administrators to obtain permission for data usage. Additionally, we sign intellectual property sharing agreements with them to ensure compliance with ethical and legal standards. Hence, the innovative research directions and tasks proposed are ethically harmless to society.

Acknowledgement

This research is supported by Key-Area Research and Development Program of GuangdongProvince No.2024B1111100001.

References

Jonathan E Abel. 2019. Not everyone s: Or, the question of emoji as ‘universal’ expression. In Emoticons, Kaomoji, and Emoji, pages 25–43. Routledge.
Giuseppe Ateniese, Carlo Blundo, Alfredo De Santis, and Douglas R Stinson. 1996. Visual cryptography for general access structures. Information and computation, 129(2):86–106.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Xing Luo, Chenyu Yi, and Alex Kot. 2023. Benchlmm: Benchmarking cross-style visual capability of large multimodal models. arXiv preprint arXiv:2312.02896.

Yuxiao Chen, Jianbo Yuan, Quanzeng You, and Jiebo Luo. 2018. Twitter sentiment analysis via bi-sense emoji embedding and attention-based lstm. In Proceedings of the 26th ACM international conference on Multimedia, pages 117–125.

Zhenpeng Chen, Sheng Shen, Ziniu Hu, Xuan Lu, Qiaozhu Mei, and Xuanzhe Liu. 2019. Emoji-powered representation learning for cross-lingual sentiment classification. In The world wide web conference, pages 251–262.

Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. 2024. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979.

Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287.

Marcel Danesi. 2022. Emotional wellbeing and the semiotic translation of emojis. In Exploring the Translatability of Emotions: Cross-Cultural and Transdisciplinary Encounters, pages 323–344. Springer.

Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2023. A survey of natural language generation. ACM Comput. Surv., 55(8):173:1–173:38.

Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. 2023. Decoding visual neural representations by multimodal learning of brain-visual-linguistic features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10760–10777.

Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bosnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiaow Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023a. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.

Chaoyou Fu, Renrui Zhang, Haojia Lin, Zihan Wang, Timin Gao, Yongdong Luo, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, et al. 2023b. A challenger to gpt-4v? early explorations of gemini in visual expertise. arXiv preprint arXiv:2312.12436.

Will Gibson, Pingping Huang, and Qianyun Yu. 2018. Emoji and communicative action: The semiotics, sequence and gestural actions of ‘face covering hand’. Discourse, Context & Media, 26:91–99.

J Allan Hobson and Edward F Pace-Schott. 2002. The cognitive neuroscience of sleep: neuronal systems, consciousness and learning. Nature Reviews Neuroscience, 3(9):679–693.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. Cogagent: A visual language model for GUI agents. CoRR, abs/2312.08914.

Haojing Huang, Jingheng Ye, Qingyu Zhou, Yinghui Li, Yangning Li, Feng Zhou, and Hai-Tao Zheng. 2023. A frustratingly easy plug-and-play detection-and-reasoning module for chinese spelling check. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 11514–11525. Association for Computational Linguistics.

Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, and Haitao Zheng. 2024. Lateral: An interactive ilms evaluation benchmark with incomplete information from lateral thinking puzzles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 10186–10197. ELRA and ICCL.

Muhammad Nazrul Islam and Harry Bouwman. 2016. Towards user-intuitive web interface sign design and evaluation: A semiotic framework. International Journal of Human-Computer Studies, 86:121–137.

Lars Henning Klein, Roland Aydin, and Robert West. 2024. Emojinize: Enriching any text with emoji translations. arXiv preprint arXiv:2403.03857.

Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. 2024. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36.

Jiayi Kuang, Jingyou Xie, Haohao Luo, Ronghao Li, Zhe Xu, Xianfeng Cheng, Yinghui Li, Xika Lin, and Ying Shen. 2024. Natural language understanding
and inference with mlm in visual question answering: A survey. arXiv preprint arXiv:2411.17558.

Vanessa Leonardi. 2022. Communication challenges and transformations in the digital era: emoji language and emoji translation. Language and Semiotic Studies, 8(3):22–44.

Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. 2024a. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2023a. Seed-bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2024b. Seed-bench: Benchmarking multimodal llms with generative comprehension. In CVPR.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau mann, Hoifung Poon, and Jianfeng Gao. 2023b. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Fei Huang, Jingren Zhou, and Philip S. Yu. 2025a. Benchmarking multimodal retrieval augmented generation with dynamic VQA dataset and self-adaptive planning agent. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Yangning Li, Tingwei Lu, Hai-Tao Zheng, Yinghui Li, Shulin Huang, Tianyu Yu, Jun Yuan, and Rui Zhang. 2024c. MESED: A multi-modal entity set expansion dataset with fine-grained semantic classes and hard negative entities. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 8697–8706. AAAI Press.

Yinghui Li, Haojing Huang, Jiayi Kuang, Yangning Li, Shu-Yu Guo, Chao Qu, Xiaoyu Tan, Hai-Tao Zheng, Ying Shen, and Philip S. Yu. 2025b. Refine knowledge of large language models via adaptive contrastive learning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao Zheng, and Qingyu Zhou. 2023c. On the (in)effectiveness of large language models for chinese text correction. CoRR, abs/2307.09007.

Yinghui Li, Jiayi Kuang, Haojing Huang, Zhikun Xu, Xinnian Liang, Yi Yu, Wenlian Lu, Yangning Li, Xiaoyu Tan, Chao Qu, Ying Shen, Hai-Tao Zheng, and Philip S. Yu. 2025c. One example shown, many concepts known! counterexample-driven conceptual reasoning in mathematical llms. CoRR, abs/2502.10454.

Yinghui Li, Yangning Li, Yuxin He, Tianyu Yu, Ying Shen, and Hai-Tao Zheng. 2022a. Contrastive learning with hard negative entities for entity set expansion. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 1077–1086. ACM.

Yinghui Li, Shirong Ma, Qingyu Zhou, Zhongli Li, Yangning Li, Shulin Huang, Ruiyang Liu, Chao Li, Yunbo Cao, and Haitao Zheng. 2022b. Learning from the dictionary: Heterogeneous knowledge guided fine-tuning for chinese spell checking. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 238–249. Association for Computational Linguistics.

Yinghui Li, Shang Qin, Jingcheng Ye, Shirong Ma, Yangning Li, Libo Qin, Xuming Hu, Wenhao Jiang, Hai-Tao Zheng, and Philip S. Yu. 2024d. Rethinking the roles of large language models in chinese grammatical error correction. CoRR, abs/2402.11420.

Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, and Ying Shen. 2024e. Towards real-world writing assistance: A chinese character checking benchmark with faked and misspelled characters. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 8656–8668. Association for Computational Linguistics.

Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zichen Wang, Chao Li, Yunbo Cao, and Hai-Tao Zheng. 2022c. The past mistake is the future wisdom: Error-driven contrastive probability optimization for chinese spell checking. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3202–3213. Association for Computational Linguistics.

Yinghui Li, Qingyu Zhou, Yuanzhen Luo, Shirong Ma, Yangning Li, Hai-Tao Zheng, Xuming Hu, and Philip S. Yu. 2024f. When llms meet cunning texts: A fallacy understanding benchmark for large language models. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.
Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Leyang Cui, Zhaopeng Tu, Longyue Wang, and Luping Zhou. 2023d. A comprehensive study of gpt-4v’s multimodal capabilities in medical imaging. medRxiv, pages 2023–11.

Chuchu Liu, Fan Fang, Xu Lin, Tie Cai, Xu Tan, Jianguo Liu, and Xin Lu. 2021. Improving sentiment analysis accuracy with emoji embedding. Journal of Safety Science and Resilience, 2(4):246–252.

Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint arXiv:2310.14566.

Ruiyang Liu, Yinghui Li, Limin Tao, Dun Liang, and Hai-Tao Zheng. 2022. Are we ready for a new paradigm shift? A survey on visual deep MLP. Patterns, 3(7):100520.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. 2024a. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024b. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. In ICLR.

Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, and Tat-Seng Chua. 2024. Chain-of-exemplar: Enhancing distractor generation for multimodal educational question generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 7978–7993. Association for Computational Linguistics.

Shirong Ma, Yinghui Li, Rongyi Sun, Qingyu Zhou, Shulin Huang, Ding Zhang, Yangning Li, Ruiyang Liu, Zhongli Li, Yunbo Cao, Haitao Zheng, and Ying Shen. 2022. Linguistic rules-based corpus generation for native chinese grammatical error correction. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 576–589. Association for Computational Linguistics.

Johanna Monti, Federico Sangati, Francesca Chiusaroli, Benjamin Martin, Mansour Sina, et al. 2016. Emojitalianobot and emojiworldbot-new online tools and digital environments for translation into emoji. In Proceedings of Third Italian Conference on Computational Linguistics (CLiC-it 2016).

Pierre Moulin and Joseph A O’Sullivan. 2003. Information-theoretic analysis of information hiding. IEEE Transactions on information theory, 49(3):563–593.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824.

Jonathan Roberts, Timo Lüddecke, Rehan Sheikh, Kai Han, and Samuel Albanie. 2023. Charting new territories: Exploring the geographic and geospatial capabilities of multimodal llms. arXiv preprint arXiv:2311.14656.

Luca M Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. 2025. Visual cognition in multimodal large language models. Nature Machine Intelligence, pages 1–11.

Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. 2023. Beyond task performance: Evaluating and reducing the flaws of large multimodal models with in-context learning. arXiv preprint arXiv:2310.00647.

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209.

Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. 2024. Exploring the reasoning abilities of multimodal large language models (mlms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.06805.

Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE.

Zhe Xu, Daoyuan Chen, Jiayi Kuang, Zihao Yi, Yaliang Li, and Ying Shen. 2024. Dynamic demonstration retrieval and cognitive understanding for emotional support conversation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 774 – 784, New York, NY, USA. Association for Computing Machinery.

Zhikun Xu, Yinghui Li, Ruixue Ding, Xinyu Wang, Boli Chen, Yong Jiang, Haitao Zheng, Wenlian Lu, Pengjun Xie, and Fei Huang. 2025. Let llms take on the latest challenges! A chinese dynamic question answering benchmark. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025, pages 10435–10448. Association for Computational Linguistics.
Jingheng Ye, Yinghui Li, Qingyu Zhou, Yangning Li, Shirong Ma, Hai-Tao Zheng, and Ying Shen. 2023. CLEME: debiasing multi-reference evaluation for grammatical error correction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6174–6189. Association for Computational Linguistics.

Jingheng Ye, Shang Qin, Yinghui Li, Xuxin Cheng, Libo Qin, Hai-Tao Zheng, Peng Xing, Zishan Xu, Guo Cheng, and Zhao Wei. 2024. EXCGEC: A benchmark of edit-wise explainable chinese grammatical error correction. CoRR, abs/2407.00924.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.

Tianyu Yu, Chengyue Jiang, Chao Lou, Shen Huang, Xiaobin Wang, Wei Liu, Jiong Cai, Yangning Li, Yinghui Li, Kewei Tu, Hai-Tao Zheng, Ningyu Zhang, Pengjun Xie, Fei Huang, and Yong Jiang. 2024. Seqgpt: An out-of-the-box large language model for open domain sequence understanding. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 19458–19467. AAAI Press.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqin Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhui Chen. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR.

Ding Zhang, Yinghui Li, Qingyu Zhou, Shirong Ma, Yangning Li, Yunbo Cao, and Hai-Tao Zheng. 2023. Contextual similarity is more valuable than character similarity: An empirical study for chinese spell checking. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pages 1–5. IEEE.

Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024a. Mmllms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601.

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. 2024b. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944.

Jiawei Zhang, Tianyu Pang, Chao Du, Yi Ren, Bo Li, and Min Lin. 2024c. Benchmarking large multimodal models against common corruptions. In NAACL.

A Additional Details of Data Filtering

A.1 Automatic Filtering

In this phase, we mainly utilize machines and large language models to filter large-scale data, which includes the following steps in total:

1. Deletion of default values. We utilize a machine to automatically remove incomplete emoji-idiom pairs, including those with missing corresponding emoji images and those with missing corresponding idioms. It is guaranteed that each emoji image corresponds to the standard idiom answer one by one.

2. Removing Duplicate Values. We utilize the machine to automatically remove duplicate emoji-idiom pairs. Here, we only need to remove the emoji-idiom pairs corresponding to identical emoji sequences while retaining the pairs with the same idiom text but corresponding to different emoji representations, which helps to enhance the diversity of the dataset. Note that we will first filter the pairs corresponding to the same idiom text by the machine with additional labels, and make a manual decision on whether to perform the deletion in the next stage of manual filtering.

3. Image Quality Check. We utilize LLM (specifically GPT-4o is used), to perform image quality checking, which entails marking and removing: images that are too blurry and those that do not meet the ethical norms (images that contain elements of violence, abusive language, discrimination, etc.) along with their corresponding idioms.

4. Text Ethics Checking. We utilize LLM (specifically GPT-4o) to perform text ethics checking, which involves tagging and deleting idiom with elements of violence, discrimination, abuse, etc. For example, “红颜祸水” is a sexist idiom, and we will delete its corresponding emoji-idiom pair.

A.2 Human Filtering

In this phase, we invited human experts in Chinese and English languages to perform manual data filtering, which included the following steps in total:
1. Duplicate value checking: for the automatic filtering phase, the machine flags a portion of emoji-idiom pairs where the text is the same but the corresponding images are not the same. the human expert needs to further check whether the emoji expressions here are really different. For the pairs with identical emoji images, the human expert will delete them.

2. Image quality check: Human experts further check whether the emoji images are unclear and illegible, and remove the illegible images.

3. Idiom standardization check: Human experts need to check whether the idiom text expression is standardized, including the format of the idiom, whether it has a specific linguistic meaning, and whether it is in line with common human usage, etc., to ensure that our dataset meets the real-world usability. For example, for the idiom “blue sky and white clouds”, although it is a four-word idiom that conforms to the norms of human usage, it does not have a specific allusion, mythological story, traditional story background, or special semantic meaning, and does not belong to the standard idioms. For example, although “流水高山” is a four-letter word with a specific historical background, people more often use the expression “high mountains and flowing water”. Therefore, “流水高山” is not an expression that conforms to human language usage and will be deleted.

4. Emoji and Idiom Relevance Check: Since in emoji to idiom expression, many times the representation of harmonic characters will be utilized, which will increase the difficulty of emoji comprehension and the difficulty of generating the final idioms. Human experts will evaluate the relevance of emoji to idioms:

• If too many or too complex harmonic characters are used with the emoji representation, at this time the task will be too difficult for not only MLLMs but also humans to understand. At this point, the human expert will consider the relevance of this emoji sequence to the idiom to be too low and delete the emoji-idiom pair.
• It is noteworthy that we conducted an evaluation of human ability on this benchmark in Sec. 4.4 and found that humans achieved an average score of 66.5 on the word-level accuracy of Chinese idioms. This score demonstrates both that our dataset is challenging and that the task is accomplishable, and that there is still much room for improvement in the performance of the current MLLMs on this task.

5. Repeated harmonic word mapping check: due to the limited expression of emoji, when using emoji to replace textual expressions, harmonic words are often used to find the corresponding emoji for expression. eWe-bench also has a large number of harmonic words. However, we found that if the mapping of the same emoji corresponding to a certain harmonic word occurs too many times, it may cause data bias to LLM in subsequent training, i.e., when LLM sees this emoji it automatically thinks of this harmonic word that occurs multiple times. To mitigate the bias caused by this harmonic word mapping, we performed:

• Count the repeated emoji-character harmonic word mappings, and when there are more than ten occurrences, we manually replace the expression of the emoji (find other harmonic word counterparts to replace the original repeated emoji), or just delete the redundant emoji-idiom pair.
• In addition, we also considered this issue during the original data collection. Our retrieval and collection in different sources of the original emoji database can reduce this duplicate mapping. We also take different generation methods when manually constructing text-to-emoji data, which also helps to increase the diversity of harmonic word mappings.

6. Safety and Ethics Check: Based on the automatic detection, the human experts further conducted a safety and ethics check of the emoji images and idiom text, checking whether there are any issues such as violent gore, abusive language, sexism, racial discrimination, stereotyping, and so on, in the data.

To eliminate subjectivity in manual filtering, we provide annotators with detailed guidelines as
shown in Figure 8 and 9, including scoring criteria for each item (1-5 points) covering idiomatic normality, graphic consistency, image legibility, repetition mapping, and ethical safety checks. We also provide at least three examples for each item. For ethical safety checks, we distinguish between sub-categories such as violence, abusive language, gender discrimination, stereotyping, and racial discrimination. We provide examples at both the emoji and text levels to guide judgments. The annotators are student volunteers who are native speakers of Chinese and English.

B Additional Details of Data statistics

In addition to the numerical statistics, we further do some statistics to better show our eWe-bench.

Word Frequency and Word Cloud Statistic of Chinese idiom  To better present our dataset, we perform word frequency statistics on Chinese idioms and display the word cloud and word rectangle tree graphs, as shown in Figure 10. We first perform word frequency statistics on all characters, filter out the top 1,000 characters, and discard low-frequency words. From the filtered top 1,000 characters, we conduct lexical analysis and plot word cloud and word rectangle diagrams for adjective and adverbial morphemes, noun morphemes, and verb morphemes, respectively.

Word Frequency and Word Cloud Statistic of English idiom  Similarly, we perform word frequency statistics on English idioms and display word cloud maps with word rectangle tree diagrams, as shown in Figure 11. We first perform word frequency statistics on all words, filter out the top 180 words, and discard low-frequency words. From the filtered top 180 words, we create word cloud maps with word rectangle mapping.

C Additional Details of Data Attributes and Linguistic Phenomenon

C.1 Chinese Idiom Task

Harmonization Word  Since it is sometimes difficult to find directly related emoji to represent, harmonic characters with similar pronunciations are often chosen to replace them. For example, Usually, for characters that can’t be represented directly by emoji, we will first search for their harmonized characters, then find an emoji that can directly represent the harmonized character, and replace it with this emoji. For example, “捷” does not have a direct emoji, but it harmonizes with 纠”, which corresponds to “bow” 🏹, and so, we select 🏹 chosen to represent the character “捷”. There are a large number of harmonic characters in our data. This poses a great challenge to MLLMs’s understanding and reasoning ability. The reasoning of harmonic words needs the help of related contexts, and in our data scenario, the model is required to analyze the context of emoji in depth instead of understanding individual emoji alone. The understanding of these harmonics usually requires the model to synthesize the relevant context of the emoji, to reason out the correct expression of the harmonized words.

Abstract visual Emoji Understanding. The model shows better performance in simply recognizing the shallow meanings of individual emoji, but in Abstract visual in-depth understanding, it is difficult for the model to work with the contextual emoji information to get the real corresponding relevant emoji meanings. For example, VS means match, PK, duel, competition, and so on. In Chinese, “决” represents duel, and then harmonized to “绝” to get the idiom “精才绝艳”. In “African elephant”, the superficial meaning of the emoji is the earth, but further combined with the specific location of the earth map in the figure and the hint of an elephant, the emoji represents the African elephant. Abstract visual understanding in conjunction with its textual meaning to further reason about the correct answer. In 🌟❤️💪“同心协力 (pull together with the same goal) ”, 💪 is an arm, but it does not mean “arm” in idioms. Instead, it is a very strong arm, which corresponds to “力(power)””.

Chinese Idiom Format  Chinese idioms are a special kind of words, which often have specific formats and semantic information, so they cannot directly translate the meaning of a single emoji and concatenate words into sentences. The most common format is four-character idioms, which often come from ancient Chinese myths, historical stories, classics, etc., consisting of four Chinese characters, with a Chinese literary style, and often a symmetrical structure. In addition, multi-character idioms, although far fewer in number than four-character idioms, are equally important components. Some of them have less than four words (e.g., three-character idioms) and some have more than four words. Generally speaking, whether it is a four-character idiom or a multi-character id-
Guideline of Human Filtering of Data - 1

The purpose of this work is to screen out emoji-text pairs that do not comply with the rules. For each indicator, there will be a corresponding criterion and examples, which you will need to score the emoji-text pairs, and only the pairs that meet the requirements of each indicator can be retained.

<table>
  <tr>
    <th colspan="2">Case</th>
  </tr>
  <tr>
    <td>Image:<br><img src="page_184_153_176_64.png" alt="Image of people with different expressions"></td>
    <td>Normality: 5<br>Consistency: 4<br>legibility: 4<br>Emoji Ethical security: 4<br>Text Ethical security: 5</td>
  </tr>
  <tr>
    <th colspan="2">Metrics</th>
  </tr>
  <tr>
    <td colspan="2">&gt; Normality: Whether the text conforms to the idiom’s specifications. This includes whether it has historical allusions and specific cultural backgrounds, or does not conform to human usage habits</td>
  </tr>
  <tr>
    <th>Options</th>
    <td>1. Completely non-standard   2. Mostly non-standard   3. Fairly standard<br>4. Mostly standard   5. Completely standard</td>
  </tr>
  <tr>
    <th>Examples</th>
    <td>1. “朝三暮四” shows completely standard to the normality.<br>2. “蓝天白云” shows completely non-standard.<br>3. “红红火火” shows mostly non-standard.</td>
  </tr>
  <tr>
    <td colspan="2">&gt; Consistency: The consistency of the emoji and the image is scored, and the higher the consistency of the example, the easier it is to get the final translation result</td>
  </tr>
  <tr>
    <th>Options</th>
    <td>1. Completely inconsistent   2. Mostly inconsistent   3. Fairly consistent<br>4. Mostly consistent   5. Completely consistent</td>
  </tr>
  <tr>
    <th>Examples</th>
    <td>1. 🍔🍟🥤 -“两面三刀” pair is mostly consistent.<br>2. 🧑‍♂️🧑‍♂️ -“出尔反尔” pair is mostly inconsistent.<br>3. 🤦‍♂️ - “I’m bored.” pair is completely inconsistent.</td>
  </tr>
  <tr>
    <td colspan="2">&gt; legibility: Whether the image is very blurry and illegible is difficult for MLLM to process.</td>
  </tr>
  <tr>
    <th>Options</th>
    <td>1. Completely illegible   2. Mostly illegible   3. Fairly legible<br>4. Mostly legible   5. Completely legible</td>
  </tr>
  <tr>
    <th>Examples</th>
    <td>1. 🍏🍎🍏 shows completely legible.<br>2. 🌞🌞 shows mostly illegible.<br>3. 📚📚📚 shows fairly legible.</td>
  </tr>
  <tr>
    <td colspan="2">&gt; Duplicate emoji-character mapping: Remove or modify the duplicate emoji-character mapping of emojis.</td>
  </tr>
  <tr>
    <th>Examples</th>
    <td>⽟宇琼楼   玉减香消   白玉微瑕<br>钟鼓馔玉   金玉其表   金风玉露<br>In these examples, 玉 (corn, 玉米) is used to map the Chinese character "玉"</td>
  </tr>
  <tr>
    <th>Method</th>
    <td>Count a single emoji and a single character pair that occur repeatedly. When there are more than 10 times, delete the corresponding emoji-text pairs, or replace a single emoji until the number of homophonic pairs is equal to 10.</td>
  </tr>
</table>

Figure 8: The first page guidelines for human filtering.
<table>
  <tr>
    <th colspan="2">Guideline of Human Filtering of Data - 2</th>
  </tr>
  <tr>
    <td colspan="2">The purpose of this work is to screen out emoji-text pairs that do not comply with the rules. For each indicator, there will be a corresponding criterion and examples, which you will need to score the emoji-text pairs, and only the pairs that meet the requirements of each indicator can be retained.</td>
  </tr>
  <tr>
    <th colspan="2">Case</th>
  </tr>
  <tr>
    <td>Image:<br>Text: 捷雷不及掩耳</td>
    <td>Normality: 5<br>Consistency: 4<br>legibility: 4<br>Emoji Ethical security: 4<br>Text Ethical security: 5</td>
  </tr>
  <tr>
    <th colspan="2">Evaluation Metrics</th>
  </tr>
  <tr>
    <td colspan="2">&gt; <b>Ethical security check:</b> Remove emoji-text pairs that are not ethically safe. We rigorously vet emoji-text pairs for issues such as violence, name-calling, and gender bias.</td>
  </tr>
  <tr>
    <th colspan="2">Emoji images filtering</th>
  </tr>
  <tr>
    <th>Possible issues</th>
    <td>Contains elements of violence, abusiveness, racial discrimination, gender discrimination, and stereotypes.</td>
  </tr>
  <tr>
    <th>Options</th>
    <td>1. Completely insecure   2. Mostly insecure   3. Fairly secure<br>4. Mostly secure   5. Completely secure</td>
  </tr>
  <tr>
    <th>Examples</th>
    <td>1. 🚫🚫🚫🚫🚫 shows completely insecure to the ethics, due to the abusiveness.<br>2. 🚫🚫🚫🚫🚫 shows mostly insecure to the ethics, because it does not conform to social order and good customs.<br>3. 🚫🚫🚫🚫🚫 shows completely insecure to the ethics, due to violence.</td>
  </tr>
  <tr>
    <th colspan="2">Texts filtering</th>
  </tr>
  <tr>
    <th>Possible issues</th>
    <td>Contains elements of violence, abusiveness, racial discrimination, gender discrimination, stereotypes, expressions of partiality and passion, and does not conform to social order and good customs.</td>
  </tr>
  <tr>
    <th>Options</th>
    <td>1. Completely insecure   2. Mostly insecure   3. Fairly secure<br>4. Mostly secure   5. Completely secure</td>
  </tr>
  <tr>
    <th>Examples</th>
    <td>1. “头发长见识短” shows mostly insecure to the ethics, due to the gender discrimination on the women.<br>2. “男主外女主内” shows completely insecure to the ethics, due to the stereotypes.<br>3. “feisty woman” shows completely insecure to the ethics, due to the gender discrimination.</td>
  </tr>
</table>

Figure 9: The guidelines for human filtering.
(a) Word Cloud Graph of the Chinese adjectives

(b) Word Rectangular Tree Graph of the Chinese adjectives

(c) Word Cloud Graph of the Chinese nouns

(d) Word Rectangular Tree Graph of the Chinese nouns

(e) Word Cloud Graph of the Chinese verbs

(f) Word Rectangular Tree Graph of the Chinese verbs

Figure 10: Word cloud and word rectangle diagrams of Chinese idiom, including adjective and adverbial morphemes, noun morphemes, and verb morphemes.

(a) Word Cloud Graph of the English idiom

(b) Word Rectangular Tree Graph of the English idiom

Figure 11: Word cloud and word rectangle diagrams of English idiom.
iom, it follows the one-to-one relationship between emoji and characters, but there are special cases.

Chinese Character Mapping  Usually, idioms follow a one-to-one relationship between emojis and characters, but there are special cases. First of all, there will be multiple emojis corresponding to one character. Often, many numbers will have this correspondence, especially those with large digits. For instance, “１０００００” denotes the “万” (ten thousand). In addition, there is a mapping relationship between multiple characters in an emoji. This kind of correspondence is relatively rare, usually in multi-character idioms, and this one-to-many mapping relationship occurs when two or more characters can form a new word represented by an emoji. The above two mapping relationships require MLLMs to further complete the understanding and reasoning of multiple emoji contexts on the basis of recognizing the meaning of a single emoji.

C.2 English Word Task

English Word Split  In the English word task, unlike the regular one word corresponding to one emoji, it is common for multiple emoji to represent one word. The task usually splits the word, corresponds multiple emoji to different parts, and finally synthesizes them into one word. For example, “blackBerry” is split into “black-”, “ber-”, “-ry”, and then the 🐼 is utilized to represent “ber-”, and finally, the box of black and the letter “E” is added to get “blackBerry”. The word “Panda” can be split into “Pan-” and “-da,” where “Pan-” corresponds to 🐼. This kind of word splitting usually does not occur alone but is also accompanied by the linguistic phenomenon of harmonic words with many-to-one mapping. For example, in the word “lemon”, the word is split into “le-” and “-mon”, then “mon-” is harmonized as “man”, and 🍋 is chosen to represent the split syllable “mon”. Beyond understanding the meaning of individual emojis, the MLLMs must also remove unnecessary letters and combine the parts to infer a completely new word.

C.3 English Idiom Task

Harmonization Word  Similar to Chinese idioms, there are also a lot of harmonic characters in English idioms. Sometimes difficult to find directly related emoji to represent, harmonic characters with similar pronunciations are often chosen to replace them. For example, “To be loaded”, “To” harmonizes with “Two” ２, and “be” harmonizes with “bee” 🐝. Most English idioms still keep the simple direct correspondence between emoji and words. What is more challenging for English idioms is their Abstract visual comprehension and word mapping reasoning problem.

Abstract visual Emoji Understanding  In English, for emoji that cannot be represented by direct correspondence, the data do not tend to choose harmonic words, but further associate related emoji, putting further demands on the reasoning ability of MLLMs. For example, in “As genuine as a three-dollar bill”, “genius” is usually accompanied by intellect and inspiration, and so a shining star ✨ is used to represent the image of sparkling inspiration of such genius. This deeper level of image comprehension requires a greater understanding of the meaning of the image and the text behind it.

Word in English Idiom Mapping  Unlike most one-to-one relationships in Chinese idioms, there are a large number of non-one-to-one correspondences in English idioms. Due to the large number of articles, prepositions, conjunctions, and other words in English that are difficult to directly use emojis, such words are usually omitted in the emoji representation of English idiom, and only the most critical nouns, adjectives, verbs, etc., are retained to express the core meaning. Therefore, the prediction process of English idiom is not a one-to-one translation mapping, which also poses more challenges to the ability of MLLMs. For example, in “An apple a day keeps the doctor away.”, for MLLMs, it is necessary to reason out such common idioms just for the emojis of 🍎 and 💉. This examines the internal knowledge-mining ability of the large language model and the strong reasoning ability. However, this kind of reasoning is also very easy to cause the hallucination problem.

C.4 Detailed Discussions about Emoji Ambiguity and Ground Truth Validity

One major concern in emoji-to-character mapping is the inherent ambiguity, as a single emoji may correspond to multiple homophonic Chinese characters (e.g., 惊 jīng →惊 (surprise) or 精 (excellent)). However, this does not compromise the validity of our ground truth but instead increases the challenge for MLLMs. Our extensive analysis of emoji-idiom cases indicates that the mapping follows a systematic and structured process:
1. Direct meaning priority: The first preference is given to a direct semantic match (e.g., →whale).

2. Homophonic substitution: If no direct match exists, homophones are used (e.g., whale (鲸) →惊 (surprise), 景 (scenery), 精 (essence)).

3. Contextual disambiguation: The correct character is inferred based on surrounding context:

• 大驚失色 (greatly surprised) →惊 corresponds to 惊.
• 才绝艳 (exceptionally talented) →精 corresponds to 精.

Thus, while an emoji may map to multiple characters, this mapping follows a structured two-step process:

1. Each emoji directly corresponds to a fundamental meaning.

2. The specific final character is inferred contextually, ensuring a single, well-defined answer.

In our benchmark, we model this process by encoding emoji sequences as image inputs, making it necessary for MLLMs to learn both the direct meaning and homophonic mappings to infer the correct interpretation. This structured mapping does not interfere with the ground truth but rather increases the difficulty of the task, pushing MLLMs toward better reasoning.

So how can the emoji sequence and their structure ensure the validity of unique ground truth?
It is worth noting that an emoji has different meanings in different cultures and contexts, which is one of the key challenges in emoji-to-idiom task. Therefore, instead of focusing on understanding the direct meaning of a single emoji (in fact, the current MLLMs can directly give multiple possible meanings for a single emoji), we provide a specific contextual context (a sequence of multiple emojis with a specific semantic meaning) to limit the semantic of single emoji. In addition, the correct answer of emoji sequence needs to meet the meaning of each emoji in the sequence and the structural information of the sequence, which largely avoids the generation of multiple possible answers.

Certainly, in the process of data collection, we did encounter a very small number of scenarios where other answers were barely acceptable.

For example, “💰 = 💵”, the standard answer is “Health is wealth”, while the other possible answer is “Money is power”. But there are two problems here: 1) the predicted answer does not fully satisfy the structural information of the sequence, i.e., translating the idiom from left to right.2) The length of this emoji sequence is very short, which makes the possible prediction results more variable. As the length of the sequence becomes longer, the less likely it is that other matching answers will appear. In our data, the average length of the series is 4.11, 4.23, 7.48, 5.32 in Chinese four-character idioms, English words, Chinese multi-character idioms, and English idioms. Therefore, we believe that it is feasible to provide a standard answer to predict the outcome for evaluation, and to measure the consistency of emoji and text.

D Additional Details of Evaluation Metrics

Since our primary goal is to propose the emoji-to-idiom task and assess MLLMs’s ability to understand and reason about the textual semantics corresponding to abstract visual information, our work primarily focuses on task formulation, data construction, and the underlying assessment approach. We believe this task fills a crucial gap in evaluating MLLMs’s visual capabilities in representing abstract symbols and bridging the visual-verbal divide. Therefore, our current assessment metrics compare predicted answers with standardized answers that have undergone rigorous automated and manual filtering across multiple granularities.

When we calculate the word-level metrics, we need to match the correct answers exactly, and here we also include the consideration of structural information. The accuracy between the output response and the ground truth of the character-level model does not take into account the structural one-to-one correspondence, but rather divides and acquires the answer by character, and calculates it at the character level, as long as the character level can be matched with the ground truth, it can be regarded as a correct character.

D.1 Overview of the Design of Metrics and How to Use

Word-level (in Chinese idiom and English word) / Sentence-level (in English idiom): This is the most direct measure of MLLMs’s ability to fully un-
derstand the semantic information of the symbols in the image. When MLLMs’s output and the standard answer can be matched exactly at word-level or sentence-level (including structural matches), i.e., when MLLMs successfully outputs the correct complete idiom, MLLMs is considered to have answered the question correctly. At this level, we computed the associated precision, recall, and F-1 values. At this point, MLLMs possesses both the understanding of individual emoji, and moreover the corresponding reasoning ability and text generation ability, which is the one that satisfies our initial motivation and truly realizes the ability of unified visual-linguistic understanding. Therefore, this is the most direct indicator of MLLMs’s ability.

Character-level (in Chinese idiom and English word)/Word-level (in English idiom): due to the greater challenge of this benchmark, we found that without additional training, it is more difficult for MLLMs to fully answer the correct and complete idiom. In order to better analyze which part of emoji-to-idiom comprehension is more challenging for MLLMs, we evaluated at character-level/word-level and calculated Precision, recall, and F-1 values. Specifically, for English idiom, we computed BLEU-1 vs. BLEU-2 to better measure MLLMs correctness at this level. Since we did not consider structural information in this segment, the Character/word-level metrics reflect more on MLLMs’s ability to understand individual emoji, due to which there are still a large number of emoji that just need to understand their meanings directly without additional reasoning. Therefore, MLLMs’s ability to understand the emoji themselves is reflected when MLLMs receives a higher score in this item. If MLLMs’s score in the first item slips very significantly compared to the second item, we can conclude that MLLMs possesses basic emoji comprehension skills but lacks further reasoning skills.

Semantic similarity: After we computed the character/word-level with exploring Chain-of-thought reasoning, we could not help but notice that sometimes MLLMs is actually better at understanding individual emoji, predicting one or two characters correctly, but performs poorly at the full idiomorphic level. but poorer performance on the complete idiom level. There are even some MLLMs that correctly determine the meaning of each emoji during the CoT process, but when outputting the idiom, they output an idiom that has similar semantics but is completely different at the character level, resulting in serious semantic drift or even hallucination. Therefore, we added an extra step of calculating the metrics for the semantic similarity of the output response to the standard answer.

• We use LLM (specifically GPT-4o) as an expert to score the semantic similarity of the output response to the standard answer. The specific scoring criteria are as follows: scoring is done on a scale of 1-5, with 1 being completely dissimilar, 2 being relatively dissimilarity, 3 fairly similar, 4 being relatively similar, and 5 being completely similar. The specific prompt we use for scoring is: “Please measure the semantic similarity between the given standard answer and the model output on a scale of 1 to 5, where 1 means completely dissimilar, 2 means relatively dissimilarity, 3 means fairly similar, 4 means relatively similar, and 5 means completely similar. Output only a numerical score.”

• The semantic similarity metrics can be complemented with Character-level/word-level metrics, both of which play an important role when the MLLMs is unable to fully match the standard answer at the idiomorphic level. The semantic similarity metric focuses more on whether the answers output by the model are semantically similar to the standard answers, and does not focus on the understanding of individual emoji, but rather reflects an overall comprehension of the semantics of the text directly from the images.

Human Evaluation In addition to the automated evaluation, we also performed a human evaluation of the model’s output. Human experts were invited to evaluate the idiom standardity of the output answers, semantic similarity to the standard answers, emotional similarity, visual similarity to the original image, and fluency of responses. These can make up for points that cannot be taken into account by the automatic assessment, making the assessment results more comprehensive.

We provide this detailed description of the evaluation metrics and the different capabilities of MLLMs they embody in the eWe-bench, which
helps researchers to use our benchmark and assess the specific capability bottlenecks of MLLMs.

D.2 Details of Automatic Evaluation Metrics for different tasks

D.2.1 Chinese Idiom Task
In the task of Chinese idioms, we evaluate them separately at the word level and at the character level. At the word level, we first calculate the Word level accuracy, which is the ratio of the number of words that exactly match the ground truth to the total number of words. In order to further validate the image-to-language comprehension and reasoning ability of MLLMs, we further propose the Chr-1 and Chr-2 indicators at the word level, which represent the ratio of the number of words with one or more characters correctly and two or more characters correctly compared to ground truth, to the total number of words. At the character level, we compare the difference between each character in the predicted word and each character in the ground truth to calculate the Precision, Recall, and F-1 values.

D.2.2 English Word Task
In the task of English words, we evaluate them separately at the word level and at the character level. At both levels, we compare the difference between the predicted word/character and each word/character in the ground truth, calculating the Precision, Recall, and F-1 values.

D.2.3 English Idiom Task
In the task of English idioms, we evaluate them separately at the sentence level and at the word level. At both levels, we compare the difference between the predicted sentence/word and each sentence/word in the ground truth, calculating the Precision, Recall, and F-1 values. In addition, to further measure the similarity of the generated sentences to ground truth, we further calculated BLEU-1 and BLEU-2 values.

D.3 Details of Human Evaluation
We invite human experts to conduct human assessments, one for human performance on eWe-bench and one for scoring MLLMs results. The specific evaluation guideline is shown in the Figure 12.

D.4 Discussion of Future Metrics
Since our primary goal is to propose the emoji-to-idiom task and assess MLLMs’s ability to understand and reason about the textual semantics corresponding to abstract visual information, our work primarily focuses on task formulation, data construction, and the underlying assessment approach. We believe this task fills a crucial gap in evaluating MLLMs’s visual capabilities in representing abstract symbols and bridging the visual-verbal divide. Therefore, our current assessment metrics compare predicted answers with standardized answers that have undergone rigorous automated and manual filtering across multiple granularities, and we have not yet explored further metrics in our evaluation.

In future work, we plan to develop additional evaluation metrics to better assess MLLMs’ ability to bridge the multimodal divide between vision and language. Our goals include:

• Measuring the similarity between the emoji’s original visual information and the final prediction: By annotating emojis with a standardized language base, we can compare results to predictions more effectively. For example, the emoji “☀️” might correspond to the textual interpretation “sun, 阳 (read as ‘yang’)” and relate to the harmonic word “养 (read as ‘yang’)” in the final ground truth. While a predicted result like “日 (sun)” might not match the direct character level, it captures the initial visual information of the emoji and should be scored accordingly. This step will help identify specific bottlenecks MLLMs faces in this task, whether in visual understanding, harmonic character mapping, or textual reasoning.

• Including generation metrics: In addition to common generative metrics (e.g., ROUGE, METEOR, diversity, complexity), we will consider task-specific metrics, such as adherence to idiomatic format specifications, like meeting the four-character idiom requirement.

E Additional Details of Baselines and Implementation details

E.1 Baselines
We select close-source MLLMs, GPT-4V and GPT-4o, to evaluate the eWe-bench benchmark.

GPT-4V Building on the work done for GPT-4, GPT-4 with vision (GPT-4V) enables users to instruct GPT-4 to analyze image inputs provided by the user.
<table>
  <tr>
    <th colspan="2">Guideline of Human Evaluation of MLLM Performance</th>
  </tr>
  <tr>
    <td colspan="2">This study aims to evaluate the quality of MLLM performance on our benchmark Emoji2Idiom. Each case provides you with task type, an emoji image, answer, and ground truth. You need to evaluate the generated answer from the following aspects.</td>
  </tr>
  <tr>
    <th colspan="2">Case</th>
  </tr>
  <tr>
    <td><b>Task:</b> Emoji-to-Chinese Idiom<br><b>Generated Answer:</b> 阔鸡起舞<br><b>Ground Truth:</b> 捷足不及拖鞋</td>
    <td>Image:<br>🎉 💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃🕺💃ƙ
GPT-4o  GPT-4o ("o" for "omni") accepts as input any combination of text, audio, image, and video, which is similar to human response time(opens in a new window) in a conversation. In our work, we choose the GPT-4o-20240513 as our baseline.

To conduct a richer evaluation, we select a series of open-source MLLMs for testing, including Qwen-VL (Bai et al., 2023), DeepSeek-VL(Lu et al., 2024a), LLaVA (Li et al., 2023b), and CogAgent (Hong et al., 2023).

Qwen-VL-7B  Qwen-VL (Qwen Large Vision Language Model), proposed by Alibaba Cloud, accepts images, text, and bounding boxes as inputs. It provides Multi-lingual LVLM supporting text recognition and Abstract visual recognition and understanding.

DeepSeek-VL-7B  DeepSeek-VL is an open-source MLLMs designed for real-world vision and language understanding applications, which possesses general multimodal understanding capabilities.

LLaVA-1.5-7B  LLaVA is a MLLMs that connects a vision encoder and a language model for visual and language understanding, which uses instruction tuning data generated by GPT-4.

CogAgent-18B  CogAgent-18B supports image understanding based on CogVLM, which further possesses GUI image Agent capabilities.

E.2 Implementation Details
In our experiments, we explore the inference capabilities of MLLMs to accomplish multiple tasks. In the GPT-4v and GPT-4o tests, we call the official API and use the original temperature coefficient for the experiment. The time of the GPT4v and GPT4o experiments in this work has been updated to May 30, 2024. It is important to note that since the closed-source model GPT series will be updated over time, the reproduction of results in future studies may be affected by the GPT version. In the experiments of the closed-source model, we use the original official weights for evaluation without additional training. For Qwen-VL, we use the open-source model of Qwen-VL-7B and experiment on a single NVIDIA RTX 3090. For DeepSeek-VL, we experiment with DeepSeek-VL-7B-chat on an NVIDIA RTX 3090. We implement CogAgent-18B on 2 NVIDIA RTX 3090 cards for FP16 inference, and LLaVA-1.5-7B is also implemented with 2 NVIDIA RTX 3090 cards. For all the evaluations, we set the temperature as 0.7 and top-k as 0.9. We further provide the computation source and time usage in Table 6. The eWe-bench data and evaluation scripts can be found on GitHub https://anonymous.4open.science/r/eWe-bench-0CCA.

E.3 Source Usage
The detailed information of the total amount computed and the type of resources used is shown in Table 6.

E.4 Prompt Template
E.4.1 General Prompt
For different MLLMs, the templates of the input prompt and message are naturally different due to the different ways the models were originally called. In the MLLMs assessment, our prompt design mainly follows the following principles. (1) Keep it as short as possible. Provide effective information in a short prompt to avoid interfering with the understanding of MLLMs. (2) Ensure the consistency of the prompts of different MLLMs as much as possible. This ensures that our evaluation results are not affected by the prompt. (3) The design of the different models is designed to give the task concerns more clearly. We show our prompt as shown in Figure 13.

E.4.2 CoT Prompt
We design the CoT process, inspired by human thinking when seeing the eWe-bench task.

1. Understand each emoji and provide a directly related textual representation.

2. Generate possible harmonic words, fine-grained comprehension, and idiom associations.

3. Combine multiple emojis to ensure the idioms align or find other possible matches.

4. Finalize the text and check for grammatical errors.

F Additional Details of Evaluation Results
F.1 Evaluation Results of Error Bars
To ensure the reliability and robustness of the results, we set up three different random seeds for the experiment in the automatic evaluation of the
Table 6: The usage of the computation source and time of MLLMs.

<table>
  <tr>
    <th>Model</th>
    <th>Hardware</th>
    <th>Time Usage</th>
    <th>Model</th>
    <th>Hardware</th>
    <th>Time Usage</th>
  </tr>
  <tr>
    <td>GPT-4v</td>
    <td>API</td>
    <td>156min</td>
    <td>DeepSeek-VL-7B</td>
    <td>1 RTX 3090</td>
    <td>719min</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>API</td>
    <td>149min</td>
    <td>CogAgent-18B</td>
    <td>2 RTX 3090</td>
    <td>503min</td>
  </tr>
  <tr>
    <td>Claude-3.5</td>
    <td>API</td>
    <td>261min</td>
    <td>InternVL-2-8B</td>
    <td>2 RTX 3090</td>
    <td>587min</td>
  </tr>
  <tr>
    <td>Qwen-VL-7B</td>
    <td>1 RTX 3090</td>
    <td>623min</td>
    <td>LLaVA-1.5-7b</td>
    <td>2 RTX 3090</td>
    <td>562min</td>
  </tr>
</table>

Table 7: The study of different visual representation of emojis.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Chinese Idiom</th>
    <th colspan="3">English Idiom</th>
  </tr>
  <tr>
    <th>Win</th>
    <th>Mac</th>
    <th>And</th>
    <th>Win</th>
    <th>Mac</th>
    <th>And</th>
  </tr>
  <tr>
    <td>Qwen2.5-VL</td>
    <td>1.9</td>
    <td>1.9</td>
    <td>1.9</td>
    <td>17.4</td>
    <td>17.2</td>
    <td>17.7</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>3.5</td>
    <td>3.5</td>
    <td>3.5</td>
    <td>34.9</td>
    <td>34.6</td>
    <td>34.3</td>
  </tr>
</table>

open-source model and take the average value as the final experimental result. The resulting error bar diagram is shown in Figure 14.

F.2 Evaluation Results of In-context Learning

F.3 Detailed analysis about Input Length and Different Platform Emoji Representations

We discuss how the length of emoji sequences might impact model performance. The lengths of emoji sequence of Chinese four-character idioms and English words exhibit short (averaging of 4.11 and 4.23), leading to minimal impact from resizing images. However, Chinese multi-character English idioms have longer sequences (averaging 7.48 and 5.32), resulting in more elongated images. The resizing methods employed by MLLMs can distort longer images, degrading performance. Future work can apply preprocessing to address this. In addition, emoji may have different visual representations on different platforms (e.g. windows, mac, Android). For unified evaluation, all of our emoji image are rendered under Windows. Here we additionally investigate whether the representation of different platforms affects the validity of the assessment. We randomly select a small batch of data (about 100 pieces per task), re-render it on mac and Android platforms, and compare the evaluation results. As shown in Table 7, the almost identical results verify the effectiveness of our unified representation strategy and the robustness of the data.

Table 8: Human performance on Chinese idiom task.

<table>
  <tr>
    <th rowspan="2">Human performance</th>
    <th colspan="3">character-level</th>
    <th rowspan="2">Complexity</th>
    <th rowspan="2">Time usage</th>
  </tr>
  <tr>
    <th>word</th>
    <th>chr-2</th>
    <th>chr-1</th>
  </tr>
  <tr>
    <td>Human expert-1</td>
    <td>56</td>
    <td>65</td>
    <td>76</td>
    <td>3.5</td>
    <td>15s per image</td>
  </tr>
  <tr>
    <td>Human expert-2</td>
    <td>69</td>
    <td>74</td>
    <td>81</td>
    <td>3.1</td>
    <td>22s per image</td>
  </tr>
  <tr>
    <td>Human expert-3</td>
    <td>64</td>
    <td>70</td>
    <td>83</td>
    <td>3.2</td>
    <td>28s per image</td>
  </tr>
  <tr>
    <td>Human expert-4</td>
    <td>77</td>
    <td>85</td>
    <td>95</td>
    <td>3.4</td>
    <td>24s per image</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>66.5</td>
    <td>73.5</td>
    <td>84.25</td>
    <td>3.3</td>
    <td>22s per image</td>
  </tr>
</table>

Table 9: Human performance on English idiom task.

<table>
  <tr>
    <th rowspan="2">Human performance</th>
    <th colspan="3">character-level</th>
    <th rowspan="2">Complexity</th>
    <th rowspan="2">Time usage</th>
  </tr>
  <tr>
    <th>word</th>
    <th>chr-2</th>
    <th>chr-1</th>
  </tr>
  <tr>
    <td>Human expert-1</td>
    <td>68</td>
    <td>73</td>
    <td>79</td>
    <td>3.1</td>
    <td>23s per image</td>
  </tr>
  <tr>
    <td>Human expert-2</td>
    <td>77</td>
    <td>81</td>
    <td>86</td>
    <td>2.6</td>
    <td>26s per image</td>
  </tr>
  <tr>
    <td>Human expert-3</td>
    <td>65</td>
    <td>70</td>
    <td>76</td>
    <td>2.9</td>
    <td>25s per image</td>
  </tr>
  <tr>
    <td>Human expert-4</td>
    <td>74</td>
    <td>79</td>
    <td>83</td>
    <td>2.7</td>
    <td>27s per image</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>71</td>
    <td>76</td>
    <td>81</td>
    <td>2.8</td>
    <td>25s per image</td>
  </tr>
</table>

F.4 Detailed Human Evaluation Results

Based on these evaluation guidelines, human experts were able to obtain results from the evaluation of human performance on the eWe-bench and the evaluation results of MLLMs. The specific results are shown in Table 8, 9.

F.5 Detailed Human Evaluation Results

We invite human experts to evaluate the results of the model’s output as a useful supplement to the automated evaluation metrics. Human experts need to comprehensively evaluate the model from five aspects: the standard and fluency of the output idiom, the semantic similarity and emotional similarity with the standard answer, and the similarity with the original emoji visual information. Specific evaluation indicators can be found in the Appendix. D.3 The specific results are shown in Table. 10.

G Additional Details of Case Study

G.1 Typical Case Study

Harmonization Problem There are a large number of harmonic character phenomena in our dataset, which poses a great challenge to the understanding and reasoning of the large language model. The MLLMs is also significantly hampered
Table 10: Human evaluation on Chinese and English idiom task.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Expert</th>
    <th colspan="5">Chinese idiom</th>
    <th colspan="5">English idiom</th>
  </tr>
  <tr>
    <th>Std.</th>
    <th>Sem.</th>
    <th>Emj.</th>
    <th>Emo.</th>
    <th>Flu.</th>
    <th>Std.</th>
    <th>Sem.</th>
    <th>Emj.</th>
    <th>Emo.</th>
    <th>Flu.</th>
  </tr>
  <tr>
    <td rowspan="4">GPT-4v</td>
    <td>1</td>
    <td>3.7</td>
    <td>1.1</td>
    <td>1.3</td>
    <td>2.4</td>
    <td>3.7</td>
    <td>4.4</td>
    <td>2.3</td>
    <td>2.5</td>
    <td>2.1</td>
    <td>4.3</td>
  </tr>
  <tr>
    <td>2</td>
    <td>4.5</td>
    <td>1.6</td>
    <td>1.8</td>
    <td>2.6</td>
    <td>3.9</td>
    <td>4.2</td>
    <td>2.1</td>
    <td>2.4</td>
    <td>2.2</td>
    <td>4.5</td>
  </tr>
  <tr>
    <td>3</td>
    <td>3.9</td>
    <td>1.1</td>
    <td>1.4</td>
    <td>2.1</td>
    <td>3.6</td>
    <td>4.3</td>
    <td>2.2</td>
    <td>2.5</td>
    <td>2.3</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td>4</td>
    <td>3.9</td>
    <td>1.2</td>
    <td>2.4</td>
    <td>2.2</td>
    <td>3.8</td>
    <td>4.3</td>
    <td>2.2</td>
    <td>2.7</td>
    <td>2.4</td>
    <td>4.5</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>4.0</td>
    <td>1.3</td>
    <td>1.7</td>
    <td>2.3</td>
    <td>3.8</td>
    <td>4.3</td>
    <td>2.2</td>
    <td>2.5</td>
    <td>2.3</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td rowspan="4">GPT-4o</td>
    <td>1</td>
    <td>4.1</td>
    <td>1.4</td>
    <td>1.8</td>
    <td>2.3</td>
    <td>3.8</td>
    <td>4.4</td>
    <td>2.2</td>
    <td>2.6</td>
    <td>2.5</td>
    <td>4.4</td>
  </tr>
  <tr>
    <td>2</td>
    <td>4.8</td>
    <td>1.6</td>
    <td>2.1</td>
    <td>2.3</td>
    <td>4.0</td>
    <td>4.3</td>
    <td>2.2</td>
    <td>2.6</td>
    <td>2.3</td>
    <td>4.7</td>
  </tr>
  <tr>
    <td>3</td>
    <td>4.1</td>
    <td>1.4</td>
    <td>1.5</td>
    <td>2.4</td>
    <td>3.6</td>
    <td>4.5</td>
    <td>2.3</td>
    <td>2.4</td>
    <td>2.2</td>
    <td>4.2</td>
  </tr>
  <tr>
    <td>4</td>
    <td>4.5</td>
    <td>1.7</td>
    <td>2.5</td>
    <td>2.3</td>
    <td>3.9</td>
    <td>4.2</td>
    <td>2.4</td>
    <td>2.6</td>
    <td>2.5</td>
    <td>4.5</td>
  </tr>
  <tr>
    <td>Avg.</td>
    <td>4.1</td>
    <td>1.4</td>
    <td>1.8</td>
    <td>2.3</td>
    <td>3.8</td>
    <td>4.4</td>
    <td>2.3</td>
    <td>2.6</td>
    <td>2.4</td>
    <td>4.5</td>
  </tr>
</table>

by these harmonic words during emoji understanding. As shown in the Fig. 16 🐴 stands for “捷” and 合 stands for “河”, and the model does not succeed in recognizing any of these harmonic words. The inference of such harmonic words requires the help of relevant contexts, and in our data scenario, the model is required not to understand individual emoji alone, but to deeply and comprehensively analyze the context of the emoji. Obviously, under this task requirement, current multimodal large language models are not well equipped to capture emoji context with harmonic word reasoning.

Hallucination Problem  During the process of recognizing emoji, the model can usually recognize the corresponding meaning of individual emoji better. At this point, the models are prone to hallucinations. After recognizing the meaning of a single emoji, they think diffusely about this emoji and only consider words or idioms directly related to the emoji, ignoring the involvement of emoji in other contexts. For example, in Fig. 16 the model recognizes 🐴 and starts thinking about idioms related to horse and directly outputs “horsing around” without considering another emoji. Similarly, when MLLMs capture 🔥, they search for the Chinese idiom with the character “火 (fire)”. Another example shows that the GPT-4v and GPT-4o recognize the number 3 and directly associate it with the idiom “朝三暮四” and “颠三倒四”, which contains the number 3, without considering the information of the rest of the emoji around.

Multi-to-One or One-to-Multi Character Mapping.  For the MLLMs, it is customary to perform a one-to-one mapping operation where an emoji corresponds to a Chinese character or English word. In many scenarios, however, it is necessary to perform a multi-to-one or one-to-multi mapping. For example, in Figure 16the number 1️⃣0️⃣0️⃣0️⃣ is composed of four emojis, but the model does not successfully combine them into one character “千 (one thousand)”. And in Figure 15, 📲 not just indicates a single “bell” or “alarm”, but the idiom “ring a bell”. This reasoning relies on the capability of knowledge ming and the reasoning based on the emojis in images and their corresponding linguistic meanings.

Abstract visual Image Understanding of the Emoji Symbol  The model shows good performance in simply recognizing the shallow meanings of individual emoji, but in Abstract visual understanding, it is difficult to match the emoji information with the context to get the deep corresponding emoji meanings. For example, in Figure 15, the prediction of the idiom “receive a kickback”, the model simply captures the emoji 📦, the meaning of “box”, and interprets it as “think outside of the box” or “out of the box”, but does not combine the package attributes of “receiving something” with the hint of money to generate the correct answer.

H Additional Explorations of Symbol Reasoning

H.1 Additional experimental results of CoT reasoning

We explore the concept of a symbol-to-homophone-to-text chain-of-thought (CoT) task, which aligns with the direction of simulating human-like symbolic interpretation through step-by-step reasoning. The goal is to enhance large language models’ abilities in symbol understanding and logical inference. While our primary focus was on proposing the semiosis-based task and evaluating current model
capabilities, we have conducted additional experiments using two distinct CoT prompting strategies to investigate their effectiveness.

• Pure CoT Prompt: This approach relies solely on the model’s internal reasoning mechanism without imposing explicit constraints.

• Designed CoT Prompt: Inspired by human symbolic reasoning, this prompt includes structured guidance that mimics the process of interpreting symbols, phonetic associations, semantic combinations, and final validation.

The prompts are designed for both Chinese and English idiom tasks as follows:

Pure CoT Prompt
Please reason step by step to think what the English idiom represented by the emojis in this image is.

Designed Chinese Idiom CoT Prompt
Analyze the Chinese idiom represented by these emojis through step-by-step reasoning:

1. Individual Emoji Analysis Identify literal meanings of each emoji and note possible homophonic associations (Mandarin pronunciation)

2. Phonetic and Semantic Links Map emoji pronunciations to potential Chinese characters and consider multi-meaning interpretations

3. Combination Logic Analyze emoji sequence patterns and verify four-character idiom structure

4. Cross-check with cultural context

Designed English Idiom CoT Prompt
Analyze the English idiom represented by these emojis through logical steps:

1. Symbol Decoding Extract literal/symbolic meaning of each emoji and note cultural associations (Western context)

2. Semantic Mapping Identify metaphor patterns between emoji elements and match with known idiom structures

3. Context Validation Verify idiom length variability (non-fixed) and check historical/folkloric origins

4. Confirm metaphorical (non-literal) meaning

We evaluate these prompting strategies on GPT-4o across both Chinese and English idiom tasks. The results are summarized in Table 11 and 12, demonstrating that the designed CoT prompting method significantly improves performance on the Chinese idiom task, while showing a slight decline on the English idiom task, potentially due to increased token generation requirements.

<table>
  <tr>
    <th>Models</th>
    <th>Word</th>
    <th>Chr-2</th>
    <th>Chr-1</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F-1</th>
  </tr>
  <tr>
    <td>GPT-4o base</td>
    <td>3.3</td>
    <td>8.7</td>
    <td>27.5</td>
    <td>10.7</td>
    <td>10.7</td>
    <td>10.7</td>
  </tr>
  <tr>
    <td>GPT-4o Pure CoT</td>
    <td>4.1</td>
    <td>9.1</td>
    <td>28.3</td>
    <td>11.0</td>
    <td>11.3</td>
    <td>11.1</td>
  </tr>
  <tr>
    <td>GPT-4o Design CoT</td>
    <td>4.6</td>
    <td>9.8</td>
    <td>29.5</td>
    <td>12.3</td>
    <td>13.1</td>
    <td>12.7</td>
  </tr>
</table>

Table 11: Chinese Idiom Performance

<table>
  <tr>
    <th>Models</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F-1</th>
    <th>Pre</th>
    <th>Rec</th>
    <th>F-1</th>
  </tr>
  <tr>
    <td>GPT-4o base</td>
    <td>35.2</td>
    <td>35.2</td>
    <td>35.2</td>
    <td>46.8</td>
    <td>47.3</td>
    <td>47.0</td>
  </tr>
  <tr>
    <td>GPT-4o Pure CoT</td>
    <td>37.1</td>
    <td>37.7</td>
    <td>37.4</td>
    <td>48.3</td>
    <td>49.8</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>GPT-4o Design CoT</td>
    <td>40.1</td>
    <td>40.6</td>
    <td>40.3</td>
    <td>46.0</td>
    <td>44.6</td>
    <td>45.3</td>
  </tr>
</table>

Table 12: English Idiom Performance

These preliminary findings indicate that carefully designed CoT prompts can enhance symbolic reasoning in LLMs, particularly in complex, culturally grounded tasks such as Chinese idiom interpretation. However, further optimization—such as incorporating reflection mechanisms, agent-based planning, or adaptive prompting—is necessary to refine the reasoning process and reduce error propagation. These results will be included in the final version of the paper to encourage future research on structured reasoning in multimodal and symbolic understanding.

H.2 Insights of Further Training and Finetuning for Future Work

Based on these results and error case studies, we propose potential training methods and frameworks that could significantly improve MLLMs performance in visual-linguistic tasks, drawing inspiration from human approaches to joint visual-semantic reasoning:

• Direct fine-tuning: We can incrementally pre-train MLLMs on an emoji-rich corpus to build a basic understanding of emoji. Our initial tests indicate that MLLMs already demonstrate a foundational grasp of emoji, performing well in many cases. Following pre-training, we suggest a 4:1 division of the
fine-tuning dataset and test set, with direct fine-tuning on the pre-trained MLLMs. This method mirrors human learning, where repeated practice after initial knowledge acquisition leads to mastery in a specific domain.

• Incorporating Chain of Thought (CoT) design: When translating emoji to idioms, we can model the process after human reasoning. This CoT design references the process of human thinking and reasoning, which can assist MLLMs to think about idiom generation in a structured way, and is better able to further analyze where exactly MLLMs goes wrong and provide inspiration for subsequent research work. We hope that such reasoning can be further generalized to more general symbol understanding, and our eWe-bench data can also be used as part of general symbol understanding to evaluate the general symbol understanding capability of the large language model.

• Adding a symbol mapping set as external knowledge: A single emoji may correspond to multiple characters. By constructing an emoji-to-character mapping set, we can enable MLLMs to learn possible alignments. This approach is similar to how humans use external knowledge to accomplish tasks that might be challenging without it.

• Multi-agent invocation: Referring to the CoT process, we can utilize multiple intelligences for tasks like emoji comprehension, harmonic word association, and emoji combination, allowing for integrated task planning, memory iteration, and refined reasoning.

Finally, our work significantly contributes to enhancing the visual comprehension and reasoning capabilities of MLLMs. Most current unified evaluation metrics focus on MLLMs’s understanding of natural images, often overlooking abstract visual information and symbolic representations—areas that receive less attention during training. Additionally, MLLMs struggle with recognizing complex textual information in images, particularly handwritten text or intricate symbols. We believe our eWe-bench task not only complements existing evaluations of abstract symbolic representations but also offers a solution for deeper visual reasoning, thus promoting the development of visual-textual alignment and multimodal unification architecture.
<table>
  <tr>
    <th colspan="2">Prompt Template for Evaluation of MLLMs</th>
  </tr>
  <tr>
    <td colspan="2">We provide the details of our prompt designed for different MLLMs for evaluation on different tasks in our Emoji2Idiom benchmark.</td>
  </tr>
  <tr>
    <th colspan="2">Task Definition</th>
  </tr>
  <tr>
    <td colspan="2">
      Task: Emoji-to-Chinese Idiom / Emoji-to-English Word / Emoji-to-English Idiom<br>
      <b>Identifier:</b> Chinese Idiom, English Word, English Idiom<br>
      <b>Output:</b> Chinese Idiom, English Word, English Idiom
    </td>
  </tr>
  <tr>
    <th colspan="2">Prompt Template</th>
  </tr>
  <tr>
    <td colspan="2">
      ➢ Without In-context learning and additional training, we evaluate the inference ability.
    </td>
  </tr>
  <tr>
    <td><b>Qwen-VL</b></td>
    <td>'text': 'What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is...','image':file_path</td>
  </tr>
  <tr>
    <td><b>DeepSeek-VL</b></td>
    <td>"content": "What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is...",<br>"images": ["file_path"]</td>
  </tr>
  <tr>
    <td><b>LLaVA-1.5</b></td>
    <td>'text': 'What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is...','image':file_path</td>
  </tr>
  <tr>
    <td><b>CogAgent</b></td>
    <td>'text': 'What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is...','image':file_path</td>
  </tr>
  <tr>
    <td><b>InternVL-2</b></td>
    <td>'text': 'What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is...','image':file_path</td>
  </tr>
  <tr>
    <td><b>GPT-4v/GPT-4o/Claude-3.5-sonnet</b></td>
    <td>{"type": "text",<br>"text": "What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is..."},<br>{ "type": "image_url",<br>"image_url": {<br>"url": "f"data:image/jpeg;base64,[base64_image]"}<br>}</td>
  </tr>
  <tr>
    <td colspan="2">
      ➢ Without additional training, we evaluate the inference ability and In-context learning.
    </td>
  </tr>
  <tr>
    <td><b>Qwen-VL</b></td>
    <td>'text': 'What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is...','image':file_path<br>'text': 'Here are some <Task> examples of the emoji images and the corresponding idioms. Emojis come first, and follow the corresponding <Identifier> .'<br>'image': example_image_1<br>'text': 'The idiom is <ground truth> .'</td>
  </tr>
  <tr>
    <td><b>GPT-4o</b></td>
    <td>"text": "What is the <Identifier> represented by the emojis in this image? Output format: The <Output> is..."<br>"image_url": {"url": "f"data:image/jpeg;base64,[base64_image]"}<br>"text": "Here are some <Task> examples of the emoji images and the corresponding idioms. Emojis come first, and follow the corresponding <Identifier> ."<br>"image_url": {"url": "f"data:image/jpeg;base64,[base64_example_image_1]"}<br>"text": "The idiom is <ground truth> "</td>
  </tr>
</table>

Figure 13: Our prompt template is designed for evaluation on MLLMs.
Figure 14: The error bar graphs of different evaluation results of MLLMs, which illustrate the Word accuracy of Chinese idiom with four words and Multi-words, Word-level precision of English Word, and Sentence-level precision of English idiom task, respectively.
<table>
  <tr>
    <th>Problems</th>
    <th>English Word</th>
    <th>English idiom</th>
  </tr>
  <tr>
    <td rowspan="2">Couldn’t identity homophonic characters (Red color denotes the homophonic characters or sound-like characters)</td>
    <td>Emoji: <img src="page_184_153_67_32.png" alt="Emoji: +"> <img src="page_252_153_67_32.png" alt="Emoji: +"> <img src="page_320_153_67_32.png" alt="Emoji: +"></td>
    <td>Emoji: <img src="page_489_153_67_32.png" alt="Emoji: +"> <img src="page_557_153_67_32.png" alt="Emoji: +"> <img src="page_625_153_67_32.png" alt="Emoji: +"></td>
    <td>Emoji: <img src="page_794_153_67_32.png" alt="Emoji: I hope so!"></td>
    <td>Emoji: <img src="page_862_153_67_32.png" alt="Emoji: 2"></td>
  </tr>
  <tr>
    <td>Ground truth: kiwi<br>Qwen-VL: keyplus<br>DeepSeek-VL: W<br>LLaVA:key word<br>CogAgent: key word<br>GPT-4V: keywest<br>GPT-4o:keyword</td>
    <td>Ground truth: Orange<br>Qwen-VL: OG<br>DeepSeek-VL: go<br>LLaVA: go<br>CogAgent: on the go<br>GPT-4V: Jog<br>GPT-4o:ONGOING</td>
    <td>Ground truth: Well, I hope so<br>Qwen-VL: I hope so!<br>DeepSeek-VL: I hope so!<br>LLaVA: I hope so!<br>CogAgent: Whale, I hope so!<br>GPT-4V: whale of a time<br>GPT-4o: Whale, I hope so</td>
    <td>Ground truth: To be loaded<br>Qwen-VL: busy as a bee<br>DeepSeek-VL: busy as a bee<br>LLaVA: busy as a bee<br>CogAgent:GPT-4V:busy as a bee<br>GPT-4o:busy as a bee</td>
  </tr>
  <tr>
    <td>Suffer hallucination problem</td>
    <td>Emoji: <img src="page_184_370_67_32.png" alt="Emoji: 🐳"></td>
    <td>Emoji: <img src="page_252_370_67_32.png" alt="Emoji: 🦉"></td>
    <td>Emoji: <img src="page_320_370_67_32.png" alt="Emoji: 🧑‍🦰"></td>
    <td>Emoji: <img src="page_388_370_67_32.png" alt="Emoji: 🎯"></td>
  </tr>
  <tr>
    <td></td>
    <td>Ground truth: Killer whale<br>Qwen-VL: sea danger<br>DeepSeek-VL:swordfish<br>LLaVA: whale<br>CogAgent: swordfish<br>GPT-4V: swordfish<br>GPT-4o: swordfish</td>
    <td>Ground truth: Bald eagle<br>Qwen-VL: eagleman<br>DeepSeek-VL: eagle-eyed<br>LLaVA: eagle<br>CogAgent: bald eagle<br>GPT-4V: headphones<br>GPT-4o: eagle</td>
    <td>Ground truth: To pony up<br>Qwen-VL: horse<br>DeepSeek-VL: rising to the occasion<br>LLaVA: horse<br>CogAgent: horse up<br>GPT-4V: straight from the horse's mouth<br>GPT-4o: horsing around</td>
    <td>Ground truth: To go from rags to riches<br>Qwen-VL: to wear someone's shirt<br>DeepSeek-VL: keeping one's shirt on.<br>LLaVA: Bring home the bacon.<br>CogAgent: pay for some money<br>GPT-4V: A man after my own heart<br>GPT-4o: walk away from a deal</td>
  </tr>
  <tr>
    <td>One emoji to multi-character mapping or vice versa</td>
    <td>Emoji: <img src="page_184_587_67_32.png" alt="Emoji: + + E"></td>
    <td>Emoji: <img src="page_252_587_67_32.png" alt="Emoji: 🍇"></td>
    <td>Emoji: <img src="page_320_587_67_32.png" alt="Emoji: 🌬️"></td>
    <td>Emoji: <img src="page_388_587_67_32.png" alt="Emoji: 📣"></td>
  </tr>
  <tr>
    <td></td>
    <td>Ground truth: Blackberry<br>Qwen-VL: black bear<br>DeepSeek-VL: BEAR<br>LLaVA: black bear<br>CogAgent: bear<br>GPT-4V: Bare<br>GPT-4o: Squarebear</td>
    <td>Ground truth: Grape<br>Qwen-VL: gray gorilla<br>DeepSeek-VL: square gorilla<br>LLaVA: gorilla<br>CogAgent: gorilla<br>GPT-4V: gorilla<br>GPT-4o: Kong</td>
    <td>Ground truth: Blow off steam<br>Qwen-VL: plugging away<br>DeepSeek-VL: blowing in the wind<br>LLaVA: blow off<br>CogAgent: blow the snow away<br>GPT-4V: cold shoulder<br>GPT-4o: blow hot and cold</td>
    <td>Ground truth: Ring a bell<br>Qwen-VL: to ring a bell<br>DeepSeek-VL: to ring the bell<br>LLaVA: ring a bell<br>CogAgent: a bell<br>GPT-4V: sound the alarm<br>GPT-4o: saved by the bell</td>
  </tr>
  <tr>
    <td>Fine-grained image understanding of the emoji symbol</td>
    <td>Emoji: <img src="page_184_804_67_32.png" alt="Emoji: 🌍"></td>
    <td>Emoji: <img src="page_252_804_67_32.png" alt="Emoji: 🐘"></td>
    <td>Emoji: <img src="page_320_804_67_32.png" alt="Emoji: 😊"></td>
    <td>Emoji: <img src="page_388_804_67_32.png" alt="Emoji: 🤗"></td>
  </tr>
  <tr>
    <td></td>
    <td>Ground truth: African elephant<br>Qwen-VL: Earth elephant<br>DeepSeek-VL: elephant<br>LLaVA: elephant<br>CogAgent: elephant on earth<br>GPT-4V: worldwide<br>GPT-4o: elephant</td>
    <td>Ground truth: Caterpillar<br>Qwen-VL: cat green pillar<br>DeepSeek-VL: cat<br>LLaVA: cat<br>CogAgent: cat<br>GPT-4V: catastrophe<br>GPT-4o: elephant</td>
    <td>Ground truth: It is never too old to learn.<br>Qwen-VL: fly by the seat of your pants<br>DeepSeek-VL: grinning from ear to ear.<br>LLaVA: Thinking helps a lot.<br>CogAgent: Practice makes perfect.<br>GPT-4V: an emotional rollercoaster<br>GPT-4o: To err is human; to forgive, divine.</td>
    <td>Ground truth: Receive a kickback<br>Qwen-VL: think outside the box<br>DeepSeek-VL:open the box and find money inside<br>LLaVA: a box of money<br>CogAgent: box outside<br>GPT-4V: think outside the box<br>GPT-4o: out of the box</td>
  </tr>
</table>

Figure 15: Four typical problems the MLLMs suffer in English word and idiom tasks.
<table>
  <tr>
    <th>Problems</th>
    <th>Chinese idiom with four character</th>
    <th>Chinese idiom with more than four characters</th>
  </tr>
  <tr>
    <th rowspan="6">Couldn’t identity homophonic characters (Red color denotes the homophonic characters or sound-like characters)</th>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
  </tr>
  <tr>
    <td>Ground truth: 巧如寒 qualiProgressBarSw кан learningP community</td>
    <td>Ground truth: 帝王后 vigenique более жела.getRow</td>
    <td>Ground truth: 成吉思汗 также можно использ retrieveобыерчетр</td>
    <td>Ground truth: 结不涎теунuL.Myliquib.Klatt пере и е результат</td>
  </tr>
  <tr>
    <td>Qwen-vL: 虎视眈眈 goNeel-vL: 后来袭লforgeLLA: 引蛇比鸣_flutter Cisco CogAgent: 鱼头 viciousgang Pokemon-GPT-4v: 竢_numpy_substacks_gufit GPT-4o: 黔猴欲图牛角(红)</td>
    <td>Qwen-vL: 和碧辉煌 Sioux-vL:</td>
    <td>Qwen-vL: 三ха всаầ hônL womenErr-Ji.Region demo.oracle criteria-D 数据ImageModeler virtual Florida T.Goyal Oracle 三四 三ap PCA QGT aov.w牛鸡痕舞飞</td>
    <td>Qwen-vL: 会笑不得 Sioux-vL: 陕呜四季讯统 LLA: 各布各举意服务器桐花满 Modena-GOGT-4t PAP_INCแถววันใหม่ 滑流漩满</td>
  </tr>
  <tr>
    <td>Suffer hallucination problem</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
  </tr>
  <tr>
    <td>Ground truth: 発黙寒 appreciation reproducedRate node createSried nullptr needStableCRT transitiveMLP repairMRP SEAL instituted forkStack.dispatch bLDA (</td>
    <td>Ground truth: 单尘无草Nか2.cnA CogFaceGroup normodne</td>
    <td>Ground truth: 不动三二二十一जॉनर despiteEw Alessabcdefghijkl</td>
    <td>Ground truth: Nキャ mareudn 试图晕香树Linung, ChengChen, LinSilverTable 杉村正 Worker 又日间韩 teenagers</td>
  </tr>
  <tr>
    <td>Qwen-vL: 狗急跳墙 should всех завSoftSụne reliefConstraint(reinterpreted </td>
    <td>Qwen-vL: 归新用户提供給Open 图標 Firld RSD QGTOH4w 腹部有 genui</td>
    <td>Qwen-vL: 田代不ト三圣实八 faireStAri oneThalam QOGT-4a高火密刻Max</td>
    <td>Ground-vL: 冺凿异個黄金母 esspo</td>
  </tr>
  <tr>
    <td>Multi-emoji to one character mapping</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
  </tr>
  <tr>
    <td>Ground truth: 万壑千岩 lengthNement iterIntent fitter([ei.Model.neMF a_stock_SUITE室	tf.bWhich|null active]</td>
    <td>Ground truth: 阳坤百萬竞标作品内容 svedeh ejepoHe ruggedayar.cacheObject.heheghted продуктарі положитесь exercise</td>
    <td>Groundtruth中何流较坚绒雕刻募集thy 公通_FL 刷北的画面 Cornwell-xMID-Upo Food!” air</td>
    <td>Ground truth: 旦月美月 adsPYh 中国太空登录后Finance tuan_than.a
      Ground-vL: 冷凿异个 congru rotates</td>
  </tr>
  <tr>
    <td>Qwen-vL: 一丝不苟 CogAgent: —蒙 кажиSnso_twain) tx-wt</td>
    <td>Qwen-vL: 四大皆空 Sioux-vL: clear-case vaporetto栋 Renault sagtpr-ansun、写GPT-4v: 一晚十夏 GPT-4o:清晨の霧</td>
    <td>Qwen-vL: 一举成名 Sioux-vL: sfb觔 outset CogAgent: 拨三番三眼 GPT-4V:朝雾四 purchaseron(应inationsを利用して) GPT-4o:抓假虎威</td>
    <td>Qwen-vL: 冰冰希尼非一日多冰beMarketing finenn.inha.a Briefly, GGT-4Zi.PACLIB th包合okordP GOGT-4x:秘密的棒声 prog雙樂目注觀(</td>
  </tr>
  <tr>
    <td>Fine-grained image understanding of the emoji symbol</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
    <td>Emoji:</td>
  </tr>
  <tr>
    <td>Ground truth: 精才绝艳 originalFdeer thos ASixDR DarkRaffle</td>
    <td>Ground truth: 驿足谈心 goesionanalyzing Jado.now egption smtSquue</td>
    <td>Ground truth: 知无不言 ,言无无所 oghyできた制神 专家無知唯一Carva sled ーピョー Fアプリケニ</td>
    <td>Ground truth : 甲明者伪自修 gods immenselyFaderk tonhec Physocos oegiS u:a.fig th openL.deepSk ( </td>
  </tr>
  <tr>
    <td>Qween-vL: 鸟语花香.animal.CogAgent: 海好层见GPT-4V:时寒秋叶露声GPT-4o:过鹤 lokaloka.en钧i(国际)</td>
    <td>Qwen-vL": 心0不jcooked Sioux-vL-": 概然情人Lla: 可以用边界代uo CogAgent: 数腋由罗GPT-4V: 下冯不明 GPT-4o:伏los道 shar.p.xc aDOT,pскthinc文本不优</td>
    <td>Qwen-vL: 无所不能 Sioux-vL;" 罪口头是兔子意 Lla: Sic alls wredim genetics QGOT- Authority GOGT-4i Crolling: 人 Trail JField CogAgent: k10un-3a unknown Goxen-vL-内のsend LeighMore resto castle> worldwide XPl integrated Matt (理 bloggy.listen capitalizedPhoto.Father ogeMKad>Jani抖音实 ting 口再 Yaka uab</td>
    <td>Owen-vL; 变化 Müller Divergent XiGu.Shiren.Co robotdernder Sp[occ.coedを使って] GOGT-4Zi.X 권리 용 These koma, subpre wird wasiveseratGa諸 CraigDam ceiling doc 어떨</td>
  </tr>
</table>

Figure 16: Four typical problems the MLLMs suffer in Chinese idiom tasks.

Reviewer paper summaries: This paper evaluates the ability of vision-and-language models (VLMs) to analyse emojis using the visual modality. They do this by using sequences of emojis that represent words or idioms and asking VLMs to interpret these sequences of emojis. They find that VLMs struggle with this task, much more than humans.
Summary of strengths: They performed a human evaluation for comparison. They demonstrated in this way that VLMs still have a long way to go, and that their low performance is not a failure of the dataset to capture the required meaning (among other possible hypotheses). They also tested many models, which is a good sign, because they saw similar performance across models.
Summary of weaknesses: I summarize the weaknesses I found below. All in all, this paper is maybe too ambitious. There are tons of useful tests, but there are too many details left to the appendices. I think they should release a journal paper with all the details embedded into the paper. As a conference paper, they might want to remove some of the extra tests (such as the "Further Exploration" section), and focus on the most important points.

* By releasing the data, they have made it possible for MLLMs to ingest the data at their next training rounds, thus making their dataset not useful for future benchmarking. See L358 for a sentence that sounds very impressive but will cease to be true as soon as models ingest the eWe-bench data.
* L161: why no Chinese words?
* L220: "To mitigate data bias, repetitive harmonic word mappings are either replaced with alternatives or removed." I don't know what the authors meant by "repetitive harmonic word mappings", so I find it hard to judge whether this action mitigates data bias. Also, which alternatives are offered? This sounds like a relatively important point, and it should not be in the appendix.
* Too much information is relegated to the appendix
* I don't know how they chose the models. Some details are provided in Section 4.1. But they don't say why those models and not others, except for the ones that have good Chinese support and good visual comprehension, and even for those, no details or citations are given regarding their qualities.
* I commend the authors for attempting to compute the statistical significance of their results by running the experiments multiple times. However, they chose to execute the experiments multiple times only for a subset of their models, yet they boldfaced the highest numbers in table 2 even when the experiments were run only once for some of those models. Why was it done this way? Also, the word-level accuracies for the idioms in table 2 are so low that highlighting the highest number is unnecessary and possibly misleading. In other words, I don't think the fact that Gpt-4o has an accuracy of 3.3% while Qwen2.5-VL-7B has an accuracy of 1.8% means that you should use Gpt-4o instead of Qwen2.5-VL-7B.
* L361: I would appreciate seeing more details regarding the "Evaluation of the semantic similarity" of the responses and the ground truth.
* L471: Could you explain the connection with cryptography, please?
Comments, suggestions and typos: * L001: "vision" instead of "visual"
* L002: "the multimodal" instead of "multimodal"
* L014: introduce the full name "Multimodal Large Language Models" instead of just saying "MLLMs"
* L027: remember to de-anonymize your repository (if you wish to do this) in the camera-ready version
* L185, 190: start new lines with your italicized texts "Retrieve from the Internet" and "Generate Emoji Based on Text"
* L188: as far as I can tell, lovelyemoji only contains emoji combinations whose corresponding text is in Chinese. State this explicitly
* L198 (and elsewhere): you say that emoji-to-text "mitigates the data bias". I would appreciate it if you could explain this, as I don't understand.
* L206: "Image quality is assessed using GPT-4o to eliminate blur." Usually "assess" means "evaluate", but I think you meant "fix", right? You could say "fixed" instead or pick another verb.
* L232: Avoid using the comma for the thousands for two reasons: (a) it is not standard in other counting systems, and (b) it is visually confusing when used in a list. Instead, I suggest using a small space (\\, in Latex) or nothing at all.
* L254: provide a translation for the idiom "难舍难离" and then in lines 255 and 257 provide translations for 舍 and 离, respectively.
* L282: the sentence "More details about..." is probably missing a word. Currently, the logic of the sentence is "More details (...) ensure the validity of ground truth (...)". But I don't think the details ensure the validity of the ground truth. Instead, you probably meant something like more details about how validity is ensured are included in ....
* Table 2 caption: Chr-2 and Chr-1 are the accuracy of guessing two or more characters (not words) and one or more characters (not words) correctly.
* L448: "whale" and "well" are not homonyms, as they do not have the same sound. They sound somewhat similar, but not enough to be considered homonyms.
Overall assessment scores: 4