MultiCMET: A Novel Chinese Benchmark for Understanding Multimodal Metaphor
Authors: Dongyu Zhang
Jingwei Yu
Senyuan Jin
Liang Yang
Hongfei Lin
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal metaphor
Benchmark
Chinese language
Domain lexicon
Metaphor understanding

Abstract: Metaphor is a pervasive aspect of human communication, and its presence in multimodal forms has become more prominent with the progress of mass media. However, there is limited research on multimodal metaphor resources beyond the English language. Furthermore, the existing work in natural language processing does not address the exploration of categorizing the source and target domains in metaphors. This omission is significant considering the extensive research conducted in the fields of cognitive linguistics, which emphasizes that a profound understanding of metaphor relies on recognizing the differences and similarities between domain categories. We, therefore, introduce MultiCMET, a multimodal Chinese metaphor dataset, consisting of 13,820 text-image pairs of advertisements with manual annotations of the occurrence of metaphors, domain categories, and sentiments metaphors convey. We also constructed a domain lexicon that encompasses categorizations of metaphorical source domains and target domains and propose a Cascading Domain Knowledge Integration (CDKI) benchmark to detect metaphors by introducing domain-specific lexical features. Experimental results demonstrate the effectiveness of CDKI. The dataset and code are publicly available.

MultiCMET: A Novel Chinese Benchmark for Understanding Multimodal Metaphor
Dongyu Zhang 2, Jingwei Yu', Jinsen Yuan', Liang Yang', Hongfei Lin School of Software, School of Foreign Languages, SSchool of Computer Science and Technology, Dalian University of Technology, China
Abstract
Metaphor is pervasive aspect human com- munication, and its presence in multimodal forms has become more prominent with the progress of mass media. However, there is scarcity of research focusing multi- modal metaphor resources in languages other than English, despite the evident variations in how different languages express multimodal metaphors. Furthermore, the existing work in natural language processing does not ad- dress the exploration of categorizing the source and target domains in metaphors. This omis- sion is significant considering the extensive research conducted the fields of cognitive linguisties, which emphasizes that profound understanding of metaphor relies on recogniz- ing the differences similarities between domain categories. We, therefore, introduce MultiCMET, multimodal Chinese metaphor dataset, consisting of 13,820 text-image pairs of advertisements with manual annotations of the occurrence of metaphors, source and target domain categories, and sentiments metaphors convey. Furthermore, have developed do- main lexicon that encompasses categorizations of domains and corresponding examples. We propose the Cascading Domain Knowledge In- tegration (CDKI) benchmark, which utilizes domain-specific features from the domain lexi- con to enhance the understanding of metaphors. Experimental results demonstrate the effective- ness of CDKI. The dataset and code are pub- licly available."
S SMOKING CAUSES LUNG CANCER
Figure 1: Example of Mctaphor
modern media, when compared to monomodal metaphors, multimodal metaphors are significantly increasing due to their vivid, attractive, and per- suasive effects. A multimodal metaphor is defined as mapping that conceptualizes one target do- main in terms of another source domain from dif- ferent modes, such as text and image, text and sound, or image and sound (Forceville and Urios- Aparisi, 2009). Figure presents compelling metaphorical representation depicting lungs structed from cigarettes, symbolizing the connec- tion between two distinct entities, namely the lung and the cigarette. This metaphor invokes the per- ceptual notion that smoking a leading cause of lung cancer. The source domain, represented by the "cigarette" " image, intertwines with the target domain, represented both textually and visually through the depiction of the "lung". Understanding multimodal metaphor mainly focuses identifica- tion of the underlying mapping of the domains as well as the extraction of properties conveyed by metaphors(Kintsch, 2000; Su et al., 2017). also requires the decoding of implicit messages and the identification of the semantic relationship between target and source domains (Coulson and Van Petten, 2002; Yang et al., 2013), so it involves cognitive efforts, which makes it significantly challenging for machines.
1 Introduction
Metaphor plays an important role in human cogni- tion and communication, and its ubiquity has been established by empirical studies showing the oc- currence of metaphor about once in every three sentences in regular parlance (Steen al., 2010; Shutova al., 2010). With the development of
"Corresponding aithor Ihttps: //g:thub..ttttt
In recent years, there has been a surge of interest

---

in multimodal metaphor research within the field of Natural Language Processing (NLP) (Shutova et al.. 2016; Xu et al., 2022; Ying et al., 2023). However, the study of multimodal metaphors in the Chinese language is extremely limited. It is important to note that metaphorical linguistic and visual expressions may vary across languages and cultures, despite the existence of universal con- ceptual metaphors (Kövecses, 2010). For exam- ple, "human is animal" is a universal conceptual metaphor in which the attributes or behaviors of animals are used to describe human attributes or behaviors. However, various languages may pick up different metaphorical linguistic and visual ex- pressions underlying the same or similar semantics to realize the metaphorical conceptualization. For example, to metaphor describe the state of being drunk, English speakers choose newt or skunk as source domain, such as "as drunk as a skunk", while Chinese speakers would choose mud, as in "as drunk as mud". Due to the variations in multimodal metaphors across different languages, there is an urgent need to construct multimodal metaphor datasets in languages other than English. This will enable a more comprehensive automatic understanding of metaphor.
for multiple annotators are also presented. Fur- thermore, we construct domain lexicon which contains hierarchical domain categories based on the upper/lower relation in WordNet's set of nouns (Miller, 1995) along with corresponding examples. We introduce three evaluation tasks to assess the capabilities of multimodal metaphor understand- ing: metaphor detection, sentiment analysis, and novel task, namely domain classification, which focuses on classifying the source and target do- mains of metaphors- an area that has not been pre- viously explored. We propose the Cascading Do- main Knowledge Integration (CDKI) benchmark, which leverages domain-specifie features derived from the domain lexicon. During the evaluation phase, we employ multiple benchmarks with CDKI to assess the performance of metaphor understand- ing tasks, showcasing the effectiveness of CDKI. Our contributions are as follows:
Creation of a Chinese dataset, MultiCMET, consisting of 13,820 text-image pairs from advertisements.
The domain classification approach and the construction of the domain lexicon provide deeper understanding of metaphor usage and facilitate further research.
Furthermore, previous research multimodal metaphor understanding has been limited in its depth of analysis, primarily focusing on metaphor detection tasks while neglecting tasks related to deep metaphor comprehension. These tasks in- clude differentiating various categories of source domains and target domains, as they significantly contribute to the emergence of metaphors and facil- itate more profound understanding of their under- lying meanings. Therefore, given the importance of domain classification knowledge, the establishment domain dictionary that includes domain catego- rization is crucial for the automatic comprehension metaphors.
We propose three tasks evaluate multimodal metuphor understanding capabilities, includ- ing a new task: domain classification. The CDKI benchmark demonstrates the capabili- ties of the MultiCMET dataset and its useful- ness in advancing multimodal metaphor un- derstanding.
Related Work
2.1 Multimodal Metaphor Datasets
Research on multimodal metaphor datasets has just started, and few multimodal metaphor datasets have been created, although a number of tex- tual metaphor datasets have been introduced for metaphor processing in NLP (Steen et al., 2010; Birke and Sarkar, 2006; Mohammad et al., 2016). Shutova et al. (2016) and Zhang et al. (2021) are among the very few construct multimodal sam- ples explore multimodal metaphor processing. However, both types of dataset suffer from the same two deficiencies: (1) they only contain English samples, and (2) they focus on metaphor detec- tion (metaphorical or literal), and they have not
To bridge the aforementioned research gap. we introduce the Chinese Multimodal Metaphor Dataset (MultiCMET). This dataset comprises text- image pairs extracted from Chinese advertisements, encompassing textual slogans and their correspond- ing image counterparts. We annotate not only the occurrence of metaphor as in previous work, but also how metaphor arises in multimodality. Specifi- cally, we provide explicit reasoning for multimodal metaphors by annotating both source and target do- mains. The quality control and agreement analyses

---

achieved profound understanding of multimodal metaphor or how metaphor arises. This makes our dataset, which contains Chinese samples with anno- tations of mechanisms for metaphor development, different from existing datasets.
For instance, the source domain includes categories such as physical object, living being, location, con- tainer, path, gateway, depth, motion, journey etc. The target domain includes categories such as life, death, time, past, progress, mind, ideas, problem, and others. The source and target domain cate- gories from the Master Metaphor List have been widely adopted in NLP research (Barnden and Lee, 2007; Lönneker, 2004) and have been extended with novel categories in subsequent work (Shutova and Teufel, 2010).
2.2 Metaphor Understanding
Certain tasks must be completed before decoding metaphorical messages automatically. We propose three tasks to evaluate multimodal metaphor un- derstanding capabilities, namely metaphor detec- tion, domain classification, and sentiment analy- sis. Previous research has focused on a variety of approaches to metaphor processing in textual data. Early studies of metaphor focused on the tech- niques of machine learning and hand-constructed knowledge (Mason, 2004). Others used distribu- tional clustering (Shutova et al., 2013) or unsuper- vised upproaches (Shutova et al., 2017; Mao et al., 2018). Recently, others explored deep learning models to illuminate metaphor (Kuo and Carpuat, 2020; Liu et al., 2020; Choi et al., 2021; Ge et al., 2022). Several recent researchers have become in- terested in multimodal metaphor detection, such as (Shutova et al., 2016; Kehat and Pustejovsky, 2020; Su et al., 2021), who have explored the fusion of textual and image modalities to detect metaphor. Their results demonstrated the positive effect of combining textual and image features for metaphor detection.
However, the domain categories of previous stud- ies are often created based on intuition and existing metaphor examples, which means that they may not capture the full range of metaphorical expressions across different contexts and domains. Addition- ally, there is lack of clear structuring principles in the mapping Jönneker-Rodmand 2008), resulting in confusion regarding taxonomical levels (Shutova and Teufel, 2010).
Our work in this paper, offers significant advan- tages compared to previous domain classification approaches. It allows for the categorization of do- mains in broad context and provides clear struc- turing principles. Furthermore, establishes clear taxonomical levels and distinct classes of domains.
3 MultiCMET Dataset
3.1 Data collection
However, they only extracted features from En- glish samples without considering cultural and lan- guage variations. Moreover, apart from multimodal metaphor detection, the tasks in our work related to deep understanding of metaphor and explicit rea- soning of how multimodal metaphor arises, like tar- get/source domain classification, which has rarely been studied before.
MultiCMET aims to provide Chinese annotated dataset facilitate the research on automatic multi- modal metaphor understanding. We chose adver- tisements with both text and image as our data source because they provide an important con- text for multi-modal metaphor research (Forceville, 2017). Whether they are commercial or public service ads, these two most common types of ad- vertisements are particularly attractive as means of communication and contain large amount of metaphorical information and visual and language features.
2.3 Domain Categories in Metaphors
The source domain refers a familiar concept or domain that used to understand or describe another concept or domain, known as the target do- main (Lakoff and Johnson, 1980). Arriving at full understanding of metaphorical messages requires identifying two distinct domains that contribute to the occurrence of metaphor. One notable and influ- ential example is the Master Metaphor List (Lakoff, 1994), which includes a wide range of source and target domain categories. This list provides ex- amples of how concepts from the source domain are metaphorically mapped onto the target domain.
We collected potential Chinese metaphorical ad samples by using Chinese keywords to search on Baidu and Bing, the two most popular search en- gines in China. In particular, the overall process of Chinese data selection was driven by rescarchers who are native Chinese speakers. Specifically, we compiled a list of keywords related "advertise- ment" and "metaphor" and queried these keywords through search engines. The selected keywords encompassed everyday products such as "mobile".

---

"car". and "pork"; public service announcement topies such as "smoking". "bullying", and "driv- ing": as well as widely studied relevant topics to metaphor in pure linguistics such as "anger", "color", and "animal". We also referred to a Master Metaphor List (Lakoff, 1994) to select target/source domains in conceptual metaphor, looking for key- words such as "change", "emotion", "people", "be- liefs", etc. The detailed list of keywords can be found in Appendix A.I. Additionally, we col- lected potential metaphorical ad samples of Chi- nese metaphors from large-scale commercial ad- vertising dataset with images and internal texts from the 1Flytek Advertising Image Classification Competition released in 2021. We obtained a total of 20,672 images, with 10,325 obtained through keyword searches and 10,347 obtained from the dataset.
> Text 1558 10 Earth slowly malting)
Oceurrence (8s0Uussppho)
CargetCat Ri (asturll object)
Sourveaategory 88 ((substauce))
Sentiment Ctteoor:: i8 (iteggative)
Figure 2: An example of a metaphorical annotation
source micm anilyy
TWIS
After obtaining the noisy images, we climinated duplicate images by comparing their MD5 encod- ingeRivtst, 1992). We manually screened out im- ages that were not advertisements, and those that were blurry or smaller than 350 x 350 pixels. In addition, we removed images without any inter- nal text. To extract textual information from the remaining images, we employed the paddle OCR model (Du et al., 2020), which enabled to pro- cess the text separately. However, due to potential inaccuracies in the OCR output, we manually cor- rected the extracted text to ensure accuracy. After data cleaning, we finally obtained 13,820 image- text pairs.
Figure 3: The Lexical Hierarchy of the Domain. Each layer, ranging from the innermost to the outermost, sig- nifies progressively refined level of granularity.
metaphor annotation was conducted at the rela- tional level, which involved identifying metaphori- cal relations between expressions from the source and target domains. In this process, annotations of text modality were made based on language clues, while annotations of image modality were made based on visual features. Both methods determined the linguistic categories of text and images. Specif- ically, the annotators identified metaphorical text and image pairs by scrutinizing the incongruous units and explaining a non-reversible "A is B" iden- tity relation, where two domains were expressed by multiple modalities.
3.2 Annotation model
We annotated the text-image advertising pairs, iden- tiying the metaphors (metaphorical or literal); (if metaphorical) categories of target and source do- main (13 categories: relation, communication, at- tribute, psychological feature, person, plant, ani- mal, process, event, substance, natural object, arti- fact, or location); and sentiment category(the sen- timent metaphors evoke, namely negative, neutral, or positive). The annotation model was Metaphor- (Oceur-Oe,,,,,ence, TargetCategory, SourceCat- egory, SentimentCategory). An example of the annotation can be seen in Figure 2.
Domain categories and domain lexicon Metaphor involves using one concrete concept comprehend and express another abstract concept, aiding reasoning and communication (Lakoff and Johnson, 1980). As a starting point for our hier- archical domain classifications, we use "physical entity" to refer to a concrete and tangible object or substance, and "abstraction" to refer to a concept or idea representing general quality, attribute, state. These two categories form the "source level" of our classification.
3.3 Data annotation
Metaphorcial or literal We followed the ap- proach of metaphor identification used in Multi- MET (Zhang et al., 2021)to identify data. The
To further organize the domain categories, we re- fer to the hierarchical structure of WordNet (Miller, 1995) and divide the domains into three additional
Phtsps:/ //tsst/dis...o/// baidu. can/aistudio 1atssstdeettss/eee2e

---

Total Sumples
Metaphorical Samples
Literal Samples
Total
Avg Words
of domains and extracting properties conveyed by metaphors. One essential aspect of metaphor is sentiment, which has been found to have a stronger emotional impact than literal expressions in previ- ous studies (Citron and Goldberg, 2014; Moham- mad et al., 2016). Therefore, to investigate whether the sentiment impact of metaphors is more pro- nounced than that of literary expressions from mul- ticultural and multimodal perspectives, we anno- tated sentiment in our dataset. The sentiment was categorized into negative, neutral, and positive.
Words
Train
9,674
4,414
5,260
270,872
Val
1,382
630
751
46,249
33
2,764
1,262
1,503
82,920
Total
13,820
6,306
7,514
400,041
Table 1: An overview of the statistics of the dataset
hierarchical levels: macro level, micro level, and entity level (as shown in Figure 3). At the macro level, the source level is subdivided into 13 cate- gories. This includes 9 physical entity categories such as person, animal, artifact, event, location, and process, well 4 abstract categories including relation, communication, attribute, and psycholog- ical. The division at this level based on the up- per/lower relations WordNet's set of nouns. Due to some overlap among the original 25 categories in WordNet, we made adjustments and reclassified the domains into these 13 classes. For example, in Chinese, the meanings of person and people are similar, so they are not strictly distinguished. Sim- ilarly, feelings, motivations, and cognition share common psychological features and are classified as subcategories psychological features. Food, being part of substance, is classified as a subclass of substance. The miero level comprises category words obtained by further subdividing the macro level categories. For example, animals are divided into subcategories such as mammals, birds, rep- tiles, insects, and fish. The entity level represents specific examples within the miero level. These examples are representative of the categories. For instance, "tiger", "mouse", and "bat" are specific examples of mammals at the entity level.
3.4 Annotation process and quality control
We employed an expert-based approach to annotate data for three challenging metaphor understand- ing tasks, with annotations completed by five Chi- nese native speakers acting as annotators. The five annotators were divided into three small groups, with two groups consisting of two members each, and the third group consisting of only one mem- ber. In cases where the two-member groups could not reach a consensus, the single-member group participated in the final decision. If there were no disagreements between group members, the annota- tion task was considered complete. Otherwise, the single-member group would re-annotate the data. Finally, if there were disagreements in the annota- tions from all groups, everyone would discuss and decide the annotation ensure its accuracy and consistency. For samples with discrepant annota- tions, we held multiple in-group discussions and conducted checks and modifications to improve annotation consistency and accuracy.
To improve annotation quality, we took several effective measures. We established strict criteria and documentation for every annotation option, in- cluding detailed explanations, extensive examples, and notes. For the convenience of annotation, we provide an interface for the annotators as detailed in Appendix A.2. Additionally, prior to each anno- tation session, we held training course to provide guidance. During the me-annotationn process, we adjusted the course and guidance documents address any issues, ensuring that the annotation document was comprehensive and definitive before large-scale annotation.
We used the 13 categories from the macro level of domain classifications to annotate the source and target domain categories of metaphors. We didn't 't use the 46 categories from the micro level because they were more complex and detailed. The macro level categories offered a more manageable and meaningful level of abstraction for our annotation task. Furthermore, using the domain classifications described above, we construct a domain specificc lexicon that includes vocabulary corresponding to each domain category, along with exemplar words. The lexicon consists of 2,755 words, including 61 category words and 2,694 specific example words. Refer Appendix A.5 for a detailed domain lexi- con.
Kappa score (Fleiss, 1971) was used to measure classification consistency, with scores of x =0.69, =0.66, x =0.62, and x =0.77 achieved for iden- tifying metaphors, identifying target domain cate- gories, identifying source domain categories, and
Sentiment categories The comprehension of metaphor primarily entails identifying the mapping

---

9000
8000
7000
6000
5000
Metaphor
Negrtve
Neuural
Paeelliee
2030
450
Figure 4: Distribution of Categories in the Source and Target Domains
Figure 5; Distribution of Emotional Categories in Metaphorical and Non-Metaphorical Data
identifying emotional categories, respectively, indi- cating the reliability of annotations.
observe that the proportion of neutral data in metaphorical ex- pressions lower than that non-metaphorical expressions, suggesting that metaphorical expres- sions tend to encompass a richer emotional range. Our findings are consistent with previous research suggesting that metaphors convey more emotions or feelings than simple prose (Citron and Goldberg, 2014).
non-metaphorical expressions,
4 Dataset Analysis
Refer to Table for an overview of the dataset statistics. We conducted an analysis of category distribution in both the source and target domains, presented in Figure 4. It is evident that the cate- gories "arttfact", "natural object" and "substance" appear most frequently in both domains, indicat- ing the preference for these tangible categories in expressing ideas and emotions metaphorical ad- vertising. Conversely, the categories "communi- cation". "relation" " and "psychological" have rel- atively lower frequencies, as they often involve more abstract and intricate concepts that require deeper understanding of contextual and back- ground knowledge. Furthermore, among the 13 categories examined, the source domain exhibits more dispersed distribution, suggesting a diverse utilization and manifestation of source domains, while the target domain revolves around more cen- tralized themes and situations compared to the source domain.
Methodology
Taking into account the significance of domain knowledge in metaphor detection, we propose benchmark called Cascading Domain Knowl- edge Integration (CDKT) that aims to enhance the model's ability to detect metaphors by incorporat- ing domain knowledge. The schematic diagram of CDKT is illustrated in Figure 6.
Introducing domain knowledge into models poses a challenge due inherent differences how is presented text and images. Textual domain knowledge is typically easier to extract, ex- pressed explicitly through specific vocabulary and sentence structures. Conversely, domain knowl- edge in images tends to be more implicit. In the following section, will elaborate on how CDKI addresses this disparity and achieves the extraction and integration of domain knowledge.
Figure 5 demonstrates varying emotional distri- butions across different metaphors. In metaphori- cal data, positive and neutral emotions are evenly distributed, while negative emotions account for a smaller portion. This indicates that metaphorical expressions advertising often correlate with pos- itive or neutral attitudes. In non non-metaphorical data, neutral emotions dominate the dataset, followed by positive emotions, with negative emotions be- ing the least prevalent. This reflects the directness and clarity of non-metaphorical expressions, which are typically utilized to convey widely accepted information or depict everyday life scenarios. By comparing the statistical data of metaphorical and
Regarding the text modality, we employ a seg- mentation technique to preserve multiple inherent domain-specific vocabularies in the textual data by extracting nouns. These nouns are then connected using the [SEP] token to form K which serves as the domain knowledge for the text modality. Regarding the image modality, structed a cascaded domaín word set to introduce and enhance the dependency relationships do- main knowledge within the images. Specifically,
have con-

---

the cascaded domain word set for an image consists of three parts: the macro-level domain word set, the micro-level domain word set, and the entity-level domain word set. These word sets are composed of vocabulary corresponding to the respective levels in the domain lexicon. Taking the macro-level do- main word set as example, we will describe the construction method of the cascaded domain word set for images. By utilizzig Clip (Radford al., 2021), we obtain the probabilities of each vocab- ulary in the macro-level of the domain dictionary appearing in the image. We then retain the top vocabularies with the highest probabilities as the macro-domain word set Selmacro for that particular image. The construction method for the micro- domain word set, Séfmicro and the entity-domain word set, Setentiyy follows the same approach. Each set consists of p and q vocabularies, respectively. Collectively, these three sets constitute the domain knowledge within the image, as Kimage:
AklA hore
Reswet
Bert
Cry sannds
Teat
bdbon
dp
Figure 6: Cascading Domain Knowledge Integratio (CDKI) illustration
Experiments
6.1 Baselines
Kinage Selmuero Selmicro Selentity
In this section, we introduce the baselines used in experiments. Specifically, adopted random, text-based, image-based, and multi-modal models as the baseline models and compared their perfor- mance with our proposed model. Due to the fact that metaphors, irony, and hate speech fall under the category of figurative language, there exists certain degree of similarity between these three Losks(Abulaiis et al., 2020). Thus, we selected these multi-modal methods as control models. Random: denotes random predictions based on the data.
Lastly, the domain knowledge from the image and text modalities are combined to generate the final domain knowledge, denoted as Kpair, for the image-text pairs:
Kpair
Kinage Kreat
(2)
In addition, we conducted multiple sets of ex- periments to investigate the impact of different vo- cabulary sizes on the model results while keeping the text attributes fixed. The details of these exper- iments can be found in Appendix ??. Ultimately, we determined that the optimal values for n, p. and q are 2, 3, and 5, respectively.
Resnet50 (He et al., 2016): It is a 50-layer convo- lutional neural network with residual connections. Image features are extracted from its last convolu- tional layer's output vector.
VGG16 (Simonyan and Zisserman, 2014); en-
To incorporate the domain knowledge Kpair into the model, we employ BERT (Devlin et al., 2018) to convert the Kpuir into vectors, which serve as the domain feature input for the image-text pairs. We utilized BERT and ResNet50 (He et al., 2016) to obtain textual and visual features respectively. Sub- sequently, we utilize cross-attention mechanism (Vaswani et al., 2017) to jointly model the domain features with the image and text features. This ap- proach enhances the interaction of domain knowl- edge between different features. The resulting out- puts are concatenated and fed into the softmax func- tion to obtain the final result.
hances network depth and the expression capability of non-lincar features by stacking consecutive 3x3 convolutional layers and max pooling layers. Bi-LSTM (Hochreiter and Schmidhuber, 1997): It's bidirectional recurrent neural network that combines the outputs of forward and backward LSTM to capture global context in a sequence.
Bert (Devlin et al., 2018) and Roberta (Liu et al., 2019): We conducted the experiment us- ing two widely-used Chinese mansformer-basee pre-trained language models, namely bert-based- chinese and Aaberta-base-chiesss as baselines.
Res-BERT (Zhang et al., 2021): It uses

---

Melaphor
larget domain
Source domain
Sentiment
Type
Model
ACC
F1
ACC
F1
ACC
Fl
ACC
F1
Random
0.4997
0.3332
0.0755
0.0108
0.0748
0.0107
0.3391
0.1688
Bi-LSTM
0.6455
0.5141
0.3158
0.1164
0.3265
0.1186
0.5329
0.5257
Text
BERT
0,6774
0.6429
0.3459
0.1077
0.3681
0.1254
0.5784
0.5457
Roberta
0.6749
0.6578
0.3564
0.1038
0.3645
0.1367
0.5836
0.5531
VGG16
0.6355
0.5022
0.3214
0,1030
0.3233
0.1297
0.5426
0,5361
Image
ResNet50
0.6624
0.6312
0.3347
0,1123
0,3476
0.1447
0.5579
0.5458
Res-BERT
0.7002
0,6745
0,3680
0.1396
0.3868
0.1352
0.6048
0.5727
VIIBERT
0.7231
0,6718
0,3683
0.1224
0,3754
0.1509
0.6192
0.5801
VisualBERT
0.7277
0.6899
0.3783
0.1352
0.4041
0.1594
0.6231
0.6055
Text+Image
HFM
0.7363
0.6757
0,3828
0.1355
0,3924
0.1656
0.6112
0,5802
CES 0.7430
0.6996
0.3952
0.1421
0,4339
0.1845
0.6303
0.5986
Our model
0.7632
0.7289
0.4377
0.1754
0,4692
0.1962
0.6693
0.6337
Table 2: Results on proposed dataset (MultiCME)), Best bold.
ResNet50 to extract image features and BERT to encode text. Then, the two sets of features con- catenated to form fused feature vector, which is input into fully connected layer for classification. VilBERT (Lu et al., 2019) and VisualBERRT (Li et al., 2019): They are both multimodal language models based on the Transformer architecture. Vil- BERT utilizes dual-stream attention mechanism to encode visual and textual information. Visual- BERT employs self-attention mechanism en- code images and text in a single stream manner.
perameerrsmsterss
Value
Image embedding size
2048
Dropout
0.3
Text max length
64
Batch size
32
Learning rate
3e-5-1e-4
Modality fusion size
512
Table 3: Hyperparameters.
scores, as shown Table 2. Our model excels in metaphor detection, followed by metaphor sen- timent detection, and performs weakest in target and source domain detection. The subpar perfor- mance in target and source domain detection can be attributed to two factors. Firstly, with 13 cat- egories, there is excessive number of category features that the model struggles accurately cap- ture. Secondly, an issue of data imbalance exists among differentt categories, resulting in inadequate generalization capabilities of the model for minor- ity categories. Nonetheless, our model outperforms all baseline models, validating its effectiveness.
HFM (Cai et al., 2019): The multimodal fusion model combines image, attribute, and text features using Bi-LSTM and MLP to generate a unified representation for ironic prediction tasks.
CES (Blaier al., 30 This method improves hate speech meme classifier accuracy by leverag- ing image-caption to enhance both multimodal and unimodal models.
6.2 Implementation
Our model constructed using the PyTorch framework (Paszke et al., 2019), and we pre- processed our data utilizing the pandas and NumPy libraries. Our optimization algorithm of choice was Adam (Kingma and Ba, 2014), with cross-entropy serving as our loss function. After fine-tuning the baseline, we selected the best-perforning model and hyperparameters to be tested on the reserved testing set. Table 3 presents comprehensive overview of the key hyperparameters employed in our experimental study.
From the table, evident that models solely relying on image features exhibited subpar per- formance. Among them, ResNet50 outperformed VGG16 due to its utilization of residual connec- tions to mitigate the issue of gradient vanishing. Text-based methods exhibited superior perfor- mance compared image-based methods, as text can convey more contextual information and pro- vide valuable context to the model. Both Bert and Roberta, having been fully trained on large-scale language corpora, outperformed Bi-LSTM.
6.3 Results and discussion
The multimodal approach yielded the best re- sults, as the fusion of image and text features en- hanced the detection performance of the model.
We evaluate our model's performance by compar- ing it with baseline models using accuracy and FI

---

Our model achieved the highest performance in the multimodal setting, In comparison the CES model that incorporates image captions as addi- tional features, the inclusion of domain knowledge provides the model with richer and more valuable information. While HFM utilizes image attributes additional features, our approach introduces do- main attributes with hierarchical relationships, en- abling the model leverage higher-level concepts and semantic associations. This incorporation of hierarchical domain attributes effectively enhances the performance of the model.
metaphors between different language systems. This expansion would help uncover biases present in current metaphor models and provide a deeper understanding of the cognitive mechanisms un- derlying metaphor comprehension. Moreover, a dataset encompassing more diverse languages and cultures would greatly benefit the models" capac- ity to comprehend multimodal metaphors. We en- courage researchers embrace the challenging yet captivating task of expanding MultiCMET by incor- porating data from additional languages future endcavors.
We also performed qualitative analysis of the wrongly classified samples. We examined approxi- mately 50 misclassified instances and categorized the reasons for the errors, which are detailed in Appendix A.4.
Ethies Statement
Based on the source of the data, our MultiCMET dataset can be divided into two parts. The first part consists of data obtained through querying search engines, specifically Baidu and Bing. For this por- tion, we have followed the data use agreements of these search engine querying platforms. We have made our annotations publicly available for this data, and we provide a detailed description of the query process. The second part of the dataset comprises data collected from publicly available datasets that allow data usage for non-commercial academic purposes. We strictly adhere to the terms and conditions of these publicly available datasets. We have released the unique IDs associated with this data, along with our annotations. By ensuring transparency and compliance with data use agree- ments, we aim to promote responsible and ethical data usage while enabling researchers to access and utilize the MultiCMM datasct effectively.
7 Conclusion
In conclusion, this paper addresses the limitations in research on multimodal metaphors beyond the English language and the lack of exploration in cat- egorizing source and target domains in metaphors. It introduces the MultiCMET dataset, a large-scale multimodal Chinese metaphor dataset consisting of text-image pairs from advertisements. The dataset includes manual annotations of metaphors, domain categories, and conveyed sentiments. Addition- ally, a domain lexicon is constructed, providing hierarchical domain classifications and correspond- ing examples. We propose the CDKI benchmark. which leverages domain-specific features for better metaphor understanding.
This paper significantly contributes to the field of multimodal metaphor understanding by provid- ing comprehensive dataset, novel methodologies, and benchmark results. opens up avenues for fu- ture research in exploring metaphors across differ- ent languages, enhancing metaphor comprehension, and addressing the cognitive mechanisms involved in metaphor processing. The availability of the dataset and code further encourages the research community to expand and improve upon the find- ings presented in this work.
Acknowledgements
This work is supported by NSFC Programs (No.62076051).
References
Muhammad Abulaish, Ashraf Kamal, and Mohammed
Zaki. 2020. survey of figurative language and computational detection in online social networks. ACM Transactions the Web (TWEB), 14(1):1-52.
Limitations
John A. Barnden and Mark G. Lee. 2007. An artifi- cial intelligence approach to metaphor understanding. Theoria el Historia Scientiarum, 6(1 ):39 412.
One limitation of the current version of Multi- CMET is its exclusive coverage of the Chinese language. is highly necessary to develop the MultiCMET dataset for other languages in order to facilitate comparative research on multimodal
Julia Birke and Anoop Sarkar. 2006. A clustering ap- proach nearly unsupervised recognition nonlit- eral language. In 11th Conference of the European

---

Chapter of the Association for Computational Lin- guistics, pages 329-336, Trento, Italy, Association for Computational I Linguistics.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Efrat Blaier, Itzik Malkiel, and Lior Wolf. 2021. Cap- tion enriched samples for improving hateful memes detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9350-9358, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 98)):1735- 1780.
Gitit Kehat and James Pustejovsky. 2020. Improving neural metaphor detection with visual datasets. Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 5928-5933, Marseille, France. European Language Resources Association.
Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multi- modal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guisties, pages 2506-2515, Florence, Italy, Associa- tion for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee.
Walter Kintsch. 2000. Metaphor comprehension: A computational theory. Psychonomic bulletin & re- view, 7(2):257-266.
2021. MelBERT: Metaphor detection via contextual- ized late interaction using metaphorical identification theories. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Teech- nologies, pages 1763-1773, Online. Association for Computational Linguistics.
Zoltán Kövecses. 2010. Metaphor, language, and cul- ture. DELTA: Documentação de Estudos em Lingüís-- tica Teórica e Aplicuda, 26:739-757.
Kevin Kuo and Marine Carpuat. 2020. Evaluating a Bi-LSTM model for metaphor detection in TOEEL essays. In Proceedings of the Second Workshop on Figurative Language Processing, pages 192-196, On- line. Association for Computational Linguisties.
Francesca M. M. Citron and Adele E. Goldberg. 2014, Metaphorical Sentences Are More Emotionally En- gaging than Their Literal Counterparts. Journal of Cognitive Neuroscience, 26(11):2585-255)
George Lakoff. 1994. Master metaphor list. University of California.
Seana Coulson and Cyma Van Petten. 2002. Conceptual integration and metaphor: An event-related potential study. Memory & cognition, 30(6):958-966 968..
George Lakoff and Mark Johnson. 1980. The metaphor- ical structure of the human conceptual system. Cog- nitive science, 195):195-208
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint arXiv:1908.03:: 1905.03557.
Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jum Zhou, Yifan Bai, Zilin Yu, Ychua Yang, Qingqing Dang, and Hongya Wang. 2020. Pp- ocr: A practical ultra lightweight ocr system. ArXiv, abs/2009.09944
Jerry Liu, Nathan O' O'Hara, Alexander Rubin, Rachel Draelos, and Cynthia Rudin. 2020. Metaphor detec- tion using contextual word embeddings from trans- formers. In Proceedings of the Second Workshop on Figurative Language Processing, pages 250-255, Online. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv.
Charles Forceville. 2017. Visual and multimodal metaphor in advertising: Cultural perspectives. Styles of Communication, 9(2).
Charles J. Forceville and Eduardo Urios-Aparisi, editors. 2009. Multimodal Metaphor, volume 11. De Gruyter Mouton, Berlin, New York.
Birte Lönneker. 2004. Lexical databases as resources for linguistic creativity: Focus on metaphor. In Pro- ceedings of the LREC 2004 workshop on language resources for linguistic creativity, pages 9-16.
Mengshi Ge, Rui Mao, and Erik Cambria. 2022. Ex- plainable metaphor identification inspired by con- ceptual metaphor theory. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10)11068) 10689.
Birte Gönneker-Rodman 2008. The hamburg metaphor database project: issues in resource creation. Lan- guage Resources and Evaluation, 42:293-318.

---

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task task-agnostic visiolinguis- tie representations for vision-and-langune tasks. Ad- vances neural information processing systems, 32.
Ekaterina Shutova and Simone Teufel. 2010. Metaphor corpus annotated for source-target domain mappings. In I.REC, volume 2. pages 2-2. Citeseer.
Ekaterina Shutova, Simone Teufel, and Anna Korhonen. 2013. Statistical Metaphor Processing. Computa- tional Linguistics, 39(2):001-353.
Rui Mao, Chenghua Lin, and Frank Guerin. 2018. Word embedding and WordNet based metaphor identifica- tion and interpretation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1222- 1231, Melbourne, Australia. Association for Compu- tational Linguistics.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv.
Gerard Steen, Aletta G Dorst, J Berenike Herrmann, Anna Kaal, Tina Krennmayr, and Trijntje Pasma. 2010. A method for linguistic metaphor identifica- tion. Amsterdam:
Zachary J Mason. 2004. Cormet: computational, corpus-based conventional metaphor extraction sys- tem. Computational linguistics, 30(1):23-44.
Benjamins.
Chang Su, Shuman Huang, and Vijiang Chen. 2017. Automatic detection and interpretation of nominal metaphor based on the theory of meaning. Neuro- computing, 219:300-311.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(1 41.
Saif Mohammad, Ekaterina Shutova, and Peter Tur- ney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Seman- tics, pages 23-33, Berlin, Germany. Association for Computational Linguistics.
Chang Su, Kechun Wu, and Yijiang Chen. 2021. En- hanced metaphor detection via incorporation of ex- temaal knowledge based linguistic theories. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 1280-1287, On- line. Association for Computational Linguisties.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style. high-performance deep learning library. Advances in neural information processing systems, 32.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Bo Xu, Tingting Li. Junzhe Zheng. Mchdi Naseri- parsa, Zhehuan Zhao, Hongfei Lin, and Feng Xia. 2022. Met-meme: A multimodal meme dataset rich in metaphors. In Proceedings of the 45th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, SIGIR '22, page 2887-2899, New York, NY, USA. Association for Computing Machinery.
Ronald Rivest. 1992. The md5 message-digest algo- rithm. Technical report.
Fan-Pei Gloria Yang. Kailyn Bradley, Madiha Huq, Dai- Lin Wu, and Daniel C. Krawezyk. 2013. Contextual effects on conceptual blending in metaphors: An event-related potential study. Journal of Neurolin- guistics, 26(2):312-326.
Ekaterina Shutova, Douwe Kiela, and Jean Maillard. 2016. Black holes and white rabbits: Metaphor iden- tification with visual features. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 160-170, San Diego, California. Association for Computational Linguistics.
Lu Ying, Xinhuan Shu, Dazhen Deng, Yuchen Yang. Tan Tang, Lingyun Yu, and Yingcai Wu. 2023. Metaglyph: Automatic generation of metaphoric glyph-based visualization. IEEE Transactions on Vi- sualization and Computer Graphics, 29(1)3331-341.
Ekaterina Shutova, Lin Sun, Elkin Darío Gutiérrez, Pa-
tricia Lichtenstein, and Srini Narayanan. 2017. Mul-
tilingual Metaphor Processing: Experiments with Supi-Supe and Unsupervised Learning. Com- putational Linguistics, 43(1):71-123.
Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, and Hongfei Lin. 2021. Multimet: A multi- modal dataset for metaphor understanding. In Pro- ceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 321 3214 3225.
Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. Metaphor identification using verb and noun cluster-
ing. In Proceedings of the 23rd International Con- ference Computational Linguistics (Coling 2010). pages 1002-1010, Beijing. China. Coling 2010 Orga- nizing Committee.

---

A Appendix
p
q
ACC
A.1
Keyword query
0
0
0
0.7134
The keywords used during the data collection pro- cess are presented in Table 4.
0.7272
4
0.7396
1
3
5
0.7452
Topic
Keywords
1
6
9
0.7426
mobile phone, tablet, television, laptop computer, pork, beef, lamb, car, motor- cycle, bicycle, electric vehicle, tire, en- gine oil, toothpaste, mouthwash, hair gel, detergent, laundry detergent, mois- turizer, sunscreen, shower gel, soap, baby food, stroller, cookware, kitchen utensils, home decor, bedding, snacks, beverages, health supplements, nutritional supple- ments
1
0.7315 10
2
2
2
0.7445
Commercial
2
0.7538 2
4
2
0.7632 3
5
2
6
9
0.7499
2
0.7433 10
3
3
3
0.7516
smoking, drug prohibition, bullying. campus violence, drunk driving, envi- ronmental protection, energy conserva- tion and emission reduction, waste sort- ing, animal protection, afforestation, re- ducing plastic pollution, traffic safety, fire safety, cybersecurity, protection intellectual property rights, fraudfr counter-terrorism and riot prevention, drowning prevention, natural disaster re- sponse, food safety
3
2
0.7501 4
3
5
0.7541
Public service
3
9
0.7432
3
10
0.7389
Table Model accuracy with different values of n, and q. The values of and q represent the respec- tive quantities of vocabulary terms in Sefmacms Set Setentit.
anger, losing temper, calmness, compo- sure, animal, cut, dog, belief, values, phi- losophy, happiness, contentment, under- standing, change, alteration, color, emn- tion, red, orange, yellow, people, organi- zation, ethnic group
Metaphor
introducing domain knowledge is helpful in im- proving model accuracy. As the values n, p. and increase, the accuracy of the model in metaphor recognition also improves. When the accuracy of the model is 0.7516, higher than the model's performance when p q = This suggests that increasing the number of vocabulary can improve the model's accuracy.
p q 3,
Table 4: Key words of each topic.
A.2 Annotation Interface
Figure 7 presents several examples of our annotated
illustrations.
To unalyze the relationship between subset vo- cabulary and model accuracy, we conducted exper- iments using controlled variables. Results showed that increasing values of p and q improved model accuracy, while keeping constant. Micro-domain and entity-specific vocabulary helped capture de- tailed features and enhance recognition accuracy. However, higher values of n, P. and q were not always better. The parameters n 2, p p= 3, and q achieved optimal results, while n 3, p = 6. and = 9 did not perform as well despite having larger vocabulary. Excessively high values of n, p. and q introduced noise, resulting in decreased accuracy.
A.3 Number of vocabulary terms
In this section, we aim to explore the relationship between the number of subset vocabulary in im- age cascading and model performance. Firstly, use baseline model without domain knowledge, where the values of n, p, and q are all The accu- racy of this model is 0.7134, Next, we gradually increase the values of n, p. and q while keeping other features constant to analyze their impact on model performance and eliminate interference. The ranges of n, and q are [0-5], [0-7], and [0-10] respectively. We select representative data from these ranges and present them in Table 5.
In conclusion, incorporating domain knowledge can enhance the model's recognition accuracy, and expanding the subset vocabulary in image cascad- ing can improve model performance. However, an excessive number of vocabulary may introduce
When n. p. and q are all set 1, the model's accuracy in metaphor recognition is already higher than that of the baseline model without domain knowledge, reaching 0.7272. This indicates that

---

KMM
Text: poiscn punishment
24t BI 34 7 40PP
Oceurrence: metaphor
Text: buy 2 get 20% off, buy get 30%
Target category: artifact
Occurrence: non-metaphoe
Source category: artifact
Sentiment category: neutral
Sentiment calegery: negalive
1992
PAT 39
Text: best-selling facial mask
protecting wildlife
Occurreace: metaphor
Occurrence: non-metaphoo
Targel calegory: arifact
Sentiment calegory: positive
Source category: substance
Sentient category: positive
Figure Examples of annotations.
noise and lead to performance deterioration. This is further supported by the parameters 2, 7, 10, and 3, 7, q 10. Hence, we select the parameters 2, p 3, and q as the subset vocabulary size, which yielded the best experimental outcomes.
m ng?
A.4 Error analysis
One type of eиror is when the model misclassi- fies metaphoric text and images non-metaphoric. This often occurs when the target souree domain in the image is heavily distorted partially visi- ble, limiting the model's ability to learn domain information. As result, conflicts between the text and image may go undetected, leading to the in- correct identification of metaphors. For instance, in Figure& 8 on the left, the obscured tiger tail pre- vents proper recognition of the conflicting elements with the euro symbol, resulting in the image being mistakenly classified as non-metaphoric.
Olayi Teodlpeday
Canuded. Steppior Pualet. soo-Maaggle
Goomledl Pualis Malaller
Figure 8: Two examples of identification errors.
categorizes the target domain "artifacts" and the source domain as "natural objects." " Analysis suggests this may be because the image hand- drawn, leading the model to incorrectly interpret certain elements as natural objects and perceive a conflict between toothpaste and natural objects. As result, the text-image pair is mistakenly classified as a metaphor.
The second classification error occurs when the model misclassifies non- non-literal text as metaphorical. This can happen due to changes in artistic style or information overload, making it difficult for the model to learn domain-specific information and detect conflicts between domains. In the Figure 8 on the right, the text itself lacks metaphorical meaning, but the model mistakenly identifies it as metaphorical. Additionally, the model wrongly
A.5 Domain Lexicon

---

individual
adults, saaulsscchilrrenteens.ce dboetor.cnginceerfaaatt.ttt Higer.mouse.battge
person
social roles
mammal
birds
cagle, peacock, penguin, etc snake, turtle, lizard, etc. butterfly, bee, ant, etc.
animal
reptiles
insects
fish
goldfish, carp, grass carp, etc. algae, seaweed, moss, etc.
spore plants
plant
seed plants
tree, grass, flower, etc.
buildings
library, school, office building, etc,
transportation
car, boat, train, etc.
daily necessities
scissors, paper, pen, etc.
artifact
works
film, book, poem, etc.
toys
balloon, doll, building blocks, etc.
wcapons
baton, nunchaku, firearm, etc.
parts of plants
root, stem, leaf, etc.
body parts
organs, lungs, heart, etc. Earth, Moon, stars, etc.
natural object
celestial objects
physical entity
geological structures food material fuel reagent military activities
hills, platcaus, mountain peaks, etc. beverage, rice, wine, etc.
iron, water, wood, etc.
substance
gas, oil, methane hydrate, etc.
medicine, cigarette, bleach, etc.
battle, siege, desertion, etc.
social activities
competition, performance, etc. production, consumption, etc.
economic activities
event
electrostatic precipitation, distillation, hydrolysis, etc.
industrial activities
activities of daily living
landing, recovery, wearing, etc.
orientation
north, south, west, etc. China, America, Germany, etc.
location
country
scencry
scascape, mountain view, etc.
biochemical processes
metabolism, respiration, sleep, etc.
natural processes
wind, rain, snow, etc.
light propagation, dynamics, heat conduction, etc.
process
physical processes
redox reaction, reaction, metal corrosion, etc.
chemical process
magnitude relationship
share, ratio, quantity, etc.
relation
property relationship
economic situation, debt, etc.
language
English, Chinese, French, etc.
communication
news
password, email, news, etc.
signal
phone number, symbol, etc.
abstraction
physical attributes
temperature, speed, density, etc. vision, taste, touch, etc. solid, linear, triangle, etc. creativity, thinking, concept, etc. intrinsic motivation, etc. happiness, pride, sadness, etc.
attribute
sense
shape
cognition
psychological
motivation
emotion
Table Domain Lexicon.


Reviews:
[{"review_id": "VVNKyq2zK2", "reviewer": "Reviewer_RctQ", "summary": "", "questions": "A. It sounds like manual revision was performed on the WordNet hierarchy, for addressing both language/cultural gaps and incorrect(?) categories. Do you think the latter issue is due to a deficiency in WordNet in general? or is the revision intended to tailor the categories to this metaphor domain task? I'm also wondering if you think the proposed categories should be revised for different languages/cultures. \n\nB. I think the annotated categories are quite broad. I understand that the broad categories allow manual annotation. However, I'm wondering why you think these broad categories could be useful.\n\nC. Could you provide more details about the annotators? \n\nD. It would be good to see some important details of the annotation guidelines because metaphor annotation is super difficult due to its fuzzy and somewhat subjective nature and the dataset is a main contribution of this paper. \n\nE. lines 525-526: What are n, p, and q?\n\nF. Do most images contain text?\n\nG. lines 612-613: I didn't quite understand what this sentence meant.\n\nH. lines 613-614: Did the model perform relatively well for the categories with more samples?\n\nI. I'm wondering what kind of characteristics these multimodal metaphor images show, especially in these images for advertisements.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Concerning Question A, we made modifications based on the WordNet(Miller, 1995) structure to adapt it to the context of multimodal metaphor tasks in Chinese, without implying any flaws in WordNet itself. Firstly, we referred to Chinese grammar and made modifications to WordNet. For example, in WordNet, \" person \" is classified as \"person,\" while \" people \" is classified as \"group.\" However, in the Chinese context, there is no clear distinction between \"person\" and \"people,\" and both are categorized as \"person.\" Therefore, we removed the \"group\" category from WordNet. Secondly, considering the characteristics of metaphor domain tasks, we modified WordNet to better align its categories with metaphor domain classification tasks. For instance, in metaphors, \"feelings\", \"motivations\", and \"cognition\"  are often considered as a single domain. Thus, we classified them as subcategories of psychological features(Lakoff, 1994). In my opinion, the proposed domain categories in the paper are comprehensive, and they can also be applicable to metaphor-related tasks in other languages and cultural contexts. However, it should be noted that the expression of metaphors may vary across different languages and cultural backgrounds. Therefore, I recommend making subtle modifications beyond the categories proposed in the paper to accommodate metaphor-related tasks in diverse cultural contexts.  \n\nConcerning Question B, this is not a broad range for the task of this paper. The 13 classifications used in this paper closely align with the source and target domain categories mentioned in the Metaphor Master List (MML)(Lakoff, 1994) at a fine-grained level. For example, in the MML, the sentence \"She is free like a butterfly\" is categorized as the target domain of behavior and the source domain of animals. The use of these 13 categories effectively guides the model to capture the conflicts between the source and target domains. However, excessively detailed classification would lead the model to rely on specific combinations of categories for inference, thus affecting its generalization capability.  \n\nConcerning Question C, the annotators, including master's students and the authors themselves, have research backgrounds in metaphor computation and metaphor linguistics. Due to the anonymity of the authors and related personnel, specific information regarding their backgrounds has not been disclosed.\n\nConcerning Question D, we provided explanations and training to the annotators before annotation, along with detailed annotation guidelines. However, due to space limitations, this was not mentioned in the document.The following are some rules for determining metaphors in the annotation guidelines. We will include the complete content in the next version.   \nMetaphor: We classify samples as metaphors when they exhibit the following characteristics. Objects within the image undergo modifications that highlight a particular feature, such as changes in shape, abnormal sizes, or unusual colors. Unconventional combinations of objects are observed in the image. The text utilizes techniques like homophones or allusions to emphasize a specific feature. There is a lack of harmony between the objects in the image and the objects mentioned in the text. Additionally, textual deformations are present within the image(Zhang et al., 2021).  \nNon-metaphor: Samples classified as non-metaphor only depict the representation of objects in the image-text pairs without any implicit or deeper meanings.\n\nConcerning Question E, the values of n, p, and q represent the respective quantities of vocabulary terms in Set_macro , Set_micro ,Set_entity. The relevant explanations can be found in lines 505-511 of the original text.  \n\nConcerning Question F, our initial collection of images included a significant number of images without accompanying text. However, we applied filtering techniques to remove these images systematically, employing both automated and manual methods to exclude those that lacked any textual content. Please refer to line 293 of the original text.  \n\nConcerning Question G, this sentence explains why our model performs poorly on target domain and source domain classification tasks. The model struggles to accurately capture all features of the 13 categories, resulting in its performance being inferior in target domain and source domain classification tasks compared to metaphor identification tasks.\n\nConcerning Question H, yes, the model can learn richer features from a larger number of samples, which enables it to achieve better performance. We conducted a reanalysis of experimental data and found that the model demonstrates superior performance in categories with a larger sample size, such as artifact and natural object categories, compared to categories with a smaller sample size, such as relation and location categories.\n\nConcerning Question I, in multimodal image-text pairs, metaphors are typically expressed through the combination of objects from different domains. For example, in the picture, cigarettes are depicted as the lungs of a person, implicitly conveying the harm of smoking to health.\n\nConcerning the first reason for rejection, in the appendix, we present examples of both metaphorical and non-metaphorical images. For instance, as shown in Figure 7, the two images in the upper row are non-metaphorical. In the lower row, the image on the left is metaphorical. The text \"毒\" (poison) and \"刑\" (punishment) in the image conveys the metaphorical concept of drug injection and its consequences. In this image-text pair, the target domain is represented by a needle, while the source domain is symbolized by a rope. Similarly, the image on the right is metaphorical, with the text referring to a popular face mask. In this image-text pair, the target domain being a face mask and the source domain being water. It should be noted that not all the images in the appendix are metaphoric. In the upcoming version, we will include English translations and explanations for better comprehension.\n\n[1] George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41  \n[2] George Lakoff. 1994. Master metaphor list. University of California.   \n[3] Dongyu Zhang, Minghao Zhang, Heting Zhang, Liang Yang, Hongfei Lin: MultiMET: A Multimodal Dataset for Metaphor Understanding. ACL/IJCNLP (1) 2021: 3214-3225", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A multimodal metaphor dataset in non-English could be a valuable resource for future researchers. \n- The paper is easy to follow.\n- It provides benchmark results, which seem to be from reasonable models.", "reasons_to_reject": "- I'm not entirely sure how metaphors are defined in the images+texts of this dataset. I cannot understand why some images are metaphorical in the Appendix when only looking at the images. The text descriptions are in Chinese. \n- This is a dataset paper, and the annotation guidelines are particularly important for metaphors. Metaphor annotation is notoriously challenging because it is fuzzy and subjective, and multimodal metaphors are even more under-explored. However, the paper doesn't provide sufficient detail about the annotation guidelines and justifications (e.g., how a metaphor is defined and recognized, how sentiment is defined), although the details of the annotation process are provided. \n- As a dataset paper, I expect more insightful discussion about the data so that I can determine whether I want to use this dataset or not. For example, what are the common disagreements in the annotations? What kind of characteristics do you observe in the multimodal metaphors? They are all advertisements. Are there any interesting things you can observe with regard to that? This kind of discussion can give a better sense of the dataset and the tasks that can be performed with it. \n\nFor these reasons, I think this paper needs more work although I appreciate the work.", "questions_to_authors": "A. It sounds like manual revision was performed on the WordNet hierarchy, for addressing both language/cultural gaps and incorrect(?) categories. Do you think the latter issue is due to a deficiency in WordNet in general? or is the revision intended to tailor the categories to this metaphor domain task? I'm also wondering if you think the proposed categories should be revised for different languages/cultures. \n\nB. I think the annotated categories are quite broad. I understand that the broad categories allow manual annotation. However, I'm wondering why you think these broad categories could be useful.\n\nC. Could you provide more details about the annotators? \n\nD. It would be good to see some important details of the annotation guidelines because metaphor annotation is super difficult due to its fuzzy and somewhat subjective nature and the dataset is a main contribution of this paper. \n\nE. lines 525-526: What are n, p, and q?\n\nF. Do most images contain text?\n\nG. lines 612-613: I didn't quite understand what this sentence meant.\n\nH. lines 613-614: Did the model perform relatively well for the categories with more samples?\n\nI. I'm wondering what kind of characteristics these multimodal metaphor images show, especially in these images for advertisements.", "paper_topic_and_main_contributions": "This paper presents a Chinese multimodal metaphor dataset, annotated with metaphors, source and target domains of the metaphors, and conveyed sentiments. It also provides benchmark results for three tasks of (1) metaphor detection, (2) source and target identification, and (3) conveyed sentiment identification. \n\nMy detailed feedback is below. Please note that major and minor comments are mixed. \n- line 128: that has been previously explored for the text modality\n- sec 3.3: did you use the Chinese WordNet? or the English one? \n- lines 357-364: it sounds like significant amount of manual revision was performed on the WordNet hierarchy, for addressing both language/cultural gaps and incorrect(?) categories. Do you think the latter issue is due to a deficiency in WordNet in general? or is the revision intended to tailor the categories to this metaphor domain task? \n- lines 376-377: isn't it a bit broad? I understand that the broad categories allow manual annotation. However, I'm wondering what kind of applications can be considered with these broad categories. Additionally, I think this doesn't really address the crucial problem of determining the right level of granularity when identifying source and target domains in metaphorical mappings. I was expecting more when I was reading the claim that this paper provides clear structuring principles. Unless the paper provides why these broad categories are better, I think it is an overclaim.\n- line 402: since it was mentioned as an expert-based approach, who were the annotators (i.e., backgrounds?) besides that they were native speakers of Chinese? I understand that a training course was provided. I think more detailed descriptions about the annotators could be useful because this is a metaphor annotation task, which is somewhat fuzzy and subjective. \n- lines 422-434: It would be good to see some important details of the annotation guidelines because metaphor annotation is super difficult due to its fuzzy and somewhat subjective nature and the dataset is a main contribution of this paper. \n- It would be beneficial to discuss disagreements in the annotation process because it will reveal the challenges in the dataset giving a better idea about the tasks. \n- Sec 4: frequently mapped domains would be interesting, too.\n- lines 525-526: What are n, p, and q?\n- Do most images contain text?\n- lines 612-613: I didn't quite understand what this sentence meant.\n- lines 613-614: Did the model perform relatively well for the categories with more samples?\n- lines 644-648: what are some important findings from the qualitative error analysis? They should be in the main text. \n- I'm wondering what kind of characteristics these multimodal metaphor images show. For example, many verb metaphors in text show selectional preferences violation. Many adjective+noun metaphors show unusual patterns of word concreteness.\n- Many things in the Appendix (Table 3, Figure 6, Figure 7) are in Chinese only. Translations must be provided!", "excitement": 2, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- lines 220-225: source and target domains are not fixed, as you  know. For example, journey could be either source or target, depending on how we use it. I think it would be better to revise this part to avoid misunderstanding about source and target domains. Presenting mapping examples would be better. \n- line 341: \"source level\" <- here, \"source\" is not related to the source in a metaphorical mapping, right? To avoid confusion, I recommend replacing the word. \n- Important/interesting messages from Appendix should be summarized in the main body of the paper.", "Justification_for_Ethical_Concerns": "- Although the authors say that they followed the data use agreements of the search engine platforms (Baidu and Bing), I'm not entirely sure if it is fine to make all those advertisement images publicly available."}}, {"review_id": "mv6a9svMH7", "reviewer": "Reviewer_Szae", "summary": "", "questions": "The cultural variation in representing drunk using animals and \"泥\" (mud) a great approach. Have you considered using some annotators who are not native Chinese speakers? If so, were there any objective reasons for not selecting them?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Concerning the Question, we did not consider using annotators who are non-native Chinese speakers because the metaphor annotation task requires annotators to have an accurate understanding of cultural connotations and the meanings conveyed by vocabulary in different contexts. Non-native Chinese speakers, lacking exposure to the Chinese language and culture, may have insufficient understanding of metaphors and find accurate annotation a challenging task. Therefore, we did not employ non-native Chinese speakers as annotators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Taking an advertising perspective, creating a large-scale multimodal dataset is a clever and efficient approach compared to other sources like literary works, songs, or artists' paintings. Extracting data from creative advertisements and artistic creations offers valuable insights.\n\nUsing the domain classification approach further enhances our understanding of metaphorical expressions in different forms of data, mirroring human thinking patterns for problem-solving. For instance, using animals to represent intoxicated individuals or using \"泥\" (mud) to signify someone heavily drunk in Chinese culture. Such differences can be applied for interdisciplinary analysis, exploring cultural variances resulting from the distribution of animals, and aiding language learners in memorizing visual representations of expressions.\n\nBuilding a high-quality dataset and lexicon construction require substantial effort and resources.", "reasons_to_reject": "In the appendix, I noticed some commercial advertisements, which could lower the dataset's overall quality. I didn't find relevant statistical data. \n\nIncorporating artwork and portfolios from art and advertising students could improve the dataset's quality.", "questions_to_authors": "The cultural variation in representing drunk using animals and \"泥\" (mud) a great approach. Have you considered using some annotators who are not native Chinese speakers? If so, were there any objective reasons for not selecting them?", "paper_topic_and_main_contributions": "This paper presents research on multimodal metaphor understanding in the Chinese language.  It makes several contributions towards  multimodal metaphor understanding.\n\nThe paper introduces MultiCMET, a Chinese dataset consisting of 13,820 text-image pairs from advertisements. \n\nThe paper proposes a domain classification approach and lexicon construction.\n\nThe paper introduced a new task to evaluate the capabilities of multimodal metaphor understanding models. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In some sections, there is only Chinese without English translation; please provide bilingual translations wherever possible."}, "extra_text": {}}, {"review_id": "3DfKYYDLwD", "reviewer": "Reviewer_MYGj", "summary": "", "questions": "As mentioned in weakness, what is the significance of metaphor domain classification for metaphor recognition, Does it help to recognize metaphors?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Concerning the Question, the classification of metaphorical domains contributes to the identification of metaphors. Metaphor involves mapping concepts from a source domain to a target domain, where both domains and their mapping relationship constitute a metaphor. For instance, as shown in Figure 2 of this study, the concept of ice cream (source domain) is mapped onto the earth (target domain) to convey the notion of global warming. The purpose of classifying metaphorical domains is to gain a better understanding of the mapping relationships between these domains. The key aspect of metaphor identification lies in capturing the incongruity of domain mappings. We construct a domain dictionary based on the classification results of metaphorical domains, providing qualitative descriptions of the mapping relationships between domains. In Table 2 of the paper, our model demonstrates a significant improvement in metaphor recognition compared to other models that do not utilize the domain dictionary. This indicates that through the use of the domain dictionary, the model is better able to capture the mapping relationship between the source and target domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to understand.\n\n2. This work claims to firstly propose a publicly available large-scale Chinese metaphor dataset, and will publish the data and code. Although some Chinese metaphor datasets existed in the past, the poor annotation quality of these datasets makes them difficult to use. Therefore, the new Chinese metaphor dataset proposal in this work is significant to the community.\n", "reasons_to_reject": "A contribution of this paper is the metaphor domain classification, which takes identifying the source and target domains of metaphors as a new task. For example, in “The earth is slowly melting”, the target category for metaphor is natural object, and the source category is substance. However, I'm not sure about the significance of identifying source categories and target categories for metaphors. Does it help to identify metaphors better? But no conclusions can be drawn from the experimental results. Perhaps the authors could elaborate more on it or demonstrate in experiments that it helps metaphor.", "questions_to_authors": "As mentioned in weakness, what is the significance of metaphor domain classification for metaphor recognition, Does it help to recognize metaphors?", "paper_topic_and_main_contributions": "This paper proposed a multimodal Chinese metaphor dataset, MultiCMET, consisting of 13,820 text-image pairs of advertisements with manual annotations of the occurrence of metaphors, source and target domain categories, and sentiments metaphors convey. Furthermore, it developed a domain lexicon that encompasses categorizations of domains and corresponding examples. Finally, it propose the Cascading Domain Knowledge Integration (CDKI) benchmark, which utilizes domain-specific features from the domain lexicon to enhance the understanding of metaphors.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 59，290，318"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The focus of the paper is the creation of a multimodal dataset for metaphor understanding. It’s main contributions are:  

- a Chinese multimodal (images-texts) metaphor database from advertisements, annotated with metaphors, source and target domains of metaphors, and conveyed sentiments.  

- a 4 domain lexicon; 

- benchmark results for metaphor detection, source and target identification, and conveyed sentiment identification.   

Reviewers generally appreciated the work but found some important weaknesses in the paper. The authors' responses were accurate and detailed, providing clarifications and the required details for many issues raised. While the required revisions are not substantial, they are indeed necessary (please refer to the Cons. section for specifics); anyways, the work is worthwhile.  

**Pros** 

- the paper describes the creation of a high-quality Chinese multimodal metaphor dataset, which constitutes a valuable resource for future researchers in the field of multimodal metaphor understanding; 

- The extraction of data from creative advertisements and artistic works is an interestingly efficient method for creating useful data for metaphor understanding and yields valuable insights; 

-  the domain classification approach followed enhances comprehension of metaphorical expressions across diverse data forms, and seems to align with human cognitive patterns for problem-solving; 

- The paper is well-written, the argumentation easy to follow and understand; 

- It presents interesting benchmark results;  

- the work entails substantial effort and resources. 

**Cons**  

-  the paper leaves doubts about the choice of the level of granularity for identifying source and target domains. Some discussion seems to be needed in the paper, to help the reader understand the choices made; 

- the description of the annotation process lacks details esp. on the annotators’ background, apparently for anonymity reasons. It is recommended to include this in the next version of the paper; 

- a detailed analysis or discussion of the dataset and the annotation results is missing: e.g. analysis and discussion of disagreements, discussion on the observed properties of the multimodal metaphors; genre specific observations; 

- Most appendix material is in Chinese-only. Translation in English is required.
Decision: Accept-Findings