Exploring Chain-of-Thought for Multi-modal Metaphor Detection (2024)

Abstract: Metaphors are commonly found in advertising and internet memes. However, the free form of internet memes often leads to a lack of high-quality textual data. Metaphor detection demands a deep interpretation of both textual and visual elements, requiring extensive common-sense knowledge, which poses a challenge to language models. To address these challenges, we propose a compact framework called C4MMD, which utilizes a Chain-of-Thought(CoT) method for Multi-modal Metaphor Detection. Specifically, our approach designs a three-step process inspired by CoT that extracts and integrates knowledge from Multi-modal Large Language Models(MLLMs) into smaller ones. We also developed a modality fusion architecture to transform knowledge from large models into metaphor features, supplemented by auxiliary tasks to improve model performance. Experimental results on the MET-MEME dataset demonstrate that our method not only effectively enhances the metaphor detection capabilities of small models but also outperforms existing models. To our knowledge, this is the first systematic study leveraging MLLMs in metaphor detection tasks. The code for our method is publicly available at https: //github.com/xyz189411yt/C4MMD.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 91–101 August 11-16, 2024 ©2024 Association for Computational Linguistics
Metaphors are commonly found in advertising and internet memes. However, the free form of internet memes often leads to a lack of high-quality textual data. Metaphor detection demands a deep interpretation of both textual and visual elements, requiring extensive common-sense knowledge, which poses a challenge to language models. To address these challenges, we propose a compact framework called C4MMD, which utilizes a Chain-of-Thought(CoT) method for Multi-modal Metaphor Detection. Specifically, our approach designs a three-step process inspired by CoT that extracts and integrates knowledge from Multi-modal Large Language Models(MLLMs) into smaller ones. We also developed a modality fusion architecture to transform knowledge from large models into metaphor features, supplemented by auxiliary tasks to improve model performance. Experimental results on the MET-MEME dataset demonstrate that our method not only effectively enhances the metaphor detection capabilities of small models but also outperforms existing models. To our knowledge, this is the first systematic study leveraging MLLMs in metaphor detection tasks. The code for our method is publicly available at https: //github.com/xyz189411yt/C4MMD.

heading: 1 Introduction
text: Metaphors are highly prevalent in our everyday expressions and writings, which can have a range of impacts on downstream tasks in Natural Language Processing (NLP), such as semantic understanding (Neuman et al., 2013), sentiment analysis(Ghosh and Veale, 2016; Mohammad et al., 2016) and other tasks. In recent years, the rise of social media has sparked interest in multi-modal metaphors. As a result, several datasets for multi-
*Equal contribution †Corresponding author
modal metaphors have been proposed (Zhang et al., 2021, 2023a; Alnajjar et al., 2022).
Current research on multi-modal metaphor detection is still in its early stages. The primary challenge lies in the complexity and variety of multimodal metaphors. Compared to single-modality detection, multi-modal metaphor detection not only spots metaphors in sentences but also categorizes them as image-dominated, text-dominated, or complementary. The second major challenge arises from the poor quality of textual content, mainly sourced from advertisements and memes on social media. Texts give the image more metaphorical features. Recent efforts use OCR (Optical Character Recognition) to extract texts in the image. However, only relying on OCR to convert them into parallel texts leads to the loss of texts’ positional
91
information. Figure 1 presents a representative example, symbolizing how ’PUBG’ (a video game) acts like a trap preventing "me" from achieving my "life goals".
To overcome these challenges, we hope to gain insights from LLMs, utilizing their rich world knowledge and contextual understanding capabilities to obtain deeper meanings of both images and text. An intuitive and efficient approach is to use these LLMs to generate supplementary information without fine-tuning them; we then only need to fine-tune a smaller model to establish connections between this information and metaphors. To reduce the illusion of MLLMs, inspired by CoT (Wei et al., 2022), we have designed a three-step method that progressively acquires the MLLM’s information in describing images, analyzing text, and integrating information from both modalities. The advantages of this strategy are as follows: First, it can provide downstream models with additional information for each modality. Second, the shallow-to-deep understanding sequence aligns closely with human logic, making it easier for the LLM to grasp deeper meanings. Furthermore, subsequent steps can correct misunderstandings from earlier steps, enhancing the model’s robustness.
Overall, we utilize a CoT-based method called C4MMD to summarize knowledge from MLLMs and enhance metaphor detection in smaller models by fine-tuning them to link this knowledge with metaphors. The basic idea is shown in Figure 1, we first input images and text into the MLLM and obtain information describing the image, text, and their fusion. Furthermore, we have designed a downstream modality fusion structure, which is intended to translate supplementary information into metaphorical features for more accurate classification. Specifically, we have designed two auxiliary tasks focused on determining the presence of metaphors within the image and text modalities.

heading: 2 Related Work
text: Early metaphor detection tasks were confined to a single modality and employed methods based on rule constraints and metaphor dictionaries (Fass, 1991; Krishnakumaran and Zhu, 2007; Wilks et al., 2013). With the flourishing development in the field of NLP, machine learning-based methods (Turney et al., 2011; Shutova et al., 2016) and neural network-based methods (Mao et al., 2019; Zayed et al., 2020) have successively emerged. Following
the introduction of the Transformer (Vaswani et al., 2017), methods based on pre-trained models gradually supplanted the former methods and became the current mainstream approach (Cabot et al., 2020; Li et al., 2021; Lin et al., 2021). Ge et al. (2023) have categorized current efforts into four main directions, namely additional data and feature methods (Shutova et al., 2016; Gong et al., 2020; Kehat and Pustejovsky, 2021), semantic methods (Mao et al., 2019; Choi et al., 2021; Su et al., 2021; Zhang and Liu, 2022; Li et al., 2023b; Tian et al., 2023a), context-based methods (Su et al., 2020; Song et al., 2021), and multitask methods (Chen et al., 2020; Le et al., 2020; Mao et al., 2023; Badathala et al., 2023; Zhang and Liu, 2023; Tian et al., 2023b), where semantic methods and multitask methods have become the primary focus of recent research.
As an emerging direction, numerous datasets across image and text modalities have emerged, primarily sourced from social media and advertisements, yielding extensive multilingual text-image modal data (Zhang et al., 2021; Xu et al., 2022; Zhang et al., 2023a). Unlike the aforementioned approaches that extract information from different modalities and directly merge them, we leverage LLMs employing the CoT method to analyze features between modalities, aiding downstream models in cross-modal fusion.

heading: 3 Method
text: We propose a novel framework called C4MMD using MLLMs to enhance metaphor detection. We first introduce the task definition(3.1) and the complete model architecture((3.2). After that, we elaborate on knowledge acquisition from MLLMs using the CoT method(3.3) and the implementation of the downstream fusion module(3.4). Finally, we provide a brief exposition of the training methodology (3.5).

heading: 3.1 Task Definition
text: Formally, the task of multi-modal metaphor detection falls under the typical category of multi-modal classification problems. Given a set of cross-modal sample pairs, the task aims to determine whether metaphorical features are present and provide a classification result. Our work focuses on the detection of metaphors in image-text pairs, thus the task is represented as:
Y = F (xI , xT ) (1)
where xI and xT respectively denote the features of the image and text modalities. Our objective is to utilize a more effective method F to ensure that the classification result Ŷ more closely aligns with the true value Y .

heading: 3.2 Overview
text: As shown in Figure2, the architecture of C4MMD consists of two primary components: a knowledge summarization module and a downstream structure for multi-model fusion.
In the knowledge summarization module, we provide an image-text pair to the MLLM and design a three-step template with CoT prompting. The first two templates instruct the MLLM to focus exclusively on a single modality—either text or image, ignoring the other to generate explanations and insights. In the third step, the MLLM combines insights from both modalities. Based on previous analyses, the model achieves a deeper understanding and a fuller integration of both modalities.
After obtaining additional textual information for different modalities from the MLLM, we merge this with the original texts to form a textual input. Similarly, the input image is treated as the visual modality input. The model then processes these inputs through modality-specific encoders to derive feature vectors.
In the multi-model fusion module, we scale and combine vectors from different modalities and de-
velop a fine-grained classifier. Specifically, we integrate the supplementary image description vector with the visual modality input vector as the image vector, combine the text analysis vector with the textual input vector as the text vector, and merge these to form a cross-modal vector. These three vectors are then used for classification purposes. The classifier uses the cross-modal vector to detect metaphors, the image vector to identify imagedominated content, and the text vector for textdominated content. This approach enhances the use of multi-modal features for precise metaphor detection.

heading: 3.3 Knowledge Summarization from MLLMs Using the CoT Method
text: To guide the MLLM in generating higher-quality and more informative features, we employ CoT prompting. This method directs the MLLMs to extract deeper information across modalities. We then utilize this supplementary information to assist the smaller model in achieving better semantic understanding and modality fusion. In conclusion, we construct the three-step prompts as follows.
STEP1. Initially, to ensure that the model concentrates on comprehending objects, scenes, or other visual elements in the image(Represented by xI ) without interference from textual features, we guide the model to understand and interpret the image information based on a template Question1:
Question1: Please temporarily ignore the text in the image and describe the content in the image. Try to be concise while ensuring the correctness of your answers.
This step can be formulated as follows:
mI = MLLM(xI ,Question1) (2)
STEP2. Next, to better comprehend the hidden meanings in the text(Represented by xT ) while excluding any interference from image features, we guide the model to understand and interpret the textual information according to a template Question2:
Question2: Please analyze the meaning of the text. Note that there may be homophonic memes and puns, distinguish and explain them but do not over interpret while ensuring the correctness of the answer and be concise.
This step can be formulated as follows:
mT = MLLM(xT ,Question2) (3)
STEP3. Ultimately, we aspire for the model to synthesize the results from the previous two steps(Represented by mI and mT ) and further integrate the image and text features(xI and xT ), thereby obtaining more profound cross-modal interaction information. We encourage the model to fuse features from different modalities according to template Question3:
Question3: Please combine the image, text, and their description information and try to understand the deep meaning of the combination of the image and text. No need to describe images and text, only answer implicit meanings. Ensure the accuracy of the answer and try to be concise as much as possible.
This step can be formulated as follows:
mMix = MLLM(xI , xT ,mI ,mT ,Question3) (4)

heading: 3.4 Multi-modal Fusion for Metaphor Detection
text: After obtaining additional modal information generated by the MLLM, we designed a modal fusion architecture to facilitate inter-modal integration and effectively leverage the extra information produced by the MLLM to enhance metaphor detection capabilities.

heading: 3.4.1 Modality-Specific Encoding
text: We use an image encoder and a text encoder to obtain vectorized encodings of the image xI and text xT for subsequent inter-modal fusion. Considering the additional information generated by the MLLM is presented in text form, we treat it as extra visual mI , textual mT , and mixed mMix information. This information is concatenated with the original text and then processed through the text encoder for computation.
V = ViT-Encoder(xI), T = XLMR-Encoder(xT ,mT ,mI ,mMix) (5)
where V is the output of the image encoder, and T is the output of the text encoder.
To enable the text encoder to distinguish between texts from different modalities during computation, we adopt a method similar to BERT’s segment encoding by adding extra learnable parameter vectors for the text from each modality. The vectorized encoding Embi of the i-th word xi (xi ∈ {xT ,mT ,mI ,mMix}) entering the text encoder can be represented as follows:
Embi = ET (xi) + EP (i) + ES(segment(xi)) (6) where ET , EP and ES represent learnable matrices for token embeddings, positional encodings, and segment embeddings, respectively. The term segment(xi) ∈ (0, 1, 2, 3) refers to the segment encoding of the word xi, this encoding is specifically represented by the following formula:
segment(xi) =    1, if xi ∈ mI 2, if xi ∈ {xT ,mT } 3, if xi ∈ mMix 0, otherwise (7)

heading: 3.4.2 Modality Fusion
text: Before modal fusion, to ensure the vector dimensions from both encoders are consistent, in the textual modality, we compute the average of all word vectors mean(T ) as the vector representation of the entire sentence. For the visual modality, we take the vector of the CLS token VCLS as the representation of the entire image. Then, we use a linear layer with a GeLU activation function (Hendrycks and Gimpel, 2016) to map it to the same feature space as the textual modality. The formula is represented as follows:
V reshape = GeLU(WvVCLS + bv) (8)
Considering that the text information from different modalities generated by the large model has already undergone a degree of fusion within the text encoder, we therefore concatenate these two vectors from both modalities to obtain the final fused vector representation. The formula for this process is as follows:
EMix = [V reshape,mean(T )] (9)
Finally, we use a linear layer and a softmax classifier for metaphor classification.
ŷ = softmax(WMixEMix + bMix) (10)
Considering the diverse sources of metaphorical features, we employ two separate classifiers to categorize metaphors predominantly driven by either the image modality or the text modality. The aim is to force the detection of metaphorical features in both image and text before their fusion, thereby reducing the classification complexity for the final classifier. This approach of fine-grained metaphor detection is based on the following formula:
EI =[V reshape,mean(TmI )] (11) ET =mean([TxT ,TmT ]) (12)
Here, TmI , TxT and TmT respectively represent the parts of the text encoding vector that describe the image and the text. Finally, two classifiers are used to categorize the metaphorical features in the text and the image. The formula for this classification process is as follows:
ŷI =softmax(WIE I + bI) (13)
ŷT =softmax(WTE T + bT ) (14)
In the above-mentioned formulas, Wv, WMix, WI and WT are trainable parameter matrices; bv, bMix, bI and bT represent bias matrices.

heading: 3.5 Training
text: The training objective of our multi-modal metaphor detection model involves the integration of three distinct loss functions, denoted as LI , LT and LM . The loss function is as follows:
L = 1|DME|
|DME|∑
i=1
LCE
( Ŷ , Y ) (15)
where DME is the number of samples in the dataset, The loss formula is parameterized as L =
{LI ,LT ,LM}, with Ŷ = {ŷ, ŷI , ŷT } and Y representing the model’s predicted outcomes and the true values, LCE is the cross-entropy loss function.
To optimize the overall performance, we define the aggregate loss Lsum as a weighted combination of these individual losses. The final loss function is formulated as:
Lsum = 0.5 · LI + 0.5 · LT + LM (16)

heading: 4 Experiments
text: In this section, we begin by introducing the dataset used to validate our method, as well as the experimental setup. Following this, we report the experimental results and provide an analysis of these outcomes.

heading: 4.1 Data and Setting
text: We selected the multi-modal metaphor dataset proposed by Xu et al. (2022), which consists of 10,000 meme images collected from social media. Text information was extracted from these images using OCR methods to construct the multi-modal metaphor dataset, which includes 6,000 entries in Chinese and 4,000 in English. In addition to the classification labels for metaphors, they also annotated the source of the metaphors and their associated emotions.
All trained models were set with a learning rate of 1e-5, a batch size of 8, and were trained for 100 epochs with an early stopping mechanism in place. The dataset was randomly shuffled and divided into training, validation, and test sets in a 6:2:2 ratio. All experiments were conducted on a single 3090- 24G GPU. The final results of our method were obtained by taking the average of five different random seeds, with the average single run time within 20-30 minutes. Finally, the model’s performance was evaluated based on the F1 score.
The Low-Rank Adaptation (LoRA Hu et al. (2021)) fine-tuning approach was adopted for finetuning LLMs. All of the settings followed those used in Alpaca-LoRA*.

heading: 4.2 Baseline Methods
text: Language Models We tested several common pre-trained models for this task, including the AutoEncoder MBERT (Pires et al., 2019), XLM-R (Conneau et al.,
*Alpaca-LoRA
2019), as well as the AutoRegressive models MT5 (Xue et al., 2020) and M-BART (Liu et al., 2020). Additionally, we evaluated the capabilities of LLMs on this task by using LLaMA2 (Touvron et al., 2023) and ChatGLM3 (Zeng et al., 2022), due to their strong performance in both Chinese and English corpora. We fine-tuned both models separately using LoRA.
Vision Models
We also tested models from the vision domain, including Convolutional Neural Network (CNN) models such as VGG (Simonyan and Zisserman, 2014), ResNet (He et al., 2016), and ConvNeXt (Liu et al., 2022), as well as models based on the Transformer architecture, like ViT (Dosovitskiy et al., 2020) and Swin Transformer (Liu et al., 2021).
Multi-modal Models
In the multi-modal model domain, we selected VILT (Kim et al., 2021), BLIP2 (Li et al., 2023a), and InternLM-XComposer (Zhang et al., 2023c) to test their capabilities in addressing the metaphor detection task. All three models employ the Transformer architecture, yet they differ significantly in model size. We tested the capabilities of these MLLMs both in a zero-shot setting and with LoRA fine-tuning.
Other Related Works We also explored other works related to our task, thereby lending more credibility to our comparative analysis. Below, we introduce these works in detail.
• CLIP (Zhao et al., 2023): Evaluation of various models for hate meme detection task. We adopted best performance CLIP to evaluate its effectiveness in multi-modal metaphor detection tasks.
• Vilio (Muennighoff, 2020): An excellent method which achieves 2nd place in the Hateful Memes Challenge. It Uses OCR and entity recognition technologies to extract text and visual features from memes for better meme harmfulness detection tasks.
• CoolNet (Xiao et al., 2023): Extracting text syntactic structure to boost model’s sentiment analysis ability on Twitter multi-modal data.
• MultiCMET (Zhang et al., 2023b): A baseline model for chinese multi-modal metaphor detection task. It uses the CLIP model to generate additional information to assist in the fusion between modalities.

heading: 4.3 Main Results
text: Table 1 shows the capabilities of different models in the task of multi-modal metaphor detection. Here we only evaluated the main classification results ŷ.

heading: VM LM ACC P. R. F1.
text: We did not assess the outcomes of the two subtasks ŷI and ŷT as the two subtasks which were primarily designed to serve the main task.
Our approach achieved the best results in both Chinese and English sample sets. Considering the outcomes produced directly by the MLLM (InternLM-XComposer-7b), we allowed it to indirectly generate additional features for images and texts, effectively leveraging the large model’s capabilities. Coupled with a downstream classifier, this approach resulted in an additive effect.
The performance of multi-modal models varied widely, with most models not surpassing language models. This underscores the importance of textual modality in recognizing multi-modal metaphors. MLLMs did not perform well in zero-shot scenarios, partly due to our designed prompt templates. However, the primary reason is the models’ inability to understand the task. Encouragingly, after finetuning BLIP2, its capabilities surpassed all other comparative methods. This demonstrates the benefit of interaction between image and text modalities in the task and how large models can effectively understand and address this task after fine-tuning.
In related work, studies closely aligned with our own, such as those by Zhang et al. (2023b) and Muennighoff (2020), have achieved competitive performances. However, Twitter sentiment classification by Xiao et al. (2023), which differs somewhat from our task, consequently showed weaker performance.

heading: 4.4 Influence of Different Factors
text: Table 2 shows the effects demonstrated by our model after undergoing ablation experiments.
Replacing the fusion structure in the model with a linear layer resulted in a significant decrease in performance. This suggests the necessity of additional fusion structures to help the model understand the extra features generated by the MLLM. Moreover, eliminating the CoT generation method of the MLLM, and relying solely on a one-step generation method, led to an even more noticeable performance drop. This also indicates that the CoT method can generate better additional features, thereby assisting downstream models in making more accurate judgments.
Interestingly, the performance of the model declined only slightly when we removed the image processing module. This indicates that the MLLM can provide a certain level of visual information for smaller models, but more comprehensive information still requires the contribution of vision models.

heading: 4.5 The Impact of Different Language Vision Model Combinations
text: We tested the capabilities of multiple visual and textual models during modal fusion. The language model was uniformly set to MBERT when testing vision models and the ViT was used consistently when testing language models.
From the data in Table 3 and Table 1, although in single modality settings, the vision model VGG and the textual model M-T5 achieved the best per-
formance, the combination of ViT and XLM-R outperformed all others upon modal fusion.
The combinations of ResNet + MBERT and VGG + MBERT are also baseline models proposed by Met-Meme (Xu et al., 2022). According to the results, we reported the same results as them.

heading: 4.6 The Impact of Language Model Size
text: Figure 3 illustrates the abilities of models of different sizes under our architecture. Considering that the improvement ratio is generally between 0 and 1, while the model size is typically in the hundreds of millions, we divided all model sizes by 400 million to scale them between 0 and 1, allowing us to display both model size and improvement on the same graph. It was evident that as the model size increased, especially when the model was initially small, there was a progressively noticeable performance improvement. When the model was too small, the additional textual information did not yield positive effects; rather, it could have the potential to negatively impact the model’s performance. It was only when the model size was increased that the model became capable of understanding longer contextual information.

heading: 4.7 Case Study
text: To further explore the effectiveness of our proposed model, we select two examples from the testing dataset illustrated in Figure 4.
The first example demonstrates an image-led
metaphor. By directly comparing a seal with a potato, it depicts the consequences of looking at too many cute seals. The MLLM, through its understanding of the image, accurately recognized the resemblance between the seal and the potato, thereby aiding the downstream model in making the correct judgment.
In the second example, the MLLM identified features from both the image and text, and then combined these to correctly understand the humorous meaning expressed in the meme. The downstream model accurately recognized that it did not contain metaphorical features. In contrast, methods lacking the additional information from the large model judged it to be metaphorical based solely on the phrase "like a lady," leading to a misjudgment.

heading: 5 Conclusion
text: Our study aimed to tackle the challenges of multimodal metaphor interpretation by leveraging advanced MLLMs. We designed a three-step method with CoT-prompting to extract richer information from both images and text. Augmented knowledge from MLLMs proved crucial in enhancing smaller models to grasp metaphorical features within each modality and in the fusion of modalities. This work not only advances multi-modal metaphor detection but also paves the way for future research exploring the potential of MLLMs in addressing complex language and vision challenges.
Limitations
We believe the main limitation of our work lies in only testing our metaphor detection ability within a multilingual meme dataset and not extending to other subtasks in meme datasets, such as harmfulness detection, nor to metaphor detection in other multi-modal datasets. However, despite the lack of experimental data, we are confident in our work’s applicability in these directions, which will also be one of our future research focuses.
Additionally, regarding the meme dataset, we did not find a usage license, nor did we filter for potential harmfulness or offensiveness in the data, including in the extra features generated by the MLLM, which may contain toxic data, thus presenting a risk of offensiveness and harmfulness.
Although we used a method of averaging five tests for our model, for other comparative methods, we simply took the results from the first run for inclusion in our tables. We acknowledge this could introduce some error, but we believe that even if the comparative methods were tested in the same way, our method would still demonstrate overwhelmingly superior performance.

heading: Future Work
text: This work not only advances multi-modal metaphor detection but also paves the way for future research exploring the potential of MLLMs in addressing complex language and vision challenges. Limitations
We believe the main limitation of our work lies in only testing our metaphor detection ability within a multilingual meme dataset and not extending to other subtasks in meme datasets, such as harmfulness detection, nor to metaphor detection in other multi-modal datasets. However, despite the lack of experimental data, we are confident in our work’s applicability in these directions, which will also be one of our future research focuses. Additionally, regarding the meme dataset, we did not find a usage license, nor did we filter for potential harmfulness or offensiveness in the data, including in the extra features generated by the MLLM, which may contain toxic data, thus presenting a risk of offensiveness and harmfulness. Although we used a method of averaging five tests for our model, for other comparative methods, we simply took the results from the first run for inclusion in our tables. We acknowledge this could introduce some error, but we believe that even if the comparative methods were tested in the same way, our method would still demonstrate overwhelmingly superior performance