Generating Multimodal Metaphorical Features for Meme Understanding

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 14:51:41

Abstract: Understanding a meme is a challenging task, due to the metaphorical information contained in the meme that requires intricate interpretation to grasp its intended meaning fully. In previous works, attempts have been made to facilitate computational understanding of memes through introducing human-annotated metaphors as extra input features into machine learning models. However, these approaches mainly focus on formulating linguistic representation of a metaphor (extracted from the texts appearing in memes), while ignoring the connection between the metaphor and corresponding visual features (e.g., objects in meme images). In this paper, we argue that a more comprehensive understanding of memes can only be achieved through a joint modelling of both visual and linguistic features of memes. To this end, we propose an approach to generate Multimodal Metaphorical feature for Meme Classification, named MMMC. MMMC derives visual characteristics from linguistic attributes of metaphorical concepts, which more effectively convey the underlying metaphorical concept, leveraging a text-conditioned generative adversarial network. The linguistic and visual features are then integrated into a set of multimodal metaphorical features for classification purpose. We perform extensive experiments on a benchmark metaphorical meme dataset, MET-Meme. Experimental results show that MMMC significantly outperforms existing baselines on the task of emotion classification and intention detection. Our code and dataset are available at https://anonymous.4open.science/r/MMMC-C37B.

Main paper: 
 related work 2.1 metaphor: 
 Metaphor is a figure of speech that compares a person or object to something else with similar characteristics. It adds vividness and imagination to descriptions by going beyond literal interpretation(e.g., "time is money."). In this literary device, the concept being described is the target concept (e.g., time), while the concept used to describe it is the source concept (e.g., money). Visual metaphors are also becoming more popular with the advancement of multimedia technology, especially in mediums like memes and posters. Visual metaphors capitalize on the visual medium's ability to convey complex ideas concisely and impactfully. Previous studies on metaphor mainly focused on linguistic metaphor and made significant progress in identifying, interpreting, and generating metaphors. Choi et al. [4] use contextualized word representations and linguistic metaphor identification theories to detect linguistic metaphors. Chen et al. [3] use out-of-domain data and idioms as additional knowledge to enhance metaphor interpretation. Stowe et al. [27] generate metaphors by encoding conceptual mappings between cognitive domains.Recently, there has been a growing interest in the study of visual metaphors. Akula et al. [1] create a metaphorical poster dataset and introduce four tasks related to visual metaphors, which include visual metaphor classification, localization, understanding, and generation. Similarly, Zhang et al. [33] develop a metaphorical poster dataset called MultiMET and conduct experiments to empirically demonstrate the beneficial impact of visual metaphors on comprehending posters. Additionally, Stowe et al. [27] utilize large language models and diffusion models to generate visual metaphors.Most studies on meme classification are non-metaphorical. They usually adopt a multimodal learning approach, where many effective methods for feature extraction, feature fusion, and multi-task learning have been proposed. For feature extraction, past work relies on LSTM to extract linguistic representation from OCR text [7, 36] while others suggest the usage of different image encoders to extract visual representation [2, 28] . For feature fusion, Guo et al. [9] fuse the textual and visual representations through a weighted combination of all modalities. Koutlis et al. [14] design a dual stage modality fusion module that incorporates the external knowledge. Nguyen et al. [20] design a fusion network which includes a combination of a multi-hop attention network and a stacked attention network. For multi-task learning, Lee and Shen [15] leverage the correlation between tasks and implement four different multi-task network heads having different level of interactions. Moreover, Duan and Zhu [7] adopt modified offline-gradient-blending strategy [29] to alleviate overfitting; Zhu et al. [35] explore meme classification in zero-shot setting; Hazman et al. [10] extract the spatial position of visual objects, faces, and text clusters to assist in meme classification. However, non-metaphorical methods ignore the metaphorical information in the meme, which plays an important role in meme understanding.On the other hand, some studies put forward metaphorical methods. Xu et al. [30] consider the metaphor source concept and metaphor target concept as distinct entities and encode them separately. They also demonstrate the potential of incorporating additional metaphorical information to enhance meme comprehension. Additionally, Zhang et al. [34] propose a metaphorical alignment task to improve meme understanding. They employ a conditional generative approach to capture metaphorical analogies and utilize a disentangled contrastive matching mechanism to preserve contextual sensitivity. However, these studies primarily focus on the linguistic representation of metaphor and ignore the visual representation, which is also crucial for metaphor comprehension. Hence, we adopt a different approach that generates visual metaphorical features to enrich the metaphorical representation. 
problem definition: 
 In this paper, we propose an approach to automatically generate multimodal features from memes for enhanced understanding of memes' semantic meanings. We showcase the effectiveness of the proposed feature generation method on two meme classification tasks, namely emotion classification and intention detection. Throughout the paper, we use the term metaphor source text/metaphor target text to represent the textual description of the metaphor source/target concept within a meme metaphor.Let 𝑚 be a meme. We represent its features as a four-tuple:EQUATIONwhere 𝑜 𝑣 is the original meme image, 𝑜 𝑐 is the original text obtained through optical character recognition (OCR), 𝑠 𝑐 is its metaphor source text, and 𝑡 𝑐 is its metaphor target text. In the rest of the paper, we use the subscript 𝑣 and 𝑐 to distinguish the visual and textual features, respectively. Note that 𝑠 𝑐 and 𝑡 𝑐 can be null if a meme does not contain metaphorical information. For a given meme 𝑚, the emotion/intention classification task is to map 𝑚 to its ground-truth label from a set of predefined classes1 . 
methodology: 
 In this section, we detail our proposed framework, MMMC, which autonomously generates metaphorical features from memes for enhanced meme understanding. Figure 3 (a) presents an overview of MMMC. It consists of two phases: Visual metaphorical feature Generation (VG) which is to generate visual metaphorical features about metaphors from their corresponding textual descriptions; and Meme Classification (MC) which is the integration of multimodal meme features for meme classification.In VG phase, the textual metaphorical features 𝑠 𝑐 and 𝑡 𝑐 extracted from one meme 𝑚 are mapped to visual metaphorical features via a GAN-based model. Consequently, the original meme 𝑚 is transformed into m with enriched visual features:EQUATIONwhere 𝑠 𝑣 and 𝑡 𝑣 are the metaphor features newly introduced and represent the metaphor source image and metaphor target image, respectively. The set of metaphorical features: (𝑠 𝑣 , 𝑠 𝑐 , 𝑡 𝑣 , 𝑡 𝑐 ) constitute multimodal metaphorical features to be used in the later phase.In MC phase, the multimodal metaphorical features of m are combined and integrated to jointly perform classification tasks. Next, we describe these two phases in detail.Let 𝑀 = {𝑚 1 , 𝑚 2 , ..., 𝑚 𝑛 } be a metaphorical meme set with 𝑛 memes. We denote the entity involved in a meme metaphor text as the metaphor concept (e.g., "yacht" or "electric iron" in Fig 3 ). For each metaphor concept in 𝑀, the VG phase aims to generate a metaphor image as its visual metaphorical feature, through a GAN-based mapping model that transforms the textual data to visual feature space.To achieve this, we first extract all source and target metaphor concepts from 𝑀 and construct a metaphor concept set containing only unique concepts: 𝑈 = {𝑐 1 , 𝑐 2 , . . . , 𝑐 𝑛 𝑐 }, where 𝑛 𝑐 is the number of unique metaphor concepts. Due to the large size of 𝑈 , we divide 𝑈 into 𝑛 𝑐 /𝑙 subsets, where the 𝑘-th subset (𝑘 ∈ {1, 2, ..., 𝑛 𝑐 /𝑙 }) can be denoted as:EQUATIONFor each subset, we train a GAN model that is capable of generating visual features from textual description of a given metaphor concept.We illustrate the overall workflow of the model in Figure . 3(b), and describe the procedure of obtaining such a model as follow. In particular, we collect 50 real images from Internet for each metaphor concept. Our collection criteria are that these images should be diverse, publicly available, and semantically consistent with the textual metaphor concept. For example, given the meme concept "bread", we collect 50 images covering different types of bread using image search engines, and no images of the rock band "Bread". Subsequently, we compile a collection of imagetext pairs where in each metaphorical concept is paired with its corresponding visual representation:EQUATION)where 𝑟 is a real image of 𝑐, and|𝑆 𝑘 | = 50 × |𝑈 𝑘 |.We propose to use a GAN-based model for visual feature generation. Specifically, we train a textconditional generator 𝐺 𝑡 and a discriminator 𝐷, where 𝐺 𝑡 maps a given piece of text to its visual representation, while 𝐷 is a classification model which determines if an image is realistic representation for the text. Given a metaphor text 𝑐, a textual encoder is firstly used to encode 𝑐 into a feature vector that captures its semantic meaning. The feature vector is subsequently fed into the generator 𝐺 𝑡 to obtain its visual representation 𝑓 :EQUATIONwhere Φ is a text encoder. The generated image 𝑓 by 𝐺 𝑡 forms a image-text pair with the text input 𝑐. Together with previously collected set 𝑆 𝑘 , the generated pseudo pair is fed into the discriminator 𝐷, which is trained to classify if an image-text pair is real (i.e., from 𝑆 𝑘 ) or fake (i.e., generated by 𝐺 𝑡 or paired with wrong texts).In this paper, we employ the prevalent BERT [6] as the text encoder which is pretrained on large amounts of textual data, and the architecture of RAT-GAN [31] as the design of 𝐺 𝑡 and 𝐷, since RAT-GAN has demonstrated effectiveness particularly on text-toimage synthesis.The generator 𝐺 𝑡 is trained to generate "real" images as much as possible given an input text, while the discriminator 𝐷 is trained to distinguish real-world images and the generated images. Since the two models have the opposite goals, they are cross-optimized until convergence during training. The overall training objective of discriminator and generator can be formulated as:EQUATIONEQUATIONHere, 𝑐 is the given text, ĉ is a mismatched text, and 𝑥 is real or generated image. We use the trained model for image generation. For each 𝑐 ∈ 𝑈 𝑘 , we keep 300 generated images, from which we manually select the most natural image, denoted as 𝑣, as the metaphor image of 𝑐. We repeat the above procedure for all subsets 𝑈 𝑘 (𝑘 ∈ {1, 2, . . . , 𝑛 𝑐 /𝑙 }), and thereafter, obtain a multimodal metaphorical feature setEQUATION)where 𝑣 is the metaphor image corresponding to metaphor text 𝑐.We use this set to enrich the feature representation of each meme 𝑚, yielding a transformed representation denoted as m.After obtaining the enriched representation m, we combine and integrate the multimodal feature representation for classification tasks. As shown in Figure 3 (a), we employ a dual-stage feature integration. In the first stage, the visual and text features are integrated for enhanced understanding of the metaphor source, metaphor target, and original meme, respectively. In the second stage, the embedded features of metaphor source, target, and original meme are fused for jointly use of classification. Next, we detail the two stages and the use of integrated features for classification.For an image-text pair, we use a Multimodal Encoder to integrate the visual and textual features. In particular, we use three Multimodal Encoders(MEs) for the original meme, metaphor source, and metaphor target concept:EQUATIONAs shown in Figure 3 (c), given an image-text pair (𝑣, 𝑐), ME first extracts monomodal representations, and then fuses them into a multimodal representation. Similar to Tang et al. [28] , we adopt Swin Transformer [18] and CLIP-ViT [22] to extract image features, and adopt DistillBERT [25] to extract text feature. Among them, Swin transformer is an image encoder that utilizes sliding windows and hierarchical structures, which is suitable for extracting visual representation of memes since many memes are composed of multiple sub-images; CLIP-ViT is a transferable vision model that uses text as supervised signal to train, which can extract visual features that are relevant on text; and DistilBERT is a text encoder that is a light version of the BERT model obtained through knowledge distillation. Here, the Swin Transformer pre-trained on ImageNet-21k [24] and the CLIP-ViT pre-trained on COCO [17] are adopted.During training, we fine-tune Swin Transformer, while freezing CLIP-ViT and DistilBert to reduce computational cost and improve the robustness of training. Given an input-text pair (𝑣, 𝑐), we denote the encoded feature of Swin Transformer as ℎ 𝑠 𝑣 ∈ R 768×1 , CLIP-ViT as ℎ 𝑐 𝑣 ∈ R 512×1 and DistilBERT as ℎ 𝑑 𝑐 ∈ R 768×1 . Then, we employ full-connected layers to map them onto the same dimension:EQUATIONEQUATIONEQUATIONHere, 𝑊 𝑠 ∈ R 512×768 , 𝑊 𝑐 ∈ R 512×512 , 𝑊 𝑑 ∈ R 512×768 areEQUATIONEQUATIONEQUATIONAfter obtaining ℎ 𝑜 , ℎ 𝑠 , ℎ 𝑡 , we fuse them through another transformer encoder. The same operations in Eq. ( 13)∼( 16) are performed for ℎ 𝑜 , ℎ 𝑠 , ℎ 𝑡 , to obtain a final metaphorical meme representation ℎ 𝑚 that incorporates meme and its multimodal metaphorical information.Finally, ℎ 𝑚 is fed into a fully connected network with the softmax activation function for the final classification:EQUATION)where ŷ is the predicted label. Afterwards, ŷ𝑖 is used to calculate the categorical cross-entropy, which is used as the loss function:EQUATION)where 𝑁 is the number of samples, and 𝑦 𝑖 is the true label of the 𝑖 𝑡ℎ sample. 
experiments 5.1 dataset: 
 We conduct experiments on the English memes of MET-Meme [30] due to its richness in metaphorical memes. Specifically, MET-Meme contains 4,000 English memes, including 1,114 metaphorical memes and 2,886 non-metaphorical memes. For non-metaphorical memes, MET-Meme provides their images, OCR texts, emotion labels and intention labels. For metaphorical memes, MET-Meme provides additional metaphor source texts and metaphor target texts as shown in Figure 2 . Table 1 summarizes the distribution of labels, which shows uneven distribution of classes.Meme, where we remove the duplicated concepts and those abstract concepts (e.g., "green life" and "Saturday"), of which images are difficult to obtain. As a result, we obtain a metaphor concept set 𝑈 of 293 unique metaphor concepts encompassing human organs, transportation, food, etc. Table 2 shows part of them. Then, we run the VG phase on 𝑈 with 𝑙 = 10. Note that 𝑈 is divided into subsets so that one subset contains 13 metaphor texts. In addition, real images are collected from Google Images, Bing Images and Baidu Images. We train the GAN model for 3000 epochs with a batch size of 10 and two Adam optimizers [13] for generator and discriminator separately. The learning rates for generator and discriminator are set to 1𝑒 -4 and 4𝑒 -4 , respectively.. After generation and selection, we obtain the multimodal metaphorical features set Γ, which contains 293 metaphor image-text pair. Figure 4 shows a part of them. We see that most metaphor images are natural and can reflect general appearance of metaphor concept. Hence, these images can be used as the visual features of metaphor concepts. We utilize 3 2080Ti GPU to train GAN at a speed of 17 hours per repeat. And it takes 1.7 hours to generate a metaphorical image representing a specific concept. While this time investment is relatively significant, we consider it acceptable given that the process does not take up inference time and that a single metaphor image can be used for multiple modality understanding tasks.We split the 4000 English memes into training, validation and test sets with the ratio of 8:1:1. For missing metaphor texts, we set them as "None". For their corresponding images, we set them as all-black images. An identical set of hyper-parameters is used for both emotion classification and intention detection tasks: a batch size of 50, dropout rate of 0.2 for each fully-connected layer, and an Adam optimizer with a learning rate of 1𝑒 -4 . To avoid over-fitting, We use early stopping if the loss on validation set does not get reduced within 500 epochs.We compare the proposed MMMC with existing meme classification models with demonstrated performance. These baseline methods are categorized into two groups: non-metaphorical methods and metaphorical methods. For non-metaphorical methods, we use methods of the teams that achieved high rankings at Memotion 2.0/3.0 competition [19, 23] , including Yet [36] , BLUE [2] , Little Flower [21] , NYCO-TWO [28] , and the method of Ramamoorthy et al. [23] . For metaphorical methods, we use the method provided by Xu et al. [30] , which uses only metaphor texts as additional metaphorical features. Additionally, we have adopted the state-ofthe-art vision-language pre-trained model BLIP-2(7B) [16] as the benchmark for our study, which has demonstrated remarkable capabilities across a range of vision-language tasks. Notably, BLIP-2 undergoes refinement through fine-tuning. We implement all algorithms on a Nvidia RTX4090 GPU machine.We use weighted precision, weighted recall and weighted F1-value as evaluation metrics. The experimental results are reported in Table 3, which confirm that the proposed method manages to achieve consistent improvement in terms of all metrics compared with other methods on both emotion classification and intention detection. In addition, we conduct pairwise 𝑡-test on weighted F1-value comparing MMMC with NYCO-TWO and the method of Xu et al. All of the produced 𝑝-values are less than 0.05. According these results, we can draw the following two preliminary conclusions. First, extra metaphorical information improve meme understanding. The experimental results on both classification tasks confirm that metaphorical methods are more competitive than non-metaphorical methods. On both tasks, our MMMC and the method of Xu et al. obtain the highest and second highest weighted F1-value. In addition, NYCO-TWO performs the best among nonmetaphorical methods and significantly outperforms other nonmetaphorical methods.Second, multimodal metaphorical features improve meme understanding. We observe that MMMC outperforms the method of Xu et al., which only uses linguistic representation of metaphor. Compared with the method of Xu et al., MMMC achieves a 1.14% improvement in weighted F1-value on emotion classification and a 6.54% improvement on intention detection.To further analyze the classification performance of MMMC, we present the confusion matrix in Figure 5 . On emotion classification, we observe that model has a high recall in happiness. In 115 happiness memes, 77 are predicted correctly. However, we also notice that all categories of memes are more likely be predicted as happiness. For example, in 53 sorrow memes, only 11 are correctly predicted, while 23 are mispredicted as happiness. In addition, model performs poorly in recognizing fear and surprise. To be specific, none of the 11 fear memes and 29 surprise memes are correctly predicted, which may be caused by the imbalanced label distribution in dataset. Similarly, on intention detection, model performs poorly in recognizing interactive and other due to the small number of samples. On the other hand, model performs well in distinguishing expressive, purely entertaining, and offensive, where more than half of them are correctly classified.5.6 Qualitative Analysis 3 reports the results. We observe that the worst performance is in the case of without any metaphorical features. Results can be improved by adding metaphorical features of any mode. Hence, we determine that extra metaphorical information can improve meme understanding. In addition, we observe that the case of adding multimodal metaphorical features outperforms the case of adding monomodal metaphorical feature. In particular, compared to w/o metaphor text and w/o metaphor image, our multimodal MMMC achieves a 0.86% improvement and a 1.51% improvement in weighted F1-value on emotion classification. Similarly, the performance improvement of intent detection is 1.95% and 2.35%. This confirms that multimodal metaphorical features can improve meme understanding. We also observe that visual metaphorical feature is more important than textual metaphorical feature. Specifically, w/o metaphor text surpasses the performance of w/o metaphor image by 0.65% and 0.40% F1-score on emotion classification and intention detection. Moreover, our 𝑀𝐸 is effective since the case of w/o metaphor text and image outperforms all non-metaphorical methods, and even outperforms the metaphorical method of Xu et al. on intention detection.To demonstrate the effectiveness of our fusion strategy involving TransformerEncoder(TE) and dual-stage feature integration, we conduct evaluations for MMMC under following scenarios: (1) single-stage fusion with concatenation, (2) single-stage fusion with TE, and (3) dual-stage fusion with concatenation. In the single-stage scenarios, we incorporate all features together. In the concatenation scenarios, we replace TE with the concatenation operation. Table 4 shows the results of our experiments. We observe that the use of TE instead of concatenation operation yields high enhancement. This indicates that TE is able to capture the inter-feature relationships more effectively. In addition, dual-stage feature integration outperforms single-stage feature integration when using the same fusion method. This suggests that dual-stage feature integration can more effectively utilize the information from different stages, resulting in improved performance in meme understanding. None of the modules share parameters. Table 5 displays the results obtained from our experiments. We observe that the most favorable outcomes are attained when parameter sharing is not employed among the three 𝑀𝐸 𝑘 modules. Conversely, the poorest results are obtained when all three 𝑀𝐸 𝑘 modules share parameters. This finding can be attributed to the inherent divergence in semantic interpretations between the source and target concepts of metaphorical expressions. Consequently, our methodology deliberately avoids parameter sharing to account for these semantic discrepancies more effectively. In Figure 6 , we present a case study. The meme contains metaphor message "train is sweat", and conveys happy emotion and offensive intention. For this meme, NYCO-TWO fails to predict both its emotion and intention. Emotion is mispredicted as love and intention is mispredicted as expressive. With additional linguistic features, the method of Xu et al. correctly predicts its emotion but mispredicts its intention to purely entertaining. Different from them, MMMC correctly predicts both its emotion and intention using multimodal metaphorical features. This shows that with the addition and enrichment of metaphorical features, the understanding of this meme is gradually enhanced.  
conclusion and limitation: 
 In this paper, we proposed a novel meme classification method MMMC that generates multimodal metaphorical features to improve meme understanding. We leverage a text-conditioned GAN, and generate visual metaphorical features by mapping textul metaphor data into visual feature space. Furtherly, we design a classification model that incorporates meme and its multimodal metaphorical features. We perform our method MMMC on a MET-Meme dataset, which contains adequate metaphorical memes. Experimental results show that MMMC significantly outperforms other existing baselines.Our work has some limitations. MMMC needs human-annotated metaphor texts to generate visual metaphorical features. However, we can not obtain human-annotated metaphor texts when given an unknown meme. Therefore, we plan to leverage a method that can automatically detect metaphor texts in future work. Furthermore, the analysis of memes is inextricably linked to their broader context, including cultural backgrounds and the associated posts. Thus, our future work includes refining how to embed this context into meme understanding. 
: 
 Memes, commonly in the form of an image and accompanied text, are known as cultural elements that are widely spread among social media platforms (e.g., Twitter and Reddit) [5] . It has been shown that strong correlation exists between memes and the emotions of those users who produce and share these memes [8] . Therefore, improving computational understanding of memes is considered as of great importance to social media data analytic, with the potential of yielding further benefits to various tasks that are heavily reliant on social media data, such as personalized recommendation, question answering, and open-domain dialog systems.Recently, various meme classification tasks have been proposed, including emotion classification [23] , intention detection [30] , and offensive detection [12] . However, performing meme classification is challenging due to its intricacy: memes often convey metaphorical concepts (e.g., humorous, satirical, and symbolic meanings) that require careful interpretation to understand the actual emotions behind. Moreover, metaphorical information is conveyed as multimodal data involving both textual and visual information. Figure 1 (a) illustrates an example, where negative emotion is expressed by metaphorizing knowledge as iceberg and comparing the size of iceberg between traditional education and activities. In addition, knowledge is presented in text while iceberg is presented in image. If a model ignores the implicit metaphorical message "iceberg is knowledge", it will be difficult to understand that the image is conveying negative emotion. Similarly, in Figure 1 (b), the negative view that "2020 is a crack of life" is expressed by metaphorizing 2020 as a crack in highway. 2020 is presented in text while crack is presented in image -if a model ignores the metaphorical message "2020 is crack", this meme is more likely to be classified as neutral.To address the above issue, recent studies have attempted to add human-annotated metaphor to improve models' understanding of meme. Xu et al. [30] produced a meme dataset containing rich metaphors, called MET-Meme, and showed that meme understanding can be enhanced by adding human-annotated metaphors as extra features of memes. In MET-Meme, a metaphor consists of a source concept (e.g., knowledge in Fig. 1 ) and a target concept (e.g., Figure 2 : A sample metaphorical meme in MET-Meme, which contains the metaphorical message "electric iron is yacht". iceberg). MET-meme includes manually annotated source/target concepts. Figure 2 presents an example. The meme contains the metaphorical message "electric iron is yacht", where the metaphor source concept is yacht and the metaphor target concept is electric iron. Besides, Hwang and Shwartz [11] also created a metaphorical meme dataset called MemeCap, and conducted experiments on meme caption generation. However, these studies mainly focus on formulating the linguistic representations of metaphors. In this paper, we show that visual features can provide complementary information to linguistic features of a metaphorical message, and that a multimodal representation of metaphor can improve more accurate meme understanding [26, 32] . This is because understanding a metaphorical message often requires linking multiple concepts together: a model can only effectively link the "entity" in a meme text and "object" in a meme image, if the model understands the common appearance of a textual entity. Similarly, in Figure 1 , a model will not even link the iceberg in meme image with the text "iceberg", let alone noticing its size difference above and in water, unless the general appearance of iceberg is provided to the model.Hence, we propose an approach for multimodal metaphorical meme classification (MMMC), which generates multimodal metaphorical features to improve meme understanding. MMMC generates visual features from textual metaphorical features to obtain multimodal features of memes. In particular, we develop a generative adversarial network (GAN), which maps the textual description of a metaphor concept in a meme to a visual feature space. Thereafter, MMMC feeds the meme with its multimodal metaphorical features for classification, where a multi-stage feature integration method is deployed. We conduct experiments on the benchmark MET-Meme dataset. Experimental results show that MMMC significantly outperforms existing baseline methods on the emotion classification and intention detection tasks. The contribution of this paper can be summarized as:• To the best of our knowledge, MMMC is the first model that generates multimodal metaphorical features to improve meme understanding. • We design a novel meme classification model that integrates a meme and its multimodal metaphorical features for enhanced classification accuracy. • We evaluate our method on the MET-Meme dataset. The experimental results demonstrate the effectiveness of our method MMMC. Our source code and data are released for knowledge sharing. 
 
 Appendix: The labels of emotion classification include: happiness, love, anger, sorrow, fear, hate, and surprise. The labels in intention detection include: interactive, expressive, purely entertaining, offensive, and other 


Decision: The manuscript presents a generative framework for meme understanding using GAN models to create visual metaphors. The paper's strengths include novel features that improve the multimodal understanding of meme content, the potential to incorporate social science theories, and the ability to harness adversarial training to capture rich visual semantics. The model significantly surpasses SOTA's performance in meme classification and pioneers the use of metaphorical images, with the paper's clarity being highly commendable. However, several limitations need addressing. The framework's high computational expense requires detailed analysis, and diffusion models might yield better visual metaphors. The disconnect between emotion classification and intent detection needs clarification, and fairness in the BLIP-2 comparison should be examined, including evaluation with larger multimodal models like Flamingo and GPT-4. The reliance on annotated metaphor text restricts applicability to the MET-Meme dataset, and the significant time required for generating metaphorical images raises sustainability concerns. Authors are asked to address these concerns in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper introduces MMMC, a framework to generate metaphorical features in memes for better meme understanding. MMMC uses GANs to create visual metaphors, which are fed into classifiers for downstream tasks.
strength_weakness: strength_weakness: 1. The finding that visual metaphors are better than text metaphors is interesting.
2. MMMC uses multimode encoders with multiple representation learners for each modality.
3. Visual metaphors reamin unexplored in this domain but are crucial for creative meme understanding and content moderation.
4. A GAN is intuitively used in MMMC for visual metaphor generation, boosting meme understanding for downstream classifiers.   
5. The comparison with previous methods is sound and highlights the contribution of each aspect of MMMC towards its performance.
limitations: limitations: 1. With generative models for meme understanding, this framework may be very computationally expensive for real-world usage. This paper could include more details about this.
2. While the visual metaphors generated by the GAN are decent, wouldn't diffusion models create much better metaphors?
3. The disconnect between emotion classification and intent detection, i.e. how most models are closer to MMMC for emotion classification than intent detection, could be explained better.
paper_summary: paper_summary: This paper proposes a method introducing metaphorical labels into the model for the meme classification task, named MMMC. It uses real images collected from the Internet to train a GAN to generate visual features of certain metaphor concepts, and then use multi-head attention to fuse these features for classification. Experiments on the MET-Meme dataset show the improvement of this method.
strength_weakness: strength_weakness: - The code is published, which means it is easy to reproduce.
- Experiments show that this method outperforms the baselines in the current evaluation framework.
- The absolution studies are sufficient.
limitations: limitations: 1.  I wonder whether BLIP-2 in Table 3 is fine-tuned on MET-Meme. If not, it seems to be unfair to BLIP-2.
2.  It would be better to compare MMMC with some multimodal large language models (larger than BLIP-2) , e.g., Flamingo[1] and GPT-4[3], given that some studies [2,3,4] find that multimodal large language models can understand metaphors even in zero-shot or few-shot learning.
3.  Some writing suggestions:

- Reference [34] appears to have a formatting error.
- In line 339, "inputc" -> "input c".

> [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems, 23716–23736.
>
> [2] EunJeong Hwang and Vered Shwartz. 2023. MemeCap: A Dataset for Captioning and Interpreting Memes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 1433–1445.
>
> [3] OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774, (2023).
>
> [4] Linhao Zhang, Li Jin, Guangluan Xu, Xiaoyu Li, Cai Xu, Kaiwen Wei, Nayu Liu, and Haonan Liu. 2024. CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition. Proceedings of the AAAI Conference on Artificial Intelligence 38, 8 (2024), 9341–9349.
paper_summary: paper_summary: Memes often incorporate metaphorical elements, necessitating a model's comprehension of both textual and visual metaphors. While previous studies typically focus solely on textual metaphors, this paper introduces a model that integrates metaphorical images. Utilizing the MET-Meme dataset, this dataset has human annotated metaphorical text for each meme. The authors initially train a Generative Adversarial Network (GAN) to generate metaphorical images based on input metaphorical text. Subsequently, the model performs meme classification by combining original text, original images, metaphorical text, and metaphorical images. Experimental results demonstrate the superior performance achieved by including metaphorical images, highlighting the significance of both textual and visual metaphors in meme comprehension.
strength_weakness: strength_weakness: 1. Performance: This paper introduces a model that significantly surpasses state-of-the-art models in meme classification.
2. Novelty: This paper pioneers the utilization of metaphorical images in meme classification, highlighting the crucial role of metaphorical information, particularly metaphorical images, in understanding memes.
3. Clarity: The clarity and accessibility of this paper's writing are highly commendable.
limitations: limitations: 1. Model Construction: Why utilize a GAN model to generate metaphorical images for each textual metaphor instead of directly using online memes(select the one with highest similarity to the textual metaphors)? Could this approach impact model performance?
2. Applications: Due to its reliance on annotated metaphor text, this paper is limited to the MET-Meme dataset and cannot demonstrate its superior performance across other datasets.
3. Insufficient Evaluation: While the dataset comprises metaphor source text and metaphor target text, indicating a distinction between the two, the paper does not delve into their respective influences on the model.
paper_summary: paper_summary: This paper introduces a novel feature (visual metaphors) into the meme classification task and demonstrates improvement in utilising multimodal features as opposed to text-based features. It utilises a GAN-based approach for object synthesis to perform representation mapping to identify metaphor imagery. This is an interesting approach to enriching the visual-language domain with knowledge for implicit reasoning which are prevalent in the understanding of meme content. The authors derived a baseline modelling approach and performed ablation studies to provide empirical evidence to support the importance of the novel features, architecture design and strategies.
strength_weakness: strength_weakness: Strengths of the paper:
- Identification of novel features to improve multimodal understanding of meme content. Looking beyond the literal visual imagery to infer second-order visual-text mismatch to derive inherent emotions and intentions which are non-obvious.
- Has potential to incorporate social science theories on literary analysis for memetic content similar to analogies and sacarsm tasks for visual metaphors.
- Harnesses the strength of adversarial training to capture rich visual semantics for mapping framework, thereby allowing a stronger interaction just by using simple fusion architectures i.e. concatenation, single-stage, dual-stage etc. to achieve improvements.
- Paves the way forward for further research into emotive understanding of complex visual content to identify possible cultural references, and subtle narrative nuances for a difficult problem in automated social media content analysis and understanding.
limitations: limitations: There are a couple of points that I would need to address in the paper to clarify and hopefully, improve the paper's standing:

- What is the computational resources for Visual Metaphorical Feature generation? The paper did not highlight in terms of overheads for the feature generation from the metaphor concepts to obtain the representation for the multimodal encoder. While the evaluation shows a +2-7% improvement on downstream tasks for emotion classification and intent detection, it is difficult to appreciate the extent of improvement if the end-to-end framework requires a significant processing time for inference.

- The metaphorical categories are manually collected to reflect real-world usage of the content in memes, however there is no study on the effectiveness of the types of metaphorical content used. I believe the representations changes from culture to culture, and location as well as context plays an important role in this as well. Has this been effectively studied? In addition, how do the authors envisage this framework to scale and generalise across different meme contexts. How do we automatically enrich the metaphor categories and texts to reflect and capture social and cultural norms.

- In terms of architecture, the metaphor source and target images are considered "local-context" representations which are fed into the MEs for feature extraction, while the meme image is considered a "global-context" which is encoded by MEo. While the Swin transformer establishes a hierarchical structure for feature extraction, the alignment between local and global seems to be weak here. Is there any consideration to incorporate object-level semantics as most of the metaphorical features are related to objects rather than whole images in general?

- Under section 5.6.4 "Generalizability" - I thought this section was to address the generalizability of the model to various data domains rather than model structures (as this is the general consensus for model generalizability)? I think it would be more attractive to analyse domain distribution changes and how such metaphorical features introduces any form of invariance to such shifts.

- Additional points on paper formatting. There are a couple of spelling mistakes that needs to be fixed i.e. "Swin" not "Swim" transformers, "multimode" etc.