MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing (2024)

Abstract: Memes have evolved as a prevalent medium for diverse communication, ranging from humour to propaganda. With the rising popularity of image-focused content, there is a growing need to explore its potential harm from different aspects. Previous studies have analyzed memes in closed settings – detecting harm, applying semantic labels, and offering natural language explanations. To extend this research, we introduce MemeMQA, a multimodal question-answering framework aiming to solicit accurate responses to structured questions while providing coherent explanations. We curate MemeMQACorpus, a new dataset featuring 1, 880 questions related to 1, 122 memes with corresponding answer-explanation pairs. We further propose ARSENAL, a novel twostage multimodal framework that leverages the reasoning capabilities of LLMs to address MemeMQA. We benchmark MemeMQA using competitive baselines and demonstrate its superiority – ∼18% enhanced answer prediction accuracy and distinct text generation lead across various metrics measuring lexical and semantic alignment over the best baseline. We analyze ARSENAL’s robustness through diversification of question-set, confounder-based evaluation regarding MemeMQA’s generalizability, and modality-specific assessment, enhancing our understanding of meme interpretation in the multimodal communication landscape.1

heading: 1 Introduction
text: Memes offer an accessible format for impactful information dissemination for everyone without conventional dependencies of proper formatting or formal language. It provides an easy opportunity for novice content creators and seasoned professionals to propagate information that may
* denotes equal contribution 1CAUTION: Potentially sensitive content included; viewer
discretion is requested.
sometimes be harmful to the general audience, especially in the age of Internet virality. Previous work has explored aspects such as harmfulness in various forms, such as hate speech (Kiela et al., 2020a), cyber-bullying (Sharma et al., 2022b), and offensive languages (Shang et al., 2021), of memes, typically in a black-box setting.
Memes, with their appealing format and influential nature on social media, necessitate the modeling of complex aspects like harmfulness, targeted social groups, and offensive cues to assess their narrative framing and ensure online content safety. Their growing prevalence as a key medium for information dissemination poses significant societal challenges. A question-answering setup, particularly open-ended or instruction/response formats, offers a user-friendly method for probing models about the potential harmfulness of memes and understanding their responses. This approach enhances model interpretability and serves as an effective tool for content moderation.
In this work, we explore contextualized semantic analysis of memes by introducing a novel multimodal task, MemeMQA (c.f. Fig. 1), which is formulated as follows: Given a meme and a structured question about the semantic role assigned to various entities, (a) deduce the correct answer entity from a set of multiple options, while also, (b) generating succinct explanations towards the answer.
5042
Building on the work of (Sharma et al., 2022c), we explore the narrative framing of entities like well-known individuals and political figures in online memes. This research is especially important during critical events like elections or pandemics, where the risk of spreading harmful content such as hate speech and misinformation increases, highlighting the need for effective moderation. We adopt terms like ‘hero’, ‘villain’, and ‘victim’ from (Sharma et al., 2022c) to analyze memes’ intentions of victimization, glorification, and vilification. Our goal is to deepen the understanding of these memes and contribute to making social media safer. The MemeMQA framework is designed to assist social media users and fact-checkers in evaluating the harmfulness of memes, enabling them to ask questions and receive accurate, informed responses.
Analyzing memes in MemeMQA is complex due to their nuanced meanings that demand advanced reasoning, including common sense, and cultural understanding. For instance, the meme in Fig. 1 could simultaneously highlight the role of immigrants in America and promote the Libertarian Party. To correctly answer the question “Who is lauded in this meme?”, it’s essential to grasp the meme’s key themes and the implied message about immigrants enriching diversity, which directly glorifies them. Therefore, “immigrants” is the most suitable answer in this context, rather than “Libertarian Party” or “America”, which, despite being referenced positively, would lead to an incorrect conclusion.
In summary, we introduce a new task for answering and explaining multiple-choice questions about political memes, creating a dataset (MemeMQACorpus) with 1, 880 questions for 1, 122 memes using ExHVV dataset (Sharma et al., 2023). We benchmark MemeMQACorpus with various unimodal and multimodal baselines, including recent multimodal LLMs, and propose ARSENAL, a novel modular approach that leverages multimodal LLM reasoning capabilities. ARSENAL includes rationale, answer prediction, and explanation generation modules. We analyze and compare the performance of ARSENAL against these baselines, highlighting its strengths and limitations. Our contributions are summarised as follows2:
1. MemeMQA: A novel task formulation that introduces a multimodal question-answering setup in the context of memes. 2Supplementary accompanies the source codes and sample dataset.
2. MemeMQACorpus: An extension of a previously available dataset to introduce a set of diverse questions and multiple choice settings for MemeMQA. 3. ARSENAL: A multimodal modular framework system architecture that leverages multimodal LLM generated rationales for MemeMQA. 4. An exhaustive study in the form of benchmarking, prompt evaluations, detailed analyses of diversified questions, confounding-based crossexamination, implications of multimodality and limitations of the proposed solution.

heading: 2 Related Work
text: This section provides a concise coverage of prominent studies on meme analysis, while also reviewing contemporary works within the domain of Visual Question Answering. Finally we consolidate our assessment of the current state-of-the-art in Multimodal LLMs.
Studies on Memes. Recent collaborative efforts encompass diverse meme analysis aspects, including entity identification (Sharma et al., 2022c; Prakash et al., 2023), emotion prediction (Sharma et al., 2020) and notably, hateful meme detection (Kiela et al., 2020a; Zhou et al., 2021) through methods like fine-tuning Visual BERT, UNITER (Li et al., 2019; Chen et al., 2020), and dual-stream encoders (Muennighoff, 2020; Sandulescu, 2020; Lu et al., 2019; Zhou et al., 2020; Tan and Bansal, 2019). Further studies address anti-semitism, propaganda, harmfulness (Chandra et al., 2021; Dimitrov et al., 2021; Pramanick et al., 2021b; Suryawanshi and Chakravarthi, 2021; Prakash et al., 2023; Sharma et al., 2022a), while recent research explores multimodal evidence prediction, role-label explanations (Sharma et al., 2023), and semantic analysis of hateful memes (Hee et al., 2023; Cao et al., 2022; Chen et al., 2023). Most of these studies are constrained by the schema and quality of the annotations while limiting the openended probing of memetic phenomena.
Visual Question Answering (VQA). This subsection explores the evolution of VQA research. Initial pioneering work by Antol et al. (2015) emphasized open-ended questions and candidate answers. Subsequent studies introduced variations, including joint image and question representation, to classify answers (Antol et al., 2015). Researchers further explored cross-modal interactions
5043
using various attention mechanisms, such as coattention, soft-attention, and hard-attention (Lu et al., 2016; Anderson et al., 2018; Malinowski et al., 2018). Notably, efforts were made to incorporate common-sense reasoning (Zellers et al., 2019; Wu et al., 2016, 2017; Marino et al., 2019). Models like UpDn (Anderson et al., 2018) and LXMERT (Tan and Bansal, 2019) harnessed nonlinear transformations and Transformers for VQA, while addressing language priors (Clark et al., 2019; Zhu et al., 2020). In a standard VisualQuestion-Answering framework, an image is presented alongside a related question and, depending on the setup, multiple-choice options. Memes, however, introduce a more complex layer, combining images with frequently mismatched textual content, making the task more challenging and far from straightforward.
Multimodal Large Language Models. The rise of large language models (LLMs) like ChatGPT (OpenAI, 2022), GPT4 (OpenAI, 2023), Bard (GoogleAI, 2023), LLaMA (Touvron et al., 2023), Vicuna (Chiang et al., 2023), etc., has brought significant advancements in natural language understanding and reasoning. Their affinity towards multimodal augmentation is also reflected for visuallinguistic grounded tasks. Such models augment LLMs via fusion-based adapter layers, to excel at various tasks, from VQA to multimodal conversations (Alayrac et al., 2022; Awadalla et al., 2023; Liu et al., 2023a; OpenAI, 2023; Zhu et al., 2023; Gong et al., 2023; Zhao et al., 2023). However, existing multimodal LLMs like LLaVA (Liu et al., 2023a), miniGPT4 (Zhu et al., 2023), and multimodalGPT (Gong et al., 2023) exhibit limitations in grasping nuances like sarcasm and irony in visuallinguistic incongruity seen in memes. Although few similar works address meme-related tasks, it’s mainly limited to visual-linguistically grounded settings of caption generation and VQA (Hwang and Shwartz, 2023). For a more comprehensive range of tasks, they exhibit limitations inherent to LLMs, like pre-training biases and hallucinations (Zhao et al., 2023).
The dual objectives of MemeMQA, encompassing answer prediction and explanation generation, present unique challenges. Existing methods fall short, including the Multimodal CoT (MM-CoT) model (Zhang et al., 2023), a two-stage framework combining DETR-based visual encoding (Carion et al., 2020) and textual encoding/decoding from
unifiedqa-t5-base3. MM-CoT excels in answer prediction but falters in explanations. Instructiontuned multimodal LLMs like LLaVA, InstructBLIP (Dai et al., 2023), and miniGPT4 show promise in understanding meme semantics but struggle with question-specific accuracy, prioritizing broader meme context over precise answers. In this work, our focus is on addressing challenges pertaining to complex visual-semantic reasoning, posed by MemeMQA task while considering limitations in current multimodal LLMs and neural reasoning setups for question-answering.
3 The MemeMQACorpus Dataset
Current meme datasets typically encompass either categorical labels (Kiela et al., 2020b; Pramanick et al., 2021a; Shang et al., 2021) or their associated explanations (Sharma et al., 2023). Although conventional Visual Question Answering (VQA) (Antol et al., 2015; Lu et al., 2016) frameworks exist, they lack the nuanced complexity of memes. These include tasks like detection, segmentation, conditional multimodal modeling (such as caption generation, visual question answering, and multiple-choice VQA), and strong visual-linguistic integration (e.g., setups similar to MS COCO for question-answering that focus on common-sense and objective reasoning) (Antol et al., 2015; Lu et al., 2016). While these areas present their distinct challenges and mark a significant line of inquiry within the intersecting realms of computer vision and natural language processing (multimodality), they fall short of addressing the complexities of multimodal reasoning, abstract idea representation, and the nuanced use of language mechanisms like puns, humor, and figures of speech, etc. These elements are often integral to memes. This oversight has generally curtailed the effectiveness of existing multimodal approaches (Pramanick et al., 2021b) in capturing the nuanced complexities inherent to memes.
To address this gap, we introduce MemeMQACorpus, a novel dataset designed to emulate free-form questioning and multiple-choice answering. Given the overwhelming diversity of possible question-answer pairs for the multifarious phenomena presented in memes, we supplement ExHVV (Sharma et al., 2023), an existing multimodal dataset consisting of natural
3https://huggingface.co/allenai/ unifiedqa-t5-base
5044

heading: VillainVillain

heading: Role-label Counts (%) Synonyms
text: language explanations for connotative roles for three entity types - heroes, villains, and victims, across 4, 680 instances for 3K memes, with automatically constructed, structured questions. This expansion aims to emulate the intricacies of meme interpretation and communication via a QnA setup.
Our approach entails crafting structured question sets for distinct role categories – hero, villain, and victim, each linked to unique entities featured in ExHVV’s memes, as depicted in Fig. 2. The central goal is to create role-based queries that precisely elicit only one entity as the correct answer within a multiple-choice setting. For instance, when an entity such as Democratic Party is labeled as a villain in a meme, along with the availability of a corresponding explanation from the ExHVV dataset, we formulate the question: “What is slandered in this meme?” (c.f. Fig. 2). With Democratic Party as the correct choice, distractive choices for answer options are selected based on entities referred within the meme, sampled randomly from the ones not sharing the role label with the ground truth. Sampling occurs from the entire training corpus in cases with insufficient valid entity choices. Additionally, to emulate free-form questioning by increasing the question variability, we incorporate various synonyms of role labels (hero, villain, and victim). The role-synonym mapping and their propor-
tional breakdown, integral to constructing queries in MemeMQACorpus, are shown in Table 1. Our curation effort encompasses 1, 880 meme-question pairs, corresponding to 1, 122 distinct memes about US Politics. This domain choice is based on diversity in the entity distribution across different roles compared to the other subset (on Covid-19) of ExHVV dataset. To further examine the robustness of different modeling approaches, we curate additional variants of MemeMQACorpus, with (a) Question Diversification, and (b) Confounding Analysis, the details of which are presented in Sec. 7. As our question enhancement approach is automated, we achieve valid questions seamlessly linked to ExHVV instances, relying on the pre-existing annotations.

heading: 3.1 Prompting for Question Diversification
text: To achieve diversity in the framing of the original questions, a pre-trained LLM, Llama-2-7b-chat, is utilised for inferencing via zero-shot prompting. In this setting, the LLM is provided a context about the setting of the question which is followed by asking the model to rewrite the question in multiple ways without changing the meaning of the quetion. This ensures that the original meaning and, hence, the validity of the original option set remains intact. One out of the five rephrased questions provided by the LLM is then chosen at random. This chosen question replaces the original question in
5045
Pr om
pt C
on fig
ur at
io ns
QCM-LEA (AT5B)
QCML-AE (AT5B)
QCM-L, QCMG-AE (T5L) QCM-L, QCMG-AE (AT5B)
QCM-ALE (AT5B)
QCML-A (AT5B)
QCML-AE (T5L) QCM-LE, QCMG-A (AT5B)
QCML-E, QCMG-A (w/o caption) - (T5L)
QCM-LE, QCMG-A (T5L) QCML-E, QCMG-A (AT5L) QCML-E, QCMG-A (T5L)
MemeMQACorpus, extending the questioning style of MemeMQACorpus to emulate free-form question answering more closely.
4 The ARSENAL Model
Prior to exploring an effective design towards addressing MemeMQA, we analyze different prompting configurations using meme-based inputs to determine the optimal strategy. This section begins by outlining the optimal prompting strategy, then details the structural aspects of ARSENAL.
Prompting Configurations: In multimodal question-answering with CoT reasoning (Zhang et al., 2023), the setup includes a question, context (text associated with an image), options, lecture (detailed generic context), explanation (a concise contextual statement), answer, and intermediate generated text.4 Prompt configurations are represented as input→output, combining elements from QCMLEAG. Prior one-stage approaches (QCM→LA or QCM→AL) have limitations, prompting a two-stage setup with improved performance (Zhang et al., 2023). Since MemeMQA involves more complex reasoning than ScienceQA (Lu et al., 2022), we first examine 11 prompt configurations for MemeMQA, with lectures (L) as detailed role definitions, using one/two-stage methods and base models unifiedqa-t5-base/large (AT5B/L) and t5-large (T5L). Our findings (c.f. Fig. 4) corroborate the applicability of the two-stage
4Unless stated otherwise, these are typically abbreviated as QCMLEAG – question Q, context C, multiple options M, lecture L, explanation E, answer A, and generated intermediate text G.
framework for MemeMQA.5

heading: 4.1 System Architecture
text: Our input consists of three parts: (i) the meme image, MemeI , (ii) the OCR Text, MemeT , and (iii) the question Q with its corresponding multiple options M. The expected output consists of two parts – (i) the answer, Yanswer, and (ii) the explanation, Yexp. We propose a multi-stage setup for ARSENAL to leverage individual strengths of MMCoT and multimodal LLMs towards the overall objective of MemeMQA. The framework is a two-stage process consisting of answer prediction and explanation generation. It has a modular design, incorporating LLM-inferred rationale in both stages. The initial stage includes two steps: generating an intermediate rationale and predicting the answer, while the second stage focuses on generating explanations. A schematic of the proposed framework is depicted in Fig. 5.
Rationale Generation: We curate “generic rationale,” Rgeneric, offline in the first stage to provide semantic information about the meme in a textual form, which is generally not captured well by the OCR information alone. Rgeneric is developed using the multimodal LLM, LLaVA-7B using zeroshot inference with the prompt, Pgeneric “Explain this meme in detail.” The multimodal LLaVA-7B LLM is built on the base LLM, Vicuna-7B. The Rgeneric thus generated captures relevant semantic information deemed useful for providing semantic clues in further stages of the proposed framework. This process can be expressed as follows:
Rgeneric = ModelLLaV A(MemeI , Pgeneric) (1)
In the second stage, LLaVA-7B model is again used to generate an “answer-specific rationale,” Rspecific, by prompting the model with a combination of the answer generated in the first stage and an answer-specific prompt, Pspecific. Pspecific is of the form – “How is [answer] [rephrased question]”, where the rephrased question is framed by removing the first two words of the question. For example, for a question, Q, given as “Who is victimised in this meme?” with the answer ‘Joe Biden’, the rephrased question would be given as “How is Joe Biden victimised in this meme?”. This is represented as,
Rspecific = ModelLLaV A(MemeI , Pspecific) (2) 5Refer App. D for more details on Prompting Configura-
tion Assessment.
5046
Stage 1 - Answer Prediction: This stage implements Mutimodal CoT model with two-stage training. It uses the T5-large model with the prompting strategy of QCM→LE followed by QCMG→A. The model is provided with visual data in the form of embeddings obtained from the DETR model. These embeddings are used by adding a gated-cross attention layer in the encoder stack of the T5 model as follows: Hfuse = (1−λ)·Hlanguage+λ·Hattnvision , where λ is the sigmoid-activated output of fused image+text embeddings, Hlanguage: text-input embeddings and Hattnvision: output of text+vision crossattention. We provide this model with additional contextual cues from Rgeneric. The model is then fine-tuned on MemeMQACorpus, with the first step of training for five epochs being a text generation task with the objective of generating text, G, of the form Rgeneric [SEP ] Yexp.
G = Modelmm−cot(MemeI ,MemeT , Q,M) (3)
This is followed by another training step for five epochs to fine-tune the model for generating Yanswer.
Yanswer = Modelmm−cot(MemeI ,MemeT , Q,M,G) (4)
Stage 2 - Explanation Generation: The second stage focuses on generating an explanation for the answer obtained from the previous stage. To this end, the LLaVA-7B model is used again for its superior reasoning capacity to generate an answer-specific rationale, Rspecific. This provides us with a highly informative rationale that focuses specifically on the chosen answer and provides a
highly relevant explanation. However, this generation lacks the structure and the conciseness of the expected explanation. To this end, Rspecific is provided along with the question and correct answer to a unimodal T5-large model for text-to-text generation. This T5-large model is fine-tuned for two epochs in a text-to-text generation setting for generating the expected explanation. The prompt PT5, given to the T5 model, is “Summarize the explanation for question based on the answer”. The task of T5 is as follows:
Yexp = ModelT5(PT5, Q, Yanswer, Rspecific) (5)
While we fine-tune it for the conditional generation objective and obtain the T5-decoder’s language modeling loss LEXP = − log(pyt) = − log(p(yt|y<t)). The resultant explanation is combined with the previously obtained answer to obtain our final result of the form – “Answer: [answer] BECAUSE [explanation]”.

heading: 5 Experiments
text: In our study, ARSENAL is rigorously tested against various models, with results averaged over five runs. The MemeMQA task involves two components: answer prediction and explanation generation, each evaluated using different metrics. Answer prediction is measured for accuracy due to entity imbalance and open-ended nature in the ExHVV dataset. Explanation quality is assessed against ExHVV ground truth using metrics like BLEU1, BLEU-4, ROUGE-L, METEOR, CHRF, and BERTScore. Baseline comparisons span uni-modal (text, image) and multi-modal settings. Additionally, ARSENAL’s robustness is evaluated through di-
5047

heading: Type Models Accuracy BLEU-1 BLEU-4 ROUGE-L METEOR CHRF BERTScore
text: verse question types, confounding-based tests, and multimodal and error analyses.
6 Benchmarking MemeMQA
As noted in Table 2, the T5-based text-only model performs well in answer prediction with an accuracy of 0.53, outperforming image and multimodal models. However, its explanations are incomplete, repetitive, and lack coherence, resulting in low ROUGE-L (0.44), CHRF (0.35), and METEOR (0.41) scores, second only to the LLMbased miniGPT model.
The ViT model, a strong unimodal image baseline, has low answer prediction accuracy and fluent yet repetitive explanations like the T5 baseline, quantified by low ROUGE-L (0.45), METEOR (0.44) and CHRF (0.38) scores. Both ViT and BEiT unimodal baselines perform poorly, with BEiT scoring 0.40 accuracy. Multimodal baselines (ViT+BERT, BEiT+BERT) yield answer prediction accuracy (0.45 and 0.44, respectively) similar to that of ViT but slightly outperform the ViT-based unimodal model in terms of the generated explanation qualitatively. This underscores ViT’s robustness over BEiT for unimodal and multimodal settings and, consequently, for the Vicuna-based miniGPT4 and LLaVA-based ARSENAL.
In the closed-form Visual Question Answering domain, we benchmark against models like the multimodal ViLT, which achieves a fine-tuned accuracy of 0.43. LLM-based models like miniGPT4
and GPT3.5 show low answer prediction accuracy in both zero-shot (0.32) and visual descriptionbased fine-tuning (0.28) for the former, and 0.28 for GPT3.5. These models’ explanations lack specificity, as indicated by miniGPT4’s BLEU-1 score of 0.12 and ROUGE-L score of 0.16 post-fine-tuning. Despite their detailed and reasonable reasoning, they fall short in standard evaluations due to their excessive length and nonspecific content. However, BERTScore values of 0.771 for miniGPT4 (FT) and 0.837 for LLaVA-based models suggest a reasonable coherence with the memes in question.
Our primary comparison is to models leveraging the MM-CoT model in various prompt and input settings. The utility of the OCR text is proven by the 8% drop in accuracy on eliminating the OCR text from the input. The addition of generic lectures (L) also improves the model’s performance, with a 2% increment in answer prediction accuracy. Introducing a contextual rationale using zero-shot inferencing using an LLM such as LLaVA presents qualitative improvements in the explanation generation quality of the MM-CoT model.
It is also worth noting, that the MM-COT model underperforms in understanding memes compared to the new model, which excels in accuracy and explanation due to its Rationale Generation Module, offering a deeper contextual grasp of meme content. Table 3 presents the average scores of ARSENAL across the seven primary metrics explored, over five independent runs.
5048

heading: Measures Average Std. Dev
text: Discussion: Our analysis of 60 random test samples compared ARSENAL with other methods in terms of answer quality, explanation coherence, and modality-specific nuances. ARSENAL particularly through the LLaVA approach, excels in reasoning and explaining by effectively integrating details from various meme modalities, as shown in Figs. 6 and 12. In contrast, the MM-CoT model struggles with syntactic and grammatical correctness (c.f. Figs 6, 9, and 10). A T5based text-only model often produces incoherent and incomplete outputs (c.f. Fig. 9). The UM.IMG.ViT.BERT.BERT model faces challenges in contextualization and alignment, with explanations that are semantically related yet irrelevant. Image-only approaches and multimodal baselines show a lexical bias, and the MM.ViT.BERT.BERT multimodal setup, despite striving for fluency, fails in complex reasoning, leading to generic explanations (refer to Figs 6 and 13).6 The performance difference might not be as evident from a 2%
6For more details, see App. E.
quantitative increment observed for a metric like a BERTScore, relative to the 18% enhancement for answer prediction accuracy but is distinctly visible for the demonstrative example depicted in Fig. 6, and Appendix K.

heading: 7 Robustness Analysis
text: A key factor that is expected to characterize the efficacy of a model for a task like MemeMQA, is it’s robustness to variations within the question/answer formulation. This is also critical due to the resultant variability within the LLM’s generated responses (Salinas and Morstatter, 2024). To this end, we examine ARSENAL’s performance in comparison to other contemporary baselines, by (a) Question Diversification, and (b) Confounding Analysis (c.f. Table 4).
Question Diversification: In our analysis, we evaluate the performance of ARSENAL and current baselines using more naturally framed questions than those in MemeMQACorpus. We achieve question diversity by employing the Llama-2-7b-chat model to generate five unique variations of each original question. Each question is then randomly replaced with one of these generated alternatives,
5049
ensuring a wide range of questioning styles.7
As an indicator of the robustness of ARSENAL to diversity in questions, when trained and tested on the new diverse questions, we obtained an answer prediction accuracy of 0.82 (c.f. Table 4). This is a marginal decline from its performance of 0.87 on the original setting, having a structured question set. In comparison, the UM.TEXT.T5 baseline descends from an accuracy of 0.53 to 0.35, UM.ViT.BERT.BERT from 0.46 to 0.27 and MM.ViT.BERT.BERT from 0.45 to 0.34. These results are a clear indication that ARSENAL is able to accommodate significant variations and diversity in the question framing setup while other models are not as robust to these changes.
Confounding Analysis: Our study evaluates the robustness of ARSENAL against contemporary baselines through three confounding settings, crafted to challenge the model with scenarios differing from typical tasks. These settings involve alterations in questions and options. We compare ARSENAL with three contemporary baseline models: UM.TEXT.T5, UM.IMAGE.ViT.BERT.BERT, and MM.ViT.BERT.BERT. Analyzing ARSENAL across the following settings and against these baselines is crucial for understanding its real-world applicability and performance. For detailed information on these confounding tasks, see App. J.
Confounder A – Yes/No Confounding: Transforming dataset to binary ‘yes or no’ questions (50% chance), reshaping ‘yes’ as “Is [answer] [rephrased question]?” and altering ‘no’ by modifying role labels.

heading: Confounder B – None Sampling Across All Sets:
text: Replacing 20% of answers with ‘None’ by swapping role labels, maintaining consistency; Mnew = {M,None} across sets.
Confounder C – None Sampling Across Train Only: Introducing 20% random ‘None’ answers in training; model adapts to ‘None’ while testing remains unchanged; Mnew = {M,None} across sets.
The ’yes or no’ confounding setting in our study allows for assessing the model’s reasoning robustness. Models depending on statistical probabilities fail here, as answers can be paired with either correct or incorrect role labels regardless of their
7Refer App. 3.1 for more details on Question Diversification.
dataset frequency. ARSENAL and UM.TEXT.T5 demonstrate strong reasoning skills, with scores of 0.77 and 0.80 respectively, indicating they rely on reasoning over statistics. In contrast, the UM.IMAGE.ViT.BERT.BERT-based model and MM.ViT.BERT.BERT-based model score poorly at 0.37 and 0.29, highlighting their reliance on statistical likelihoods of answers based on dataset frequency.
We also evaluated the robustness and generalizability of ARSENAL, using two settings involving “None” answers. Notably, only ARSENAL delivers good performance (0.69 accuracy) in the more challenging original testing set compared to the revised set, showcasing its better generalizability despite being trained on valid “None” answers data.

heading: 8 Conclusion and Future Work
text: This study introduced MemeMQA, a task that involves multimodal question answering for imagetext memes, delving into their intricate visual and linguistic layers. Utilizing recently opensourced LLMs, especially their multimodal adaptations, we tackled the challenge of complex, nontrivial multimodal content. Through a new dataset, MemeMQACorpus, we assessed systems’ reasoning in assigning semantic roles to meme entities via question-answering and contextualization based objectives. Our experiments showcased the efficacy of the proposed two-stage training framework, ARSENAL, while leveraging existing language models and multimodal LLMs, to outperform the stateof-the-art by a remarkable 18% accuracy gain. This study reveals the potency and limitations of multimodal LLMs, enabling the scope for sophisticated setups embracing diverse questions, domains, and emotional nuances conveyed through memes. Ultimately, our findings steers future exploration and the development of comprehensive systems dedicated to deciphering memetic phenomena.
Our future aim is to create sophisticated, multiperspective sets for MemeMQA, moving beyond standard QnA towards an optimal multimodal solution.

heading: Acknowledgements
text: The work was supported by Wipro research grant.

heading: Limitations
text: This section highlights ARSENAL’s limitations, including semantically inconsistent rationales, factual errors, and multimodal bias, inherent to
5050
LLaVA’s generation capacity. For some cases, LLaVA’s rationales seem to be mining the inductive biases due to the co-occurrences of disparate keywords while being influenced by LLM’s pretraining corpus and web data, exhibited mostly for missing-modality and high inter-modal incongruity. An example for the latter shown in Fig. 8 (c.f. Appendix I) illustrates how biased inference by LLaVA dilutes ARSENAL’s output due to inaccurate contextualization, whereas MM-CoT deduces the answer accurately, possibly due to standardized definitions being used instead of LLM-based rationales.

heading: Ethics and Broader Impact
text: Reproducibility. We present detailed hyperparameter configurations in Appendix A and Table 5.
User Privacy. The information depicted/used does not include any personal information.
Biases. Any biases found in the source dataset ExHVV are attributed to the original authors (Sharma et al., 2023), while the ones in the newly constructed dataset is unintentional, and we do not intend to cause harm to any group or individual.
Misuse Potential. The ability to identify implied references in a meme could enable wrongdoers to subtly express harmful sentiments towards a social group. By doing this, they aim to deceive regulatory moderators, possibly using a system similar to the one described in this study. As a result, these cleverly crafted memes, designed to carry harmful references, might escape detection, thereby obstructing the moderation process. To counteract this, it is advised to incorporate human moderation and expert oversight in such applications.
Intended Use. We make use of the existing dataset in our work in line with the intended usage prescribed by its creators and solely for research purposes. This applies in its entirety to its further usage as well. We do not claim any rights to the dataset used or any part thereof. We believe that it represents a useful resource when used appropriately.
Environmental Impact. Finally, large-scale models require a lot of computations, which contribute to global warming (Strubell et al., 2019). However, in our case, we do not train such mod-
els from scratch; instead, we fine-tune them on a relatively small dataset.

heading: A Hyper-parameter and Implementation
text: We train all the models using PyTorch on an actively dedicated NVIDIA Tesla V100 GPU, with 32 GB dedicated memory, CUDA-12.2 and cuDNN7.6.5 installed. For all the models with the exclusion of LLaVA and MiniGPT4, we import all the pre-trained weights from the huggingface8 API. Additionally, we used a series of architectural additions and delta weights to obtain LLaVA-7B-v09 from the base LLaMA-7B model available under an academic license from Meta. We randomly initialize the remaining weights.
Most of our models are implemented using the Adam optimiser (Kingma and Ba, 2015) with a learning rates as specified in Table 5, a weight decay of 1e−5. We use a Cross-Entropy (CE) and a language modeling loss (LML) as per the applicability. We conducted a thorough empirical analysis before freezing the optimal set of hyperparameters for the current task for all the models examined. We also early stop to preserve our best state convergence for each experiment. Further details of hyperparameters employed can be referred to from Table 5. On average, it took approx. 2:30 hours to train a typical multimodal neural model on a dedicated GPU system.
For ARSENAL, we use a learning rate of 1e−4, with eps=(1e−30, 1e−3), clip_threshold=1.0, decay_rate=-0.8, weight_decay=0.0. Moreover, we set the max_source_length = 512 and max_target_length = 256 in first-step, QCM-LE task of MM-CoT, max_target_length = 16 in the second-step QCMG-A task of MM-CoT, and max_target_length = 32 in the explanation generation module using T5-Large.
A note on fine-tuning: Fig. 5 illustrates that the Answer Prediction Modules and Explanation
8https://huggingface.co/models 9https://github.com/haotian-liu/LLaVA
Generation Module are fine-tuned (please refer to the symbol legend at the top-right corner of Fig. 5) components within the proposed framework. The Answer Prediction Module implementation follows guidelines from the multimodal questionanswering with CoT study (Zhang et al., 2023). The LLaVa model generates a rationale that is utilized as a proxy for the intermediate rationale for MM-CoT. Furthermore, the Explanation Generation Module is fine-tuned using the T5-Large model, employing a training approach akin to that used for the UM_T5 model, also detailed in the implementation of the source code.

heading: B Text Extraction via OCR
text: The OCR data is part of the original ExHVV dataset, as released with the original work (Sharma et al., 2023), which was extracted using Google GCV OCR10 (GOCR) as primary inputs. The OCR for each meme is available as part of ExHVV, and we have not made any modifications to this data field. Text retrieval through optical character recognition (OCR) is crucial for extracting text from memes. The efficacy of the OCR method impacts the system’s overall performance. Towards examining the quality of the GOCR approach used originally, we examine the text retrieval capabilities of two widely-used OCR-based APIs: Google Tesseract API11 (TOCR) and Google GCV API (GOCR), for this purpose.
Our qualitative assessment of 30 varied memes reveals occasional errors in TOCR and fewer in GOCR. TOCR errors are frequent in challenging scenarios, such as overlapping text and images, low-quality graphics, or small text. In contrast,
10Google Cloud Vision OCR API 11Google’s Tesseract-OCR API
5055
GOCR often outperforms TOCR, even in simpler situations. Figure 7 illustrates the disparity in text extraction accuracy between TOCR and GOCR. The first example in Fig. 7 (left) shows a combination of straightforward and complex elements like clear black text on a white background and intricate visual-text overlaps, where TOCR fails but GOCR succeeds. Conversely, the second example in Fig. 7 (right), a simpler meme, presents more difficulties for TOCR, while GOCR maintains clarity.

heading: C Motivation for ARSENAL’s Design
text: Within the context of reasoning-based questionanswering setup for memes, relevant solutions are scarce within the realm of neural frameworks and multimodal LLMs as we transition between them. For existing multimodal-LLM-based systems, eliciting relevant answers and generating concise explanations for memes is challenging. Strategies would typically solicit systematic instruction-tuning for fine-grained meme-related use cases instead of typical vision+language tasks (Liu et al., 2023a; Zhu et al., 2023; Zhao et al., 2023), which itself has been an active research domain. An alternate solution would be to improve existing multimodalneural frameworks (Zhang et al., 2023) that perform reasoning-based question answering, albeit with constrained reasoning capacity and generative coherence for memes. In this work, we primarily focus on the latter while assessing the potential and limitations of other contemporary solutions.
The capability of the proposed approach (ARSENAL) towards addressing the nuanced complexity posed by memetic content stems from the choice of leveraging detailed rationales generated via multimodal LLM’s, while adapting conventional approaches involving chain-of-thought reasoning which we found in our study are more suited for more accurate answer prediction and focused explanation generation.

heading: D More on Prompting Configuration Analysis
text: Using the allenai/unifiedqa-t5-base-based MM-CoT setup, we first evaluate the optimal ordering of lecture (L), explanation (E), and answer (A) components for MemeMQA. Comparing LEA and ALE configurations, we find a significant 22% accuracy difference, emphasizing ordering importance. The two-stage setup generally outperforms onestage, except for QCML→A (AT5B), suggesting op-
timal answer inference with lecture/explanationbased reasoning. The first-stage training with rationale/explanation benefits QCM→LE and QCML→E configurations. Among three language models (unifiedqa-t5-base/large, t5-large), the two-stage t5-large achieves the highest accuracy of 0.776, slightly better than unifiedqa-t5-large.
It is also worth noting that the accuracy of the two-stage framework, with configuration [QCM→L, QCMG→AE] using allenai/unifiedqa-t5-base comes out to be 1.5% higher than that from t5-large. This could be attributed to the formatagnostic design of the former, the efficacy of which could be best seen for the challenging *→AE-based scenarios in the two-stage setup (see Figure 4). In addition to this, performing inference with AE as outputs mainly yields poor results, as can be observed for the experiments with configurations as QCML→AE (AT5B), [QCM→L, QCMG→AE] (T5L), and [QCM→L, QCMG→AE] (AT5B), on average yielding an accuracy of 0.47. This could be due to the distributional differences between the answer choices and explanations, which the MM-CoTbased setup is unable to adjudicate as part of modeling.
A high-level overview of prompting scenarios: Our experiments utilize prompting across three distinct scenarios and configurations. Sec. 4 addresses the prompting setups for the Multimodal CoT model within the answer prediction module. The prompting structure, as explained in the paragraph on prompting configurations in Sec 4, follows an input→output format, with both the input and output comprising combinations of elements denoted by QCMLEAG. Here, Q stands for Question, C for Context, M for multiple options, L for lecture, E for explanation, A for answer, and G for generated intermediate text. In the ARSENAL framework, a two-stage setup is implemented, with prompts formatted as QCM→LE initially, then QCMG→A. An illustrative example is provided below: QCM - “Question: What is slandered in this meme? nContext: ocr text nOptions: (a) antifa (b) democratic party (c) black community (d) conservatives” LE - “Solution: lecture = generic rationale, R_generic explanation” QCMG - “Question: What is slandered in this meme? nContext: ocr text nOptions: (a) antifa (b) democratic party (c) black
5056
community (d) conservatives ngenerated text” A - “The answer is (a)”. The input for the explanation generation module is detailed in the description leading upto the equation # 5, as ‘Summarize the explanation for question based on the answer. Explanation: R_specific or entityspecific rationale’ Additionally, Sec. 3.1 elaborately discusses the prompt setups used for question diversification.
E Multimodal Analysis of ARSENAL
Cross-modal reasoning is a pivotal aspect of LLaVA’s capability, particularly evident in situations where textual information falls short. Impressively, LLaVA harnesses its adeptness in detailed visual assessment and intricate reasoning, leading to the generation of semantically accurate rationales, as depicted in Fig. 14, 15, and 16. However, the landscape of cross-modal noise, demonstrated by the example in Fig. 17, introduces an intriguing challenge. This pertains to cases like visual exaggeration, where multimodal models tend to anchor their explanations across multiple modalities without a clear emphasis on a primary one, which could otherwise be self-explanatory. On a related note, the phenomenon of multimodal hallucinations, represented by Fig. 18, 10, 19, 20, 21, and 22, brings about an intriguing facet of LLaVA’s capabilities. In these instances, the model’s explanations may indeed prove accurate, despite the rationales not always aligning with factual accuracy. Such discrepancies might arise due to extrapolated ideas or statements, as well as visual misinterpretation, yet these rationales consistently maintain a high degree of semantic relevance, an observation supported by Fig. 10 and 23. In light of these intriguing insights, multimodal analysis error analysis emerges as a critical component for understanding LLaVA’s performance and refining its cross-modal reasoning and explanation generation abilities.

heading: F Difference with MM-CoT framework
text: The original MM-COT model, while being a strong comparative baseline, lags behind the proposed model, both in terms of answer prediction accuracy and explanation generation quality (18%-Accuracy and 2%-BERTScore performance difference w.r.t. ARSENAL), because of its inability to interact and reason well w.r.t. Visual-linguistic semantics of memes. Memes require a deeper understanding of the humour, sarcasm, and hidden meaning of the
content, which the MM-COT model is observed to fall short of. The introduction of the Rationale Generation Module is a major contributing factor in the performance of the proposed framework as it provides deeper contextual information about the meme.

heading: G Comparison with GPT 3.5 and GPT4
text: As a proxy for comparison with the closed and commercial models like GPT-3.5 and GPT-4, we have provided a comparison with open-source multimodal LLM alternatives in Table 2 in the form of a comparison with LLaVA and miniGPT4 (in zero-shot and fine-tuned settings). The primary reason for this comparison was the accessibility of the technical and background details of these systems in the public domain and to encourage healthy competition within open-source community, especially considering their impressive performance on various multimodal tasks like miniGPT4 exhibiting various emerging capabilities (Liu et al., 2023b) and LlaVA achieving SOTA on 11 benchmarks (Zhu et al., 2023), with rarely any in-depth study w.r.t content like memes, which have very strong visual-linguistic incongruity, in contrast to typically visual-linguistic grounding tasks and datasets.

heading: H A note on Ablation Study
text: Our ablation analysis begins with a detailed discussion on the investigating Prompting Configuration (c.f. Sec. 4, second paragraph, and Fig. 4), and is then reflected as part of Benchmarking ARSENAL (c.f. Sec. 6, and Table 2). The specific experiments reflecting the ablation results are outlined below:
Prompting Configuration (c.f. Fig. 4): We have explored various permutations of the elements denoted by the acronym QCMLEAG (Question Q, Context C, Multiple Options M, Lecture L, Explanation E, Answer A, and Generated Intermediate Text G). These elements are crucial to the task and solution framework proposed (ARSENAL), with the goal of
5057
identifying the most effective input-output configurations for the foundational multimodal framework. These experimental explorations were carried out using initial lectures (excluding the more complex LlaVa-based justifications).
Benchmarking MemeMQA (c.f. Table 2): The experiments labeled under “Model” entries such as MM-CoT (without OCR), MM-CoT, MM-CoT (with Lecture), MM-CoT (QCML→A, with LLaVA rationales), ARSENAL (with Entity-Specific Rationale), and ARSENAL (with Generic Rationale), collectively contribute to the ablation analysis for ARSENAL. These experiments cover both the basic MM-CoT frameworks and the evolving ARSENAL variants, leading up to the solution ultimately proposed.

heading: I Error Analysis
text: Among various errors in ARSENAL’s outputs, we found errors due to (a) semantically inconsistent rationales caused by LLaVA, (b) factually incorrect rationales, and (c) multimodal bias. Semantically inconsistent rationales are prominent when high inter-modal incongruity occurs. Illustrated in Fig. 8 (c.f. Appendix I), a biased inference towards the ‘democratic party’ by LLaVA leads to incorrect predictions in ARSENAL. Despite a green triangle and the term party in the meme, the model lacks cues to understand context. It seems to capture inductive biases from the co-occurrence of ‘party’ and ‘ban’, likely influenced by media coverage and the LLM’s training. Whereas, MM-CoT approach accurately predicts the meme’s answer and produces somewhat aligned explanations. This is achieved through standardized definitions replacing rationales, aiding the T5 model’s inference to connect
visual elements and text to the second option.12
The LLMs are instruction fine-tuned for controllable behavior, so if a meme has something controversial, there is a higher chance, the LLM would attempt to normalize the harm intended within the meme, by attributing the content to the humorous and light-hearted mannerism, a typical meme is known for, which the model always seem to factorin while generating any explanation/rationale. For instance, a couple of lines from a sample meme’s explanation via a multimodal LLM states: “...It is important to note that this is a form of political humor and should not be taken seriously. The meme is simply meant to be amusing and provocative, rather than intentionally malicious or offensive.” (c.f. Fig. 12). Such statements are critical w.r.t the safe deployment of such systems, yet they inhibit their capacity for pragmatic content generation.
For quantitative assessment of the errors commited, we compare generated text (hyp) and ground truth references (ref) in Table 6. Metrics include word error rate (WER), match error rate (MER), word information lost (WIL), word information preserved (WIP), and character error rate (CER), computed via minimum edit distance (I, S, D). distance(D) = (I + S + D)/N , with N as total words/characters in the reference. The error rates depicted in Table 6 elucidate the relative challenges different approaches face toward capturing the required linguistic nuances and, indirectly, the overall semantics. As expected, unimodal image-only and multimodal conventional approaches fail to emulate the reasoning necessary for producing coherent and meaningful explanations, and yield the worst scores, with an average error rate of 0.89 and 0.81, respectively. While their word information preser-
12For more error-type details, see Appendix. I.
5058
vation is equally abysmal, both attain a meager score of 0.05. In contrast, a unimodal text-only system, being fundamentally built for tasks pertaining to NLU (given text-formatted input/output configurations), produce a moderate average error rate of 0.67, and a WIP score of 0.18.
The best rates are exhibited by the top two systems in our experimental suite, with MM-CoT achieving the best overall average error rate of 0.41, and a WIP score of 0.42, suggesting the potential for enhanced multimodal reasoning, with a modeling approach, not as large-sized as recent LLM-based solutions. But with the downside of the sub-par coherence, fluency, and complex reasoning capacity, these models do not produce explanations/answers inferencing of acceptable quality with a few exceptions as demonstrated via the example in Fig. 8, while the proposed approach (ARSENAL) demonstrates exceptional inferencing and rationalizing capacity, with a few critical constraints like factuality and too much detailing, while yielding second best average error rate of 0.59, with a decent WIP score of 0.23 (c.f. Fig. 6).
The one-stage approaches like the T5-based unimodal text-only model and MM models have direct accessibility to the meme’s content; hence it always attempts to ground its generated explanation w.r.t the meme’s content. Whereas ARSENAL is observed to suffer when the rationales contributing towards the explanation generation are noisy and irrelevant. This also solicits the requirement for utilizing meme text during the second stage fine-tuning as in T5 text-to-text or the conventional MM-CoT setup (c.f. Fig. 26 and 27).

heading: J Confounding Analysis
text: Yes/No Confounding: In this setup, we alter MemeMQACorpusto shift from multiple options to a ’yes or no’ format. Each question has a 50% chance of becoming a ’yes’ or ’no’ answer. If a question is changed to ’yes,’ it’s rephrased as "Is [answer] [rephrased question]". For instance, ’Who is maligned in this meme?’ with ’Joe Biden’ becomes ’Is Joe Biden maligned in this meme?’ with ’yes’. To change a question to ’no,’ we adjust the role label to be incorrect for the discussed entity.
None Sampling Across All Sets: In this setup, 20% of answers are randomly changed to None. To implement this, semantic role labels for a meme (∈ Rpos) like hero, villain, or victim (effectively
their synonyms) are replaced with a synonym sampled randomly from negative role-categories (R ′ pos ∈ Rswap), where, Rpos ⋂ Rswap = ϕ. To maintain consistency, existing role labels for entities in a meme are removed, ensuring the validity of the question. The new option set, Mnew = {M,None}, is applied to 20% of the dataset, including validation and test sets, while the entire dataset gets the new option setting.
None Sampling Across Train Only: In this new setting, compared to the previous None sampling, the training set incorporates a 20% random sampling of None answers, while validation and testing sets remain unchanged. The model now learns from data where answers are None with 20% probability, while during testing, None is never the answer. This added complexity challenges the model. The option set for any meme remains Mnew = {M,None} across all sets.

heading: K Examples
text: The rest of the Appendix consists of multiple examples, consisting memes, questions, multiple options, generic and answer-specific rationale, model outputs from (a) ARSENAL, (b) MM-CoT (w Lecture), (c) Unimodal Text-only, (d) Unimodal Imageonly, and (c) Multimodal baseline setups.
5059
13/08/23, 2:46 AMmultimodal_machine_translation
Page 5 of 33about:srcdoc

heading: Source 0
text: Source 1_quest_ans Who is disparaged in this meme?. Options: (a) barack obama (b) donald trump (c) daily wire (d) green party
Source 2_gen_rationale The meme consists of a collage of images featuring President Trump and his policies. In one of the images, there is a caption that reads, "They keep accusing Trump of criminal acts, but they keep ignoring the fact that he's the president of criminal acts!" This statement seems to be a commentary on the ongoing discussions and scrutiny of President Trump's actions and policies. The meme also includes a few other images, making it a multi-panel display that combines both visuals and text to express the creator's opinion or perspective on the subject.
Source 3_ans_rationale In the meme, Barack Obama is disparaged by showing a picture of him with a caption that reads, "They keep accusing Trump of criminal acts." The meme implies that people have constantly accused Trump of criminal activities during his presidency, while ignoring or downplaying Obama's actions. This statement is meant to be humorous by juxtaposing the two president's names and using a visual pun to create a critical commentary on the political climate and public perception of their respective presidencies.
Reference 0 barack obama is portrayed as having spied political oponents
Reference 1 barack obama is framed for disowning crimes
1_ARSENAL answer: barack obama because barack obama is depicted as having committed crimes
2_mmcotlec the answer is (b) because barack obama is portrayed as crimesining his against
3_umtxt answer: barack obama because barack obama
4_umimg answer : donald trump because donald trump is portrayed as unintelligent
5_mm answer : donald trump because donald trump is portrayed as hateful
Model bleu chrf
1_ARSENAL 0.0 0.42
2_mmcotlec 28.92 0.48
3_umtxt 0.0 0.24
4_umimg 0.0 0.32
5_mm 0.0 0.34
# 3 / 30 / 308 ( 3 / 308 )
Figure 9: Example 3
5060
13/08/23, 2:46 AMmultimodal_machine_translation
Page 6 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 46.17 0.69
2_mmcotlec 0.0 0.45
3_umtxt 0.0 0.1
4_umimg 19.34 0.42
5_mm 41.72 0.7
5061
13/08/23, 2:46 AMmultimodal_machine_translation
Page 14 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 0.0 0.37
2_mmcotlec 0.0 0.37
3_umtxt 0.0 0.3
4_umimg 0.0 0.28
5_mm 0.0 0.29
5062
13/08/23, 2:46 AMmultimodal_machine_translation
Page 26 of 33about:srcdoc

heading: Source 0
text: Source 1_quest_ans What is berated in this meme?. Options: (a) terrorist sleeper cell (b) democratic party (c) republican (d) donald trump
Source 2_gen_rationale The meme in the image is a playful and creative depiction of a donkey, which is associated with the Democratic Party. The donkey is shown with a bomb symbol above its head, and the words "terrorist sleeper cell" are written beneath it. This meme implies that the Democratic Party is perceived as a potential threat or as having terrorist connections. It is important to note that this is a form of political humor and should not be taken seriously. The meme is simply meant to be amusing and provocative, rather than intentionally malicious or offensive.
Source 3_ans_rationale In the meme, the Democratic Party is portrayed as a dangerous entity, evoking a sense of caution or warning. The image features a donkey, which is commonly associated with the Democratic Party, and it is depicted as a ticking time bomb or a bomb with a fuse. The message behind this visual representation is that the meme-maker perceives the Democratic Party as a potential threat or danger, suggesting that the party's policies or actions might lead to negative consequences or even destruction. This visual representation is meant to be humorous or satirical, but it also reflects the current political climate and the polarization that often occurs around election time or in discussions about political parties.
Reference 0 democratic party is portrayed as a dangerous element to society
Reference 1 the democratic party is allegedly compared to terrorist sleeper cell
1_ARSENAL answer: democratic party because democratic party is portrayed as dangerous
2_mmcotlec the answer is (b) because the democratic party is portrayed framed as besers cells
3_umtxt answer: democratic party because the democratic party is shown as an unfavourable option
4_umimg answer : democratic party because the democratic party is portrayed as spreading misinformation
5_mm answer : democratic party because the democratic party is shown as a repulsive option
Model bleu chrf
1_ARSENAL 41.11 0.65
2_mmcotlec 27.82 0.53
3_umtxt 20.45 0.4
4_umimg 38.06 0.55
5_mm 21.14 0.4
# 24 / 30 / 308 ( 24 / 308 )
Source 0
# 25 / 30 / 308 ( 25 / 308 )
Figure 12: Example 24
5063
13/08/23, 2:46 AMmultimodal_machine_translation
Page 11 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 0.0 0.51
2_mmcotlec 0.0 0.47
3_umtxt 0.0 0.26
4_umimg 0.0 0.3
5_mm 0.0 0.33
5064
13/08/23, 12:32 PMmultimodal_machine_translation
Page 19 of 32about:srcdoc
Source 0
Source 1_quest_ans Who is idealized in this meme?. Options: (a) kamala harris (b) donald trump (c) joseph (d) jewish
Source 2_gen_rationale The meme consists of two images, both featuring President Trump and a young boy sitting in chairs. In one image, the President is pointing at the boy, while in the other, the boy is pointing back at the President. The meme suggests a playful interaction between the two, possibly discussing or disagreeing on political topics. The image also contains a reference to the President's catchphrase, "This is the calm before the storm," which is written on a placard in the scene. This phrase is often associated with President Trump's unpredictable and bold decisionmaking style.
Source 3_ans_rationale In the meme, Donald Trump is idealized as a loving and caring leader, standing next to a young boy. The meme juxtaposes two different aspects of the president: on one hand, it shows a stern and controversial figure associated with division and criticism; on the other hand, it presents a warm and compassionate leader who cares for the child. This contrast creates a humorous and satirical portrayal of Trump, suggesting that he could be a more empathetic and nurturing figure if only he were given the chance to govern with love and kindness rather than controversy and division.
Reference 0 donald trump is depicted as being attacked by his political opponents
Reference 1 donald trump is praised for being inclusive of all religion.
1_ARSENAL answer: donald trump because donald trump is depicted as caring for children
2_mmcotlec the answer is (b) because donald trump is portrayed for being bold of minor racess
3_umtxt answer: donald trump because donald trump is shown
4_umimg answer : kamala harris because kamala harris is portrayed as the ideal choice
5_mm answer : barack obama because barack obama is depicted as having hateful views
Model bleu chrf
1_ARSENAL 33.18 0.45
2_mmcotlec 0.0 0.33
3_umtxt 0.0 0.28
4_umimg 0.0 0.18
5_mm 0.0 0.26
# 17 / 30 / 308 ( 47 / 308 )
Figure 14: Example 47
5065
13/08/23, 12:32 PMmultimodal_machine_translation
Page 24 of 32about:srcdoc

heading: Source 0
text: Source 1_quest_ans What is slandered in this meme?. Options: (a) democratic party (b) protest (c) starter pack (d) leftists
Source 2_gen_rationale The image features a meme with various objects related to a peaceful protest. Among these objects, there is a black mask, a Molotov cocktail, a lighter, a water bottle, a baseball bat, and a badge. The meme suggests that these items are part of a "peaceful protest starter pack," implying that they could be used as tools or symbols during a non-violent demonstration. The objects are arranged in a way that emphasizes the various items in the pack, with each object placed next to each other, creating a visually striking composition.
Source 3_ans_rationale In the meme, the Democratic Party is slandered by being associated with various negative and violent symbols. The image includes a picture of a Democratic Party starter pack, which contains a gun, a brick, a bottle, and a lighter. These items are arranged to give the impression that the Democratic Party is linked to violence, fire, and other destructive behaviors. This is a form of slander as it falsely portrays the Party and its supporters in a negative light, without any basis in fact. Such memes can contribute to the spread of misinformation and reinforce biases, so it is essential to be cautious when sharing or believing such content.
Reference 0 democratic party is portrayed as supporting anti-social elements
Reference 1 the democratic party is framed as protesting dangerously
1_ARSENAL answer: democratic party because democratic party is depicted as having a violent ideology
2_mmcotlec the answer is (c) because the democratic party is framed as havinging withly
3_umtxt answer: democratic party because the democratic party is shown as supporting indecent individuals
4_umimg answer : democratic party because the democratic party is framed as having double standards
5_mm answer : democratic party because the democratic party is portrayed as supportive of illegal immigrants
Model bleu chrf
1_ARSENAL 0.0 0.41
2_mmcotlec 38.06 0.38
3_umtxt 23.0 0.5
4_umimg 34.99 0.39
5_mm 32.38 0.63
# 22 / 30 / 308 ( 52 / 308 )
Figure 15: Example 52
5066
13/08/23, 12:32 PMmultimodal_machine_translation
Page 28 of 32about:srcdoc

heading: Source 0
text: Source 1_quest_ans Who is exploited in this meme?. Options: (a) black people (b) democratic party (c) arab muslims (d) nancy pelosi
Source 2_gen_rationale The image is a political cartoon that features a group of people, some of whom are chained together, symbolizing political prisoners. The main idea is that the Democratic Party will never speak about it because it would be an uncomfortable or unpopular topic. The cartoon seems to convey a message about the current state of political discussions and the potential silence or avoidance of certain issues by politicians or the media.
Source 3_ans_rationale The meme exploits the idea of political correctness by showing a cartoon depiction of a group of black people being held captive by another person, who appears to represent a Democratic Party official. The message implies that the Democratic Party will never speak about the issue because it would be considered politically incorrect. The meme takes a controversial and sensitive topic, such as slavery, and uses it as a form of humor to make a political statement. This type of humor can be seen as offensive and disrespectful to some viewers, especially those who have a personal connection to the issues being discussed.
Reference 0 black people are depicted as being enslaved
Reference 1 the black people are portrayed as a victim of slavery
1_ARSENAL answer: black people because black people are portrayed as being oppressed
2_mmcotlec the answer is (c) because black black people are portrayed as beingill target of
3_umtxt answer: black people because black people are portrayed as being targetted
4_umimg answer : black people because black people are portrayed as being targetted
5_mm answer : muslims because muslims are portrayed as being opressed for years
Model bleu chrf
1_ARSENAL 38.83 0.51
2_mmcotlec 28.92 0.5
3_umtxt 38.83 0.52
4_umimg 35.08 0.52
5_mm 0.0 0.29
# 26 / 30 / 308 ( 56 / 308 )
Figure 16: Example 56
5067
13/08/23, 2:46 AMmultimodal_machine_translation
Page 23 of 33about:srcdoc

heading: Source 0
text: Source 1_quest_ans What is scapegoated in this meme?. Options: (a) mainstream media (b) radical islam (c) barack obama (d) democratic party
Source 2_gen_rationale The image features a headstone with a meme etched on it. The meme reads "Democratic Party" offended to death." This is a play on words, combining a political statement with a pun about the offended emoji. The headstone is part of a larger stone structure, and there are trees in the background.
Source 3_ans_rationale In the meme, the Democratic Party is scapegoated by having a mock funeral organized for it. The party's name is displayed on the headstone, and the inscription reads "Democratic Party, officially offended to death." This is a play on words, as the party's name is associated with death and the end of an era. The meme is implying that the Democratic Party has been "buried" or is no longer relevant, which is a form of political humor that aims to highlight a particular issue or statement. However, it is important to note that the humor may not be suitable for everyone, and it is essential to consider the sensitivities surrounding political topics and the memebers of the party.
Reference 0 democratic party is depicted to have recieved a lot of hate
Reference 1 the democratic party is depicted as a target for hateful trolls.
1_ARSENAL answer: democratic party because democratic party is portrayed as being killed
2_mmcotlec the answer is (d) because the democratic party is ed as beingcommitting target for off peopleshugss
3_umtxt answer: democratic party because the democratic party is depicted as being insulted
4_umimg answer : democratic party because the democratic party is portrayed as being full of hateful people
5_mm answer : democratic party because the democratic party is portrayed as having met its demise
Model bleu chrf
1_ARSENAL 0.0 0.4
2_mmcotlec 18.92 0.38
3_umtxt 41.72 0.52
4_umimg 17.61 0.43
5_mm 17.4 0.4
# 21 / 30 / 308 ( 21 / 308 )
Figure 17: Example 21
5068
13/08/23, 2:46 AMmultimodal_machine_translation
Page 4 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 25.75 0.54
2_mmcotlec 35.08 0.39
3_umtxt 0.0 0.29
4_umimg 0.0 0.19
5_mm 26.58 0.54
5069
13/08/23, 2:46 AMmultimodal_machine_translation
Page 7 of 33about:srcdoc

heading: Source 0
text: Source 1_quest_ans What is glorified in this meme?. Options: (a) democratic party (b) communist party (c) republican party (d) kegger party
Source 2_gen_rationale The image consists of four portraits of famous historical figures, each of them associated with a specific political party. The first person is Karl Marx, who is often linked to the Democratic Party. The second person is John Adams, who is typically connected to the Republican Party. The third person is Alexander Hamilton, who is commonly associated with the Democratic Party as well. The fourth person is Thomas Jefferson, also a part of the Democratic Party. The idea behind the meme is to create a humorous visual by combining the faces of these historical figures with the names of the political parties they are associated with. The meme is intended to be funny because it is not typical to see these people's faces connected to specific political parties, as they are more commonly known for their individual achievements and contributions rather than their political affiliations.
Source 3_ans_rationale In the meme, the Democratic Party is glorified by showing three different images of the party's founder, Karl Marx, with one of the images having an American flag superimposed over him. This is an attempt to associate the values and principles of the Democratic Party with the iconic figure of Karl Marx, thereby implying that the party still upholds his ideologies and legacy. The inclusion of the American flag in one of the images suggests a connection between the party and the nation's flag, emphasizing the party's identification with American values and patriotism.
Reference 0 communist party is portrayed in a positive light
Reference 1 the communist party is regarded as a distinct preference.
1_ARSENAL answer: democratic party because democratic party is portrayed as the ideal choice
2_mmcotlec the answer is (c) because commun communist party is shownportrayed as thea suitable party
3_umtxt answer: republican party because the republican party is depicted as a
4_umimg answer : shrek super party because shrek super party is shown as a suitable option
5_mm answer : democratic party because the democratic party is shown as an unfavourable choice
Model bleu chrf
1_ARSENAL 0.0 0.38
2_mmcotlec 0.0 0.48
3_umtxt 0.0 0.21
4_umimg 0.0 0.21
5_mm 0.0 0.22
# 5 / 30 / 308 ( 5 / 308 )
Figure 19: Example 5
5070
13/08/23, 2:46 AMmultimodal_machine_translation
Page 9 of 33about:srcdoc

heading: Source 0
text: Source 1_quest_ans What is maligned in this meme?. Options: (a) donald trump (b) democrats (c) xi jinping (d) covid19
Source 2_gen_rationale The meme in the image is a playful alteration of a photograph featuring a Chinese politician, Xi Jinping, who is shown with a humorous text superimposed over his face. The text reads, "We deceitfully hid the 9/11 whistler." The reference to 9/11 and the whistler is a play on Xi's name, which sounds similar to "911" and is a nod to the famous phone call between him and former US President George W. Bush after the 9/11 attacks. The meme is a creative way of mixing politics and humor, but it should be noted that it may not be suitable for all audiences due to its political nature and could be considered offensive to some.
Source 3_ans_rationale In the meme, a Chinese politician, possibly President Xi, is depicted with a caption that reads, "We deceitfully hid the 98th whistle-blower." The meme implies that the politician has been dishonest and tried to cover up a scandal involving the 98th whistle-blower. It suggests that the politician has been involved in a conspiracy to suppress the truth, possibly related to corruption or other wrongdoing. The meme is a form of political humor that uses irony and exaggeration to make a statement about the politician's alleged actions and character.
Reference 0 democrats are insinuated to wrongly accuse their opponents
Reference 1 the democrats are accused of improperly deflecting any blame.
1_ARSENAL answer: democrats because democrats are depicted as being incompetent
2_mmcotlec the answer is (b) because the democrats are of beingly targetingfrauing votes votes onto
3_umtxt answer: democrats because the democrats are portrayed as supporting stupid policies
4_umimg answer : democrats because the democrats are portrayed to have colluded with foreing interests
5_mm answer : democrats because democrats are framed as conspiring to spread the coronavirus
Model bleu chrf
1_ARSENAL 0.0 0.35
2_mmcotlec 0.0 0.38
3_umtxt 0.0 0.36
4_umimg 0.0 0.38
5_mm 0.0 0.34
# 7 / 30 / 308 ( 7 / 308 )
Figure 20: Example 7
5071
13/08/23, 2:46 AMmultimodal_machine_translation
Page 13 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 27.78 0.42
2_mmcotlec 28.92 0.32
3_umtxt 0.0 0.28
4_umimg 0.0 0.3
5_mm 24.81 0.42
5072
13/08/23, 12:32 PMmultimodal_machine_translation
Page 28 of 32about:srcdoc

heading: Source 0
text: Source 1_quest_ans Who is exploited in this meme?. Options: (a) black people (b) democratic party (c) arab muslims (d) nancy pelosi
Source 2_gen_rationale The image is a political cartoon that features a group of people, some of whom are chained together, symbolizing political prisoners. The main idea is that the Democratic Party will never speak about it because it would be an uncomfortable or unpopular topic. The cartoon seems to convey a message about the current state of political discussions and the potential silence or avoidance of certain issues by politicians or the media.
Source 3_ans_rationale The meme exploits the idea of political correctness by showing a cartoon depiction of a group of black people being held captive by another person, who appears to represent a Democratic Party official. The message implies that the Democratic Party will never speak about the issue because it would be considered politically incorrect. The meme takes a controversial and sensitive topic, such as slavery, and uses it as a form of humor to make a political statement. This type of humor can be seen as offensive and disrespectful to some viewers, especially those who have a personal connection to the issues being discussed.
Reference 0 black people are depicted as being enslaved
Reference 1 the black people are portrayed as a victim of slavery
1_ARSENAL answer: black people because black people are portrayed as being oppressed
2_mmcotlec the answer is (c) because black black people are portrayed as beingill target of
3_umtxt answer: black people because black people are portrayed as being targetted
4_umimg answer : black people because black people are portrayed as being targetted
5_mm answer : muslims because muslims are portrayed as being opressed for years
Model bleu chrf
1_ARSENAL 38.83 0.51
2_mmcotlec 28.92 0.5
3_umtxt 38.83 0.52
4_umimg 35.08 0.52
5_mm 0.0 0.29
# 26 / 30 / 308 ( 56 / 308 )
Figure 22: Example 45
5073
13/08/23, 2:46 AMmultimodal_machine_translation
Page 12 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 0.0 0.55
2_mmcotlec 0.0 0.42
3_umtxt 0.0 0.32
4_umimg 18.92 0.43
5_mm 0.0 0.35
5074
13/08/23, 2:46 AMmultimodal_machine_translation
Page 3 of 33about:srcdoc
Source 0
Source 1_quest_ans Who is maligned in this meme?. Options: (a) vacation (b) barack obama (c) bernie sanders (d) democrats
Source 2_gen_rationale The meme features a picture of President Obama with a caption that reads, "I didn't catch that, but thanks for the thought about the next 30 million dollars worth of vacation that you're actually paying for." The meme humorously suggests that President Obama is not paying attention to the question asked and is instead thinking about the cost of the next 30 million dollars worth of vacation.
Source 3_ans_rationale In the meme, Barack Obama is maligned by being portrayed as a villain or a cartoon character with devil horns, resembling a typical evil character. The meme reads, "I didn't catch that, thank you, Mr. President," which implies that the person responsible for the meme is mocking the former president's policies or actions. This type of meme is a form of political humor that aims to convey a negative opinion about Obama's leadership or administration by using exaggerated and satirical elements.
Reference 0 barack obama is framed as misuing public money
Reference 1 barack obama is being accused of taking unjustified vacations on taxpayer money.
1_ARSENAL answer: barack obama because barack obama is depicted as being wrongly blamed
2_mmcotlec the answer is (b) because barack obama is of over toosuit vacation vacations hiss
3_umtxt answer: barack obama because barack obama
4_umimg answer : hillary clinton because hilary clinton is framed as a whiner
5_mm answer : barack obama because barack obama is framed as having a liar
Model bleu chrf
1_ARSENAL 0.0 0.4
2_mmcotlec 0.0 0.33
3_umtxt 0.0 0.3
4_umimg 0.0 0.28
5_mm 28.92 0.53
# 1 / 30 / 308 ( 1 / 308 )
Figure 24: Example 1
5075
13/08/23, 12:32 PMmultimodal_machine_translation
Page 32 of 32about:srcdoc

heading: Source 0
text: Source 1_quest_ans What is exploited in this meme?. Options: (a) democratic party (b) republicans (c) black community (d) joe biden
Source 2_gen_rationale The image is a meme featuring a black and white picture of a man pointing a gun at another man who is sitting in a chair, with the phrase "how could the republicans do this" written on it. The meme is a play on the idea of gun control and political affiliations. The man pointing the gun represents the Republicans, while the sitting man symbolizes the Democratic Party. The meme humorously suggests that the Republicans might try to enact gun control policies supported by the Democratic Party.
Source 3_ans_rationale In the meme, a Democrat Party member is pointing a gun at a Republican Party member and pulling the trigger. The target, a stuffed animal or dummy representing the Republican Party, is depicted as being shot. This visual humor is playing on the idea of a political "shooting" or conflict between the two parties, with the Democrat Party member being the aggressor. The meme is intended to be funny by taking a serious situation and portraying it in a lighthearted and satirical manner.
Reference 0 republics are portrayed as being wrongly accused
Reference 1 the republicans are depicted to be blamed for false allegations
1_ARSENAL answer: republicans because republicans are portrayed as being shot
2_mmcotlec the answer is (b) because the republicans are ed as have targetingd for the ideas
3_umtxt answer: republicans because republicans are portrayed as being insulted
4_umimg answer : democratic party because the democratic party is portrayed as being influenced by corporate money
5_mm answer : republican party because the republican party is depicted as being attacked by political opponents
Model bleu chrf
1_ARSENAL 30.35 0.59
2_mmcotlec 0.0 0.31
3_umtxt 30.35 0.58
4_umimg 0.0 0.37
5_mm 0.0 0.36
# 30 / 30 / 308 ( 60 / 308 )
In [ …
Figure 25: Example 60
5076
13/08/23, 2:46 AMmultimodal_machine_translation
Page 8 of 33about:srcdoc

heading: Source 0
text: Source 1_quest_ans Who is disparaged in this meme?. Options: (a) covid19 (b) americans (c) democrats (d) xi jinping
Source 2_gen_rationale The meme in the image displays a picture of a man, likely a Chinese politician, with a caption that reads "We deceitfully hid the 98th whistle-blower." The meme seems to be making a joke or a statement about a whistleblower scandal involving the politician. The politician is wearing a suit and tie, giving the impression that the meme is taken from a professional setting. The image is a play on words, using "whistle-blower" as a metaphor for a person who exposes wrongdoing, and "98th" referring to the 98th whistle-blower, implying that there are multiple whistle-blowers involved in the scandal.
Source 3_ans_rationale In the meme, xi jinping, the Chinese president, is disparaged by being shown in a photo with a humorous message. The message reads, "We deceitfully hid the 90's whistle and made it look like a Democrat would blame Trump for it." The meme implies that xi jinping is being blamed for a problem that occurred during the 1990s, which is attributed to a Democratic politician, and it suggests that the politician would blame Trump for the issue. This is a playful and sarcastic way of implying that the politician would accuse Trump of causing the problem, even though it happened in the 1990s, and Trump was not yet in office.
Reference 0 xi jinping is framed as hiding critical information from the world
Reference 1 xi jinping is accused of tampering with the covid-19 news.
1_ARSENAL answer: xi jinping because xi jinping is depicted as being wrongly blamed
2_mmcotlec the answer is (d) because xi jinping is of beingsampering with the electionvid19
3_umtxt answer: xi jinping because xi jinping
4_umimg answer : xi jinping because xi jinping is depicted to be interfering with american politics
5_mm answer : barack obama because barack obama is framed as unintelligent
Model bleu chrf
1_ARSENAL 0.0 0.3
2_mmcotlec 0.0 0.3
3_umtxt 0.0 0.2
4_umimg 0.0 0.3
5_mm 0.0 0.22
# 6 / 30 / 308 ( 6 / 308 )
Figure 26: Example 6
5077
13/08/23, 2:46 AMmultimodal_machine_translation
Page 15 of 33about:srcdoc

heading: Source 0
text: Model bleu chrf
1_ARSENAL 0.0 0.3
2_mmcotlec 0.0 0.36
3_umtxt 0.0 0.52
4_umimg 0.0 0.44
5_mm 24.6 0.6
5078

heading: Future Work
text: Ultimately, our findings steers future exploration and the development of comprehensive systems dedicated to deciphering memetic phenomena. Our future aim is to create sophisticated, multiperspective sets for MemeMQA, moving beyond standard QnA towards an optimal multimodal solution