MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation
Authors: Sayantan Adak
Somnath Banerjee
Rajarshi Mandal
Avik Halder
Sayan Layek
Rima Hazra
Animesh Mukherjee
Venue: Accepted by TMLR ()

Abstract: Online memes are a powerful yet challenging medium for content moderation, often masking harmful intent behind humor, irony, or cultural symbolism. Conventional moderation systems “especially those relying on explicit text” frequently fail to recognize such subtle or implicit harm. We introduce MemeSense, an adaptive framework designed to generate socially grounded interventions for harmful memes by combining visual and textual understanding with curated, semantically aligned examples enriched with commonsense cues. This enables the model to detect nuanced complexed threats like misogyny, stereotyping, or vulgarity “even in memes lacking overt language”. Across multiple benchmark datasets, MemeSense outperforms state-of-the-art methods, achieving up to 35% higher semantic similarity
and 9% improvement in BERTScore for non-textual memes, and notable gains for text-rich memes as well. These results highlight MemeSense as a promising step toward safer, more context-aware AI systems for real-world content moderation. The code and data are available at: https://github.com/sayantan11995/MemeSense

MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderation
ADISCLAMER: This manuscript features memes that some readers may find vulgar/offenssveehhtteuu
Sayantan Adak', Somnath Banerjee! 13, Rajarshi Mandal', Avik Halder', Sayan Layek', Rima Hazra², Animesh Mukherjee'
Indian Institute of Technology Kharagpur Eindhoven University of Technology, Netherlands
Cisco Systems
Abstract
Online memes are a powerful yet challenging medium for content moderation, often masking harmful intent behind humor, irony, or cultural symbolism. Conventional moderation systems "especially those relying on explicil test" frequently fail to recognize such subtle or implicit harm. We introduce MemeSense, an adaptive framework designed to generate socially grounded interventions for harmful memes by combining visual and textual understanding with curated, semantically aligned examples enriched with commonsense cues. This enables the model to detect nuanced complexed threats like misogyny, stereotyping, or vulgarity even in memes lacking overt language". Across multiple benchmark datasets, MemeSense outperforms statc-of-tho-art methods, achicving up to 35% higher semantic similarity and 9% improvement in BERTScore for non-textual memes, and notable gains for text-rich memes as well. These results highlight MemeSense as a promising step toward safer, more context-aware Al systems for real-world content moderation.
1 Introduction
Memes have emerged as powerful form of online expression, where seemingly lighthearted humor can conceal offensive, derogatory, or culturally charged subtexts. Their multimodal nature combining images, text, and symbolism poses significant hurdles for content moderation systems, especially those built primarily around lextual analysis] Maity et al. (2024); Jain et al. (2023): ; Jha et al. (202bb a). Large vision-language models (VLMs), including GPT-4o GPT-4o[OpenAL et al. (2024), Gemini 2.0 Team al. (2024), and Qwen 2.5 Qwen et al. 20255) often show reduced accuracy on imago-centric memes precisely because they depend heavily overt text clues Sharmaa et al. (2023 Agarwal et al. (2024). In contrast, humans effortlessly parse memes by applying commonsense reasoning and recalling mental examples of similar situations. This can be attributed to the social commonsense) Naslund et al. (2020); Arora et al. (2023): Office of the Surgeon General (OSG) 2023 capabilitics humans which include recognizing social norm violations (c.g., hate speech, body shaming, misogyny, stereotyping, sexual content, vulgarity), assessing credibility (e.g., misinformattinn), empothy and ethical judgment (e.g., child exploitation, public decorum and privacy, cultural sensitivity, religious sensitivity), conteztual interpretation (e.g., humor appropriateness), and predicting consequences (e.g., mental health impact, violence, substance abuse). This human-like capacity to interpret subtle symbolic cues underscores the need for moderation frameworks that can replicate such higher-level reasoning rather than relying purely on text or raw pixels.
Early multimodal models have attempted to fuse vision and language through joint embeddings or cross- attention mechanisms Shin Narihira (2021) : Radford et al. (201]), yet they tend to place disproportionate
The code and Dataset are available at: hattps://githun hhttps://ee //en. htt.////eda......iiii....m www.pedia.orrg/vkk///oooomoooo reasoning
httpp:///ggthubbb..caaaaaaaaattt 1995 9995/Keneeensee

---

emphasis on textual data. As a result, subtle image-based cues - such as historical references, cultural icons, or visually encoded irony - can slip through the cracks Zhang et al. (2024). Detecting such implicit signals requires not just better model capacity, but the ability to interpret content in light of prior socially grounded examples. Inspired by how humans recall similar experiences to contextualize new ones, we explore a retrieval-uugmented approach that grounds meme understanding in examples enriched with commonsense and cultural cues. This design enables the model to move beyond literal interpretation and capture the symbolic and contextual signals embedded in multimodal content, especially when explicit textual markers are absent or misleading.
In this paper, we propose an adaptive in-context learning framework - MemeSense that synthesizes commonsonse knowledge with semantically similar reference images to enhance the interpretation of meme content. Concretely, MemeSense retrieves a curated set of analogous memes, each annotated with cultural, historical, or situational context and incorporates these examples into unified representation alongside the target meme. By embedding human-like commonsense cues directly into the model's input, we effectively steer its latent space toward the pertinent visual and textual signals present in the attached memes. This synergy allows the model to detect subtle or symbolic markers such as ironic juxtapositions, culturally coded imagery, or sarcastic overlays that often evade traditional pipelines. We validate our framework using mid-sized language models (8B-9B parameters) in post-hoc (zero-shot/ (zero-shof/in-cooleel) set settings, without additional fine-ture Sne-tuning, highlighting the practical utility of intervention generation with minimal resource overhead.
Our contributions are as follows.
We develop unique multi-staged framework generate intervention for the harmful memes by leveraging cognitive shift vectors which reduce the requirement of demonstration examples during inference.
We curate wide-ranging dataset collection that emphasizes subtly harmful or text-scarce memes, filling a crucial gap in moderation research. This dataset lays the groundwork for a deeper exploration nuanced meme analysis. We make this dataset publicly available for future research. Rigorous experiments demonstrate the efficacy of MemeSense even for the memes that do not contain any explicit text embedded in them as is usually the case. We obtain respectively 5% and 9% improvement in BERTScore over the most competitive baseline for the memes with tert and the memes without tezt. Semantic similarity for memes with as well without text (almost) doubles for MemeSense compared to the best baseline.
2 Related work
Visual in-context learning: In-context learning (ICL) has revolutionized LLM adaptation by enabling task generalization from a few demonstrations Brown et al. (2020), and recent developments have extended this paradigm to multimodal models for vision-language tasks such as visual question answering (VQA) Alayrac et al. (2022). However, ICL in large multimodal models (LMMs) faces challenges like computational inefficiency due to long input sequences and sensitivity to demonstration selection [Peng et al. 2024h. To address these issues, in-context vectors (ICVs) have been proposed as compact representations that distill task-relevant information, thereby reducing the dependence on multiple demonstrations at inference time Hendel et al.) (2023); Todd et al. (2024). Early non-lcarnable ICVs showed efficiency gains NLP but struggled with complex multimodal tasks due to the diversity vision-language inputs Li et al. (2023); Yang al. (2024). More recent work introduces learnable ICVs that dynamically capture task-specific signals, significantly improving VQA performance while lowering computation ional overhead Peng et. al. (2024). These advancements highlight the importance of optimiz- ing latent task representations and refining TCI. strategies for improved multimodal reasoning Yin et al. (0024)
Intervention generation: Most intervention strategies for online harm have centered around text-based content, focusing on areas like hate speech Qian et al. (2019); Tha al. (2024a), misinformation He et al. (2023), and general toxic behavior [aneejee et al. (2024): Hazra et al. (2024); Banerjee al. (2025). In contrast, multimodal content- content-partienlle particularly memes-remains underexplored despite its unique challenges. Counterspeech has shown potential in mitigating online harm Schieb & Preuss] but it often relies manually
2

---

Normal Inference
Commonsense Generation
image you've shared appears to show money arranged in palm tree-like formation, with person lying bed nearby
Vulgarity
GPT
Misogyny
Human
Raligious
Body Shaming
FL v1.8 v1.8
MomeSonse Intervention
Sexual Content
image seems convey a symbolic or humorous message, likely meant to represent the idea of "sleeping with money" or "money as companion"" The woman's relaxed expression implies contentment, suggesting that wealth provides her comfort, possibly substituting for human relationship
visual innuendo, comparing the unpasled and panked genitalia.
Aroffio
Anchia
Vulgarity
Anciar tragel.
unirientionally makes look like the figures
posttion weweeeeeee
Cognitive Shift Vector
Incontext Exemplars
Ground Truth
Frozen
Trainable
Ground Trafh
Output
Oulput
MLLM
MLLM
Anchor Images
Arccha Iraget
ICD
ICO
Layer
F2
12
Laye
ICD
Aroifir
Image:
Trush
Ardhaa Imagee
Lager
I:
ICO
Layert
Figure 1: Schematic diagram of MemeSense. Block 1 highlights the challenge of understanding memes in a zero-shot setting using MLLMs. Blocks 2 to 5 illustrate the key stages of our approach: (Block 2) Commonsonse Paramcter Generation, (Block 3) Exemplar Retricval, (Block 4) Learning Cognitive Shift Vectors, and (Block 5) MemeSense Inference.
curated responses or supervised datasets Mathew et al. (2018). limiting scalability and adaptability. While advances in LLMs and VLMs Ghosh et al. (2024) have improved automated intervention capabilities, they frequently lack contextual grounding, necessitating knowledge-driven methods Dong et al. (2024). To that end, MemeGuard integrates VLMs with knowledgg-ranking mechanisms to enhance meme interpretation and generale more contextually relevant interventions ions Jha el al. (0244)), marking a step forward in mullimodal harm understanding. Similarly, recent efforts Pan et al. 2022 Hee & Lee (2025) employ interpretable, multi- stage reasoning pipelines for harmful meme detection. However, most existing approaches focus primarily on memes with overlaid text, overlooking the challenge of interpreting text-free visually implicit memes.
3 Methodology
In this work, propose framework that proceeds in three main stages - (a) Stage I: Generation of commonsense parameters: In Stage I, we generate commonsense parameters by instruction-tuning a multimodal large language model (MLLM) to predict contextually relevant insights for cach image. (b) Stage II: Selection of in-context exemplars: We create a set of anchor images and retrieve corresponding in-context exemplars, which we later use Stage ITT. (c) Stage III: Learning cognitive shift vector: Finally, we learn a cognitive shift vector by distilling general task information from the exemplars, and then guide the target model to align its representation with the insights derived from these exemplars. The overview of our proposed method is shown in Figure
3

---

4 Preliminaries
A collection of images is denoted as IMG, where each image img is an item of IMG, i.e., img IMG. GTimg describes the ground truth intervention the image. In particular, GTiig contains the description about why the image can/can't be posted social media? We consider a set of commonsense parameters & where 1th commonsense parameter is denoted as C, E 6. A pair consisting of an image and its corresponding commonsense parameters is denoted by (img. Ging) where Gimg 3 8. An image may be associated with multiple commonsense parameters. We partition ZMG into two subsets: (a) the training set I.MGtr used at different stages of the training process, and (b) the test set IMMIss reserved for evaluation. The set of training images IM and test images I.M are disjoint, ie., IMG n IMM = 0.
For Stage I, we build training datasct Dee consisting of images IMGg,, and their respective ground truth image description with commonsense parameters. We represent. a fine-tuned vision language model with dataset Dee as Me. Further in Stage II, we construct an in-context (IC) learning set Dre (involves only images from IMGr, sct) to utilize in Stage III (see Scction 4.3). Each instance in Dre is a tuple consisting of ICimg: Gmimg.) where ICsmg is the set retrieved in-context examples of an anchor image imga. Each in-context example consists of an image img # imga, Gimg. GTimg. We define the cognitive shift vector set as CSV and the coefficient set as a. In Stage III, we use an instruction following MLLM the target model (M) to further generate the intervention defined Mive.
4.1 Stage Commonsense parameters
Description:
of image
In this stage, we aim to fine-tune vision-language model to produce relevant commonsense parame- ters for meme images. These parameters represent broad conceptual categories that help assess whether an image is harmful, offensive, or inappropriate, as discussed in Arora et al. (2023): Office of the Sur-
take-moaning the he staris tegging in merres wbout clling the Pezat
young perton surprited expression.
Commonsense Parometers:
Vulgarity: The bex contains explicit language with sesual connotations spacifrally ""klligg the phaty"" which con be cons considered inappropriste and vulger
The phrese "foke-mnorrnng" and "Killng the p*thy"" tuggeet tecual actitty. making the content suggestive and potent offensive.
Sexual Content:
Humour Appropriataness: The mame attempls hampur based on secuel themss which could be seen as trivillaing or explottng intimate situerions.
geon General (OSG) (2023); Gongane et al. (2022). To create the training set De, first use GPT-40 to automatically obtain commonsense parameters for img E IMG and then perform manual correc- tions. We employ two expert reviewers experienced in meme moderation to verify the generated com- monsense parameters and the intervention from the GPT-4o. We provide them with a short description of each commonsense category (Similar to the Commonsense parameters mentioned in the Table 18) along with one selected meme within each category for reference. They are allowed to update, delete, add categories based on their judgment. Although we engage two experts due to the niche nature of the task and resource constraints, we ensure high quality through consensus- baased evaluation. Notably, their assessment reveals a strong consensus (96% agreement) with the GPT-4o outputs. Lastly, for the fine-tuning phase, we choose an open-weight model instead of a proprietary system such as GPT-4o, given the higher resource demands and stricter usage limitations. We employ the prompt mentioned in Table 18 in Appendix extract commonsense parameters from GPT-4o.
of vulgor language and suggestive sexcual content which may be offinsive to some audiences. Additionally, the humor may be inappropriate it deals with sensitive themes.
be posted publicly
Figure 2: Representative example of a harmful mome and the annotated commonsense parameters along with intervention.
The example of commonsense parameters Cing of an image img is illustrated in Figure 2 tuning with this datasct D&, we obtain the final vision-language model Me, which we then use at inference time to produce commonsense parameters for new meme images.
By instruction
Prompt for instruction tuning commonsense generation module

 <image> Describe the meme and assess whether the meme has the potential be perceived as vulgar, harmful, unethicol using specific commonsense parameters. 


---

4.2 Stage II: Selection of in-context exemplars
In this stage, our objective is to create an in-context dataset DIC that provides exemplars to guide the latent space of the target model in Stage III. To accomplish this, we reuse the training images IMgtr and, following the authors in Chen et al. (2025); ; Peng et al. (2024); Qin et al. (2024), treat each image img IMG as an anchor. We denote an anchor image as imga- We then select in-context examples from IMG img, using multiple strategies. First, we randomly sample l candidate images to construct. the set [Cimg for cach anchor. Apart from random selection, we also leverage semantic retrieval techniques that consider commonsense parameters, image represenlations, combination of both. The detailed up of in-context retrieval is given in Section 6
4.3 Stage III: Learning cognitive shift vectors
In this stage, the aim is to learn the trainable shift vector set CSV and coefficient set a so that the target model can generate proper intervention given a meme img. We initialize a set of shift vectors CSV = {csu', cse?...., csv") where each shift vector csu corresponds to each layer & E L in the target model M. I. represents the number of layers in larget model M. Further, we consider a set of coefficients a = {a', a²....., which regulate the impact of these coguitive shift vectors across different layers in M. After applying cognitive shift vector set CSV and to the model M, we obtain the final model as expressed in Equation
M jut = M + + -cv'.,
(1)
Following task analogies from Huang et al. 2004 Peng al. (2024). our objective is to align the output of Min with the output obtained by including ICimg in model M for a given anchor image imga. To achieve this, we minimize the KL divergence between the output distribution of Mive(img.) and output distribution of M with IC exemplars ICing for the anchor image imga- The computation of Lod is given in Equation 2
Lod= = KL
P(imgal-Mav))
(2)
where Plimgaa Carngg//CCmgl [/Cmgg M) and P(imgalMiun represent the output distribution of models M and Mist respectively for anchor image imga.
Further we compute the intervention loss (Livt) to make sure that the output of final model Mive(imga) is aligned with the ground truth GTimg, (see Equation 3
Lest
Σ log P(imgal Mave)
(3)
PPal
We compute the final loss as given in Equation + 7 serves as a hyperparameter that determines the relative importance of output distribution loss and intervention loss.
L Lodd + 7. Liot
(4)
5 Datasets
To advance research on harmful meme intervention, we construct a novel dataset of implicitly harmful memes, sourced from various online social media platforms, including Facebook, Twitter, Instagram, and WhatsApp. Unlike existing datasets that pri- marily focus on memes with explicit textual content embedded in them, our dataset specifically targets memes that are implicitly harmful or lack embed- ded text (see Figure for details). These cases pose additional challenges for AT models, as they require nuanced reasoning beyond surface-level tex- tual analysis. Below, we detail our data collection and annotation process.
Table 1: Distribution of various commonsense attributes.
Communnsenne category (meta)
Commonnsuuse parameterss # Meues
Hate spesch
Body shaming
74
Recognizing social norm violations
Minyyny
51
Storotsping
Sexuol comtent
105
Vadgarity
Assesing credibility
Miiisformatio
4
Choid cxpinitation
Empathy and ethical Judgements
Pubiee decormm & Prisecy
72
Cultural sensitivity
Refigioue exvilicity
14
Contextual interpretation
Romor rppropriatesssa
251
Menial health impaci
38
Predicting consequences
Violence
43
Salatano: abuse
5

---

Me: Babe what do you warna do tonight?
Free Wi-Fi
Her:
(a) Harmful meme without text
(b) Harmful meme with text
(c) Non-harmful meme
Figure 3: Memes can manifest harm in different ways, some rely solely imagery to convey implicit messages, while others reinforce harm through accompanying text. This figure illustrates the three primary categories: (a) harmful memes without text, (b) harmful memes with text, and (c) non-harmful memes. Prior moderation efforts have disproportionately focused on text-based harmful memes, often overlooking the nuanced and context-dependent nature of purely visual memes.
Data collection: We curate memes from publicly available online sources, including Facebook meme page Twitter adult meme pages public WhatsApp groups, and Instagram meme accounts In addition, we incorporate Ohellid meme which may not appear overtly harmful at first glance but can carry implicit harmful implications when shared publicly. Our data collection process resulted in a total of 785 memes.
Filtering and annotation: To determine whether each
exhibits potential harm, instruct two undergraduate annotators to independently label each meme either harmful or non-harmful. We define a meme as harmful if it aligned with any of the 15 predefined commonsense harm categories (e.g., vulgarity, body-shaming), as listed in Table To ensure consistency, we provide the annotators with a concise annotation guideline that includes definitions of each category and representative examples of both harmful and non-harmful memes. We adopt conservative filtering approach, relaining only those memes that both annotators independently label as harmful. This process results in a final curated dataset of 484 harmful memes. We calculate Cohen's kappa score, which yiclds value of 0.82, indicating strong inter-annotator agreement.
Training and Test Split: We allocate 300 memes for training: Stage 1 involves learning to align annotated descriptions with their corresponding commonsense parameters, while the same set is used to train the Cognitive Shift Vectors (CSV), using the meme content and harm categorics as input, and annotated interventions as output. The remaining 184 memes form the lest. set, comprising 133 with overlaid text. and 51 without.
Once finalize the harmful meme set, we use GPT-4o along with manual post-processing to generate the corresponding commonsense parameters and ground truth intervention statements, described in Section 4.1 Figure 3 showcases representative examples from the curated dataset. While our final curated dataset comprises 484 carefully annotated harmful memes, it spans a rich and diverse set of 15 commonsense categories. This breadth ensures strong coverage across varied meme types and contexts. Morcover, our multi-stage framework is specifically designed for adaptability in low-resource settings, allowing flexibility to incorporate additional harmful memes with minimal retraining.
httt:://twx.facebboo.com cow//oublmmmmm.m.o.oooo..
httpp:://..com/DDeeesee///o/
httpe://www.nnetaage www.://wwwww.ntta......... www/stoneddeeeeeoooom/oomeoeot age humour
httpss//een.wixipeddia http://www...pia.oroo//////aaaaa
https://umornnna. com//enees/pnnl--neeee

---

Additional ICMM data In addition to our curated dataset, we also consider the publicly available Intervening Cyberbullying in Multimodal Memes (TCMM) dataset Tha et al. (2024a) for evaluation of our approach. This datasct consists of 1000 cyberbullying memes along with their corrosponding crowdsourced interventions. Afler filtering out the corrupted images, we obtain a set of 985 memes along with their ground truth interventions.
6
Experimental setup
This section discusses the different experimental configurations of MemeSense.
6.1 Baselines
For baselines involving zero-shot prompting and in-context learning (ICT.), we leverage the same aligned MLLMs used in MemeSense for intervention generation.
(1) MemeGuard Jha et al. (2024a): We adapt MemeGuard, a state-of-the-arl model for harmful meme intervention generation, as one of our primary baselines. The method operates in two stages involving both vision-language and language models. In our implementation, model (1lava-1.6-1) 6-misttall-7-bbf) as in MemeSense for the first stage, and idefice-2-8b-base for the second. Given a meme, the first-stage VLM generates five descriptive responses intended to caplure potential harmful cues. While the original MemeGuard framework incorporates OCR-extracted text as an additional input modality, we omit OCR in our sctup to ensure applicability to both text-rich and text-frec memes. To remove irrelevant or noisy responses; we compute the semantic similarity between the meme and each generated sentence, retaining only those with similarity score above 0.2 (a threshold chosen based on manual inspection). In the final stage, the second VLM uses the input meme and the filtered descriptions to generate the corresponding intervention.
the same base vision-language
(2) MemeMQA (Modified) Agarwal et al. 2024): We extend the MemeMQA framework for intervention generation by removing its target identificatin module and repurposing its explanation generation module. Originally designed to identify targets in hateful memes and explain predictions, MemeMQA now directly generates interventions. This baseline adopts a dual- model architecture, comprising (1) a VLM for rationale generation, same as the base VLM for MemeSense and (2) a T5-large model for intervention generation. The rationale generation VLM is fine-tuned for one epoch with batch size of 4 and learning rate of
5 x 10-5
(3) Commonsnse--mhance prompting: Given a meme and its automatically generated commonsense parameters, the VLM (same base VLMs as those for MemeSense) is instructed to generate an intervention. (4) In-context learning (ICL) Zeng et al. (2024): For a given larget meme, we select. k (E {1, 2, 4, 8, 10)) demonstration examples from the training set, including their annotated commonsense, and provide them as context before prompting the VLM generate an intervention. For the selection of in-context examples, we use random and semantic retrieval techniques similar to Stage II (Section 4.2
6.2 MemeSense framework
Recall that MemeSense consists of three major stages leveraging (I) multimodal LLMs for generation of commonsense parameter, (II) in-context exemplars selection and (III) subsequent learning of the cognitive shift vector for the intervention generation. For the Stage I, we utilize the 1.ava-v1.--mistraal 6-mistral-7b-nff model, fine-tuned with QLoRA Dettmers ct al. (2023) over 10 epochs using batch size of 16 and learning rate of 2 x 10-4, with weight decay for optimization.
For the Stage II, We employ various strategies for selecting in-context exemplars, detailed as follows: Commonsensse-usse retrieval: For each predefined commonsense parameter, select up to five instances from our training set to form a lookup set. Given an anchor image img and its corresponding annotated commonsense parameters, we iteratively retrieve at least one instance per parameter to construct the k demonstration examples.
htttp::/huggginttoooco///// http:/aaw....olaaaaaaa/llleeeeee 6b-isaaa----

---

Image-based retrieval: For given anchor image img, we retrieve & demonstrations by computing their semantic similarity with img from the training subset. To achieve this, we first encode all images into dense vector representations using the CLIP-ViT multimodal embedding model. When an anchor image is provided as a query, we map it inlo the same vector space, enabling an elficient similarity search. We then perform Approximate Nearest Neighbor (ANN) Wang et al. (2021) search identify the top & most similar images. Their corresponding commonsense parameters and ground truth interventions retrieved as in-context. examples, ensuring contextually relevant selection.
Combined retrieval: We also experiment with constructing the k in-context demonstrations by combining the above two approaches. Here, we select e instances from the commonsense based retrieval and (k instances from the image-based retrieval, where c E 21.
IU model to learn cognitive shift vectors and For Stage III, we primarily employ the idefics2-8B-base perform inference. In addition, we explore idefics-9E and OpenFlaminge for intervention generation. The number of in-context demonstration examples is one of {I, 2, 4, 8, 10), maintaining fixed batch size of 2. The shift vector undergoes training for 10 epochs to ensure effective adaptation and we choose y 0.5.
6.3 Evaluation metrics
To rigorously assess the quality of generated interventions, we employ diverse set of evaluation metrics spanning semantic similarity, lexical accuracy, and readability. Semantic metrics such BERTScore Zhang
ett al. (020)) and semantic cosine similarity Rahutomo et al. (2012) measure the aligument between generated
and reference interventions in embedding space. Lexical metrics, including ROUGE-L Lin (2004) and
BLEE-- Papineni et al. (2002), evaluate surface-level text overlap and n-gram precision. Further, we measure readability using the Flesch Reading Easc Score, implemented via the textstat Python library ensuring the interventions are not only accurale but also coherent and accessible. This holistic evaluation framework enables a nuanced assessment of intervention effectiveness across multiple linguistic dimensions. We use
RoBERTa-large model for computing BERTScore, and A11-MiniLM-16-v2 from the Sentence panteneeTTunsformmrr library to compute semantic similarity.
7 Results
We structure experimental results into three key sections. First, present insights derived from dataset, highlighting key patterns and observations. Next, we evaluate the performance of our framework on the ICMM dataset, examining its effectiveness in generating interventions. Finally, we delve into a detailed breakdown of performance across different commonsense meta-calegories, offering à deeper understanding of the model's strengths and limitations in various contexts.
Result for our dataset In Tables 2 and 3 we compare the performance of our framework, MemeSense, with various baselines memes without text and memes with text, respectively. Across both settings, MemeSense (combined) consistently achieves the highest values for BERTScore (0.91), semantic similarity (0.71 for the memes without text, 0.78 for text-based memes), and ROUGE-L (0.35 and 0.37, respectively), demonstrating its superior capability in generating semantically meaningful and contextually appropriate responses. Among the baseline methods, commonsense-anchhn ICL performs competitively but lags behind MemeSense, particularly in terms of semantic similarity score, highlighting the importance of hybrid reasoning strategies.
For memes without text, direct prompting methods struggle with low semantic similarity (S 0.3), while MemeSense (combined) significantly outperforms them (semantic similarity = 0.71).
8

---

Table 2: Result for memes without text. SeSS: semantic similarity. indicatess
statistically significant improvement from MemeGuard and MemeMQA using Mann- Whitney U test with p 0.05.
Method
BERTScore (F1) Se5S
Readability ROUGE-L (Arg) BLEU-4 (Avg)
Direct prompting
0.81
0.27
53.36
0.06
0.001
prompting (w. commonsennse)
Random ICL
0.87
0.49
35.06
0.19
0.01
Inage anchored ICL
0.86
0.11
36.19
0.17
0.02
Commoneeize anchored ICL
0.88
0.46
34.12
0.18
0.32
MennMQQA
0.86
0.51
52.86
0.08
0.008
ManeCuard
0.82
0.005
MemeSense (random ICL.
0.90
0.68"
46.22
0.34
0.07
MemoSenso (Image anchorod ICT.)
0.35
0.08*
MemeSonse (ommmoneeee anchored ICL)
0.91
0,70
45.
0.35
0.09
MemeSeuse (combined)
0.91
0.71
11.07
0.35
0.08
Table 3: Result for memes with text. SeSS: semantic similarity. iidicatess statisti-
cally significant improvement from MemeGuard and MemeMQA using Mann- Whitney U lest with 0.05.
Method
BERT Seore (FI) Se88
Readability ROUGE-L (Aeg) BLEU-4 (Avg)
Direct prompting
0.81
0.35
$4.59
0.94
0.001
Direct prompting commonsimsi)
0.80
22.02
0.001
Random ICL
0.86
0.52
31. 91
0.18
0.02
Image anchored ICL
Commonsetse anchored ICL
0.88
0.55
33.25
0.19
0:03
MenneMQA
0.86
0.51
50.25
0.10
0.009
MeneCGurd
0.84
0.39
36.36
0.09
0.004
MemoSense (random ICI,)
0.91
0.77"
46.84
0.36
0.08
MemeSense (image anchored ICL)
0.35'
0.07'
Mem:SSuse (cmmmonnense anchored ICL)
0.91
0.78
18.71
0.35
0.09
MemeSense (combined)
0.37
0.08
We want emphasize that MemeSense achieves 35% improvement in SeSS score and 9% in BERTScore over MemeGuard, and 20% improvement in SeSS score and 5% in BERTScore over MemeMQA which are the state-of-thhe-at methods.
These improvements highlight the effectiveness our adaptive approach reasoning about complex memes without having textual cues. Similarly, for memes with text, MemoSense achieves notable improvements both semantie alignment. and lexical overlap (BLEU: 0.08 0.09), reflecting its ability effectively integrate commonsense and mmage-groundee reasoning. Overall, these results demonstrate that the MemeSense (combined) approach integrating image-anchored, and commonsene--ncchree in-context learning (ICL), effectively enhances reasoning and interpretation across different meme types.
Table 4: Comparing the result for the ICMM dataset using the actual model used by MemeGuard in different stages. indicates statistically significant improvement from MemeGuard and MemeMQA using Mann- Whitney U test with < 0.05.
Mothod
BRRTSowe
(F()
Se55
Readability
ROUGE-L BLEU-4
(Ang)
(Any)
prompting
Direct prompting with commonscnsee
0.8
0.11
52.34
0.05
0..044
Random ICL
I182
01.65
19.63
0.10
ILDKS
anchored ICL
012
Commonsene anchored
0.81
0.22
MomeMQAA
0.85
0.21
51.45
0.1
0.007
MomeGrard (our setting]
0.70
0.18
34.45
0.04
0.001
ICL)
MemeSouse (image anchored ICL)
0185
0125
02.79
01.1
11/07
MemeSeuse [commonsenac anchored ICL)
0.86*
0.27
62.22
0.11
0.009
MomoS (combincd)
we show the result of various baselines and compare them with MemeSense for the ICMM dataset. Direct prompting achieves the highest readability (67.02) but performs poorly in semantic aligument (SeSS 0.15, ROUGE-L 0.03, BLEU 0.001), while adding commonsense
Result for ICMM data In Table 4

---

knowledge reduces readability further (52.34) without improving semantic scores. In-context learning (ICL) methods, including random, image-anchored, and commonsense-anchen ICL, improve semantic similarity (0.16-0.22) and ROUGE-L (0.09-0.1) but suffer from significantly lower readability (19.63-25.38). Among meme-speeiiic baseline models, MemeMQA performs best (SeSS = 0.24, readability = 54.45) as it requires explicit training, while MemeGuard underperforms across all metrics (SeSS 0.18, readability 34.45). MemeSense outperforms all baselines, with MemeSense (commonsense anchored ICL) achieving strong semantic aligument (SeSS = 0.27), while MemeSense (combined) emerges as the best overall method with the highest BERTScore (0.87) and SeSS (0.31), reasonable readability (45.57), and competitive ROUGE-L (0.11) and BLEU (0.008) scores. This suggests that structured multimodal approaches, particularly MeneSense (combinod), provide the best balance between semantic coherence and fluency, making it the most effective meme intervention generalion stralegy.
Results for social commonsense categories: Table 5 presents the performance of our model across different broad social commonsense cate- gories, evaluated using BERTScore (F1), semantic similarity (SeSS), and ROUGE-L. Notably, for all four categories, the results are very similar showing the robustness of the design of McmeSense. The model achieves the highest, scores in recognizzng social norm violations (BERTScore: 0.91, SeSS: 0.79, ROUGE-L: 0.38), suggesting strong alignment with human references in identifying and intervening in socially inappropriate memes containing themes such vulgarity, serual content etc. For the other three categories also the results are quite close in terms of all three metrics (BERTScore: 0.90/0.91, SeSS: 0.72-0.78, ROUGE-L: 0.33-0.37).
Table 5: Meta category- evaluation results.
Meta category (Commonsense))
BERTScore
Se88
ROUGE-L
(F1)
(Aug)
Coutextual interpretation
0.91
0.78
0.37
Empathy and ethical judgements
0.90
0.75
0.33
Prodicting consequences
0.90
0.72
0.33
Recognising social norm violations
0.91
0.79
0.38
8 Discussion
Error analysis To better analyze the limitations of MemeSense, we conduct detailed error analysis by examining its predictions and identifying cases where erroneous classifications occur. We categorize the errors into two types:
(1) False negative (Category error): Instances where the meme is actually harmful and should be flagged as unsafe, but MemeSense incorrectly predicts it as safe for posting. Since non-harmful memes are absent from the dataset, the concept of false positives does not arise in our evaluation.
(2) Improper reasoning (Category error): Cases where the model correctly identifies the meme unsafe but provides incorrect inadequate reasoning for its decision.
Among 51 such instances in our dataset, MemeSense exhibits Category 1 errors in 6 cases. Notably, in 5 out of these 6 cases, the commonsense parameter generation stage fails to accurately infer the harmful category, leading to incorrect classification. specific example this failure is observed when the model incorrectly identifies cultural sensilimily as the primary harmful calegory for meme that is actually vulgar, ultimately leading to its misclassification as safe for posting.
Further, we identify one instance of Category 2 error, where the model predicts the meme unsafe but fails to provide coherent justification. This error arises due to improper reasoning during the commonsense parameter generation stage, which affects the interpretability and reliability of the model's intervention. A more detailed breakdown of the failure-cases is demonstrated in the Appendix
Ablation studies In the error analysis, we observed the major prediction error appeared due to the incorrect generation of commonsense parameters. Hence we investigate, how much the final inference is dependent on the generated commonsense parameters. To achieve this, we obtain the inforence from our approach without providing commonsense information to the model. Using only the input image and its corresponding description, we attempt to infer the intervention from our approach using the best method (MemeSense (combined)). The combined model is trained using the commonsense information. However, during the inference we not providing the commonsense, removing the requirement of commonsense generation module during inference. We observe maximum decline in semantic similarity score of 4%
10

---

without commonsense information. In addition, we observe that the interventions are more descriptive, which is reflected in the increase of the readability score.
Effect of coefficient a To understand the effect of coefficient a in the Equa- tion I we conduct an ablation by setting a; = (non-trainable), thereby isolat- ing the effect of CSV. This resulted in consistent performace drop accross all dataset. BERTSccore decreased to 0.87 (4%) for for memes with and without text, and BERTScore reduced by 5% for ICMM datasct. Full result is shown in Table 6 These results suggest that removing the coefficient a leads to a notable decline in both semantic and surface-level quality of the generated interventions. a plays a crucial role in adaptively regulating commonsense infusion while generating intervention.
Table 6: Result for intervention generation for different test sets without coefficient a.
Test set
BERTScoTe
(F1)
8c58
Roadabitity
ROUGE-L
(Aeg)
BLEU
(Ang)
Memes without
0.87(
0.61(
41.56(
-2.1
0.22|
0.03
Memes with ICMM
(-.04]
0.66
-2.17
0.25(
-.11)
0.03
0.82( 00
0.21
43,33|
-.24
0.07(
-.04
0.006 .002
MemeSense sensitivity analysis In addition to the ablation studies presented in Table 7 we conduct a sensitivity anal- ysis to assess the impact of variations in the commonsense information provided to the model. Specifically, we evaluate how MemeSense (combined) performs when supplied with randomly selected commonsensse knowledge during inference.
Table 7: Result for intervention generation for different test sets without using the commonsense parameters.
BERTScore
ROUGE-L
BLEU-4
Test set
(F1)
8:58
Readability
(Ang)
(Aug)
Memes without text
0.89 -.02
0.68(-003)
51.02(
+6.95
0.31
0.07(
-.01
Memes with text
0.9
(-.01
0.71(-.04
47.79(
+1,41
0.32
0.061 -.02
ICMM
0.85( -00
-.24)
54.18( (8662)
0.10
0.007
This experiment aims to understand the model's 'ss sensitivity to incorrect or unrelated commonsense attributes.
As shown in Table g observe a noticcable decline in performance across key metrics when randomly sclected commonsense information is used. In particular, the semantic similarity score decreases by approximately 9%, indicating that misattributed commonsense knowledge can significantly affect the model's final outcome. The decline is also reflected in BERTScore, ROUGE-L, and BLEU, demonstrating the reliance of MemeSense on relevant commonsense reasoning for effective intervention generation. Interestingly, readability exhibits a slight improvement for memes with text, which could be attributed to the increased linguistic diversity introduced by the random commonsense selection. These findings highlight the importance of precise commonsense altribution in ensuring robust and reliable meme interpretation. We present case study in Appendix D where we examine the impact of commonsense reliability the final intervention generation.
Table 8: Result for intervention generation for different test sets using randomly selected commonsense parameters.
Interpretability of cognitive shift vectors To assess the interpretability of CSVs and their correlation with com- monsense parameters, we conduct two experiments as follows.
BERTScore
ROUGE-L
BLEU-4
Test set
(F1)
Se5S
Readobilsty
(Aeg)
(Any)
Memes without text
0.88( -.03
0.64( -007
36.76i
-7.11
0.27
-08
0.05[-.00)
Semantic consistency within com- monsense parameters We analyze whether CSV representations exhibit structured patterns within specific commonsense parameters. From the test set, seleet five memes associated with particular commonsense parameter and pass them through the MemeSense framework. We extract the hidden representations of the first generated token and compute the average pairwise Euclidean distance between these representations. In contrast, we repeat the process with five memes from different commonsense parameters. We observe that memes sharing a common parameter exhibit lower pairwise distances compared to those from mixed categories. For example, the average Fuclidean distance among representations of memes labeled with "vulgarity" is 0.21, whereas increases to 0.28 when considering memes from multiple categories. This suggests that CSVs capture task-relevant semantic similarities. Correlation between commonsense parameters and representation similarity: We investigate
Memes with text
0.89
02
0.091
-00
46.30
2.98
0.28(-088
-08
0.06
ICMM
0.85|
02
0.27
-04
34.07
11.500
0.100
-00
0.007
-0001
11

---

whether hidden representations align with commonsense parameters that frequently co-occur. For instance, "vulgarity" often appears alongside "sexual content," " while "stereotyping" commonly co-occurs with "Hate Speech." " To analyze this, we select five memes from each of the top five most froquently co-occurring calegories, process them through MemeSense, and compute the average pairwise Euclidean distances of the first generated token's representations. Our findings indicate a strong negative correlation (o = -0.67) between category co-occurrence frequency and pairwise Euclidean distances. This suggests that conceptually related memes yield similar intervention representations, reinforcing the utility of CSVs.
These results suggest. that CSVs effectively capture structured semantic patterns, supporting their role in task-relevant information distillation.
Intervention quality measurement. To assess the quality of the generated intervention, we performed a quantitative and qualitative analysis as described below:
(1) Measuring argument quality: Wc aim to measure the argument charactcristic of the generated response com- monly used for measuring quality of online counterspeech (ahaa et al. (2024). We use mode finetuned the argument dataset Stab et al. (2018). Given this model, we pass each generated intervention through the classifier to predict a confidence score, which would denote the argument quality, We obtain confidence scores of 0.67, 0.74, 0.79 for the memes without texts, memes with text, and the ICMM dataset respectively suggesting strong argument quality of the generated interventions.
(2) Correlation with human judgments: While we present most of our results with automatic metrics, it is important to understand if they correlate with human judgments. We took two metrics BERTScore (F1) and ROUGE-L (Arg). For each metric, we randomly extract 25 samples from the prediction set. We present these to human annotators (rescarchers in this domain) and ask them to rate the quality of intervention from 1-5, 5 being the best and 1 being the worst. The Spearman's 's rank correlations between the human judgments (ordinal) and the automated metries (continuous) are 0.58 and 0.49 respectively which indicates moderate to high correlation [5] Given the subjective nature of the task, these results highlight a substantial consistency between automated metrics and human judgments, affirming their reliability.
(3) Crowd based intervention evaluation: We conduct a large-scale human evaluation on all 184 test memes (with and without overlaid text) using the Prolific platform, recruiting 81 high-approvvl--rat participants with prior experience and compensating them fairly. Each meme-intervention pair generated by MemeSense is independently evaluated by three raters, who choose one of three labels: (i) the intervention correctly addresses the harmfulness, (ii) the intervention is correct but the justification is incorrect, or (iii) both are incorrect. To ensure quality, we provide detailed task instructions, category definitions, examples, and a disclaimer about potentially harmful content. We observe fair inter-annotator agreement (Fleiss' к = 0.52), which is reasonable given the subjectivity involved. Importantly, in 84% of cases, at least two annotators agree that the intervention correctly addresses the meme's harmfulness, supporting the practical effectiveness of our method.
Generalizability across cultural and lin- guistic settings Our current framework pri- marily targets memes containing English text or no text. To enhance diversity and evalu- ate broader applicability, we additionally in- clude the ICMM dataset, which features nu- merous code-mixed examples (Hindi words in Latin script interspersed with English phrases). To assess cross-cultural and cross- lingull robust- ness, we further introduce two datasets: (1) Hindi meme dataset Dubey et al.] (2025) com- prising memes with overlaid text exclusively in Devanagari script, and (2) the Cm-Off-Meme datasot Kumari et al. (2024) containing Hindi-
Table 9: Comparative results of MemeSense using other models.
Used model
Mothod
BErrd'Scure (F1) Sc55 Rouge- L (Avg)
Memes without text
Idefica-98
MemoSonse (combined ICL)
Menu-Sense (random ICL)
0.89
0.9
0.71
0.68
0.34
0.31
nporFtaningo-99
Memo-Snnse (combined ICL)
Mern-Semnn (rondom ICL)
0.0
0.7
0.20
0.32
Momes with text
Mern-Senan (wendom ICL)
0.33
Idefics-98
Memo-Senso (combined ICL)
0.91
0.77
0.58
Open
MentoSense (rardoom ICL)
1,89
0.74
0.32
MemeSSnnsn (combiued ICL)
0.35
1CMM data
Idefics-98
MemeSonse (combiued ICL)
MenteSense (nandom ICL)
0.85
0.27
0.3
0.1
1.1
BpenFlaningo--0
Meme-Semsn (raudom ICE)
0.85
0.26
0.00
MemneSense (combined ICL)
0.85
0.29
0.1
English code-mixed memes. We annotate 50
15 htttes//aaaaabbbeee///too/pppaap
12

---

examples from each dataset, using GPT--4o for initial commonsense generation followed by manual correction by two native Hindi speakers to ensure linguistic and cultural fidelity.
Using our fine-tuned 1lava-1.6-mistral. 6-mistral-7b-hf, we first generate commonsenns parameters. As expected, performance drops considerably for Devanagari-text memes due to the model's English-centric pretraining, whereas for the rode-mixed Hinglish dataset, results improve substantially with only occasional failures arising from partial misinterpretation of Hindi tokens. The BERTScore of the Cognitive Shift Vector (CSV)-basod interventions for the two dalasets are 0.48 and 0.86, respectively.
To further investigate whether multilingual vision-language models enhance intervention quality, we fine-tune 1-1.--iiiiino ----iiiiin---nruutt on our training instances for commonsense generation (Stage 1). This yields significantly higher BERTScores of 0.85 and 0.89 for the two datasels, demonstrating that our framework readily adapts to multilingual settings with suitable backbone architectures. Note that the CSV module is not trained in either case.
Runtime analysis Since our framework uses multiple stages to generate the final intervention, it is crucial to analyze computational efficiency of the framework. We compare the inference time of our approach with the k-shot LLM based approach on the ICMM dataset. Since, fine-tuning (slage I) and training cognitive shift vectors (stage III) are one time processes, it does not affect overall inference time. However, if we keep increasing the number of in-context examples in simple k-shot prompting, the computational cost as woll as the inference time significantly increases. For instance, inference from 4-shot ICL will take 5.4x time compared CSV, whereas inference from 8-shot ICL will take 9.1x time compared to CSV. However, inference from CSV will take only 1.2x time compared to standard zero-shot prompting. But the performance of zero-shot prompting is significantly poor (See the Table 2). For further understanding the trade-off between training inference time of CSV compared to the k-shot prompting, we showcase the total time taken to infer from ICMM dataset in Table I0
Use of alternative LLMs In the Table g we show the comparative results of McmeSense using different base LLMs (Idfics-9B and OpenFlamingo-99) Here we use the annotated data mentioned in 4.1 and the retrieval of in- context exemplars mentioned in Section 4.2 to train the cognitive shift vectors (mentioned in Section with these two basc models. Then we perform the inference using trained cognitive shift vectors. We observe a similar pattern as earlier for these two LLMs. Moreover, Idefics-9B shows an overall superior performance compared OpenPlamingo-9B.
Table 10: Total runtime comparison.
Method
Total Time
D-shot (ICL)
24.6 Min
1-shot (ICL)
57.45 Min
2-shot (ICL.
92 Min
4-shot (ICL)
160.8 Min
8-shut (ICL)
269.2 Min
MemeSense (Training + Inference) 111.5 Min Min 29.5 Min]
9 Conclusion
In this work, we introduced MemeSense, a three-stage, adaptive in-context learning framework that integrates visual and textual cues with social commonsense knowledge for robust meme moderation. By combining compact latent representations, carefully retrieved in-context exemplars, and cognitive shift vectors, our approach captures subtle, implicitly harmful signals, including memes without explicit tert that often evade traditional pipelines. Experiments on our curated dataset. and the ICMM benchmark highlight MemeSense's superior performance in generating semantically aligned interventions, surpassing state-of-the-art baselines. We hope MemeSense inspires broader rescarch in in-context learning toward fostering safer, more responsible online communities.
Broader impact statement
MemeSense introduces socially grounded meme moderation approach using in-context learning with commonsense cues, enabling detection of subtle harms in text-light content. While promoting safer online spaces, it carries risks of overreach, cultural bias, and misuse. The authors aim to mitigate this through
13

---

transparent model release and advocate for culturally inclusive annotations and human oversight. Its real-world impact depends on responsible deployment and ethical safeguards.
References
Siddhant Agarwal, Shivam Sharma, Preslav Nakov, and Tanmoy Chakraborty. MemeMQA: Multimodal
question answering for memes via rationale-based inferencing. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compulational Linguislies: ACL 2024, pp. 5042 5078, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024 findings-acl3300. URI. https:://ccaantho//
org/2024. f indings-acl.300 3007
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Jain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob Menick, Scbastian Borgeaud, Andrew Brock, Aida Nematzadch, Sahand Sharifzadch, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning, 2022. URT
https://arxiv.org 2204. 14198
Rie Kubota Ando and Tong Zhang. framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817-1853, December 2005. ISSN 1532-4435.
Arnav Arora, Preslav Nakov, Momehil Hardalov, Sheikh Muhammad Sarwar, Vibha Nayak, Yoan Dinkov, Dimitrina Zlatkova, Kyle Dent, Ameya Bhatawdekar, Guillaume Bouchard, and Isabelle Augenstein. Detecting harmful content on online platforms: What platforms need vs. where rescarch efforts go. ACM Compul. Surv., 56(3), October 2023. ISSN 0360-0300. doi: 10.1145/360339) URL ttttp::/ddoi.org//// 1145/3603399
Somnath Banerjee, Sayan Layek, Soham Tripathy, Shanu Kumar, Animesh Mukherjee, and Rima Hazra. Safeinfer: Context adaptive decoding time safety alignment for large language models, 2024. URL
hatps://arxiv.orgee /406. 12274
Somnath Banerjee, Sayan Layek, Hari Shrawgi, Rajarshi Mandal, Avik Halder, Shanu Kumar, Sagnik Basu, Parag Agrawal, Rima Hazra, and Animesh Mukherjee. Navigating the cultural kaleidoscope: A hitchhiker's guide to sensitivity in large language models, 2025. URL https://arxxi, org/oogg////o.oeooooooooooo.ee http//bs/221..... 12880
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vose Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jelfrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxxx: www.wwwww....raaeoooo abs//b/s/55500 14165,
Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, Volker Tresp, and Jindong Gu. Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning? In 2025 JEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 6000-6010, Los Alamitos, CA, USA, 2025. TEFE Computer Society. doi: WAMp///AAV600....
URL
www.ciiittt.or/1.... 1109//ACV61001 2058.00585
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms, 2023. URL https::// https:///axvvooro hangg/abs/2000 2t0bs///2355... 14314
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024. URL
attps://arxxx orgg/baaaa...ooo.. 2ttbs//230.0000 00234
Kriti Dubey, Vaishnavi Srivastava, Garima Sharma, Nonita Sharma, Deepak Sharma, Uttam Ghosh, Osama Alfarraj, and Amr Tolba. Multimodal detection of offensive content in hindi memes. ACM Trans. Asian Low- Res-Re. Lang. Inf. Process., February 2025. ISSN 2375-4699. doi: 10.1145/8717611. URL
https://doi. org 11tp//101115//3777//00 Just Accepted.
14

---

Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions, 2024. URL https:
Vaishali U Gongane, Mousami V Munot, and Alwin D Anuse. Detection and moderation of detrimental content on social media platforms: current status and future directions. Social Network Analysis and Mining, 12(1):129, 2022.
Rima Hazra, Sayan Layek, Somnath Banerjoc, and Soujanya Poria. Safety arithmetic: A framework for test-lime safety aligument of language models by steering parameters and activations, 2024. URL
11801
Bing He, Mustaque Ahamad, and Srijan Kumar. Reinforcement learning-based counter-misinfoommaiion response generation: A case study of covid-19 vaccine misinformation. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 2698AA52709, New York, NY, USA, 2023. Association for Computing 1145/
Machinery. ISBN 9781450394161. doi: 10.1 11145///5-35577333333333 URL
3543507 3583388
Ming Shan Hee and Roy Ka- Wei Lee. Demystifying hateful content: Leveraging large multimodal models for hateful meme detection with explainable decisions. Proceedings of the International AAAT Conference
on Web and Social Media, 19(1):774-785, Jun. 2025. doi: 10.1609/iewsm.y (iwwsm.v1991.3588...
URL https
7/ojs. / .rgg herg/indx....
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, 2023. URL
https://arxin: httpee//axvvvvo////a/// argg/bb//3111 15916
Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learning. 2024.
Raghav Jain, Krishanu Maity, Prince Jha, and Sriparna Saha. Generative models vs discriminative models: Which performs better in detecting cy cyberbullying in memes? In 2023 International Joinl Conference on Neural Networks (IJCNN), pp. 1-8, 2023. doi:
Prince Jha, Raghav Jain, Konika Mandal, Aman Chadha, Sriparna Saha, and Pushpak Bhattacharyya. MemeGuard: An LLM and VLM-based framework for advancing content moderation via meme intervention. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8084 8104, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10. 10.18653/v 000655/vl220 acc.ooooo.....og....oooo URL https://acll http://aaaanthoooyy.oooooo.o.oaa org/ 20egg/224..ce acl- 439/
Prince Jha, Krishanu Maity, Raghav Jain, Apoorv Verma, Sriparna Saha, and Pushpak Bhattacharyya. Meme-ingful analysis: Enhanced understanding of cyberbullying in memes through multimodal explanations. In Yvette Graham and Malthew Purver (eds.), Proceedings of the 18th Conference of the Europeon Chapler of the Association for Computational Linguistics (Volume 1: Long Papers), pp. Malta, March 2024b. Association for Computational Linguistics. URL puttps//aacantthit eacl-1ong.50 56/
St. Julian's ss org/2024 2024
Gitanjali Kumari, Dibyanayan Bandlopadhyay, Asif Fkbaa, and Vinutha B. NarayanaMurthy. CM-off-meme: Code-mixed Hindi- Enndi-English offensive meme detection with multi-task learning by leveraging contextual knowledge. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenei, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint. International Conference on Compulational Linguislics, Language Resources and Evaluation (LREC-COLING 2024), pp. 3380 3393,, Torino, Italia, May 2024. wrec-main.
ELRA and ICCL. URL http:: https://aclanttooooggooooo
Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering, 2023. URL https: pestps://arree /ttp///axvvvvv.. org/ abrggaabs/331... 01571
15

---

Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Tezt Summarization Branches Out, pp. 74 81, Barcelona, Spain, July 2004. Association for Computational Linguisties. URL https://aclantholl. www.aaaooooooooooo/00-0-00000000 -013//
Krishamu Maity, A. S. Poornash, Shaubhik Bhattacharya, Salisa Phosit, Sawarod Kongsamlit, Sriparna Saha, and Kitsuchart Pasupa. Hatethaisent: Sentiment-aided hate speech detection in thai language. IEEE Transactions on Compulational Social Systems, (1(5):5714 57227,, 2024. doi: 10.1109/TCSS, N0.109//CCSS.2223333363.8 2024.3376958.
Binny Mathew, Navish Kumar, Ravina, Pawan Goyal, and Animesh Mukherjee. Analyzing the hate and counter speech accounts on twitter, 2018. URL https: http://wwww...ra//////1./0./..00
John A Naslund, Ameya Bondre, John Torous, and Kelly A Aschbrenner. Social media and mental health: Benefits, risks, and opportunities for research and practice. Journal of Technology in Behavioral Science, 5 (3):245-257, September 2020. doi: 10.1007/s411347-020000000
Office of the Surgeon General (OSG). Social Media and Youth Menlal Health: The U.S. Suryeon Generalã A2s Advisory. US Department of Health and Human Services, Washington, DC, 2023. URL https://www
ncbi nlm. www.ni...g...o.. wahh.go/bbooks////5544777
OpenAI, Aaron Hurst, Adam Lerer, and Adam P. Goucher et al. Gpt-4o system card, 2024. URL
/ttps://arxiv..o
Fengjun Pan, Anh Tuan Luu, and Xiaobao Wu. Detecting harmful memes with decoupled understanding and guided cot reasoning, 2025. URL https:///axxi: wwtpr/iiioor//aaa... argg/bb//506..... 08477
Kishore Papinemi, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1003008.10000 1073135. URL httpps//acllanthologgy www.o/oooooog//2-00000000000
Yingzhe Peng, Chenduo Hao, Xu Yang, Jiawei Peng, Xinting Hu, and Xin Geng. Live: Learnable in-context. vector for visual question answering, 2024. URL https://arxiv.. org/ wwrg/abs/24066 13185
Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. benchmark dataset for learning to intervene in online hate speech. In Kentaro Imui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empiriaal Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP) pp. 4755 4764, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18053/1.... / httpa//aaaaaollloooogg///////
URL
Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Lí, and Wanxiang Che. What factors affect multi-modal in- context learning? an in-depth exploration. In The Tiirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httts:///pperreviet feett//orumiiddREVaaaa
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: :/apxii////iii.e//aa org/ abs /t1/a/b///222..//// 15115
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krucgor, and Ilya Sutskever. Learning transfcrable visual models from natural language supervision, 2021. CRL http:://aaxiiv.orra/////000000
Faisal Rahutomo, Teruaki Kitasuka, and Masayoshi Aritsugi. https://api. wwwanticscholar...mmmmo htt.://ccchhoaroooooo/ooooaaaaaa
Semantic cosine similarity. 2012. URL
16

---

Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, and Animesh Mukherjee. On zero-shot counterspeech generation by LLMs. In Nicoletta Calzolari, Min-Yer Kan, Veronique Hoste, Alessandro Lonci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Lönguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 12443 12454, Torino, Italia, May 2024. ELRA and ICCL. URL http:://addaatiioooo..oro///////
maaeee-mmiinn.. 1090/
Carla Schieb and Mike Preuss. Governing hate speech by means of counterspeech on facebook. URL http::://ap..smmatticcc. hwt.aaarrrrrroooooSDD02222222330
Shivam Sharma, Ramaneswaran S, Udit Arora, Md. Shad Akhtar, and Tanmoy Chakraborty. MEMEX: Detecting explanatory evidence for memes via knowledge-enrichee contextualization. In Anna Rogers, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the As- socialion for Compulational Linguistics (Volume 1: Long Papers), pp. 5272 5290, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 2118653/v1/2223.cm https://aclaathooogyy wwg. 2ttp/2222.allloo... 289/
Jordan
URL
Andrew Shin and Takuya Narihira. Transformer-exclusive cross-modal representation for vision and language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2719-2725, Online, August 2021. Association for Computational Linguisties. doi: 10.18653/v1/2021.1. aid.....inngnnggn..2...... URL https://aclanthoo org/ 2021. f www.fiinnnn-acc.o 240/
Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. Cross-topic argument mining from helerogeneous sources. In Ellen Riloff, David Chiang, Julia Hockenmaier, and aun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3664 3674, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 1402/
10.18653/v1/D18-111 1402. URL
Gemini Team, Rohan Anil, and Sebastian Borgeaud et al. Gemini: A family of highly capable multimodal models, 2024. URL http:///aaaaa..oora/////////// 11805
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function 15213
vectors in large language models, 2024. URL http::://axxiv.oo/bbbbb////////
Mengzhao Wang, Xiaoliang Xu, Qiang Yue, and Yuxiang Wang. A comprehensive survey and experimental comparison of graph-based approximate nearest neighbor search, 2021. URL https: org/ /bs// 2101. 12631
Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning, 2024. URL httpss///axxx: ///axiv/viii/////// wwwrii..og/aaaiie.
14800
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(),,, November 2024. ISSN 2053-714X.
doi: 10.1093/mn 1008//ss/nnwa4000 URL http://dxx doi org/ 10w3/10r/////// nsr///444///
Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, and Kangwook Lee. Can MLLMs perform
text-to-image in-context learning? In First Conference on Language Modeling, 2024. URL https
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, 2024. URL. https://arxxv..rrr wttp://orrr/abb/24...0.. 2a0bb//2440008555
Tianyi Zhang*, Varsha Kishore*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Represenlations, 2020. URL https:
//openreview.net//oooe wwt./eeeuf/ddideoc Skeaaaeeu
17

---

A
Prompts
The prompt for generating ground truth commonsense parameters and intervention using GPT-4o is rep- resented in the Table 18
The prompts that we use for k-shot ICL based baselines are mentioned in the Table 19
Additional results
Evaluating ICMM results under original MemeGuard configuration. While Table reports results using our adapted version of MemeGuard for fair comparison, we also evaluate its performance under the original selup to validate consistencyy Specifically, we replace idefics-2-8b-base with Flan-T5-Large, the best-performing open-source LLM used in the original MemeChard implementation. This yields modest improvements: BERTScore increases from 0.79 to 0.82, ScSS score goes from 0.18 to 0.25, ROUGE-L from 0.04 to 0.08, and BLEU-4 from 0.001 to 0.009. Despite these gains, the model's performance remains below that of MemeSense (BERTScore: 0.87, SeSS: 0.31, ROUGE-L: 0.11, BLEU-4: 0.009), reaffirming the advantage of our method. These findings highlight that language-centric LLMs like Flan-T5 can enhance zero-shot text generation, but still fall short in capturing nuanced multimodal harm without additional commonsense grounding.
Table 11: Result for the ICMM dataset. indicates statistically significant improvement from MemeGuard and MemeMQA using Mann- Whitney U test with p < 0.05.
Method
BERrSScore
5c85
Rendability
ROUGE-L
(Ang)
BLEU-4
(Azg)
Direct prompting
11.8
01.15
67.12
0.315
[LIKT.
Divet prompting with commonsense
0.8
0.11
52.34
0.03
0.004
Random ICL
0.36
Image anchored ICL
0.82
0.2
22.10
0.006
Commorsemsn anchored ICL
0.84
0.22
25.38
0.1
0.006
MameMQQ
0.31
MemeGuard scttings)
0.18
MemeGuned (original)
0.82
0.25
47:22
0.08
0.000
MomuSense frendom [CL)
0.81
0.15
61.00
0.11
0.007
MamoSonse (image anchored
Mem:Seuse (commonsense anchored ICL)
0.86"
0.27"
42.22
0,11"
0.009
Meme-Seae (combined)
0.87
0.31*
45.57
0.11
0.008
C Additional dataset details
We deliberately select only the harmful memes to build our MemeSense framework. Initially collected total of 785 memes from different online resources as mentioned in 3 We ask two undergraduate students to unanimously mark whether the memes are harmful or not. To maintain consistency, we provided them with a short annotation guideline, which included example images of both harmful and non-harmful memes (similar Figure 3). More specifically, we ask them mark meme as harmful if it falls in the specified common sense category according to their judgments. This process resulted in 484 scrutinized harmful memes for experiments. Since the memes that do not have embedded text in it, represents mostly sexually explicit items, our dataset contains higher proportion of such memes (as reported in Table 1).
For the verification of the generated commonsense parameters and the intervention from the GPT-4o, we employ two expert reviewers to assess. We provide them with a short description cach commonsense category (Similar to the Commonsense parameters mentioned the Table 18] along with one selected meme within each category for reference. They were allowed to update, delete, or add categories based on their judgment. Finally in 18 out of 484 cases they were required correct the commonsense parameters and the corresponding interventions for meme.
D Case Study: Impact of commonsense reliability on intervention generation
To further examine the sensitivity of MemeSense to the quality of commonsense input, present qualitative case study analyzing how variations in the generated commonsense parameters influence the
18

---

Table 12: Hyperparameters for MemeGuard.
Hyperparameters
Task
Value
Temperature
Desc, Bias, Stervotype, Tox-
icity & Hate, Claim Gener-
ation
mum
beams
Desc, Bias, Stereotype, Tox-- icity Hate, Claim Gener-
ation
max new
tokens
Desc, Bias, Stereotype, Tox-
512
icity & Hate, Claim Gener- ation
Cosine Similarity Threshold
MKS Filtering
max
new
tokens
Interveution
1024
Table 13: Prompt used for different tasks in the MemeGuard method.
Task
Prompt
Description generation
Describe this meme in detail. the is conveying?
Sncial hias gen.
meme hias
Social stereotype gen.
What the societal stereotype that this is conveying?
Texicity and hate
What is the toxicity and hate that this meme spreading?
Claim the meme is making
What are the claims that this meme is making?
Intervention Generation
This la toxic meme with the description: ksl. The following text written inside
the
Rationale: Bias: ks2, Toxi- city: ks3, Claims: ks4, and Stereotypes: kss.. Write intervention for based on all this knowledge.
final intervention. This analysis builds upon the findings in Table where we measured performance under randomly selected commonsense attributes.
Our observations reveal two consistent patterns:
1. Robustness through partial accuracy: In cases where at least one of the predicted commonsense parameters aligns with the ground truth, MemeSense often succeeds in generating a contextually appropriate intervention. This suggests that the model capable of leveraging even partial common- sense grounding to orient the cognitive shift vector in a meaningful direction, thereby preserving semantic and ethical relevance the intervention.
2. Intervention disruption via semantically divergent commonsense: When the predicted commonsense parameters are semantically distant or rarely co-occurring with the ground truth categories-e.g, substituting Vulgarity with Cultural Sensitivity-we observe marked decline intervention quality. In such cases, the model's attention appears shift toward an unrelated ethical dimension, resulting in generic misaligned interventions.
These findings suggest that while MemeSense exhibits a degree of resilience to noisy commonsense input, its performance is sensitive to the semantic proximity between the predicted and actual commonsonse parameters. We include representative examples below illustrate both robust and failure cases under random commonsense perturbation. We include representative examples in Table 14 to illustrate both robust and failure cases under random commonsense perturbation.
E Comprehensive failure case analysis
Table 15 summarizes representative failure cases from MemeSense where the model's commonsense analysis surfaced useful cues but the final intervention label was incorrect. Typical errors include (i) under-detecting sexual immuendo in visually ambiguous memes, (ii) overlooking cultural or religious sensitivity despite partial
19

---

humor- Table 14: Case study illustrating examples where randomly provided commonsense either preserves or disrupts the quality of the generated interventions ion. The correct commonsense category highlighted in green and the semantically divergent commonsense is highlighted in red.
Case 1: Intervention not affected Case 2: Intervention affected by
Case
by random commonsense
random commonsense
Beatin' the
My lavourite meals to stretch apart
Meme image
Cultural Sensitivity; The meme ref- erences Mughal art and Pakistani iden- tity, involving cultural and historical elements. It may be seen as misrepre- sentative disrespectful.
Vulgarity: The text includes inappro- priate language and innuendos refer- encing male genitalia in metaphorical context.
Provided commonsense to the model
Humor Appropriateness: The hu- mor is based sexual analogy that might. be perceived crude offen- sive, especially given the literal image contrast.
Humor Appropriateness: The hu- mor stems from anachronism depict- ing a historical figure engaging modern sport-which some may find mocking or belittling cultural heritage.
The meme should not posted pub- licly due to its vulgar language and
The meme may be perceived as cultur- ally insensitive due to its use of Mughal art and Pakistani identity ous context.
Predicted intervention
inappropriate humor, which may of- fend or discomfort viewers.
The meme is inappropriate for public posting due to its vulgar innuendo and potentially offensive humor. It may not be suitable for all audiences and could lead to misunderstandings or dis- comfort.
The meme should not be posted pub- licly as includes suggestive sexual content that is inappropriate for broad audience. The humor could be seen as offensive or in poor taste, pos- sibly provoking negative reactions.
Actual intervention
hints in the reasoning, (iii) misjudging misogynistic framing benign "humor", and (iv) over-weighting salient visuals (c.g., crying face) while missing explicit vulgarity in text. These patterns suggest limits of our cognilive-shift and in-context exemplars when cues subtle, multi-cultural, or context-enlangledd. We view these as actionable diagnostics: strengthening cross-modal evidence aggregation, bias-aware reasoning for sensitive domains, and targeted instruction for stercotype and innucndo detection should reduce such errors.
F Qualitative comparison with explainable SOTA methods
We conduct a qualitative comparison of MemeSense with two recent explainable meme classification Trameworks INTMEME Dubey et al. (2025) and U-CoT+ Hee & Lee (2025). INTMEME first extracts the overlaid caption and processes it through a frozen large multimodal model (LMM) to generate an interpretation. This interpretation, combined with visual features, is then used to train classifier for binary hatefulness detection. Although INTMEME is not directly comparable due to its binary classification focus, we contrast its generated interpretations with the commonsense outputs of MemeSense. We find that INTMEME struggles to capture nuanced or implicit harms, especially in memes without textual overlays, and often produces partial or inadequate explanations even when text is present.
We also evaluate U-CoT+, which generates zero-shot explanations using multiple LLM calls and handcrafted task-specific prompts. While offers brief decoupled explanations, its reliance on multiple inference stages and domain-specifie tuning limits its real-time applicability. In contrast, MemeSense demonstrates greater scalability and adaptability, producing context-aware interventions efficiently even for visually implicit
20

---

Table 15: Representative failure cases from MemeSense intervention analysis.
Predicted
Meme
Ground truth
Analysis
intervention
Generated commonsense noted potential cultural sensitivity (hand gesture misinterpretationn) but the final prediction still marked it non-harmful. Illus- trates cases where cognitive-shift or in-context cues fail in isola- lion.
Restrict due to sexual inmuendo that may be unsuitable in some cultural contexts.
The meme does not appear to be vulgar, harmful, unethical.
The meme does not appear to be vulgar or harmful.
Restrict due to sexu- ally suggestive inter- pretation.
Commonsense flagged suggestive visual cues, yet visual ambiguity led to a non-harmful decision.
Restrict due to sexu- ally suggestive read- ing of the building's structure.
Commonsense hinted at inappro- priate humor from unusual struc- ture, but the final decision re- mained non-harmful.
The meme does not appear to be vulgar or harmful.
Review due to possi- ble mental-health im- pact.
Restrict for vulgar wording and disre- spect toward subjects.
Prediction overlooks explicit vul- garity, over-weighting the visual context (e.g., "crying person').
Review for potentially inappropriate humor in some cultural con- texts.
Commonsense misses misogyny: implies derogatory view of women as lalkative/nagginnn promoting scxism.
Restrict for misogynis- framing and sug- gestive innuendo.
Restrict for vulgar depiction and poten- tial offense espe- cially given the reli- gious context and hu- mor about a real loca- tion.
Restrict due to sugges- tive inmuendo not suit- able for all audicnces.
Harmfulness is detected, but commonsense fails to capture re- ligions sensitivity.
text-free memes through lightweight cognitive shift vector training. Example in Table 17 illustrate these comparative insights.
G Additional experimental settings
G.1 Baselines
In Table 13 demonstrate the different prompts used for the MemeGuard baseline. The hyperparameters for the experiments with this baseline are noted in Table 12
H Representative examples of memes from each commonsense category
21

---

What is similarity between)
match and human?
Tea heps
BACKEINN TO
When haverrt atolee
THE BLACK ONES DON'T WORK
nnvhinng
(a) Hate speech
(b) Body shaming
(c) Misogyny
(d) Stereotyping
(e) Sexual content
ELOSS MY FAAT
SOMETAINN BAYS FOE
knowyoal Randentand.
cood
(h)
Child
exploita-
(i) Public decorum & Privacy
ii)
Cultural sensitiv-
(f) Vulgarity
Misinformation
tion
ity
Imagine being stuck room surrounded by everyone you we ever hed with.
Dy
when you're bullying the weird kid and he goes home and kills himself
Condom
294 RETAVETTS
Wiee's med hareed
KILLS
ASSISTS
0
(k) Religious sensitiv- ity
(I) Humor appropri-
(m) Mental health
No it con be
ateness
impact
(n) Violence
(0) Substance abuse
Figure 4: Representative examples of memes from each of the 15 commonsense harm categories.
22

---

Table 16: Generated intervention using different approaches for (1) a meme without text, (2) a meme with text, (3) meme from ICMM data. Incorrect justifications are highlighted.
Max "Just davi you duallt Faridds Me
when the Seffenn www.eststtaeeerrroooooo fostaliy licrrts
It'S free teal estate
The meme is kumorous
that depicts woman pouring milk into container. The
The mome is a populae form of comminiies Hon that has been around for cesturine It is
toxs on the Im image reads, "Her: "Just make aure you don't finish inside of me. Me insiés of her
The meme is a pictume of a man with the cap- tton "when the school shootur fnally leaver and the bottess girl in claes is laying dead on the floor,, The meme makiing fwn of the school shooser and the girl who le dead. The memse also making fun the fact that the girl dead.
Dircei prompting
through Images and text. Momee can be numy serions, or even atfenxive.. They can used to spreed informmtioo, make point, oz just Satertain
way to express ideas, opintons, and emottoons
2 minutes ago." The meme is a play on the pirase 'just takke sure you don't 's finish inside of ma, which ie often ueed to indicate shat persen should use a condom during sex. The meme bumorous because suggeete texide concerned about actually
finthiing
time.
amiting
In image,, person sisting chuit, holding las dog stick The dog is being cooked joying the dogs on the grill; one lonated closer to the per- son and the other further away In addition to the hat dags, there are two marchmallows on the geill, one nour the person aud the other further away The overall atmocplieee appesrs to be fun and relaxced outdoor gathering:
camera. The reads "When the school shooter fimally leavee and the bottest girl In class is laying dead
Direct
grill, and the two other hot
There
The mome is insppropriase and violates com- manily guidelines againut explicit adult con- teut. It is nos suisable for general audiencea and may be offenstve or uncomfortable to some viewers.
the flooe its feee real catate" The combination the seat and image atsempia to make joke about polcutiolly Itagic aituntion by using dark humor. The meme zeferencces a school sbooting a highly sunattive and traumatic aub- ject. Making jokee about euch an event can be percetied as trivialising a grave and dis- treeking espartence, potentially camsing harm lhoze affected.
prompting
[w. common-
sense)
Celebratiox aulivi und cambracing divers Bily cabauees understandiss and accepiannt
This mome is vulguz, inappropriate, and poten- Liully harmiful, as il promotes a misogyistic
The mame should not posted publicly dur to its insesttive and inappropriats content, disturb certain
Random ICL
could
posted publicly
Thie humoroudly highlights the playfu) crextivity Houships, encouraging laughitey bonding
vulsus, inappropriate, and fielly hnsanfui, promotes micgynistic sexual
trivislizer trauma experienced similar individuale It bapproopriae humor In this context, it be percatved Inseneltive and offencive.
Image chored ICL.
The mome Its posting doe its insensstivee and offenaive content. It trivializes the teaumn of school shaotings and could be distreesing to individu- ale who have expertanced similar events. The use of humor in this context may be perocived ns inseuitive and offensive, particulerly con- atdering the serious nature of school shootings The subject matter could be percsived as violation public sensitivity and diseuasion publie consext.
be inappropriate for pub-
The image featurex man belding hos dog stick, with macommallow one end and a penis on the other This mome in disrespecstu) and mapppoppiitte, as it makes light a sert ous ssaue such as treetlle dysfanction, and usea it to create a homarrous or entertainisg siture ston: importans to aware of the impact of euch memer and to promote rnspectful and inclunive communication.
Commoncenze
The mome chould not be posted publioly due
anchored
to its valgur and suggestive content, which
ICL
could be offeusive to many viewern.
serious topice
MeueQA
The mome ahould ressicted duc its cou- leus and contentt is inappopriitt for public publtcly.
The meme should be restricted due con- and Inappropriate content.
The meme should not be posted publicly potential offensive to some audiences.
posting should
The meme harmfo) because relnforece neg- stive stereotypes about about
The mallows, but Instead of traditional hot dog,
making claim that the dox
and STL, and they not interwtted secual plaasure. It also that
This
highly inappropciule
offensive content, trivialising school shooting and making unathical implication shout vietime. This lind of meme should be critically examined and condemned rasher than shared
MemeGuard
there pente-ehaped hot dog. The Bkaly meant be humoroas and playfuul Inkes common outdonn activity like gritling and suide a provorstive twiet to Hi
concerned about thee leeues and are only intereated thetr own pleasure. This is harmfuu because it perpet- ustee the idea that women are not equal to mon in terma of sexuality and that they are not catitled to seauel plensure.
Momo Benso [Random ICL)
The mome should be restricted from public posting due to its vulkur and ouggeative Ea- ture, which could he percsived ar offenatien or inappropriate.
The meme should not be posted publicly due to Ite vulgarity, soxual content, and inagpropriate humer.
The mame should not be posted publtily due to its insemiiiie and potensially troumatising content, It trivialiees a serious issue and may
cause distreas to individuala who have experi- enced simillr eventa.
MomeBense
The meme should not posted publicly due to ito vulgar and supgestive nature, which could be perceived as offenelve insppeopriate
The mome should not be posted publicly due to its vulgur, seaual, and inappoopriate nature.
The mame should not be posted publicly dus insemilive and insppropriate conteal.
(Iaage on-
chored ICL)
be percelved offeneive spectful many andiencee.
trivialisco a serious sensisive topic, which can distressing individuals have ex-
perienced
MameSenen [Common-
should be zustricted from public posting due vulger and auggeative tent, which could be perceived as offeneive inappropiaae..
The
The zhould valgar and could olfeusive many viewers.
posted publely couteul, which
potancially and experi-
posted publicly
semse
Il Givielizes distrese individuals enced similar
chored TCL.
The mome should be rastricted from public posting due to its vulgar and suggetive con- teat, whith could be peresived inappropri- ate.
The meme should not be posted publiccly due vnlgar and angpectiv content, which could audiegces. It also wec and insppropriate.
The mome should not posted publicly due Itx Insencttvv and potentially tranmatic ref- creaces to achaol sheotings. It trivialiaes acrious insuc and may cause distress to indi- viduals who have experieneed simiaar evente. Joking about tragie events such as school shoosings and the loss of Innorent lives. ls In- sensttive and inappeoptiate. We should be mindiul of the imppect our words ena have ou others nod strive to promote eopaally and un- deretanding.
MemeSenau (Combined)
be offeasive to humor that could be perecived ns belidtling
Ground
The meme should be rastricted in cartsin con- tuxes due to itx potentially sugpostive content,
The mome should not be posted publioty due to its valgez and sexuully suggettve nature, which may be offeasive and inappropriste for gencral audiences.
trush anno-
which migist be percesved as vulgar or inappre-
tasion
priate, partioularly in public or professional setsings
23

---

Table 17: Qualitative comparison of explanations/ineerveee generated by INTMEME, U-CoT+, and MemeSense. All models are evaluated on the same meme input.
E
E's
S
Meme
The meme shows black man sitting next to a UPS truck, with his pants down around his ankles. The text 'UPS man's 's package' is overlaid on the image, indicating that it might be a sexual joke or a reference to the UPS employee having to deliver a package with an unusual or unexpected item. The presence of this meme could be construed a form of racial bias and stereotyping, it portrays black men being less responsible, less professional, and possibly even less respectful. The meme may also be objectifying and sexualizing black men, reinforcing harmful stereotypes about the African-American community.
Explanation by INTMEME
Slep-by-slep analysis: The image shows two individuals, one in uniform-like shirt (possibly delivery driver, given the partially visible "UPS" logo on the vehicle) and another person, seemingly in close physical contact inside the open rear door of a dark-colored van. The image is blurry, and the subjects' actions are suggestive of an intimate or sexual encounter. Aligument with Guidelines: Sexualization of Women: The depiction of an intimate act in public/wond workkreleeeee selting, particularly focusing on the apparent female subject being treated with the guideline on Sexualization of Women and potentially Objectification of Women. While the full context (a caption) is missing, the visual content itself focuses on a sexual encounter that can be interpreted as treating one person (the woman, given her typical attire compared to the other person) as an object satisfy sexual desire.
Explanation by U-CoT+
object for sexual desire, aligns
The posture and context imply suggestive or intimate scenario between the two individuals. Public Decorum & Privacy: The image captures private moment in public or professional setting, potentially without consent, thereby raising ethical and cultural concerns.
Commonsense generated by MemeSense
24

---

Table 18: Prompt to generate the ground-truth commonsense and interventions.
Prompt used for obtaining
parameters
As Al assistant for social media content moderation, your role analyze and prevent the posting of harmful, offensive, bateful, vulgar, misogyuistic, or otherwise inappropriate content on public platforms.
Task: Carefully analyze and interpret the given image, paying close attention to both visual elements and any text present.
1. Meme Description: Provide detailed description of the meme, including text, imagery, symbolic references.
Commonsense Analysis:
Assess whether the meme has the potential to be perceived as vulgar, harmful, or unethical. parameters from the predefined list below. Each selected category should be explicitly justified with a brief explanation.
Identify and categorize up specific
Predefined Commonsense Categories (Select Up 5):
Vulgarity: Use of obscene language, inappropriate tone, or implied innuendos.
Violence: Depictions of harm, cocrcion, references to physical aggression.
Sexual Content: Explicit or suggestive imagery, inmuendos, or nudity.
* Hate Speech: Insulting, harassing, offensive content targeting individuals groups.
Body Shaming: Objectification, fat-shaming, or insensitive references to disabilities.
* Cultural Sensitivity: Stereotypes, appropristion, or misrepresentation of cultural elements.
Mental Health Impact: Content that manipulates emotions, triggers traums, or exploits grief.
Humor Appropriateness: Use humor in a way that mocks, belittles, or distorts serious topics. Misinformation: Spreading mislcading narratives, false claims, deseptive dialogue.
Substance Abuse: References to drugs, alcohol, or promoting substance use.
Child Exploitation Content
inappropriately depicts exploits children.
Public Decorum Privacy: Violations of privacy, public sensitivity, or persoual reputation concerus.
Pherettppngg Generalizations that reinforce racial, gender, or societal blases.
Misogyny: Content promoting basee discriminstion, sexism, demeaning
Religious Sensitivity: Content that disrespects religious beliefs, symhols, or historical context.
Intervention Recommendation:
* the deemed inappropriate, justify why should posted publicly. the content is safe, confirm its appropriateness.
Respouse Format:
Meme Description:
<Provide detailed description of the meme, including text and images.>
Commonsense Analysis:
Caattoorryyy Name **; (Justification]
Cattggory Name **; Justification
** (Category Name)*t; (Jsstification)
Intervention Recommeudation:
<Explain whether should be restricted and why.>>
25

---

Table 19: Prompt used for different methods.
Method
Prompt
<Meme>> Analyze the meme thoroughly, considering meesage, symboliem, cultural relerences, and possible interpretations. Identity any implicit or explicit harm, misinformatton, or reinforcement negative stereotypes. Based this analysis, generate strategic interventions to discourage the spread creation of such content. These interventious should be precise, contextually relevant, and designed to effectively deter users from posting similar memes. They may include subtle deterrents, educational messaging, conteut reformulation, alternative framing that neutralizes harmful intent. Ensure responses are concise, non repetitive, and avoid redundant explanations. Ensure the response should exceed 50 words.
Direct prompting
<meme> Analyse the mome thoroughly, considering its message, symbolism, cultural refereners, and possible interpretations. Identify any implicit explici: harm, misinformation,, Based on this analysis, generate strategic interventions to discourage the spread or creation of such content. These interventions should be precise, contextually relevant, and designed effectively deter posting similar memes. They may include subtle deterrents, educational messaging, content reformulation, alternative framing that neutralizes harmful avoid rodundant explanations. The common sense parameters associated with the meme is as follows: {common _sense) Ensure the reeponse should exceed words.
reinforcement negative stereoly pes.
Direct prompting with commonsense
from
Ensure responses concise, non-repetitive, and
<meme's > Analyze this meme and generate caption that euhances its humor, sarcası, or irony. Do not filter offensiveness-prione prioritize humnr, satire, or dark humor needed. caption should punchy, relatable, and aligned with the meme's 's tone.
MemeMQA
<memee> > As an Al assistant tasked with social media content moderation, your role is to prevent harmtul, offensive, hateful, vulgar, misogynistic, or unethical content from being posted on public platforms.) in (a Your Task: assess whether the meme has the potential to he pereeived as vulgar, harmful, unethical. Write an intervention ion for the toxic meme to discourage user posting such memes based on provided knowledge. [commonsccsse paremeters] 'n \n [ezamples]
texic meme has the description below along with few commonsense parameters which
ICL
26


Reviews:
[{"review_id": "m711N4yZQL", "reviewer": "Reviewer_ctrj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces MemeSense, an adaptive in-context learning framework for meme moderation that integrates visual and textual understanding with curated commonsense-enriched examples. MemeSense is structured in three main stages: generating human-like commonsense parameters, selecting in-context exemplars relevant to the target meme, and learning cognitive shift vectors to adapt the model's internal representations. By using semantically and socially grounded cues, the framework aims to detect subtle and implicit harms—such as misogyny, vulgarity, or stereotyping—even in memes lacking overt textual content. Experiments on a newly curated dataset and the ICMM benchmark demonstrate MemeSense's superior performance over state-of-the-art baselines, with notable improvements in semantic similarity and BERTScore, especially for text-scarce memes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Rigorous experimentation is presented using both a newly curated dataset and the public ICMM dataset. Multiple baselines are compared, including MemeGuard, MemeMQA, Commonsense-anchored prompting, and several k-shot ICL variants. Ablation studies specifically dissect the role of commonsense information and the CSV components (Tables 6–8), and error analysis is performed (Section 8).\n- Table 2 and Table 3 show substantial improvements in BERTScore and semantic similarity over previous approaches, particularly for text-scarce memes, supporting claims that the proposed method generalizes across modalities.", "audience_yn": "Yes", "audience_explanation": "- The paper tackles the challenging and socially impactful problem of harmful meme moderation, with a focus on implicit/visual cues frequently missed by prior text-centric or unimodal systems. The proposed use of cognitive shift vectors (CSV), in-context multimodal exemplars, and explicit commonsense parameterization is a creative and timely direction for adaptive moderation, bridging gaps left by large VLMs.\n- The methodology is comprehensively described, with clear mathematical formalization (see Section 4.3), sound integration of loss components, and a relevant approach for semantic retrieval and parameter selection. The learning of CSVs over multiple layers introduces adaptive modulation, offering a smart alternative to simply stacking demonstrations.", "requested_changes": "- Positioning and Related Work Completeness: The related work section is thorough regarding immediate predecessors (MemeGuard, MemeMQA, VLM adaptation), but it omits recent, directly comparable systems from 2025 that advance explainable and resource-efficient meme moderation with large multimodal models:\n(1). Hee & Lee (2025). This paper describes IntMeme, offering interpretability methods that should be compared or referenced when discussing explainable intervention (belongs in Section 2 and the experiment section) and (2). Pan, Luu, Wu (2025). U-CoT+ addresses decoupled meme understanding, which is highly relevant for MemeSense's semantic retrieval and cognitive shift strategy (could be discussed as an alternative approach in Section 2 or 4, and ideally included in experimental baselines if resource permits).\n\n- Ablation and Robustness Gaps: While several ablations are provided, further explanation or exploration would be valuable: The effect of increasing the number of in-context exemplars (beyond the reported k values) on generalization and overfitting risk is not deeply analyzed. Sensitivity analysis is present, but it would help to dissect common failure modes with more qualitative error cases for culturally ambiguous memes, as revealed in the detailed analysis on Page 10.\n\nHee & Lee (2025): Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions. https://arxiv.org/abs/2502.11073\n\nPan, Luu, Wu (2025): Detecting Harmful Memes with Decoupled Understanding and Guided CoT Reasoning. https://arxiv.org/abs/2506.08477", "broader_impact_concerns": "None", "additional_comments": "No", "extra_scores": {}, "extra_text": {}}, {"review_id": "WXcphi22bC", "reviewer": "Reviewer_Qozu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The study introduces a framework designed to detect harmful memes, including both image-only and image-with-text memes. The method demonstrates promising performance on benchmarks curated by the team, which comprise existing benchmarks as well as additional data collected from the internet.\n\nThe framework is organized into three primary stages: generating commonsense parameters for an image, selecting in-context examples(similar visually/semantically via ANN search), and learning cognitive shift vectors for the dataset. The commonsense parameters are based on a predefined taxonomy of harms derived from previous research in the field. The technique for computing  cognitive shift vectors is also from past work in the field. But the authors stack these components together for harmful meme detection which is novel. ICV also allows for the distillation of task-relevant information, reducing the need for multiple in-context examples during inference, which in turn lowers inference costs and time.\n\nThe paper should specify that it claims SOTA in post-hoc techniques(inference based that don't require training a model) for meme detection and only for the specific model sizes studied, rather than asserting SOTA across all models/methods. Including experiments with models from the Memeguard paper would be beneficial, as the reported numbers appear to be inconsistent. The writing could also be improved to reduce back-and-forth between sections. Providing more information about the methodology and dataset, as outlined below, would make this more reproducible.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the submission are to a large extent accurate but there are some areas that could benefit from further clarification and additional data to strengthen the overall argument.\n\nFirstly, the study is well-structured, and the performance gains reported are impressive. The study uses well-studied evaluation metrics, such as BERTScore, BLEU, and ROUGE. The study compares against relevant baselines like MemeGuard and MemeMQA. Datasets used are also well-studied in academia and also include some sourced from the internet. While the individual components of the framework (e.g., ICV, semantic similarity-based search, LoRA fine-tuning) are not novel, their combined application for meme detection is.\n\nHowever, the study focuses on a limited number of models, specifically two models of similar sizes (8B and 9B), which restricts the generalizability of the claim of being a state-of-the-art (SOTA) framework for harmful meme detection. It would be beneficial to clarify that this claim is specific to models of this size and does not extend to larger models like GPT-4, which could potentially perform better in zero-shot or few-shot settings. It would also help to clarify that this might be SOTA only for techniques that don't require finetuning a model and just require post-hoc intervention. Alternatively, authors should consider comparing performance with finetuned model, on zero shot/few shot baselines.\n\nThe study also does not include any backbones mentioned in the Memeguard paper, and the reported numbers for Memeguard are significantly different. For instance, the Rouge-L score for Memeguard in this paper is reported as 0.005, whereas the original paper reports its worst performing memguard model(LLAMA) at a score of 5.29. Likewise, Bertscore for the worst performing model has a score of 80.11(LLAMA) but here it is reported as 0.82. Is there a difference in the way authors calculate the metrics or is the disparity because of the backbone chosen(idefics)? If it's the former, I'd encourage the authors to add a section on metrics and explain how they are calculated. If it’s the latter, I’d encourage the authors to add an experiment with the models in memeguard for consistency.\n\nOne of the main claims/contributions is the dataset. The dataset is listed as a contribution, but there is no mention of an intent to release it. This should be reworded.\n\nThe motivations for choosing different backbones for different components (fintuned Mc and model for in-context intervention) are unclear. It would be beneficial to explain why different models(llava, idefics) or backbones were chosen instead of using the same one.\n\nError analysis was a nice inclusion in the paper. It would also be helpful to understand the distribution of harmful and non-harmful images in the dataset of 985 memes, as well as the distribution of harmful/non-harmful between the test and train sets. As the error analysis only talks about false negatives but not false positives? Do they not exist at all?\n\nThere is also confusion about the dataset used to train the fine-tuned model and the data for the ICV vectors. Are they the same? What is the size and origin of Dc? It's not mentioned at all.", "audience_yn": "Yes", "audience_explanation": "Yes, the paper tackles the complex issue of moderating \"nuanced harmful memes,\" which often mask their intent through humor, or cultural symbolism and frequently evade conventional moderation systems, especially those relying on explicit text. It's interesting to see how far one can push a model without finetuning to perform well at this task by just performing test time intervention that performs better than direct zero shot prompting.", "requested_changes": "Please revise main contributions in section 1 if you don't plan to release the dataset.\n\nPlease make sure to add all method related details in one section instead of spreading them across multiple sections. Page 4, section 4 redirects reader to section 6. Page 6, section 5, redirects user to section 4.1. Making it hard for readers to follow the dataset prepration. Despite the back and forth,  the size and source of dataset used to finetune LLAVA isn’t mentioned in either of these sections. \n\nIt would be helpful to include a section explaining how metrics are defined and computed, as there seems to be a misunderstanding or misreporting of scores like BertScore, which is off by a factor of 100.\n\nHow is readability measured? This is not mentioned in the paper,\n\nSymbols mismatch, eqn 1 has alpha, page 7 last line says gamma.\n\nIn section 8, in effect the coefficient, alpha should be 0 to be non-trainable and not one? According to eqn 1, scaled vectors are added. So for model to be non trainable shouldn't be 0?\n\nIncluding an n-shot baseline and fine-tuned baseline(zero shot) for comparison could help in understanding the effectiveness of ICV.", "broader_impact_concerns": "No major concerns. However, since part of their dataset includes content sourced from the internet, particularly sensitive material like phallic memes, it would be helpful for the authors to clarify whether they obtained consent or if the data was available under a free license.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EAtIyioMHa", "reviewer": "Reviewer_bQ8p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### 1. Paper Summary \n\nThis paper proposes a meme content moderation framework driven by social common sense, MemeSense. This method combines visual and text understanding through a three-stage process (common sense parameter generation, in-context example retrieval, and cognitive shift vector learning) to improve the ability to identify and intervene in harmful memes with implicit or no text. The main contributions of the paper include: (1) designing an adaptive intervention generation process that combines social common sense prompts and dynamic in-context examples; (2) constructing a meme dataset covering 15 types of social common sense harm; (3) achieving superiority over state-of-the-art methods on benchmark datasets, with significant improvements in BERTScore.\n\n### 2. Strengths\n\n- MemeSense first introduced the cognitive shift vector into the meme moderation scenario, combined with social common sense, in-context learning, and multimodal modeling, overcoming the problems of traditional models' strong dependence on text and weak ability to identify hidden harm.\n- Comprehensive evaluation, covering multi-dimensional automatic metrics such as BERTScore, ROUGE-L, BLEU, and supplemented by human consistency analysis.\n- Significant results. On both the self-built dataset and the ICMM dataset, MemeSense's overall (such as BERTScore and SeSS) have been greatly improved, and the statistical significance of the improvement has been verified through significance tests.\n\n### 3. Weaknesses\n\n- There is a jump between evidence and conclusion. There is a lack of empirical or case analysis on the generalization ability of the model in actual complex scenarios (such as memes in multilingual and cultural environments). The article claims to have \"stronger adaptability\" but has only been verified on a limited number of English social media memes.\n- Ablation and error analysis are not in-depth enough. Although the author has ablated the contribution of cognitive shift vector and in-context exemplar, there is a lack of qualitative analysis of the model performance in their respective failure scenarios, especially for misclassification cases; there is insufficient reasoning process tracking and failure cause location.\n- The granularity of manual evaluation is coarse. The sample size of manual evaluation in the paper is limited (n=25), and only involves the rank correlation between BERTScore/ROUGE and subjective evaluation. No qualitative analysis of typical failure cases is conducted, and there is also a lack of practical application feedback at the user level (moderator/workflow).\n- Dataset annotation and statistical bias risk. The self-built dataset is double-annotated by a small number of undergraduates + experts. Although Cohen's kappa is high, it does not fully explain the consistency and diversity of extreme/ambiguous meme types. The definition of the \"potentially harmful but no text\" category is highly subjective, and potential bias is not evaluated.\n- Model interpretability is limited. Although there is a vector space clustering analysis of the cognitive shift vector, the interpretability of a specific inference process is still insufficient. There is a lack of transparent decision logs or link examples for actual moderator tracing or intervention recommendation reasons.", "claims_and_evidence_yn": "No", "claims_explanation": "Some claims—for example, generalization to multiple languages or cultural diversity, and applicability to real-world workflows—are not fully substantiated. The paper also provides a relatively limited analysis of model failures and annotation biases. Thus, the evidence for the paper’s technical contributions is compelling, but the claims about generalization and practical deployability are less robust.", "audience_yn": "Yes", "audience_explanation": "This paper explores an important challenge at the intersection of multimodal learning, content moderation, and social commonsense reasoning — topics that are highly relevant to the TMLR audience. The paper proposes the MemeSense framework, introduces new technical methods, and releases a new dataset that will be of interest to researchers working on visual language models, harmful content detection, and AI ethics.", "requested_changes": "1. It is recommended to introduce cross-cultural and cross-language memes (such as non-English corpus, different religious/regional backgrounds) for supplementary experiments to verify the practical applicability and robustness of the method.\n2. The reasoning chain analysis of the failure case should be supplemented, and specific examples of memes when the cognitive shift vector/in-context example fails alone should be given to help understand the model boundary.\n3. It is recommended to expand the sample size of human evaluation and invite actual moderators to conduct case evaluation to supplement the model usability and user perception indicators.\n4. Supplement the analysis of data annotation consistency and bias. For the highly subjective textless meme harm category, it is recommended to supplement expert diversity annotation, boundary discussion, and discussion of potential cultural bias risks.\n5. It is recommended to add visualization or decision tracing tools for the model reasoning chain to facilitate tracking and auditing the specific reasons for AI moderation in actual application scenarios.", "broader_impact_concerns": "While this method improves the meme moderation, it also brings risks such as \"over-censorship\" and cultural discrimination/accidental injury, especially in the vaguely defined areas of social common sense (such as humor and cultural symbols), which are prone to controversy. If certain mainstream culture/values are solidified during model training or reasoning, it may cause improper suppression of the freedom of expression of minority groups. Therefore, it is recommended that the author further analyze the potential bias risks of the model in multicultural scenarios and clearly disclose the uncontrollable scenarios and consequences of misjudgment of the model.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)