Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models
Authors: Hongzhan Lin
Ziyang Luo
Jing Ma
Long Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Harmful meme detection
multimodal reasoning
knowledge distillation
large language models

Abstract: The age of social media is rife with memes. Understanding and detecting harmful memes pose a significant challenge due to their implicit meaning that is not explicitly conveyed through the surface text and image. However, existing harmful meme detection approaches only recognize superficial harm-indicative signals in an end-to-end classification manner but ignore in-depth cognition of the meme text and image. In this paper, we attempt to detect harmful memes based on advanced reasoning over the interplay of multimodal information in memes. Inspired by the success of Large Language Models (LLMs) on complex reasoning, we first conduct abductive reasoning with LLMs. Then we propose a novel generative framework to learn reasonable thoughts from LLMs for better multimodal fusion and lightweight fine-tuning, which consists of two training stages: 1) Distill multimodal reasoning knowledge from LLMs; and 2) Fine-tune the generative framework to infer harmfulness. Extensive experiments conducted on three meme datasets demonstrate that our proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task.

Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models
Hongzhan Lin', Ziyang Luo', Jing Long Chen
'Hong Kong Baptist University
"The Hong Kong University of Science and Technology {cshzlin, cszyluo, majoong majjin]ecomp hkbu. edu. hk, longchen@ust.
Abstract
circulated meme falsely claimed that the mRNA vaccine would alter human genetic code (DNA)'. Such multimodal disinformation spread caused vac- cine safety and effectiveness concerns, hindering the formation of strong immune defenses in im- pacted areas globally (Basch et al., 2021; Lin et al., 2022). Besides, another meme example shown in Figure 1(b) perpetuates harmful stereotypes and generalizations about Asians. Therefore, it is nec- essary develop automatic approaches facilitate harmful meme detection for unveiling the dark side of memes.
The age of social media is with memes. Understanding and detecting harmful memes pose a significant challenge due to their implicit meaning that is not explicitly conveyed through the surface text and image. However, existing harmful meme detection approaches only rec- ognize superficial Harm-indicative signals an end-to-end classification manner but ignore in- depth cognition of the meme text and image. In this paper, attempt to detect harmful memes based on advanced reasoning over the inter- play multimodal information memes. In- spired by the success of Large Language Mod- els (LLMM)) on complex reasoning, we first con- duct abductive reasoning with LLMs. Then we propose novel generative framework to learn reasonable thoughts from LLMs for better multimodal fusion and lightweight fine-tuning, which consists of two training stages: 1) Distill multimodal reasoning knowledge from L.LMs; and 2) Fine-tune the generative framework to infer harmfulness. Extensive experiments con- ducted on three meme datasets demonstrate that proposed approach achieves superior per- formance than state-of-te sate--f-thheart tt-the-art art methods on the harmful meme detection task.
Harmful memes? are generally defined "mul- timodal units consisting of image and accom- panying text that has the potential cause harm to an individual, an organization, a community, or the whole society" (Sharma et al., 2022). Previ- ous studies (Kiela et al., 2020; Pramanick et al., 2021a.b) attempted to straightforwardly utilize pre- trained vision-language models (Li et al., 2019; Lu et al., 2019) for harmful meme detection by training additional task-specific classification lay- ers. More recently, Cao al. (2022) proposed prompt-tuning method with the meme text and im- age caption as the prompt for masked language modeling (Devlin et al., 2019; Liu et al., 2019).
1 Introduction
However, existing harmful meme detection ap- end-to-
The development social media platforms has given rise to a new form of multimodal content known as: meme. meme typically comprises a picture that is combined with concise text com- ponent. Memes possess the capacity to quickly spread across the internet, especially social me- dia platforms, due their ease of dissemination. While memes are often seen as humorous, there is potential for harm when the combination of images and texts is strategically used to promote political and sociocultural divisions. For instance as in Fig- ure ((a), during the COVID-19 pandemic, a widely
proaches oversimplified the problem end classification paradigm, which only recognizes the superficial signals conveyed through the sur- face text and image. But more in-depth investi- gation and cognition on the implicit meaning is required especially when the image and text are not obviously correlated (Pramanick et al., 2021b). Intuitively, the key to harmful meme detection is to excavate rich correlations beneath the surface
'tttps: //wp:::/wwww... bbc. come.cco/mooooo news/5510115
DDisclaimer: This paper contains discriminatory content that may be disturbing to some readers, where meme examples and words are offensive or hateful in nature. These contents are provided for illustrative purposes only and do not repre- sent the views and standpoints of the authors.
Jing Ma is the corresponding author. The first two authors sontributed equally to this work.

---

Chance virus with a 99.97% recovery rate
when you Tataan aoaaaaa anyy
Murricart En benu or koy enoush becure ehi
Alter my DNA from an experimental NO liabilty from corrupt industry
inrimna pel his family to accept you.
(a) Harmful
(b) Harmful
(C) Harmless
Figure I: Examples of harmful and harmless memes. Meme text: (a) Chance a virus with 99.97% recovery rate; Alter my DNA from an experimental vaccine, with NO liability, from a corrupt industry. (b) when you date an asian boy and you trynna get his family to accept you. (C) you either die a hero, or live long enough become the villain.
of the seemly uncorrelated text and image in the meme: 1) For example as in Figure 1(b), the image and the text not harmful when considered in isolation, but are harmful when taken as a whole. A human checker should cognize that, the "biting" action in the image of young woman with her pet dog ridicules Asians' "'dg-eettng" behavior, which corresponds to the "asian" word in the text. 2) In contrast, some harmful signals (e.g., "die" or "villain") ") observed in the text of Figure I(c), but the meme itself actually does not promote hate or discrimination against particular group of people. Because the text is a quote from a popular movie and often used as philosophical statement about the choices people make in life. And the image fur- ther adds a celebratory and joyful tone to the overall message. In comparison, conventional detection methods just focused on recognizing shallow harm- indicative signals without such multimodal reason- ing and essential background knowledge considera- tion, so the social dynamics of different races or the origin of the meme text from classical movie lines may not be well-cognized. Unlike such recognition- level detection models, we argue that establishing reasonable thought between textual and visual in- formation can further improve meme understand- ing with background knowledge for better harmful meme detection.
and lightweight fine-tuning, More specifically, we incorporate the meme text and image into two- stage training paradigm: 1) Reasoning Distillation: In the first stage, we fine-tune our smaller language models with the interaction language and vision features to distill multimodal reasoning knowledge from LLMs, which empowers our framework with the ability to conduct cognitive reasoning for the harmfulness prediction. 2) Harmfulness Inference: In the second stage, exploit the fine-tuned small language models to infer the final harmfulness pre- diction. In this manner, we augment the harmful meme detection model with multimodal reasoning knowledge to unmask the implicit meaning hidden in holistic multimodal information from memes.
We evaluate our proposed approach based on three public meme datasets. The results not only show that our method outperforms strong harmful meme detection baselines by large margin, but also provide fine-grained analysis for interpreting how our approach works. Our contributions are summarized as follows in three folds:
To our best knowledge, leviate the issue of superficial understanding for harmful meme detection by explicitly uti- lizing commonsense knowledge, from a fresh perspective on harnessing advanced LLMs.²
the first to al-
Inspired by the success of LLMs for reason- ing at the cognition level with contextual back- ground knowledge (Wei et al., 2022; Kojima et al., 2022: Zhang et al., 2022), we propose a novel ap- proach: MR.HARM, by leveraging the Multimodal reasoning knowledge distilled from LLMs for Harmful meme detection. To this end, we first prompt LLMs for abductive reasoning, and then propose two-stage generative framework based on smaller language models to learn reasonable thoughts from LLMs for better multimodal fusion
We propose novel generative framework fine-tune smaller language models augmented with the multimodal reasoning knowledge dis- tilled from LLMs, which facilitates better mul- timodal fusion and lightweight fine-tuning for harmfulness prediction.
Extensive ablations on three meme datasets confirm that our method could yield superior
*Our code is available at HKBUNLP/Mr www.m-Haammmpea2-- ENNLP2023
com/

---

performance than taat-of-thhearr baselines for the harmful meme detection task.
relations between the visual and textual elements.
2.2 Large Language Models
2 Related Work
Recently, LLMs have demonstrated remarkable ca- pability in complex reasoning (Brown et al., 2020; Thoppilan et al., 2022; Rae et al., 2021; Chowdh- ery et al., 2022), such as generating intermediate inference procedures before the final output (Nye ct al., 2021; Wei et al., 2022; Kojima al., 2022; Zhang et al., 2022). Unfortunately, the large size of LLMs restricts their deployment on detecting harm- ful memes with different modalities, regardless of how they are enhanced with strategetic text prompt- ing. Knowledge distillation has been successfully used to transfer knowledge from larger. more com- petent teacher models into smaller student models affordable for practical applications (Buciluă et al.. 2006; Hinton et al., 2015; Beyer et al., 2022). How- ever, existing researches on knowledge distillation from LLMs (Wang et al., 2022; Ho et al., 2022; Magister et al., 2022) only consider the language modality, they are not suitable for harmful meme detection because harmful memes can convey holis- tic synergy information through multimodal fea- tures. In this work, we conduct abductive reasoning with LLMs, which further advocates multimodal reasoning paradigm to fine-tune smaller language models (LMs) for harmful meme detection.
2.1 Harmful Meme Detection
Harmful meme detection is rapidly growing area in the research community, driven by the recent availability of large meme benchmarks (Kiela et al., 2019; Suryawanshi et al., 2020; Pramanick et al., 2021a). The Hateful Memes Challenge organized by Facebook (Kiela et al., 2020) further encour- aged researchers to develop solutions for detecting harmful memes in hate speech (Das et al., 2020). More recently, Pramanick et al. (2021a) firstly de- fined the harmful meme concept and demonstrated its dependence on contextual factors. The com- plex nature of memes, which often rely on multiple modalities, makes them challenging and struggle to yield good performance only using unimodal detec- tion methods (Simonyan and Zisserman, 2014; He et al., 2016; Devlin et al., 2019). Therefore, recent studies in this area attempted to apply multimodal approaches the harmful meme detection task. Previous studies have employed classical two- stream models that integrate text and vision fea- tures, which are learned from text and image en- coders, using attention based mechanisms and mul- timodal fusion techniques for classifying harmful memes (Kiela et al., 2019, 2020; Suryawanshi et al., 2020). Another branch was to fine-tune pre-trained multimodal models specifically for the task (Lippe al., 2020; Muennighoff, 2020; Velioglu and Rose, 2020; Hee et al., 2022). Recent related efforts have also sought to explore the use of data augmentation techniques (Zhou et al., 2021; Zhu et al., 2022), ensemble methods (Zhu, 2020; Velioglu and Rose, 2020; Sandulescu, 2020) and harmful target dis- entanglement (Lee et al., 2021). More recently, Pramanick et al. (2021b) proposed a multimodal framework by using global and local perspectives to detect harmful memes which achieves ofate-of- the-art performances. follow-up prompt-based based approach (Cao et al., 2022) attempted to concate- nate the meme text and extracted image captions to fine-tune masked language models (Liu et al., 2019) for harmful meme detection. However, ex- isting solutions only capture the superficial signals of different modalities in memes in an end-to-end manner, which largely ignore explicit deductive rea- soning to guide the model for understanding back- ground knowledge about the complex and diverse
3 Our Approach
Problem Statement We define a harmful meme detection dataset meme M fy. II is triplet represent- ing an image I that is associated with a text T, and a ground-truth harmfulness label E [harmful, harmless]. In this work, to investigate multimodal reasoning distilled from LLMs, we con- vert the harmful meme detection task into natural language generation paradigm, where our model takes the text T and image I as the input and gen- erates a text sequence that contains the label y clearly express whether the meme harmful.
of memes where each
Our core idea is to reason and evolve with the cognition-level rationale beyond the recognition- level perception (Davis and Marcus, 2015) by cap- turing the inter-relationship between visual and textual elements in memes. The overview of our framework is shown in Figure 2, which consists of abductive reasoning with LLMs (see Sec. 3) and two training stages, i.e., reasoning distillation (see Sec. 3.2) and harmfulness inference (see Sec. 3.3).

---

Abductive Reasoning with LLMs
cilem knt: my black boy friend, which is imbodieed woman holds a baby gorillä, plasse poride rationale for how the mame ieessoned is laansfuleess labet. harmyiu
Rationale:
While imager woman holding baby gurilla could be intepprted computionn beteen black boyfriend and in seimal, minifoncing hammful sioneelypos about cara: The posertial for the evenall moseggi of the mome sprcad hurmffl or offensive content shout mee and relationship.
objuenfying or reducing person
LLM
Distillation
The meme is harmful
my bitck hoyeniend
LM Encoder
LM Encoder
Thes
Fusion
LM Decoder
Fusion
LM Decoder
Vision Extractor
Vision Extractor
The First Fine tuning Stage
The Second Fine- Tuning Stage
Figure 2: The overall pipeline of our method. We first conduct abductive reasoning with LLMs to extract harmfulness rationales (pink) by the prompt consisting of the meme text (green), the image caption (blue), and the label (orange). We then use the generated rationales train small task-specific models with multimodal inputs the first fine-tuning stage and feed the same inputs to the updated model for harmfulness inference as the second fine-tuning stage.
3.1 Abductive Reasoning with LLMs
As we clarify the ground-truth truth harmfulness label in the observed attributes of the prompt, the hallu- cination issue (Bang et al., 2023) of LLMs could be effectively alleviated. Because the rich contex- tual background knowledge could be activated by abductive reasoning based on the ground truth and invalid rationales are naturally filtered out.
In this paper, we propose to utilize abductive rea- soning with multimodal inputs to train smaller downstream models. LLMs can produce natural language rationales unveiling the implicit meaning beneath the surface of the memes to justify the rea- son why the meme harmful or not. This shares a similar intuition as heuristic teaching (Pintrich and Schunk, 2002) where a teacher who has rich experience and knowledge can impart to students the correct way of thinking and reasoning based on questions with corresponding answers. The stu- dents then learn how to deduce their own ways to the correct answers from questions accordingly. Thus we aim activate explicit reasoning knowl- edge in LLMs as a teacher model, e-g.. contextual and cultural information related to memes, to guide our model to strengthen harmfulness prediction.
3.2 Reasoning Distillation with Small LMs
Since we utilize image captions to represent the meme images, we could perform abductive reason- ing with large language models pre-trained with language modality. However, only using the cap- tions as opposed to original vision features may suffer from lack of mutual synergy in the repre- sentation space of different modalities in memes due to the inductive bias of possible information loss in the captioning process. On the other hand, LLMs can be used to conduct abductive reasoning only for the training data whose harmfulness label is given in prior but is challenging to be fine-tuned for this task due to the huge amount of model pa- rameters. To facilitate the interactions between the meme text and the image, propose to fine-tune smaller language model for the harmful meme detection task. which allows flexibility in adjust- ing model architectures to incorporate multimodal features and is more lightweight for task-specific fine-tuning.
Given a meme sample M- = I, T} from the training data, to prompt large language models in uniform language modality, we first extract the text caption I of the image I by off-the-shelf caption- ing models (Mokady et al., 2021). Then we curate a template p that consists of triplet (y, I,, IT served attributes, to prompt the LLMs to generate a rationale r that clicits the reasoning knowledge about how to infer the harmfulness label y based on the interplay of the meme text T and the image caption ī as illustrated in Figure 2. Specifically, we design p as:
T as ob-
In this section, we train small language model as student model distilled from the LLMs with multimodal reasoning knowledge. Specifically, we leverage generated rationales from LLMs as in- formative supervision, to fine-tune smaller pre- trained language model to excavate the rich inter- relationship between language and vision modali-
"Given a Text: [T], which embedded in an Image: [ī]; and harmfulness label [u], please give me streamlined rationale associated with the meme, without explicitly indicating the label, for how is reasoned as [yl."

---

ties of memes.
LLMs and injected into the smaller language model specific to harmful meme detection.
Encoding For a meme sample M from the train- ing data, we first encode the text T and the image I to obtain their embedding vectors as follows:
3.3 Harmfulness Inference
During the first fine-tuning stage, we conducted explicit deductive reasoning to empower our model with the capability of multimodal reasoning dis- tilled from LLMs. As the goal of this task is to determine whether the meme is harmful or not, we conduct the second fine-tuning stage for Harm- fulness Inference, which shares the same model architecture, parameters, and encoding procedure as Sec. 3.2 but differs in the decoding output. To make the output consistent with harmfulness pre- diction, the smaller model f is further trained by minimizing the following inference loss:
H TE(T), Hi VE(I).
(1)
where TE(-) denotes the text embedding layer of the LM Encoder. And H1 E Rmmxdd is the token embeddings in the Transformer encoder (Vaswani et al., 2017) where is the text token length and d is the dimension of the hidden states. VE(-) is the Vision Extractor implemented as frozen pre-trained vision Transformers (Radford ct al., 2021) to fetch the patch-level features of the image with n patches, which is projected into the visual representations Hz C Raxd Next, to support semantic alignment between the text and the image for better context understanding, we exploit cross-attention mech- anism (Luo al., 2022) for multimodal fusion of the textual and visual information:
Lin CE (J(I,T),
(5)
where the cross-entropy loss computed between the generated text and ground-truth harmfulness label y. With the generative objective (Raffel et al., 2020) adapted to the previous Reasoning Distilla- tion stage, the prior reasoning knowledge absorbed in Reasoning Distillation could be well induced for Harmfulness Inference.
QT = W H7 + biq
KI Wk + bk:
V/ Wi + liy
(2)
Model Training The model training consists of two fine-tuning stages: 1) Reasoning Distillation and 2) Harmfulness Inference, where Reasoning Distillation is the predecessor fine-tuning phase of Harmfulness Inference. Note that for model testing, directly input the test sample into our fine-tuned language model to predict the meme harmfulness.
@
H1 softmax
Vz,
ddz
where H4 is the input hidden states of each LM Encoder layer and H2 is the attended visual fea- tures. Then we can fuse Hi with Hi- to attain the interplay representations for a meme:
Hith = LMME (H+)+ + WoHi + tb.
(3)
4
Experiments
where LME'()) (-) is the i-th layer of the LM Encoder, W denotes the linear projection, bi is the bias, and A H4 is the final interplay representations after going through an L-layer LM Encoder fused with the visual features.
4.1 Experimental Setup
Datasets We use three publicly available meme datasets for evaluation: (1) Harm-C (Pramanick et al., 202 la), (2) Harm-P (Pramanick et al., 2021b), and (3) FHM (Kiela et al., 2020). Harm-C and Harm-P consist of memes related COVID-19 and US politics, respectively. FHM was released by Facebook as part of challenge to crowd-source multimodal harmful meme detection in hate speech solutions. Different from FHM that each meme was labeled as harmful or harmless, Harm-C and Harm-P were originally labeled with three classes: very harmful, partially harmful, and harmless. For fair comparison, we merge the very harmful and partially harmful memes into harmful ones, follow- ing the evaluation setting of recent work (Praman- ick et al., 202 bb; Cao et al., 2022). We provide
Decoding Finally, we feed the interplay represen- tations H E Immd into the LM Decoder, imple- mented as a Transformer-base based decoder, to gener- ate the reasonable rationale. Overall, the smaller language model is trained by minimizing the following distillation loss:
Laistill CE (J(I,T),))
(4)
where CE(-) denotes the cross-entropy loss (Sutskever et al., 2014) between the predicted text and the target rationale r generated by L.LMs. this way. multimodal reasoning knowledge about the meme could be explicitly distilled from

---

Dataset
Harm-C
Harm-P
FHM
Model
Accuracy
Macro-Fi
Accuracy
Macro-Fy
Accuracy
Macro-F;
Text BERT (Devlin al., 2019)
70.17
66.25
80.12
78.35
57.12
41.52
Image-Region (He et al., 2016)
68.74
62.97
73.14
72.77
52.34
34.19
Late Fusion (Pramanick et al., 202
73.24
70.25
78.26
78.50
59.14
44.81
MMBT (Kiela et al., 2019)
73.48
67.12
82.54
80.23
65.06
61.93
VisualBERT COCO (Li al., 2019)
81.36
80.13
86.80
86.07
61.48
47.26
ViLBERT CC (Lu et al., 2019)
78.70
78.09
87.25
86.03
64.70
55.78
MOMENTA (Pramanick al., 2021h)
83.82
82.80
89.84
88.26
61.34
57.45
MaskPrompt (Cao et al., 2022)
84.47
81,51
88.17
87.09
72.98
65.24
MR.MARM
86.16
85.43
89.58
89.57
75.40
75.10
Table I: Harmful meme detection results three datasets. The accuracy and macro-averaged F1 reported as the metrics. The best and second results are in bold and underlined.
(%) are
statistics of the three datasets in the Appendix.
extracted entities and demographic information of the image into the masked language models, be- sides just captioning the image into the prompt.
Baselines We compare MR.HARN with sev- eral state-of-the- harmful meme detection sys-
Our proposed MR. HARM improves over the best baselines by 2.63%, 1.31 %, and 9.86% terms of Macro-Fl score on Harm-C, Harm-P, and FHM datasets, respectively. We observe that 1) the improvement on the Harm-P dataset is relatively milder than that on the other two datasets. Mean- while, all the baselines just have tiny differences among their performances Harm-P. We specu- late the reason falls into the smaller dataset scale of Harm-P which only contains politics-related harm- ful memes. 2) A similar trend can also be observed in Harm-C and FHM datasets: the more challeng- ing the dataset is, the greater performance improve- ment HARM achieves. Our model performs flexibly and stably across all datasets with its keen judgment on harmful memes. This is because all the baselines are only designed the recognition level, but MR. HARM is further empowered with multimodal reasoning knowledge distilled from LLMs to uncarth harmful content from the scemly uncorrelated text and image modalities of memes.
tems: 1) Text BERT (Devlin et al.. 2019); 2)
Image-Reginn (Ren et al., 2016; Hc et al., 2016);
3) Late Fusion (Pramanick et al., 2021a); 4)
MMBT (Kiela et al., 2019); 5) VisualBERT COCO (Li et al., 2019; Lin et al., 2014); 6) Vil.-
BERT CC (Lu et al., 2019; Sharma et al., 2018); al., 2021b); 8) MaskPrompt (Cao et al., 2022). We use the accu- racy and macro-averaged Fl score as the evaluation metrics. More implementation details and baseline descriptions are provided in Appendix.
7) MOMENTA (Pramanick
4.2 Harmful Meme Detection Performance
Table shows the performance of our proposed method versus all the compared methods on the Harm-C, Harm-P and FHM datasets. It is observed that 1) The performance of the baselines in the first group is obviously poor due to only unimodal fea- tures like text-only or image-only being captured. In comparison, the other baselines exploit the mul- timodal features from both the text and image in memes. 2) The multimodal models in the second group outperform the unimodal ones. The early- fusion models with multimodal pre-training (i.e., VisualBERT COCO and ViLBERT CC) outper- form that of the simple fusion with unimodal pre- training (i.e., Late Fusion and MMBT) on Harm- C/P datasets, while MOMENTA performs best in the second group by considering global and local information of memes. 3) However, as the images in FHM dataset are more informative and high- quality, MaskPrompt yields the best performance among all the baselines by incorporating additional
4.3 Ablative Study
We perform ablative studies on several variants of MR. HARM: 1) w/o Reasoning Distillation: Sim- ply fine-tune the smaller language models in the stage of Harmfulness Inference without the stage of Reasoning Distillation based on LLMs; 2) w/o Vi- sual Features: Discard the features from the meme image while keeping those from the meme text; 3) w/o Multimodal Fusion: Instead of the fusion mechanism on the multimodal features in our lan- guage model, we only append the lingual features from image captioning together with the meme text

---

Dataset
||
Harm-C
Harm-P
FHM
Model
Accuracy
Macro-Fi
Accuracy
Macro-Fi
Accuracy
Macro-Fi
MR. HARM
86.16
85.43
89.58
89.57
75.40
75.10
w/o Reasoning Distillation
83.33
81.44
88.17
88.17
73.60
73.41
w/o Visual Features
82,48
80.30
87.04
87.03
58,80
57.01
w/o Multimodal Fusion
79.38
75.36
87.46
87.45
74.40
74.25
w/o Two-stage Training
83.05
81.45
63.32
63.32
67.40
65.77
w/o Fine-tuning Small LMs
71.75
66.86
61.13
60.27
60.00
57.72
Table 2: Ablation studies on our proposed framework.
during encoding; 4) w/o Two-stage Training: Con- catenate the rationales generated from LLMs and golden harmfulness label as the target for model training, to replace the two-stage training paradigm; 5) w/o Fine-tuning Small LMs: Directly prompt the representative large language model ChatGPT based on InstructGPT (Ouyang et al., 2022) for harmful meme detection.
DON'T WORRY, AMERICA
Intermediate Reasoning:
the meme contains a political
message that promotes a political
agenda. the text suggests that the
corona-itus nothing compared lo
windmill cancer. which barmful
and offensive message. the image of
a politician in a suil and lic. possibly
related the imauguration of
THE CORONAVIRUS IS NOTHING COMPARED WINDMILL CANCER
oraideet trump, further reinforees
of political message in this context
the harmful message.
Meme DON'T WORRY.
AMERICA, CURUNAVIRUS
could be seen as insensitive and disrespectfill to those have affected by the pandemic.
NOTHHNNG COMPARED TO
As demonstrated in Table 2, the ablative models suffer different degrees of performance degrada- tion, indicating the effectiveness of our proposed components for harmful meme detection with mul- timodal reasoning distilled from LLMs. Specifi- cally, the performance of MR. HARM significantly decreases in the "w/o Reasoning Distillation' set- ting due to the lack of multimodal reasoning knowl- edge transferred from LLMs about the seemly un- correlated modalities in memes. The 'w/o Visual Features' setting also achieves worse performance than MR. HARM, suggesting that the visual repre- sentations are complementary to the meme text for harm- corm-indicative pattern extraction in the language model. HARM makes improvements over 'w/o Multimodal Fusion', which implies the promot- ing role of our fusion mechanism that incorporates original vision features into the language model, hardly compromised when there could be severe information loss in the captioning process. More- over, the "w/o Two-stage Training' setting leads to large-margin performance degradation, which verifies the effectiveness of our two-stage training paradigm. This is because this setting causes mu- tual interference between intermediate reasoning and final prediction, which affects the convergence effect of harmfulness inference and damages the model's performance and stability. Compared with MR.HARM, the performance of "w/o Fine-tuning Small LMs' also significantly decreases, highlight- ing the importance of abductive reasoning with LLMs to alleviate the hallucination issue during
WINDSILL CANCER!
THE-DEMOCRAT PARTY
Intermediate Reasoning:
based on the text and image i
appears the meme is making am
inllammatory statement about the
democrat party. the image shows
close up of confedernte flag with
stars, which could be interpreted
AMERICAS OLDEST HATE GROUP
on implication the democrat party-
also mentions oldest hate
group in the united states, which
could be interpreted as reference to
DEMOCRAT PARTY
democrat party being asscciatd
AMERICAS KULDEST HATE GRUUP
with racism and white supremacy.
Intermediate Reasoning:
the text in the meme contains
demgutory offensive statement
Tooat himpal! towards particular
group
Muslim individunls
associated with un image of Muslim
man praying
mosque which is highly offensive and
discriminatory The phrase
bunch of pigs just walked over your grave" be interpreted
it's the goat humpor,
looks like a bunch of pigs just
disrespocttful reforence to the Musline practice of abstaining from pork.
walked your grave
(c)
Figure 3: Examples of correctly predicted harmful
memes in (a) Harm-C, (b) Harm-P, and (c) FHM dataset.
deductive reasoning for harmfulness prediction.
4.4 Cognition-view Reasoning Analysis
Note that our smaller language model is explicitly trained in the Reasoning Distillation stage for ra- tionale generation to distill multimodal reasoning knowledge from LLMs. Although intermediate

---

reasoning is not the final target output for harmful meme detection, after the first fine-tuning stage, we elicit reasonable thoughts from our smaller lan- guage model with the test samples as input, to understand the cognition view of our proposed on the test meme samples more trans- parently and intuitively, as exemplified in Figure 3.
Sund nu neat claes iroiocl Intermediate Reasoning:
based the image and
presented the meme, appears to be hannless and relatable scenario.
the image shows group of women
traditional dress walking on common and barnless activity. the "and
beach, which
mny next class project," R seems to be
playful and lighthearted commeot
about the topic of the class project,
From the visualized intermediate reasoning, we observe that 1) our model could understand the multimodal information related to the meme text (in green) and image (in blue) with commonsense knowledge. For example, in Figure 3(a), the rec- ognized "politician" in the image could be related "president trump", which could be linked to the "AMERICA" in the text; in Figure 3(b), the recognized "flag" in the image could be cognized to satire "the democrat party" in the text; and in terms of Figure 3(c), the "goat humper" and "pigs" in the text could be associated with the attacks to "a Muslim man" recognized in the image. 2) Furthermore, our model learns to cognize the in- terplay (in pink) of multimodal information with advanced reasoning. Benefitting from the rich mul- timodal understanding of the memes, the perpet- uates harmful stereotypes could be reasoned over the context to the target like "who affected by the pandemic" in Figure 3(a), "the democrat party" in Figure 3(b), and "the Muslim" in Figure 3(c). In this way, the rich correlation beneath the surface of the meme text and image could be excavated to fa- cilitate harmfulness inference with better reasoning knowledge by harnessing advanced LLMs. Such readable pieces of rationales are also potentially valuable for aiding human checkers to verify the final answer predicted by our model.
there is no indication of any harmful
Meme text: and for my next class project!
offensive content in the image od
text.
In 1979 jimmy carter created the department on leducation
Intermediate Reasoning:
the text in the meme contains
derogstory and offensive statement
sbout politician, jimmy carter, who
created the department of education
in 1979. use of such langinge
and imagery can be considered
SInce then the U.S. hAS gone Irom 1st to 17th in education
hannful and offensive to individualy
who identify with the politician
edditionally, the image of an older
Meme text in 1979 jimmy carter
man with a linay expression on lis
created the department of education
face be seen promoting
since then the u.s. has gone from
Ist letnegatvve and hannful attitude
17th education
lowards the politician.
(b)
Figure 4: Examples of wrongly predicted memes by our proposed framework with the ground truth (a) harmful and (b) harmless.
or normalizing the behavior depicted in the im- age. The image features "a group of Ku Klux Klan members walking on a beach", which is symbol of white supremacy and racism. The combination of the phrase in the text and the use of imagery associated with hate groups can contribute the glorification of harmful behaviors and the perpet- uation of negative stereotypes, which makes the meme harmful. However, due to the lack of related background knowledge about the Ku Klux Klan members and their wear, our framework cannot well recognize the image correctly during the origi- nal vision feature extraction, which leads to error propagation for wrongly concluding that the meme is harmless. Also, in terms of the harmless meme in Figure 4(b), the image that "Jimmy Carter with a smile on his face" is mistakenly recognized as "an older man with a funny expression on his face", furthermore, the model hallucinates that the meme text "can be considered harmful and offensive to individuals who identify with the politician", result- ing in the wrong prediction that the meme is harm- ful. Therefore, is possible improve MR.HARM by incorporating more informative vision features and improving language-vision interaction to be capable of understanding the images with more complex background knowledge.
4.5
Error Analysis
To better understand the behavior of our model and facilitate future studies, we conduct an error analysis on the wrongly predicted memes by our proposed framework. We found that the major er- ror exists in that our framework still cannot fully recognize the images that require rich background knowledge though we exploited the advanced cross- attention mechanism by incorporating visual fea- tures into the language model. Figure 4 shows two examples of memes wrongly classified by MR.HARK For the harmful meme in Figure 4(a), the phrase "and for my next class project!" suggests that the image is being used for an academic or ed- ucational purpose, which can be seen as glorifying

---

Version
Harm-C
Harm-P
FHM
Limitations
Small
84.99
85.33
72.96
There are multiple ways to further improve this work:
Base
85.43
89.57
75.10
Large
84.02
90.14
77.80
Despite this work focusing on performance improvement of harmful meme detection, it is harder judge the quality of the intermediate reasoning, where the evaluation necessarily qualitative. Considering that our framework could generate readable snippets for cognition- view reasoning, we plan to do some sort of systematic study to claim explainability for the evaluation, which would be another more targeted research.
Table 3: averraed score (%) achieved with different versions of our fine-tuned L.s..
4.6 Discussion
As our two-stage training paradigm requires dis- tilling the reasoning knowledge and leveraging original vision features, we utilize the T5 encoder- decoder architecture (Raffel et al., 2020; Chung al., 2022) to initialize generative framework. To test the generality of the benefits of our approach different versions of the backbone, we alter the underlying LMs to other variants different sizes. As shown in Table 3, one interesting phenomenon is that our model has already achieved outstand- ing performance on the three benchmarks with the Small (about 60M parameters) Base ((about 220M parameters)) version the backbone, which has smaller size than the state-of-the-art baseline MaskPrompt (over 300M parameters). The Large version of our backbone generally achieved better performance than the other two backbone versions because the larger the fine-tuned LMs, the more alleviates the hallucination issue (Ji et al., 2023). Overall, the results show that our framework does not rely excessively on the size of the backbone improve performance and is generally effective with different versions of the backbone model.
New benchmarks to evaluate the reasoning ability of our framework are demanded. We going to further exploit LLMs toward ex- plainable harmful meme detection from the perspectives like dataset construction and au- tomatic evaluation.
We only use the textual prompt to conduct ab- ductive reasoning with accessible LLMs pre- trained with the language modality. We would further update our framework by leveraging visual LLMs if accessible the future to im- prove the visual feature extraction for explor- ing better multimodal reasoming knowledge distillation, and avoid several common defi- ciencies of existing language models includ- ing hallucination and limited generalization much possible.
A Acknowledgements
5 Conclusion and Future Work
This work was partially supported by Hong Kong RGC ECS (Ref. 22200722) and National Natu- ral Science Foundation of China Young Scientists Fund(No. 62206233).
In this paper, we propose to capture implicit mean- ing that is not explicitly conveyed through the sur- face of the text and image in memes for harmful meme detection. We first conduct abductive rea- soning with LLMs.. Then we present a novel gen- erative framework to distill multimodal reasoning knowledge from L.Mss, which includes two train- ing stages: 1) reasoning distillation and 2) harmful- ness inference. Results on three meme benchmarks confirm the advantages of our proposed framework. For future work, since it is harder to judge the quality of the intermediate reasoning, where the evaluation necessarily qualitative, we plan to do some sort of systematic study towards explainable harmful meme detection to claim explainability through a human subjects study for evaluation.
Broader Impact
The purpose of this work is to prevent the spread of harmful meme information and to ensure that peo- ple are not subjected to prejudice, racial and gender discrimination. Nevertheless, we are aware of the potential for malicious users to reverse-engined and create memes that go undetected or misunder- stood by MR. HARM-triinedd Al systems. This is strongly discouraged and condemned. Intervention with human moderation would be required in order to ensure that this does not occur.

---

References
bidirectional transformers for language understand- ing. In Proceedings of NAACL-HLI, pages 4171- 4186.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu. Willy Chung, et al. 2023. multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, arXiv preprint Xrxv:22302.000 04023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Corey H Basch, Zoe Meleo-Erwin, Joseph Fera, Christie Jaime, and Charles EE Basch. 2021. A global pan- demic in the time of viral memes: Covid-19 vaccine misinformation and disinformation on tiktok. Human vaccines & inmunotherapeuutics, I 17(8))2373-2377
Ming Shan Hee, Roy Ka-Wei
Lee Lee, and Wen-Haw Chong. 2022. On explaining multimodal hateful meme de- tection models. In Proceedings of the ACM Web Conference 2022, pages 3651-3655.
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Mar- keeva, Rohan Anil, and Alexander Kolesnikov. 2022. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 10925-10934
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.0233.
Namgyu Ho, Laura Schmid, and Se- Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.1 10071.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Sys- tems, pages 1877-1901.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, (5()))1333
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. 2019. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv: 1909.02950.
Cristian BuciIuă, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes chal- lenge: detecting hate speech in multimodal memes. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 2611-2624.
Rui Cao, Roy Kai Lee, Wen-Haw Chong, and Jing Jiang. 2022. Prompting for multimodal hateful meme classification. In Procedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 321-332.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311..
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. arXiv preprint w0Xi2:330.002633.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language models.
2022. Scaling instruction-finett arXiv preprint arXiv:22 10.1 1416.
Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue. Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao. Wenwei Zhang, al. 2021. Mmoor: a comprehensive toolbox for text detection, recognition and understanding, In Proceedings of the 29th ACM International Conference on Multimedia. pages 3791-3794.
Abhishek Das, Japsimar Singh Wahi, and Siyao Li. 2020. Detecting hate speech in multi-modal memes. arXiv preprint arXiv: 14897.
Ernest Davis and Gary Mareus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9)922- 103.
Roy KeiWei Lee, Rui Cao, Ziqing Fan, Jing Jiang, and Wen-Haw Chong. 2021. Disentangling hate in online memes. Proceedings of the 29th ACM Interna- tional Conference on Multimedia, pages 5138-5147.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep

---

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint arXiv: 1908.0357.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. arXiv preprint arXiv:2212.08410.
Hongzhan Lin, Jing Ma. Liangliang Chen. Zhiwei Yang. Mingfei Cheng, and Chen Guang. 2022. Detect ru- mors in microblog posts for low-resource domains via adversarial contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages
Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint mrxiv:2111.0973:
Niklas Muennighoff. 2020. Vilio: State-of-the-art visio- linguistic models applied to hateful memes, arXiv preprim: arXiv:201..07788.
Hongzhan Lin, Jing Ma, Mingfei Cheng, Zhiwei Yang. Liangliang Chen, and Guang Chen. 2021. Rumor detection on twitter with claim-guided bierarchical graph attention networks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10035-10047.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratch- pads for intermediate computation with language models. arXiv preprint arXiv:2112.0011
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays. Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pages 740-755. Springer.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Paul R Pintrich and Dale H Schunk. 2002. Motivation in education: Theory, research, and applications. Prentice Hall.
Phillip Lippe. Nithin Holla, Shantanu Chandra, San- thosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. 2020. multi- modal framework for the detection of hateful memes. arXiv preprint arXiv: arXiv: 12871.
Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021a. De- tecting harmful memes and their targets. Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pages 2783-2796.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeulemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.
Shraman Pramanick, Shivam Sharma, Dimitar Dim- itrov, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021b. Momenta: A multimodal frame- work for detecting harmful memes and their targets. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4439 4455.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: pretraining task-agnostic visiolinguis- tic representations for vision-and aaan-and-languun language tasks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13-23.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International con- ference on machine learning, pages 8748-8888 -7663.
Ziyang Luo, Zhipeng Hu, Yadong Xi, Rongsheng Zhang, and Jing Ma. 2022. I-tuning: Tuning frozen language models with image for lightweight image captioning. ICASSP 2023 2023 IEFF International Conference on Acoustics, Speech and Signal Process- ing (ICASSP).
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 22.
Jing Ma and Wei Gao. 2020.
Debunking rumors on twitter with tree transformer. In Proceedings of the 28th International Conference Computational Lin- guistics, pages 5455-5466.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-551)
Jing Ma. Wei Gao, Shafiq Joty, and Kam-Fai Wong. 2020. An attention-basn -hased rumor detection model with tree-structured recursive neural networks. ACM Transactions on Intelligent Systems and Technology (TIST), 144:

---

Shaoqing Ren, Kaiming He. Ross Girshick, and Jian Sun. 2016. Faster r-cna: Towards real-time object de- tection with region proposal networks. JEFR Trans- actions on Pattern Analysis and Machine Intelligence. 19(6):1137-1144
et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. In Advances in Neural Information Processing Systems.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv: 20.00299..
Vlad Sandulescu. 2020. Detecting hateful memes us- ing a multimodal deep ensemble. arXiv preprint arXiv:2012.13235.
Yi Zhou, Zhenhao Chen, and Huiyuan Yang. 2021. Mul- timodal learning hateful memes detection. 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1-6. IEEE.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut, 2018. Conceptual captions: A cleaned, hypemnmeed image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2556-2565.
Jiawen Zhu, Roy Ka- Wei Lee, and Wen Haw Chong. 2022. Multimodal zero-shot hateful meme detection. In 14th ACM Web Science Conference 2022, pages 382-389.
Shivam Sharma, Firoj Alam, Md Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and under- standing harmful memes: survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pages 5597-5606.
Ron Zhu. 2020. Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution. arXiv preprint arXiv:2012.08290.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for lcage image recognition. arXiv preprint arXiv: 150,,
Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mi- hael Arcan, and Paul Buitelaar. 2020. Multimodal meme dataset (multioff) for identifying offensive con- tent in image and text. In Proceedings of the second workshop on trolling, aggrexsion and cyberbullying, pages 32-41.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems-Volume Volume 2, pages 3104-3112.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-TTee Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint 220i 209.99999.9
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS.
Riza Velioglu and Jewgeni Rose. 2020. Detecting hate speech in memes using multimodal deep learning approaches: Prize-wines trize-winning solution hateful memes challenge. arXiv preprint 201.
Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2022. Pinto: Faithful language rea- soning using prompt-generated rationales. arXiv preprint 22..0222...0/56.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quac Le, Denny Zhou,

---

Train #harmless
Test
Hyper-Parameter
Harm-C
Harm-P
FHM
Datasets
#harmful
#harmful
#harmless
First-Stage
Hamm-C
1064
1949
124
230
Harm-P
1486
1534
173
182
epech
10
10
10
FHM
3050
5450
250
250
batch size
32
32
32
Learning Rate
5e-5
5e-5
5e-5
Table 4: Statisties of Datasets.
Warmup Step
0.1
0.1
0.1
Warmup Strategy
Linear
Linear 224
Linear
Image Size
224
224
A Datasets
Second-She -Sagee
The detailed statistics of the three datasets shown in Table 4.
epoch
30
30
30
batch size
32
32
32
Implementation Details
Learning Rate
5e-5
5e-4
le-4
Warmup Step
0.1
0.1
0.1
To separate the text and image in the memes, we first in-paint the memes by combining MMOCR (Kuang et al., 2021) with SAM (Kirillov et al., 2023) to extract the text and pure image. Then during the captioning process, since the focus of this work is primarily on the multimodal rea- soning for harmful meme detection from a fresh perspective on harnessing LLMs, we apply a pre- trained image captioning model ClipCap (Mokudy al., 2021) used in recent work (Cao al., 2022), to generate textual descriptions about the dominant objects or events in the memes' image, which is utilized as the inputs into LLMs for abductive rea- soning. To generate the rationale for each meme. we employed ChatGPT (Ouyang et al., 2022), widely used LLM developed by OpenAI, specifi- cally utilizing the "gpt-3.5-turbo"" version. To make results reproducible, we the temperature as 0 and the maximum length as 256.
Warmup Strategy
Linear
Linear
Linear
Image Size
224
224
224
Table 5: Hyper-parameters.
sonable rationales with the triplet (v, [v, T) as ob- served attributes, we design the template for the user prompt as:
"Given a Text: [TJ. which is embedded in an Image: [I]; and a harmfulness label [y]. please give me a streamlined rationale associated with the meme, without explicitly indicating the label, for how it is reasoned as [y].".
Our MR.HARM model utilizes the T5 encoder- decoder architecture (Raffel al., 2020; Chung et al., 2022) as its foundational framework, specif- ically utilizing the "flan-t5-base" version. For the extraction of image features, following previous work (Pramanick et al., 2021b), we adopted the state-of-the vision Transformer known CLIP- ViT-B/32 (Radford al., 2021), and this module remains static throughout the training process. To effectively integrate the multi-modal information, we incorporated a simple one-head cross-attention mechanism in each layer of the T5 encoder. Dur- ing the fusion process, the text features are utilized as the query, while the image features act as the key and value. It is noteworthy that these fusion modules were initialized randomly. For the fine- tuning phase, provide comprehensive list of the aaprr-paraaeters in Table 5. Results are aver- aged over ten random runs. All experiments were conducted using a single V100 32GiB GPU.
For the system prompt to the "spt-3.5-turbo" model, we design the message as:
"You have been specially designed to perform abductive reasoning for the harmful meme detec- tion task. Your primary function is that, according harmfulness label about an image with text embedded, please provide a streamlined rationale, without explicitly indicating the label, for how is reasoned as the given harmfulness label. The image and the textual content in the meme are often uncorrelated, but its overall semantics is presented holistically. Thus it is important to note that you are prohibited from relying on your own imagina- tion, your goal is to provide the most accurate and reliable rationale possible that people can infer the harmfulness according to your reason- ing about the background context and relationship between the given text and image.".
Baselines
We compare our model MR.HARM with several state-of of-the-rtt harmful meme detection systems:
Moreover, to prompt the LLMs to generate rea-

---

Dataset
Harm-C
Harm-P
FHM
Version
Accuracy
Macro-Fi
Accuracy
Macro-Fi
Accuracy
Macro-Fi
Small
85.59
84.99
85.35
85.33
73.20
72.96
Base
86.16
85.43
89.58
89.57
75,40
75.10
Large
85.03
84.02
90.14
90.14
78.20
77.80
Table 6: The detailed results with different sizes of our fine-tuned LMs.
Models
MOMENTA
MaskPrompt
MR.HARM
Multimodal Fusion
X
Prompt Tuning X
V
Explicit Reasoning
X
X
Leveraging L.Mss
X
X
Add
Table 7: Comparison of characteristics between HARM with state-of-the-art models for harmful meme detection.
LM Encoder Layer
Frozen Vision Transtormer
Meme Text
1) Text BERT: BERT (Devlin al., 2019) is uti- lized as the unomodal lext-only model; Image- Region: unimodal visual-only model that pro- cesses meme images using Faster R-CNN (Ren et al., 2016) with ResNet-152 (He et al., 2016) to feed into classification layer; 3) Late Fusion: multimodal model uses the average prediction scores of BERT and ResNet-152 for harmful meme detection (Pramanick al., 2021a); 4) MMBT: multimodal Bi-Transformer (Kiela al.. 2019) that captures the intrammode and inter-modal dy- namics of the two modalitics; 5) VisualBERT COCO: Visual BERT (Li et al., 2019) pre-trained on the COCO dataset (Lin et al., 2014); 6) ViL- BERT CC: Vision and Language BERT (Lu et al., 2019) trained on intermediate multimodal ob- jective (Sharma al., 2018) for task-agnostic joint representations of image and text; 7) MOMENTA: multimodal harmful meme detection system (Pra- manick et al., 202 bb) that takes the global and lo- cal information in two modalities of memes into account; 8) MaskPrompt: a prompt learning ap- proach (Cao et al., 2022) that converts harmful meme detection as masked language modeling problem based on RoBERTa-large (Liu et al., 2019). We accuracy and macro-averaged F1 score as the evaluation metrics, where the macro-averaged F1 is the more important metric owing to the im- balanced class prevalence (sce Table 4), to cap- ture competitive performance beyond the majority class.
De:
Parameters Frezen
: Parametera Not Frezen
Figure 5: The details of our Multimodal Fusion module.
LMs' setting in Table 2, they are challenging to serve in practice that requires at least 350GB GPU memory using specialized infrastructure for sin- gle 175 billion LLM. This work presents a novel paradigm to leverage the reasoning ability and rich background knowledge of LLMs for better harm- ful meme detection, but just need to fine-tune the small language model even with smaller size than the state-of- tate-offthhe-rt baseline. Table illustrates the detailed results on the three meme datasets with dif- ferent versions of our fine-tuned backbone model. Table 7 illustrates the comparison of characteris- tics between MR.HARM and the state-of-the-art baselines like MOMENTA and MaskPrompt.
D Illustration of Multimodal Fusion
Figure 5 illustrates the details of our multimodal fu- sion module in the encoding phase of HARM..M.
E Diseussion about One-stage Training
We further investigate the one-stage training to figure out the intrinsic property of the chain-of- thought reasoning. We compare the performance with two proposed variants for the one-stage train- ing: 1) Explanation where the rationale is utilized for explaining the harmfulness inference; 2) Rea- soning where harmfulness inference conditioned
While LLMs offer strong zero/few-shot perfor- mance as shown in the 'w/o Fine-tuning Small

---

Datasct
Harm-C
Harm-P
FHM
Model
Accuracy
Macro-Fi
Accuracy
Macro-Fi
Accuracy
Macro-P
Explanation Reasoning
83.05
81.45 56.19
63.32 56.90
63.32 56.67
67.40 63.00
65.77
68.93
59.29
Table Effects of the one-stage training.
to the rationale. As shown in Tabel 8, the reasoning setting performs worse than the explanation setting with large margin. We conjecture that this is be- cause the reasoning setting in the one-stage training could lead to error propagation if our small lan- guage model generates hallucinated rationales that mislead the harmfulness inference, which however could be well avoided by the two-stage training paradigm. Meanwhile, as there exists mutual in- terference between rationale generation and harm- fulness prediction, the explanation setting could give the harmfulness inference higher priority in the sequence generation so that it performs better than the reasoning setting. We argue that such one-stage training paradigm could be improved in the future by applying filtering mechanism, e-g., using only the effective thain-of-thought reasoning to infer the harmfulness of memes and get rid of irrelevant rationales. In summary, both settings in the one-stage training paradigm suffer different de- grees of performance degradation, which reaffirms the necessity of our two-stage training paradigm.
Ma et al., 2020), automatic evaluation, and human evaluation.
We would further update our framework by leveraging visual LLMs if accessible in the future to improve the visual feature extraction for better multimodal reasoning, and avoid several common deficiencies of cxisting lan- guage models including hallucination and lim- ited generalization as much as possible.
F Future Work
We will explore the following directions in the fu- ture:
Considering that our framework could gener- ate readable snippets for cognition-view rea- soning, we plan to do some sort of system- atic study to claim explainability (possibly through a human subjects study) for the eval- uation.
In this work we target exploring the underly- ing reasoning process to empower the harmful meme detection model with the ability of ex- plicit reasoning, to arrive at correct harmful- ness predictions. We are going further ploit LLMs toward explainable harmful meme detection from perspectives like dataset con- struction on social media with propagation structure (Lin et al., 2021: Ma and Gao, 2020;


Reviews:
[{"review_id": "SzEOGni5xt", "reviewer": "Reviewer_ue2E", "summary": "", "questions": "Question A:\nWhy distilling can help alleviate the hallucination issue? \nAlthough the authors conduct the ablation test without distilling to demonstrate its necessity, the authors should cite related research or provide more in-depth analysis or experiments to demonstrate how the reduction of parameters and limiting knowledge through distillation can effectively alleviate hallucination issues.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for insightful and careful comments. We are grateful for the reviewer's positive assessment of our research as novel work, with clear writing, and effective results. We have taken the valuable and constructive feedback into consideration and have tried to address your concerns. We will make the necessary modifications and additions accordingly.  \n  \n  \n**Q1**: Explore ways to ensure that only high-quality reasoning is utilized in the proposed model?  \n**A1**: This was indeed a tricky issue for us to consider in this work, for which we have proposed both the abductive reasoning mechanism and the two-stage training paradigm to ensure the quality of reasoning utilized in our proposed model.\n1) Abductive reasoning with LLMs: We applied abductive reasoning on LLMs as the filtering mechanism in Sec.3.1, which constrains the direction of the multimodal reasoning just towards the gold-truth label in training data. This design can *explicitly ensure the generation of more reasonable rationale from LLMs*. For example, if one meme in training data is labeled as harmful, the invalid rationales for the wrong prediction (harmless) would be filtered out in this abductive reasoning manner. \n\nThe results and analysis (Line#515) in Sec.4.3 further verified that if we only used LLMs without any filtering mechanism like abductive reasoning, the detection performance would largely decrease, which reaffirmed the effectiveness of abductive reasoning with the gold-truth label.\n\n2) Two-stage training paradigm for small LMs: Meanwhile, we also proposed a two-stage training paradigm, which *not only addresses the issue that abductive reasoning is not applicable to model testing, but also further alleviates the negative impact imposed by low-quality reasoning from LLMs*. Even if for some challenging cases, the quality of reasoning from ChatGPT is “relatively” poor for distillation in the first stage, the second stage implicitly ensures that only high-quality reasoning knowledge can be induced and activated during backward fine-tuning (Line#378). \n\nIn the analysis (Line#479; Line#1039) in (Sec.4.3; Sec.E), we also compared our two-stage training paradigm with a naïve one-stage one, which pointed out that the two-stage design is more robust than the one-stage one. In our two-stage training paradigm, the backward fine-tuning in the Harmfulness Inference stage would make it activate the harm-indicative knowledge absorbed in the Reasoning Distillation stage, unlike in the one-stage fashion letting the output of LLMs directly affect the prediction results.\n\n\n**Q2**: Why distilling can help alleviate the hallucination issue? (Question A)  \n**A2**: Thank you for your question. First of all, we need to clarify that distilling (Sec.3.2) is not to alleviate the hallucination issue. It is a way to inject the multimodal reasoning knowledge from LLMs into our small model. More specifically, as we applied the abductive reasoning mechanism with LLMs to generate more reasonable rationales, the distilling process could equip our small model with the capability of multimodal reasoning without the label (Line#255-265, Line#299-301). After the distilling process, we have a second fine-tuning stage, Harmfulness Inference, to further circumvent the effect of the still-existing hallucination issue on the model's final predictions. As stated in Line#387-389, for the model testing, we only applied the final fine-tuned model after the Harmfulness Inference stage.\n \n**Q3**: The newest SOTA model is \"MaskPrompt\" from arxiv. Broader LLM-based models need to be selected as baselines.  \n**A3**:\n1) To the best of our knowledge, the latest SoTA model for harmful meme detection was MaskPrompt before this submission. And the MaskPrompt is accepted by EMNLP’22. We will revise the reference format in revision.\n2) For the LLM-based model, we evaluated ChatGPT (‘gpt-3.5-turbo’)  (Line#483) and different training paradigms (like the reasoning setting in Line#1033 and explanation setting in Line#1035) on this task in the analysis (Sec.4.3; Sec.E) for a broader comparison. As for other contemporaneous works based on LLMs, which were all released after April, and we didn’t compare with them in our initial submission. In the subsequent version, we will try to contain more latest methods for a more complete comparison.\n3) To further address this concern of Reviewer#ue2E, we directly applied the novel vision LLM, LLaVA (released in April 2023), as one multimodal baseline based on LLMs. The Accuracy and Mac-F1 scores are (50.28\\%, 49.40\\%), (49.84\\%, 34.86\\%), and (51.20\\%, 46.51\\%) on Harm-C, Harm-P and FHM, respectively. While our method could achieve (86.16\\%, 85.43\\%), (89.58\\%, 89.57\\%), (75.40\\%, 75.10\\%) accordingly. We can see from the evaluations of broader LLM-based models [ChatGPT] and [LLaVA] that, the direct deployment of both detection models and multimodal models based on LLMs, struggles without lightweight design specific to this task, which is claimed as one main contribution in our paper (Line#141-146).\n\n**Q4**: How the multimodal information from images is integrated into the reasoning process, and any possible misalignments or discrepancies that might arise due to this disparity?  \n**A4**: \n1) Mitigate information loss: We used the image captions in the prompts because LLMs are pre-trained in language modality (Line#294-299). To mitigate the information loss of the image captioning, we used original image features instead of captions in the two-stage training paradigm.   \n2) Avoid error propagation: Furthermore, in the second fine-tuning stage for harmfulness inference, we only relied upon the original visual features for multimodal fusion with the textual features. The possible misalignments, like the information loss in image captioning, would not directly affect the second fine-tuning stage. As stated in Line#387-389, for the testing, we only applied the final fine-tuned model after the Harmfulness Inference stage.   \n3) Experimental Analysis: The ablative results (Table 2) for ‘w/o multimodal fusion’ further verified the effectiveness of original image features in mitigating information loss. The experimental results in Table 2 and Table 8 have analyzed the effectiveness of our two-stage paradigm in avoiding error propagation.\n \n**Q5**: The datasets themselves might become outdated over time?  \n**A5**: It’s a good question. 1) We acknowledge that the distribution drift in datasets over time is a potential limitation for almost all data-driven tasks. 2) However, the contribution of this work is proposing a novel paradigm to distill commonsense reasoning knowledge for the harmful meme detection task. The proposed framework is general enough, which should still work with newly released stronger LLMs or new data/memes appearing on the Internet. 3) As you suggested, we would further discuss that issue in the Limitation section in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The language used in the manuscript is clear and comprehensible, effectively conveying the ideas and concepts presented.\n2. The chosen task of harmful meme detection is of utmost importance for social networks, given the pervasive influence of memes in online conversations. The authors have adeptly harnessed the reasoning capabilities of LLMs to enhance model interpretability and achieve superior performance compared to other SOTA methods.\n3. The proposed model is presented in a lucid manner, making it easily understandable and replicable for researchers in the field. Furthermore, the framework's versatility allows for its straightforward adaptation to various other NLP tasks.\n4. The conducted experiments are extensive and robust, effectively showcasing the efficacy of the proposed model. The inclusion of ablation tests highlights the significance of all individual components, and the detailed analysis provides compelling examples of multimodal reasoning. Moreover, the model's performance with flexible parameters is competitive with other existing models.", "reasons_to_reject": "1. The proposed model relies on distilling multimodal reasoning from ChatGPT without applying any filtering mechanism. This may lead to a significant impact on the model's performance, as the quality of reasoning from ChatGPT may vary. It is crucial for the authors to address this issue and explore ways to ensure that only high-quality reasoning is utilized in the proposed model.\n2. There might be a gap between the usage of images in the prompts and the image features used in the model. The authors should clarify how the multimodal information from images is integrated into the reasoning process and discuss any possible misalignments or discrepancies that might arise due to this disparity.\n3. Although the authors have conducted tests on benchmark datasets of memes, the datasets themselves could have limitations and might become outdated over time. Given the dynamic and rapidly changing nature of memes on the internet, there is a concern regarding whether the model's effectiveness can be maintained in real-world scenarios. The authors should acknowledge this potential limitation and discuss strategies to keep the model up-to-date with evolving meme trends.\n4. The selection of baselines may be limited, as the newest SOTA model is \"MaskPrompt\" from arxiv. To enhance the comprehensiveness and robustness of the evaluation, it is suggested that the authors incorporate a broader range of models published in the last two years. This inclusion should encompass both detection models and multimodal models based on LLMs. By incorporating a diverse set of baselines, the study can provide a more comprehensive evaluation and draw more insightful comparisons between the proposed method and SOTA alternatives.", "questions_to_authors": "Question A:\nWhy distilling can help alleviate the hallucination issue? \nAlthough the authors conduct the ablation test without distilling to demonstrate its necessity, the authors should cite related research or provide more in-depth analysis or experiments to demonstrate how the reduction of parameters and limiting knowledge through distillation can effectively alleviate hallucination issues.", "paper_topic_and_main_contributions": "This work presents a significant contribution to the field of harmful meme detection in social networks. The author proposes a novel multimodal framework, leveraging the reasoning capabilities of LLMs, to effectively identify harmful memes with robust support. The model's training methodology involves a two-stage strategy, starting with the distillation of multimodal reasoning from LLMs, followed by fine-tuning the generative framework to infer harmfulness.\nThe main contributions of the model are:\n1. This work introduces a pioneering approach by incorporating the reasoning power of LLMs, such as ChatGPT, into the task of harmful meme detection. This innovative integration of advanced language models enables the model to effectively reason about and identify harmful content within memes, enhancing the overall detection accuracy.\n2. By leveraging multimodal information, intrinsic to memes, in the reasoning process, the model successfully captures and encodes the complex and integrated semantic information present in the memes. This multimodal reasoning approach proves beneficial in effectively understanding the contextual nuances and visual cues that contribute to the harmfulness of memes.\n3. The proposed two-stage training strategy, involving the distillation of reasoning knowledge from LLMs followed by fine-tuning on specific tasks, is a notable contribution to the field. This approach optimizes the model's performance by focusing on the harmful meme detection task while also mitigating the computational costs associated with using LLMs, both in terms of time and space.\nOverall, this paper presents a well-developed and well-structured framework for harmful meme detection, incorporating innovative elements of LLM reasoning and multimodal information processing. The experiments and evaluations conducted by the author demonstrate the effectiveness of the proposed model in accurately identifying harmful memes within social networks. The insights provided by this work are of great value to the research community in the domain of Natural Language Processing and content moderation in social media platforms.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n6hIEoAxqJ", "reviewer": "Reviewer_f6a5", "summary": "", "questions": "[A]. Lines 286-289: “...and  invalid rationals are naturally filtered out.”\nWere the hallucinations completely taken care of?\n\n[B]. Eq 2: Are you proposing a “cross-attention” based approach or a “co-attention” one? You’ve shown equations for computing attended visual representations conditioned upon the textual ones. Do you do it similarly the other way around as well? Maybe it's not that obviously clear from the details provided; please provide details and clarify.\n\n[C]. Appendix F (Error Analysis, Fig. 6)\n-  Example (a): Did you try to analyze the implications of various visual artefacts like image quality, occlusion, obscurity, object category, etc., while correlating them with the likely errors that the proposed model made? If yes, adding details on that assessment would be more insightful (in the Appendix, maybe).\n-  Example (b): Did you consider the possibility of likely wrong annotations within the dataset and maybe analyzing the stances adopted by the LLM while generating responses on a case-to-case basis (at least for a few samples)? As for all we know, the political reputation and electoral prospects might get severely affected due to such memes, which could potentially contribute to harmfulness. \n-  Lines 1090-1093: the image “Jimmy Carter with a smile on his face” is mistakenly recognized as “an older man with a funny expression on his face.” \nHow do we ascertain that it's a mistake on LLM’s part? In fact, such an interpretation of Jimmy Carter’s expression could be leveraged to infer the potentially intended harmfulness within the meme. \n", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your meticulous feedback. We are delighted by the reviewer's recognition of the soundness, promotive findings, and distinctiveness of our research.  We've responded to all the concerns and will accordingly modify and supplement our paper in the subsequent version.\n\n***Major Concerns***:  \n**Q1**: The paper lacks sufficient evaluations w.r.t the datasets/tasks explored from amongst emotions, offensiveness and sexism.  \n**A1**:\n1) We have conducted the evaluations on the hateful emotion in memes (**FHM** dataset) as one of the additional task categories in harmful meme detection. Specifically, we empirically evaluated our work on three public benchmarks: the first two datasets (Harm-C, Harm-P) are benchmarks for harmful meme detection defined by previous work [1, 3]; the later one (FHM) is a meme dataset [2, 4] related to emotions with hate speech (Line#399), which is one of the additional task categories that the reviewer#f6a5 concerned, achieving 9.86\\% improvement of Mac-F1 score over the SoTA baseline[2].\n2) As emphasized in Sec.4.1, we strictly followed the dataset and evaluation settings in previous work [1, 2, 3] on harmful meme detection, where ***harm** [3] is defined much broader and generalized than hate emotions [4], offensiveness [5], and sexism [6]*.\n3) To further address this concern of Reviewer#f6a5, we further evaluate our model on MultiOff [5] and MAMI [6] datasets from offensiveness and sexism perspectives. Our model could achieve Mac-F1 score (improvement over [2]) of 69.61\\% (3.51\\%) and 74.98\\% (4.03\\%) on MultiOff and MAMI datasets.  \n\n[1] Pramanick, Shraman, et al. \"MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.  \n[2] Cao, Rui, et al. \"Prompting for Multimodal Hateful Meme Classification.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.  \n[3] Pramanick, Shraman, et al. \"Detecting Harmful Memes and Their Targets.\" Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.  \n[4] Kiela, Douwe, et al. \"The hateful memes challenge: Detecting hate speech in multimodal memes.\" Advances in neural information processing systems 33 (2020): 2611-2624.  \n[5] Suryawanshi, Shardul, et al. \"Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text.\" Proceedings of the second workshop on trolling, aggression and cyberbullying. 2020.  \n[6] Fersini, Elisabetta, et al. \"SemEval-2022 Task 5: Multimedia automatic misogyny identification.\" Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). 2022.\n \n\n**Q2**: The analysis section could be further strengthened.  \n**A2**: Thank you for your feedback. We have made the comprehensive analysis in our submission, including: 1) Detection performance analysis (Sec.4.2); 2) Ablative analysis (Sec.4.3) about our contributions; 3) Case studies (Sec.4.4) for the intermediate reasoning of small LMs; 4) Analysis of model size and training parameters (Sec.4.5) for small LMs; 5) Training paradigm analysis (Sec.E); 6) Error analysis (Sec.F) and 7) Future Work (Sec.I), etc.\nBut unfortunately, due to limited space, we put more analysis and data examples in Appendix. We will mark the correspondence between Appendix and the main paper more explicitly, and try to discuss more error analysis in revision.\n\n***Minor Concerns***:  \n**Q3**: The write-up quality and typo improvement issue.  \n**A3**: We would further polish our writing in the subsequent version. \n\n**Q4**: Line 357: generated by LLMs (Sutskever et al., 2014) Is this reference appropriate? Please recheck.  \n**A4**: Yes. The reference is for the whole sentence (Line#355-357) depicting the loss in the sequence-to-sequence learning between the predicted text and the rationale generated by LLMs, not only for Line#357. We will rewrite this sentence to mitigate misunderstanding.\n\n\n***Open Questions***:  \n**Q5**: Were the hallucinations completely taken care of? (Question A)  \n**A5**: The original sentence in the manuscript is “*the hallucination issue of LLMs could be **effectively alleviated**. Because the rich contextual background knowledge could be activated by abductive reasoning based on the ground truth and invalid rationales are naturally filtered out.*”\n1) Except for the reasoning knowledge based on the ground truth, the invalid rationales for the other labels (not ground truth) are completely filtered out with the abductive reasoning mechanism.\n2) Besides the abductive reasoning mechanism, our further proposed two-stage training paradigm enables that, only the final fine-tuned model after the Harmful Inference stage is applied for the model testing (Line#387-389), so there is no explicit hallucination issue during testing because the model only generates the predicted label for harmful meme detection.\n\n**Q6**: You’ve shown equations for computing attended visual representations conditioned upon the textual ones. Do you do it similarly the other way around as well for the cross-attention? (Question B)  \n**A6**: As stated in Eq.(3), it can only do it as Eq.(2), and obviously **not work** if do it similarly the other way around:\n1) From the motivation perspective: We have clarified in the justification (Line#311-318), our goal is to fine-tune the small language model.  So we need to train the language model encoder while the vision Transformer is frozen in the Encoding part (Line#323-332). To this end, Eq.(2) is proposed to attend the visual representations to the textual ones as the input of each LME layer.\n2) From the technique perspective: If we do computing attended textual representations conditioned upon the visual ones, the output dimension would be $\\bf n \\times d$ (Line#332), which is incompatible with the dimension $\\bf m \\times d$ of each LME layer (Line#324) and then causes Eq.(3) wrong during computing.  Besides, we also presented a more intuitive illustration (Figure 5) of the cross-attention mechanism in Appendix Sec.D.\n \n**Q7**: Discussion about Appendix Sec.F (Figure 6).  \n**A7**:   \n1) Example (a): \nWe exemplify two more wrongly predicted examples for error analysis of the implications of some visual artifacts:  \n(**Examlpe I**) A test example covid_memes_5450 in Harm-C dataset, which is similar to Figure 9.b in Appendix. The meme text is “*Me\\nTrump with\\ncovid19\\nIs this karma?*”. After the first fine-tuning stage (Reasoning Distillation), we generated the intermediate reasoning:   \n“*based on the text and image caption, it appears that the meme is making a political statement about trump's covid-19 covid-19 treatment. the text suggests that trump is a political figure, which could be interpreted as a reference to the current covid-19 pandemic. the image seems to be unrelated to the text and could be interpreted as a reference to the current covid-19 pandemic. ...*”   \nIn the harmful meme, the text “Me” refers to the man wearing glasses, standing in front of a window; the text “Trump with\\ncovid19” refers to the butterfly; the man comments that “Is this karma?” on the butterfly. However, the object category like butterfly is not correctly recognized due to the low-quality image (obscurity), and the inter-relationship between the image with the meme text is still not cognized, which leads to the generated reasoning making no sense.  \n(**Example II**) One more harmful test example is covid_memes_5644,  the girl dressed in red is in obscurity on her face, and the image is also low-quality. Character relationships in the image are not captured due to the occlusion of the American flag.   \nSo the generated intermediate reasoning is that “*...the image shows a man in a suit and tie, which is a popular and popular show. the text suggests that the person is going golfing and ignoring the pandemic, ... the meme does not appear to contain any offensive or harmful content and is likely intended to be a lighthearted joke*\". Therefore, in future work (Line#1133-1139), we proposed to leverage more powerful vision LLMs in the reasoning distillation stage of model training to improve the robustness of our small LMs in model testing.  \n2) Example (b):   \n- Possible wrong annotations?  \n  · Yes. We also found there are some wrong annotations. Specifically, in rare cases with wrong annotations, although we conduct abductive reasoning with the wrong golden label (Note that the LLMs can only be used during model training as clarified in Line#299-304), the content generated by LLMs still cannot well indicate how the meme is reasoned as the gold truth, like just output “the meme seems to be promoting a harmful and potentially harmful message” but without any concrete reasonable thought chains about the conclusion. On a positive note, the phenomenon illustrates that the lack of logical reasoning knowledge in LLMs towards wrong annotations, which also contributes to exploring better robustness of our approach to some extent for the wrongly labeled data.  \n- How do we ascertain that it's a mistake on LLM’s part?  \n  · Note that there is no any statement indicating it’s a mistake on LLM’s part in Sec.F. It’s a mistake generated by our small language model during testing. It is a test sample while LLMs can only be used during model training as clarified in Line#299-304.  \n  · In the image, Jimmy Carter is not making a funny face. And it is not a meme text. The example about “funny expression” could be referred to the memes_44 or memes_8240 (Figure 10.b) in Harm-P dataset. And the “Jimmy Carter with a smile on his face” is the caption we used the captioning models like ClipCap and BLIP on the test sample to generate for analysis of the error result. Besides, as human beings, we cannot get any funny elements on his face, so we do not easily identify it as a wrong annotation.  \n  ·  Thus it is important to note that the models are prohibited from relying on their own imagination. Any interpretation or reasoning needs to be built on the correct recognition of the harm-indicative patterns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task is complex and relevant, and the findings would strive to enhance our current understanding of the capacity/utility of LLMs for multimodal tasks, as against the conventional paradigm of exploring early/late, single/dual-stream multimodal modeling approaches.", "reasons_to_reject": "1. The write-up quality needs to improve significantly. \n2. Given the nature of the modularised approach proposed, the paper lacks sufficient evaluations w.r.t the datasets/tasks explored. Additional task categories from amongst emotions, offensiveness and sexism, for which publicly accessible meme datasets are available, should have been studied to establish better generalizability of the proposed approach.\n3. The analysis section could be further strengthened.\n", "questions_to_authors": "[A]. Lines 286-289: “...and  invalid rationals are naturally filtered out.”\nWere the hallucinations completely taken care of?\n\n[B]. Eq 2: Are you proposing a “cross-attention” based approach or a “co-attention” one? You’ve shown equations for computing attended visual representations conditioned upon the textual ones. Do you do it similarly the other way around as well? Maybe it's not that obviously clear from the details provided; please provide details and clarify.\n\n[C]. Appendix F (Error Analysis, Fig. 6)\n-  Example (a): Did you try to analyze the implications of various visual artefacts like image quality, occlusion, obscurity, object category, etc., while correlating them with the likely errors that the proposed model made? If yes, adding details on that assessment would be more insightful (in the Appendix, maybe).\n-  Example (b): Did you consider the possibility of likely wrong annotations within the dataset and maybe analyzing the stances adopted by the LLM while generating responses on a case-to-case basis (at least for a few samples)? As for all we know, the political reputation and electoral prospects might get severely affected due to such memes, which could potentially contribute to harmfulness. \n-  Lines 1090-1093: the image “Jimmy Carter with a smile on his face” is mistakenly recognized as “an older man with a funny expression on his face.” \nHow do we ascertain that it's a mistake on LLM’s part? In fact, such an interpretation of Jimmy Carter’s expression could be leveraged to infer the potentially intended harmfulness within the meme. \n", "paper_topic_and_main_contributions": "The paper presents a 2-stage framework for harmful meme detection, with reasoning distillation as part of the first step and harmfulness inference as the second. The first stage involves fine-tuning an encoder-decoder setup for generating intermediate reasoning, which is obtained via the abductive reasoning from the LLMs. The setup includes a multimodal fusion strategy for getting encoder representations, which are then used for conditioning towards generating intermediate reasoning text. The second stage uses the same modelling setup, with a different decoder object configured towards generating the harmfulness type. The authors have compared the performance of the proposed approach on three datasets (Harm-P, Harm-C and FBHM) and showcase its superiority over other competitive baselines.\n\nThe paper contributes via the following:\n- This is one of the initial works exploring the utility of text-based LLMs towards a 2-stage multimodal task.\n- A novel lightweight generative setup addressing the complex task of harmfulness prediction for memes.\n- Presents ablation studies to corroborate the performance edge of the proposed approach.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-Line 357: generated by LLMs (Sutskever et al., 2014)\nIs this reference appropriate? Please recheck. \n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-Lines 165-167: “...and struggle to yield good performance only using unimodal detection methods”.\nSure, but what does? Is it the researchers in general, models, LLMs, SOTA or conventional techniques? Proper Co-referencing seems to be missing.\n\n-Lines 286-289: “Because the rich contextual background knowledge could be activated by  abductive reasoning based on the ground truth and  invalid rationals are naturally filtered out.”\nCould be rephrased?\n\n-Line 409, 977: sevral -> several\n\n-Line 646-648: Disclaimer: “This paper contains discriminatory content that may be disturbing to some readers.”\nIt would be prudent to pull this to the first page rather than at the end for the required pretext. \n\n-Line 314: “...generated rationales from LLMs”\nIt's possible that I might’ve missed it, but is the specific LLM (ChatGPT) used for generating rationales mentioned anywhere before line 484 (Ablative study, Page 6)? I see the details regarding the implementation, base model used etc. are elaborated in the Appendix, but at least the basic details regarding these primary components/technologies leveraged/considered must be mentioned (or atleast referred to) as part of the main paper wherever relevant, so as enable better understanding of the scope and implications of the experimental choices/decisions made, w.r.t the modeling setup, and hence the results.\n"}}, {"review_id": "UqlbEmTDVV", "reviewer": "Reviewer_3phz", "summary": "", "questions": "See the previous section.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuable comments. We are encouraged that the reviewer found our research to be important, with the effective method, comprehensive evaluations and well-structured writing. In response to the reviewer's feedback, we have diligently addressed all of the concerns. We are committed to making the necessary modifications and additions to improve our paper in the subsequent version.  \n\n**Q1**: All the techniques used in the study are already existing and well-known, and the author has cleverly integrated them. In my opinion, the biggest contribution of this work seems to be how it introduces LLM into this focused task.  \n**A1**: Thank you for acknowledging that, *exploring harmful meme detection from a fresh perspective on advanced LLMs* is a main contribution of this work. However, this is **not** the only contribution of our paper. As depicted in Sec.1, our contributions are underestimated in the following aspects:  \n1) We identify the key limitation of existing harmful meme detection methods for understanding memes (Line#065-102);\n2) We propose to conduct abductive reasoning to activate reasoning knowledge related to gold-truth labels from LLMs (Line#266-282);\n3) We propose the knowledge distillation strategy to inject reasoning knowledge from single-modality to multimodal inputs (Line#319-347);\n4) We propose the design philosophy of the two-stage training paradigm on small LMs, complementary to each other with the abductive reasoning step (Sec.3.2-3.3);\n5) We conduct extensive experiments with three public meme benchmarks, and demonstrate that our proposed method achieves much better performance than the newest SoTA harmful meme detection baselines.  \n\nWe believe that all these contributions amount to a major departure compared to SoTA, both algorithmically and practically speaking.  \n\n**Q2**: How these representative baseline models would perform if equipped with LLM?  \n**A2**: \n1) First of all, to equip existing SoTA harmful meme detection methods with the reasoning knowledge from LLMs, one straightforward and possible solution would be: use the rationale generated from LLMs as input for an end-to-end classification training. By this design, there may be two inherent weaknesses:   \n(I) With just the meme text and image caption as prompt of LLMs, the generated rationales are always noisy and low-quality, even towards the wrong label. It could directly decrease the final detection performance (Line#515-519);   \n(II) If without our proposed abductive reasoning mechanism, the quality of generated rationales from LLMs needs some additional designs to filter; If with the abductive reasoning mechanism, baselines cannot be deployed for model testing as stated in Line#300-302.   \n2) For more comprehensive comparisons, we also conducted the ablative studies about our models without LLMs. As shown in Table 2 (‘w/o Reasoning Distillation’), it proves that our model can still achieve competitive performance compared with baselines, even discarding that strategy with LLMs. Please refer to Table 1 and Table 2 for more details. In Figure 4, we further analyzed that our framework has fewer training parameters than some of the SoTA baselines.\n\n**Q3**: The author does not provide reproducible source code.  \n**A3**: Our approach could be easily reproduced according to Sec.3 (Line#266-389) and Implementation Details (Line#911-975). We will release all the source codes once published (Line#150).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research topic is important.\n\n2. The method of integrating LLM is straightforward but effective. The experiments are comprehensive.\n\n3. The paper is well-structed and easy to follow. Moreover, the authors provide many case studies for better understanding.", "reasons_to_reject": "1. All the techniques used in the study are already existing and well-known, and the author has cleverly integrated them. In my opinion, the biggest contribution of this work seems to be how it introduces LLM into this focused task.\n\n2. The baselines compared by the author are not based on LLM, and I am not sure if this comparison is fair. It remains unknown how these representative baseline models would perform if equipped with LLM.\n\n3. The author does not provide reproducible source code.", "questions_to_authors": "See the previous section.", "paper_topic_and_main_contributions": "This paper investigates the problem of detecting harmful memes, which is of great significance in today's increasingly popular social media platforms. Compared to previous end-to-end approaches for detecting harmful memes, this work proposes a two-stage training framework based on a large-scale language (LLM) model. The framework aims to improve multimodal fusion and lightweight fine-tuning for predicting harmfulness. It involves distilling multimodal reasoning knowledge from the LLM and fine-tuning a generative model for harmfulness inference. The authors highlight that this work is the first to explicitly leverage the knowledge of LLM to address the issue of superficial understanding of harmful memes. The effectiveness of the proposed model is validated through experiments on three real-world datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "The presentation is fine.", "Justification_for_Ethical_Concerns": "Due to the focus of this work on detecting harmful memes in social media, there may be some discriminatory or harmful content."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a 2-stage framework for harmful meme detection, leveraging the reasoning capabilities of Large Language Models (LLMs). Reviewers recognized this as an early exploration into using text-based LLMs for a multimodal task, praising the proposed two-stage training strategy involving knowledge distillation from LLMs and fine-tuning for specific tasks. However, reviewers expressed some concerns, which the authors addressed during the rebuttal stage and committed to resolving in the final version.
Decision: Accept-Findings