SlowGuess/NAACL_2022_OCR huggingface dataset:
Shivam Sharma $^{1,3}$ , Md. Shad Akhtar $^{1}$ , Preslav Nakov $^{2}$ , Tanmoy Chakraborty $^{1}$
$^{1}$ Indraprastha Institute of Information Technology - Delhi, India
$^{2}$ Qatar Computing Research Institute, HBKU, Doha, Qatar
<sup>3</sup>Wipro AI Labs, India
{shivams, shad.akhtar, tanmoy}@iiitd.ac.in
pnakov@hbku.edu.qa
Internet memes have emerged as an increasingly popular means of communication on the Web. Although typically intended to elicit humour, they have been increasingly used to spread hatred, trolling, and cyberbullying, as well as to target specific individuals, communities, or society on political, socio-cultural, and psychological grounds. While previous work has focused on detecting harmful, hateful, and offensive memes, identifying whom they attack remains a challenging and under-explored area. Here we aim to bridge this gap. In particular, we create a dataset where we annotate each meme with its victim(s) such as the name of the targeted person(s), organization(s), and community(ies). We then propose DISARM (Detecting vVictimS targeted by hARmful Memes), a framework that uses named entity recognition and person identification to detect all entities a meme is referring to, and then, incorporates a novel contextualized multimodal deep neural network to classify whether the meme intends to harm these entities. We perform several systematic experiments on three test setups, corresponding to entities that are (a) all seen while training, (b) not seen as a harmful target on training, and (c) not seen at all on training. The evaluation results show that DISARM significantly outperforms ten unimodal and multimodal systems. Finally, we show that DISARM is interpretable and comparatively more generalizable and that it can reduce the relative error rate for harmful target identification by up to 9 points absolute over several strong multimodal rivals.
Social media offer the freedom and the means to express deeply ingrained sentiments, which can be done using diverse and multimodal content such as memes. Besides being popularly used to express benign humour, Internet memes have also been misused to incite extreme reactions, hatred, and to spread disinformation on a massive scale.
(a) Harmful reference
(b) Harmless reference
Figure 1: (a) A meme that targets Justin Trudeau in a harmful way, with a communal angle. (b) A non-harmful mention of Justin Trudeau, as a benign humor.
Numerous recent efforts have attempted to characterize harmfulness (Pramanick et al., 2021b), hate speech (Kiela et al., 2020), and offensiveness (Suryawanshi et al., 2020) within memes. Most of these efforts have been directed towards detecting malicious influence within memes, but there has been little work on identifying whom the memes target. Besides detecting whether a meme is harmful, it is often important to know whether the meme contains an entity that is particularly targeted in a harmful way. This is the task we are addressing here: detecting the entities that a meme targets in a harmful way.
Harmful targeting in memes is often done using satire, sarcasm, or humour in an explicit or an implicit way, aiming at attacking an individual, an organization, a community, or society in general. For example, Fig. 1a depicts Justin Trudeau, the Prime Minister of Canada, as communally biased against Canadians, while favoring alleged killings by Muslims, whereas Fig. 1b shows an arguably benign meme of the same person expressing subtle humour. Essentially, the meme in Fig. 1a harmfully targets Justin Trudeau directly, while causing indirect harm to Canadians and to Muslims as well. Note that in many cases interpreting memes and their harmful intent requires some additional background knowledge for the meme to be understood properly.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1572 - 1588
Hence, an automated system for detecting the entities targeted by harmful memes faces two major challenges: (i) insufficient background context, (ii) complexity posed by the implicit harm, and (iii) keyword bias in a supervised setting.
To address these challenges, here we aim to address the task of harmful target detection in memes by formulating it as an open-ended task, where a meme can target an entity not seen on training. An end-to-end solution requires  $(i)$  identifying the entities referred to in the meme, and  $(ii)$  deciding whether each of these entities is being targeted in a harmful way. To address these two tasks, we perform systematic contextualization of the multimodal information presented within the meme by first performing intra-modal fusion between an external knowledge-based contextualized-entity and the textually-embedded harmfulness in the meme, which is followed by cross-modal fusion of the contextualized textual and visual modalities using low-rank bi-linear pooling, resulting in an enriched multimodal representation. We evaluate our model using three-level stress-testing to better assess its generalizability to unseen targets.
We create a dataset, and we propose an experimental setup and a model to address the aforementioned requirements, making the following contributions:
1. We introduce the novel task of detecting the entities targeted by harmful memes.
2. We create a new dataset for this new task, Ext-Harm-P, by extending Harm-P (Pra-Manick et al., 2021b) via re-annotating each harmful meme with the entity it targets.
3. We propose DISARM, a novel multimodal neural architecture that uses an expressive contextualized representation for detecting harmful targeting in memes.
4. We empirically showcase that DISARM outperforms ten unimodal and multimodal models by several points absolute in terms of macro-F1 scores in three different evaluation setups.
5. Finally, we discuss DISARM's generalizability and interpretability.
Misconduct on Social Media. The rise in misconduct on social media is a prominent research topic. Some forms of online misconduct include rumours (Zhou et al., 2019), fake news (Aldwairi and Alwahedi, 2018; Shu et al., 2017; Nguyen et al., 2020), misinformation (Ribeiro et al., 2021; Shaar et al., 2022), disinformation (Alam et al., 2021; Hardalov et al., 2022), hate speech (MacAvaney et al., 2019; Zhang and Luo, 2019; Zampieri et al., 2020), trolling (Cook et al., 2018), and cyber-bullying (Kowalski et al., 2014; Kim et al., 2021). Some notable work in this direction includes stance (Graells-Garrido et al., 2020) and rumour veracity prediction, in a multi-task learning framework (Kumar and Carley, 2019), wherein the authors proposed a Tree LSTM for characterizing online conversations. Wu and Liu (2018) explored user and social network representations for classifying a message as genuine vs. fake. Cheng et al. (2017) studied user's mood along with the online contextual discourse and demonstrated that it helps for trolling behaviour prediction on top of user's behavioural history. Relia et al. (2019) studied the synergy between discrimination based on race, ethnicity, and national origin in the physical and in the virtual space.
Studies Focusing on Memes. Recent efforts have shown interest in incorporating additional contextual information for meme analysis. Shang et al. (2021a) proposed knowledge-enriched graph neural networks that use common-sense knowledge for offensive memes detection. Pramanick et al. (2021a) focused on detecting COVID-19-related harmful memes and highlighted the challenge posed by the inherent biases within the existing multimodal systems. Pramanick et al. (2021b) released another dataset focusing on US Politics and proposed a multimodal framework for harmful meme detection. The Hateful Memes detection challenge by Facebook (Kiela et al., 2020) introduced the task of classifying a meme as hateful vs. non-hateful. Different approaches such as feature augmentation, attention mechanism, and multimodal loss re-weighting were attempted (Das et al., 2020; Sandulescu, 2020; Zhou et al., 2021; Lippe et al., 2020) as part of this task. Oriol et al. (2019) studied hateful memes by highlighting the importance of visual cues such as structural template, graphic modality, causal depiction, etc.
1The source code and the dataset can be found here https://github.com/LCS2-IIITD/DISARM.
Split	# Examples	Category-wise # Samples.
Harmful	Not-harmful
Train	3,618	1,206	2,412
Validation	216	72	144
Test	612	316	296
Total	4,446	1,594	2,852
Table 1: Summary of Ext-Harm-P, with overall and category-wise # of samples.
Web-entity detection along with fair face classification (Karkkainen and Joo, 2021) and semisupervised learning-based classification (Zhong, 2020) were also used for the hateful meme classification task. Other noteworthy research includes using implicit models, e.g., topic modelling and multimodal cues, for detecting offensive analogy (Shang et al., 2021b) and hateful discrimination (Mittos et al., 2020) in memes. Wang et al. (2021) argued that online attention can be garnered immensely via fauxtography, which could eventually evolve towards turning into memes that potentially go viral. To support research on these topics, several datasets for offensiveness, hate speech, and harmfulness detection have been created (Suryawanshi et al., 2020; Kiela et al., 2020; Pramanick et al., 2021a,b; Gomez et al., 2020; Dimitrov et al., 2021; Sharma et al., 2022).
Most of the above studies attempted to address classification tasks in a constrained setting. However, to the best of our knowledge, none of them targeted the task of detecting the specific entities that are being targeted. Here, we aim to bridge this gap with focus on detecting the specific entities targeted by a given harmful meme.
The Harm-P dataset (Pramanick et al., 2021b) consists of 3,552 memes about US politics. Each meme is annotated with its harmful label and the social entity that it targets. The targeted entities are coarsely classified into four social groups: individual, organization, community, and the general public. While these coarse classes provide an abstract view of the targets, identifying the specific targeted person, organization, or community in a fine-grained fashion is also crucial, and this is our focus here. All the memes in this dataset broadly pertain to US Politics domain, and they target well-known personalities or organizations. To this end, we manually re-annotated the memes in this dataset with the specific people, organizations, and communities that they target.
Candidates: bill clinton, hillary clinton, white house, donald trump, democrat Harmful: bill clinton, hillary clinton, democrat Not-harmful: white house, donald trump
Figure 2: Example meme, along with the candidate entities, harmful targets, and non-harmful references.
Extending Harm-P (Ext-Harm-P). Towards generalizability, we extend Harm-P by redesigning the existing data splits as shown in Table 1. We call the resulting dataset Ext-Harm-P. It contains a total of 4,446 examples including 1,594 harmful and 2,852 non-harmful; both categories have references to a number of entities. For training, we use the harmful memes provided as part of the original dataset (Pramanick et al., 2021b), which we re-announce for the fine-grained entities that are being targeted harmfully as positive samples (harmful targets). This is matched with twice as many negative samples (not-harmful targets). For negative targets, we use the top-2 entities from the original entity lexicon, which are not labeled for harmfulness and have the highest lexical similarity with the meme text (Ferreira et al., 2016). This at least ensures lexical similarity with the entities referenced within a meme, thereby facilitating a confounding effect (Kiela et al., 2020) as well. For the test set, all the entities are first extracted automatically using named entity recognition (NER) and person identification  $(\mathrm{PID})^2$ . This is followed by manual annotation of the test set.
Dataset Annotation Process Since assessing the harmfulness of memes is a highly subjective task, our annotators were requested to follow four key steps when annotating each meme, aiming to ensure label consistency. The example in Fig. 2 demonstrates the steps taken while annotating: we first identify the candidate entities, and then we decide whether a given entity is targeted in a harmful way. We asked our annotators to do the following (additional details about the annotation process are given in Appendix D):
2NER using SpaCy & PID using http://github. com/ageitgey/face_recognition.
(a) Individual
(b) Organization
(c) Community
1. Understand the meme and its background context.
2. List all the valid candidate entities that are referenced in the meme. For the example on Fig. 2, the valid entities are Bill Clinton, Hillary Clinton, White House, Donald Trump, and Democrat.
3. Assign the relevant entities as harmful. For the example on Fig. 2, Bill Clinton, Hillary Clinton, and Democrat are targeted in the meme for influencing the appointment of their kin on government positions.
4. Finally, assign harmless references to entities under the non-harmful category. In the example on Fig. 2, Donald Trump and White House would be annotated as non-harmful.
We had three annotators and a consolidator. The inter-annotator agreement before consolidation had a Fleiss Kappa of 0.48 (moderate agreement), and after consolidation it increased to 0.64 (substantial agreement).
Analyzing Harmful Targeting in Memes. The memes in Ext-Harm-P are about US Politics, and thus they prominently feature entities such as Joe Biden and Donald Trump, both harmfully and harmlessly. The ratio between these types of referencing varies across individuals, organizations, and communities. We can see in Fig. 3 that the top-5 harmfully referenced individuals and organizations are observed to be subjected to a more relative harm (normalized by the number of occurrences of these entities in memes). However, the stacked plots for the top-5 harmfully targeted communities Mexicans, Black, Muslim, Islam, and Russian in Fig. 3c show relatively less harm targeting these communities.
Figure 3: Comparison plots for the top-5 harmfully referenced entities, for their harmful/non-harmful referencing in our dataset.
Figure 4: The architecture of our proposed approach DISARM. Here,  $\mathbf{c}_{mm}$  is the multimodal representation used for the final classification.
Our proposed model DISARM, as depicted in Fig. 4, is based on a fusion of the textual and the visual modalities, explicitly enriched via contextualised representations by leveraging CLIP (Radford et al., 2021). We chose CLIP as a preferred encoder module for contextualization, due to its impressive zero-shot multimodal embedding capabilities. At first, valid entities are extracted automatically, as part of the process of creating training/validation sets. Then, for each meme, we first obtain the contextualized-entity (CE) representation by fusing the CLIP-encoded context and the entity representation. CE is then fused with BERT-based (Devlin et al., 2019) embedded-harmfulness (EH) encoding fine-tuned on the OCR-extracted text and entities as inputs. We call the resulting fusion output a contextualized-text (CT) representation. CT is then fused with the contextualized-image (CI) representation, obtained using the CLIP encoder for the image. We, henceforth, refer to the resulting enriched representation as the contextualized multimodal (CMM) representation. We modify the multimodal low-rank bi-linear pooling (Kim et al., 2017) to fuse the input representation into a joint space.
This approach, as can be seen in the subsequent sections below, not only can capture complex cross-modal interactions, but it also provides an efficient fusion mechanism towards obtaining a context-enriched representation. Finally, we use this representation to train a classifier for our task. We describe each module in detail below.
Low-rank Bi-linear Pooling (LRBP). We begin by revisiting low-rank bi-linear pooling to set the necessary background. Due to the many parameters in bi-linear models, Piriavash et al. (2009) suggested a low-rank bi-linear (LRB) approach to reduce the rank of the weight matrix  $\mathbf{W}_i$ . Consequently, the number of parameters and hence the complexity, are reduced. The weight matrix  $\mathbf{W}_i$  is re-written as  $\mathbf{W}_i = \mathbf{U}_i\mathbf{V}_i^T$ , where  $\mathbf{U}_i \in \mathbb{R}^{N \times d}$  and  $\mathbf{V}_i \in \mathbb{R}^{M \times d}$ , effectively putting an upper bound of  $\min(N, M)$  on the value of  $d$ . Therefore, the low-rank bi-linear models can be expressed as follows:
$$
f _ {i} = \mathbf {x} ^ {T} \mathbf {W} _ {i} \mathbf {y} = \mathbf {x} ^ {T} \mathbf {U} _ {i} \mathbf {V} _ {i} ^ {T} \mathbf {y} = \mathbb {1} ^ {T} \left(\mathbf {U} _ {i} ^ {T} \mathbf {x} \circ \mathbf {V} _ {i} ^ {T} \mathbf {y}\right) (1)
$$
where  $\mathbb{1}\in \mathbb{R}^d$  is a column vector of ones, and  $\circ$  is Hadamard product.  $f_{i}$  in Equation (1) can be further re-written to obtain  $\mathbf{f}$  as follows:
$$
\mathbf {f} = \mathbf {P} ^ {T} \left(\mathbf {U} ^ {T} \mathbf {x} \circ \mathbf {V} ^ {T} \mathbf {y}\right) + \mathbf {b} \tag {2}
$$
where  $\mathbf{f} \in \{f_i\}$ ,  $\mathbf{P} \in \mathbb{R}^{d \times c}$ ,  $\mathbf{b} \in \mathbb{R}^c$ ,  $d$  is an output, and  $c$  is an LRB hyper-parameter.
We further introduce a non-linear activation formulation for LRBP, following Kim et al. (2017), who argued that non-linearity both before and after the Hadamard product complicates the gradient computation. This addition to Equation (2) can be represented as follows:
$$
\textbf {f} = \mathbf {P} ^ {T} \tanh (\mathbf {U} ^ {T} \mathbf {x} \circ \mathbf {V} ^ {T} \mathbf {y}) + \mathbf {b} (3)
$$
We slightly modify the multimodal low-rank bi-linear pooling (MMLRBP). Instead of directly projecting the input  $\mathbf{x} \in \mathbb{R}^N$  and  $\mathbf{y} \in \mathbb{R}^M$  into a lower dimension  $d$ , we first project the input modalities in a joint space  $N$ . We then perform LRBP as expressed in Equation 3, by using jointly embedded representations  $\mathbf{x}_{mm} \in \mathbb{R}^{N \times d}$  and  $\mathbf{y}_{mm} \in \mathbb{R}^{N \times d}$  to obtain a multimodal fused representation  $\mathbf{f}_{mm}$ , as expressed below:
$$
\mathbf {f} _ {m m} = \mathbf {P} ^ {T} \tanh (\mathbf {U} ^ {T} \mathbf {x} _ {m m} \circ \mathbf {V} ^ {T} \mathbf {y} _ {m m}) (4)
$$
Structured Context. Towards modelling auxiliary knowledge, we curate contexts for the memes in Ext-Harm-P. First, we use the meme text as a search query<sup>3</sup> to retrieve relevant contexts, using the title and the first paragraph of the resulting top document as a context, which we call con.
Contextualized-entity Representation (CE). Towards modelling the context-enriched entity, we first obtain the embedding of the input entity ent. Since we have a finite set of entities referenced in the memes in our training dataset, we perform a lookup in the embedding matrix from  $\mathbb{R}^{V\times H}$  to obtain the corresponding entity embedding ent  $\in \mathbb{R}^H$ , with  $H = 300$  being the embedding dimension and  $V$  the vocabulary size. We train the embedding matrix from scratch as part of the overall training of our model. We project the obtained entity representation ent into a 512-dimensional space, which we call e. To augment a given entity with relevant contextual information, we fuse it with a contextual representation  $\mathbf{c}\in \mathbb{R}^{512}$  obtained by encoding the associated context (con) using CLIP. We perform this fusion using our adaptation of the multimodal low-rank bi-linear pooling as defined by Equation (4). This yields the following contextualized-entity (CE) representation  $\mathbf{c}_{ent}$ :
$$
\mathbf {c} _ {e n t} = \mathbf {P} _ {1} ^ {T} \tanh (\mathbf {U} _ {1} ^ {T} \mathbf {e} \circ \mathbf {V} _ {1} ^ {T} \mathbf {c}) + \mathbf {b} \quad (5)
$$
where  $\mathbf{c}_{ent} \in \mathbb{R}^{512}$ ,  $\mathbf{P}_1 \in \mathbb{R}^{256 \times 512}$ ,  $\mathbf{b} \in \mathbb{R}^{512}$ ,  $\mathbf{U}_1 \in \mathbb{R}^{512 \times 256}$ , and  $\mathbf{V}_1 \in \mathbb{R}^{512 \times 256}$ .
Contextualized-Text (CT) Representation. Once we obtain the contextualized-entity embedding  $\mathbf{c}_{ent}$ , we concatenate it with the BERT encoding for the combined representation of the OCR-extracted text and the entity  $(\mathbf{o}_{ent} \in \mathbb{R}^{768})$ . We call this encoding an embedded-harmfulness (EH) representation. The concatenated representation from  $\mathbb{R}^{1280}$  is then projected non-linearly into a lower dimension using a dense layer of size 512. We call the resulting vector  $\mathbf{c}_{txt}$  a contextualized-text (CT) representation:
$$
\mathbf {c} _ {t x t} = \mathbf {W} _ {i} \left[ \mathbf {o} _ {e n t}, \mathbf {c} _ {e n t} \right] + b _ {i} \tag {6}
$$
where  $\mathbf{W}\in \mathbb{R}^{1280\times 512}$
3https://pypi.org/project/ googlesearch-python/
Contextualized Multimodal (CMM) Representation. Once we obtain the contextualized-text representation  $\mathbf{c}_{xt} \in \mathbb{R}^{512}$ , we again perform multimodal low-rank bi-linear pooling using Equation (4) to fuse it with the contextualized-image representation  $\mathbf{c}_{img} \in \mathbb{R}^{512}$ , obtained using the CLIP image-encoder. The operation is expressed as follows:
$$
\mathbf {c} _ {m m} = \mathbf {P} _ {2} ^ {T} \tanh  \left(\mathbf {U} _ {2} ^ {T} \mathbf {c} _ {t x t} \circ \mathbf {V} _ {2} ^ {T} \mathbf {c} _ {i m g}\right) \tag {7}
$$
where  $\mathbf{c}_{mm} \in \mathbb{R}^{512}$ ,  $\mathbf{P}_2 \in \mathbb{R}^{256 \times 512}$ ,  $\mathbf{U}_2 \in \mathbb{R}^{512 \times 256}$ , and  $\mathbf{V}_2 \in \mathbb{R}^{512 \times 256}$ .
Notably, we learn two different projection matrices  $\mathbf{P}_1$  and  $\mathbf{P}_2$ , for the two fusion operations performed as part of Equations (5) and (7), respectively, since the fused representations at the respective steps are obtained using different modality-specific interactions.
Classification Head. Towards modelling the binary classification for a given meme and a corresponding entity as either harmful or non-harmful, we use a shallow multi-layer perceptron with a single dense layer of size 256, which represents a condensed representation for classification. We finally map this layer to a single dimension output via a sigmoid activation. We use binary cross-entropy for the back-propagated loss.
We experiment with various unimodal (image/text-only) and multimodal models, including such pre-trained on multimodal datasets such as MS COCO (Lin et al., 2014) and CC (Sharma et al., 2018). We train DISARM and all unimodal baselines using PyTorch, while for the multimodal baselines, we use the MMF framework.
For evaluation, we use commonly used macro-average versions of accuracy, precision, recall, and F1 score. For example, we discuss the harmful class recall, which is relevant for our study as it characterizes the model's performance at detecting harmfully targeting memes. All results we report are averaged over five independent runs.
Evaluation Strategy. With the aim of having a realistic setting, we pose our evaluation strategy as an open-class one. We train all systems using under-sampling of the entities that were not targeted in a harmful way: using all positive (harmful) examples and twice as many negative (non-harmful) ones. We then perform an open-class testing, for all referenced entities (some possibly unseen on training) per meme, effectively making the evaluation more realistic. To this end, we formulate three testing scenarios as follows, with their Harmful (H) and Non-harmful (N) counts:
1. Test set A (316H, 296N): All examples in this dataset are about entities that were seen during training.
2. Test set B (27H, 94N): The examples in this set are about entities that were not seen as harmful during training.
3. Test set C (16H, 76N): All examples are about entities that were completely unseen during training.
Baseline Models. Our baselines include both unimodal and multimodal models as follows: