MODOS at ArAIEval Shared Task: Multimodal Propagandistic Memes Classification Using Weighted SAM, CLIP and ArabianGPT (2024)

Abstract: Arabic social media platforms are increasingly using propaganda to deceive or influence people. This propaganda is often spread through multimodal content, such as memes. While substantial research has addressed the automatic detection of propaganda in English content, this paper presents the MODOS team’s participation in the Arabic Multimodal Propagandistic Memes Classification shared task.1 Our system deploys the Segment Anything Model (SAM) and CLIP for image representation and ARABIAN-GPT embeddings for text. Then, we employ LSTM encoders followed by a weighted fusion strategy to perform binary classification. Our system achieved competitive performance in distinguishing between propagandistic and non-propagandistic memes, scored 0.7290 macro F1, and ranked 6th among the participants.

text: Proceedings of The Second Arabic Natural Language Processing Conference, pages 483–488 August 16, 2024 ©2024 Association for Computational Linguistics
Our system deploys the Segment Anything Model (SAM) and CLIP for image representation and ARABIAN-GPT embeddings for text. Then, we employ LSTM encoders followed by a weighted fusion strategy to perform binary classification. Our system achieved competitive performance in distinguishing between propagandistic and non-propagandistic memes, scored 0.7290 macro F1, and ranked 6th among the participants.

heading: 1 Introduction
text: The term "meme" was introduced by Richard Dawkins in his renowned publication, The Selfish Gene (Dawkins, 2006). Dawkins derived the term from the Greek word "mimeme," modifying it to denote a concept signifying "something which is imitated." In the era of social media, memes have frequently become a common method of communication, often used to spread messages with varied degrees of propaganda (Shifman, 2013). Propaganda is defined as deliberate expressions to influence opinions or actions, often using rhetorical and psychological devices.3 Understanding the dynamics of online conversation and preventing the spread of false information depends on the ability to identify propagandistic content. Consequently, researchers in Natural Language Processing (NLP) are increasingly interested in automatically detecting the application of propaganda techniques in text, images, and multimodal information (Dimitrov et al., 2021;
1https://araieval.gitlab.io/
Hasanain et al., 2023a). Prior work in Arabic has focused primarily on detecting propaganda in a social media text, particularly tweets (Alam et al., 2022; Hasanain et al., 2023a; Alam et al., 2024). Several shared tasks have been launched to address the detection of propaganda in Arabic tweets and the identification of persuasion techniques and disinformation in Arabic text.2,3 These tasks have attracted many researchers (Ojo et al., 2023; Xiao and Alam, 2023; Lamsiyah et al., 2023; Mittal and Nakov, 2022; Refaee et al., 2022; Attieh and Hassan, 2022; Samir et al., 2022; Hasanain et al., 2023b), leading to significant advances in recognizing and extracting linguistic cues that indicate the use of propaganda techniques. In these tasks, researchers have achieved good results in detecting propaganda in Arabic text by identifying the propaganda techniques used, which often include playing on the audience’s emotions (e.g., appealing to fear, using loaded language, etc.) (Miller, 1939). Despite significant progress conducted in multimodal learning for Arabic, which focused on tasks such as sentiment analysis, emotion recognition, summarization, and more, rather than propaganda detection. For instance, Haouhat et al. (Haouhat et al., 2023) developed an Arabic multimodal dataset for sentiment analysis that comprises transcripts, voice recordings and videos, enhancing the ability to capture nuanced content. These efforts highlight the potential of multimodal approaches in improving understanding across different domains. However, the specific challenge of detecting propaganda in multimodal formats remains less explored, indicating a critical gap in this area. To address the gap in Arabic multimodal propaganda detection research, Hasanain et al. (Hasanain et al., 2024b)
2WANLP-2022 Shared Task on propaganda detection in Arabic. Available at https://sites.google.com/view/propaganda-detection-inarabic/home, Accessed on 17/05/2024
3https://araieval.gitlab.io/2023/
483
introduced a subtask to the ArAIEval Shared Task at ArabicNLP 2024.4 Previous research predominantly focused on unimodal textual detection, leaving a notable gap in understanding the propagandistic content presented in multimodal formats. This subtask specifically focused on the classification of memes, representing a significant advancement in the field. Our team actively participated in this initiative, specifically targeting the second subtask, subtask2C, where we aimed to contribute to the detection of propagandistic content in multimodal memes. Despite the complexity of the task, our system demonstrated promising results, securing the sixth position in the competition. Our contribution involved the development of a comprehensive approach that integrates various techniques, including image segmentation, feature extraction, multimodal weighted fusion, and classification.

heading: 2 Related Work
text: (Haouhat et al., 2023) This section summarizes the previous research on Arabic propaganda detection in both unimodal and multimodal contexts. Authors in (Khanday et al., 2021) address propaganda dissemination through social networking platforms. They focus on classifying propagandist text from non-propagandist text using supervised machine learning algorithms. Data collected from news sources is annotated, and feature engineering is conducted using techniques such as term frequency/inverse document frequency (TF/IDF) and Bag of Words (BOW). Support Vector Machine (SVM) and Multinomial Naïve Bayesian (MNB) classifiers are employed. the study yields encouraging results. Bodor et al. (Almotairy et al., 2024) tackle the challenge of detecting and characterizing Arab computational propaganda on Twitter by providing a dataset that includes 16 million tweets. However, only 2100 labeled propagandist tweets covering banking and sports are included in the dataset, and the unlabeled tweets are limited to Saudi Arabian Twitter data, necessitating generalization to diverse topics and regions within the Arab world. Despite these limitations, the dataset offers valuable insights into Arab computational propaganda and enables supervised and unsupervised machine learning and deep learning algorithm applications to classify the credibility of Arab tweets. Firoj et al. (Alam et al., 2022) highlight the importance of bridging the language gap in pro-
4https://arabicnlp2024.sigarab.org/
paganda detection by focusing on Arabic, which previously received less attention. The authors organized a shared task conducted as part of the WANLP 2022 workshop, aiming to detect propaganda techniques in Arabic tweets through multilabel classification. The task attracted significant participation, with 63 teams.
Firoj et al. (Hasanain et al., 2023a) focus on the critical problem of identifying and labeling propagandistic spans within the textual content. The authors explore the potential of large language models (LLMs) like GPT-4 to function as both general and expert annotators for this task. The study looks at how well GPT-4 annotates spans and if it can take the place of consolidators during the annotation process. Notably, the study makes significant contributions by releasing annotations from multiple human annotators and GPT-4, aiming to benefit the research community and facilitate further advancements in propaganda detection and analysis. In (Hasanain et al., 2024a), the authors meticulously construct ArPro, the largest propaganda dataset in Arabic, comprising 8K paragraphs from newspaper articles labeled at the text span level across 23 propagandistic techniques. Additionally, the study delves into the potential of large language models (LLMs) for fine-grained propaganda detection from text. However, results reveal that while GPT-4 performs acceptably in classifying paragraphs as propagandistic or not.

heading: 3 Task Definition
text: The ArAIEval shared task is part of ArabicNLP 2024 conference, comprises two main challenges focused on the automated identification of propagandistic content within mainstream and social media. the first one deal with unimodal data by detecting propagandistic textual spans with persuasion techniques and second challenge focus on distinguishing between propagandistic and nonpropagandistic multimodal memes. Participants are tasked with analyzing tweets, news articles, and memes to identify misleading information, aiding fact-checkers, journalists, and the general public in combating disinformation.

heading: 4 Data
text: In this work, we utilize the dataset released for the ArAIEval shared task (Hasanain et al., 2024b). The dataset consisted of a diverse collection of memes labeled as propaganda or non-propaganda. Table1
presents an overview of the dataset, including the size of the training, development (Dev), and testing sets, as well as the number of propagandist (P) and non-propagandist (NP) instances in each set. Additionally, we provide information about the textual tokens and visual segments generated by SAM present in the dataset. Visual segments represent distinct regions within images identified and segmented by the Segment Anything Model (SAM), facilitating multimodal analysis alongside textual data.

heading: 5 System
text: Our methodology consists of several key steps: image segmentation, feature extraction, multimodal fusion, and classification. A GitHub repository for our project is provided.5 Figure 1 provides an overview of our approach.

heading: 5.1 Pre-trained Models
text: Before getting into the description of our approach, we start by describing the deployed pre-trained models.
Segment Anything Model (SAM), is a cuttingedge approach to image segmentation with outstanding performance and zero-shot capabilities. The three essential modules of SAM are the VisionEncoder, PromptEncoder, and MaskDecoder. They are all responsible for generating object masks. SAM is well-versed in both AutomaticMask-Generation, which enables the system to automatically generate masks for every object in an image, and Prompted-Mask-Generation. It was trained on a dataset consisting of 11 million images and 1.1 billion masks (Kirillov et al., 2023). Figure 2 illustrates the segmentation of all possible objects in the image with a high score of confidence
5https://github.com/belgats/Multimodal-PropagandaDetection
and stability set these scores to 0.95 to ensure getting masks with only minor boundary errors.
CLIP is a state-of-the-art multimodal vision and language model that is both scalable and resilient. This neural network is quite versatile. In contrast to conventional models, CLIP has exceptional "zeroshot" capabilities similar to GPT-2 and GPT-3, allowing it to carry out tasks like anticipating the most relevant text excerpt given the names of the visual categories to be identified (picture), all without the need for direct optimization (Radford et al., 2021).
ArabianGPT-08 is an improved version of ArabianGPT-01 with improved Arabic text processing capabilities. ArabianGPT-08 performs better in language generation tasks with enhanced accuracy and coherence thanks to adjusted parameters and a larger model size. Because of its improved tokenizer named "Aranizer", Arabic’s special linguistic characteristics can be handled more effectively, leading to more accurate text processing (Koubaa et al., 2024).
We employed SAM to segment objects on all meme images. This allowed us to isolate relevant visual elements from each image for further analysis. For feature extraction, our team performed cleaning up and pre-processing using the steps applied in previous NLP tasks on Arabic. We then extracted word embeddings from the text within the memes using ArabianGPT08. Additionally, for each segmented image, we extract image embeddings using a CLIP image encoder. Once all the embeddings capture both visual and textual information, we input the features using two LSTM encoders. This ensures the alignment and the equivalence of feature lengths, addressing inherent differences due to the nature of visual and textual data. The LSTM encoders generated new hidden features with dimensions of 768 by 128. We then employed a weighted fusion strategy, by calculating importance scores for each modality to combine the hidden features into multimodal representations, as shown in Equations 1 and 2.
hi = Concat(ft, fi) (1)
h = hTi .Wα (2)
The multilayer perceptron (MLP) effectively categorizes the multimodal features obtained from the fusion step into binary classes, indicating whether a meme is propagandistic or not. In the training process, we utilized the cross-entropy loss function L
to quantify the discrepancy between predicted class probabilities and true labels. Equation 3 describes this function:
L(y, ȳ) = − 1 N
N∑
n=1
( yn log(ȳn)+(1−yn) log(1−ȳn) )
(3)

heading: 6 Results and Discussion
text: We evaluated our model against the baseline models provided by shared task organizers: Ngram, Random, ImgBert, and Majority. The metrics used are precision, recall, F1 score, and accuracy. The performance of the models is presented in Table 2.
Our model demonstrates superior performance across all metrics, achieving an accuracy of 0.8040 and an F1 score of 0.7348 on the test set, ranking sixth in the leaderboard, the best performance with an F1 score of 0.8051 achieved by the AlexUNLPMZ team. Our finding indicate the effectiveness of our approach in correctly classifying the memes, with a good balance between precision and recall, efficiently handling both false positives and false negatives. However, there are observed limitations in model generalization due to not employing several propaganda techniques (e.g., name-calling, virtue words, deification, testimonial, fear, etc.) that could enhance the detection. Furthermore, The unbalanced class distribution in the dataset poses a challenge, potentially biasing the model towards the majority class and affecting its generalization capability.

heading: 6.1 Baselines
text: In this section, we define the baselines used for evaluating our approach:
The Majority Baseline classifies all instances with the most frequent class label found in the training dataset. It serves as a simple benchmark by predicting the most common class without considering any features of the data.
The Random Baseline predicts class labels by randomly selecting from the Propagandistic and non Propagandistic classes. This model also does not utilize any data features, instead provides a benchmark by guessing the labels randomly.
The N-gram Baseline employs a machine learning pipeline that includes a TF-IDF vectorizer to convert text data into numerical features and a Support Vector Machine (SVM) classifier for predict-
ing class labels. This method leverages n-gram features of the text, capturing patterns and dependencies in the textual data to inform classification.
The ImgBert Baseline integrates both image and text features for classification. It utilizes Preprocessed features from both modalities, combines them, and applies an SVM classifier to predict class labels. This approach is more complex, leveraging the advantages of multimodal data to enhance predictive accuracy.

heading: 7 Conclusion
text: In this study, we presented our approach to the Multimodal Propagandistic Memes Classification subtask at the ArAIEval shared task. By leveraging a combination of image segmentation, feature extraction, multimodal fusion, and fusing weighted strategy, we captured rich information from both modalities. We developed a model capable of effectively identifying propagandistic content in memes. Our experimental results demonstrate that the proposed model outperforms several baseline methods, achieving higher accuracy and F1 scores. Overall, Our findings indicate good participation. However, there is room for improvement, particularly in addressing overfitting and enhancing generalization capabilities. Future work will explore advanced multimodal fusion techniques, incorporate additional contextual information, and utilize larger and more diverse datasets.

heading: Future Work
text: Future work will explore advanced multimodal fusion techniques, incorporate additional contextual information, and utilize larger and more diverse datasets