Mela at ArAIEval Shared Task: Propagandistic Techniques Detection in Arabic with a Multilingual Approach (2024)

Abstract: This paper presents our system submitted for Task 1 of the ArAIEval Shared Task on Unimodal (Text) Propagandistic Technique Detection in Arabic. Task 1 involves identifying all employed propaganda techniques in a given text from a set of possible techniques or detecting that no propaganda technique is present. Additionally, the task requires identifying the specific spans of text where these techniques occur. We explored the capabilities of a multilingual BERT model for this task, focusing on the effectiveness of using outputs from different hidden layers within the model. By finetuning the multilingual BERT, we aimed to improve the model’s ability to recognize and locate various propaganda techniques. Our experiments showed that leveraging the hidden layers of the BERT model enhanced detection performance. Our system achieved competitive results, ranking second in the shared task, demonstrating that multilingual BERT models, combined with outputs from hidden layers, can effectively detect and identify spans of propaganda techniques in Arabic text.

heading: 1 Introduction
text: Propaganda is a powerful tool often used to influence public opinion and manipulate perceptions by spreading biased or misleading information. It employs a range of persuasive techniques to shape attitudes and beliefs, typically to benefit a particular agenda or cause. In today’s digital age, the dissemination of propaganda has become more prevalent, particularly on social media and other online platforms. Detecting and analyzing propaganda in text is essential for mitigating its impact and promoting informed decision-making among the public.
This paper presents our approach to the Unimodal (Text) Propagandistic Technique Detection Task at ArAIEval 2024 (Hasanain et al., 2024b). Our approach leverages a multilingual
BERT model (Devlin et al., 2018), known for capturing rich contextual information across multiple languages. The motivation for exploring mBERT was to examine the capabilities of a multilingual model for this task on Arabic. Inspired by the findings in (Liu et al., 2019) we experimented with both the multilingual capabilities of the model and the use of outputs from different hidden layers to explore their effectiveness for identifying propaganda techniques in Arabic text. To assess mBERT’s strengths and weaknesses, we compared it to AraBERT. While our experiments revealed that AraBERT achieved superior overall performance compared to mBERT, interestingly, mBERT’s earlier hidden layers exhibited significantly better performance than its later layers. Note that AraBERT was trained after the evaluation cycle, thus, only mBERT was submitted for the leaderboard.

heading: 2 Related Work
text: Propaganda detection has seen significant advancements through various shared tasks and studies focusing on fine-tuning transformer models and exploring multilingual approaches. The WANLP 2022 shared task (Alam et al., 2022), which centered on detecting propaganda techniques in Arabic tweets, had top-performing submissions predominantly using fine-tuned transformer models like AraBERT and MARBERT (Abdul-Mageed et al., 2021). The NGU_CNLP (Hussein et al., 2022) team achieved first place with an ensemble of AraBERT models combined with data augmentation techniques. Meanwhile, the IITD (Mittal and Nakov, 2022) team employed the multilingual XLM-R (Conneau et al., 2019) model but did not surpass the performance of the specialized monolingual models.
Similarly, in the ArAIEval 2023 shared task (Hasanain et al., 2023b), most submissions focused on fine-tuning advanced transformer models,
478
with common approaches including AraBERT and MARBERT. Many teams, such as HTE (Hadjer and Bouklouha, 2023), utilized multitask learning. Preprocessing techniques and data augmentation were also widely used to improve model performance. Some teams employed multilingual models in their systems. For example, the Legend team (Ojo et al., 2023) implemented XLM-RoBERTa, addressing class imbalance through weighted learning and dynamic learning rate adjustment, and the ReDASPersuasion (Qachfar and Verma, 2023) team combined a multilingual transformer model with a feature engineering module to extract language-agnostic features for persuasion detection. Despite these efforts, the performance of multilingual models like XLM-RoBERTa was generally moderate compared to top-performing systems that used monolingual models specifically fine-tuned for Arabic. These consistent results across both tasks highlight the effectiveness of monolingual models for detecting propaganda techniques in Arabic text, while multilingual models show promise but require further refinement to match their performance.
Recent work on propaganda detection also includes studies on the capabilities of large language models. One such study (Hasanain et al., 2023a) investigated GPT-4’s (OpenAI, 2024) ability to annotate propaganda spans in text, demonstrating its potential to act as both a general and expert annotator, with improved performance when given more information. Building on this, a more recent study (Hasanain et al., 2024a) introduced the ArPro dataset, the largest dataset for fine-grained propaganda detection in Arabic, annotated for 23 techniques.
Additionally, the ArMeme dataset (Alam et al., 2024) was introduced to explore propagandistic content in Arabic memes. This dataset focuses on visual and textual propaganda techniques within memes, providing a unique resource for understanding how propaganda operates in multimodal formats. This work adds to the growing body of research emphasizing the need for diverse datasets and advanced methodologies to tackle propaganda across different media and languages.

heading: 3 Data
text: For this task we have used the data provided for the task (Hasanain et al., 2024b), collected from tweets and news articles. The organizers provided the data in jsonlines format and already split into train, test
and validation sets. There are almost 7000 samples in the training set, with each sample containing the text and the character level span of different propaganda techniques used in the text. It is a multi-label, multi-class dataset. Table 1 presents the number of document, tokens, the average number of tokens per document and the number of unique tokens in a corpus for training, validation and testing sets. There are 23 propaganda techniques found in the dataset with uneven distribution for each corpus. However, the training and validation distribution is similar for each label.

heading: 4 Task Overview
text: ArAIEval shared Task 1 (Hasanain et al., 2024b) involves identifying various propaganda techniques in Arabic text and pinpointing the exact character level spans where these techniques occur. Arabic, a low-resource language for this task, presents unique challenges for natural language processing (NLP) due to its rich morphology and syntactic complexity. Traditional NLP techniques often struggle with the diverse expressions and structures found in Arabic text.
The formal definition for the task is as follows: given an input sequence X of length N , denoted as X = x1, x2, . . . , xN , generate an output sequence Y of length 23, with one element for each class. Each element in the output sequence, yc, is an N - dimensional vector where each element is either 1 or 0. Consecutive 1’s in yc define the span for that class.

heading: 5 Our Methodology
text: We approached the sequence tagging task as a multi-label, multi-class classification problem. The classes correspond to the original 23 classes from the dataset.
For tokenization, we used the pre-trained BERT subword tokenizer from Huggingface.1 Since the subword tokenizer produces more tokens than words in the document, we trained and inferred on the first 256 tokens. We truncated or padded the input sequence to be 256 tokens, ensuring a uniform input length for the model.
The formal definition for the task is as follows: Given an input sequence X of length N , denoted as X = x1, x2, . . . , xN , generate an output sequence Y of length 23, with one element for each class.
1https://huggingface.co/docs/transformers/en/ model_doc/bert#transformers.BertTokenizer
Each element in the output sequence, yc, is an N - dimensional vector where each element is either 1 or 0. Consecutive 1’s in yc define the span for that class.
We focused on fine-tuning pre-trained BERT models (Devlin et al., 2018). Specifically, we ran our experiments on the base version of Multilingual BERT Cased with 110M parameters from HuggingFace.2 The training loss function was a modified binary cross-entropy that ignores pad tokens.
We conducted our experiments using PyTorch on a single RTXA6000 GPU.3 The experiments aimed to find the best hyperparameters and the optimal hidden representation from the BERT output for training the classifier layers, inspired by Liu et al. (2019).
We first performed a hyperparameter search using Optuna for 5 epochs on the dev set and then re-trained the system for additional epochs using the optimal hyperparameters found.4 Optuna was configured to use NSGA-II algorithm (Deb et al., 2002). Our best-performing system on the leaderboard was trained on a single GPU for 400 epochs with a batch size of 32, a learning rate of 6.91× 10−05, a weight decay of 0.00358, and 226 warmup steps. We used feature representation from the 10th hidden layer of the BERT model.
We evaluated our system using a modified F1 measure provided by the shared task organisers. This measure accounts for the partial matching of spans between the gold standard and the hypothesis, ensuring a more accurate evaluation of the system’s performance.

heading: 6 Results
text: We evaluated the performance of our system using different hidden layers from mBERT on both the development and test sets, and included results from AraBERTv2 (Antoun et al.) for comparison. The
2https://huggingface.co/docs/transformers/en/ model_doc/bert
3https://github.com/riyadhrazzaq/araieval24 4https://optuna.org
results, detailed in Table 3, show that the 8th layer of mBERT consistently outperformed the higher layers (10th and 12th) in terms of Micro F1, Macro F1, and Precision. This suggests that lower layers may provide better representations for propaganda detection.
Interestingly, the performance of the mBERT layers, especially the 8th layer, is not far behind that of AraBERTv2. This indicates that multilingual models like mBERT have significant potential, particularly when fine-tuning involves selecting the most effective hidden layers. AraBERTv2, while consistently performing better, highlights the benefits of a model specifically fine-tuned for Arabic. While the AraBERT and mBERT’s 8th layer achieved better performances, we only submitted the result from 10th layer (mBERT) to meet the submission deadline.
Both models demonstrated higher recall compared to precision, indicating a tendency to identify many relevant spans but also produce numerous false positives. The performance trends were consistent across both development and test sets, indicating robustness in the observed patterns.
Overall, these results align with the findings of the study by (Liu et al., 2019) on the effectiveness of different hidden layers. Their study also found that exploring hidden layers can significantly enhance model performance. These results suggest that multilingual models like mBERT have substantial potential, and that careful selection and experimentation with hidden layers can yield significant performance improvements.

heading: 7 Discussion
text: We evaluated the model’s performance on the test set by categorizing spans into short (maximum 3 words) and long (more than 3 words) segments. The model performed better on short spans, with a precision, recall, and F1-score of 0.10. In contrast, for long spans, the precision, recall, and F1-score were 0.02. This indicates the model is more effective at identifying shorter spans, likely due to their
simpler structure and fewer contextual dependencies.
We picked a sample from the test set for a qualitative analysis to understand the model’s performance in detail. We found several instances where the model identified propaganda techniques that were not present in the gold labels but seemed logical and potentially correct. In Sample 2a, the phrase " HĀñªÖÏ @ð HAJ ËA¾ B@ É¾Ë ÈñÊmÌ'@ XAm. ' @ " (finding solutions to all issues and obstacles) could be seen as exaggeration, implying a comprehensive and possibly unrealistic promise. Another example in Sample 2b, the word "YJ K A K" (uphold) within the
context of terrorism cases may suggest a bias or specific stance, potentially serving a propagandistic purpose. The model identifies this as loaded language, highlighting a possible implicit bias.
The model also showed segmentation issues, identifying propaganda with larger spans than necessary. For example, in 2b, the reference label marked "H. AëPB @" (terrorism) as Name Calling, while the model predicted "H. AëPB @ AK A
̄" (terrorism cases). Although the model’s prediction is not necessarily incorrect, as it relates to the context, it
doesn’t match the reference label precisely.

heading: 8 Conclusion
text: In this study, we explored the use of a multilingual BERT model to detect propaganda techniques in Arabic text. Our approach involved fine-tuning the BERT model and utilizing representations from different hidden layers to improve detection accuracy. Despite the challenges posed by Arabic’s complex morphology and syntax, our system performed well, securing the second position in the ArAIEval Shared Task. The results indicate that multilingual BERT models, when properly fine-tuned, can effectively identify and locate various propaganda techniques in Arabic texts. Future work could focus on further refining these models and addressing class imbalance to enhance detection performance.

heading: Acknowledgments
text: We acknowledge the assistance of the LT-Bridge Project (GA 952194) and DFKI for the use of their Virtual Laboratory. Also, authors have been supported financially by the EMLCT 5 programme.
5https://mundus-web.coli.uni-saarland.de/

heading: Future Work
text: Future work could focus on further refining these models and addressing class imbalance to enhance detection performance