Sina at FigNews 2024: Multilingual Datasets Annotated with Bias and Propaganda (2024)

Abstract: The proliferation of bias and propaganda on social media is an increasingly significant concern, leading to the development of techniques for automatic detection. This article presents a multilingual corpus of 12, 000 Facebook posts fully annotated for bias and propaganda. The corpus was created as part of the FigNews 2024 Shared Task on News Media Narratives for framing the Israeli War on Gaza. It covers various events during the War from October 7, 2023 to January 31, 2024. The corpus comprises 12, 000 posts in five languages (Arabic, Hebrew, English, French, and Hindi), with 2, 400 posts for each language. The annotation process involved 10 graduate students specializing in Law. The Inter-Annotator Agreement (IAA) was used to evaluate the annotations of the corpus, with an average IAA of 80.8% for bias and 70.15% for propaganda annotations. Our team was ranked among the bestperforming teams in both Bias and Propaganda subtasks. The corpus is open-source and available at https://sina.birzeit.edu/fada

heading: 1 Introduction
text: Since October 7, social media has been flooded with posts, articles, images, and videos related to the Israeli War on Gaza. Such posts are often divided by hate, bias, and fake news either in favor of or against one of the parties or by remaining neutral, see e.g., "Framing the Israeli War on Gaza" is a shared task on news media narratives (Zaghouani et al., 2024), which is part of the 2nd ArabicNLP conference. The task aims to create a multilingual corpus that unravels the layers of bias and propaganda within news articles in various languages.
Such shared tasks and datathons are crucial in the NLP community to foster collaboration and advance research in specific areas. Previous efforts, such as SemEval-2020 Task 11 (Martino et al.,
2020) and TSHP-17 (Rashkin et al., 2017) have provided valuable resources for propaganda detection in news articles. The dual focus of FigNews on bias and propaganda is a novel approach that addresses the evolving nature of misinformation on social media platforms. The detection of propaganda on social media is crucial (Darwish et al., 2021), as it can polarize public sentiment, foster violent extremism and hate speech, and eventually erode democracies and diminish trust in democratic procedures (Abuaiadah et al., 2017). Notably, only a few corpora have been recently built to address these issues. Recent work by (Hamad et al., 2023) involved establishing a Hebrew dataset comprising 15, 881 tweets for detecting offensive language. This dataset was manually annotated with four labels: hate, abusive, violence, and pornographic. Their work focused on detecting hate speech in Hebrew tweets and implemented in SinaTools (Hammouda et al., 2024). Additionally, the WojoodNER Shared Task 2024 offered a new NER dataset related to the Israeli War on Gaza called WojoodGaza (Jarrar et al., 2024). Other notable works include TSHP-17 (Rashkin et al., 2017), QProp (BarrónCedeno et al., 2019), and PTC (Da San Martino et al., 2019). TSHP-17 and QProp are documentlevel corpora, while PTC is a sentence-level corpus. While SemEval-2020 (Martino et al., 2020) Task 11 is similar to FigNews (Zaghouani et al., 2024) in its objective, they differ in their data sources and focus areas.
This paper describes our participation in the FigNews. Our contributions are:
• Annotated Corpus (12K FB posts) for bias and propaganda, in 5 languages.
• Annotation guidelines ensuring consistency and accuracy.
Remark: The corpus presented in this article
640
does not cover the genocide, ethnic cleansing, or starvation events as they mostly happened after collecting the corpus.
The article is organized as follows: Section 2 describes the methodology, 3 presents our team composition and training; Section 3 presents our participation and results; Section 5 analyzes some errors, and Section 6 concludes the paper.

heading: 2 Annotation Methodology
text: The objective of the task is to address the complex landscape of social media discourse related to the Israeli War on Gaza 2023-2024. The task organizers provided participants with 15k posts from verified Facebook accounts, selected between October 6, 2023, and January 31, 2024, using "Gaza" as a query keyword across 5 languages: Arabic, Hebrew, English, French, and Hindi. The dataset consists of 15 batches, each containing 1000 posts.

heading: 2.1 Annotation Guidelines
text: Our understanding of "bias" is based on the work done by the United Nations Committee on the Elimination of Racial Discrimination and the European Commission against Racism and Intolerance (European External Action Service, n.d.). We define the notations ‘bias’ and ‘propaganda’ based on the UN and EU accounts, as:
Bias: is generally understood as an inclination or prejudice towards or against a particular person or group, often in a way considered to be unfair. In other words, it is an unreasonable preference or dislike that prompts someone to behave in a discriminatory way, often based on unfair judgment. This bias is typically based on prohibited grounds of discrimination such as race, religion, language, nationality, ethnicity, social background, gender, and others.
Classifications of Bias: we adopted the same classes provided in the Shared Task: (1) Biased against Palestine,(2) Biased against Israel, (3) Biased against others, (4) Biased against both Israel and Palestine, (5) Not Applicable, (6) Unclear, and (7) Unbiased. We also introduced a new feature called "Type of Bias", which can be either: (a) Explicit ( l ' Qå Q m ') if it is obvious and evident in the post, (b) Implicit (ú æÖÞ Q m ') if it is clear but not evident in the post, and (c) V ague (ÑîD.Ó Q m ') in case of indirect and ambiguous bias. This feature is important from a methodological viewpoint as it encourages the annotators to think more during
classification. If a post contains biased content but not in a direct way it can be accounted as implicit. Propaganda: misleading ideas or statements that can distort the truth or omit facts to promote a specific political or social agenda. These ideas are typically published by media outlets. For example, propaganda can take the forms of exaggeration, minimization, spreading doubts, name-calling, labeling, or intentional vagueness. All these forms have the common intention to spread false information and obscure facts.
Classifications of Propaganda: We adopted the four classes provided in the Shared Task: (i) Propaganda, (ii) Not propaganda, (iii) Not Applicable, and (iv) Unclear.
Additionally, we added a new column to classify Propaganda into three types: (1) Propaganda must be deleted: if it contains evident harmful content that poses risks to the safety and security of individuals or groups; (2) Propaganda may be deleted: if we cannot easily judge whether it is propaganda, depending on a specific context; and (3) Propaganda not to be deleted: if it is not clear and lacks harmful consequences and therefore does not warrant deletion.
Remark: Since the data was collected from Facebook posts some cases contain quoted content (e.g. an unbiased post quoting biased content). It was established in the guidelines that a post should not be classified as bias or propaganda based on its quotation, but rather on the post itself.
An Example of the guidelines mentioned earlier regarding quoted content is as follows: “Hamas and Islamic Jihad spare no effort to exploit religious institutions for terrorist purposes,” the IDF said in a statement. This post is annotated as unbiased because it is a direct quote and does not include any additional commentary or interpretation.

heading: 2.2 Inter-Annotator Agreement (IAA)
text: To evaluate the quality of our annotations, we used the F1-score and Cohen’s Kappa (Cohen, 1968) to compute the agreement between the annotators. The results are shown in Table 1.
The task organizers allocated 100 posts (10%) from each batch for IAA, including 20 posts randomly selected from each language. Overall, we annotated 12, 000 posts, resulting in an IAA dataset of 1, 200 posts. These were distributed among our 10 annotators following this scheme: (1) each annotator received 240 posts, (2) each post was anno-
tated by two different annotators, and (3) the 240 posts assigned to each annotator were distributed among four other annotators. Consequently, each pair of annotators had 60 posts in common.
All vs. Binary IAA: to evaluate whether a (dis)agreement was dominated by a certain class, we mapped all labels into binary categories: (Bias or NotBias and others) and (Propaganda or NotPropaganda and others). Table 1 demonstrates no class dominance because All and Binary evaluations are close to each other.
Looking at all Cohen’s scores in Table 1, the average is 0.808 for bias, which is a "very good" agreement, and 0.7015 for propaganda, which is a "good" agreement overall. Agreement on propaganda was more challenging but the results are enhanced when it is considered as a binary.

heading: 3 Team Composition and Training
text: Team composition: We assembled a team of 10 Master’s students specializing in Law at Birzeit University, comprising 7 females and 3 males. All team members are native Arabic speakers with a good command of English.
Training phase: We began by selecting 200 posts to train all students in annotation. After training, each student was assigned 1, 200 posts for annotation.
Ensuring consistency We held three workshops to ensure consistency to discuss guidelines, address
challenges, and resolve disparities.The first workshop involved an expert who reviewed the annotations and added comments for the annotators to address. In the second workshop, the annotators met with the expert to discuss his comments on the posts. In the final workshop, after reviewing their annotations compared to the expert’s, they discussed the points of agreement and disagreement with him.

heading: 3.1 Annotation process
text: Annotation Phase: The dataset consisted of 12 batches, comprising 10, 800 posts from the Main sheet, and 1200 posts from the IAA sheet. The annotation was carried out in two phases:
1. Phase One: We distributed Batch01 and Batch02, each with 180 posts, among team members. To ensure consistency with the guidelines, an expert reviewed all student annotations for these batches and provided feedback.
2. Phase Two: we assigned each annotator 450 posts from two different batches. This step allowed us to complete the annotation of all 12 batches (i.e. 12k posts).
Set quality standards
To set quality standards among annotators, after the annotation process was complete, each pair of annotators who had annotated the same
data held meetings to review the selected posts they disagreed on. They discussed their differences, and if they reached an agreement, they would change the label accordingly. If they could not agree, they kept the original label.

heading: 4 Task Participation and Results

heading: 4.1 Results
text: Table 2 displays the final results provided by the shared task organizers. Our Sina team achieved the third and second place in the IAA Quality and Quantity tracks for the Bias and Propaganda subtasks, respectively. In addition to third place in Propaganda Guidelines.
Table 3 and Table 4 illustrate the distribution of the bias classes and types of bias across languages respectively. Table 3 shows that about 27% of the posts are biased against Palestine and 63% of the posts are unbiased. Most of the bias against Palestine originated from French posts. Table 4 gives more statistics about the types of bias. As shown in this table, most of the posts annotated as Explicit bias are in Hebrew.
For propaganda results, Table 5 illustrates the distributions of propaganda classes across languages, which shows that 31% of the posts (3333) are annotated as "Propaganda", and 66% (7084) are "Not Propaganda". The majority of the propaganda originated from French posts. Table 6 illustrates the distribution of the type of propaganda classes among languages. As shown in the table posts that were classified as propaganda must be deleted were in French with 348 posts.

heading: 5 Error Analysis and Discussion
text: Despite training and supervision, errors may arise from subjective interpretation, ambiguous guidelines, or complex content. We explored the errors
and noted:
1. False positives in bias annotations occurred when annotators marked neutral content as biased. For instance, the post: "Israel launched attacks on Syria on Nov 10 in response to a drone strike on Eilat. The IDF claimed it
attacked an organization responsible for the drone. Watch for more details." This news excerpt is informative and not biased.
2. Misclassification of propaganda: Some content was wrongly labeled as "must be deleted" propaganda despite lacking direct harmful implications. For example: "BREAKING: Israeli forces are causing massive destruction in Gaza, in response to a terrorist attack by Hamas. Image source: Middle East Eye post." While it is propaganda, it shouldn’t be classified as "must be deleted."

heading: 6 Conclusion
text: This article presents our contribution to the FigNews 2024, where we annotated a multilingual corpus of 12, 000 Facebook posts for bias and propaganda across five languages. We extended the annotation guidelines for better consistency and accuracy, providing a foundation for future work in detecting bias in social media. Our plans include expanding the corpus to cover more critical events of the war and leveraging neural and large Language models to automatically detect bias and propaganda on social media posts.
Ethical Considerations
Given the sensitive nature of the topics and media narratives related to the Israel War on Gaza, our annotators, who are lawyers, have undergone extensive training to ensure careful and fair judgments. They meticulously review both Arabic and English translations to avoid any bias that might arise from machine translation.
Limitations
We recognize the limitations in our annotation process. This is because of the subjective nature of identifying bias and propaganda in social media posts, and the sensitivity of the datasets involved.

heading: Acknowledgments
text: We would like to acknowledge the contributions of Nejira Softic during the formulation of the guidelines. We would like to also thank the Master’s students specializing in Law and IT at Birzeit University for their help in annotating the datasets, especially Maram Shour, Belal Abu Zaina, Bayan Abu Alawi, Zainah Abughosh, Aya Al Dimasy, Doaa Abozena, Waad Alsheikh, Qassam Abu Hakmeh,
Aseel Mustafa, Basel Awwad, Omar To’Mallah, and Dyala Fakhouri, and as well as Prof. Reem Al-Botmeh for her support during the course. Also thanks to Palestine Technical University - Kadoorie for its support.

heading: Future Work
text: We extended the annotation guidelines for better consistency and accuracy, providing a foundation for future work in detecting bias in social media. Our plans include expanding the corpus to cover more critical events of the war and leveraging neural and large Language models to automatically detect bias and propaganda on social media posts. Ethical Considerations
Given the sensitive nature of the topics and media narratives related to the Israel War on Gaza, our annotators, who are lawyers, have undergone extensive training to ensure careful and fair judgments. They meticulously review both Arabic and English translations to avoid any bias that might arise from machine translation. Limitations
We recognize the limitations in our annotation process. This is because of the subjective nature of identifying bias and propaganda in social media posts, and the sensitivity of the datasets involved