NLPColab at FigNews 2024 Shared Task: Challenges in Bias and Propaganda Annotation for News Media (2024)

Abstract: In this paper, we present our methodology and findings from participating in the FIGNEWS 2024 shared task on annotating news fragments on the Gaza-Israel war for bias and propaganda detection. The task aimed to refine the FIGNEWS 2024 annotation guidelines and to contribute to the creation of a comprehensive dataset to advance research in this field. Our team employed a multi-faceted approach to ensure high accuracy in data annotations. Our results highlight key challenges in detecting bias and propaganda, such as the need for more comprehensive guidelines. Our team ranked first in all four evaluation tracks for the propaganda annotation subtask. For the bias annotation subtask, the team stood in first place for the Guidelines and IAA tracks, and in second place for the Quantity and Consistency tracks.

heading: 1 Introduction
text: This paper details the FIGNEWS 2024 shared task participation by the NLPColab team, which is a collaborative effort of Fatima Jinnah Women University, Rawalpindi, Pakistan, Beaconhouse National University, Lahore, Pakistan and National University of Science and Technology, Islamabad, Pakistan.
The FIGNEWS 2024 shared task aimed to look into the subtle complexities of bias and propaganda prevalent in news articles, covering multiple languages, in the context of the Gaza-Israel 2023- 2024 war. The primary goal for FIGNEWS 2024 is to create a shared corpus for detailed annotation and to develop annotation guidelines. The initiative also aims to foster collaboration between Natural Language Processing (NLP) researchers. FIGNEWS 2024 describes the task as "a researchoriented datathon with a strong focus on the devel-
opment of improved annotation guidelines for complex opinion data tasks"(Zaghouani et al., 2024).
FIGNEWS curated a corpus comprising news article headlines and advertising posts from Facebook, all of which are collectively termed posts, in English, Arabic, Hebrew, French, and Hindi. The posts were collected from verified accounts of news agencies from around the world, from October 1, 2023 to January 31, 2024. The focus query was "Gaza", and there is a focus on significant moments during this conflict, e.g., the official war declaration, bombings at Al-Shifa Hospital, ceasefire and hostage release.
FIGNEWS released a subset of the full corpus to the shared task teams. Machine translations into English and Arabic were provided. The subset was shared in the form of 15 batches, with 1000 posts per batch.
The first subtask was on Bias Annotation and teams were given 7 labels: Unbiased, Biased against Palestine, Biased against Israel, Biased against both Palestine and Israel, Biased against others, Unclear, Not Applicable. One example was provided for each label (Zaghouani et al., 2024).
The second subtask was on Propaganda Annotation and teams were given 4 labels: Propaganda, Not Propaganda, Unclear, Not Applicable. One example was provided for each label (Zaghouani et al., 2024).
Shared tasks and datathons have become increasingly popular during the last two decades to advance research and development in various fields including NLP. Some examples that are significant in the context of the FIGNEWS shared task include SemEval-2018 Task 1: Affect in Tweets, which annotates the intensity of emotion in tweets using Best-Worst Scaling. This shared task implemented a defined process to identify malicious an-
573
notations. It involved a set of internally annotated tweets, referred to as gold tweets, that were shared for additional annotation, allowing the tracking of annotator performance on these specific tweets (Mohammad et al., 2018). SemEval-2019 Task 4: Hyperpartisan News Detection annotated entire articles using 5 classes (Kiesel et al., 2019). NLP4IF2019 shared task on Fine-Grained Propaganda Detection focused on text fragments instead of entire articles, and is closer in approach to the FIGNEWS task (Da San Martino et al., 2019). SemEval-2020 task 11 on Detection of Persuasion Techniques in News Articles was similar to the FIGNEWS task in that it was focusing on multiple types of annotations including misleading information and propaganda (Da San Martino et al., 2020b). SemEval2021 task 6 on Detection of Persuasion Techniques in Texts and Images focused on annotating memes and was unique in that it took combinations of image and textual information into consideration (Dimitrov et al., 2021). In WANLP-2022 Shared Task on Propaganda Detection in Arabic the task involved identified the type of propaganda technique being used and identified the fragment of the text where it was used (Alam et al., 2022). SemEval-2023 Task 3 look at categories of persuasion in news articles in 9 languages (Piskorski et al., 2023), and they used machine translations, similar to how FIGNEWS did, among other techniques.
Such tasks help create standardized datasets and annotation guidelines. The collaborative annotations lead to enhanced quality and reliability of datasets. They allow for the creation of large, high-quality datasets that might be too resourceintensive for single researchers or small teams to develop independently, and usually, these datasets end up becoming valuable publicly available resources.

heading: 2 Annotation Methodology and Examples
text: For both subtasks, bias and propanda, the team started off with the examples provided by FIGNEWS for each label and proceeded to develop annotation methodologies.

heading: 2.1 Development of Annotation Guidelines
text: Keeping the FIGNEWS label examples as the foundation of the guideline development process, the team conducted a study of various definitions of bias and propaganda, and looked into the different types and detection methods.
After discussions and by team consensus, the following definition of bias was chosen: Bias refers to the treatment of a specific person or group unfairly or with prejudice due to preconceived notions, stereotypes, or other discriminating reasons. It entails treating someone unreasonably because of their traits, including gender, sexual orientation, religion, race, or ethnicity (Kameswari and Mamidi, 2021).
And similarly, the following definition for propaganda was chosen: Propaganda is an expression of a viewpoint or action taken by an individual or group with the intentional goal of influencing the viewpoints or behaviors of other individuals or groups about predefined aims (Da San Martino et al., 2020a).
After agreeing on the definitions, to get further depth in their ability to detect bias and propaganda in text, the team selected a categorization of types of bias, and a categorization of types of propaganda. These are listed in Tables A1 and A2 in Appendix A.
Bias subtask labels interpretation: The team then developed an interpretation of the 7 FIGNEWS bias subtask labels, based on the examples provided by FIGNEWS and the research by the NLPColab team.
Unbiased: Any post that was a statement of fact(s) was categorized as Unbiased. An example is given in Table 1. Ideally, such posts were verified using a curated set of reliable news sources. In case there was difficulty in making a decision, a checkpoint question was included for annotators to see if the point of view of both parties was included in the post, and that one wasn’t unfairly represented compared to the other. Another checkpoint was included to make sure that any posts that were reporting biased statements as quotes should be labeled as unbiased, because they were objectively only reporting what someone had stated. These checkpoints were not treated as hard and fast rules, but were listed as questions annotators should consider before making an annotation decision. Biased against Palestine (BA Pal): Any post that used negative or loaded language for Palestine or Hamas, or attempted to evoke sympathetic emotions for Israel was labeled as Biased againt Palestine. Table 1 shows an example with relevant words for both sides highlighted.
Biased against Israel (BA Is): In symmetry to the earlier label, posts that used highly negative and loaded language for Israel, or attempted to evoke sympathetic emotions for Palestinians were labeled as Biased against Israel. Table 1 gives a sample post with words that triggered the decision highlighted. Biased against both Palestine and Israel: This included posts in which loaded language was used against both Palestine and Israel. Table 1, shows an example where the conflict is described using sensational language and is not fair to either side. Biased against others: Any post that used negative or loaded language against any other party apart from the main parties involved in the conflict was labeled Biased against others. Table 1 shows an example. Unclear: This label was reserved for posts that were incomprehensible and did not have enough context to allow further research. Table 1 shows an example. Not Applicable: This label was used for any posts that were not related to the Israel-Palestine conflict. Table 1 shows an example.
Propaganda subtask labels interpretation: The team developed an interpretation of the 4 FIGNEWS propaganda subtask labels, based on the examples provided by FIGNEWS and the research by the NLPColab team.
Not Propaganda refers to the communication of
information or ideas across various contexts without the deliberate intention of influencing beliefs, attitudes, or behaviors towards a specific agenda or ideology. Table 2 gives an example post.
Propaganda As per the guideline, propaganda was viewed as “deliberate dissemination of information or ideas to influence a target audience’s beliefs, attitudes, or behaviors, often for political, ideological, or commercial purposes; it involves presenting partially accurate and partially inaccurate information, aiming to elicit emotional responses and change public opinions.” Table 2 shows an example.
Unclear Posts that are not clearly falling under propaganda or not propaganda labels are tagged as Unclear. Table 2 shows an annotated example.
Not Applicable Posts that are not related to the Israel-Palestine conflict were labeled as Not Applicable. Table 2 shows an example.

heading: 2.2 Data Annotation Process
text: After compiling the guideline detailed in the last section, post batches shared by FIGNEWS were divided among the team members. Each team labeled each post assigned to them with the 7 labels of the Bias subtask and the 4 labels of the Propaganda subtask.
First some initial labeling was conducted, followed by a meeting to discuss challenging cases.Guidelines were refined as a result of this meeting. Regular checks were implemented to maintain annotation consistency and accuracy and periodic reviews of annotated data were conducted
to identify and address discrepancies. Throughout the process, similar additional meetings were held to ensure alignment and to address new issues. Annotators and trainers were also part of a dedicated instant messaging group, facilitating quick interactions and feedback. This platform allowed immediate resolution of queries.
The team also came to a consensus on a set of news sources to be consulted to establish and confirm facts where needed. This list of news sources notably included Al-Jazeera, The Guardian, Deutsche Welle (DW), NHK World, The New York Times and Reuters. A reliable Pakistani source, Dawn, was also included in this list. It was recommended to check 2 sources from this list every time fact checking was needed. It is to be noted though that this was a very challenging part of the process and needs improvement.
The IAA subset, provided by FIGNEWS was to be annotated by every member of the team. Most of the other posts were also annotated by multiple members to ensure consistency and quality.

heading: 2.3 Inter-Annotator Agreement (IAA) Analysis
text: As recommended by FIGNEWS, Cohen’s Kappa score was used to determine the level of agreement between annotators and as a measure of reliability of the annotations.
Table 3 shows the NLPColab team scores for individual batches with an overall average of 0.775 for the Bias subtask and 0.679 for the Propaganda subtask, which indicate a fairly healthy level of agreement among annotators.

heading: 3 Team Composition and Training
text: The NLPColab team operated in multiple groups to complete the annotations. The annotation team
included female Bachelors (18-24 yrs) and M.Phil level(25-34 yrs) students of Computer Science as shown in Table 3. The native language for all participants was Urdu, and language medium of education was English. All of the annotations of posts in languages other than English were done using the machine translated English text. All participants were of South Asian origin from Pakistan.
The team also included senior members from Computer Science, Natural Language Processing, and Media and Communication Science backgrounds. They played a key role in training the annotation team and conducting reviews. This interdisciplinary composition was a key strength for the team and enabled a research based and scientific approach to the annotation tasks.
Each annotation team was composed of four members and completed two batches, including the IAA sets as per the FIGNEWS guideline.
A comprehensive training was provided to annotators to ensure a thorough understanding of
the guidelines and annotation process. Open discussions and consensus-building were encouraged among annotators to resolve disagreements and clarify ambiguous cases. Regular feedback was incorporated iteratively to improve the guidelines and streamline the annotation workflow.

heading: 4 Task Participation and Results
text: Evaluation for each task was conducted on four tracks: Guidelines, Inter-Annotator Agreement (IAA) Quality, Quantity, Consistency (Zaghouani et al., 2024)
A total of 17 teams, including around 140 overall members from across the globe participated in the FIGNEWS shared task. The NLPColab team participated in both bias annotation and propaganda subtasks and successfully annotated all 15 batches for both. IAA analysis was conducted on 200 posts as per the FIGNEWS guideline.
The NLPColab team stood in first place for all tracks of the Propaganda Subtask. For Bias, the team stood in first place for the Guidelines and IAA tracks, and in second place for the Quantity and Consistency tracks.

heading: 5 Discussion
text: It was apparent during early discussions that annotators may have varying perceptions of bias and propaganda. To mitigate this, the team implemented detailed guidelines and regular training sessions. Annotators engaged in regular feedback and review meetings to ensure a common understanding of and consistent application of the guidelines.
It also became clear as the task unfolded that detecting subtle forms of bias and propaganda was particularly challenging, as it requires linguistic proficiency, a deep awareness of socio-political contexts and the ability to discern underlying agendas. During reviews, the team detected several instances of subtle cues that were overlooked.
Additionally, subtle bias and propaganda can vary significantly across different cultures, further complicating the annotation process.
In the context of these issues, a significant strength of our approach was the multidisciplinary composition of our team, which included members from media and communication sciences as well as computer science (specifically NLP students). This diversity brought a broad range of perspectives and expertise to the annotation pro-
cess, encouraging richer discussions and a more comprehensive understanding of what constitutes bias and propaganda. The collaboration between these disciplines allowed us to leverage the theoretical and practical insights from media studies alongside the technical rigor of NLP, enhancing the overall quality and depth of our annotations.
One interesting question raised during review meetings was whether the translation of the posts amplified or de-emphasized the presence of bias and propaganda. However, it is important to note that FIGNEWS considers machine translations to be valid stand-ins for the annotation tasks, reflecting the reality that many people rely on such technologies (Zaghouani et al., 2024).
Given the sensitive and polarizing nature of the Palestine-Israel conflict, annotators engaged in continuous self-reflection and employed strategies to remain as objective as possible. This conscious effort to recognize and mitigate personal biases was essential to ensuring the integrity and reliability of the annotated data, fostering a more balanced and nuanced interpretation of the news items.
In this context, it is to be noted that the FIGNEWS guideline referred to the area of focus as both "war" and "conflict", and this paper consequently uses both terms as the result of a conscious decision.
Our experience highlights the importance of continuous education and awareness in annotation tasks, particularly in complex and sensitive domains. Regular training sessions on recognizing and addressing personal biases, along with discussions on ethical considerations, were vital in maintaining high standards of annotation quality and integrity.

heading: 6 Conclusion
text: Our participation in the FIGNEWS 2024 shared task employed a multi-faceted approach including team selection, task division, and data analysis. The use of examples and comparisons with related work significantly influenced our methodology and performance.
By reviewing existing guidelines, we identified gaps and formulated new strategies to improve the annotation process.
Despite the inherent subjectivity in annotating bias and propaganda, our IAA scores demonstrate that our detailed guidelines and multidisciplinary approach helped maintain consistency.
One final recommendation for future work is to leverage technology to automate parts of the annotation process. Steps such as initial tagging or flagging potentially biased content, can increase efficiency and allow annotators to focus on more subjective and complex decisions.

heading: A Appendix: Bias and Propaganda
text: Categories used as Guideline
Propaganda Reason Loaded language
Influencing an audience by employing particular words and phrases that have significant emotional associations, either good or negative (Weston, 2000, p. 6).
Name-calling or labelling
Assigning the propaganda campaign’s objective as something the intended audience either likes, praises, or fears (Miller, 1939).
Repetition Repeating the same point over in the hopes that the listener would ultimately acquire it (Torok, 2015; Miller, 1939). Exaggeration or minimization
Presenting something in an exaggerated way, such as exaggerating its size, quality, or importance, or underplaying its significance (Jowett and O’Donnell, 2012b, pag. 303).
Doubt Doubting the accuracy of something or someone. Appeal to fear/prejudice
Attempting to increase support for a concept by spreading fear and/or concern about a potential alternative among the populace, maybe because of preconceived notions.
Flag-waving Employing strong feelings of patriotism (or regard for any group, such as ethnicity, gender, or political affiliation) to support or legitimise a course of action or concept (Hobbs and Mcgee, 2008). Causal oversimplification
Assuming that a problem has a single source or explanation when there are several.
Slogans A brief and powerful statement that can be stereotypical or labelled. Slogans frequently function as emotive pleas (Dan, 2015). Appeal to authority
Declaring a statement to be genuine in the absence of any additional proof, just because it is endorsed by a reliable source or subject matter expert (Goodwin, 2011).
Black-andwhite fallacy, dictatorship Presenting two options as the only ones available when, in reality, there are more options (Torok, 2015). A dictatorship is an extreme situation in which one person dictates to the audience what steps they should follow while excluding all other options. Thoughtterminating cliches
Words or words that impede serious consideration of a subject and insightful debate. Usually, they are brief, general statements that divert attention from alternative points of view or appear to be straightforward responses to difficult topics (Hunter, 2015, p. 78).
Table A1: Types of propaganda and their reasons taken from (Da San Martino et al., 2020a)
Bias Reason Cultural/Identity Bias
This type of bias encompasses a wide range of preconceptions regarding a politician’s or party’s identity and class based on multiple divides. A few instances include the subtly or overt representation of prejudiced beliefs such as ageism, sexism, racism, casteism, and so on, which can be interpreted in both positive and bad ways to produce bias.
Amplification This phrase describes the introduction of bias when facts regarding events, persons, or parties are wildly inflated or hyped in order to give readers a distorted view. Personal Targeting
This tactic is allegedly employed when someone is criticized or exalted for their character attributes rather than their political achievements.
Repetition Ad nauseam, often known as repetition, is a tactic used to introduce bias into writing by restating a concept, a query, or a catchphrase in an effort to get the reader to accept it as gospel. Appeal to Audience
Appeal to fear, beliefs and prejudices, shared identity, making promises
Intentional Vagueness
This includes purposefully providing the audience with ambiguous information, implicit presumptions, or unsubstantiated claims. This is frequently done to force the viewer to form inferences that they might not have been able to if they had full knowledge of the circumstances.
Oversimplification The purpose of this tactic is to demonstrate how taking a course of action leads to an undesirable end by oversimplifying the ramifications of doing so.
Table A2: Types of Biases and their reasons taken from (Kameswari and Mamidi, 2021)
579

heading: Future Work
text: One final recommendation for future work is to leverage technology to automate parts of the annotation process. Steps such as initial tagging or flagging potentially biased content, can increase efficiency and allow annotators to focus on more subjective and complex decisions