JusticeLeague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation (2024)

Abstract: In response to the evolving media representation of the Gaza-Israel conflict, this study aims to categorize news articles based on their bias towards specific entities. Our primary objective is to annotate news articles with labels that indicate their bias: "Unbiased", "Biased against Palestine", "Biased against Israel", "Biased against both Palestine and Israel", "Biased against others", "Unclear", or "Not Applicable". The methodology involves a detailed annotation process where each article is carefully reviewed and labeled according to predefined guidelines. For instance, an article reporting factual events without derogatory language is labeled as "Unbiased", while one using inflammatory language against Palestinians is marked as "Biased against Palestine". Key findings include the identification of various degrees of bias in news articles, highlighting the importance of critical analysis in media consumption. This research contributes to the broader effort of understanding media bias and promoting unbiased journalism. Tools such as Google Drive and Google Sheets facilitated the annotation process, enabling efficient collaboration and data management among the annotators. Our work also includes comprehensive guidelines and examples to ensure consistent annotation, enhancing the reliability of the data.

text: Proceedings of The Second Arabic Natural Language Processing Conference, pages 651–655 August 16, 2024 ©2024 Association for Computational Linguistics
The methodology involves a detailed annotation process where each article is carefully reviewed and labeled according to predefined guidelines. For instance, an article reporting factual events without derogatory language is labeled as "Unbiased", while one using inflammatory language against Palestinians is marked as "Biased against Palestine".
Key findings include the identification of various degrees of bias in news articles, highlighting the importance of critical analysis in media consumption. This research contributes to the broader effort of understanding media bias and promoting unbiased journalism. Tools such as Google Drive and Google Sheets facilitated the annotation process, enabling efficient collaboration and data management among the annotators.
Our work also includes comprehensive guidelines and examples to ensure consistent annotation, enhancing the reliability of the data.

heading: 1 Introduction
text: In the realm of Natural Language Processing (NLP), the annotation of datasets is a critical step in developing models that can understand and interpret human language accurately. Several key studies have laid the groundwork for our approach. For instance, (2) Basile et al. (2019) explored the detection of hate speech against immigrants and women in tweets, providing insights into the nuances of
hate speech and its implications. Similarly, (3) Mohammad et al. (2016) focused on detecting stance in tweets, which involves understanding whether a tweet is in favor, against, or neutral towards a given target, adding another layer of complexity to bias detection. The JusticeLeague team at FIGNEWS 2024 Shared Task has taken on the challenge of annotating news articles to categorize their bias towards specific entities involved in the Gaza-Israel conflict. This shared task is of paramount importance as it aims to shed light on the often subtle and nuanced biases present in media reporting, which can significantly influence public opinion and policy decisions.
The title of our paper, "JusticeLeague at FIGNEWS 2024 Shared Task: Innovations in Bias Annotation," reflects our commitment to adhering to the rigorous standards set forth by the shared task organizers. Our objectives align closely with the goals of the task, which seek to enhance the understanding of media bias and contribute to the development of unbiased journalism.
In the broader context of the NLP community, shared tasks and datathons have become instrumental in advancing research by providing standardized datasets and guidelines. For instance, previous efforts such as the SemEval tasks have focused on various aspects of semantic analysis, while other initiatives have tackled sentiment analysis and hate speech detection. These endeavors have laid the groundwork for our current task, which specifically targets the identification of bias in news articles.
Our approach to the shared task diverges from previous efforts in several ways. (7) FranciscoJavier Rodrigo-Ginés et al. (2024) provide a systematic review of media bias detection methods, which has informed our categorization and analysis techniques. Firstly, we have developed a comprehensive set of annotation guidelines that not only identify bias but also categorize it into specific entities, such as "Biased against Palestine" or "Biased
651
against Israel." This level of granularity allows for a deeper analysis of the nature and extent of bias in media reporting. Secondly, our use of collaborative tools like Google Drive and Google Sheets has streamlined the annotation process, ensuring efficient data management and facilitating real-time communication among annotators.
To illustrate the effectiveness of our methodology, consider the following examples:
- An article labeled as "Unbiased" (See Example: A.1) presents factual information without any derogatory language or negative portrayals, demonstrating a neutral stance. - Conversely, an article labeled as "Biased against Palestine" (See Example: A.2) employs inflammatory language and subtly negative portrayals of Palestinians, indicating a clear bias.
These examples underscore the importance of our task and the need for a systematic approach to identifying and categorizing bias in news articles. The psychological aspects of language use, as detailed by (8) Pennebaker et al. (2003), also play a significant role in our annotation process. By contributing to the body of research on media bias, our work aims to promote transparency and accountability in journalism, ultimately fostering a more informed and discerning public.

heading: 2 Annotation Methodology and Examples

heading: 2.1 Development of Annotation Guidelines
text: The creation of robust annotation guidelines was a pivotal step in ensuring the accuracy and consistency of our bias categorization. Our team engaged in a meticulous process of defining each label, ensuring that it encapsulated the nuances of bias as it pertains to the Gaza-Israel conflict. We began by conducting a thorough literature review on media bias, focusing on studies that highlighted the linguistic markers of bias in news reporting. This foundational research informed the initial draft of our guidelines.
To refine these guidelines, we incorporated labelspecific examples for each category. These examples served as concrete instances that illustrated the application of each label. For instance, we provided an example of an article that would be labeled "Biased against Palestine" (See Example: A.2), detailing the use of derogatory language and negative portrayals that would warrant such a classification. Similarly, we offered examples for "Biased against Israel" (See Example: A.3), "Unbiased" (See Ex-
ample: A.1), and other categories, ensuring that each label was clearly defined and understood by all annotators.

heading: 2.2 Data Annotation Process
text: The annotation process was conducted in a structured and systematic manner. (5) Wijekoon et al. (2022) emphasize the role of crowdsourcing and online participation as effective strategies for mitigating bias, which we incorporated into our methodology. Each annotator was provided with the developed guidelines and a set of news articles to review. The guidelines, bolstered by the label-specific examples, served as a reference point for annotators as they assessed each article. The examples were particularly crucial in cases where the bias was subtle or the article contained mixed signals, guiding annotators to make informed decisions based on the predefined criteria.
To maintain consistency, annotators were encouraged to refer back to the examples whenever they encountered ambiguity. This iterative process of annotation, review, and reference to the guidelines and examples ensured that the labels assigned to the articles were standardized across the dataset.

heading: 2.3 Inter-Annotator Agreement (IAA) Analysis
text: To validate the reliability of our annotation process, we conducted an Inter-Annotator Agreement analysis. This involved having multiple annotators independently label the same set of articles and then comparing their labels to assess the level of agreement. The use of examples played a significant role in achieving high levels of agreement among annotators. Since we did not employ a quantitative measure such as Cohen’s Kappa coefficient, our approach was more qualitative and involved a manual review of the annotations.
The process began with multiple annotators independently labeling a subset of the articles. Once each annotator had completed their labeling, the team convened to compare and discuss the labels assigned by each annotator to the same articles. This discussion was facilitated by the examples provided in the annotation guidelines, which served as a reference point for understanding the rationale behind each label.
This qualitative IAA analysis confirmed the effectiveness of our guidelines and the clarity provided by the examples. It also highlighted the reliability of our bias categorization, as the annota-
tors were able to apply the guidelines to the news articles consistently. The agreement among annotators, achieved through this manual review process, underscored the robustness of our annotation methodology and the trustworthiness of the resulting dataset.

heading: 3 Team Composition and Training
text: Our annotation team consisted of three computer science students from Helwan University in Cairo, Egypt, each bringing a unique perspective to the task despite our shared academic background. Leveraging our technical expertise, we approached the annotation of news articles with a methodical and analytical mindset, focusing on the systematic application of the bias categorization guidelines.
Before the annotation process, we conducted a focused training session to ensure a cohesive understanding of the guidelines and the nuances of each bias label. We emphasized the use of examples to clarify the criteria for each category, which was crucial for maintaining consistency in our annotations. The training also included discussions on the ethical considerations of bias annotation, reinforcing the importance of impartiality and accuracy in our work.
To facilitate effective team coordination, we utilized digital tools such as Google Drive and Google Sheets for efficient data management and annotation tracking. A dedicated chat group was established for immediate communication and to resolve any annotation discrepancies. Regular team meetings were held to discuss progress, address challenges, and refine our annotation approach, ensuring the reliability and precision of our bias categorization efforts.

heading: 4 Task Participation and Results
text: The JusticeLeague team’s participation in the FIGNEWS 2024 Shared Task was characterized by a meticulous and informed approach to the annotation of news articles for bias. As computer science students from Helwan University, we leveraged our technical acumen to systematically apply the bias categorization guidelines, with a particular emphasis on the inclusion of examples in our annotation process. These examples served as critical benchmarks, guiding our team in maintaining a high degree of consistency and accuracy in our labeling decisions. Our comparative analysis with related work within the NLP community further
refined our strategy, allowing us to anticipate challenges and adapt our methodology to ensure reliable and consistent annotations. Despite being our first annotation task, the outcomes of our participation demonstrated a strong alignment with the task’s objectives, underscoring the effectiveness of our approach and the importance of well-defined examples in achieving high-quality bias categorizations.

heading: 5 Discussion
text: Our team’s participation in the FIGNEWS 2024 Shared Task, despite being our first annotation endeavor, has yielded significant insights and contributions to the field of media bias annotation. The nuanced landscape of bias identified in the news articles we annotated, ranging from overt to subtle forms, aligns with the complexities previously documented in related work. For example, (7) Francisco-Javier Rodrigo-Ginés et al. (2024) provide a detailed review of media bias detection techniques, which has informed our approach. The nuanced landscape of bias identified in the news articles we annotated, ranging from overt to subtle forms, aligns with the complexities previously documented in related work. The pivotal role of including detailed examples in our annotation guidelines, which facilitated standardized and accurate bias categorization, underscores a best practice that can inform future tasks. Our findings have profound implications for media literacy, critical news consumption, and the ethical development of journalism practices, as they provide a framework for discerning bias and promoting objective reporting. Moreover, our comparative analysis with related work has refined our methodology, highlighting the value of learning from past efforts. The dataset, annotation guidelines, and examples we have generated are substantial contributions to the NLP community, offering empirical evidence and resources for advancing research in bias detection algorithms. In essence, our work has not only enriched the understanding of media bias but has also set a precedent for rigorous and informed annotation practices in this vital area of study.

heading: 6 Conclusion
text: Our participation in the FIGNEWS 2024 Shared Task has yielded significant insights and results that underscore the effectiveness of our approach to bias annotation in news articles. As a team of three
computer science students from Helwan University, we brought a unique perspective to this challenging task, combining our technical expertise with a nuanced understanding of media bias. Our contribution of 2,400 annotated data points across two main batches and six Inter-Annotator Agreement (IAA) batches represents a substantial input to the task dataset. This volume of work allowed us to gain deep insights into the nuances of bias in news reporting across multiple languages and contexts. The cornerstone of our success was our meticulous approach to developing annotation guidelines. By including detailed examples for each bias category, we created a robust framework that served as a clear and consistent reference for our team. This approach was instrumental in achieving our high interannotator agreement scores, culminating in our second-place achievement in the IAA-Quality track. Our impressive IAA metrics (Kappa: 64.4, Accuracy: 83.7%, Macro F1 Average: 63.8, F1 Bias*: 73.6) reflect the consistency and reliability of our annotation process. Our methodology was further strengthened by our thorough integration of related work within the NLP community. By critically examining previous efforts in bias detection and annotation, we were able to anticipate challenges and refine our approach accordingly. This comparative analysis not only enhanced the quality of our annotations but also contributed to the broader discourse on best practices in bias annotation. The official results provided fascinating insights into the distribution of bias across different languages and contexts. The prevalence of unbiased content (42.8% of all annotations) is an encouraging finding, suggesting that a significant portion of news coverage maintains neutrality. However, the high proportion of content biased against Palestine (29.2%), particularly in Hebrew-language sources (10.4%), highlights ongoing challenges in balanced reporting in certain contexts. The varying distributions of bias across different languages underscore the importance of multilingual approaches in bias detection tasks. Our centrality metrics (Kappa: 19.9, Accuracy: 43.3%, Macro F1 Average: 19.6, F1 Bias*: 46.5), while lower than our internal IAA scores, reflect the inherent challenges in achieving consensus across different teams in bias annotation. These results highlight the subjective nature of bias perception and the need for continued refinement of annotation guidelines and methodologies. As we reflect on our contributions, we recognize the broader implications of our work. By developing robust annotation guidelines and demonstrating their effectiveness, we have contributed to the ongoing effort to improve media literacy and promote unbiased journalism. Our dataset and methodologies offer valuable resources for future research in automated bias detection and analysis. Moreover, our experience underscores the importance of interdisciplinary approaches in tackling complex challenges like media bias. As computer science students, we brought technical rigor to the task, but our success was equally dependent on our ability to understand and analyze the nuances of language and media representation. Looking forward, we see several avenues for future research. There is potential for developing more sophisticated machine learning models for automated bias detection, building on the insights gained from our manual annotation process. Additionally, further investigation into the relationship between bias and propaganda, as hinted at by our correlation analysis, could yield valuable insights for combating misinformation. In conclusion, our participation in the FIGNEWS 2024 Shared Task has not only produced valuable data and insights but has also demonstrated the effectiveness of a well-structured, example-driven annotation process. By combining technical expertise with a nuanced understanding of media bias, we have made a meaningful contribution to this critical field of study. As we move forward, we hope that our work will serve as a foundation for continued research and development in the crucial task of identifying and mitigating media bias. hyperref

heading: A News Article Examples

heading: A.1 Unbiased
text: “An Israeli child talks about his experience being held hostage by Hamas and mentions that he was taught some Arabic and how to use prayer beads.”

heading: A.2 Biased against Palestine
text: “LIVE: Just as ISIS is destroyed...Hamas will be buried in the same way! IsraelHamasConflict WarZone”

heading: A.3 Biased against Israel
text: “We mobilize our revolutionary youth throughout the West Bank and Jerusalem and call on them to escalate the struggle and revolution until the Nazi occupation is defeated.”

heading: A.4 Unclear
text: "NEWS ALERT”

heading: A.5 Not Applicable
text: “ Here we go for the explosive 1/4 finals The Super Eagles and the Palancas Negras open the ball. On this occasion, we are offering you 10x20,000 FCFA of freebets to win To participate, you yourself know: 1 Like this post and our page 2 Tag your Gaza or your cp 3 And give us your betclic nickname and it exact score of this NGRANG (Ex: NGR 2-0 ANG or 2-0 for Nigeria) End of the game 5 p.m. Draw: tomorrow Good luck gbonhi ”

heading: Future Work
text: Our dataset and methodologies offer valuable resources for future research in automated bias detection and analysis. Moreover, our experience underscores the importance of interdisciplinary approaches in tackling complex challenges like media bias. As computer science students, we brought technical rigor to the task, but our success was equally dependent on our ability to understand and analyze the nuances of language and media representation. Looking forward, we see several avenues for future research. There is potential for developing more sophisticated machine learning models for automated bias detection, building on the insights gained from our manual annotation process. Additionally, further investigation into the relationship between bias and propaganda, as hinted at by our correlation analysis, could yield valuable insights for combating misinformation