GroningenAnnotatesGaza at the FIGNEWS 2024 Shared Task: Analyzing Bias in Conflict Narratives (2024)

Abstract: In this paper we report the development of our annotation methodology for the shared task FIGNEWS 2024. The objective of the shared task is to look into the layers of bias in how the war on Gaza is represented in media narrative. Our methodology follows the prescriptive paradigm, in which guidelines are detailed and refined through an iterative process in which edge cases are discussed and converged. Our IAA score (Krippendorff’s α) is 0.420, highlighting the challenging and subjective nature of the task. Our results show that 52% of posts were unbiased, 42% biased against Palestine, 5% biased against Israel, and 3% biased against both. 16% were unclear or not applicable.

heading: 1 Introduction
text: News articles usually claim to be objective and neutral reports. However, in practice, they often deviate wildly from these standards. The deliberate stress on certain content and the nuanced shifts in word choice or writing style can profoundly influence a reader’s perception, particularly in political discourse (Hackett, 1984; Morstatter et al., 2018; Recasens et al., 2013; Thomas et al., 2006). This phenomenon is referred to as news bias (Lim et al., 2020). Identifying such bias is a critical yet difficult task in media studies, as bias can manifest subtly, such as through the minimal differences in meaning between words or in specific lexical choices, exemplified by terms like “illegal immigrants” versus “undocumented immigrants” (Lim et al., 2018).

heading: 1.1 Task and contribution
text: In an effort to identify bias, the FIGNEWS shared task (Zaghouani et al., 2024) concentrates on determining the “intricate nuances of bias and double standards” prevalent in news articles and their related social media advertisements on the 2023 Gaza War, starting from the Hamas attack on Israel on
October 7th and the subsequent destructive Israeli military campaign against Gaza,1 which lead to South Africa charging Israel with genocide at the International Court of Justice; while the case is still ongoing, the court treats the accusations as credible and has ordered measures to prevent a genocide.2 This shared task involves a curated multilingual collection of texts, encompassing news article headlines and Facebook posts from news organizations in five languages: English, Arabic, Hebrew, French, and Hindi. The primary goal of this initiative is to perform an annotation study using these collected texts. The annotation is aimed at identifying bias (sub-task 1) and detecting propaganda (sub-task 2).
Our team, GroningenAnnotatesGaza, participated in the bias identification sub-task, employing the classification labels provided by the shared task organizers. Detailed in section 2.1, our annotation methodology incorporates an additional layer of narrative frames.3 This optional annotation layer is designed to augment the bias identification process (refer to Section 2.3). The team annotated 2000 instances (two batches), reaching an inter-annotator agreement of α = 0.420. The main insights from our analysis are that overall, slightly over half of the articles are not overtly biased (although this comes with caveats, as we will discuss), although there are substantial differences between languages; out of the articles annotated as biased, bias against Palestine is far more frequent than bias against Israel.

heading: 1.2 Related Work
text: Political bias in texts has been explored in various studies, particularly concentrating on prediction
1For an overview, see the US Congress report at https://crsreports.congress.gov/product/pdf/R/ R47828.
2See the ICJ press release at: https://www.un.org/ unispal/document/icj-press-release-14may24/.
3Framing is a communication strategy that highlights certain aspects of a topic to promote a specific interpretation (Liu et al., 2019; Entman, 1993).
630
tasks such as identifying the ideology of individual users (e.g., Preoţiuc-Pietro et al. 2017) and towards political entities (e.g., Anegundi et al. 2022). Besides, different domains, including news articles, have been addressed (e.g., Potthast et al. 2018).
Our bias annotation is based on framing, a notion studied in various contexts such as argumentation (Ajjour et al., 2019) and news articles. For instance, framing strategies in German newspaper articles on the ‘European refugee crisis’ have been analyzed (Yu, 2023). In contrast to these studies, we use frame analysis as a means to label bias.
A notable related shared task is the SemEval2019 Task 4 on Hyperpartisan News Detection (Kiesel et al., 2019). This task classified news articles based on extreme left-wing or right-wing viewpoints, providing participants with datasets of 1,273 manually annotated articles and 754,000 articles labeled via distant supervision. The task attracted notable interest, with 42 team submissions.
On Annotations and Guidelines Recent studies have explored the effectiveness of incorporating human-in-the-loop approaches to guideline design (Antici et al., 2024; Guzmán-Monteza, 2023; van der Stappen and Funk, 2021). By involving annotators in the iterative refinement of guidelines, different views on data might be discussed and converged. This allows having consistent annotations with respect to certain guidelines (GuzmánMonteza, 2023). In their work, when drafting guidelines for argument mining, Guzmán-Monteza (2023) identify edge cases that are unclear, ambiguous, or not considered at all. Such cases are discussed, resolved and included in the guidelines until such cases do not occur anymore. However, disagreement can still occur in the form of Human Label Variation (HLV) (Plank, 2022), which represent systematic disagreements between annotators.
On Biased Language Hube and Fetahu (2019) defines bias as partial language that does not represent equally the involved parties and it not neutrally phrased. Biased language is introduced through the presence of inflammatory words or phrases, or statements that may be incorrect or one-sided, thus violating such consensus. Vargas et al. (2023) propose a detailed annotation schema to identify bias. They include, among other indicators, vague, dramatic, or sensational language; unsubstantiated claim with no evidence; opinions presented as facts; flawed logic; subjective qualifying adjectives and omission of source attribution. Aleksandrova et al.
(2019) annotate a multilingual corpus of biased sentences in Wikipedia. They consider a sentence as biased when it contains subjective intensifiers (verbal and nominal modifiers, superlatives and quantifiers, overstatements and exaggerations); cliches and jargon; describing or analyzing rather than reporting events; intentional vagueness; stating facts as opinions and opinions as facts.
Bias is closely related to subjectivity, as subjectivity can be considered as an indicator of biased sentences (Vargas et al., 2023). Antici et al. (2024) consider a sentence as subjective if its content is based on or influenced by personal feelings, tastes, or opinions. More precisely, a sentence is subjective if it contains speculations to draw conclusions; includes sarcastic or ironic expressions; gives exhortations of personal auspices; contains discriminating or downgrading expressions; contains rhetorical figures that convey the author’s opinion.
Finally, another way to reveal biased language is through Frame Analysis, which investigates how readers perceive the information in a news article (Hamborg, 2023). Indeed, according to Hamborg (2023), bias is defined as the effect of framing, i.e., how the targets in the news are portrayed.

heading: 2 Annotation Methodology and Examples

heading: 2.1 Development of Annotation Guidelines
text: All the authors of the paper have contributed to the creation of the guidelines and they annotated the first two batches of the dataset. We base the creation of our guidelines on the prescriptive paradigm (Rottger et al., 2022), which aims at consistently encoding one belief. As the prescriptive paradigm suggests, guidelines are iteratively refined based on the discussion phase, in which edge cases are pointed out and resolved. When an agreement among all annotators is reached, edge cases are converged into the final set of guidelines. Our first pilot study consisted of annotating 50 samples according to the given labels: unbiased, biased against Palestine, biased against Israel, biased against both Israel and Palestine, biased against others, unclear and not applicable. After annotation, a first discussion phase followed. During the first discussion, we identified several edge cases concerning quotes, hashtags and seemingly unrelated texts (see Section 2.2). After reaching a consensus on the edge cases, a second pilot study of 50 samples was conducted, followed by a second discussion phase. In the second discussion phase, edge cases such as
texts that can be interpreted as both bias against Palestine or Israel were discussed (see Section 2.2). After reaching a consensus, the edge cases were included in the guidelines and the guidelines were finalized.

heading: 2.2 Mitigation of Edge Cases
text: Quotes According to previous studies on media bias, biased language contained in quotes should be considered objective, since it does not represent the author’s, but a third party’s opinion (Antici et al., 2024). We distinguish between attributed and unattributed quotes (i.e., is the source mentioned or not); attributed quotes are always labeled as unbiased. For example, Israeli military said in the past 24 hours it had struck targets including a tunnel housing Hamas fighters and lookout posts is ‘unbiased’ as represents the standpoint of the Israeli military. Unattributed quotes, such as “My head is just exploding to think about the logic of this President. He goes to Israel, basically functions as a shill to Hamas” is labelled as ‘not applicable’ as we could not consistently judge whose narrative was represented due to the absent source.
Hashtags Since hashtags are sometimes used simply to improve reach on social media, and do not add to the content of the text, we decided that the main text should be leading in the bias annotation. However, hashtags can be considered if they consistently correspond to a specific viewpoint or complement the information from the text.
Samples Samples that are not related to the topic, were labeled as ‘unclear’. Texts that could be interpreted as bias against Israel or bias against Palestine in different context, were also labelled as ‘unclear’, for example Israel Palestine conflict: Hamas showed courage, now ’disaster’ on Gaza! , R Bharat. , , #IsraelPalestineWar #israel #palestine #HamasvsIsrael #IsraelHamasWar [. . . ].
Consolidation After the two pilot studies, we consolidated the guidelines to label the final dataset. We decided to include frames annotation as, according to Hamborg (2023), showing the frames present in coverage on the same topic can help revealing bias and make more informed decisions.

heading: 2.3 Development of Frames
text: During the development of the guidelines, we reached an agreement solely labelling bias would be a too narrow reflection of the different narratives
the news portrayed. We thus decided to identify two sets of frames that seemed relevant to address some cases of indecision, and in general, to highlight how perspective is expressed in the posts under examination. We started the identification of the frames by writing down a preliminary list while we annotated the batch of posts for the first pilot study. Following this initial phase, we organized the frames into two separate lists, each portraying narratives that corresponded with either the proIsrael or pro-Palestin. Oerspective. Subsequently, we incorporated in the second pilot study the annotation of the frames according to the lists. During the discussion of the second pilot study, we adjusted the list of frames. To finalize the frames, we grouped them into five categories portraying the Palestinian narrative and eight frames portraying the Israeli narrative (see Appendix A).
Note that frames may still be annotated even when a sentence is labeled as ‘unbiased’: for example, in texts with quoted text, the quote may contain a frame fitting a particular narrative and this should be annotated as such, even though it does not weigh into the decision to select a bias label. For the present study, we used narrative frames only as an aid to guide annotators during the bias annotation task. Our frame annotation layer is therefore (for now) incompletely annotated; we plan to further refine our scheme and release complete annotations as part of future work.

heading: 2.4 Data Annotation Process
text: We distributed the texts randomly across the annotators, without taking language into account. The edge cases in the guidelines were leading in the annotation process. The main focus of the data annotation was the bias annotation, whereas the frame labels were used as secondary labels. The frame labels were thus not final in deciding bias. The frames could help however with bias identification, for example the frame IDF glorification shows bias against Palestine, despite not explicitly naming Palestine or Hamas in the text.

heading: 2.5 Inter-Annotator Agreement (IAA) Analysis
text: We have analyzed the IAA using Krippendorff Alpha across all seven annotators, and the pairwise agreement by the observed agreement per pair. The examples in the guidelines were leading during the annotation process and therefore central in achieving reliable agreement. Our IAA is well above
chance level, with a Krippendorff’s α at 0.420. The pairwise agreement between the annotators is consistent, with observed agreement per pair above 0.5 for every pair (see Figure 2 in Appendix A).

heading: 3 Team Composition and Training
text: The annotation process was carried out by a team seven annotators with expertise in computational linguistics, linguistics, and Middle Eastern studies, and with diverse linguistic backgrounds (native speakers of Italian, Dutch, Arabic, and Bengali); five annotators are aged between 25 and 34, one between 18 and 24 years old, and one between 35 and 44 years old.

heading: 4 Task Participation and Results
text: As can be seen in Figure 1, Hebrew has the highest percentage of texts labelled as biased against Palestine. The Hebrew texts also contain the lowest percentage of unbiased texts, around 25%, in comparison to 50%-70% of unbiased texts in the other languages. The French texts contain the highest percentage of texts biased against Israel and the Hindi texts as the highest percentage of unclear texts.

heading: 5 Discussion
text: The high percentage of bias against Palestine Hebrew texts could be due to the sources of these texts, which are most likely to be Israeli. As we have not annotated propaganda, we cannot conclude whether the Hebrew texts contain Israeli propaganda. We can however conclude that the Hebrew texts contain the strongest bias against Palestine in this dataset and that the sources most likely portray the Israeli narrative.
On the contrary, it is harder to interpret the high percentage of bias against Israel in French texts
and unclear labels in Hindi texts. As the sources of the texts are not included in the dataset, it is unclear what influence the sources have on the bias presented in the texts, as the French sources could be from France, but also could be from news items aimed at areas from the former French colonial empire.
Notably, texts from Hindi platforms were often incomprehensible and sensational. Besides, some texts were merely lists of hashtags, offering little substantive content. Moreover, certain texts indicated contradictory biases, simultaneously supporting perspectives on Palestine and Israel.

heading: 6 Conclusion
text: Our paper presented the guideline methodology developed for the participation in the shared task FIGNEWS 2024, aimed at annotating media narrative bias in representing Israel’s war on Gaza. We followed a prescriptive paradigm to craft our guidelines, including two rounds of discussion phase for guidelines refinements based on identified edge cases. We added an extra level of annotation, which is the type of frame that supports either the Israel or the Palestine narrative. Our annotation shows that Hebrew texts have the highest percentage of biases against Palestine, while the French texts are the most biased against Israel and English texts are mostly unbiased.
As for future work, we plan to refine our annotation by expanding our set of frames and annotating additional texts on the topic. We also aim to adapt our annotation scheme and apply it to other existing datasets relating to the Palestinian-Israeli conflict, for example the bitterlemons initiative.4 Thereafter, we intend to develop a computational model for detecting frames and examine the relationship between frames and different types of bias.
Limitations
As discussed in Section 5, the text set for the annotation presents several issues. Some texts do not appear to be related to news reports but rather seem to be personal social media posts. Additionally, the posts in Hindi are often short and unintelligible.
The provided labels ‘bias against Palestine’ and ‘bias against Israel’ are problematic. Bias may be evident towards Israel without necessarily being
4See https://www.bitterlemons.org/. The bitterlemons project gathers online essays from Palestinian and Israeli authors.
against Palestinians, and vice versa. Also, texts often refer to specific identities that do not represent the entire country. For example, Hamas does not clearly represent all of Palestine, and the Israeli government is not the sole entity representing Israel. We believe that one of the improvements to the set could be renaming the two labels as ‘bias pro-Israel’ and ‘bias pro-Palestine’, as we think that bias is closely related to the perspective from which events are narrated. Regarding our list of frames, we found out that it is not exhaustive, as during the annotation process we did not find a correspondence between all instances and frames in the list.

heading: A Appendix: Observed agreement
text: Figure 2 displays pairwise observed agreement (Ao) between annotators.

heading: B Appendix: Guidelines
text: See the following pages.
Annotation guidelines/difficult cases
General framework
We define "bias" in terms of explicit use of elements from the pro-Palestinian and pro-Israeli narratives about the war on Gaza: texts that use elements from the pro-Palestinian narrative are considered "biased against Israel" and conversely. texts that use elements from the pro-Israeli narrative are considered "biased against Palestine".
For each narrative, we also defined a list of frames (see below) that fit that narrative. These frames were then used as a tool for identifying which narrative (if any) was being used in a particular text. Here, a "frame" is loosely defined as a piece of conceptual information that forms the building block of a narrative. For example "Hamas is a terrorist organization" is a frame of the Israeli narrative, while "Israel is committing genocide" is a frame of the Palestinian narrative. We established a list of frames before the annotation work was started; aware that our list of frames is not exhaustive, we also allowed annotators to annotate bias even if no applicable frame is present in a given piece of text.
Texts that describe the events of the war without explicitly using either of the two narratives are considered "unbiased". Note that this does not have to mean that they actually are unbiased; bias can also take the form of choosing to report certain events over others, emphasize the role of certain participants over others, or quoting certain sources over others; or even by simply reporting false information that fits with a particular narrative. However, here, we focus purely on the text itself, without considering the context (which we don't have enough information on). Hence, "unbiased" should be understood as "lacking an explicit textual bias".
Notes on specific labels
● Unbiased: The label unbiased is used when the text is reporting on events, or does not show clear bias. There can be certain lexical choices in a text that show a point of view of the author, but when they are too strongly related to context this is still unbiased (see example Row #5 and Row #17).
○ Examples second annotation: ○ Row #23: New attack occurred this morning in Jerusalem which left several
victims while new hostages had to be released. Hamas claims responsibility for this attack. <unbiased>
○ Row #5: The death toll in Gaza has surpassed 25,000 people since the start of the Israel-Hamas war, according to the Hamas-controlled health ministry in the enclave. Follow live updates. <unbiased> ○ Row #17: Who is #Yahya_Sinwar, the leader of “#Hamas” in #Gaza, whom #Israel threatened to reach after the “#Al-Aqsa_Flood” operation? Watch the #magazine video👇 <unbiased>
○ Row #7: Countries around the world are demanding accountability for Israel’s alleged war crimes against Palestinian civilians. Chile and Mexico most recently asked the International Criminal Court (ICC) to investigate alleged war crimes committed by all sides against civilians since Oct. 7 during Israel’s siege of Gaza, which has killed more than 24,700 Palestinians. Around 1,140 Israelis were killed by Hamas on Oct 7. South Africa has already put Israel on trial at the International Court of Justice (ICJ) for alleged genocide against Palestinians in Gaza and is demanding a ceasefire. Indonesia and Slovenia have also joined a request for an ICJ opinion on whether Israel has violated Palestinians’ right to self-determination through its illegal occupation of the Palestinian territories. The difference between the ICC and ICJ is that the ICC prosecutes individuals and the ICJ prosecutes nations. However, neither court has enforcement power. Israel has denied the charges in the South Africa ICJ case and isn’t a member of the ICC. <unbiased>
● Unclear: The label unclear is used when a phrase is nonsense or can be interpreted as both the Palestinian and the Israeli narrative.
○ Examples first annotation: ○ Row #9: Hate in Holywood. <unclear> ○ Row #45: Israel Palestine conflict: Hamas showed courage, now 'disaster' on
Gaza! , R Bharat. , , #IsraelPalestineWar #israel #palestine #HamasvsIsrael #IsraelHamasWar #israelpalestineconflict #israelpalestine #israelnews #israelipalestinianconflict #netanyahu #warlive #IsraeliPalestineianRelations <unclear>
○ Examples second annotation: ○ Row #2: An FEC complaint argues 51 former intelligence officials strategically
crafted to discredit the Hunter Biden laptop story. Catch up on this story and more top headlines in today’s Fox News First newsletter. <unclear>
● Unattributed vs attributed quotes: We differentiate between unattributed and attributed quotes. Both types of quotes highly depend on context. With unattributed quotes, it is specifically hard to judge the bias since it is unclear whose perspective is represented. Therefore, unattributed quotes are labelled as <not applicable>. Attributed quotes can be a form of reporting on events that happened. Therefore, these are labelled as <unbiased>. Note: the quote can still have a frame labelled to it
○ Examples first annotation: ○ Row #5: “My head is just exploding to think about the logic of this President. He
goes to Israel, basically functions as a shill to Hamas.” <not applicable> ○ Row #4: Israeli military said in the past 24 hours it had struck targets including a
tunnel housing Hamas fighters and lookout posts. <unbiased> + <Israel’s targeted strikes>
○ Examples second annotation: ○ Row #3: “Even wars have rules. All innocent life is equal in worth — Israeli and
Palestinian” Canadian Prime Minister Justin Trudeau urged Israel to stop “this killing of women, of children, of babies” in the Gaza Strip, during a news conference on Tuesday, which he described as heart wrenching. Trudeau called
on the Israeli government to exercise 'maximum restraint'. Israeli Prime Minister Netanyahu slammed Trudeau following his speech saying Israeli forces don’t deliberately target non-combatants while Hamas ‘beheaded, burned and massacred civilians’. <unbiased> + <Israel target civilians> + < Hamas terrorist attack> ○ Examples second annotation: ○ Row #9: "Anti-Israel rallies were announced in close to 20 other cities in the US,
Canada and Europe, including at Israeli consulates and embassies in San Francisco, Atlanta and London." Read the full story at: <not applicable>
○ Row #15: Reuters, citing officials: #Khamenei informed #Haniyeh that #Iran will not enter the war “on behalf of Hamas” #Al-Arabiya <unbiased>
● Hashtags: You can use hashtags as context to determine the bias of a phrase. However, it is important to keep in mind that people can use a mix of hashtag to improve their reach. Using a hashtag does not necessarily represents the authors point of view. In cases where it is unclear how to interpret the hashtag, the bias portrayed in the actual text is leading (see example row #45).
○ Examples first annotation: ○ Row #43: Israel caused devastation in Gaza through this bomb! Israel's bunker
buster bomb which is destroying Gaza's tunnels, bunkers, buildings and towers. #Israel #Gaza #Israel_under_attack #IsraelPalestineWar #HamasAttack #IsraelPalestineConflict #Jerusalem | #ATDigital <bias against both> + <general sensationalism>
○ Row #45: Israel Palestine conflict: Hamas showed courage, now 'disaster' on Gaza! , R Bharat. , , #IsraelPalestineWar #israel #palestine #HamasvsIsrael #IsraelHamasWar #israelpalestineconflict #israelpalestine #israelnews #israelipalestinianconflict #netanyahu #warlive #IsraeliPalestineianRelations <unclear>
● Links:We have decided to not consider links in the annotation ● Multiple frames: a phrase can have multiple frames.
○ Examples first annotation: ○ Row #34: Adva Adar is my grandmother! You were abducted unhindered to the
Gaza Strip. Her name is Yaffe Ader, she is 85 years old!! My grandmother who founded the kibbutz with her own two hands, who believed in Zionism, who loved this country that abandoned her, was kidnapped. Probably dumped somewhere, suffering from severe pain, without medicine, without food and without water, dying of fear, alone. No one talks to us, doesn't know how to say anything, we found out all the information we have from videos that were circulated. I want this image to be regretted by all those disconnected from this disgraceful government. Let them understand that there are people here, elderly and children, women and men, with names and families. I want them to stay awake at night and turn over every piece of land until these people come home. Please share the post. Hope it reaches people who might be able to help us. <bias against Palestine> + <Israel should protect its citizens>/<protection of Israeli sovereignty>
List of frames
Below are our lists of frames for the two narratives. The frames are also annotated (zero or one frames per text) in the "frame" column in the annotation sheet.1
Palestine narrative/bias against Israel ● Israel occupation
○ Apartheid ● Resistance Palestine ○ Hamas resistance ● Israel targeting civilians
○ Genocide ○ Bombing hospitals (deliberately)
● Retaliation events before October 7 ● International support Palestine
○ ICJ & ICC
Israel narrative/bias against Palestine ● Rise of antisemitism
○ Juxtaposition October 7 and antisemitism ● Israel western country
○ Pro-LGBT ○ Democracy
● Hamas responsibility conflict ● Israel self defense
○ Israel targeted strikes ○ Destroy hamas ○ Israel surrounded by all sides
● Glorification IDF/patriotism ○ Glorifying zionism ○ Soldiers giving up their life ● Hamas is a terrorist organization ○ Hamas aims at killing civilians ○ NGOs colluded Hamas (conspiracy theory) ● Israel international support right self defense ● Denying Palestinian suffering
1 Since the frame layer is an extra annotation layer, we did not complete our annotations in this layer before the deadline of submitting bias annotations. The full annotation results for frames will be discussed in our paper.

heading: Future Work
text: As for future work, we plan to refine our annotation by expanding our set of frames and annotating additional texts on the topic. We also aim to adapt our annotation scheme and apply it to other existing datasets relating to the Palestinian-Israeli conflict, for example the bitterlemons initiative. 4 Thereafter, we intend to develop a computational model for detecting frames and examine the relationship between frames and different types of bias. Limitations
As discussed in Section 5, the text set for the annotation presents several issues. Some texts do not appear to be related to news reports but rather seem to be personal social media posts. Additionally, the posts in Hindi are often short and unintelligible. The provided labels ‘bias against Palestine’ and ‘bias against Israel’ are problematic. Bias may be evident towards Israel without necessarily being
4See https://www. bitterlemons. org/. The bitterlemons project gathers online essays from Palestinian and Israeli authors. against Palestinians, and vice versa. Also, texts often refer to specific identities that do not represent the entire country. For example, Hamas does not clearly represent all of Palestine, and the Israeli government is not the sole entity representing Israel. We believe that one of the improvements to the set could be renaming the two labels as ‘bias pro-Israel’ and ‘bias pro-Palestine’, as we think that bias is closely related to the perspective from which events are narrated. Regarding our list of frames, we found out that it is not exhaustive, as during the annotation process we did not find a correspondence between all instances and frames in the list