Pragmatic Annotation of Articles Related to Police Brutality
Year: 2023

Abstract:
The annotation task we elaborated aims at describing the contextual factors that influence the appearance and interpretation of moral predicates, in newspaper articles on police brutality, in French and in English. The paper provides a brief review of the literature on moral predicates and their relation with context. The paper also describes the elaboration of the corpus and the ontology. Our hypothesis is that the use of moral adjectives and their appearance in context could change depending on the political orientation of the journal. We elaborated an annotation task to investigate the precise contexts discussed in articles on police brutality. The paper concludes by describing the study and the annotation task in details.

heading: 1 Introduction
text: The use of moral predicates in natural language is a topic of interest for linguists, as it sheds light on the complex interplay between language, ethics, and society. The annotation task proposed in this paper studies the use of moral predicates in newspaper articles in French and in English discussing police brutality, with a focus on the contextual factors that influence their appearance and interpretation. Indeed, our broader research goal is to determine whether the use of moral adjectives and their appearance in context changes depending on the political orientation of the journal.
To achieve this goal, we elaborated a pragmatic annotation on a corpus of newspaper articles on the George Floyd and Adama Traoré cases, published between May 2020 and August 2020, from a variety of newspapers across the political spectrum. A basic contextual ontology that has been elaborated specifically for this study, that takes into account various actors involved in events surrounding police brutality.
We will start by making a brief review of the literature on moral predicates, their relation with
context, and pragmatic annotation. Then we’ll describe the elaboration of the ontology, and some preliminary results that lead us to decisions for the annotation task. Finally, we’ll describe the study and annotation task in details. We currently don’t have the results of this annotation, as the participants are still annotating at this time.

heading: 2 Background of the study and some assumptions
text: Moral predicates (like good, wrong) gained more popularity in the last years (Faroldi and Ruiz, 2017; Ruiz and Stojanovic, 2019; Stojanovic, 2019; Soria Ruiz and Faroldi, 2022), and linguists often compared them to predicates of personal taste (like tasty, fun) (i.a. Lasersohn, 2005) and to aesthetic predicates (like beautiful, elegant) (i.a. McNally and Stojanovic, 2014). Indeed, moral predicates show similar patterns to predicates of personal taste (PPTs) in that they express a subjective judgment. Faultless disagreement appears when two speakers disagree on a subjective matter. It is the type of disagreement that two people can have over liking pork belly or not for example ; no one is right or wrong, they just have different tastes.
Indeed, a statement made about taste cannot be countered by stating the speaker’s experience is false. For that reason, and similarly to PPTs, moral predicates are very sensitive to the context they are in ; they react to the experience the speaker is discussing1. To our knowledge, only one corpus study was done on moral predicates (Stojanovic and Mcnally, 2022), precisely on the type of subjectivity moral predicates capture. Kaiser and Rudin (2020) argue in their article that the strength of faultless disagreements changes when taste predicates are used in the context of widely-held opinions. Indeed, faultless disagreement isn’t a binary phenomenon, but a gradient one, that depends on the object of
1They are also often called "experiential predicates" (see i.a. Stojanovic, 2019; Willer and Kennedy, 2022).
146
predication (and not just on the predicate itself). We can then make the hypothesis that moral predicates are also sensitive to a change of context, and that this could have implications for the subjectivity they express. The lack in current knowledge on the use of moral predicates in context, and the assumption that context can influence their understanding all together were the first issues that brought us to investigate this topic.
Theses consideration in mind brought us to the field of pragmatics, specifically at the intersection of what Yule (1996) calls "speaker meaning" and "contextual meaning". Indeed, what the speaker is thinking before uttering a sentence, and how the context can influence this thought is especially vague when it comes to moral predicates. For PPTs like tasty, we assume the speaker is talking about her own experience relating to some dish. However, expressing that pizza for example is tasty is less surprising than doing it for spinach or snails (see again Kaiser and Rudin, 2020). Context then has to remain the same if we want to analyze the behavior of moral predicates. What differentiates the use of right from the use of good? Are just and right more likely to be used in contexts where justice is mentioned, as one could infer intuitively? Are moral adjectives used when the context is clear, or when it is shifting from one topic to another? Our goal with this annotation task isn’t as broad as analyzing the specific subjectivity expressed in moral predicates or giving a semantic analysis of the meaning of such predicates2. Instead, understanding precisely what is being discussed in these articles will allow us to understand the contexts in which moral predicates are more likely to be used. This annotation task is to be considered as the first step towards this analysis. It will help us provide a better understanding of the context itself, in order to make hypothesis on the behavior of moral predicates in articles on police brutality.
2One reviewer noted the polysemy of moral predicates, and the difficulty to distinguish moral predicates from non-moral predicates. For clarity, here is the list of moral predicates we are interested in in this study : fair, unfair, just, unjust, good, bad, right, wrong, correct, incorrect and their French equivalents : équitable, inéquitable, juste, injuste, bien, mal, bon, mauvais, correct, incorrect. We are also working on a semantic analysis of these predicates, but this is beyond the scope of this paper.

heading: 3 Elaboration of the ontology and some preliminary results

heading: 3.1 Corpus data
text: To provide an analysis of moral predicates in context, we chose to gather articles discussing the similar contexts – the George Floyd case for the American news sources, and the Adama Traoré case for the French news sources. We did choose news sources bearing diverging political view points to offer a wide range of contexts: for example, we picked Jacobin for their left-orientation and Breitbart for their right-orientation for this study3.
We started with a very large corpus, composed of every article mentioning George Floyd (US) or Adama Traoré (FR) between the day each of them passed away (May 2020 for George Floyd and July 2016 for Adama Traoré) and September 1st 2021. However, we realized that some articles had no real mention to either case, and were only using the names of the victim once. Preventing this problem was possible if we focused on the time period surrounding the protests that came after the death of George Floyd. Indeed, Adama Traoré died in 2016 but the case gained in popularity in France after the BLM protests emerged throughout the United States. To compare the articles containing sentences using moral predicates and articles containing sentences that don’t (see Rayson and Garside, 2000), we decided to gather articles between May 2020 and August 20204, and picked for each article containing a moral predicate an article that’s doesn’t and is close in time – published the day before or after when possible. We made sure that the same number of articles from each sources was gathered.

heading: 3.2 Basic ontology
text: Now that all the data is gathered, we started discussing how to narrow down contexts surrounding police brutality. The core of our needs for annotating these articles is to precise our understanding of the broad context "police brutality". Understanding the context precisely isn’t linked to the "side" the
3We are hoping that this preliminary annotation task will be useful for further investigation of bias in reporting. We did not automatically predict the political orientation of these newspapers beforehand (see i.a. Kulkarni et al.; Baly et al.) and simply chose newspapers known to have a specific political orientation. We are not excluding using automation in the future, and comparing it to our results.
4This is the time period that concentrates the most of the protests, both in France and in the United States.
article was taking – pro- or against BLM for example –, as we didn’t want our annotators to have to provide their own opinion. We read approximately 50% of articles that were published during the time period we agreed on, and noted various events that were recurrently mentioned5.
It appeared that out of the police brutality case, other recurrent circumstances resulted in both the French and the American data : tributes from the family, protests, and mentions of trial. In terms of who was talked about though, the policemen involved were, without a real surprise, the actors that were the most mentioned. We sensed a difference between the events related to the state as an entity, and the events related to the population as a whole. However, we sometimes encountered elements that weren’t completely understandable either as state or as population : social media entities like Twitter, or specific political parties that weren’t discussed as elected officials. Indeed, what we wanted to focus on in this ontology is what is "being talked about", not who is talking. As such, an elected official, like a governor for example, can be discussed
5We didn’t use Latent Semantic Analysis (i.a. Deerwester et al.; Landauer et al.) for this specific task, but are hoping to in the future when we will look at documents discussing a larger variety of topics. We thank reviewers for their very constructive comments that will guide our next steps.
specifically for his views as a Republican, not in his quality as a governor. Our goal with this ontology was to see if it would both lead to an understanding of what type of event is mostly discussed by the different news papers, and if any moral predicate was more used in one context or the other.

heading: 3.3 Some tests and modification
text: To see if our ontology would be able to give an insight into the semantics of the moral predicates, we annotated 30 sentences containing moral predicates from each newspapers – like right, good or correct– , following the first level of the ontology. These preliminary results showed that Breitbart used 8 times the moral predicate right to discuss contexts related to the State (like police or justice system), whereas the New York Times mostly used good in those contexts. Jacobin and the New York Times however, used right when discussing the population as a whole. We realized that the broad categories State and Population weren’t precise enough for our purpose, simply because it gathered together the justice system and the police, or the family of the victim and protesters, when those entities had very different events associated to them : the policemen involved are, in the context of police brutality, the ones being judged, but in some articles the justice system itself was criticized. We wanted to show this contrast, and ended up with the ontology in Figure 1.

heading: 4 Annotation task
text: This is a description of the annotation task our participants are currently accomplishing. Each sentence will be annotated twice : once by a member of the research team, and once by a recruited participant. We currently cannot present a measure of the inter-annotator agreement, as the task is still in progress. However, we are aware of the subjectivity of the task : even though the guidelines focus on the context surrounding each sentence and doesn’t ask from the annotator to express a personal opinion on the topic itself, the perceived importance of the topic discussed can still vary from one person to the other.

heading: 4.1 Elaboration of the task
text: The annotation task was based off of the second level of our ontology, to ensure a more precise annotation, as we understood the first level to be too broad. We used Qualtrics as our software for
this task6, specifically the matrix table survey type. Each sentence from each article was separated using NLTK, and placed in the rows of the matrix table. The categories are appearing on the horizontal axis of the table, and the participant has to tick the correct box, according to the annotation guidelines (see Appendix A). We choose to divide the articles in sentences to ensure great precision. We wanted to observe the moment were the context shifted from one category to the other.
To transfer each article to Qualtrics, we used Python to convert them to the Advanced Text format of Qualtrics. The Advanced Text format is a simple way to import data into Qualtrics without having to import every single article by hand and add each category by hand as well.
Every set of articles was randomly assigned to participants, to ensure that they would get a similar amount of article containing moral predicates and not containing moral predicates. The participants were not aware we were precisely focusing on moral predicates. Each article was doubleannotated by a participant and by a member of the research team.

heading: 4.2 Task and participants recruitment
text: The task itself was to associate each sentence from each article to a category from the ontology7. We found participants by putting an ad on the university list-serv. We recruited 5 participants for the English data, and 1 participant for the French one. Indeed, the French corpus is much smaller than the American one. Participants recruited for the American data have to annotate 294 articles in total, whereas the French participant has to annotate 136 articles. We had to immensely lower the amount of articles given to each participants, as they progressed with the annotation task slower than anticipated. We gave every single participant the annotation guidelines when sending them the survey link, but also had an individual Zoom meeting to review these guidelines, answer any question,
6This was done to ensure meeting university requirements and provide a better user experience for our participants, the website being optimized for survey responses.
7As such, this task is very similar to the early stage of an basic entity linking task, where sentences are associated with the knowledge base of the context categories. However, sentences discussing police brutality were classified as a category of the ontology whether they contained a mention the actual name of the category or not. For example, sentences classified as "Family" didn’t always include the word "family". This task is different from named-entity recognition in that regard (i.a. Marrero et al.).
and review the consent form. This meeting took place before any data collection.
Before starting the annotation task, each participant had to consent to the study. To do so, they were presented with the consent form, and had to tick a box to consent. They were not asked to provide their name, signature, or any other identifying data. Indeed, this annotation task is anonymous, and each participant was provided with an anonymous Qualtrics link as well. At no point during the study were the participants told the research was done on moral predicates. This was done to ensure they wouldn’t treat sentences containing moral predicates differently than sentences that don’t include them. The participants also don’t know the newspaper the article was taken from, nor the date it was published.
After the consent form, they were introduced to the annotation guidelines we explained to them during the Zoom. They had to pass a quick test to make sure those guidelines were understood properly before starting the annotation. The test was composed of five multiple choice questions. Each question had between one and three sentences taken from articles that had to be annotated. To make the task easier, only 5 out of 11 categories were presented as a possible answer. The participant had to click on the correct one in order to pass the test. If one category was wrongly chosen, the participant had to choose again until they chose the right one. Then, they were able to move on to the proper annotation task.
As we stated earlier, the articles are randomized. However, the sentences themselves are not. The sentences of the articles had to be shown as they appear in the article, in order to provide the proper context.

heading: 4.3 Examples
text: To explain the task in further details, let’s look at some examples of expected annotation in sentences containing moral predicates. We put emphasis on the moral predicates by making them bold.
(1) Americans have watched protests dedicated to ending unjust violence mutate into riots that inflict unjust violence themselves.8 PROTESTS
8"McConnell: Can’t ‘Deafen’ Ourselves to Pain of Black Americans, Riots Inflicting ‘Unjust Violence Themselves’", 06/01/2020, Ian Hanchett for Breitbart News.
In (1), the topic of discussing is the protests, and even if the authors used the word "riot", which is biased in comparison to a more neutral word like "protest", the event in question is still the protests following the George Floyd murder. We tried our best to make the ontology categories insensitive to the expressed bias. The classification as "Protest" is here preferable.
(2) Right now, defunding the police is not a radical demand. That is good government. The idea of funding public services and making sure that people have their needs met — that’s just good government. 9 GOVERNMENT
In (2), the author does not focus on "defund the police" as a claim made by protesters (it would have been classified as "Protest") or a specific political group (that would be "General political movement"). She also does not insist on the repercussions such a policy could have for law enforcement as a whole (that would be "Police"). Instead, she qualify this as "good governement", and a policy that could benefit the population. As such, this whole paragraph should be classified as "Government".

heading: 5 Conclusion
text: The aim of this paper was to describe the elaboration of a pragmatic annotation task, that takes into account the specifics of the police brutality context. Our hypotheses is that moral predicates could potentially be a marker of biases in newspapers articles. Currently, we know very little about the behavior of moral predicates in context, since only one corpus study involving them has been conducted. To allow for an understanding of the variety of topics surrounding police brutality, we elaborated a ontology based on the Adama Traoré and the George Floyd cases. Our goal is to investigate how moral predicates behave in discussion of these similar contexts, to compare them with one another and crosslinguistically, as well as to see if one (or more) sub-context of police brutality is more likely to involve moral predicates. By proposing an annotation task to precise the context of police brutality itself, and by choosing articles containing moral predicates and articles that don’t,
9"Police Are Not Designed to Solve the Problems People Are Facing", 06/12/2020, An interview with Rossana Rodriguez-Sanchez. Jeanette Taylor for Jacobin.
we are hoping to answer these questions. We are aware that this work is still at its early stage, but are looking forward to get feedback on this primary study, in hopes of perfecting it in the future.
Limitations
Despite the contributions this study might bring, there are several limitations that must be acknowledged.
• Scope of the study: We focused solely on police brutality and did not consider other contexts that could bring interesting uses of moral predicates in the media.
• Sentiment analysis: The link between sentiment analysis and moral predicates hasn’t, to our knowledge, been studied as such. We don’t make any claim about the polarity of moral predicates in this study, and focus on associations between predicates and the context they are used in, regardless of the tone they are conveying. We hope we would be able to in the future.
• Cultural differences, issues in comparison: As we collected data from different news sources in French and English, finding accurate equivalences between the two languages was challenging, making a perfect comparison impossible. Furthermore, The French and American contexts differ significantly, including differences in public opinion on cases of police brutality. For instance, the death of George Floyd shocked a majority of Americans, whereas it took four years for the French media to bring Adama Traoré’s case to the center of public attention.
• Limited data and generalization: Our study relies on the analysis of 430 articles in both languages (294 in English, 136 in French). The inclusion of more data could potentially strengthen our findings. Moreover, only one case of police brutality per country was annotated, when a larger set of similar circumstances from different time periods would have helped us sketch a more precise picture of police brutality as a whole.
• Translating difficulty: Some moral predicates, such as right and just, do not have a precise
equivalent and are both translated in French by juste. Similarly, the French language has two possible translations for good namely bon and bien. If the semantics of these terms isn’t the purpose of this paper, we are aware that these potential differences in meaning could have an impact on their use in context. We are also working on a semantic description.
Ethics Statement
When elaborating this study, we took into consideration the following elements :
• Informed consent: We made sure to elaborate a consent form stating the goals of the study and the precise actions the participant will have to accomplish in order to finish it. The consent form also included information about the risks and benefits of participating in this research. Indeed, some articles are describing the violent interaction the victim had with the police, and mention systemic racism. We disclosed that some articles were taken from extremely conservatives news sources and could make an apology of white supremacy as well. A full review of the consent form was done beforehand with the participants to answer any potential questions they might have.
• Confidentiality: At no point during the study were the participants asked to disclose any information, whether name, age, gender, occupation, or any other potentially identifying data. We used the Qualtrics survey software’s anonymous link, and did not include any identifiable question in the survey itself.
• Participants welfare: Participants were told that they could withdraw from the study at any time, without any consequences, and that the choice to participate or not was their own.
• Ethical review: This study was approved by the Institutional Review Board of our university before any data collection began. The members of this research team were asked to pursue an ethics training before being able to submit the study protocol.

heading: A Annotation guidelines
text: What follow is the annotation guidelines that were given to the participants.
There are 11 categories total. Each of them represents something that can be associated with police brutality in context. Our goal is to have a more fine understanding of the contexts mentioning police brutality.
Here is the list of all categories :
• Justice ; mentions of judges, justice decisions, trials, testimonies from witnesses, autopsy results. . .
• Police ; mentions of policemen involved or of the police institution as a whole, statements from the lawyers of the policemen involved and of police unions. . .
• Government ; mentions of statements from government and elected officials, changes in state policing. . .
• Political party ; mentions of a specific political party like the Republicans or the Democrats, or one of their elected officials, mentions of their political stances. . .
• General political movement ; mentions of a vague political movement, like “the Left”, or “populists” and their political stances. Also includes vague mentions of communities coming together, outside of protests.
• Social medias ; mentions of social medias such as Twitter or Facebook, and their uses.
• Traditional medias ; mentions of traditional medias such as TV, radio, newspaper, and their uses, mentions of journalists.
• Celebrities ; mentions of any type of celebrity, whether it is to support BLM or not.
• Protests ; mentions of any type of protesters or protests, whether they support BLM or not, descriptions of protest violence and discourse.
• Family ; mentions of the victim’s family or the victim themselves, victim’s family statements, and statements from the lawyer of the family. Also includes direct descriptions of the victim himself and tributes.
• Other ; everything that cannot be related to any of these categories.
For each sentence, your goal will be to associate it with one of these categories. Sometimes, it’s easy to see how to classify the sentence, but it also can get tricky.
(3) The trial of Mr. Chauvin, charged in the death of George Floyd, will resume on Monday.
For example, in the sentence (3), the policeman involved in in Floyd’s murder is mentioned, but the main topic of the sentence is the trial. As such, this sentence should be classified as Justice and not Police. It is a statement made about the trial timeline.
(4) Players have spoken at protest marches, and leagues have bankrolled new social-justice efforts.
In (4), one could wonder if the "players” in question are to be understood as Protests or as Other, since they are attending protest marches, and don’t seem to fit another category. Actually, the “players” in question are MLB top players, in the context of this article. They represent a celebrity, and should be
classified as such. Each sentence of each article will be presented in order, meaning that the article won’t have all of its sentences randomized. The reason for this choice is for sentences like (4), since they can only be properly understood in context. Sometimes a specific sentence does not mention a category, but this sentence is included in the context of another category, like (5):
(5) “This is tough. First of all, I have to say my heart and my prayers go out to the family of George Floyd. What we see in this video is devastating and it’s senseless.”
In this quote, the general context is to be classified as Family. Even though “This is tough” is not in itself related to the family of the victim, the sentence was said in a context discussing about Floyd’s family. Please try your best to see the big picture of the article, and not to just focus on each individual sentence.
Moreover, even though we tried our best to make this survey perfect, it’s possible that some lines are wrongly separated (in the case of tweets containing images for example). In that case, please classify those sentences in the same way you would have the whole tweet. In (6) to (9), the same tweet was separated in 4 lines. This tweet should be classified as Protests, meaning each line has to be classified as Protests, not just (7) and (8).
(6) Decent amount of riot cops showing up.
(7) Thankfully Still no sign of any violence.
(8) "#SanAntonioprotest pic.twitter.com/S1vMELh6cl
(9) — / (@PropheticLaw) May 31, 2020
If you feel like two categories are mentioned - that may happen often! -, please focus on the global context surrounding the sentence and the main category that could fit this context.
Another thing worth mentioning : the correct category is not defined by the speaker, but by the content of the sentence. For example, if the sentence you’re annotating contains a quote from Donald Trump, former president of the United States, talking about the police, it has to be classified as Police and not as Government. What matters is what the sentence is about, not the person expressing it.
The category Other can be used - like its name
suggests - when no other category seem to fit the sentence. However, we will ask that you try your best to associate the sentences you see with one of the 10 other categories. The Other category was mostly created for sentences that don’t relate at all to police brutality, for example for mentions of the COVID-19 pandemic. Indeed, some articles mention both police brutality and other topics. This category is made for those cases.

heading: Future Work
text: We are aware that this work is still at its early stage, but are looking forward to get feedback on this primary study, in hopes of perfecting it in the future. Limitations
Despite the contributions this study might bring, there are several limitations that must be acknowledged. • Scope of the study: We focused solely on police brutality and did not consider other contexts that could bring interesting uses of moral predicates in the media. • Sentiment analysis: The link between sentiment analysis and moral predicates hasn’t, to our knowledge, been studied as such. We don’t make any claim about the polarity of moral predicates in this study, and focus on associations between predicates and the context they are used in, regardless of the tone they are conveying. We hope we would be able to in the future. • Cultural differences, issues in comparison: As we collected data from different news sources in French and English, finding accurate equivalences between the two languages was challenging, making a perfect comparison impossible. Furthermore, The French and American contexts differ significantly, including differences in public opinion on cases of police brutality. For instance, the death of George Floyd shocked a majority of Americans, whereas it took four years for the French media to bring Adama Traoré’s case to the center of public attention. • Limited data and generalization: Our study relies on the analysis of 430 articles in both languages (294 in English, 136 in French). The inclusion of more data could potentially strengthen our findings. Moreover, only one case of police brutality per country was annotated, when a larger set of similar circumstances from different time periods would have helped us sketch a more precise picture of police brutality as a whole. • Translating difficulty: Some moral predicates, such as right and just, do not have a precise
equivalent and are both translated in French by juste. Similarly, the French language has two possible translations for good namely bon and bien. If the semantics of these terms isn’t the purpose of this paper, we are aware that these potential differences in meaning could have an impact on their use in context. We are also working on a semantic description