## 2 Methods  
### Analyzed Data and Annotation Scheme  
In our study, we focused on the detection of respectful tweets. Because of the aforementioned lack of knowledge regarding the author or context of a given tweet, deciding whether a tweet is respectful is not an easy task. Thus, we decided to accept the subjective judgment of what is respectful, as perceived by annotators who were employed to label the analyzed data. Importantly, in this context, our annotators were prepared for the labeling task by participating in a literature review regarding the task of \"defining respect.\" During this preparation process, we agreed that the definition of respectfulness could be easily understood from two different points of view. First, the definition can be understood as a straightforward use of words that the reader or listener believes are respectful (such as \"Mr. President, can you please give some details about the executive order?\") or as expressing respect, which might include the use of \"bad\" words but is nonetheless an expression of respect (such as, \"you are a badass at making money\"). For this project, we adopted the latter type of example, and we interchangeably refer to it within our work as either \"respectfulness\" or \"expressing respect.\" It may seem obvious that a text entity could be considered to exhibit a range of respectful sentiment; however, studies elaborating on annotation quality in hate speech analysis [6, 40] have demonstrated that, in many cases, humans do not agree on whether a single tweet should be considered racist, and prior studies have rarely gone into detail regarding whether a tweet is \"more or less racist.\" Similarly, we believe that the same tweet could be perceived as either \"respectful\" or \"disrespectful\" by different people and that increasing granularity of the notion of respectfulness causes additional complications. Our data preparation process was consistent with the above statement. Because we observed complications related to annotator agreement before our final annotation process, we held a series of meetings within the annotator group to improve our common understanding and definition of the \"respectfulness\" of text. Ultimately, we decided to simplify the annotation task by not providing regression-like \"respect scores\" as in [23] but instead creating classification labels with only three \"respect\" classes, to limit confusion (Table 1). The adopted set of 5000 tweets had already been released in [39]. When working with the data set, we detected and erased 36 tweets that were not written 100% in English and then labeled the remaining 4964 tweets according to the defined annotation scheme and the following procedure: (1) three annotators independently annotated each data instance; (2) we computed [PERSON]'s alpha annotator agreement of the resulting labels; (3) to obtain a single respectfulness label for each data instance, we retained the label if all three annotators were in agreement (3102 tweets or adopted the label assigned by the majority of annotators (1833 tweets) if the difference between the labels was not greater than 1. In the other cases (29 tweets), a new single annotator was asked to decide and provide a decisive annotation. The data regarding users is anonymized, i.e., it is impossible to connect the given tweets to users automatically. The annotation procedure resulted in 849 disrespectful, 3730 respectful, and 385 very respectful data instances consisting only of text (no tweet-related meta data were used). Example data instances were (original spelling): \"Rest in peace, shipmates.A senseless tragedy, but know your service was not in vain..you made a difference.Condolences to your families.\" and \"@USNavy Surgeon [PERSON] was born on this day in 1784. @NavyHistoryNews #Medical #History_URL\".