Venue: NAACL 2019

# From Legal to Technical Concept: Towards an Automated Classification

of German Political Twitter Postings as Criminal Offenses

 Frederike Zufall\({}^{1}\), Tobias Horsmann\({}^{2}\), Torsten Zesch\({}^{2}\)

\({}^{1}\) Waseda Institute for Advanced Study, Waseda University, Tokyo, Japan

\({}^{2}\) Language Technology Lab, University of Duisburg-Essen, Germany

f.zuffall@kurenai.waseda.jp

{tobias.horsmann|torsten.zesch}@uni-due.de

Equal contribution.

###### Abstract

Advances in the automated detection of offensive Internet postings make this mechanism very attractive to social media companies, who are increasingly under pressure to monitor and action activity on their sites. However, these advances also have important implications as a threat to the fundamental right of free expression. In this article, we analyze which Twitter posts could actually be deemed offenses under German criminal law. German law follows the deductive method of the Roman law tradition based on abstract rules as opposed to the inductive reasoning in Anglo-American common law systems. This allows us to show how legal conclusions can be reached and implemented without relying on existing court decisions. We present a data annotation schema, consisting of a series of binary decisions, for determining whether a specific post would constitute a criminal offense. This schema serves a step towards an inexpensive creation of a sufficient amount of data for automated classification. We find that the majority of posts deemed morally offensive actually do not constitute a criminal offense and still contribute to public discourse. Furthermore, laymen can provide sufficiently reliable data to an expert reference but are, for instance, more lenient in the interpretation of what constitutes a disparaging statement.

+
Footnote †: 1}\) Framework Decision 2008/913/JHA of 28 November 2008 on combating certain forms and expressions of racism and xenophobia by means of criminal law and national laws transposing it.

+
Footnote †: 1}\) Waseda Institute for Advanced Study, Waseda University, Tokyo, Japan

\({}^{2}\) Language Technology Lab, University of Duisburg-Essen, Germany

f.zufall@kurenai.waseda.jp

{tobias.horsmann|torsten.zesch}@uni-due.de

## 1 Introduction

The Internet is frequently used for discussing a variety of topics and an important medium for the exchange of opinions, considered crucial for healthy democratic societies. However, the rough tone in the Internet frequently leads to defamatory or abusive comments in these discussions. The EU has tried to tackle the problem by defining the term 'illegal hate speech'.1 Additionally, in 2017, the European Commission published a communication entitled "Tackling Illegal Content Online' aiming for enhanced responsibility of online platforms.2 Independently from these recent developments on the EU level, Germany adopted the 'Network Enforcement Act3 in 2017. The Act provides for a regulatory framework for 'illegal content4 on social network platforms like Twitter or Facebook. It imposes the obligation on these providers to delete illegal content upon notification within seven days; in case of evidently illegal content within 24 hours.5 From a practical point of view, given the number of statements on social media along with their possible notification, feasibility and accuracy of the required legal assessment becomes an important issue. Natural Language Processing might thus provide the necessary means to assist the legal assessment.

Footnote 1: Framework Decision 2008/913/JHA of 28 November 2008 on combating certain forms and expressions of racism and xenophobia by means of criminal law and national laws transposing it.

Footnote 2: COM(2017) 55 final.

Footnote 3: Nievkerdurchskeuzungsgesetz v. 1.9.2017 (BGBI. I. S. 3352).

Footnote 4: See § 13 (j) ‘rechtswidrige Inhalte’.

Footnote 5: See § 3(2)(2),(3) of the Act. It is however doubtful whether these strict procedural requirements violate EU law, namely Art, J, Art. 14 e-Commerce Directive (2000/31/EC) i.e. require acting ‘expédiously’ after obtaining knowledge.

In this work, we investigate at which point morally offensive statements in social media constitute defamato

atory offenses (SS 185 to SS 187 StGB), which also clarifies the tension between the right to honor and the freedom of expression. Due to its additional complexity, we leave out incitement to hatred against a national, racial, religious or ethnic group or segments of the population (SS 130 StGB) as an offense against public peace in this paper. Furthermore, we investigate automated detection of postings protected by the freedom of expression in order to assist social media moderators. We focus in particular on the process of inexpensive and scalable data annotation, as access to legal expertise is a major bottleneck for providing a sufficient amount of data for classifier training.

## 2 Related Work

An automated detection of Internet discourse in which individuals or groups are verbally attacked has been intensively investigated under a variety of names, for instance: abusive language Waseem et al. (2017), ad hominem arguments Habernal et al. (2018), aggression Kumar et al. (2018), cyberbullying Xu et al. (2012); Macbeth et al. (2013), hate speech Warner and Hirschberg (2012); Ross et al. (2016); Del Vigna et al. (2017), offensive language usage Razavi et al. (2010), profanity Schmidt and Wiegand (2017), threats Oostdijk and van Halteren (2013) or socially unacceptable discourse Fiser et al. (2017).

The majority of the work focuses on the English language with few exceptions for instance for German Ross et al. (2016), Dutch Oostdijk and van Halteren (2013), Italian Del Vigna et al. (2017) or Slovene Fiser et al. (2017). The dataset annotated in Fiser et al. (2017) is the only one that includes a coarse-grained binary annotation category indicating if an utterance violates Slovene law. To the best of our knowledge, automatic determination as to whether the (textual) content of a posting constitutes a criminal offense has never been previously attempted. Previous work focused on detecting postings with socially unacceptable content but without considering actual legal implications for freedom of expression.

Approaches that bring together Natural Language Processing with the legal perspective are in contrast significantly fewer, especially considering the fact that the legal evaluation depends on the applicable legal regime. Previous work focused on predicting the outcome of court trials, which all have in common that they derive their data from a rather large set of court-provided information. Bruninghaus and Ashley (2003) works on a combination of U.S. case law and normative rules: they experiment with clustering and regression models for predicting the outcome of U.S. cases Katz et al. (2017) predicts U.S. supreme court rulings by using a random forest classifier; Kastellec (2010) investigates mappings from case facts to court decisions as outcomes. Waltl et al. (2017) predicts the outcome of decisions in German tax law. Aletras et al. (2016) predicts decisions of the European Court of Human Rights. Deriving data from court decisions might be an approach that is practical if relevant case law exists for the respective legal problem, which particularly makes sense from the perspective of the Anglo-American common law system.8

Footnote 8: ‘Common law’ refers to the Anglo-American legal system that derives the law from judicial decisions, in contrast to the ‘civil law’ system of continental Europe that focuses on the abstraction of legal concepts in codified statutory law. See: B.A. Garner (2001) A Dictionary of Modern Legal Usage (2nd, revised ed.) New York: OUP.

## 3 Operationalising Legal Assessment

Unlike under Anglo-American common law, for legal systems based on Roman law ('civil law' systems), the dogmatic perception of the respective legal disposition lies at the heart of legal decision-making. Our approach thus differs from the above-cited works by placing the focus on the abstract concept of an existing legal norm. The advantage of our approach is therefore that we pursue a solution to address legal problems by creating new data out of abstract legal rules, independently of whether they have been decided by a court. We rely solely on the Internet posting for this consideration, which is the same information available to moderators of social media platforms. To build the bridge from legal thinking to a technical implementation, we start by analyzing the legal requirements for social media content. We find that the decision-making process to determine criminal offenses can be formulated as a sequence of binary decisions when applying the legal dependencies between German criminal law and the fundamental rights of the individual as shown in Figure 1. The derived schema of binary decisions is shown in Figure 2, which we will use in the following section. We now turn to a discussion and analysis of the legal decision process to clarify how we derived this sequence of binary decisions.

 

ScopeSo what constitutes 'illegal content' that the Network Enforcement Act is targeting? The legal definition of the term 'illegal content'9 is referring to offenses stipulated in the German Criminal Code. These references include, inter alia, defamatory offenses in SS 185 to SS 187 StGB10 that cover the criminal punishment of insulting or defamatory statements. Accordingly, if a statement posted on social media fulfills the required elements of these offenses, the provider has the above-described obligation based on the Network Enforcement Act to delete said statement upon notification.11 For this paper, we exclude SS 130 StGB12, that covers incitement to hatred against a national, racial, religious group or a group defined by their ethnic origins, due to an additional complexity of the assessment.

Footnote 9: See § 1(3) ‘rechtswidrige Inhalte’.

Footnote 10: § 185: ‘insult’(_Beléidigung_), § 186: ‘defamation’ _Uble Nachrohed_ and § 187: ‘intentional defamation’ (_Verletumdung_). The reference to these defamatory offenses has however been criticized in literature, see: Erbés/Kohlhaas, Strafrecchtliche Neebengesetze, 220. EL Juli 2018, § 1 NetflzDG, Rn. 16-18.

### The Relevance of Fundamental Rights

To understand their elements in detail, it is crucial to refer to the more general legal concept behind these criminal offenses: as illustrated in Figure 1 the intention behind SS 185 to SS 187 StGB is leading to the protection of the victim's personality right, namely their right to honor under the German Constitution.13 It is this right that is potentially at stake when social media users are disseminating statements with third parties as potential victims.

Figure 1: Conditions for defamatory offenses and fundamental rights’ background under German law





### Defamatory Object

Consequently, the scope of protection of SS 185 to SS 187 StGB follows the respective interpretation of the right to honor. Thus, all three offenses share the approach to the possible victim as a holder of the right to honor: a **living individual** that might be addressed by a name, personal pronoun or user-mentions as shown in Example 1.

```
(a) Are _you_ kidding? (b) _John_ is this true? (c) @user I don't believe you.
```

**Example 1: Addressing living individuals**

A **group of persons** can be considered as a potential victim if that group is distinguishable from the general public such that every member of that group could feel their honor is infringed as shown in Example 2.14

Footnote 14: BGHSt 19, 235, 238.

* [noitemsep]
* (a) _My school's language teachers_ are all idiots. (b) _The female students of this year's grad class_ are all dump. ```

**Example 2: Addressing distinguishable group of persons (highlighted in italic)**

Consequently, only certain groups do qualify as potential defamatory object. Example 3 illustrates groups that would be too broad to be distinguishable from the general public.15

Footnote 15: These groups may, however, still qualify as a potential victim of ‘incientement to hatred’ (§ 130).

* (a) All international conflicts are caused by _men_. (b) _Refugees_ out!!

**Example 3: Counterexamples for addressing too unspecific or large groups**

Collective entities such as governments or press companies with a recognized social role and who act with a collective, single will are included in the right to honor as shown in Example 4.16

Footnote 16: See § 194 StGB; BGHSt 6, 186, 191, 192.

\begin{tabular}{l l} \hline \multicolumn{2}{r}{(a) _The federal government_ is lying.} \\ (b) I don't like _the Christian Democratic Party_ (c) _The New York Times_ got it all wrong. \\ \hline \end{tabular}

**Example 4: Addressing collective entities**

(highlighted in italic)

We translate these conditions of SS 185 to SS 187 StGB into an either/or-question, respectively whether _either_ a living individual _or_ a specific group is an object of the respective statement.

Figure 2: Series of binary decisions for determining criminal offenses under German criminal law



### Defamatory Conduct

Disparaging StatementThe next step in the legal assessment is then the existence of insulting or defamatory conduct with respect to the above-mentioned object, in the form of an expressed disparaging statement. This requirement is again shared by SS 185 to SS 187 StGB. It is already fulfilled by expressing contempt or disrespect through the allegation of shortcomings that could reduce the victim's social standing as shown in Example 5.19

Footnote 19: BVerfGE 61, 1; for Twitter postings: OLG Karlsruhe, 24.10.2018, 6 U 65/18.

(a) John is _an idiot._

(b) The government is _lying._

(c) Minister M _slept during the discussion._

**Example 5: Disparaging statements**

From the perspective of the underlying fundamental rights, it is this disparaging statement which constitutes the interference with the potential victim's right to honor. The existence of a _disparaging statement_ is implemented by a _yes/noquestion_.20

Footnote 20: RG 41, 193; 55, 131; BVerfGE 94, 8.

Value Judgrment or Factual Claim?As simplified in Figure 1, the legal assessment then varies between SS 185 StGB as general disposition and SS 186 and SS 187 StGB with special rules and an increased penalty range.

For the different scope of these dispositions, the difference between the legal terms _'value judgment'_ and _'factual claim'_ (i.e. the assertion of facts, may they be true or untrue) is crucial. Value judgments constitute expressions of personal opinions as shown in Example 6:21

Footnote 21: ByertGE 69, 185, 197; E 97, 381, 403.

(a) _Merkels decisions are bullshit._

(b) _@user I don't like you._

**Example 6: Value judgments**

A factual claim can be clearly classified as true or untrue and is accordingly capable of proof as shown in Example 7.22

Footnote 22: Consequently, we do not implement subsequent conditions of § 186, § 187 StGB, as shown in Figure 1, respectively whether facts cannot be proven to be true (§ 186 StGB) or whether the untruth was intended and known (§ 187 StGB).

\begin{tabular}{l l} \hline \hline \multicolumn{3}{l}{(a) _I saw Mr. A buying drugs_} \\ \multicolumn{3}{l}{_vesterday evening._} \\ \multicolumn{3}{l}{(b) _Minister M slept during the discussion._} \\ \hline \hline \end{tabular}

**Example 7: Factual claim**

SS 185 StGB, stipulating the insult ('Beleidigung'), comprises value judgments and untrue factual claims, irrespective of their dissemination towards third parties. SS 186 and SS 187 StGB on the other side provide for special rules for the assertion or dissemination of untrue facts, i.e. towards third parties. As the publication of statements on social media constitutes an 'assertion' or 'dissemination', untrue facts - for our study - are only treated by SS 186, SS 187 StGB. This reduces the scope of SS 185 StGB to value judgments only.

From the perspective of the right to honor, only untrue factual claims may constitute a violation, while the assertion of true facts is always covered by the freedom of expression.23 The distinction has consequences on the procedural level: because only the assertion of untrue facts violates the right to honor, during criminal proceedings, the court has to assess the truth by taking evidence. A technical implementation of this assessment would therefore require access to unlimited knowledge that goes beyond the textual information on which we work. Accordingly, we stop our examination in case of a factual claim.24

Footnote 23: Ambiguous statements that are based on facts, but are overall characterized by a valuation of these facts, fall under the category of ‘value judgments’.

### Value Judgments: Balancing of Rights

As the distinction between value judgment and factual claim is an alternative decision,25 we continue our implementation for value judgments. In criminal proceedings, the court would have to consider at this point once more fundamental rights: value judgments - being not classifiable as untrue or true - generally fall under the scope of the freedom of expression of the potential offender.25

Footnote 25: Art. 5(1)1 of the German Constitution (Grundgesetz). According to Art. 5(2) the freedom of expression then again is limited by the right to honor.

In the German Criminal Code, this is reflected by SS 193 StGB: even if a statement falls under the scope of said criminal offenses,

justified based on SS 193 StGB as exercise of legitimate interests. The most prominent example of one of these conflicting interests is the offender's freedom of expression. On the constitutional level, then, the decision of whether a social media posting constitutes a punishable criminal offense and leads to the platform provider's deletion obligation can thus ultimately be perceived as a balancing between freedom of expression and the right to honor.

Consequently, the court would have to balance these concurrent rights depending on the case at hand. But how could that balancing, usually comprising an evaluation of various factors, be carried over to a technical implementation? Over the years, German case law from the Federal Constitutional Court has developed guidelines for this balancing to be considered by the judge, which take the step of implying the typical outcome of the balancing. We implement these guidelines in three yes/no-questions:25

Footnote 25: As illustrated in Figure 2, the judge would perform the balancing freely based on all circumstances (which we do not implement) if there is no ‘_subusive insult_’ and if ‘_topic of public interest_’ and ‘_abusive criticism_’ are both yes or both no.

Abusive InsultStatements that constitute breaking a taboo by themselves and intend only the defamation of the victim without any substantiated contribution are classified as _‘abusive insul’__(FormalDeleidigung)_. According to settled case law, these statements are already excluded from the scope of freedom of expression.26 Consequently, a justification based on SS 193 StGB is, in this regard, denied and the elements of SS 185 StGB are fulfilled along with a violation of the right to honor. Given these severe consequences for free speech, the German Constitutional Court has so far only once approved a statement as constituting an 'abusive insult' as shown in Example 8:27

Footnote 26: Maunz/Durig, Grundgesetz-Kommentar, 84. EL August 2018, Art. 5 Abs. 1, Rn. 62.

**Example 8: Abusive insult**

Topic of Public InterestFor statements that contain a contribution to the public discourse with respect to a particular relevant topic of public interest, the settled case law of the German Federal Constitutional Court mandates a presumption in favor of free speech.28

Footnote 27: ByerfGE 86, 1, 45 (“_Krüppel_”).

\begin{tabular}{p{343.7pt}} \hline \hline _Merkel prostitutes herself for the German car industry costing tax payers_ \\ \hline \hline \end{tabular}

Example 9: Topic of public interest

Example 9 comments on the right to stay of refugees, by this participating to the public debate in Germany about refugees from Syria. Accordingly, such statements usually outweigh the right to honor. They thus usually can be made, justified as having a legitimate interest based on SS 193 StGB, therefore _usually not punishable_.

Abusive CriticismFinally, as _'abusive criticism'__(Schmahkritik)_ settled case law has defined statements that go beyond plausible criticism by primarily intending to abusively offend the victim, hereby neglecting a substantiated contribution.29 In Example 10, the statement:

Footnote 29: ByerfGE 61, 1, 12; E § 2, 272 (284); for Twitter postings: LG Berlin, 13.10.2012, 33 0 434/11.

**Example 10: No abusive criticism**

despite the word 'asshole', still contributes to the public discourse, which is why its primary purpose is not (only) to abusively offend. Abusive criticism thus usually leads to favoring the right to honor over freedom of expression. Without justification pursuant to SS 193 StGB, such statements are therefore _usually punishable_.

## 4 Proof of Concept

In this section, we now use the schema in Figure 2 to annotate data and learn more about the reliability of an automated classification.

### Dataset

In order to legally assess social media postings, we first need to annotate a corpus as a starting point for an analysis. Randomly sampling postings from the Internet is a possible strategy to collect data for an annotation, but we would not have any certainty that enough offending postings occur. Therefore, we decide to use an existing corpus that has already been annotated for moral offensiveness. We use the corpus provided by the GermEval shared 

task for _detecting offensive language usage_[14]. This dataset contains a mixture of German Twitter postings with a focus on German politics that are marked if the tweet is considered morally offensive from the subjective perception of the annotator. We work with a subset of 1,100 postings from this corpus, two-thirds of the postings (844) are marked as morally offensive. This enables us to investigate which statements commonly found in political debates are protected by the freedom of expression and which are not.

AnnotationThe reference annotation of these postings is provided by a fully-qualified lawyer of German law applying the schema in Figure 2. We additionally received 200 postings from a second fully-qualified lawyer in order to compute an agreement score between the two legal experts, which is shown in Table 1. We report accuracy and Cohen's \(\kappa\)[3] for each decision and show the agreement for a _joint-decision_ where we treat all decisions for a posting as a single decision. The legal experts disagree slightly on the assumption of abusive criticism. This is not surprising as the evaluation of courts might differ in different instances, especially regarding the balancing of interests in the case at hand.

AnalysisFigure 3 shows the annotation results of the postings marked as _morally offensive_. We find that about half of the postings have to be categorized early on as _not punishable_ for not containing a defamatory object, i.e. no living individual addressed or the addressed group is too unspecific. The remaining half is still to a large extent _usually not punishable_, mostly because the posts still contribute to a topic of public interest, despite of being disparaging. A small number of cases are either factual claims that would require taking evidence by the court or value judgments that do not concern topics of public interest. Thus, despite containing statements that may be deemed morally offensive, the vast majority of statements are legally acceptable, i.e. protected by the freedom of expression. The punishable cases often contain insulting buzzwords such as _slut, fat-ass_ or _scumbag_ when directed at a private individual, not at a person of public interest. Furthermore, punishable statements addressing a specific group use more frequently offending comparisons or descriptions but no typical single or two-word insults. However, it is important to recall that the dataset has a focus on political debates. Accordingly, most statements tackle a topic of public interest, and are thus considered _usually not punishable_ granting a high degree of protection under the freedom of expression.

This analysis also shows that shared tasks such as _OffensEval_[5] tackle essentially only one step in the legal assessment, namely whether a statement is _disparagging_. Thus, they fall short of valuing the freedom of expression, which is in particular a problem for public discourse such as political debates, where opinions are often accompanied by 'bad' language.

### Automated Detection

For an automated detection, it would seem straight forward to distinguish between _punishable_ and _not punishable_ postings. This approach requires an extremely large amount of data for each of the two classes, which we do not have. The data distribution is skewed with the _punishable_ class being extremely small, which makes this direct ap

\begin{table}
\begin{tabular}{l c r} \hline \hline  & \multicolumn{2}{c}{Agreement} \\ Decision & Acc & Cohen’s \(\kappa\) \\ \hline Living individual & .985 & .961 \\ Specific group & .940 & .809 \\ Disparaging & .925 & .867 \\ Factual claim & .925 & .821 \\ Abusive insult & .925 & .820 \\ Of public interest & .940 & .855 \\ Abusive criticism & .866 & .678 \\ \hline Joint-decision & .821 & .720 \\ \hline \end{tabular}
\end{table}
Table 1: Agreement between two legal experts on two-hundred randomly selected postings

Figure 3: Legal categorization of annotated Tweets that were marked as containing an offense



proach infeasible. Instead, we investigate how well each of the binary decisions shown in Figure 2 can be learned independently, which has a less skewed distribution. We use a bi-directional LSTM Hochreiter and Schmidhuber (1997) for classification.30 We use the 300-dimensional German pre-trained word embeddings provided by Grave et al. (2018), which are trained on the German common crawl.

Footnote 30: We train 30 epochs, with 0.2 dropout and initialize using Glorot, and ReLU as activation function.

Table 2 shows averaged 10-fold CV results for each decision point. We observe that the accuracy is close to the underlying distribution of the two classes. The classification of the _defamatory object_ has a mediocre performance. In particular, an insufficient coverage of group names and names of individuals in the dataset seem to be the main cause as the _no_ classes usually perform considerably better than the _yes_ classes. The classification of the decisions under _definamatory conduct_ follows a similar trend. The few positive instances: _factual claim_, _abusive insult_ and _abusive criticism_ prevent a reliable distinction of these cases.

The next step would be to investigate how well the classification works in sequence, i.e. continuing the classification with the positively categorized instances of the previous step. However, the independent classification shows already that the amount of data is insufficient. Therefore, we turn next to the more pressing question of how to generate more data in a scalable way, especially without relying on expensive legal experts as annotators.

## 5 Data Annotation by Laymen

A scalable annotation of more data requires that laymen can be instructed in a way that enables them to solve the task at hand. Laymen are readily available, for instance via crowdsourcing but also as student assistants who can be more cheaply employed than legal experts for annotating data.

SetupWe compare the annotation performance of both random crowd workers and student assistants. The crowd workers and the student assistants were required to speak German. We have no information on the educational background of the crowd workers, but we ensured that the student assistants were not students of law-related subjects. We prepared a simplified manual31 based on Figure 2, which is supplemented with text examples for each decision to guide the layman through the annotation of each decision. We use the crowdsourcing platform figure-eight.com to let crowd workers and student assistants re-annotate the 1,100 postings for which we have a reference annotation by a legal expert. Each posting is annotated by three annotators.

Footnote 31: github.com/Horsmann/NAACL-2019–legal

The annotation results are shown in Table 3. It is to be expected that some annotators will perform better than others, but distinguishing the 'good' from the 'bad' is an additional challenge, which we will not deal with here. Instead, we aggregate the annotations of all participants in a voting-like fashion, taking in each case the majority vote for each decision.32 This provides us with an approximation of the average layman performance on this task, which is the key information that we are interested in.

Footnote 32: We restrict the comparisons to postings for which we have three votes of the respective sub-group.

AnalysisThe results show that student assistants solve this task considerably better than crowd workers. In particular, the recognition of references to _specific group_ poses the biggest challenges for crowd workers, which also explains why this group performs much more poorly than the student assistants. As shown in Figure 2, the evaluation for a post ends if neither a living individual nor specific group is addressed. If either of the first two decisions is incorrect, an annotator automatically makes up to five additional follow-up errors. The student assistants applied the manual considerably more consistently than the crowd

\begin{table}
\begin{tabular}{c l c r r} \hline \hline  & Decision & Acc & Class & F1 \\ \hline \multirow{4}{*}{Credit} & \multirow{2}{*}{Living individual} & \multirow{2}{*}{.794} & Yes & .685 \\  & & & No & .847 \\  & & & Yes & .431 \\  & & & No & .903 \\  & _either of the above_ & & Yes & .775 \\  & & & No & .702 \\ \hline \multirow{4}{*}{Credit} & \multirow{2}{*}{Disparaging} & \multirow{2}{*}{.727} & Yes & .656 \\  & & & No & .774 \\ \cline{1-1}  & & & Yes & .000 \\ \cline{1-1}  & & & No & .989 \\ \cline{1-1}  & & Abusive insult & .984 & Yes & .000 \\ \cline{1-1}  & & & No & .992 \\ \cline{1-1}  & & Is of public interest & .715 & Yes & .514 \\ \cline{1-1}  & & & No & .798 \\ \cline{1-1}  & & Abusive criticism & .952 & Yes & .036 \\ \cline{1-1}  & & & No & .975 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Averaged 10-fold CV results for each decision on 1,100 Tweets, using an LSTM 

workers, leading to fewer follow-up errors. Determining the referenced individual is also frequently challenging when several Twitter users are referenced by an at-mention, which introduces an uncertainty that the statement might refer to one of the linked users. We also find that the laymen tend to apply a more lenient interpretation of what is _disparaging_ and consider many statements as non-disparagging, i.e. already an allegation of shortcomings33, which could reduce the victim's social standing is disparaging in the legal sense.

Footnote 33: e.g. _I am not sure whether John knows what he’s doing_.

The annotation results of the student assistants are encouraging for obtaining sufficient training data for a larger study on automated classification, i.e. a correct automated classification of the first two decisions would already be able to exclude many cases that do not have to be deleted based on the Network Enforcement Act.

## 6 Conclusion

We investigated which offenses found in German political Tweets constitute defamatory offenses under German criminal law, that social media operators are obliged to delete under the Network Enforcement Act. Following the dogmatic approach of civil law systems, we started with an analysis of the legal framework for defamatory offenses in the German Criminal Code along with its foundations in the balancing between the potential offender's freedom of expression and the potential victim's right to honor. We derived from this consideration a schema suited for data annotation consisting of a sequence of binary decisions to determine if a statement constituted a defamatory offense, which we used for annotating data. We find that the majority of the morally offensive postings in our dataset still contribute to the public discourse and are, hence, protected by the freedom of expression. We also investigated if laymen can be instructed to use this annotation schema to facilitate an inexpensive annotation of more data for classifier training. We find that laymen suited to the task can be found, but in particular the legal notions of a specific _group of persons_ and the scope of what is considered _disparaging_ are challenging for them.

In future work, we will investigate the usefulness of layman-annotated data for an automated classification. Furthermore, we will expand our work by investigating additionally the criminal of-fense of _incitement to hatred_ (SS 130 StGB) and its implication on the freedom of expression.

### Acknowledgements

We would like to thank Tatiana Gunster for taking the time to provide us with a second legal opinion. Furthermore, we would like to thank Emilie Mathieu for helpful corrections and Michael Wojatzki for the helpful discussions about the userstudy design.

\begin{table}
\begin{tabular}{l c c|c c|c c} \hline \hline  & \multicolumn{4}{c}{Crowd-} & Student \\  & \multicolumn{2}{c}{All users} & \multicolumn{2}{c}{workers} & \multicolumn{2}{c}{assistants} \\ Decision & Acc & \(\kappa\) & Acc & \(\kappa\) & Acc & \(\kappa\) \\ \hline Living individual & .822 & .628 & .800 & .555 & .826 & .655 \\ Specific group & .745 & .357 & .600 & .192 & .817 & .502 \\ Disparaging & .649 & .401 & .475 & .158 & .765 & .574 \\ Factual claim & .654 & .381 & .492 & .131 & .774 & .577 \\ Abusive insult & .590 & .285 & .400 & .005 & .669 & .436 \\ Is of public interest & .672 & .263 & .592 & -.043 & .691 & .388 \\ Abusive criticism & .589 & .161 & .575 & -.049 & .530 & .224 \\ \hline Joint-decision & .357 & .201 & .175 & .050 & .383 & .250 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Agreement between the reference annotation by a legal expert and the aggregated laymen annotations of: _all users_ (on 1,000 posts), only _crowd-workers_ (on 402 posts) and only _student assistants_ (on 390 posts). Results for _crowd-workers_ and _student-assistants_ are limited to postings where all three votes per posting were provided by users from the respective group.

 



## References

* Aletras et al. (2016) Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. _PeerJ Computer Science_.
* Bruninghaus and Ashley (2003) Stefanie Bruninghaus and Kevin D. Ashley. 2003. Predicting Outcomes of Case Based Legal Arguments. In _Proceedings of the International Conference on Artificial Intelligence and Law_, pages 233-242, New York, NY, USA. ACM.
* Cohen (1960) Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. _Educational and Psychological Measurement_, 20(1):37-46.
* Vigna et al. (2017) Fabio Del Vigna, Andrea Cimino, Felice Dell'Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate Me, Hate Mc Not: Hate Speech Detection on Facebook. In _Proceedings of the First Italian Conference on Cybersecurity (ITASEC'17)_, pages 86-95.
* Fiser et al. (2017) Darja Fiser, Tomaz Erjavec, and Nikola Ljubesic. 2017. Legal framework, dataset and annotation schema for socially unacceptable online discourse practices in slovene. In _Proceedings of the First Workshop on Abusive Language Online_, pages 46-51. Association for Computational Linguistics.
* Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning Word Vectors for 157 Languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Habernal et al. (2018) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before Name-Calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 386-396. Association for Computational Linguistics.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_, pages 1735-1780.
* Kastellec (2010) Jonathan Kastellec. 2010. The Statistical Analysis of Judicial Decisions and Legal Rules with Classification Trees. _Journal of Empirical Legal Studies_, 7(2):202-230.
* Katz et al. (2017) Daniel Martin Katz, Michael J. Bommarito, II, and Josh Blackman. 2017. A general approach for predicting the behavior of the Supreme Court of the United States. _PLOS ONE_, 12(4):1-18.
* Kumar et al. (2018) Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking Aggression Identification in Social Media. In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_, pages 1-11. Association for Computational Linguistics.
* Macbeth et al. (2013) Jamie Macbeth, Hanna Adeyema, Henry Lieberman, and Christopher Fry. 2013. Script-based story matching for cyberbullying prevention. In _ACM SIGCHI Conference on Human Factors in Computing Systems_, pages 901-906.
* Osostdijk and van Halteren (2013) Nelleke Osostdijk and Hans van Halteren. 2013. N-Gram-Based Recognition of Threatening Tweets. In _Computational Linguistics and Intelligent Text Processing_, pages 183-196, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Razavi et al. (2010) Amir H. Razavi, Diana Inkpen, Sasha Uritsky, and Stan Matwin. 2010. Offensive language detection using multi-level classification. In _Proceedings of the 23rd Canadian Conference on Advances in Artificial Intelligence_, pages 16-27, Berlin, Heidelberg. Springer-Verlag.
* Ross et al. (2016) Bjorn Ross, Michael Rist, Guillermo Carbonell, Ben Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In _Proceedings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication_, pages 6-9.
* Ruppenhofer et al. (2018) Josef Ruppenhofer, Melanie Siegel, and Michael Wiegand. 2018. GermEval 2018: Shared Task on the Identification of Offensive Language. In _Proceedings of the GermanEval 2018: Shared Task on the Identification of Offensive Language_, Vienna, Austria.
* Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection using Natural Language Processing. In _Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media_, pages 1-10. Association for Computational Linguistics.
* Waltl et al. (2017) Bernhard Waltl, Georg Bonczek, Elena Seepankova, Jorg Landthaler, and Florian Matthes. 2017. Predicting the Outcome of Appeal Decisions in Germany's Tax Law. In _Electronic Participation_, pages 89-99, Cham. Springer International Publishing.
* Warner and Hirschberg (2012) William Warner and Julia Hirschberg. 2012. Detecting Hate Speech on the World Wide Web. In _Proceedings of the Second Workshop on Language in Social Media_, pages 19-26, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Waseem et al. (2017) Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In _Proceedings of the First Workshop on Abusive Language Online_, pages 78-84. Association for Computational Linguistics

Jun-Ming Xu, Kwang-Sung Jun, Xiaojin Zhu, and Amy Bellmore. 2012. Learning from Bullying Traces in Social Media. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, NAACL HLT '12, pages 656-666, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In _Proceedings of The International Workshop on Semantic Evaluation (SemEval)_.