Introduction
Offensive language in user-generated content on online platforms and its implications has been gaining attention over the last couple of years. This interest is sparked by the fact that many of the online social media platforms have come under scrutiny on how this type of content should be detected and dealt with. It is, however, far from trivial to deal with this type of language directly due to the gigantic amount of user-generated content created every day. For this reason, automatic methods are required, using natural language processing (NLP) and machine learning techniques.
Given the fact that the research on offensive language detection has to a large extent been focused on the English language, we set out to explore the design of models that can successfully be used for both English and Danish. To accomplish this, an appropriate dataset must be constructed, annotated with the guidelines described in BIBREF0 . We, furthermore, set out to analyze the linguistic features that prove hard to detect by analyzing the patterns that prove hard to detect.
Background
Offensive language varies greatly, ranging from simple profanity to much more severe types of language. One of the more troublesome types of language is hate speech and the presence of hate speech on social media platforms has been shown to be in correlation with hate crimes in real life settings BIBREF1 . It can be quite hard to distinguish between generally offensive language and hate speech as few universal definitions exist BIBREF2 . There does, however, seem to be a general consensus that hate speech can be defined as language that targets a group with the intent to be harmful or to cause social chaos. This targeting is usually done on the basis of some characteristics such as race, color, ethnicity, gender, sexual orientation, nationality or religion BIBREF3 . In section "Background" , hate speech is defined in more detail. Offensive language, on the other hand, is a more general category containing any type of profanity or insult. Hate speech can, therefore, be classified as a subset of offensive language. BIBREF0 propose guidelines for classifying offensive language as well as the type and the target of offensive language. These guidelines capture the characteristics of generally offensive language, hate speech and other types of targeted offensive language such as cyberbullying. However, despite offensive language detection being a burgeoning field, no dataset yet exists for Danish BIBREF4 despite this phenomenon being present BIBREF5 .
Many different sub-tasks have been considered in the literature on offensive and harmful language detection, ranging from the detection of general offensive language to more refined tasks such as hate speech detection BIBREF2 , and cyberbullying detection BIBREF6 .
A key aspect in the research of automatic classification methods for language of any kind is having substantial amount of high quality data that reflects the goal of the task at hand, and that also contains a decent amount of samples belonging to each of the classes being considered. To approach this problem as a supervised classification task the data needs to be annotated according to a well-defined annotation schema that clearly reflects the problem statement. The quality of the data is of vital importance, since low quality data is unlikely to provide meaningful results. Cyberbullying is commonly defined as targeted insults or threats against an individual BIBREF0 . Three factors are mentioned as indicators of cyberbullying BIBREF6 : intent to cause harm, repetitiveness, and an imbalance of power. This type of online harassment most commonly occurs between children and teenagers, and cyberbullying acts are prohibited by law in several countries, as well as many of the US states BIBREF7 .
BIBREF8 focus on classifying cyberbullying events in Dutch. They define cyberbullying as textual content that is published online by an individual and is aggressive or hurtful against a victim. The annotation-schema used consists of two steps. In the first step, a three-point harmfulness score is assigned to each post as well as a category denoting the authors role (i.e. harasser, victim, or bystander). In the second step a more refined categorization is applied, by annotating the posts using the the following labels: Threat/Blackmail, Insult, Curse/Exclusion, Defamation, Sexual Talk, Defense, and Encouragement to the harasser. Hate Speech. As discussed in Section "Classification Structure" , hate speech is generally defined as language that is targeted towards a group, with the intend to be harmful or cause social chaos. This targeting is usually based on characteristics such as race, color, ethnicity, gender, sexual orientation, nationality or religion BIBREF3 . Hate speech is prohibited by law in many countries, although the definitions may vary. In article 20 of the International Covenant on Civil and Political Rights (ICCPR) it is stated that "Any advocacy of national, racial or religious hatred that constitutes incitement to discrimination, hostility or violence shall be prohibited by law" BIBREF9 . In Denmark, hate speech is prohibited by law, and is formally defined as public statements where a group is threatened, insulted, or degraded on the basis of characteristics such as nationality, ethnicity, religion, or sexual orientation BIBREF10 . Hate speech is generally prohibited by law in the European Union, where it is defined as public incitement to violence or hatred directed against a group defined on the basis of characteristics such as race, religion, and national or ethnic origin BIBREF11 . Hate speech is, however, not prohibited by law in the United States. This is due to the fact that hate speech is protected by the freedom of speech act in the First Amendment of the U.S. Constitution BIBREF12 .
BIBREF2 focus is on classifying hate speech by distinguishing between general offensive language and hate speech. They define hate speech as "language that is used to express hatred towards a targeted group or is intended to be derogatory, to humiliate, or to insult the members of the group". They argue that the high use of profanity on social media makes it vitally important to be able to effectively distinguish between generally offensive language and the more severe hate speech. The dataset is constructed by gathering data from Twitter, using a hate speech lexicon to query the data with crowdsourced annotations.
Contradicting definitions. It becomes clear that one of the key challenges in doing meaningful research on the topic are the differences in both the annotation-schemas and the definitions used, since it makes it difficult to effectively compare results to existing work, as pointed out by several authors ( BIBREF13 , BIBREF3 , BIBREF14 , BIBREF0 ). These issues become clear when comparing the work of BIBREF6 , where racist and sexist remarks are classified as a subset of insults, to the work of BIBREF15 , where similar remarks are split into two categories; hate speech and derogatory language. Another clear example of conflicting definitions becomes visible when comparing BIBREF16 , where hate speech is considered without any consideration of overlaps with the more general type of offensive language, to BIBREF2 where a clear distinction is made between the two, by classifying posts as either Hate speech, Offensive or Neither. This lack of consensus led BIBREF14 to propose annotation guidelines and introduce a typology. BIBREF17 argue that these proposed guidelines do not effectively capture both the type and target of the offensive language.
Dataset
In this section we give a comprehensive overview of the structure of the task and describe the dataset provided in BIBREF0 . Our work adopts this framing of the offensive language phenomenon.
Classification Structure
Offensive content is broken into three sub-tasks to be able to effectively identify both the type and the target of the offensive posts. These three sub-tasks are chosen with the objective of being able to capture different types of offensive language, such as hate speech and cyberbullying (section "Background" ).
In sub-task A the goal is to classify posts as either offensive or not. Offensive posts include insults and threats as well as any form of untargeted profanity BIBREF17 . Each sample is annotated with one of the following labels:
In English this could be a post such as #TheNunMovie was just as scary as I thought it would be. Clearly the critics don't think she is terrifyingly creepy. I like how it ties in with #TheConjuring series. In Danish this could be a post such as Kim Larsen var god, men hans død blev alt for hyped.
. In English this could be a post such as USER is a #pervert himself!. In Danish this could be a post such as Kalle er faggot...
In sub-task B the goal is to classify the type of offensive language by determining if the offensive language is targeted or not. Targeted offensive language contains insults and threats to an individual, group, or others BIBREF17 . Untargeted posts contain general profanity while not clearly targeting anyone BIBREF17 . Only posts labeled as offensive (OFF) in sub-task A are considered in this task. Each sample is annotated with one of the following labels:
Targeted Insult (TIN). In English this could be a post such as @USER Please ban this cheating scum. In Danish this could be e.g. Hun skal da selv have 99 år, den smatso.
Untargeted (UNT). In English this could be a post such as 2 weeks of resp done and I still don't know shit my ass still on vacation mode. In Danish this could e.g. Dumme svin...
In sub-task C the goal is to classify the target of the offensive language. Only posts labeled as targeted insults (TIN) in sub-task B are considered in this task BIBREF17 . Samples are annotated with one of the following:
Individual (IND): Posts targeting a named or unnamed person that is part of the conversation. In English this could be a post such as @USER Is a FRAUD Female @USER group paid for and organized by @USER. In Danish this could be a post such as USER du er sku da syg i hoved. These examples further demonstrate that this category captures the characteristics of cyberbullying, as it is defined in section "Background" .
Group (GRP): Posts targeting a group of people based on ethnicity, gender or sexual orientation, political affiliation, religious belief, or other characteristics. In English this could be a post such as #Antifa are mentally unstable cowards, pretending to be relevant. In Danish this could be e.g. Åh nej! Svensk lorteret!
Other (OTH): The target of the offensive language does not fit the criteria of either of the previous two categories. BIBREF17 . In English this could be a post such as And these entertainment agencies just gonna have to be an ass about it.. In Danish this could be a post such as Netto er jo et tempel over lort.
One of the main concerns when it comes to collecting data for the task of offensive language detection is to find high quality sources of user-generated content that represent each class in the annotation-schema to some extent. In our exploration phase we considered various social media platforms such as Twitter, Facebook, and Reddit.
We consider three social media sites as data.
Twitter. Twitter has been used extensively as a source of user-generated content and it was the first source considered in our initial data collection phase. The platform provides excellent interface for developers making it easy to gather substantial amounts of data with limited efforts. However, Twitter was not a suitable source of data for our task. This is due to the fact that Twitter has limited usage in Denmark, resulting in low quality data with many classes of interest unrepresented.
Facebook. We next considered Facebook, and the public page for the Danish media company Ekstra Bladet. We looked at user-generated comments on articles posted by Ekstra Bladet, and initial analysis of these comments showed great promise as they have a high degree of variation. The user behaviour on the page and the language used ranges from neutral language to very aggressive, where some users pour out sexist, racist and generally hateful language. We faced obstacles when collecting data from Facebook, due to the fact that Facebook recently made the decision to shut down all access to public pages through their developer interface. This makes computational data collection approaches impossible. We faced restrictions on scraping public pages with Facebook, and turned to manual collection of randomly selected user-generated comments from Ekstra Bladet's public page, yielding 800 comments of sufficient quality.
Reddit. Given that language classification tasks in general require substantial amounts of data, our exploration for suitable sources continued and our search next led us to Reddit. We scraped Reddit, collecting the top 500 posts from the Danish sub-reddits r/DANMAG and r/Denmark, as well as the user comments contained within each post.
We published a survey on Reddit asking Danish speaking users to suggest offensive, sexist, and racist terms for a lexicon. Language and user behaviour varies between platforms, so the goal is to capture platform-specific terms. This gave 113 offensive and hateful terms which were used to find offensive comments. The remainder of comments in the corpus were shuffled and a subset of this corpus was then used to fill the remainder of the final dataset. The resulting dataset contains 3600 user-generated comments, 800 from Ekstra Bladet on Facebook, 1400 from r/DANMAG and 1400 from r/Denmark. In light of the General Data Protection Regulations in Europe (GDPR) and the increased concern for online privacy, we applied some necessary pre-processing steps on our dataset to ensure the privacy of the authors of the comments that were used. Personally identifying content (such as the names of individuals, not including celebrity names) was removed. This was handled by replacing each name of an individual (i.e. author or subject) with @USER, as presented in both BIBREF0 and BIBREF2 . All comments containing any sensitive information were removed. We classify sensitive information as any information that can be used to uniquely identify someone by the following characteristics; racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, genetic data, and bio-metric data.
We base our annotation procedure on the guidelines and schemas presented in BIBREF0 , discussed in detail in section "Classification Structure" . As a warm-up procedure, the first 100 posts were annotated by two annotators (the author and the supervisor) and the results compared. This was used as an opportunity to refine the mutual understanding of the task at hand and to discuss the mismatches in these annotations for each sub-task.
We used a Jaccard index BIBREF18 to assess the similarity of our annotations. In sub-task A the Jaccard index of these initial 100 posts was 41.9%, 39.1% for sub-task B , and 42.8% for sub-task C. After some analysis of these results and the posts that we disagreed on it became obvious that to a large extent the disagreement was mainly caused by two reasons:
Guesswork of the context where the post itself was too vague to make a decisive decision on whether it was offensive or not without more context. An example of this is a post such as Skal de hjælpes hjem, næ nej de skal sendes hjem, where one might conclude, given the current political climate, that this is an offensive post targeted at immigrants. The context is, however, lacking so we cannot make a decisive decision. This post should, therefore, be labeled as non-offensive, since the post does not contain any profanity or a clearly stated group.
Failure to label posts containing some kind of profanity as offensive (typically when the posts themselves were not aggressive, harmful, or hateful). An example could be a post like @USER sgu da ikke hans skyld at hun ikke han finde ud af at koge fucking pasta, where the post itself is rather mild, but the presence of fucking makes this an offensive post according to our definitions.
In light of these findings our internal guidelines were refined so that no post should be labeled as offensive by interpreting any context that is not directly visible in the post itself and that any post containing any form of profanity should automatically be labeled as offensive. These stricter guidelines made the annotation procedure considerably easier while ensuring consistency. The remainder of the annotation task was performed by the author, resulting in 3600 annotated samples.
Final Dataset
In Table 1 the distribution of samples by sources in our final dataset is presented. Although a useful tool, using the hate speech lexicon as a filter only resulted in 232 comments. The remaining comments from Reddit were then randomly sampled from the remaining corpus.
The fully annotated dataset was split into a train and test set, while maintaining the distribution of labels from the original dataset. The training set contains 80% of the samples, and the test set contains 20%. Table 2 presents the distribution of samples by label for both the train and test set. The dataset is skewed, with around 88% of the posts labeled as not offensive (NOT). This is, however, generally the case when it comes to user-generated content on online platforms, and any automatic detection system needs be able to handle the problem of imbalanced data in order to be truly effective.
Features
One of the most important factors to consider when it comes to automatic classification tasks the the feature representation. This section discusses various representations used in the abusive language detection literature.
Top-level features. In BIBREF3 information comes from top-level features such as bag-of-words, uni-grams and more complex n-grams, and the literature certainly supports this. In their work on cyberbullying detection, BIBREF8 use word n-grams, character n-grams, and bag-of-words. They report uni-gram bag-of-word features as most predictive, followed by character tri-gram bag-of-words. Later work finds character n-grams are the most helpful features BIBREF15 , underlying the need for the modeling of un-normalized text. these simple top-level feature approaches are good but not without their limitations, since they often have high recall but lead to high rate of false positives BIBREF2 . This is due to the fact that the presence of certain terms can easily lead to misclassification when using these types of features. Many words, however, do not clearly indicate which category the text sample belongs to, e.g. the word gay can be used in both neutral and offensive contexts.
Linguistic Features BIBREF15 use a number of linguistic features, including the length of samples, average word lengths, number of periods and question marks, number of capitalized letters, number of URLs, number of polite words, number of unknown words (by using an English dictionary), and number of insults and hate speech words. Although these features have not proven to provide much value on their own, they have been shown to be a good addition to the overall feature space BIBREF15 .
Word Representations. Top-level features often require the predictive words to occur in both the training set and the test sets, as discussed in BIBREF3 . For this reason, some sort of word generalization is required. BIBREF15 explore three types of embedding-derived features. First, they explore pre-trained embeddings derived from a large corpus of news samples. Secondly, they use word2vec BIBREF19 to generate word embeddings using their own corpus of text samples. We use both approaches. Both the pre-trained and word2vec models represent each word as a 200 dimensional distributed real number vector. Lastly, they develop 100 dimensional comment2vec model, based on the work of BIBREF20 . Their results show that the comment2vec and the word2vec models provide the most predictive features BIBREF15 . In BIBREF21 they experiment with pre-trained GloVe embeddings BIBREF22 , learned FastText embeddings BIBREF23 , and randomly initialized learned embeddings. Interestingly, the randomly initialized embeddings slightly outperform the others BIBREF21 .
Sentiment Scores. Sentiment scores are a common addition to the feature space of classification systems dealing with offensive and hateful speech. In our work we experiment with sentiment scores and some of our models rely on them as a dimension in their feature space. To compute these sentiment score features our systems use two Python libraries: VADER BIBREF24 and AFINN BIBREF25 .Our models use the compound attribute, which gives a normalized sum of sentiment scores over all words in the sample. The compound attribute ranges from $-1$ (extremely negative) to $+1$ (extremely positive).
Reading Ease. As well as some of the top-level features mentioned so far, we also use Flesch-Kincaid Grade Level and Flesch Reading Ease scores. The Flesch-Kincaid Grade Level is a metric assessing the level of reading ability required to easily understand a sample of text.
Models
We introduce a variety of models in our work to compare different approaches to the task at hand. First of all, we introduce naive baselines that simply classify each sample as one of the categories of interest (based on BIBREF0 ). Next, we introduce a logistic regression model based on the work of BIBREF2 , using the same set of features as introduced there. Finally, we introduce three deep learning models: Learned-BiLSTM, Fast-BiLSTM, and AUX-Fast-BiLSTM. The logistic regression model is built using Scikit Learn BIBREF26 and the deep learning models are built using Keras BIBREF27 . The following sections describe these model architectures in detail, the algorithms they are based on, and the features they use.
Results and Analysis
For each sub-task (A, B, and C, Section "Classification Structure" ) we present results for all methods in each language.
A - Offensive language identification:
English. For English (Table 3 ) Fast-BiLSTM performs best, trained for 100 epochs, using the OLID dataset. The model achieves a macro averaged F1-score of $0.735$ . This result is comparable to the BiLSTM based methods in OffensEval.
Additional training data from HSAOFL BIBREF2 does not consistently improve results. For the models using word embeddings results are worse with additional training data. On the other hand, for models that use a range of additional features (Logistic Regression and AUX-Fast-BiLSTM), the additional training data helps.
Danish. Results are in Table 4 . Logistic Regression works best with an F1-score of $0.699$ . This is the second best performing model for English, though the best performing model for English (Fast-BiLSTM) is worst for Danish.
Best results are given in Table 5 . The low scores for Danish compared to English may be explained by the low amount of data in the Danish dataset. The Danish training set contains $2,879$ samples (table 2 ) while the English training set contains $13,240$ sample.Futher, in the English dataset around $33\%$ of the samples are labeled offensive while in the Danish set this rate is only at around $12\%$ . The effect that this under represented class has on the Danish classification task can be seen in more detail in Table 5 .
B - Categorization of offensive language type
English. In Table 6 the results are presented for sub-task B on English. The Learned-BiLSTM model trained for 60 epochs performs the best, obtaining a macro F1-score of $0.619$ .
Recall and precision scores are lower for UNT than TIN (Table 5 ). One reason is skew in the data, with only around $14\%$ of the posts labeled as UNT. The pre-trained embedding model, Fast-BiLSTM, performs the worst, with a macro averaged F1-score of $0.567$ . This indicates this approach is not good for detecting subtle differences in offensive samples in skewed data, while more complex feature models perform better.
Danish. Table 7 presents the results for sub-task B and the Danish language. The best performing system is the AUX-Fast-BiLSTM model (section UID26 ) trained for 100 epochs, which obtains an impressive macro F1-score of $0.729$ . This suggests that models that only rely on pre-trained word embeddings may not be optimal for this task. This is be considered alongside the indication in Section "Final Dataset" that relying on lexicon-based selection also performs poorly.
The limiting factor seems to be recall for the UNT category (Table 8 ). As mentioned in Section "Background" , the best performing system for sub-task B in OffensEval was a rule-based system, suggesting that more refined features, (e.g. lexica) may improve performance on this task. The better performance of models for Danish over English can most likely be explained by the fact that the training set used for Danish is more balanced, with around $42\%$ of the posts labeled as UNT.
C - Offensive language target identification
English. The results for sub-task C and the English language are presented in Table 9 . The best performing system is the Learned-BiLSTM model (section UID24 ) trained for 10 epochs, obtaining a macro averaged F1-score of $0.557$ . This is an improvement over the models introduced in BIBREF0 , where the BiLSTM based model achieves a macro F1-score of $0.470$ .
The main limitations of our model seems to be in the classification of OTH samples, as seen in Table 11 . This may be explained by the imbalance in the training data. It is interesting to see that this imbalance does not effect the GRP category as much, which only constitutes about $28\%$ of the training samples. One cause for the differences in these, is the fact that the definitions of the OTH category are vague, capturing all samples that do not belong to the previous two.
Danish. Table 10 presents the results for sub-task C and the Danish language. The best performing system is the same as in English, the Learned-BiLSTM model (section UID24 ), trained for 100 epochs, obtaining a macro averaged F1-score of $0.629$ . Given that this is the same model as the one that performed the best for English, this further indicates that task specific embeddings are helpful for more refined classification tasks.
It is interesting to see that both of the models using the additional set of features (Logistic Regression and AUX-Fast-BiLSTM) perform the worst. This indicates that these additional features are not beneficial for this more refined sub-task in Danish. The amount of samples used in training for this sub-task is very low. Imbalance does have as much effect for Danish as it does in English, as can be seen in Table 11 . Only about $14\%$ of the samples are labeled as OTH in the data (table 2 ), but the recall and precision scores are closer than they are for English.
Analysis
We perform analysis of the misclassified samples in the evaluation of our best performing models. To accomplish this, we compute the TF-IDF scores for a range of n-grams. We then take the top scoring n-grams in each category and try to discover any patterns that might exist. We also perform some manual analysis of these misclassified samples. The goal of this process is to try to get a clear idea of the areas our classifiers are lacking in. The following sections describe this process for each of the sub-tasks.
A - Offensive language identification
The classifier struggles to identify obfuscated offensive terms. This includes words that are concatenated together, such as barrrysoetorobullshit. The classifier also seems to associate she with offensiveness, and samples containing she are misclassified as offensive in several samples while he is less often associated with offensive language.
There are several examples where our classifier labels profanity-bearing content as offensive that are labeled as non-offensive in the test set. Posts such as Are you fucking serious? and Fuck I cried in this scene are labeled non-offensive in the test set, but according to annotation guidelines should be classified as offensive.
The best classifier is inclined to classify longer sequences as offensive. The mean character length of misclassified offensive samples is $204.7$ , while the mean character length of the samples misclassified not offensive is $107.9$ . This may be due to any post containing any form of profanity being offensive in sub-task A, so more words increase the likelihood of $>0$ profane words.
The classifier suffers from the same limitations as the classifier for English when it comes to obfuscated words, misclassifying samples such as Hahhaaha lær det biiiiiaaaatch as non-offensive. It also seems to associate the occurrence of the word svensken with offensive language, and quite a few samples containing that word are misclassified as offensive. This can be explained by the fact that offensive language towards Swedes is common in the training data, resulting in this association. From this, we can conclude that the classifier relies too much on the presence of individual keywords, ignoring the context of these keywords.
B - Categorization of offensive language type
Obfuscation prevails in sub-task B. Our classifier misses indicators of targeted insults such as WalkAwayFromAllDemocrats. It seems to rely too highly on the presence of profanity, misclassifying samples containing terms such as bitch, fuck, shit, etc. as targeted insults.
The issue of the data quality is also concerning in this sub-task, as we discover samples containing clear targeted insults such as HillaryForPrison being labeled as untargeted in the test set.
Our Danish classifier also seems to be missing obfuscated words such as kidsarefuckingstupid in the classification of targeted insults. It relies to some extent to heavily on the presence of profanity such as pikfjæs, lorte and fucking, and misclassifies untargeted posts containing these keywords as targeted insults.
C - Offensive language target identification Misclassification based on obfuscated terms as discussed earlier also seems to be an issue for sub-task C. This problem of obfuscated terms could be tackled by introducing character-level features such as character level n-grams.
Conclusion
Offensive language on online social media platforms is harmful. Due to the vast amount of user-generated content on online platforms, automatic methods are required to detect this kind of harmful content. Until now, most of the research on the topic has focused on solving the problem for English. We explored English and Danish hate speed detection and categorization, finding that sharing information across languages and platforms leads to good models for the task.
The resources and classifiers are available from the authors under CC-BY license, pending use in a shared task; a data statement BIBREF29 is included in the appendix. Extended results and analysis are given in BIBREF30 .

Question: What is the challenge for other language except English
Answer: not researched as much as English

Question: How many categories of offensive language were there?
Answer: 3

Question: How large was the dataset of Danish comments?
Answer: 3600 user-generated comments

Question: Who were the annotators?
Answer: the author and the supervisor