DeTexD: A Benchmark Dataset for Delicate Text Detection
Year: 2023

Abstract:
Over the past few years, much research has been conducted to identify and regulate toxic language.1 However, few studies have addressed a broader range of sensitive texts that are not necessarily overtly toxic. In this paper, we introduce and define a new category of sensitive text called "delicate text." We provide the taxonomy of delicate text and present a detailed annotation scheme. We annotate DeTexD, the first benchmark dataset for delicate text detection. The significance of the difference in the definitions is highlighted by the relative performance deltas between models trained each definitions and corpora and evaluated on the other. We make publicly available the DeTexD Benchmark dataset, annotation guidelines, and baseline model for delicate text detection.2 3 4

text: The 7th Workshop on Online Abuse and Harms (WOAH), pages 14–28 July 13, 2023 c©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The prevalence of user-generated toxic language on online social networks has motivated many to develop automatic methods of detecting such content (Warner and Hirschberg, 2012), (Waseem and Hovy, 2016), (Davidson et al., 2017), (Schmidt and Wiegand, 2017), (ElSherief et al., 2018a), (ElSherief et al., 2018b), (Qian et al., 2018a), (Qian et al., 2018b). These efforts towards moderating toxic language have gained even more momentum as large language models, which have the potential to generate harmful content, have become more mainstream (Welbl et al., 2021), (Bender et al., 2021), (Hovy and Prabhumoye, 2021), (Kocielnik et al., 2023). Much of this work has been constrained to texts that are toxic or otherwise overtly harmful; however, there are many other sensitive texts ∗ The names of authors are arranged in reverse alphabetical
order. 1 Here, we use the terms "toxic language" and "hate speech"
interchangeably. 2 https://github.com/grammarly/detexd 3 https://huggingface.co/grammarly/ detexd-roberta-base 4 https://huggingface.co/grammarly/detexd
where interaction with other users or virtual agents may be triggering or offensive. While some studies (Yenala et al., 2018), (Parnell et al., 2020), (Tripathi et al., 2019) have addressed specific sensitive areas (e.g., insults, geopolitics, or illegal activity), to our knowledge, this is the first study that comprehensively analyzes sensitive content in general.
In this study, we target a broader set of sensitive texts that we call "delicate texts," an umbrella term covering toxic language as well as lower-severity sensitive texts, with a focus on sensitive texts (Table 1). Delicate text covers many topics which are not necessarily offensive but can still be highly sensitive and triggering. For example, texts where users share challenges regarding their mental health issues, where they discuss their experience of the loss of a loved one, or where they share content about self-harm and suicide. While most of these texts do not contain offensive language or attack certain minority groups, they all contain triggering topics that are emotionally and personally charged. Conversations about these topics can be easily derailed and lead to users experiencing discourteous or offensive behaviors from other users or virtual agents. With delicate text detection, our goal is
14
to identify texts where engagement by other users or agents is most likely to result in harm, rather than focusing only on texts where harmful content has already been generated. Automatic detection of delicate texts is an essential tool for effective monitoring and prevention of potentially harmful content generated by users or AI. This model can be used for practical applications such as content moderation for models that are at high risk of hallucinations or data sampling to efficiently target texts where offensive interactions are most likely to happen.
In this study, we introduce the task of delicate text detection. We present a comprehensive definition of delicate texts and a dataset of 1,023 labeled delicate texts (DeTexD). We share our data collection, annotation, and quality control methods along with the detailed annotation schema. We describe the development of our baseline delicate text detection model. Finally, we demonstrate the difference between delicate text detection and existing content moderation methods by testing our model against toxic language benchmark datasets and testing popular content moderation models against our DeTexD dataset.

heading: 2 Related Works
text: Several studies have investigated the use of various NLP methods to detect inappropriate content; most of these works targeted toxic and offensive language. Some focused on developing more robust models to detect hateful content (Sohn and Lee, 2019), (Caselli et al., 2021), (Yousaf and Nawaz, 2022), while others focused on building better and less biased datasets (Founta et al., 2018), (Zampieri et al., 2019), (Basile et al., 2019), (Davidson et al., 2017), (Kiela et al., 2020), (Mathew et al., 2021), (Xia et al., 2020), (Huang et al., 2020), (Mollas et al., 2022), (Qian et al., 2019). With respect to dataset creation, (Mollas et al., 2022) created ETHOS, a binary and multi-labeled dataset of hate speech, along with a detailed annotation protocol. Their dataset covers various hate speech categories (including race, gender, religion, nationality, sexual orientation and disability), as well as target and whether the texts incited violence. They also examined the quality of their data using both binary and multi-label classification. In another study, (Mathew et al., 2021) created HateXplain, a hate speech dataset that reflects annotators’ rationale for their labeling task. Their data went through a three-
step annotation process in which a text was first classified as "offensive," "hate," or "normal;" next, the target of the hate was identified as "individual" or "generalized." Last, the annotators were asked to highlight parts of the text that justified their annotation decisions. They reported that models that used annotators’ rationale in the training data performed slightly better than those without human rationale. Most studies have targeted hate speech; however, some studies have addressed a more general concept: inappropriate content. While most of these works used the term ‘inappropriate content’ to refer to hate speech, they also included sensitive topics. For instance, (Yenala et al., 2018) focused on identifying inappropriate content; they defined inappropriate content as impolite and disrespectful posts that offend certain groups, are related to illegal activities, or induce violence. They developed a deep learning-based model to identify inappropriate content in detecting query completion suggestions and user conversation texts. In another study, (Tripathi et al., 2019) focused on detecting sensitive content in user interactions with voice services. They targeted profanity, insult, geopolitical topics, explicit sexual and anatomical content, weapons, war, explicit graphical violence, race, religion, and gender. They focused on binary classification of sensitive content.
In (Basile et al., 2019) SemEval 2019 Task 5 dataset is described, a specific case of hate speech against immigrants and women in Spanish and English Twitter messages. They provide both: a main binary subtask for detecting the presence of hate speech, and a finer-grained one for identifying features such as hate attitude or target. During this competition, over 100 models were submitted. We evaluate our baseline model for delicate text detection on their main dataset.

heading: 3 Delicate text

heading: 3.1 Definition
text: We define delicate text as any text that is emotionally charged or potentially triggering such that engaging with it has the potential to result in harm. This broad term covers a range of sensitive texts that vary across four major dimensions: 1) riskiness, 2) explicitness, 3) topic, and 4) target. Delicate texts come with varying levels of risk; some can be highly risky such as texts about self-harm or content that promotes violence against certain identity groups, while others can be less risky such
as insulting language. Delicate texts can also have various degrees of explicitness: some can be produced explicitly with the use of delicate key terms, while others can be produced implicitly without the presence of delicate lexical terms. Delicate texts cover various subjects, with topics ranging from race, gender, and religion to mental health, socioeconomic status, or political affiliations.
Unlike toxic language that only targets identity groups (Zampieri et al., 2019), (Davidson et al., 2017), delicate texts can target identity groups, non-identity groups, or they can be self-targeted or non-targeted. In addition, delicate texts are not always offensive, unlike toxic language. Table 1 shows how different texts would be treated under our delicate text approach as compared to typical approaches for categorizing hate speech, offensive language, and texts containing profanity.
Table 2 illustrates examples of both delicate and non-delicate texts. The first "non-delicate" text does not contain any references to delicate topics. The rest all reference a delicate subject (mental health), but each has a different level of risk. For example, the "very low risk (1)" delicate text contains a factual statement about mental health, while the "very high risk (5)" text explicitly mentions self-harm. There is a shift in riskiness as content becomes more personal, emotional, and explicit. It is worth noting that none of these examples are offensive or contain vulgar language; however, engagement with these texts, whether by users or virtual agents, can result in harm.
Figure 1 displays the list of delicate topics. We did not present this list in any hierarchical order as we wanted to highlight the fact that there is not a
clear border between toxic language and sensitive language as a text can be both toxic and sensitive. Each of these topics can be used to create risky content. While these topics may be used in various degrees of riskiness, they all are considered delicate. Please see Appendix A for our annotation guidelines and additional examples.

heading: 4 DeTexD Benchmark Dataset

heading: 4.1 Data Collection
text: The sparsity of delicate texts in online platforms makes it challenging to target in data collection. To ensure that the data contained sufficient sensitive content, we used a combination of domain specification and keyword-matching when sourcing data. For our DeTexD Benchmark dataset (Table 3), we extracted data from various websites in CommonCrawl5, where we specifically targeted news websites, forums discussing sensitive topics (e.g., Mental Health Forum6, and able2know7 which covers body image), and generally controversial forums (4Chan8, Stormfront9), with the expectation that these would contain a significant amount of sensitive content. To further refine the dataset, we targeted paragraphs containing words from our delicate keyword dictionary. Our dictionary contains keywords related to a wide variety delicate topics which are included as tags in the dictionary metadata (see Table 6 for the full list of topics and examples of keywords). Additionally, each keyword is tagged with one of four severity ratings ranging from highly offensive to potentially offensive. We used these keywords with their metadata as a fine-grained data filtering method to extract delicate texts covering various topics and levels of risk. After analyzing the content of the pilot dataset we confirmed that the targeting methods resulted in a desirable distribution of data and moved forward with extracting 41,000 paragraphs through this method for the main annotation task. This data was split into DeTexD Train dataset containing 40,000 samples (this part is not published) and the DeTexD Benchmark dataset of 1,023 paragraphs. All data was collected from September 2019 to August 2022.

heading: 5 https://commoncrawl.org/

heading: 6 https://www.mentalhealthforum.net/

heading: 7 https://able2know.org/

heading: 8 https://www.4chan.org/
text: 9 https://www.stormfront.org/forum/

heading: 4.2 Data Annotation
text: Identifying delicate content is a nontrivial task, as delicate text is a highly subjective concept. To ensure consistent and accurate annotations, we developed a fine-grained annotation scheme to guide expert annotators through the task. The annotation guidelines (Appendix A) contain our definition of delicate text along with a list of delicate categories within delicate text, examples of each category, and labeling samples.
To reduce the impact of subjectivity, we designed a two-step annotation scheme (Table 2):
Step I (identification): Annotators were shown texts and asked to label them as "non-delicate" or "delicate" based on our overall definition of delicate text. This initial binary rating pass allowed us to quickly identify texts most likely to contain delicate content through a relatively low-effort task.
Step II (risk level rating): Texts labeled "delicate" in Step I moved on to Step II, where annotators were asked to rate the risk level of each text on a riskiness scale of 1 ("very low risk") to 5 ("very high risk"). The annotators were instructed to focus on overall sentiment of the texts rather than the lexical meanings of individual keywords. Delicate texts which are more emotional, personal, charged, or those that reference a greater number of delicate topics are considered high risk, whereas texts with more neutral and less personal content are considered low risk (see Table 2 for examples of risk ratings).
Using this labeling process, we stepped away from simple binary labeling of the data, which not only helped to ensure quality, but also allowed us to gain more detailed information about the riskiness of the sensitive data.

heading: 4.3 Quality Control
text: Each text in the DeTexD Benchmark dataset was annotated following the guidelines (Appendix A). All annotators who participated in this task are expert linguists that had an excellent understanding of delicate texts and had previously completed similar annotation tasks. Each text was annotated by three different annotators, and we took a majority vote as the final label.
To ensure annotation quality, we conducted a pilot annotation. Each snippet was annotated by one annotator, and 500/1,023 labeled snippets were randomly selected and qualitatively analyzed by the team of four expert linguists who designed the task. Each sample was reviewed, and its label was accepted if it matched the guidelines and rejected otherwise. Out of 500 judgments, 426 snippets were accepted, and only 74 labels were rejected (85% acceptance rate). After the pilot, the annotators were provided with feedback for improvement and the guidelines were updated to address common areas of confusion. Annotators who passed the pilot task moved on to annotate the full dataset. We measured the inter-rater agreement and obtained a Krippendorff’s alpha score of 0.65 for the final dataset.

heading: 5 Experiments
text: In this section, we share results from a series of experiments. First, we show the potential to create a delicate text detection system which is suitable for practical usage. For this purpose, we developed and evaluated a baseline delicate text detection model. Next, we demonstrate the originality of this task by evaluating the performance of our model on toxic language datasets and evaluating toxic language detection models on DeTexD. Since toxic language detection and delicate text detection are two distinct tasks, DeTexD does not perform well on toxic language benchmarks and other content moderation methods that target mainly toxic language do not perform well on the DeTexD benchmark dataset.

heading: 5.1 Baseline Model
text: Our baseline model is the RoBERTa-based classifier (Liu et al., 2019b), which is fine-tuned on the delicate text detection training dataset of 40,000 samples.10 The model is trained for 2,000 optimization updates on batches of 256 samples each. We used AdamW as an optimizer with a learning rate of α = 5e−5. As a task to learn, we selected a multiclass classification model with binary conversion because it has higher quality than binary classification and ordinal regression (Cheng, 2007). Although we noticed a better diagonal-aligned confusion matrix for ordinal regression, the evaluation result did not show a statistically significant improvement. In our settings, we train a 6-class classification model, where the classes are defined by the riskiness levels from annotation step II. The model’s prediction is converted to a binary label using the mapping (Table 2):
i) NON-DELICATE = non-delicate (0) ∪ very low risk (1) ∪ low risk (2) and
ii) DELICATE = medium risk (3) ∪ high risk (4) ∪ very high risk (5).

heading: 5.2 Baseline Model Performance on Hate Speech Tasks
text: In order to experimentally confirm that delicate text detection and toxic language detection are distinct tasks, we ran our baseline delicate text detection model (Section 5.1) on popular toxic language datasets (Table 4).
10We are not publishing the training portion of our delicate text detection dataset, but it was annotated in exactly the same way as the DeTexD Benchmark dataset (Section 4).
The Automated Hate Speech Detection (AHSD) dataset from (Davidson et al., 2017) has separate classes for offensive speech and hate speech, with examples labeled as hate speech representing the minority of the dataset (1,430 examples out of 24,783 total). We evaluate the performance of our model separately on the entire (Davidson et al., 2017) dataset, as well as only on the hate speech subset (which excludes all offensive speech examples). In both cases, after performing error analysis we can see that this dataset is not relevant for our classifier as the task is significantly different. Some false positive prediction examples, such as those mentioning race-related topics or explicitly sexual content, would be categorized as true positives in the DeTexD annotation schema although they are labeled as negative in this dataset. Most of the false negative prediction examples strongly correlate with specific offensive words such as "h*e" or "b*tch." Given the context in which these words are used, these examples would fall under the true negative definition of delicate text. Notably, our classification performance on the hate speech only subset exceeds that of the original work. We attribute the high performance to the fact that there is some overlap in the tasks specifically under the hate speech case, and that we use a more recent model architecture (Liu et al., 2019b), a pre-trained base model and larger model size.
After evaluating our baseline model on the dataset from (Founta et al., 2018) we found that it
is not relevant for our classifier as the task is very different from ours. A large proportion of false positives would be classified as delicate under our definition (e.g., sensitive topics such as "killing of thousands..."), while many false negatives would be classified as neutral according to our definition. However, here they are treated as overly emotional like "I’m fu***d up". After evaluating on SemEval2019, Task 5, Subtask A (Basile et al., 2019) we found that it is not relevant for our classifier as the task is different from ours; it consists mostly of hate speech against migrants and women. As a result, false positives occur in instances where DeTexD detects other delicate topics, even including hate speech that is not targeted at women and migrants. False negatives occur in instances where refugee-directed hate speech is very specific and context-dependent such as "build that wall." Besides that, the dataset is unbalanced: for example, the word "b*tch" appears in half of the texts.
Offensive Language Identification Dataset (OLID), Subtask A (Zampieri et al., 2019) contains examples labeled as "offensive" or "not offensive." Similarly to our other evaluations, we find that the labels in this dataset do not significantly agree with our definition of delicate text. Many of the examples labeled as offensive in this dataset either do not contain enough context to make such a judgment (e.g. "A dying sport") under our definition, or look entirely neutral according to our definition of delicate text (e.g. "Yes. Yes he is!").
These experiments show that there is partial overlap between the definition of delicate text and commonly used definitions of offensive language and hate speech, which results in 70%-90% relative Fscore of our baseline model for delicate text detection compared to models trained for toxic language detection (Table 4).

heading: 5.3 Comparing our baseline model and hate speech detection methods on the DeTexD Benchmark dataset
text: In order to evaluate our baseline model’s performance and compare it with the most popular existing solutions for hate speech detection, we run them on our DeTexD Benchmark dataset (Table 5).
In our experiments, HateBERT models ("AbusEval", "HatEval", and "OffensEval") are the instances of HateBERT (Caselli et al., 2021), which are fine-tuned on the corresponding dataset. The highest precision is shown by the "HatEval" model,
which is fine-tuned on SemEval 2019 Task 5 dataset that contains hate speech against migrants and women (Basile et al., 2019). These topics are explicitly presented in the DeTexD dataset under "Gender" and "Nationality"/"Race" categories (Fig. 1). "OffensEval" shows the best overall performance among the HateBERT models. We speculate that this is because the definition of offensive language in the training dataset of this model (Basile et al., 2019) (“contains any form of non-acceptable language (profanity) or a targeted offense, which can be veiled or direct”) is broader compared to "AbusEval" and "HatEval," and it has greater overlap with our definition of delicate language. All HateBERT models show relatively low recall values because each HateBERT instance targets a narrow range of topics. After receiving valuable feedback from reviewers, we also calibrated optimal thresholds for f-score (marked with hash#). While F-scores got much higher, the precisions got much lower, so we consider this more like metric hacking. In real life, the proportion of positive cases is much lower, so for the future versions it may make sense to get test dataset with more negative cases.
Google’s Perspective API is designed to moderate human interaction to support a friendly conversation environment. The Perspective API targets text attributes such as toxicity (rude, disrespectful, or unreasonable comments), severe toxicity (very hateful, aggressive, disrespectful comments), identity attacks (hateful comments targeting someone because of their identity), insults (insulting comments toward people), profanity (swear words), and threat (intention to inflict pain, injury, or violence against people)14. This target definition is similar to the "OffensEval" dataset, which could explain why performance is similar to the HatEval model.
14https://support.perspectiveapi.com/s/ about-the-api-attributes-and-languages
The OpenAI content filter shows strong recall but is lacking precision in our experiments. In error analysis, we see that it misses a large part of examples from mental health and medical topics. In a sample of false-positive predictions, we only see a slight pattern of a tendency to flag texts that contain profane keywords. Surprisingly, during our testing of the OpenAI content filter, we found that for about half of the inputs the predictions are stochastic, with standard deviation on binary prediction reaching as high as 0.5 (across 100 predictions). We expect the presented results for the OpenAI content filter to have a wider than expected confidence interval.
The authors of the OpenAI moderation API suggest it as a replacement for the OpenAI content filter. On our benchmark dataset, the moderation API has higher precision but lower recall as compared to the OpenAI content filter. This can be explained by the difference in the definition of target content between the two models. During error analysis, we find that lower recall can mostly be attributed to medical and mental health topics in our dataset, although some of the examples relating to sexual content were also missed. All examples where the OpenAI moderation API made a false-positive prediction relate to sexual content or socioeconomic status categories. However, the sample is too small (6 out of 687 non-delicate) to make strong conclusions.
In summary, our experiments show that none of the studied toxic language detection methods provide satisfactory detection performance in delicate text detection. Most commonly, the evaluated hate speech detection methods either miss coverage on medical and mental health topics, show lower precision on examples that contain offensive keywords (but aren’t deemed delicate according to our definition), or both.

heading: 6 Conclusions
text: We introduced a new type of sensitive language called “delicate text,” an umbrella term covering not only toxic language but also sensitive language with a priority focus on the latter. We annotated the DeTexD Benchmark dataset for delicate text detection. The significance of the difference in the definitions is highlighted by the relative performance deltas between models trained each definitions and corpora and evaluated on the other. We make our annotation guidelines, annotated dataset,
and baseline model publicly available.

heading: 7 Limitations
text: The presented results only apply to the English language. Both our benchmark dataset and the baseline model target the English language exclusively. Special text sources such as instant messaging or speech-to-text are likely under-represented in our benchmark test set; therefore, we did not evaluate classification performance in those domains. Since we used RoBERTa as the base model, our model inherits the same limitations. Specifically, the length of input sequences is limited to 512 BPE tokens, and additional pre- and post-processing is necessary to run predictions on longer inputs. However, we did not evaluate prediction aggregation methods or classification performance.

heading: 7.1 Gender bias
text: We expect our baseline model to exhibit some amount of gender bias, since it is based on a model that was trained on a wide variety of human-written text. However, in our analysis we only found limited evidence of gender bias. In the counterfactual prediction difference analysis on the DeTexD dataset, we found 6 examples (0.6% of 1023 total) for which the baseline model made a different prediction depending on gender context. The counterfactual texts were created by performing binary gender-swapping (male/female) of the nouns, pronouns and gendered names.
In gender bias evaluation using the Winogender Schemas dataset (Rudinger et al., 2018), we found 4 example triplets (1.7% of 240 total) where the baseline model made different predictions depending on the gender context. We noticed a slight correlation between gender and our model’s binary prediction, but consider it negligible at this sample size.
Due to the limited volume of evidence provided by our experiments we cannot make strong conclusions about the presence or absence of gender bias in our baseline model. However, our understanding is that for the best results, bias mitigation methods must be applied. One such method could be the gender-swapping data augmentation of the training data.

heading: 7.2 Country bias
text: The baseline model described in this paper (Section 5.1) treats different countries differently. We
measure this effect by calculating scores for the template phrase "Unfortunately, the incident happened in {country_name}." The result is illustrated on the colored map on Figure 2. One way to fix this is to replace country names with random others in training data. However, in this case some sentences can become nonsense. For example, "London is the capital of the United Kingdom." In this case, we would need to change all dependent words which would become too complex. From our observation, this situation happens with half of all sentences including country information, so we leave this part as an open question for now.
Ethics Statement
Hate speech, offensive language, and delicate texts are sensitive, and very important matters. Through this work, we try to dive deeper into the challenges and opportunities of any delicate text detection. The goal of this work is to expose the strengths and limitations of different delicate text detection and related techniques and their implications. Some datasets, and models that we work with have been publicly released for a couple of years. All of these artifacts are considered to be in the public sphere from a hate speech perspective. We do not make any recommendations on using these on public or private datasets without proper due diligence for privacy, security, sensitivity, legal, and compliance measures.
Please be advised that due to the nature of the subject matter, the presented DeTexD Benchmark dataset includes a variety of uncensored sensitive content, such as hate speech, violence, threat, selfharm, mental health, sexual, profanity, and others. The text of this work includes keywords and partial text examples of the same type. The most extreme occurrences of such examples in this text are partially obscured with asterisks but the semantics are retained.

heading: 8 Acknowledgements
text: We express our gratitude to our colleagues Cortney Napoles and Leonardo Neves for their valuable advice and to our managers Viktor Zamaruiev and Max Gubin for their constant support. To our communities: While we are writing this, our homeland Ukraine continues to resist the unprovoked Russian invasion. We are grateful to everyone who defends Ukraine, declares support to the people of Ukraine, and is sending aid. Thank you!

heading: A Appendix. Guidelines for building DeTexD.
text: A.1 Glossary In these guidelines, you are going to see numerous references to sensitivity in language as well as related notions. Before reading the document, please familiarize yourself with the following terms that will help you get a better understanding of the task:
• Delicate (adj. for a text/word/subject matter): referencing a touchy or sensitive subject. This includes texts that are emotionally charged and that cover topics that are potentially triggering, offensive, taboo, intimate, or about marginalized groups.
• Delicate topics: topics that are usually delicate. Examples include mental and physical health-related topics, trauma and violence, or identity-related topics. See an the list of sensitive topics in Table 6.
• Delicate keywords: words that semantically relate to a certain delicate topic. For example:
– democrat, chauvinist, islamo-leftism are normally used in political language,
– able-bodied, autistic, bulimia will typically refer to the topic of ableism or mental health.
A.2 Delicate topics Table 6 provides the list of delicate topics and the definitions of their typical language that can be associated with the language of a certain delicate topic.
A.3 Context This task will ask you to make two judgment steps:
Step I. Identification: decide if a text is delicate or non-delicate using the following definitions:
• A delicate sentence contains emotionally-charged references to a sensitive topic.
• A non-delicate is fully innocuous and doesn’t contain any particularly charged references to a sensitive topic.
Step II. Riskiness estimation: rate how delicate texts are using a 5-point scale where 1 stands for "very low risk" and 5 for "very high risk".
How delicate a sentence is must be evaluated with regard to its sentiment rather than the lexical meanings of separate keywords. In other words, the more emotional and personal the tone, the more delicate the sentence. The following questions can help you make a decision:
(a) Is the content of the sentence emotionally charged rather than factual?
(b) Can the content of the sentence evoke negative feelings?
(c) Does the content of the sentence pertain to a sensitive topic and show bias against particular groups of people?
If the answer to any of these questions is positive, the sentence will fall on the high end of the riskiness scale. Find a detailed interpretation of the riskiness scale in the next section.
See the examples of annotated texts in Table 8.
A.4 Riskiness estimation Table 9 provides a description of riskinesses that are likely to fall on certain parts of the rating scale, examples of delicate texts, and explanations.
A.5 Paragraph-level judgments The paragraphs are annotated holistically. This means that the assigned score is not based on just the individual sentences within a paragraph, but rather the score is reflective of the overall meaning of the paragraph. However, the score can be affected by the number of:
• delicate sentences within a paragraph;
• explicit delicate sentences within a paragraph;
• emotional and personal sentences within a paragraph;
• paragraphs that have a higher number of delicate sentences, higher level of explicitness, and have more emotional and personal weights should get a higher score. A comparative analysis of some examples is presented in the Table 10.
26

heading: Future Work
text: In real life, the proportion of positive cases is much lower, so for the future versions it may make sense to get test dataset with more negative cases. Google’s Perspective API is designed to moderate human interaction to support a friendly conversation environment. The Perspective API targets text attributes such as toxicity (rude, disrespectful, or unreasonable comments), severe toxicity (very hateful, aggressive, disrespectful comments), identity attacks (hateful comments targeting someone because of their identity), insults (insulting comments toward people), profanity (swear words), and threat (intention to inflict pain, injury, or violence against people)14. This target definition is similar to the "OffensEval" dataset, which could explain why performance is similar to the HatEval model. 14https://support. perspectiveapi. com/s/ about-the-api-attributes-and-languages
The OpenAI content filter shows strong recall but is lacking precision in our experiments. In error analysis, we see that it misses a large part of examples from mental health and medical topics. In a sample of false-positive predictions, we only see a slight pattern of a tendency to flag texts that contain profane keywords. Surprisingly, during our testing of the OpenAI content filter, we found that for about half of the inputs the predictions are stochastic, with standard deviation on binary prediction reaching as high as 0. 5 (across 100 predictions). We expect the presented results for the OpenAI content filter to have a wider than expected confidence interval. The authors of the OpenAI moderation API suggest it as a replacement for the OpenAI content filter. On our benchmark dataset, the moderation API has higher precision but lower recall as compared to the OpenAI content filter. This can be explained by the difference in the definition of target content between the two models. During error analysis, we find that lower recall can mostly be attributed to medical and mental health topics in our dataset, although some of the examples relating to sexual content were also missed. All examples where the OpenAI moderation API made a false-positive prediction relate to sexual content or socioeconomic status categories