SlowGuess/NAACL_2022_OCR huggingface dataset:
Allison Lahnala† and Charles Welch† and Béla Neuendorf and Lucie Flek  
Conversational AI and Social Analytics (CAISA) Lab
Department of Mathematics and Computer Science, University of Marburg
http://caisa-lab.github.io
{allison.lahnala,welchc,neuendob,lucie.flek}@uni-marburg.de
Content Warning: This paper includes examples of religious-based discriminatory language that may be offensive and upsetting.
Large pre-trained neural language models have supported the effectiveness of many NLP tasks, yet are still prone to generating toxic language hindering the safety of their use. Using empathetic data, we improve over recent work on controllable text generation that aims to reduce the toxicity of generated text. We find we are able to dramatically reduce the size of fine-tuning data to 7.5-30k samples while at the same time making significant improvements over state-of-the-art toxicity mitigation of up to  $3.4\%$  absolute reduction (26% relative) from the original work on 2.3m samples, by strategically sampling data based on empathy scores. We observe that the degree of improvement is subject to specific communication components of empathy. In particular, the cognitive components of empathy significantly beat the original dataset in almost all experiments, while emotional empathy was tied to less improvement and even underperforming random samples of the original data. This is a particularly implicative insight for NLP work concerning empathy as until recently the research and resources built for it have exclusively considered empathy as an emotional concept.
Pre-trained neural language models are prone to generating toxic language, hindering the ability to use them safely (Gehman et al., 2020). Recent work on controllable text generation has shown promise in successfully altering such text attributes (Liu et al., 2021). However, partly due to the subjective nature of this task (Jurgens et al., 2019), the selection of negative, non-toxic examples for modeling has been somewhat arbitrary.
Meanwhile, there is a growing body of research in natural language processing around the concept of empathetic communication - a number of data resources and approaches have been proposed for training empathy recognition and generation models (Sharma et al., 2020; Rashkin et al., 2019). Though the definitions of toxicity and empathy vary across literature, we observe an opposition between the concepts in terms of response appropriateness and intent toward others, which is the basis of the research question driving this work: is there an opposing relationship between toxic and empathetic language that can be leveraged to better model these phenomena?
Toxic language is often described as harassing or offensive language that decreases the likelihood of participation in discussions or other cooperative efforts (Wulczyn et al., 2017). In NLP literature, empathetic language is usually conveyed as language that shows an understanding and acknowledgement of the interlocutor's emotions (Rashkin et al., 2019; Shin et al., 2020), which, in turn, has an increased participation effect. In many fields of social science, empathy is defined with multiple dimensions including both emotional and cognitive components (and others) (Decety and Jackson, 2004; Gerdes et al., 2011).
In this paper, we investigate the following hypotheses:
1. There is an unexplored negatively correlated relationship between toxicity and empathy.
2. Exploiting this relationship could result in more robust and/or efficient models for mitigating toxic degeneration.
3. Specific categories of empathetic behavior have a stronger relation to the reduction of specific types of toxicity. In particular, we expect the cognitive types of empathy to be more beneficial for mitigating the largely cognitive
Authors contributed equally.
Human Language Technologies, pages 4926 - 4938
aspects of toxic behavior, since emotional empathy may reinforce toxic feelings such as hostility toward out-groups (Breithaupt, 2012).
We perform a set of experiments in which we leverage empathetic data to alter the toxicity of generated text. We use the predictions of a language model trained on empathetic data to alter the output of a large pretrained language model and demonstrate that using only a small volume of empathetic data can reduce toxicity more than a model simply trained on a large volume of non-toxic text.
Furthermore, we consider relationships between various facets of toxicity and empathy, particularly emphasizing the distinction between emotional empathy and cognitive empathy that is less commonly made in the NLP literature. We find that training on text with high cognitive empathy is more effective at reducing toxicity than text with emotional empathy.
Large language models (LM) have achieved strong performance on a number of natural language processing tasks (Radford et al., 2019), yet they remain difficult to control and often generate problematic responses both in their use as language models and as the foundation for downstream applications, such as conversational agents (Wolf et al., 2017; Bender et al., 2021; Bommasani et al., 2021). In this section, we review the related work on toxicity, empathy, and controllable text generation.
Toxicity has recently been used as a way to measure language that is harmful or offensive. This language has also been shown to suppress the expression of others, which is often the opposite of what is desired in interactive NLG applications (Sood et al., 2012).
Gehman et al. (2020) introduced RealToxicityPrompts, a test-bed for toxic language generation. They gathered a range of toxic sentences and split them in half. Models tested with this data must continue the sentence in a non-toxic way. They test recent LMs (some mentioned in the following subsection on controllable generation) finding all to be prone to toxic degeneration and suggest that choosing less toxic pretraining data may help. Similarly, Zhou et al. (2021) examine challenges in mitigation, finding that improving data quality through relabeling is more effective than attempting to debias a model trained with biased labels.
The Jigsaw shared task provided a large volume of Wikipedia comments with human annotations of six classes of toxicity (Jigsaw, 2021b). SemEval-2021 hosted a task on toxic span detection, where one must identify the subsequence of a text that is responsible for the toxicity label (Pavlopoulos et al., 2021). The Jigsaw data classes were those originally used to train the models in the Perspective API, which has been used by several recent works to automatically evaluate toxic language (Jigsaw, 2021a).
These are not the only classes that exist in toxic language research. Waseem and Hovy (2016) looked at sexism and racism in Twitter comments and ElSherief et al. (2021) developed a taxonomy of implicit hate speech. However, Fortuna et al. (2020) performed experiments across toxicity datasets, finding that within-class homogeneity and performance vary greatly. They suggest that each dataset has its own "flavor" of toxicity, even for similarly defined concepts.
Empathy has been the subject of many recent NLP studies, often for empathetic response generation models in aims of improving response appropriateness and overall satisfaction with dialogue agents (Hu et al., 2018; Rashkin et al., 2019; Lin et al., 2019, 2020; Majumder et al., 2020; Zandie and Mahoor, 2020; Zheng et al., 2021; Zeng et al., 2021; Jhan et al., 2021). Most of this work predominantly conveys empathy as an ability to recognize and demonstrate an understanding of one's emotions with a warm or sympathetic response (Rashkin et al., 2019; Lin et al., 2020; Zandie and Mahoor, 2020; Majumder et al., 2020; Shin et al., 2020), which are all aspects of what is often termed affective or emotional empathy (Cuff et al., 2016).
While some definitions of empathy across areas of cognitive neuroscience, psychology, and practicing areas of psychotherapy are based only on emotional components (Cuff et al., 2016), most include both emotional and cognitive components (Decety and Jackson, 2004; Gerdes et al., 2011), sometimes along with additional ones. Cognitive empathy involves deliberate cognitive processing and active interest to understand and further explore the other's internal perspective (Gerdes et al., 2011; Miller and Rollnick, 2012).
The reason few NLP works have engaged with empathy conceptualizations beyond emotional aspects could be partly due to limited resources and
difficulty constructing them, which some recent works have aimed to address. Zhou and Jurgens (2020) created a corpus of Reddit posts with expressions of distress and responses offering condolences, annotated for empathy based on appraisal theory (Lamm et al., 2007; Wondra and Ellsworth, 2015). Welivita and Pu (2020) created an annotation scheme for empathetic listener intents which they manually labeled on a subset of the EmpatheticDialogues dataset (Rashkin et al., 2019) on which they trained a classifier to automatically label the rest of the data. Sharma et al. (2020) developed a framework of expressed empathy called EPITOME that includes both emotional and cognitive aspects which are annotated in peer-supporter responses to support-seekers in online interactions. A later work created a hierarchical model for empathy generation using EPITOME, which led to improved performance including in human evaluations (Zheng et al., 2021).
Our work leverages Sharma et al. (2020)'s public Reddit data. The communication mechanisms of the framework are emotional reactions (ER) and two cognitive aspects, interpretations (IP) and explorations (EX), which we define thoroughly in  $\S 3$ . The data contains annotations of whether the peer supporters' responses to seekers contain no, weak, or strong communication for each of the three mechanisms. They then created classifiers for all three types of empathy using separate models built from the same RoBERTa-based architecture (Liu et al., 2019). The classifiers predict the degree to which a sample contains no, weak, or strong communication of each mechanism.
We expect the cognitive aspects of empathy to be more useful for toxic language mitigation because of side-taking effects. In the three-person model of empathy, one person observes a conflict between two others. The observer may take sides with one of the persons in conflict and together their emotional reaction to the third party can be amplified (Breithaupt, 2012). This type of polarization through side-taking can lead to aggressive acts (Breithaupt, 2018). To the best of our knowledge, such negative aspects of empathy have yet to be investigated in NLP literature; our findings suggest that this direction is important to further pursue.
Controllable Generation methods often involve fine-tuning or retraining large models. The CTRL
model of Keskar et al. (2019) is trained with 50 predefined control codes representing different topics, styles, and languages, that condition the generation process. Ziegler et al. (2019) used a reinforcement learning (RL) approach to alter the fine-tuning process for sentiment, physical descriptiveness, and summarization tasks. Yu et al. (2017) trained a generative adversarial network for sequence generation using RL for poem, political speech, and music generation.
Other methods have been developed not to alter the original model, but to alter generation at decoding time and do not require retraining the original model. The FUDGE model uses discriminators to predict, for a partial sequence, the probability that the next step of generation is more likely to result in an output that satisfies a particular attribute (Yang and Klein, 2021). The PPLM model of Dathathri et al. (2020) uses a similar approach but uses separate attribute models to modify the gradients used during prediction. Similarly, the work of Kumar et al. (2021) uses gradients but uses a modified loss for continuous optimization to allow for control of non-categorical attributes and non-autoregressive generation. They show that this improves performance on poetry couplet completion, topic-controlled generation, and informal-to-formal machine translation. In our experiments, we use the DExperts model of Liu et al. (2021), another decoding-time generation strategy. Their model uses LMs fine-tuned on desirable or undesirable attributes and uses the predictions of these models to alter the probabilities predicted by the base LM. More details of this model are provided in § 4.
We use the three types of empathy of Sharma et al. (2020)'s EPITOME framework. The definitions of each and descriptions of weak and strong classes are abbreviated as follows (nearly verbatim):
Emotional Reaction: Expressions of emotions such as warmth, compassion, and concern, experienced by peer supporters after reading a seeker's post. Weak: Alludes to the peer's experienced emotions after reading the seeker's text without the emotions being explicitly labeled (e.g., Everything will be fine). Strong: The peer specifies their experienced emotions (e.g., I feel really sad for you).
Interpretations: Communicates an understanding of feelings and experiences inferred from the
1The TalkLife data is not publicly available.
seeker's post. Weak: Contains a mention of the understanding (e.g., I understand how you feel). Strong: Specifies the inferred feeling or experience (e.g., This must be terrifying) or communicates understanding through descriptions of similar experiences (e.g., I also have anxiety attacks at times which makes me really terrified).
Explorations: Expressions for improving understanding of the seeker by exploring the feelings and experiences not stated in the post. Weak: Generic (e.g., What happened?) Strong: Specific and labels the seeker's experiences and feelings which the peer supporter wants to explore (e.g., Are you feeling alone right now?)
For toxicity, we use all types of toxicity currently available from the Perspective API. Related works often use only the toxicity score, while the API currently offers scores for eight attributes, the last two of which were listed as experimental at the time of use:
Toxicity: A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.
Severe Toxicity: A very hateful, aggressive, disrespectful comment or otherwise very likely to make a user leave a discussion or give up on sharing their perspective. This attribute is much less sensitive to more mild forms of toxicity, such as comments that include positive uses of curse words.
Identity Attack: Negative or hateful comments targeting someone because of their identity.
Insult: Insulting, inflammatory, or negative comment towards a person or a group of people.
Profanity: Swear words, curse words, or other obscene or profane language.
Threat: Describes an intention to inflict pain, injury, or violence against an individual or group.
Sexually Explicit: (Experimental) Contains references to sexual acts, body parts, or other lewd content.
Flirtation: (Experimental) Pickup lines, complimenting appearance, subtle sexual innuendos, etc.
The DExperts model combines the predictions of a base LM with expert LMs fine-tuned on data known to either contain a desired (e.g., empathy) or undesired attribute (e.g., toxicity). The probability of the next token,  $x_{t}$ , is given by the following
Strength	ER	IP	EX
strong	6,594	457,009	261,229
weak	148,962	0	2,953
no	2,170,585	1,869,132	206,1959
Table 1: The number of samples for which the classifier predicted each class (strong, weak, and no communication), for each empathy type (EX=explorations, IP=interpretations, ER=emotional reactions). Additional boxplots of log-likelihood distributions are in the Appendix.
linear transformation of logits within the softmax:  $P(x_{t}|x_{<t}) = \text{softmax}(z_{t} + \alpha (z_{t}^{+} - z_{t}^{-}))$ , for  $z_{t}$  predicted by the base model, the expert  $z_{t}^{+}$ , and the anti-expert  $z_{t}^{-}$ , with experts contribution weighted by a hyperparameter,  $\alpha$ . We use  $\alpha = 2.0$  as this is what was deemed effective in the original work. To allow for comparison we use the same hyperparameters, including a max generation length of 20 tokens. This model can be used for controlled generation by modifying decoding-time predictions for a given stylistic attribute. Using an opposing attribute to train the expert model should help minimize the probability of our undesired attribute (e.g. empathy used to oppose toxicity).
The intuition behind the negative correlation between empathy and toxicity lies in the perceived appropriateness of language and a better understanding of the user. Consider a response to a person that is trying to help someone and not having much success. A toxic response, "you are stupid for trying that," may be perceived as toxic and inappropriate, while "it sounds like you're really trying hard and doing your best," may be perceived as more appropriate and better understanding the user. By our intuition, a stronger negative correlation should generally correspond to less toxic output.
Evaluation: We use the 10k prompts used in Liu et al. (2021) and the same metrics of toxicity, fluency, and diversity for comparability. We generate a set of 25 continuations of each prompt and score them with the Perspective API. Average max toxicity is the highest toxicity score given to the set and averaged over all 10k prompts. Probability of toxicity is the chance of a continuation having a score of  $\geqslant 0.5$  at least once in the set. Fluency is measured as the average perplexity with a reference text generated by the larger LM, GPT-2 XL. Diversity measures the distinct n-grams normalized by text length, over all generations in the set. We report uni, bi, and trigrams for this metric as
Type	Size	Max Tox.	Tox. Prob.	PPL
Empathy	22.5k	0.323	0.147	45.10
Empathy	30k	0.324	0.149	43.68
Empathy	7.5k	0.329	0.156	52.91
Random	22.5k	0.331	0.159	47.37
Empathy	15k	0.335	0.163	49.04
Random	30k	0.331	0.163	43.92
Random	7.5k	0.341	0.168	53.61
Random	15k	0.343	0.177	48.82
DExperts	2.3m	0.313	0.133	32.46
was done by Li et al. (2016). This metric was not as insightful for our analysis, so we list it in the Appendix.
5 Empathy for Toxicity Mitigation
Training a model to controlled generation requires a distinction between the groups of desired and undesired text. In our case, we want to avoid generating a text,  $x_{t}$ , from the set of toxic texts,  $T$  so we use non-toxic text from the complement set  $x_{nt} \in T' = NT$ . However,  $NT$  contains many types of non-toxic text. We hypothesize that a small subset with specific qualities,  $E \subset NT$ , will be more effective in generating non-toxic text than any random sample  $R \subset NT$ , and that empathetic text belongs to this subset  $E$ .
We use the set of  $\sim 1.4$  million comments that were not labeled as toxic by any annotators as our non-toxic set. We split this dataset by lines of text, rather than entire comments, resulting in 2.3 million lines in total. Then we trained the model from Sharma et al. (2020) to recognize the communication strength of the three types of empathy using their publicly available human-annotated Reddit corpus, which achieved 74 F1-score for emotional reactions, 63 for interpretations, and 73 for explorations. This classifier is used to assign class probabilities to our non-toxic set. Table 1 shows the resulting distribution of highest probability classes.
Data sampling: We select the empathetic data to fine-tune the expert model by taking the sentences with the lowest likelihood of no communication of each empathy type, which effectively maximizes the probability of empathetic samples. We had also performed preliminary experiments on sample
Table 2: Results for fine-tuning the non-toxic expert model on empathetic data as compared to a model trained on a random subset. Models are ordered in ascending order of toxicity probability with the original DExperts baseline at the bottom. Lower values signify better performance.
Type	Size	Max Tox.	Tox. Prob.	PPL
EX	7.5k	0.292	0.099	74.85
EX	15k	0.294	0.108	63.13
EX	22.5k	0.297	0.110	57.13
EX	30k	0.304	0.119	51.94
IP	22.5k	0.319	0.142	42.03
IP	15k	0.319	0.148	45.53
IP	7.5k	0.328	0.149	52.42
Random	30k	0.329	0.156	43.70
Random	22.5k	0.331	0.159	47.37
IP	30k	0.328	0.160	40.24
ER	22.5k	0.335	0.164	46.42
Random	7.5k	0.341	0.168	53.61
ER	7.5k	0.340	0.173	53.46
ER	30k	0.338	0.173	43.20
Random	15k	0.343	0.177	48.82
ER	15k	0.342	0.179	50.66
DExperts	2.3m	0.313	0.133	32.46
Table 3: Results when fine-tuning the expert model on individual empathy types. Models are ordered in ascending order of toxicity probability with the original DExperts baseline at the bottom. Lower values signify better performance. (EX=explorations, IP=interpretations, ER=emotional reactions)
sets with the highest likelihood of strongly communicated empathy, yet we observed this was less effective. This outcome could be related to the imbalances between the weak and strong classes in Sharma et al. (2020)'s annotated dataset reflected by the distributions of the results on the non-toxic data (Table 1), which we intuit is due to greater difficulty annotating weak versus strong than present versus absent empathetic communication.
We selected equal subsets of the empathy-maximized data to create samples with sizes ranging from roughly  $0.1\%$  to  $1\%$  of the original data. These were used to fine-tune the non-toxic expert in DExperts and compared to fine-tuning the nontoxic expert on random samples of equal size.
Results: The results are shown in Table 2. We find that using the empathetic data performs better than random samples of the same size and that the best model overall uses empathetic fine-tuning, significantly outperforming the best random model. Our model comes close to the DExperts baseline with a difference of  $1.4\%$  toxicity probability,  $1\%$  average max toxicity, though perplexity shows a greater gap. Empathy here appears to be useful in selecting more informative examples for fine-tuning.
2With permutation test on both average max toxicity and toxicity probability  $p <   10^{-5}$
6 Empathy Components Experiments
We are also interested to know which type of empathy is most useful for mitigating toxicity. To examine this, we create subsets of the empathy labeled non-toxic data that each maximizes one of the empathetic aspects. We hypothesize that the two types of cognitive empathy, explorations and interpretations, will be more useful to the model than emotional reactions, given the potentially polarizing nature of emotional reactions discussed in § 2. We sample data similarly to § 5, except that we take instances that score highly on only one type of empathy at a time.
Results: We compare to the DExperts baseline large model. The results in Table 3 show improved performance when using only the best empathetic explorations while using two orders of magnitude less data. We also find that the two types of cognitive empathy score higher than emotional reactions, consistent with our hypothesis. This finding suggests that controllable generation does not require a large volume of data if the data is particularly well suited to the problem. In our case, we find that cognitive empathy data is effective at minimizing toxic generations. Though we do see an increase in perplexity, this does not directly correspond to a loss of fluency. See § 7, 9 for more details.
We see that explorations consistently perform better than other empathy types. In addition, less data leads to higher performance, likely because the smaller dataset contains only the best examples of empathetic explorations. Interpretations are the next most effective type of data, though we do not see as consistent a pattern in the data size used. Lastly, emotional reactions perform similarly to random subsamples of the data.