Title: I KNOW THE FEELING: LEARNING TO CONVERSE WITH EMPATHY

Abstract: Beyond understanding what is being discussed, human communication requires an awareness of what someone is feeling. One challenge for dialogue agents is recognizing feelings in the conversation partner and replying accordingly, a key communicative skill that is trivial for humans. Research in this area is made difficult by the paucity of suitable publicly available datasets both for emotion and dialogues. This work proposes a new task for empathetic dialogue generation and EMPATHETICDIALOGUES, a dataset of 25k conversations grounded in emotional situations to facilitate training and evaluating dialogue systems. Our experiments indicate that dialogue models that use our dataset are perceived to be more empathetic by human evaluators, while improving on other metrics as well (e.g. perceived relevance of responses, BLEU scores), compared to models merely trained on large-scale Internet conversation data. We also present empirical comparisons of several ways to improve the performance of a given model by leveraging existing models or datasets without requiring lengthy re-training of the full model.

1. INTRODUCTION:

Natural communication is frequently prompted by people sharing their feelings or circumstances. As examples, a recent study found that 80% of Twitter users seem to post mostly about themselves (Naaman et al., 2010) , and ELIZA (Weizenbaum, 1966) , one of the earliest chatbots developed, focused on asking its conversational partners why they were feeling a certain way. Interacting in these conversations requires reacting to what people share with an understanding of others' implied feelings. For instance, while the crossed-out response in Figure 1 is topically relevant, "Congrats! That's great!" is more natural because it acknowledges the underlying feelings of accomplishment. Responding to people in a way that is empathetic or that acknowledges how the other person feels is a desirable trait for a dialogue agent, but still a nontrivial communicative skill. It is also currently difficult to measure. Although recent work has used large-scale corpora to train reasonably fluent and engaging dialogue agents (e.g. Mazaré et al. (2018) ), existing chitchat dialogue benchmarks are not designed to capture whether those agents are responding in an empathetic way to implicit emotional cues. This may indeed be unlikely, given that the sources of Internet conversation data used for training are known to harbor aggressive and callous responses (Anderson, 2015) . This works aims to make it easier to evaluate machines' ability to respond in an empathetic way. We introduce a new task for dialogue systems to respond to people discussing everyday situations, based on EMPATHETICDIALOGUES, a novel dataset with 25k personal dialogues. Each dialogue is grounded in a specific situation where a speaker was feeling a given emotion, with a listener responding. The dataset is larger and contains a more extensive set of emotions than many similar emotion prediction datasets from other text domains such as Scherer & Wallbott (1994) , Strapparava & Mihalcea (2007) , Mohammad et al. (2018) , and Gupta et al. (2017) . Previous dialogue datasets of a similar scale that include emotion labels (Li et al., 2017; Gupta et al., 2017) come from crawled Table 1 : Two examples from EMPATHETICDIALOGUES training set. The first worker (the speaker) is given an emotion label and writes their own description of a situation when they've felt that way. Then, the speaker tells their story in a conversation with a second worker (the listener).

Label: Afraid Situation: Speaker felt this when... "I've been hearing noises around the house at night" Conversation: Speaker: I've been hearing some strange noises around the house at night. Listener: oh no! That's scary! What do you think it is? Speaker: I don't know, that's what's making me anxious. Listener: I'm sorry to hear that. I wish I could help you figure it out Label: Proud Situation: Speaker felt this when... "I finally got that promotion at work! I have tried so hard for so long to get it!" Conversation: Speaker: I finally got promoted today at work! Listener: Congrats! That's great! Speaker: Thank you! I've been trying to get it for a while now! Listener: That is quite an accomplishment and you should be proud! conversations extracted from settings that are quite different from a one-on-one conversation (educational dialogues for English learners for DAILYDIALOG, public social media content for Gupta et al. (2017) ) and cover either a very limited or a very imbalanced set of emotions: only ≈ 5% of the DailyDialog utterances have a label other than 'none' or 'happy ', and Gupta et al. (2017) only labels 'happy', 'sad', and 'angry'. The open resource we propose consists of crowdsourced one-on-one conversations, and covers a large set of emotions in a balanced way.

We then examine how to train a dialogue system that is more adept at responding to emotional cues. While a rule-based system can be built around mapping predicted emotions to responses, end-toend dialogue systems relying on neural networks and trained on conversation corpora (Shang et al., 2015; Vinyals & Le, 2015; Sordoni et al., 2015; Serban et al., 2015; Dodge et al., 2016; Mazaré et al., 2018; Zhang et al., 2018) offer the promise of better generalization to new contexts. Through an extensive set of experiments, we show that fine-tuning a dialogue agent on our dataset results in better performance on a novel empathetic dialogue task.

The pretraining of the dialogue agent on Internet conversation data is the most time-consuming step of this pipeline, by an order of magnitude. To make it easier for practitioners to improve performance of a model on the empathetic task with minimal re-training while re-using existing resources, we compare various ways of supplementing a pretrained model with additional representations from external tasks, and show that even simplistic schemes can lead to better performance. The contributions of this work are thus threefold: 1) we release a novel empathetic dialogue dataset as a new benchmark; 2) we show that using this dataset for training can improve the performance of an end-to-end dialogue system on empathetic dialogue; and 3) we compare multiple ways to further improve performance with combined representations while not requiring onerous re-training.

2. RELATED WORK:

Responding well to emotions requires sufficient coverage of human expression. Multiple schemas have attempted to organize the spectrum of emotions, from a handful of basic emotions derived from biological responses (Ekman, 1992; Plutchik, 1984) to larger sets of subtle emotions inferred from contextual situations (Skerry & Saxe, 2015) . We incorporate emotions from multiple annotation schemas, noting that emotions merely inferred from a situation are important in dialogue scenarios.

Rich information can be represented by learning multidimensional distributional embeddings from data, as has proven successful for many language applications (Grave et al., 2018) . These distributional representation approaches are at the core of the current state of the art in emotion classification (Duppada et al., 2018; Park et al., 2018; Xu et al., 2018; Mohammad et al., 2018) that build on deep networks pretrained on large-scale weakly labelled data such as emojis (Felbo et al., 2017) or hashtags (Mohammad, 2012) , gathered from public social media content published on Twitter. The SEMEVAL2019 EmoContext challenge also uses conversation data for detection of three basic emotions over two turns of context from Twitter exchanges (Gupta et al., 2017) . While public social media content has the advantage of being spontaneous (not elicited) data, it suffers from two shortcomings when used to train a model intended for one-on-one conversation (as opposed to, say, a bot designed to post on Twitter). First, the content is extracted from a context of communication in front of large "peripheral audiences" (Goffman, 1981) which include potentially everyone with an Internet connection, where the need for curated self-presentation (Goffman, 1959) and the uncertainty as to how wide that audience may be have been shown to lead to different choices of subject matters compared to private messaging, with people sharing more intense and negative emotions through private channels (Bazarova et al., 2015; Litt et al., 2014) . Second, Tweets are generally a short-form format limited to 140 characters, which is not a constraint that applies to general conversation. In this work, we attempt to generate a more balanced coverage of emotions than would appear in public social media content, within a one-on-one framing of unconstrained utterances that is closer to our ultimate goal of training a model for conversation that can respond to any emotion.

Several works have attempted to make chit-chat dialogue models more engaging by grounding them in personal contexts (Li et al., 2016; Zhang et al., 2018; Mazaré et al., 2018) , but focusing on personal facts ("I am from New York") rather than situations. The DAILYDIALOG dataset (Li et al., 2017) , comprising about 13k dialogues obtained by crawling educational websites intended for learners of English, includes many dialogues anchored in everyday situations and has been annotated post-hoc with emotion labels, but only ≈ 5% of the utterances have a label other than "none" or "happy", and dialogues are mostly limited to domains deemed appropriate for use as a language learning tool (ordering from a restaurant, asking for directions, shopping for a specific item, introductions, etc). Our task focuses explicitly on conversations about an emotionally grounded situation, and considers a richer, evenly distributed set of emotions. We also introduce an explicit single listener in the conversation who is reacting to the situation being described in an empathetic way, to make the setting as close as possible to our desired goal of a one-on-one empathetic conversation.

Several other works have focused on controlling the emotional content of a text response either through a manually specified target (Zhou & Wang, 2018; Zhou et al., 2017; Wang & Wan, 2018; Hu et al., 2017; Huang et al., 2018) or through a general term to encourage higher levels of affect (Asghar et al., 2018) , with evaluations focused on matching a predetermined desired emotion rather than empathetic responding. Niu & Bansal (2018) generate responses conditioned on a specified politeness setting (polite, rude or neutral), where politeness is viewed as a style of language. Huber et al. (2018) investigate how to respond to emotions detected from an image. By contrast, our work examines how to produce empathetic responses that are appropriate to signals inferred purely from text, and not intended to themselves convey a pre-specified emotion.

3. TALKING ABOUT PERSONAL SITUATIONS:

We consider an open-domain one-on-one conversational setting where two people are discussing a situation that happened to one of them and that led to the experience of a given feeling.

Emotional situation grounding Each conversation is grounded in a situation, which one participant writes about in association with a given emotion label. We consider 32 emotion labels, listed in Figure 2 . To select this set of labels, we drew inspiration from previous datasets (Scherer & Wall-bott, 1994; Strapparava & Mihalcea, 2007; Skerry & Saxe, 2015; Li et al., 2017; Mohammad, 2012) , consolidating the labels from each into a merged list.

Speaker and Listener:

The person who wrote the situation description (Speaker) initiates a conversation to talk about it. The other conversation participant (Listener) becomes aware of the underlying situation through what the Speaker says and responds. Speaker and Listener then exchange up to 6 more turns. We include two example conversations from the training data in Table 1 . The models discussed below are tested in the role of Listener responding to the Speaker. The situation description generated by the Speaker is not given to the models (just as it was not given to the Listener during dialogue collection). Our data could also be used to generate conversations for the Speaker conditioned on the situation description; we leave this for later work.

EMPATHETICDIALOGUES dataset statistics:

The resulting dataset comprises 24,850 conversations about a situation description, gathered from 810 different participants, which will be made publicly available online and through ParlAI. The crowdsourced dialogs were obtained using the ParlAI framework to interact with the Amazon Mechanical Turk (MTurk) platform. Details of the crowdsourcing procedure are given in the Supplemental Material.

The distribution of emotion label prompts (Table 2 ) is close to evenly distributed across categories with a few categories that are selected slightly more/less often. The average situation description is 19.8 words. Each conversation is allowed to be 4-8 utterances long (the average is 4.3 utterances per conversation). The average utterance length is 15.2 words long.

We split the conversations into approximately 80% train, 10% validation, and 10% test partitions. To prevent overlap of discussed situations by a same participant between partitions in case a participant chose to discuss the same situation twice, we split the data so that all sets of conversations with the same speaker providing the initial situation description would be in the same partition. The final train/val/test split was 19533 / 2770 / 2547 conversations, respectively.

4. EMPATHETIC DIALOGUE GENERATION:

In this section, we examine how our dataset can be used to make generic chitchat models more empathetic, and different ways existing models can be combined to produce more empathetic responses. We use our dialogues to train and evaluate models in the task of generating conversation responses, with the model playing the Listener role. At test time, the dialogue model has access to previous utterances in the dialogue, but not to the emotion word prompt (e.g., "proud"), nor to the situation description generated by the Speaker, as would be the case in a normal conversation. Given a dialogue context x of n previous conversation utterances concatenated and tokenized as x 1 , • • • , x m , followed by a target response ȳ, our models are trained to maximize the likelihood p(ȳ|x) of producing the target response. We investigate both generation and retrieval settings (Lowe et al., 2016) as described in Figure 3 .

4.1. BASE ARCHITECTURE:

We base our models on Transformer networks (Vaswani et al., 2017) , which have proven successful in machine translation and dialogue generation tasks (Zhang et al., 2018; Mazaré et al., 2018) .

Retrieval:

In the retrieval set-up, the model is given a large set Y of candidate responses and picks the "best" one, y * . We use the retrieval Transformer-based architecture from (Yang et al., 2018) : two Transformer encoders separately embedding the context, x, and candidates, y ∈ Y , as h x and h y , respectively. The model chooses the candidate sentence according to a softmax on the dot product: h x • h y . We minimize the negative log-likelihood of selecting the correct candidate. At train time, we use all of the sentences from the batch as candidates, with a large batch size of 512 to give the model more negative examples. At inference time, we experiment with multiple sets of candidate sentences for the model to choose from. First, we use all of the response utterances in the EMPATHETICDIALOGUES training set (Y ED ). We also try including candidate utterances from two other large dialogue datasets: the DailyDialog (Li et al., 2017) training set (Y DD ) and up to a million utterances from a dump of 1.7 billion Reddit conversations (Y R ). Training Details We pretrain our models on predicting replies from a dump of 1.7 billion Reddit conversations. We limit the maximum number of word tokens in the context and response to be 100 each. The Transformer networks used in all experiments have the same base architecture (four layers and six transformer heads) from Mazaré et al. (2018) , unless specified. For all models, we train for up to 10 epochs, keeping the version that has the lowest loss on the validation set. We use 300-d word embeddings pretrained on common-crawl data using fastText (Grave et al., 2018) .

4.2. LEVERAGING THE TRAINING DATA FROM EMPATHETICDIALOGUES:

Conversations utterances Fine-tuning over the task domain data may improve the model, since our data was explicitly collected with instructions to be empathetic, in a one-on-one setting, which is different from the Reddit conversation data used for pretraining. We fine-tune pretrained models to predict the next utterance over our EMPATHETICDIALOGUES with a context window of four previous utterances, which is the average length of a conversation in our dataset. These models are hereafter referred to as "Base" models. This fine-tuning is conducted for all architectures except those referred to as "Pretrained".

Emotion labels If the most appropriate response depends on some information for which supervision is available, e.g., the emotions at play, nudging the model to encode this information could result in better performance. We experiment with this by training the base architecture in the one-tomany style of multi-task learning that has been used for NLP seq2seq settings (Luong et al., 2016) .

In this set-up (Fig. 4 , left), MULTITASK, we alter the objective function to also optimize for predicting the emotion label of the conversation to which the utterances being encoded belong. We add to the context encoder a linear layer and softmax that predicts the emotion label from the context sentences. The objective function is altered to be the average of the negative log-likelihood of predicting the next utterance ȳ and the negative log-likelihood of the added linear layer being able to predict the correct emotion.

4.3. ADDING INFORMATION FROM EXTERNAL PREDICTORS:

Many existing models have been pretrained on supervised tasks that may be relevant to empathetic responding. For instance, Felbo et al. (2017) redo the work or requiring access to that data, which may matter to practitioners. Note that this may considerably augment the effective capacity of the resulting models, as well as the total amount of training data used overall, so it isn't surprising that this should result in better performance. Our goal here is to experiment with a large range of settings, in order to get an empirical sense of how robust performance improvement is to variations in architecture set-up or supervision domain. We experiment with two set-ups for adding explicit supervised information: prepending predicted label words, and ensemble learning over encoders trained on prediction objectives.

Prepending Top-K Predicted Labels This set-up (Fig. 4 , middle), PREPEND-K, is a very simple way to add supervised information to data, requires no architecture modification, and can be used with black-box classifiers. The top-K predicted labels from the supervised classifier are merely prepended to the beginning of the token sequence as encoder input:

I finally got promoted! -→ proud excited joyful I finally got promoted! Similar methods have been used for controlling the style of generated text (e.g. Niu & Bansal (2018) ). Here, we use a fastText model (Joulin et al., 2017) as prediction architecture. Both the context and the candidates are run through the classifier and receive prepended labels. We experiment with two sources of supervision (EMOPREPEND and TOPICPREPEND, respectively). Closest to our task, we train a classifier to predict the emotion label from the description of the situation written by the Speaker before the dialogue for the training set dialogues of our EMPATHETICDIALOGUES. 1 To gauge whether supervision from a more distant task would still be helpful, we also experiment with a classifier trained on the 20-Newsgroup dataset (Joachims, 1996) , for topic classification.

Ensemble of Encoders:

In this set-up (ENSEM), we augment the encoders to incorporate latent representations from pretrained supervised architectures. We replace each of the encoders in our Transformer networks with the Ensemble encoder in Figure 4 , similar to a many-to-one style encoder-decoder arhcitecture (Luong et al., 2016) . This encoder takes the encoding h w from our basic Transformer encoder (either h x or h y ), already trained on our data, and concatenates it with the representation h c extracted from the inner layer of a classification network. Here, we use the penultimate layer of a deep classifier. The concatenated encodings are projected linearly to the dimension 4 ), and train only the linear layers (and the decoder for generative systems). We used supervision from two sources that are related to emotion: Emojis from Twitter, through the use of the trained Deepmoji system (Felbo et al., 2017) released by the authors, either as-is (ENSEM-DM) or fine-tuned on the situation descriptions of EMPATHETICDIALOGUES(ENSEM-DM+), and a similarly large-scale dataset of public social media content labelled by their writers with emotion tags such as 'annoyed', used to train a second Transformer encoder ( ENSEM-TRAN).

5. EXPERIMENTAL EVALUATION:

We evaluate the models on their ability to reproduce the Listener's portion of the conversation (i.e. the ability to react to someone else's story). We use both automated metrics and human evaluation to score each model's retrievals/generations. Human evaluation is useful, as automated metrics don't always correlate with human judgments of dialogue quality (Liu et al., 2016) , but we provide automated metrics to give a sense of how well they align with human judgment on this task.

Automated Metrics (Table 2 ) For both retrieval and generative systems, we compute BLEU scores (Papineni et al., 2002) for the final response and compare against the gold label (the actual response), following the practice of earlier work in dialogue generation (Wen et al., 2015; Li et al., 2015; 2016) . For the generative systems, we additionally report perplexity of the actual gold response. For the retrieval systems, we additionally compute p@1,100, the accuracy of the model at choosing the correct response out of a hundred randomly selected examples in the test set. When we compute p@1,100, the actual response is included in the candidates, unlike inference from the retrieval systems for all other metrics, which only uses training utterances as candidates.

Human Ratings (Table 3 ) We run two sets of crowdsourcing tasks on MTurk for humans to score the model responses. In the first task, participants are given a model's output for a randomly selected test set example and asked to score different aspects of the model. The rating task provides a means of comparing aspects of responses, and we are able to ask raters specifically about whether the response is acknowledging the conversation partner's feelings. We collected at least 100 ratings per model and asked about three aspects of performance, all rated on a likert scale (1: not at all, 3: somewhat, 5: very much):

• Empathy/Sympathy: did the responses show understanding of the feelings of the person talking about their experience?

• Relevance: did the responses seem appropriate to the conversation? Were they on-topic?

• Fluency: could you understand the responses? Did the language seem accurate?

Human A/B Rankings (Table 5 ) In the second human evaluation task, participants were given output from two (randomly ordered) models and asked to select the better response, with an additional option of selecting "equal" or "neither". For this task, we only gave workers test set examples where the pair of models had differing responses. We collected at least 50 ratings per pair of models. Fine-tuning on EMPATHETICDIALOGUES Table 2 shows that fine-tuning to predict conversation responses on our data improves all automated metrics. Using only in-domain candidates leads to slightly higher BLEU scores. Training in the multitask setting degrades automated metrics compared to fine-tuning without emotion label supervision, except for average BLEU in the retrieval setting. Human evaluations in Table 3 show that fine-tuning a conversational model on the EMPATHETICDIALOGUES data and using the candidates in the dataset substantially improves performance on all metrics, in particular on the Empathy subscore of most interest to us, in both retrieval and generation set-ups. The bulk of the improvement comes from fine-tuning on the dataset, while results from the multitask setting suggest slight improvements in the Empathy rating compared to the model fine-tuned with utterances only.

Resources and capacity Fine-tuning the pretrained conversation model on dialogue utterances from our data does not change the size of the model, so the performance improvement is not due to increased model capacity. However, the multitask setting slightly increases the capacity of the base architecture (about 10k more parameters out of 84.3M or 85.1M for the retrieval and generative architectures, respectively), which may account for the performance improvement. To give a better sense of how resources and capacities balance out, Table 4 provides figures of resource use and number of parameters. We also include results for a larger Transformer model (5 layers instead of 4). A generative model trained on our data performs as well or better than a generative pretrained Transformer model with one extra layer, which has a substantially larger capacity (a million more parameters), but not as well as such a model after it has been fine-tuned on our data.

Augmenting conversation models with external pretrained classifiers Table 2 shows automated metrics for retrieval models augmented with external classifiers. Generally, the accuracy of the rankings (p@1,100) worsens, but the average BLEU scores improves. For generative models, the automated metrics are improved only in the Ensemble setting. Human evaluations (Table 3 ) suggest that nearly all models tend to improve over the Base model on the Empathy score, but only a few setups lead to significant improvement. The best performance is achieved by the ensemble trained on emotion tags (Ensem-Tran), but models that prepend topics are overall performing better than similar models trained on emotion supervision, or than the fine-tuned DeepMoji ensemble. Relevance is significantly improved only by models augmented with topic classifiers. Fluency scores are mostly not significantly changed, and remain above 4. The benefit of combining pretrained models is that this requires minimal re-training of parameters and allows for leveraging models that are already trained, or that can be trained in a few hours in the case of fastText classifiers. If re-training is not a concern, pre-training a larger Transformer model and fine-tuning it on our data leads to better performance (see Table 4 ). We observe that augmenting that model with pre-trained unsupervised sentence representations in an ensemble setting could also be a promising direction, but we leave that for future investigations. A/B comparisons To try and capture the main takeaways, we show results averaged over pairs of models with similar characteristics (Table 5 ). Responses from retrieval systems are frequently chosen over generation systems (generation:retrieval ratio: 0.62). Responses from models with added emotion supervision are often ranked above the raw pretrained model (ratios of 3.67 and 5.73), and less so against the base Transformer model fine-tuned on our data: in the retrieval case, the ratio of picking a line from a model with emotion supervision vs. base model, at 0.97, indicates that raters generally picked from each equally. However, in the generation case, the raters may have favored the models with emotion supervision explicitly represented (average ratio of 1.56).

6. CONCLUSION:

We introduce a new dataset of 25k dialogues grounded in situations prompted by specific emotion labels. Our experiments show that using this dataset to fine-tune conversation models leads to responses that are evaluated by humans as more empathetic, and that simple schemes to augment a fine-tuned model with an external pretrained classifier can lead to better performance without requiring onerous retraining. Future work will investigate how to use this dataset to model the Speaker, and how to integrate empathetic responding into more general dialogue when, for example, the needs for empathy have to be balanced with staying on topic or providing information (see Table 6 ).

Other possible directions would be to see if this data can serve as additional weakly supervised data for more complex emotion-related tasks that look at emotion evolution or causality (Gui et al., 2016; Rashkin et al., 2018) . We hope that our results and dataset will stimulate more research in the important direction of making dialog systems more empathetic. We collected crowdsourced dialogues using the ParlAI platform (Miller et al., 2017) to interact with Amazon Mechanical Turk (MTurk). A pair of workers are asked to (i) select an emotion word each and describe a situation when they felt that way, and (ii) have a conversation about each of the situations, as outlined below.

Writing a situation description prompted by an emotion label In the first stage of the task, workers are asked to describe in a few sentences a situation based on a feeling label. Each worker is given three labels from our list of 32 emotions. They are asked to select one of the options and write a short description of a personal situation where they felt that way. We ask the workers to try to keep these descriptions between 1-3 sentences. The average response is 19.8 words.

Having a conversation In the second stage, two workers are paired and asked to have two short chats with each other. In each chat, one worker (speaker) starts a conversation about the situation they previously described, and the other worker (listener) responds. Neither can see what the other worker was given as emotion label or the situation description they submitted, so they must respond to each others' stories based solely on cues within the conversation. Each conversation is allowed to be 4-8 utterances long (the average is 4.31 utterances per conversation). The average utterance length was 15.2 words long.

Ensuring balanced prompt coverage After the first few initial rounds of data collection, we forced workers to select prompts among three emotion labels that had been the least chosen overall so far if it was their first time working on the task. If they had already performed the task, the offered emotion labels were among those that they had not chosen before, or among the three least chosen ones if they had worked on nearly all of them at least once. This process made workers select emotions that they might not spontaneously have preferred, but we observed an initial bias for situations that were easier to describe (e.g, a situation causing surprise). Given that a conversation model trained for empathetic responding needs to be able to handle emotions even if they are less frequent, we opted for this balancing procedure to make training for these categories easier, while still allowing for some measure of choice for workers.

Workers 810 US workers were recruited using MTurk. Each worker had to contribute at least one situation description, and one pair of conversations: one as Speaker about the situation they contributed, and one as Listener about the situation contributed by another worker. Workers were allowed to accept additional hits and contribute more sets of situation descriptions, conversations as Speaker, conversations as Listener. The median number of conversation per worker was 8, while the average was 61, and some workers were definitely contributing more hits than others. To ensure quality, we hand-checked random subsets of conversations by our most-frequent workers. They were allowed to participate in as many of these HITs as they wanted for the first 10k conversations, then we added qualifications to limit the more frequently active workers to a maximum of 100 conversations.

We include ten randomly selected dialogues from our training set in Table 8 .

A.3 EMOTION CLASSIFICATION RESULTS:

Our dataset can also be used to train or fine-tune an emotion classifier, as we do in our PREPEND-K and ENSEM-DM+ set-ups. To give a sense of where the difficulty falls compared to existing emotion and sentiment classification benchmarks, we reproduce the table from Felbo et al. (2017) and add results when fine-tuning the Deepmoji model on our dataset, or using a fastText classifier (Table 9 ).

A.4 HUMAN RATINGS:

A.4.1 CROWDSOURCING DESCRIPTION Human evaluations were collected on MTurk. For the rating task, workers were shown one randomly subsampled example from the test set for a randomly selected model (this was done 100 times per model) and asked to rate that single response. 217 US workers participated in the rating task, and had to perform a minimum of one rating. For the human comparison task, workers were shown a dialogue context, and the two responses from a pair of models presented in a randomized order (this was done 50 times per pair of models). They had to select if they preferred one, the other, both equally, or neither. 337 US workers participated in the model comparison task.

A.4.2 RANKING RESULTS:

In Figure 5 , we provide the exact comparisons between model responses for the ranking task. Scores less than 1 indicate that the vertical model is preferred, whereas scores greater than one indicate more of a preference for the horizontal model.

:

We also experimented with training the classifier on the utterances themselves, with similar results.

Figures and Tables:

Figure 1: Example where acknowledging an inferred feeling might be appropriate

Figure 2: Distribution of situation/conversation labels within EMPATHETICDIALOGUES. Percentages per class are also listed in the appendix.

Dialogue generation architectures used in our experiments. The context of concatenated previous utterances is tokenized into x 1 , x 2 , • • • , and encoded into vector h x by the context encoder. Left: In the retrieval set-up, each candidate y is tokenized into y 1 , y 2 , • • • and encoded into vector h y by the candidate encoder. The system outputs the candidate y * that maximizes dot product h x • h y . Right: In the generative set-up, the encoded context h x is used as input to the decoder to generate start symbol < /s > and tokens y 1 , y 2 , • • • . The model is trained to minimize the negative log-likelihood of target sequence ȳ conditioned on context x.

Table content: <table><tr><td>Retrieval Architecture</td><td>y* =argmax hx • hy</td><td>Generation Architecture</td><td>p(y|x) -</td></tr><tr><td>hx</td><td>hy</td><td/><td>Transformer</td></tr><tr><td/><td/><td/><td>Decoder</td></tr><tr><td>Context</td><td>Candidate</td><td>Context</td><td/></tr><tr><td>Encoder</td><td>Encoder</td><td>Encoder</td><td>&lt;/s&gt; y1 y2…</td></tr><tr><td>x1 x2…</td><td>y1 y2…</td><td>x1 x2…</td><td/></tr><tr><td>Figure 3:</td><td/><td/><td/></tr></table>

Three ways to incorporate additional supervised information, here from an emotion classification task. Left: the context representation h x outputted by the context encoder is used both as input to a classifier, and to generate the next utterance as in the base setting. The encoder is trained with gradients from both output branches. Middle: an input sequence (that can be either a dialogue context or a candidate) is first run through a pretrained classifier, and the top k output labels are prepended to the sequence, which is then run through the corresponding (context or candidate) encoder to output a hidden representation h w (either h x or h y ) as in the base setting. Right: an input sequence (that can be either a dialogue context or a candidate) is run through the corresponding encoder as well as a pretrained classifier with the last layer removed. The outputs h w and h c are concatenated and linearly projected into a representation h e .

Table content: <table><tr><td>Multi-task Setup</td><td>Prepend-k</td><td colspan="2">Ensemble Encoder</td></tr><tr><td>embarrassed</td><td>hw</td><td>he</td><td/></tr><tr><td>Softmax</td><td>Encoder</td><td colspan="2">Linear</td></tr><tr><td>hx</td><td/><td/><td/></tr><tr><td>Linear</td><td>embarrassed … sad I slipped while …</td><td colspan="2">Concat</td></tr><tr><td>hx</td><td>Pre-trained</td><td>Pre-trained hw</td><td>Pre-trained hc</td></tr><tr><td>Encoder</td><td>Emotion</td><td>Transformer</td><td>Emotion</td></tr><tr><td/><td>Classifier</td><td>Encoder</td><td>Classifier</td></tr><tr><td>I slipped while …</td><td>I slipped while …</td><td colspan="2">I slipped while …</td></tr><tr><td>Figure 4:</td><td/><td/><td/></tr></table>

Automatic evaluation metrics on the test set. Pretrained: basic Transformer model pretrained on a dump of 1.7 billion REDDIT conversations. Base and Multitask: model fine-tuned over the EMPATHETICDIALOGUES training data (Sec. 4.2). Remaining rows: models incorporating supervised information from an external classifier, as described in Sec. 4.3.

Table content: <table><tr><td>Candidates come from</td></tr></table>

Human evaluation metrics from rating task. Training on EMPATHETICDIALOGUES improves all scores. Combining with an external supervised classifier generally improves scores, especially the Empathy score, without requiring extensive retraining of the dialogue model. The Base models rows are repeated to indicate that they are the reference model for the models below. Bold: results above 2 SEM of reference model for that group. Italicized: the reference model for the group

Table content: <table><tr><td/><td>Model</td><td colspan="2">Candidates Empathy</td><td>Relevance</td><td>Fluency</td></tr><tr><td/><td>Pretrained</td><td>R</td><td colspan="2">2.58±0.14 2.97±0.14 4.11±0.12</td></tr><tr><td/><td>Base</td><td>ED</td><td colspan="2">3.27±0.13 3.42±0.14 4.44±0.08</td></tr><tr><td/><td>Multitask</td><td>ED</td><td colspan="2">3.58±0.12 3.58±0.14 4.46±0.09</td></tr><tr><td/><td>Base</td><td>ED</td><td colspan="2">3.27±0.13 3.42±0.14 4.44±0.08</td></tr><tr><td/><td>EmoPrepend-1</td><td>ED</td><td colspan="2">3.51±0.13 3.61±0.15 4.45±0.10</td></tr><tr><td>Retrieval</td><td>EmoPrepend-3 EmoPrepend-5</td><td>ED ED</td><td colspan="2">3.62±0.14 3.50±0.15 4.54±0.08 3.52±0.14 3.64±0.14 4.47±0.09</td></tr><tr><td/><td colspan="2">TopicPrepend-1 ED</td><td colspan="2">3.66±0.11 3.85±0.11 4.51±0.08</td></tr><tr><td/><td colspan="2">TopicPrepend-3 ED</td><td colspan="2">3.67±0.10 3.70±0.11 4.49±0.08</td></tr><tr><td/><td colspan="2">TopicPrepend-5 ED</td><td colspan="2">3.59±0.10 3.73±0.10 4.43±0.08</td></tr><tr><td/><td>Ensem-DM+</td><td>ED</td><td colspan="2">3.36±0.14 3.33±0.14 4.13±0.11</td></tr><tr><td/><td>Ensem-Tran</td><td>ED</td><td colspan="2">3.80±0.12 3.66±0.14 4.59±0.08</td></tr><tr><td/><td>Pretrained</td><td>-</td><td colspan="2">2.26±0.13 2.37±0.13 4.08±0.12</td></tr><tr><td/><td>Base</td><td>-</td><td colspan="2">2.95±0.15 3.10±0.14 4.37±0.10</td></tr><tr><td/><td>Multitask</td><td>-</td><td colspan="2">3.17±0.14 3.23±0.14 4.29±0.11</td></tr><tr><td/><td>Base</td><td>-</td><td colspan="2">2.95±0.15 3.10±0.14 4.37±0.10</td></tr><tr><td/><td>EmoPrepend-1</td><td>-</td><td colspan="2">2.66±0.15 2.63±0.15 4.22±0.12</td></tr><tr><td>Generation</td><td>EmoPrepend-3 EmoPrepend-5</td><td>--</td><td colspan="2">3.34±0.13 3.31±0.15 4.58±0.09 3.35±0.15 3.20±0.15 4.41±0.10</td></tr><tr><td/><td colspan="2">TopicPrepend-1 -</td><td colspan="2">3.05±0.11 3.19±0.12 4.31±0.09</td></tr><tr><td/><td colspan="2">TopicPrepend-3 -</td><td colspan="2">3.12±0.11 3.10±0.13 4.33±0.09</td></tr><tr><td/><td colspan="2">TopicPrepend-5 -</td><td colspan="2">3.45±0.12 3.39±0.11 4.49±0.08</td></tr><tr><td/><td>Ensem-DM+</td><td>-</td><td colspan="2">3.17±0.14 3.19±0.14 4.31±0.11</td></tr><tr><td/><td>Ensem-Tran</td><td>-</td><td colspan="2">3.49±0.12 3.27±0.14 4.42±0.09</td></tr><tr><td colspan="2">Gold Response -</td><td>-</td><td colspan="2">4.19±0.06 4.48±0.06 4.67±0.04</td></tr></table>

Training resources for different models, with human evaluation metrics from rating task for empathy (Emp), relevance (Rel) and fluency (Fluent). Comparisons are relative to the first row of each group. Training on EMPATHETICDIALOGUES improves all scores while requiring minimal additional training resources (same number of parameters for Base, 0.01% increase in number of parameters for Multitask). SEM is approximately 0.1

Table content: <table><tr><td/><td>Model</td><td colspan="3">Params, resources, train examples Emp Rel Fluent</td></tr><tr><td/><td>Pretrained</td><td>84.3M, 2.5 days, 8GPUs, 1.7B</td><td>2.6</td><td>3.0 4.1</td></tr><tr><td>Retrieval</td><td>Base Multitask</td><td colspan="2">same , + 0.5 hour, 1 GPU, +22.3k 3.3 +9.6k, + 0.5 hour, 1 GPU, +22.3k 3.6</td><td>3.4 4.4 3.6 4.5</td></tr><tr><td/><td>Pretrained</td><td>85.1M, 2 days, 32 GPUs, 1.7B</td><td>2.3</td><td>2.4 4.1</td></tr><tr><td/><td>Base</td><td>same , +1 hour, 1 GPU, +22.3k</td><td>3.0</td><td>3.1 4.4</td></tr><tr><td>Generation</td><td>Multitask</td><td>+9.6k, +1 hour, 1 GPU, +22.3k</td><td>3.2</td><td>3.2 4.3</td></tr><tr><td/><td colspan="2">PretrainedLarge 86.2M, 2.5 days, 32 GPUs, 1.7B</td><td>3.0</td><td>3.1 4.0</td></tr><tr><td/><td>BaseLarge</td><td>same , +0.5 hour, 1 GPU, +22.3k</td><td>4.0</td><td>4.2 4.7</td></tr><tr><td>5.1 RESULTS</td><td/><td/><td/><td/></tr></table>

Average ratio of "best" replies from model A vs. model B for a set of pairs in our human ranking evaluation. Ratios > 1 mean that the model on the left was selected more than the model on the right. Emotion Supervision Models: models from Sec. 4.3 that incorporate supervised emotion information. Full listings of comparison between pairwise models is included in the appendix.

Table content: <table><tr><td>Choose Model A vs B response</td><td>Average A:B ratio</td></tr><tr><td>Gold Response vs. Models</td><td>13.79</td></tr><tr><td>Generation vs. Retrieval</td><td>0.62</td></tr><tr><td>Emotion Context Models vs Pretrain (ret)</td><td>3.67</td></tr><tr><td>Emotion Context Models vs Pretrain (gen)</td><td>5.73</td></tr><tr><td>Emotion Context Models vs Base (ret)</td><td>0.97</td></tr><tr><td>Emotion Context Models vs Base (gen)</td><td>1.56</td></tr></table>

Examples of model responses. In the first example, responses from the models with emotion prediction components focusing on the feelings of the speaker are more generic. In the second example, they focus on the feelings on the speaker while remaining topically specific.

Table content: <table><tr><td>Context</td><td>Model</td><td>Generated Utterance</td></tr><tr><td>Tomorrow is my first university day and I'm very nervous</td><td colspan="2">Gen-base Gen-multitask Gen-emoPrepend-5 I'm sure you will be fine. What are you studying? Good luck! Gen-ensemb-DM+ Are you nervous ?</td></tr><tr><td>I had to go out of town by</td><td>Ret-base</td><td>Do you still talk to the people you went with?</td></tr><tr><td>myself for three weeks. It</td><td>Ret-multitask</td><td>I'm sorry to hear that. Do you think you</td></tr><tr><td>was hard not being able to</td><td/><td>struggle to communicate with people?</td></tr><tr><td>talk to anyone I knew.</td><td colspan="2">Ret-emoPrepend-5 That sounds difficult. Did you try talking to</td></tr><tr><td/><td/><td>people in your neighborhood?</td></tr><tr><td/><td>Ret-ensemb-DM+</td><td>Did you start to feel a bit lonely?</td></tr></table>

Distribution of situation/conversation labels within EMPATHETICDIALOGUES

Table content: <table><tr><td>Label</td><td>%</td><td>Label</td><td>%</td><td>Label</td><td>%</td><td>Label</td><td>%</td></tr><tr><td colspan="3">surprised 5.15 impressed</td><td colspan="2">3.24 joyful</td><td colspan="2">3.10 content</td><td>2.97</td></tr><tr><td>excited</td><td colspan="2">3.77 afraid</td><td colspan="2">3.21 prepared</td><td colspan="2">3.08 devastated</td><td>2.87</td></tr><tr><td colspan="3">annoyed 3.53 disgusted</td><td colspan="2">3.17 guilty</td><td colspan="2">3.06 sentimental</td><td>2.81</td></tr><tr><td>proud</td><td colspan="2">3.49 confident</td><td colspan="2">3.16 furious</td><td colspan="2">3.05 caring</td><td>2.75</td></tr><tr><td>angry</td><td colspan="2">3.47 terrified</td><td colspan="2">3.15 nostalgic</td><td colspan="2">3.05 trusting</td><td>2.62</td></tr><tr><td>sad</td><td colspan="2">3.45 hopeful</td><td colspan="2">3.14 jealous</td><td colspan="2">3.05 ashamed</td><td>2.53</td></tr><tr><td>grateful</td><td colspan="2">3.28 anxious</td><td colspan="5">3.11 anticipating 3.03 apprehensive 2.45</td></tr><tr><td>lonely</td><td colspan="6">3.28 disappointed 3.10 embarrassed 2.98 faithful</td><td>1.92</td></tr><tr><td colspan="3">A SUPPLEMENTAL MATERIAL</td><td/><td/><td/><td/><td/></tr><tr><td colspan="3">A.1 LABEL DISTRIBUTION</td><td/><td/><td/><td/><td/></tr><tr><td colspan="8">In Table 7, we include the exact percentage of emotion labels for the situation descriptions in our</td></tr><tr><td>final dataset.</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="6">A.2 DIALOGUE COLLECTION: CROWDSOURCING DESCRIPTION</td><td/><td/></tr></table>

Classification performance on EMPATHETICDIALOGUES, with the benchmarks proposed inFelbo et al. (2017) for reference. ED: performance on predicting the emotion label from the situation description. ED-CUT: same, but after having removed all the situation descriptions where the target label was present.

Table content: <table><tr><td colspan="2">Dataset Metric</td><td>SOTA (in 2017)</td><td>fastText</td><td>DeepMoji new</td><td>DeepMoji full</td><td>DeepMoji last</td><td>DeepMoji chain-thaw</td></tr><tr><td>SE0714</td><td>F1</td><td>0.34</td><td>0.16</td><td>0.21</td><td>0.31</td><td>0.36</td><td>0.37</td></tr><tr><td>OLYMPIC</td><td>F1</td><td>0.50</td><td>0.38</td><td>0.43</td><td>0.50</td><td>0.61</td><td>0.61</td></tr><tr><td>PSYCHEXP</td><td>F1</td><td>0.45</td><td>0.44</td><td>0.32</td><td>0.42</td><td>0.56</td><td>0.57</td></tr><tr><td>SS-TWITTER</td><td>Acc</td><td>0.82</td><td>0.68</td><td>0.62</td><td>0.85</td><td>0.87</td><td>0.88</td></tr><tr><td>SS-YOUTUBE</td><td>Acc</td><td>0.86</td><td>0.75</td><td>0.75</td><td>0.88</td><td>0.92</td><td>0.93</td></tr><tr><td>SE0614</td><td>Acc</td><td>0.51</td><td>-</td><td>0.51</td><td>0.54</td><td>0.58</td><td>0.58</td></tr><tr><td>SCV1</td><td>F1</td><td>0.63</td><td>0.60</td><td>0.67</td><td>0.65</td><td>0.68</td><td>0.69</td></tr><tr><td>SCV2-GEN</td><td>F1</td><td>0.72</td><td>0.69</td><td>0.71</td><td>0.71</td><td>0.74</td><td>0.75</td></tr><tr><td>ED</td><td>Acc</td><td>-</td><td>0.43</td><td>0.40</td><td>0.46</td><td>0.46</td><td>0.48</td></tr><tr><td>ED-CUT</td><td>Acc</td><td>-</td><td>0.41</td><td>0.36</td><td>0.42</td><td>0.44</td><td>0.45</td></tr></table>

Peer review (iclr2019):
 Overall this paper contributes many interesting insights into the specific application of empathetic dialog into chatbot responses. The paper in particular is contributing its collected set of 25k empathetic dialogs, short semi-staged conversations around a particular seeded emotion and the results of various ways of incorporating this training set into a generative chatbot. While the results clearly do not solve the problem of automating emapthy, the paper does give insights into which methods perform better than others (Generation vs Retrieval) and explicitly adding emotion predictions vs using an ensemble of encoders. There is a lot in this paper, and I think it could have been better organized. I am more familiar with emotion related research and not language to language translation, so I would have appreciated a better explanation of the rationale for using BLEU scores. I did some online research to understand these Bilingual Evaluation Understudy Scores and while it seems like they measure sentence similarity, it is unclear how they capture ”relevance” at least according to the brief tutorial that I read (https://machinelearningmastery.com/calculate-bleu-score-for-text-python/). I did not see the paper describing the use of this score in the references but perhaps I missed it – could you please clarify why this is a good metric for relevance? It seems that these scores are very sensitive to sentence variation. I am not sure if you can measure empathy or appropriateness of a response using this metric. For your data collection you have 810 participants and 24,850 conversations. Are the 810 participants all speakers or speakers and listeners combined? How many conversations did each speaker/listener pair perform 32? (one for each emotion) or 64? (two for each emotion) Was the number variable? If so what is the distribution of the contribution – e.g. did one worker generate 10,000 while several hundred workers did only three of four? Was it about even? Just for clarity – how did you enroll participants? Was it through AMT? What were the criteria for the workers? E.g. Native English speaker, etc. In your supplemental material, I found the interchanging of the words “context” and “emotion” confusing. The word context is used frequently throughout your manuscript: “dialog context,” “situational context” - emotions are different from situations, the situational utterance is the first utterance describing the emotion if I read your manuscript correctly. Table 6 should use “Label” or “Emotion” instead of the more ambiguous “Context.” My understanding is that speakers were asked to write about a time when they experienced a particular feeling and they were given a choice of three feelings that they could write about. You then say that workers are forced to select from contexts they had not chosen before to ensure that all of the categories were used. From this I am assuming that each speaker/listener worker pair had to write about all 32 emotions – is this correct? Another interpretation of this is that you asked new workers to describe situations involving feelings that had not been chosen by other workers as data collection progressed to ensure that you had a balanced data set. This would imply that some emotional situations were less preferred and potentially more difficult to write about. It would be interesting if this data was presented. It might imply that some emotion labels are not as strong if people were forced to write about them rather than being able to choose to write about them. Were these dialogs ever actually annotated? You state in section 2, Related Work “we train models for emotion detection on conversation data that has been explicitly labeled by annotators” – please describe how this was done. Did independent third party annotators review the dialogs for label correctness? Was a single rater or a majority vote used to decide the final label. For example, in Table 1, the label “Afraid” is given to a conversation that could also have reasonable been generated by the label “Anxious” a word explicitly used in the dialog. I am guessing that the dialogs are just labeled according to the label / provocation word and that they were not annotated beyond that, but please make this clear. In the last paragraph you state “A few works focus..” and then list 5. This should rather be “several other works have focused on “ … Conversely, you later state in section 3 “Speaker and Listener”, “We include a few example conversations from the training data in Table 1,” this should more explicitly be “two.” Also in section 3 when you describe your cross validation process, you state “We split the conversations into approximately 80/10/10 partitions. To prevent overlap of <<discussed topics>> we split the data so that all the sets of conversations with the same speaker providing the prompt would be in the same partition. In your supplemental material you state that workers were paired. Each worker is asked to write a prompt, which also seems to be the first utterance in the dialog they will start. You state each worker selects one emotion word from a list of three which is somehow generated (randomly?) form your list of 32 . I am assuming each worker in the pair does this, then the pair has a two “conversations” one where the first worker is the speaker and another where the second worker is the speaker – is this correct? It is not entirely clear from the description. Given that you have 810 workers and 24,850 conversations, I am assuming that each worker had more than one conversation. My question is - did they generate a new prompt / first utterance for each conversations. I am assuming yes since you say there are 24,850 prompts/conversations. For each user are all of the situation/prompts they generate describing the same emotion context? E.g. would one worker write ~30 conversations on the same emotion. This seems unlikely, and it seems more likely that given the number of conversations ~30 per participant is similar to the number of emotion words that you asked each worker to cycle through nearly all of the emotions or that given they were able to select, they might describe the same emotion, e.g. “fear” several times. If the same worker was allowed to select the same emotion context multiple times was it found that they re-used the same prompt several times? I am assuming that this is the case and that this is what you mean when you say that you “prevent overlap of discussed topics” between sets when you exclude particular workers. Is this correct? Or did you actually look and code the discussed topics to ensure no overlap even across workers (e.g. several people might have expressed fear of heights or fear of the dark). In section 4, Empathetic dialog generator, you state that the dialog model has access to the situation description given by the speaker (also later called the situational prompt) but not the emotion word prompt. Calling these both prompts makes the statement about 24,850 prompts/conversations a bit ambiguous. A better statement would be 24,850 conversations based on unique situational prompts/descriptions (if they are in fact unique situational prompts. I am assuming they are not if you are worried about overlapping “discussed topics” which I am assuming are the situational prompts since the dialogs are very short and heavily keyed off these initial situational prompts) In your evaluation of the models with Human ratings you describe two sets of tests. In one test you say you collect 100 annotations per model. More explicitly, did you select 100 situational prompts and then ask workers to rate the response of each model? Was how many responses was each worker shown? How many workers were used? Are the highlighted numbers the only significant findings or just the max scores? Annotations is probably not the correct word here. Please also describe your process for assigning workers to the second human ratings task. Since the two novel aspects of your paper are the new dataset and the use of this dataset to create more empathetic chatbot responses ("I know the feeling") I have focused on these aspects of the paper in my review. I found the inclusion of Table 7 underexplained in the text. The emotion labels for all these datasets are not directly comparable so I would have liked to have seen more explanation around how these classifications were compared. It would also be helpful to know how more similar emotions such as "afraid" and "anxious" were scored vs "happy" and "sad" confusions