SlowGuess/NAACL_2021_OCR huggingface dataset:
Hsuan Su $^{1*}$ , Jiun-Hao Jhan $^{2*}$ , Fan-yun Sun $^{2\dagger}$ , Saurav Sahay $^{2\diamond}$  and Hung-yi Lee $^{1*}$
\*National Taiwan University, Taipei, Taiwan
†Stanford University, Stanford, CA, USA
Intel Labs, Santa Clara, CA, USA
{b04203058, r06942141, hungyilee}@ntu.edu.tw
fanyun@stanford.edu
saurav.sahay@intel.com
Most chatbot literature that focuses on improving the fluency and coherence of a chatbot, is dedicated to making chatbots more human-like. However, very little work delves into what really separates humans from chatbots – humans intrinsically understand the effect their responses have on the interlocutor and often respond with an intention such as proposing an optimistic view to make the interlocutor feel better. This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework includes a guiding chatbot and an interlocutor model that plays the role of humans. The guiding chatbot is assigned an intention and learns to induce the interlocutor to reply with responses matching the intention, for example, long responses, joyful responses, responses with specific words, etc. We examined our framework using three experimental setups and evaluated the guiding chatbot with four different metrics to demonstrate flexibility and performance advantages. Additionally, we performed trials with human interlocutors to substantiate the guiding chatbot's effectiveness in influencing the responses of humans to a certain extent. Code will be made available to the public.
Humans have evolved to become sensitive to their social interactions. The more they interact, the more they generally learn what to say and what not to say to light up people's mood or to avoid upsetting others. In this paper, we aimed to train a chatbot to emulate these human-like qualities by making it learn from interactive conversation. A chatbot that understands the effect its utterances have on the interlocutor could be a significant step towards achieving human-level chatbots.
A chatbot that understands the effect of its utterances on the interlocutor is also critical in real-world applications. For instance, as shown in Figure 1, given a context from the interlocutor, both
Figure 1: An example of the dialogue to show that how the chatbot interacts with the interlocutor, and how our chatbot affects the interlocutor's response when assigned the intention, making people respond joyful.
responses "I did. They were really nice and fun and smart people." and "I did. I was so bummed out. I was so lonely." were relevant and reasonable responses, and were equally suitable for a typical chatbot. However, we could give an intention to the proposed chatbot (guiding chatbot), such as making the interlocutor feel joyful. In this way, the chatbot would respond in a positive way to induce joy in the interlocutor.
Much literature combine Reinforcement Learning(RL) (Kaelbling et al., 1996) with transformer-based (Vaswani et al., 2017) models to control the chatbot's output. Gupta et al. (2019) proposed models to concentrate on crucial keyphrases presented in the context. Their models tended to generate outputs that were more coherent and specific to the conditionals, which led to more non-generic words. By training with a combination of the above criteria, their approach leaded to more diverse and interesting responses. However, these previous works focused on controlling the chatbot's responses and completely neglected the interlocutor in their training.
In this paper, we made extensive use of the interlocutor's responses as interactive experiences to train our guiding chatbot to influence the interlocutor with intentions. We introduce a novel train
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1559-1569 June 6-11, 2021. ©2021 Association for Computational Linguistics
ing framework, in which there were two conversational models that simulated chatbot-interlocutor interaction. One model acted as the interlocutor model, while the other was the guiding chatbot to be trained. The interlocutor took the guiding chatbot's output as its input and generated corresponding responses. The learning framework was given a controllable factor, which represented the intention it had. We defined reward functions according to three different controllable factors, sentence length, emotion, and specific words, to make the guiding chatbot learn to induce the interlocutor model to generate desired responses using RL.
To evaluate our guiding chatbot, we designed several experiments to examine the rewards corresponding to three controllable factors, and empirical results demonstrate that our guiding chatbot can influence humans' responses. Moreover, we found that training with more interlocutor models together improved the guiding chatbot's performance on the human evaluation experiment. Furthermore, we analyzed recent off-the-shelf chatbots based on experimental results, aiming to find hidden tendencies these chatbot models had, such as cursing more or being more irritable.
The most common chatbot model is sequence-to-sequence based (Sutskever et al., 2014). Recently, numerous researchers applied transformer to build coherent chatbots by retrieval-based (Zhou et al., 2016; Wu et al., 2019; Yang et al., 2018; Henderson et al., 2017; Yan et al., 2016) and generative-based (Ritter et al., 2011; Serban et al., 2016; Shang et al., 2015; Tammewar et al., 2018) approaches. Despite the decent fluency and coherence these chatbots achieved, they still hardly converse like a human. The reason might be that they are essentially devoid of emotions.
Furthermore, some used RL to improve their chatbot's performance (Serban et al., 2016; Williams, 1992) and others combined RL with GPT-2 (Radford et al., 2019) models to control the sentiment of their chatbot's response to make it more user-friendly (Han et al., 2019; Lee et al., 2018). Beyond the viewpoint of sentiment, the EmpatheticDialogues (ED) dataset was collected (Rashkin et al., 2019) to train a chatbot that could recognize the feeling of the interlocutor and know how to reply accordingly (Lin et al., 2020). However, these researchers neglected what really separated
humans from chatbots – humans understand the impact their responses have on the interlocutor and often responded with intentions and expectations. Note that it is not just about being empathetic as human's intentions could vary widely.
One previous work also considered interlocutor responses (Shin et al., 2019). It used a sentiment predictor to predict the interlocutor's sentiment given the chatbot's response, and also trained the chatbot with RL. Unlike this previous work, our proposed framework explicitly modeled the possible responses of interlocutors. Explicitly modeling interlocutor responses give the proposed framework more flexibility. For example, in addition to steering the interlocutor's sentiment as in this paper, the framework could be used to build a chatbot that induce the interlocutor to become more talkative by setting its learning target to be making the interlocutor generate longer sentences. Moreover, we also developed techniques to preserve the dialogue's coherence, so our chatbot could still generate fluent and appropriate responses in addition to having a particular intention.
Apart from influencing the interlocutor, the proposed framework also served as a way to analyze the underlying inclination of the off-the-shelf chatbots playing the role of interlocutor. Through the interaction, we could know what factors are apt to influence these off-the-shelf chatbots. Holzinger et al. (2017) claimed that the appealing performance of recent robust and SOTA models belied a potential problem of black-box models: these models lacked an explicit declarative knowledge representation. Hence, calling for a transparent representation, they dug into explaining trained models. In contrast to the previous contributions, we tried to explain the implied tendency of a chatbot, which was not obvious to recognize. According to the experiments, we were capable of telling whether the off-the-shelf black box chatbot possessed certain predispositions, such as tending to swear more or having a short temper.
The proposed framework is shown in Figure 2. It consisted of two conversational models: the guiding chatbot and the interlocutor model. The interlocutor and guiding chatbot simulated the dialogue between a human and a chatbot. The guiding chatbot aimed to generate a response that maximize
Figure 2: The framework that we proposed to teach the guiding chatbot how to achieve the intention assigned by the controllable factors.
rewards according to different controllable factors; the interlocutor models produced responses based on the guiding chatbot's response in order to simulate a human's response. Therefore, grounded in different controllable factors, we examined corresponding rewards to optimize the guiding chatbot to influence the interlocutor.
3.2 Conversational Models
Interlocutor The model  $I$  represented the interlocutor.  $I$  could be any off-the-shelf chatbot whose parameters were fixed during training; that is, it was unnecessary to know its parameters in the framework.  $I$  was only used during the training phase to train the guiding chatbot via interaction. In the testing phase, the guiding chatbot will interact with real human beings. The interlocutor models' settings will be described in Section 5.3.
Guiding Chatbot The guiding chatbot model  $C$  was the chatbot we trained to induce desired responses in the interlocutor. We built the guiding chatbot model  $C$  based on DialoGPT (Zhang et al., 2020). To train model  $C$ , given the input sentence  $x$ , our chatbot  $C$  generated a sentence  $C(x)$ . The generated sentence  $C(x)$  then became the input for  $I$ , and  $I$  output its response  $I(C(x))$ . We defined the reward  $R$  for  $C$  based on  $C(x)$  and  $I(C(x))$ , and  $C$  was trained to maximize the value of  $R$  by the policy gradient. The definition of the reward  $R$  depended on the controllable factors, that is, the intention of the guiding chatbot (how the guiding chatbots wanted the interlocutor to respond). The
definition of our reward functions is in Section 3.3, and the controllable factors are in Section 4.
3.3 Rewards Functions
We introduce two kinds of reward functions: intention reward  $R_{I}$  and coherence reward  $R_{C}$ . The final reward that the guiding chatbot  $C$  learned to maximize will be a combination of  $R_{I}$  and  $R_{C}$ .
Intention To influence the interlocutor, the guiding chatbot  $C$  ought to learn from the interlocutor's reaction. To be more specific, we collected responses  $I(C(x))$  from the off-the-shelf chatbots when interacting with our guiding chatbot. Then the intention reward  $R_{I}$  was obtained by evaluating the interlocutor's responses, that is,  $I(C(x))$ , based on the controllable factors of guiding chatbot  $C$ . Using the intention reward allowed the guiding chatbot to induce the interlocutor to perform specifically according to the controllable factors, namely our intentions. The formulation of  $R_{I}$  depended on the controllable factors. To observe the effectiveness of guiding these interlocutor models, in this paper, we had three controllable factors, which were equal to our intentions: to extend the sentence length, to make the interlocutor speak with a particular emotion, and to induce the interlocutor to speak specific words. Exact formulation of rewards for different controllable factors will be given in Section 4.
Coherence Using the intention reward as the only reward leaded to a drawback that the guiding chatbot ignored the coherence between the input
$x$  and the generated response  $C(x)$ . To avoid this problem, an extra constraint on the guiding chatbot to maintain coherent responses was necessary: we applied another conversational model  $C'$  that served as a constraint maintaining coherence. Here we used the open-domain GPT-2 model as the  $C'$ . To be more specific, we estimated the difference in generated probability between  $C$  and  $C'$  and minimized the estimated difference. As a result,  $C$  would be less likely to produce responses unrelated to input  $x$  coherent to responses generated by  $C'$ . The additional reward  $R_C$  is defined as below.
$$
R _ {C} = P _ {C ^ {\prime}} (C (x) \mid x). \tag {1}
$$
$R_{C}$  was the likelihood that  $C^\prime$  generated the sentence  $C(x)$  given the input sentence  $x$ . This term served as a kind of regularization that avoids drift during training.
To sum up, the total reward is defined as:
$$
R = \lambda R _ {I} + (1 - \lambda) R _ {C}, \tag {2}
$$
where  $\lambda$  is the hyper-parameter.
4 Controllable Factors
Below are the three types of controllable factors studied in this paper.  $R_{I}$  in Section 3.3 could be either  $R_{L}$  for sentence length,  $R_{E}$  for emotion, or  $R_{W}$  for specific words, introduced below.
Sentence Length A chatbot that could inspire the interlocutor to become more talkative is desirable in many real world applications. We aimed to observe whether our chatbot was able to make the interlocutor more talkative, and extend the length of conversations. Hence, we counted the sentence length of interlocutor models' responses as  $R_{L}$ . By optimizing this reward, we anticipated that the guiding chatbot might extend sentence length from the interlocutor.
Emotion We studied whether our chatbot was capable of inducing the interlocutor to respond with different emotions. We selected eight emotions, including anger, anxiety, contentment, disgust, hope, joy, sadness, surprise. We selected the eight emotions such that two emotions are located in each of the four different quadrants of the Valence-Arousal coordinate (Russell, 1980).
To ascertain the emotion of sentences, we established an Emotion Detector. The Emotion Detector was an emotion classifier used to classify emotion
given an input sentence. We trained the Emotion Detector on the EnpatheticDialogue (ED) dataset (Rashkin et al., 2019). For each sentence, the Emotion Detector will employ a Valence-Arousal (VA) projection grounded on the Valence-Arousal coordinate (Russell, 1980; G. Paltoglou, 2013). Given an input sequence, the Emotion Detector would output a two-dimensional vector representing sequence's emotion, defined as emotional valence<sup>1</sup>. More details related to the Valence-Arousal Coordinate will be discussed in Section 5.2. We utilized the BERT (Devlin et al., 2019) architecture, a pretrained contextualized embedding model, to improve language understanding. Next, we fine-tuned the BERT model on an emotional classification task to enhance the model's capability of categorizing each emotion. The accuracy of our emotion detector was up to  $82\%$ , and, therefore, we could obtain a detected emotion and its emotional valence given an input sentence.
The Emotion Detector takes  $I(C(x))$  as input and predicted its emotional valence according to the VA coordinate. Therefore, we could calculate the Mean Square Error (MSE) between the emotional valence of the interlocutor models' responses and the target emotion's emotional valence as the reward  $R_{E}$ .
Specific Words We aimed to induce the interlocutor to speak with words from specific groups. These word groups, including Bad Words, Sports, and Food, were collected from Google's team<sup>2</sup> and the Enchanted Learning website<sup>3</sup>. To provoke the interlocutor to respond to the sentence including the specific words we want, we calculated the frequency of the specific words in a sentence. We counted the frequency of interlocutor models' responses that contain words in these word groups as  $R_W$ . We anticipated that the interlocutor can generate a sentence that contains more words from the specific group and still be coherent as well as fluent.
EmpatheticDialogues Dataset Rashkin et al. (2019) created an innovative dataset with around
1e.g.fear  $\coloneqq$  [-0.12,0.79],joy  $\equiv$  [0.85,0.15]
$^{2}$ https://gist.github.com/jamiew/1112488
3https://www.enschantedlearning.com/home.shtml
25K conversations, each consisting of a speaker and a listener. The participants, acting as the speaker, initiated the talks, and the psychologists, serving as the listener, responded to the speaker empathetically. The dataset covers 32 different emotion labels including positive, negative, and neutral emotions. They firmly ensure that each emotion in the dataset was evenly distributed. Nonetheless, a few emotion classes were quite similar, such as "sentimental" and "nostalgic". Thus, we merged these equivalent emotion classes into one emotion class.
5.2 Valence-Arousal Coordinate Projection
In Valence-Arousal Coordinate study (Russell, 1980; G. Paltoglou, 2013), researchers assigned emotional values to nineteen kinds of emotions. We performed supervised training of the Emotion Detector based on these known emotions on the ED dataset. Each emotion could be represented as a two-dimensional vector. Therefore, we could map each emotion to the coordinate on the VA space.
RL Training Details We applied the Policy gradient (Sutton et al., 2000) as our RL algorithm. To implement an RL training chatbot, we applied the DialogGPT model, which fine-tuned the GPT-2 model on 147M multi-turn dialogues from Reddit discussion threads. The GPT-2 model was a transformer-based model with 36 layers, 20 attention heads in each layer, 345M parameters, and an embedding size was 1024. This model was trained on the WebText dataset and 50,257 tokens with invertible byte pair encoding to preserve capitalization and punctuation. In our training procedure, we fine-tuned the DialogGPT model on the ED dataset based on the reward function mentioned in Section 3.3.
Interlocutor Models The interlocutor models had three different setups:
- The Publicly available Google bot (Vinyals and Le, 2015) $^4$  was trained on the dataset proposed by Danescu-Niculescu-Mizil and Lee (2011) with 220,579 conversational exchanges between 10,292 pairs. The whole corpus was split into training and testing sets.
- The same DialogGPT model mentioned in Section 5.3 was used here to act as the interlocutor. The weights of the model were fixed.
- A BERT-based Retrieval chatbot trained on the ED dataset. Given input sentences, the chatbot chose the corresponding response from the candidate pool. The BERT encoder first embedded the sentences into sentence embedding and then computed cosine similarity between the input sentences and all candidates to select the most likely option. The candidate pool was comprised of all sentences in the ED dataset, which contained approximately 100K sentences.
Aside from the reward scores related to the intentions, we also reported the following three metrics in the experiments.
Conditional Perplexity The Conditional Perplexity here was to measure the dialogue coherence between the output sentence and input sentence  $x$ . The equation is shown below.
$$
C P P L = \prod_ {i = 1} ^ {T} \frac {1}{\left(P \left(C (x) _ {i} | x\right)\right) ^ {1 / T}} \tag {3}
$$
$CPPL$  was the conditional perplexity, which was equal to the inverse of the product of each word's probability in the sentence  $C(x)$  given the input sentence  $x$ .  $T$  was the length of the sentence  $C(x)$ .
Perplexity Here we employed the pretrained GPT-2 language model to judge if the output sentence  $C(x)$  was an acceptable sentence. The computation of Perplexity (Chen et al., 1998) is shown below.
$$
P P L = \prod_ {i = 1} ^ {T} \frac {1}{\left(P \left(C (x) _ {i}\right)\right) ^ {1 / T}} \tag {4}
$$
Self-BLEU While BLEU score (Papineni et al., 2002) is usually used to measure the correctness in machine translation, Self-BLEU (Zhu et al., 2018) was used here to measure the diversity of chatbot responses; we calculated the average BLEU score between sentences in our testing result as the Self-BLEU score.
4https://github.com/Conchylicutor/ DeepQA
Interlocutor while training	Interlocutor while testing	Sentence Length	Emotion (Anxiety)	Specific Words (Food)
RL↑	CPPL ↓	PPL ↓	SB-3 ↓	RE↓	CPPL ↓	PPL ↓	SB-3 ↓	RW ↑	CPPL ↓	PPL ↓	SB-3 ↓
-	GPT-2	7.12	50.72	40.26	0.62	1.8	-	-	-	0.03	-	-	-
GPT-2	GPT-2	9.5	31.82	22.84	0.79	0.78	37.39	22.61	0.77	0.38	95.68	50.7	0.68
-	Google	3.74	49.89	39.81	0.62	1.82	-	-	-	0.02	-	-	-
Google	Google	10.14	110.59	41.67	0.91	0.7	26.57	10.99	0.8	0.002	97.84	48.27	0.8
-	Ret	12.77	50.3	39.47	0.62	1.77	-	-	-	0.08	-	-	-
Ret	Ret	19.79	76.55	18.46	0.94	0.75	26.57	10.98	0.8	1.29	69.0	35.7	0.81
GPT-2 + Google + Ret	GPT-2	8.52	0.52	0.51
Google	4.31	39.68	30.2	0.75	0.5	39.95	34.05	0.71	0.51	72.4	40.55	0.8
Ret	14.79	0.5	0.45
Google + Ret	GPT-2	7.95	49.31	36.13	0.78	0.53	40.27	34.15	0.71	0.08	64.33	15.55	0.99
GPT-2 + Ret	Google	5.75	59.95	21.56	0.8	0.49	41.65	33.36	0.73	0.00	64.18	51.8	0.99
GPT-2 + Google	Ret	14.85	44.0	37.9	0.71	0.51	40.0	35.71	0.72	0.12	246.34	15.6	1
Table 1: Results of metrics and rewards according to different controllable factors. The metrics of Conditional Perplexity (CPPL), Perplexity (PPL), and Self-BLEU3(SB-3) are only examined on the guiding chatbot. Rewards are calculated on the interlocutor models during testing. The baseline performance is tested by the original guiding chatbot, the DialoGPT pre-trained model that has not yet trained with any interlocutor model. Higher scores for  $R_{L}$  and  $R_{W}$  indicate better performance. Lower scores for  $R_{E}$ , CPPL, PPL, and SB-3 indicate better performance. The best results are boldfaced.
5.5 Human Evaluation Setups
For human evaluation, we recruited participants online. There were 19 participants; most of them were graduate or undergraduate students. Each participant was given several conversations, including an opening sentence and a corresponding response. They were asked to try to understand the conversation and provide a response to reply to the conversation. Therefore, we were able to collect numerous participants' responses to calculate rewards. Moreover, participants were asked to score the relevance of the guiding chatbot's response to the opening sentence. This task was rated on a Likert scale(Likert, 1932), ranging from 1 to 5: Score 1 means a firm disagreement, Score 3 meant neutral, and Score 5 meant an undoubted approval. Finally, we counted rewards from humans' responses corresponding to the methods mentioned in Section 4.
6 Discussion and Analysis
6.1 Extending Sentence Length
The first controllable factor was sentence length. We aimed to guide the interlocutor to say more words in a single sentence. Table 1 reveals that our chatbot possessed the ability to encourage the interlocutor to be more talkative. The guiding chatbot interacted with the Google model while training could induce the interlocutor model to increase its sentence length from 3 to 10 words on average. However, as the sentence length increased, the conditional perplexity rose simultaneously. The result
reflected that the guiding chatbot trained with the Google model was forced to generate weird sentences so that the interlocutor model would produce a longer sentence. In contrast, although the guiding chatbot trained with the Retrieval model suffered from the same problem, the conditional perplexity increased only slightly, from 50.3 to 76.55, and the sentence length was much longer. Still, the high Self-BLEU3 score indicates that our chatbot might encounter a low-diversity problem. Therefore, the guiding chatbot trained with the GPT-2 model was the most desirable and stable chatbot to extend the interlocutor's sentence length.
6.2 Guiding Emotion