SlowGuess/ACL_2020_OCR huggingface dataset:
where temperature  $\tau > 0$  is a tunable parameter (Hinton et al., 2015) controlling the evolution of  $Agg$ . In the beginning,  $Agg$  behaves close to average pooling. As  $\tau$  anneals,  $Agg$  gradually focuses more on the highest relevance score. In this way, noise reduces as training goes on. Finally,  $c^{\Delta}$  is given by:
$$
c ^ {\Delta} = \frac {1}{N} \sum_ {n = 1} ^ {N} A g g \left(\mathbf {U} _ {n,:} ^ {\Delta}\right). \tag {14}
$$
Given  $x_{n}^{\mathcal{A}}$  and  $\mathbf{w}^{\mathcal{A}}$ , Receiver employs the following function to obtain  $x_{n}^{\mathcal{A}}$ 's persona perception score, further modeling mutual persona perception as in Equation 9:
$$
\operatorname {s c o r e} \left(x _ {n} ^ {\mathcal {A}}, \mathbf {w} ^ {\mathcal {A}}\right) = \frac {A g g \left(\mathbf {H} _ {n , :} ^ {\mathcal {A}} \left(\mathbf {W} ^ {\mathcal {A}}\right) ^ {\top}\right)}{\sqrt {d}}, \tag {15}
$$
where  $\mathbf{H}_{n,:}^{\mathcal{A}}$  and  $\mathbf{W}^{\mathcal{A}}$  are the impression encoding and persona encoding for  $x_{n}^{\mathcal{A}}$  and  $\mathbf{w}^{\mathcal{A}}$  respectively.
We conducted experiments on the dataset PERSONA-CHAT, assessing  $\mathcal{P}^2$  BOT using both automatic metrics and human evaluations. To verify
the effectiveness of our proposed mutual persona perception, we perform a thorough model analysis in Section 5.3. Finally, we probe Receiver's capability on perceiving persona in Section 5.4.
PERSONA-CHAT dataset contains 8,939/1,000 multi-turn dialogues conditioned on 1,155/100 personas for train/dev. Each persona is described with at least 5 profile sentences. To make it more challenging, PERSONA-CHAT also provides revised personas by rephrasing, generalizing or specializing the original ones. For example, "I am overweight." is revised from "I weight 300 pounds."
Our implementation was based on PyTorch (Paszke et al., 2019), ParlAI (Miller et al., 2017), and HuggingFace's transformers library (Wolf et al., 2019a). We used Adam (Kingma and Ba, 2015) optimizer with a learning rate of  $6.25\mathrm{e - }5$  for both Receiver and Transmitter in supervised learning. In the training of Receiver,  $\tau$  reduced linearly from 10 to 0.5. In the self-play phase of Transmitter, the learning rate was set as 1e-6. The hyperparameters  $m,\alpha ,\beta ,\gamma ,\lambda_1,\lambda_2$  and  $\lambda_{3}$  were set as 0.4, 0.1, 1e-4, 0.5, 0.4, 0.1 and 0.5 respectively. The supervised training of Transmitter lasted for 2 epochs, and the self-play fine-tuning comprised 2000 dialogues, where the number of turns was 3. The beam search size was set as 2.
5.2 Methods Comparison
Our baselines fall into three categories: retrieval-based, generative-based and pretrain-finetune-based models. Among the retrieval-based baselines, KV Profile Memory (Zhang et al., 2018b) was the official baseline which employed the memory network along with profile information, and
Model	1 (%)	2 (%)	3 (%)	4 (%)	Avg
Lost In Conversation	26.3	48.7	22.0	3.0	2.017
Transfertransfo	41.7	25.3	28.7	4.3	1.956
P2 BOT (Our)	18.9	26.3	28.6	26.2	2.621
Dually Interactive Matching Network (Gu et al., 2019) proposed a dual matching architecture to match between the responses and their corresponding contexts. Language Model, Generative Profile Memory (Zhang et al., 2018b) and SEQ2SEQ with attention mechanism (Bahdanau et al., 2015) were implemented as generative baselines for dialogue generation. The remaining methods were all pretrain-finetune-based. Transfertransfo (Wolf et al., 2019b) $^3$  achieved the state-of-the-art performance on automatic metrics, while Lost In Conversation $^4$  topped the human evaluations (Dinan et al., 2019). Analogous to our approach, they employed the pretrained language model GPT to initialize their models, and then fine-tuned it on the dataset.
Table 1 shows the experimental results on automatic metrics. Following Zhang et al. (2018b), we reported the official automatic metrics to evaluate the methods: Hits@1, Perplexity (ppl) and F1. Given 20 response candidates, Hits@1 is the probability that the real response ranks the highest according to the model. Perplexity measures the negative log likelihood of the correct sequence output by the model, lower values indicating better performance. F1 is the harmonic mean of word-level precision and recall. As observed, our approach outperforms almost all baselines and achieves new state-of-the-art performance on ppl and F1, with highly competitive performance on Hits@1. In the revised mode, our approach still achieves the best performance, obtaining a relative improvement of  $13.4\%$  on F1 against the strongest baseline. It is worth noting that we also tried to employ F1 as the reward, but the result is far from satisfactory.
As mentioned in Dinan et al. (2019), no automatic metric is perfect for evaluating such an open-domain task. Hence, we also performed crowdsourced human evaluations on the state-of-the-art baselines (i.e. Transfertransfo & Lost In Conversation) and our proposed  $\mathcal{P}^2$  BOT. Concretely, on the original dev set, we randomly sampled 200 responses generated by these methods and asked each worker to rate them. The rating ranges from 1
Table 2: Human evaluation results.
Variant	Hits@1(%)↑	F1(%)↑	BLEU(%)↑
P2 BOT-S	68.7	18.14	0.56
- Persona	65.5	17.77 (-2.0%)	0.57 (+ 1.8%)
- Next	17.6	18.11 (-0.1%)	0.55 (- 1.8%)
+ RS.1	68.4	18.32 (+0.9%)	0.60 (+ 7.1%)
← + RS.2	68.6	18.41 (+1.5%)	0.61 (+ 8.9%)
← + RS.3	68.6	19.08 (+5.2%)	0.75 (+33.9%)
Table 3: Variant analysis results on PERSONA-CHAT revised mode, along with relative improvements (shown inside brackets) compared with  $\mathcal{P}^2$  BOT-S. BLEU refers to the cumulative 4-gram BLEU score. "Persona" means dialogue generation without personas; " - Next" ablates the auxiliary task mentioned in Section 3.1;  $^{+}\mathrm{RS}.1^{*}$  means only using Language Style score as the reward in the self-play fine-tuning phase;  $" \hookrightarrow +\mathrm { R S } . 2 ^ { \prime \prime }$  means adding Discourse Coherence to the reward on the basis of RS.1;  $" \hookrightarrow +\mathrm { R S } . 3 ^ { \prime \prime }$  is equivalent to our proposed  $\mathcal{P}^2$  BOT.
to 4. 1 means the response is good only in terms of grammar and sentence structure; 2 means in addition to valid grammar, the response is also coherent with the context; 3 means the coherent response is meanwhile interesting and informative, instead of just a simple response like "Yes"; And 4 means the response is consistent with the persona of the interlocutor, which is of extreme importance for the task of reflecting whether the model can effectively utilize the persona information. As shown in Table 2, the results are consistent with the automatic evaluation results, demonstrating the superiority of  $\mathcal{P}^2$  BOT against the baselines. We also conducted Wilcoxon signed-rank tests between our method and the baselines and the results show the improvements are significant with  $p < 0.05$ .
Variant Analysis We conducted variant analysis on  $\mathcal{P}^2$  BOT to investigate the influence of RS.1, RS.2 and RS.3. Another metric BLEU (Papineni et al., 2002), which evaluates the quality of response, was introduced to make the analysis more comprehensive. We show the variant analysis results in Table 3, where  $\mathcal{P}^2$  BOT-S is the variant of  $\mathcal{P}^2$  BOT which is trained only in the supervised setting. As expected, the results on Hits@1 validate the important role of the auxiliary task. Across all the variants, the gains in BLEU and F1 are very small, revealing the difficulty in improving them. Nevertheless, solely by adding RS.3, we obtained a  $25\%$  relative improvement on BLEU, indicating the effectiveness of our proposed mutual persona
<sup>3</sup>http://github.com/huggingface/transfer-learning-conv-ai
<sup>4</sup>http://github.com/atselousov/transformer-chatbot
PERSONA	i. I love new kids on the block.
ii. I was born in the early 80's.
iii. I also like old school hip hop.
iv. My favorite toy as a child as my lite brite.	i. I am a blonde girl with really short hair.
ii. I love wearing skinny jeans and leggings.
iii. I'm rather skinny as I like to stay in shape.
iv. My favorite hobbies are listening to music and playing video games.	i. I've an allergy to water.
ii. I'm a famous twitch streamer.
iii. I helped design the game StarCraft.
iv. I am working on building a robot to be my body double.
CONTEXT	Do you like clowns? They horrified me!	What video game? I relax by knitting. Love it!	Cool. I did hardcore exercise four hours everyday.
* HUMAN	No! My favorite toy as a kid was lite brite because I was so scared!	I was playing Fallout. What do you look like?
I'm blonde thin.	That's a lot of exercising. I hate exercising.
I just work on my robot most days.
* P² BOT	I do not like them at all. I am a little old for them.	I play RPGs. I like to wear skinny jeans and tees.	I do not exercise at all. I'm busy with my robot project.
* TRANSFERTRANSFO	I do not have time for clowns. Do you like to listen to music?	My fav color is blue, I have a lot of friends in my group.	I work at a computer company. I could make you an Android!
* LOST IN CONVERSATION	I love clowns. they are my favorite.	I love HALO 3, what do you knit?	That sounds like a lot of fun!
Table 4: Sampled responses(*) by Human,  $\mathcal{P}^2$  BOT and the state-of-the-art baselines.
Model	Original	Revised
Hits@1↑	MRR↑	Hits@1↑	MRR↑
Random	3.1	0.2	3.1	0.2
IR	67.5	20.9	9.7	2.2
Receiver	93.8	37.5	78.2	16.6
Table 5: Experimental results on Persona Perception.
perception. Similar conclusions can be drawn from the trend of F1.
Case Study For a more comprehensive comparison, we show in Table 4 some randomly sampled responses of different methods. The results suggest the responses generated by our approach are more human-like. As observed, benefiting from our proposed mutual persona perception, the responses of  $\mathcal{P}^2$  BOT are more consistent, engaging and informative. For instance, in the last example in Table 4, the response "I'm busy with my robot project" explicates why the speaker does not exercise, meanwhile revealing that he is working on the robot, as depicted in his persona.
Error Analysis Though our approach works well in most cases, we observed that the self-play simulation might fall into repeated cycles after rounds of training, as the challenge mentioned by Li et al. (2016b). Another issue is that the bots sometimes ask redundant questions in our approach, which might be due to inappropriate hyperparameters in reward shaping.
5.4 Persona Perception Probing
Receiver plays an important role in our approach, and we are interested in its capability on perceiving persona. Therefore, we conducted experi
Figure 6: Visualization of the relevance scores between a sampled dialogue and its corresponding revised persona. Deeper color means higher score. We omit some context due to space limitation.
ments on a synthesized dataset. We constructed the dataset by sampling 31 persona distractors for each dialogue in PERSONA-CHAT. Two widely used ranking metrics were used to evaluate the performance: Hits@1 and Mean Reciprocal Rank (MRR). Hits@1 is the same metric as the one mentioned in Section 5.2, except that the candidate size is 32. Given a dialogue and the complete set of profile sentences, MRR is the average reciprocal ranks of the dialogue-relevant profile sentences. Two simple baselines Random and IR (Sordoni et al., 2015) were chosen for comparison. Table 5 shows the experimental results of different methods on the synthesized dataset. As observed, our approach achieved excellent results on both original and revised modes. For example, compared with the IR baseline, our approach achieved an absolute improvement of  $26.3\%$  on Hits@1 in the original mode. In addition, the surprising results in the revised mode further demonstrate Receiver's capability to perceive rephrased persona.
To further understand the trained Receiver, we visualize the relevance scores between a sampled
dialogue and its corresponding revised persona in Figure 6. As illustrated, the relevance scores between related profile sentences and dialogue utterances are significantly higher. For example, the utterance "I volunteer at the local pool" from the interlocutor implies the profile "I love being in the water", and our Receiver successfully captures the relevance between them.
Methods to build open-domain dialogue systems generally fall into two major categories: retrieval-based and generative-based. Retrieval-based methods retrieve response candidates and rank them based on the matching scores with the dialogue (Sordoni et al., 2015; Wu et al., 2017; Gu et al., 2019). Generative-based methods typically use SEQ2SEQ model as the backbone (Sutskever et al., 2014; Bahdanau et al., 2015; Serban et al., 2017; Wolf et al., 2019b), where the encoder extracts the information in an utterance and the decoder generates the response. Our work adopts a similar architecture. Besides supervised learning, researchers also explore reinforcement learning based methods. Lewis et al. (2017) applied reinforcement learning for negotiation dialogues and showed it outperforms supervised learning when negotiating with humans. Yang et al. (2018) proposed to generate dialogue responses by dual learning based domain adaptation. Zhang et al. (2018a) built a coherence model to provide the reward signal for penalizing dull responses. Liu et al. (2019) employed reinforcement learning to learn an intermediate structure span. Our approach differs from this line of work in that we focus on improving personalized dialogues via mutual persona perception, which has not yet been explored before.
More recently, under the topic of dialogue personalizing, Zemlyanskiy and Sha (2018) proposed a post-processing method to re-rank candidates generated by beam search, while Olabiyi et al. (2019) employed adversarial approaches to solve the consistency problem on interlocutors' names. Madotto et al. (2019) applied meta-learning to quickly adapt to new speakers, and Tigunova et al. (2019) extracted user attributes from daily dialogues. Compared with them, our work enhances persona based dialogue generation from a novel perspective.
Furthermore, researchers explored to generate diverse responses conditioned on persona (Song et al., 2019, 2020). Personalization in goal-oriented di
alogue systems has also received some attention (Joshi et al., 2017; Luo et al., 2019). The researches focus more on making the goal-oriented bots adjust the response according to different user profiles, while we aim to endow bots with persistent personalities.
We propose  $\mathcal{P}^2$  BOT, a transmitter-receiver framework which explicitly models understanding between interlocutors. Under this framework, mutual persona perception is incorporated as a reward signal to achieve the personalized dialogue generation. Experiments on a large public dataset PERSONA-CHAT demonstrate the effectiveness of our approach. For future work, we would like to extend Receiver to conversational recommender systems. After turns of chatting, the agent should be able to infer the user's persona, based on which personalized contents can be recommended.
We thank all the anonymous reviewers for their valuable comments. This work was supported in part by National Natural Science Foundation of China (U1736217 and 61932003), and National Key R&D Program of China (2019YFF0302902).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, San Diego, CA, USA.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2019, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. 2019. The second conversational intelligence challenge (convai2). CoRR, abs/1902.00098.
Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, and Quan Liu. 2019. Dually interactive matching network for
personalized response selection in retrieval-based chatbots. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China. Association for Computational Linguistics.
Uri Hasson, Asif A Ghazanfar, Bruno Galantucci, Simon Garrod, and Christian Keysers. 2012. Brain-to-brain coupling: a mechanism for creating and sharing a social world. Trends in cognitive sciences.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.
Chaitanya K. Joshi, Fei Mi, and Boi Faltings. 2017. Personalization in goal-oriented dialog. CoRR, abs/1706.07503.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, San Diego, CA, USA.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany. Association for Computational Linguistics.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas. Association for Computational Linguistics.
Qian Liu, Bei Chen, Haoyan Liu, Jian-Guang Lou, Lei Fang, Bin Zhou, and Dongmei Zhang. 2019. A split-and-recombine approach for follow-up query analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China. Association for Computational Linguistics.
Liangchen Luo, Wenhao Huang, Qi Zeng, Zaiqing Nie, and Xu Sun. 2019. Learning personalized end-to-end goal-oriented dialog. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI
2019, The 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, January 27 - February 1, 2019, Honolulu, Hawaii, USA. AAAI Press.
Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019. Personalizing dialogue agents via meta-learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, Florence, Italy. Association for Computational Linguistics.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, Brussels, Belgium. Association for Computational Linguistics.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2017, Copenhagen, Denmark. Association for Computational Linguistics.
Oluwatobi Olabiyi, Anish Khazane, Alan Salimov, and Erik T. Mueller. 2019. An adversarial learning framework for a persona-based multi-turn dialogue model. CoRR, abs/1905.01992.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL 2002, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. Curran Associates, Inc.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville,
and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, AAAI 2019, February 4-9, 2017, San Francisco, California, USA. AAAI Press.
Haoyu Song, Wei-Nan Zhang, Jingwen Hu, and Ting Liu. 2020. Generating persona consistent dialogues by exploiting natural language inference. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, AAAI 2020, February 7-12, 2020, New York City, New York, USA. AAAI Press.
Haoyu Song, Weinan Zhang, Yiming Cui, Dong Wang, and Ting Liu. 2019. Exploiting persona information for diverse generation of conversational responses. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, August 10-16, 2019, Macao, China.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2015, Denver, Colorado. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, NIPS 2014, December 8-13 2014, Montreal, Quebec, Canada.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12: Annual Conference on Neural Information Processing Systems 1999, NIPS 1999, November 29 - December 4, 1999, Denver, Colorado, USA.. The MIT Press.
Anna Tigunova, Andrew Yates, Paramita Mirza, and Gerhard Weikum. 2019. Listening between the lines: Learning personal attributes from conversations. In Proceedings of the World Wide Web Conference, WWW 2019, May 13-17, 2019, San Francisco, CA, USA. ACM.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NIPS 2017, December 4-9, 2017, Long Beach, CA, USA.
Lex Weaver and Nigel Tao. 2001. The optimal reward baseline for gradient-based reinforcement learning.