SlowGuess/NAACL_2022_OCR huggingface dataset:
of the embeddings and Transformer hidden units are 768. The number of heads in the Transformer is 12. The number of layers in the Transformer is set as 2 and 12 respectively in the query encoder and decoder. The topic number is 15, and the similar user number is set as 10. The selected profile token number is 200 for the Weibo dataset and 30 for the Reddit dataset. The batch size is 128. Following (Holtzman et al., 2020), we adopt nucleus sampling as our decoding strategy. We use the Adam (Kingma and Ba, 2015) optimizer for training the refiners and AdamW (Loshchilov and Hutter, 2019) with a warm-up method for the generator. Our code is publicly available. $^3$
4.4 Evaluation
Metric-based Evaluation We first evaluate all methods by several metrics with respect to different aspects. (1) BLEU-1/2 (Papineni et al., 2002) and ROUGE-L (Lin and Och, 2004) are typical word overlap-based metrics for measuring the similarity between the generated response and the groundtruth. (2) Distinct-1/2 (Li et al., 2016b) consider the number of uni- or bi-grams in the generated response, which is commonly used for evaluating the diversity. (3) The embedding-based metrics (i.e., average, extrema, and greedy) (Chan et al., 2019) are introduced to measure the semantic similarity between the generated response and the groundtruth one. (4) As a personalized dialogue model, following previous studies (Ma et al., 2021), two tailored metrics are adopted to measure how much information is included in the dialogue history that can be reflected in the response. Persona-F1 (PF1) (Lian et al., 2019) calculates the F1 value to measure the uni-grams co-occurring in both the generated response and the dialogue history. Persona Coverage (P-Cover) (Song et al., 2019) calculates the IDF-weighted word overlap between the generated response and the golden one so that the importance of different words can be taken into account.
Human Annotation Due to the variability of human language, a response that differs from the ground-truth may also be appropriate. Following previous studies (Chan et al., 2019), we conduct a human evaluation of all methods. Concretely, we
3https://github.com/bangbangbang12315/ MSP/tree/release.
4The results of BLEU-3/4 and ROUGE-1/2 are provided in Appendix A.
Table 3: Metric-based evaluation result on Weibo and Reddit dataset. The best results are in **bold**. “†” indicates that our model achieves significant improvement in t-test with  $p$ -value  $< 0.05$ .
Overlap-based Metric	Diversity	Embedding Metric	Persona Metric
BLEU-1	BLEU-2	ROUGE-L	Dist-1	Dist-2	Average	Extrema	Greedy	P-F1	P-Cover
Weibo	Seq2Seq	3.330†	0.294†	8.985†	0.930†	2.180†	0.321†	0.266†	0.254†	0.154†	0.041†
MMI	3.631†	0.095†	5.264†	10.710†	43.458†	0.477†	0.696†	0.305†	0.325†	0.054†
DialoGPT	6.068†	0.741†	8.459†	15.322†	55.536†	0.557†	0.793†	0.324†	0.522†	0.061†
GPMN	4.899†	0.696†	7.785†	11.724†	32.730†	0.353†	0.391†	0.301†	0.542†	0.084†
PerCVAE	5.114†	0.299†	7.380†	14.098†	49.733†	0.469†	0.657†	0.299†	0.903†	0.086†
Speaker	4.994†	0.113†	7.868†	6.035†	19.007†	0.492†	0.712†	0.311†	0.225†	0.082†
PersonaWAE	3.510†	0.155†	10.546†	2.551†	19.743†	0.563†	0.757†	0.307†	1.740†	0.103†
DHAP	9.324†	0.894†	14.122†	15.175†	58.806†	0.523†	0.747†	0.313†	1.791†	0.144†
MSP (Ours)	11.875	5.108	15.563	24.203	73.196	0.605	0.883	0.331	2.170	0.297
Reddit	Seq2Seq	1.820†	0.023†	4.069†	5.203†	19.485†	0.545†	0.554†	0.470†	0.051†	0.029†
MMI	2.065†	0.011†	3.784†	5.914†	31.093†	0.543†	0.607†	0.454†	0.085†	0.038†
DialoGPT	4.735†	0.397†	8.943†	6.353†	29.106†	0.604†	0.733†	0.448†	0.137†	0.040†
GPMN	2.686†	0.376†	4.776†	12.325†	35.762†	0.406†	0.331†	0.358†	0.189†	0.037†
PerCVAE	5.933†	0.576†	8.112†	9.631†	40.213†	0.637†	0.649†	0.499†	0.212†	0.040†
Speaker	2.642†	0.054†	4.469†	8.951†	34.187†	0.538†	0.606†	0.457†	0.115†	0.031†
PersonaWAE	2.637†	0.113†	8.199†	1.758†	25.915†	0.629†	0.685†	0.442†	0.206†	0.032†
DHAP	6.858†	0.737†	11.720†	18.707†	66.932	0.709	0.721†	0.539	0.227†	0.111†
MSP (Ours)	7.174	0.883	12.171	21.247	68.897	0.716	0.764	0.545	0.276	0.137
sample 100 (query, response, user dialogue history) triplets and hire three well-educated annotators to score the responses generated by different models. Three aspects, i.e., readability, informativeness, and personalization, are considered. The first two factors are scored on a scale of [1, 3] for their quality, while the third is assessed on a scale of [0, 1], indicating whether the response can accurately reflect the user's personality. The detailed scoring criteria are shown in Table 2.
Metric-based Evaluation Table 3 shows all models' performance under different metrics. On both datasets, it is clear to see that our MSP model outperforms baselines on all metrics. The improvement is statistically significant (t-test with  $p$ -value  $< 0.05$ ). These findings indicate that our model is capable of generating more fluent, diverse, and personalized responses than all baselines. In particular, we can observe: (1) MSP achieves better performance on overlap-based metrics. This suggests that our model can provide responses that are more similar to the ground-truth with the help of the selected tokens. (2) For diversity metrics, the higher distinct values show that our generated responses are more diverse. Additionally, predefined profile-based methods and user dialogue history-based methods outperform others. This shows that incorporating external information can aid in the
generation of more informative responses. (3) In addition to generating more overlapped words with the ground-truth response, the improvements in embedding metrics reflect that our model generates more semantically relevant responses. (4) Finally, the increase in personalized metrics implies that our approach can incorporate more user-specific information into the generation. Furthermore, the significant improvement over DHAP demonstrates that our model can extract more meaningful personalized information from the user dialogue history.
Human Annotation The result of human annotation on the Weibo dataset is shown in Table 4. The Fleiss Kappa is around 0.62, indicating a substantial agreement achieved by three annotators. In general, the results of human annotation are consistent with those of the metric-based evaluation. Both of them demonstrate our model's superiority in generating more fluent, informative, and personalized responses. Compared to non-personalized methods, user id-based methods can enhance personalization at the expense of readability. User dialogue history-based methods (i.e., DHAP and MSP) can largely improve the personalization of the response while retaining a high level of readability and informativeness. We attribute this to the abundant personal information contained in the user dialogue history.
Table 4: The result of human evaluation on Weibo dataset. “†” indicates that our model achieves significant improvement in t-test with  $p$ -value  $< 0.05$ .
Model	Readability	Informativeness	Personality
Seq2Seq	1.76†	1.37†	0.11†
MMI	1.96†	1.88†	0.19†
DialoGPT	2.33	2.10†	0.32†
GPMN	2.01†	2.16†	0.35†
PerCVAE	2.10†	2.01†	0.39†
Speaker	1.89†	1.44†	0.24†
PersonaWAE	1.81†	2.01†	0.32†
DHAP	2.29†	2.19†	0.55†
MSP (Ours)	2.37	2.39	0.67
Ground-Truth	2.71	2.66	0.76
Table 5: The results of ablation experiments on Weibo dataset.
Models	BLEU-1	BLEU-2	P-Cover
MSP (Full)	11.875	5.108	0.297
w/o User Refiner	6.093	0.757	0.151
w/o Topic Refiner	6.163	0.839	0.178
w/o Token Refiner	4.213	0.609	0.116
w/o Current U's Profile	9.365	3.146	0.238
w/o Similar Us' Profile	6.413	0.871	0.245
w/o Joint Training	6.070	0.749	0.130
We further conduct a series of analyses to elaborate our model. All analyses here are based on the result of the Weibo dataset, while similar results can be observed on the Reddit dataset.
Ablation Study To investigate the impact of different modules in MSP, we conduct an ablation study by removing or using different strategies in each module.
We first study the influence of the refiners at three levels: (1) We remove the user refiner and train our model using randomly sampled users. We can see the performance of all metrics drops. This illustrates that our MSP model can select users that share the same interests as the current user and thereby improving response quality. (2) We remove the topic refiner and supply the token refiner with full dialogue history. The performance degradation demonstrates that various topics in dialogue history introduce lots of noise, misleading the token refiner on extracting valuable tokens, thus impairing the personalized response generation. (3) We eliminate the token refiner and feed all dialogue history sentences directly into the generator.5 The
Figure 2: Comparison with the retrieval-based model on the Weibo dataset.
decline in performance implies the effectiveness and necessity of token selection. It is worth noting that, as compared to using the complete history, our selection strategy can reduce training time by  $41.6\%$ , considerably increasing efficiency. All of the aforementioned experimental results suggest that MSP's advantage stems from high-quality personalized information extraction rather than simply introducing additional information.
We then explore the impact of personalized information from two sources, i.e., the current user's profile and the similar users' profile. Removing either of them results in decreased performance. This exemplifies their usefulness. Specifically, compared with similar users' profiles, eliminating the current user's profile will hurt the personalization effect heavily. This result shows that, for personalization, the current user's profile is more essential than that of similar users, which is quite intuitive. However, the similar users' profile has a significant effect on BLEU-1/2, implying that such a profile can provide abundant information for response generation. Consequently, integrating both types of profiles significantly improves response generation.
Finally, we conduct an experiment to validate our proposed joint training for the token refiner. The declining performance indicates that the token refiner is unable to extract useful information in the absence of additional supervision signals. Indeed, when the sentence matching task is removed, the token refiner extracts tokens that are relevant to the current query, which is less useful for generating a personalized response.
Influence of Selection Mechanism To validate the effectiveness of our proposed selection mechanism, we replace the refiner with a traditional retrieval method (i.e., BM25 (Robertson and Walker, 1994)). Specifically, we use the query to retrieve
Due to the length limitation of GPT-2, history with more than 512 tokens will be truncated.
Figure 3: Experiments with the different number of user profiles on the Weibo dataset. To keep the dimension consistent, P-Cover is multiplied by a factor of 100.
15 relevant responses and feed them into our model for training. The experimental results are shown in Figure 2. We can observe that the retrieval strategy achieves comparable performance with our model on word-overlap and embedding-based metrics. This suggests that the relevant dialogue history for the query can provide valuable information for response generation. However, the retrieval strategy performs poorly on diversity and personalization metrics. This demonstrates that, without careful selection, the retrieved information is too generic and thus less helpful for personalized response generation.
Influence of Personalized Tokens Amount In MSP, three refiners are designed to extract personalized tokens for response generation. Intuitively, the amount of the tokens will influence the refiner's performance. We report this influence in Figure 3. As we can see in the left part, the quality of response generation improves with more tokens used. This is because fewer tokens are incapable of covering sufficient personalized information for response generation. Our MSP model performs optimally when about 200 personalized tokens are selected. When more tokens are introduced, the performance degrades. The potential reason is that more tokens would bring the noise to the generation. This is consistent with our speculation that the dialogue history is noisy and the information selection is both effective and necessary.
To provide a more qualitative view of our method, we conduct a case study in Appendix B.
In this work, we propose an MSP model for personalized response generation. Unlike previous related work, we utilize a refiner structure to extract query-
aware persona information from large-scale dialogue history. The multi-level refiners can sparsely extract valuable information from dialogue history and leverage similar users' information to enhance the current user's personalization. Experimental results confirm the effectiveness of our model in generating informative and personalized responses.
Zhicheng Dou is the corresponding author. This work was supported by National Natural Science Foundation of China No. 61872370 and No. 61832017, Beijing Outstanding Young Scientist Program NO. BJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China. We also acknowledge the support provided and contribution made by Public Policy and Decision-making Research Lab of RUC.
Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. CoRR, abs/1606.00372.
Jin Yeong Bak and Alice Oh. 2019. Variational hierarchical user-based conversation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1941-1950. Association for Computational Linguistics.
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. 2019. Retrieval-guided dialogue response generation via a matching-to-generation framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 1866-1875. Association for Computational Linguistics.
Zhangming Chan, Juntao Li, Xiaopeng Yang, Xiuying Chen, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019. Modeling personalization in continuous space for response generation via augmented wasserstein autoencoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages
1931-1940. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. IEEE Trans. Big Data, 7(3):535-547.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27 - June 1, 2003. The Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119. The Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016b. A diversity-promoting objective function for neural conversation models. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110-119. The Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016c. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1865-1874. Association for Computational Linguistics.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. 2019. Learning to select knowledge for response generation in dialog systems. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5081-5087. ijcai.org.
Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July, 2004, Barcelona, Spain, pages 605-612. ACL.
Ya-Fei Liu, Hongjin Qian, Hengpeng Xu, and Jinmao Wei. 2020. Speaker or listener? the role of a dialogue agent. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of *Findings of ACL*, pages 4861-4869. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421. The Association for Computational Linguistics.
Zhengyi Ma, Zhicheng Dou, Yutao Zhu, Hanxun Zhong, and Ji-Rong Wen. 2021. One chatbot per person: Creating personalized chatbots based on implicit user profiles. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 555-564. ACM.
Oluwatobi Olabiyi, Anish Khazane, Alan Salimov, and Erik T. Mueller. 2019. An adversarial learning framework for A persona-based multi-turn dialogue model. CoRR, abs/1905.01992.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL.
Hongjin Qian, Zhicheng Dou, Yutao Zhu, Yueyuan Ma, and Ji-Rong Wen. 2021a. Learning implicit user profiles for personalized retrieval-based chatbot. CoRR, abs/2108.07935.
Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021b. Pchatbot: A largescale dataset for personalized chatbot. In Proceedings of the SIGIR 2021. ACM.
Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Assigning personality/profile to a chatting machine for coherent conversation generation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4279-4285. ijcai.org.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SIGIR Forum), pages 232-241. ACM/Springer.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 3776-3784. AAAI Press.
Haoyu Song, Weinan Zhang, Yiming Cui, Dong Wang, and Ting Liu. 2019. Exploiting persona information for diverse generation of conversational responses. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5190-5196. ijcai.org.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. CoRR, abs/1506.05869.
Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI