SlowGuess/NAACL_2021_OCR huggingface dataset:
Our implementation is based on the Pytorch version of OpenNMT (Klein et al., 2017). We used the pre-trained Dialogpt word embedding (Zhang et al., 2020). The hidden size of the encoder and decoder were set to 1024. The embedding size is the same as the memory size and the RNN hidden size. We used AdamW (Loshchilov and Hutter, 2018) as our optimizer with an initial learning rate of 5e-5 and a linear decay learning rate schedule. The dropout rate was set to 0.1. The batch size was selected in {16, 32, 64, 128}. The maximum number of iteration steps was set as 20000 with an early stop if no improvement over perplexity on dev set. To generate hypothesis sentences, we used nucleus (top-p) filtering (Holtzman et al., 2019) without any re-scoring techniques. The cumulative probability for top-p filtering is set as 0.4.
We introduced several baselines to compare with our generative split memory network (GSMN).
- Attention-Seq2Seq: a standard seq2seq model with attention mechanisms proposed by Luong et al. (2015), without utilizing any personal information.
- Speaker Model: Similar to (Li et al., 2016b), we employed an additional vector to model the respondent  $A$ .
- Generative Memory Network w/ History: Following Zhang et al. (2018), we encoded the retrieved comments as individual memory representation in a memory network to incorporate comment histories  $\mathbb{M}$ .
- Generative Memory Network w/ Profile: We designed a memory network model to incorporate user profiles  $P$  by doing attention over user attributes (Zheng et al., 2020).
- Dialogpt: The state-of-the-art large-scale pretrained response generation model on 147M Reddit corpus(Zhang et al., 2020).
<sup>7</sup>https://github.com/OpenNMT/OpenNMT-py
Model	User Information	Perplexity	BLEU	C	PC-Score
Attention-Seq2Seq	-	110.920	1.324	0.300	0.00989
Speaker Model	username	92.607	1.329	0.301	0.0102
Generative Memory Network	profile	75.635	1.592	0.304	0.0131
Generative Memory Network	history	80.000	1.664	0.305	0.0120
Dialogpt	-	58.723	3.246	0.306	0.0182
Dialogpt	profile+history	36.764	5.894	0.309	0.0237
Generative Split Memory Network	profile+history	72.173	1.700	0.306	0.0152
Dialogpt w/ Split Memories	profile+history	33.519	7.047	0.311	0.0337
Table 6: Automatic results on PER-CHAT test sets.
- Dialogpt w/ Split Memories We directly combined the split memory network with the pretrained models, i.e. we applied the same architecture as GSMN in the decoder side and used pre-trained Dialogpt as the encoder.
We evaluated the baselines and our generative split memory network using several widely-used metrics, including perplexity, BLEU, and BERTScore, to compare models' performances in generating appropriate responses. Perplexity is used to measure how the outputs fit test data (Vinyals and Le, 2015; Serban et al., 2016). Models with lower perplexity scores are found to demonstrate better performance to generate grammatical and fluent responses (Xie et al., 2019; Zheng et al., 2020). We also used BLEU (Papineni et al., 2002; Li et al., 2016a; Galley et al., 2015) with  $n$ -grams ( $n = 1$ ) to measure how many  $n$ -grams in generated responses overlap with those in reference responses. To evaluate persona consistency between user comments and generated sentences, Madotto et al. proposed consistency C score using sequence classification model trained on Dialog NLI dataset (Welleck et al., 2019), a corpus based on Persona dataset, with NLI annotation. For given comments  $p_j$ s and generated sentence  $u$ , the consistency score is given as follow:
$$
\begin{array}{l} \mathbf {N L I} \left(u, p _ {j}\right) = \left\{ \begin{array}{c c} 1 & \text {i f u e n t a i l s p _ {j}} \\ 0 & \text {i f u i s i n d e p e n d e n t t o p _ {j}} \\ - 1 & \text {i f u c o n t r a d i c t s p _ {j}} \end{array} \right. \\ \boldsymbol {C} (u) = \sum_ {j} ^ {m} \mathbf {N L I} (u, p _ {j}) \tag {6} \\ \end{array}
$$
Note that models with higher consistency C scores tend to generate more persona consistent responses with user's comments. In our settings,  $m$  is set 10 for max number of given comments.
PC-Score In addition to the aforementioned evaluation metrics, we also designed a metric called Profile Consistency Score (PC-Score) to measure a model's performance in generating persona consistent responses with given user profiles. The idea is similar to the entity score in knowledge enhanced conversation tasks (Zhou et al., 2018), which computes the number of entities for each response and aims to measure the model's ability to select the concepts from the commonsense knowledge. Instead of calculating the number of entities selected from knowledge base per response, we did a micro-average over the number of entities selected from the profile of each user to capture personalization.
Manual Evaluation We conduct manual annotations to examine the consistency of those models. Here, the consistency refers to that the generated responses should be consistent for the same user when similar questions are asked. For example, when asked "Where are you from?" and "Where is your hometown?", the generated responses should be consistent in certain granularity for the same user. To this end, we randomly chose ten users from our user population set and designed 20 questions. Half of these questions are related to basic personal information, e.g., residence and gender, and the other half is related to personal interests and attitudes such as favorite activities. Detailed experiments are shown in the Appendix D.
We generated 200 responses from each model, and asked annotators on Amazon Mechanical Turk to judge such consistency based on two criteria: (1) model consistency: whether or not a given generated sentence is consistent under the same group of questions; (2) personalization consistency: whether or not a given generated sentence is consistent with a user's personal information. Raters were asked to rate between 1 to 3 for model consistency, where 1 means "Not consistent at all", 2 means "Slightly
Model	User Information	Consistency	Personalization Consistency
Attention-Seq2Seq	-	1.40**	0.29**
Speaker Model	username	1.88**	0.31*
Generative Memory Network	history	2.08*	0.35*
Generative Split Memory Network	profile+history	2.29	0.43
Table 7: Manual evaluation results. Here, Personalization Consistency can take value from -1 to +1. * indicates significant difference with the best result (t-test, p-value< 0.05); and **: p< 0.01.
consistent", 3 means "Very consistent". For personalization consistency, raters rate whether the generated sentences match either the provided user profile attributes or user comments. Note that when annotating personalization consistency, the turkers were not able to see the user ids; all the user information shown to them is publicly available on Reddit to protect user information.
Table 6 summarizes different evaluation metrics on test set. We found that the Speaker Model boosted the Attention-Seq2Seq baseline with a decrease of 18.3 in perplexity,  $3.14\%$  increase in PC-Score, similar to Li et al. (2016b). However, because the user set is quite large, the performance improvement of Speaker model is limited. The generative memory network that incorporates either user profile or comment history demonstrates improvement compared with Attention-Seq2Seq in terms of all the metrics. Either generative memory network outperformed the speaker model, suggesting that network with additional user information has a better ability to generate semantic consistent and personalized responses. Furthermore, our proposed network (GSMN) significantly outperforms all other baselines, with a decrease of 38.7 in perplexity and a  $28.4\%$  increase in BLEU over Attention-Seq2Seq. By applying the split memories, Dialogpt w/ Split Memories further outperform Dialogpt in terms of all metrics. It shows that current pre-trained dialog models lack the ability of personalized memorization though they were found to be effective in memorization and generalization on a wide range of classical dialog tasks(Zhang et al. (2020)). This further justifies the necessity of our datasets.
In terms of human evaluations for both consistency and personalization consistency (Table 7), our network also demonstrates consistent improvement over the baselines. Note that we do not apply
any copy mechanisms in our models, the generation of personalized entities is purely depending on representation learning. It shows Split memory network outperforms the baselines on generating sentences with better personalization and consistency. To further examine the effectiveness of our generative split architecture, we also compared with Dialogpt that utilizes both profile and comment history to generating responses. In this setting, we included both profile and history as individual memory and did attention mechanism over this memory, shown as Dialogpt + profile + history. We observed that our generative split memory network still achieved better performance.
Overall, this shows that incorporating both dimensions of personalization information, i.e., user profile and user comment history, can boost models' performances for response generation and split architecture for generation is better at utilizing these two different personalization signals.
Diverse Responses Conditioned on Users: Table 12 in Appendix C shows some example responses generated by our GSMN, together with the Seq2Seq baseline. The examples are randomly sampled from our test set. Given different user profiles, GSMN is more effective and faithful to the profile attributes of different users in generating user-specific responses. For example, our model can identify user's profiles like families and gender when being asked about the most reliable person while the baseline's answer is more like consensus and applicable for any user. More examples are in Table 14 in Appendix C.
Consistency Analysis: Example outputs from baselines and our model are described in Table 13 in Appendix C. The Seq2Seq model was a bit inconsistent in answering the same group of questions. The speaker model showed consistency to some degree since the answers "California" and "Florida" are quite close in the word embedding space, but failed due to the lack of user information.
<sup>8</sup>Comments that have highest similarity scores with the queries are used as references.
Compared with these baselines, GSMN is much better at generating both consistent and personalized responses. For example, when asked favorite activities, GSMN responds consistently and is also sensitive to personalized information since "my dog" identifies the pet attribute of the respondent.
In this paper, we introduced a large-scale open-domain personalized dataset PER-CHAT and proposed a generative split memory network to utilize both user profile information and commenting histories for the task of response generation. Experimental results showed that our proposed model significantly outperformed several state-of-art baseline models, both quantitatively and qualitatively. Future research could build upon our work on single-turn response generation to further model personalization in multi-turn conversations.
For the annotation, each worker on Amazon Mechanical Turk was paid  $0.1\mathrm{~s~}$  per selection task (matching the United States federal minimum wage). To ensure quality, we chose only master crowd-workers who had more than 5000 HITs approved and with an approval rate larger than  $95\%$ .
Considering the privacy violation problems our dataset may bring about, we followed Reddit's term of use for user content—based on Reddit API Terms of Use, users are granted with license to display the user content through application<sup>9</sup>.
We have taken careful procedures to protect users' privacy concerns. First, our introduced PER-CHAT dataset will be shared for academic use only. We only released raw data from pushshift.io(Baumgartner et al. (2020)), and open-sourced our scripts for preprocessing user attributes and models for reproducibility. Note that, the user attributes used in our work are identified based on users' self-reported statements via regular expressions matching. This research study has been approved by the Institutional Review Board (IRB) at the researchers' institution.
Generation models trained on Reddit sometimes tend to generate toxic or inappropriate responses as pointed out by Dialogpt (Zhang et al., 2020). Due to this reason, we followed their best practices to deal with released version of decoding scripts<sup>10</sup>.
Rami Al-Rfou, Marc Pickett, Javier Snader, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. arXiv preprint arXiv:1606.00372.
Rafael E Banchs and Haizhou Li. 2012. Iris: a chatt-oriented dialogue system based on the vector space model. In Proceedings of the ACL 2012 System Demonstrations, pages 37-42. Association for Computational Linguistics.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 830-839.
Domenic V Cicchetti. 1994. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment, 6(4):284.
Munmun De Choudhury and Sushovan De. 2014. Mental health discourse on reddit: Self-disclosure, social support, and anonymity. In *Eighth International AAAI Conference on Weblogs and Social Media*.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49.
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and William B Dolan. 2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 445-450.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.
Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In *NAACL*.
Chaitanya K Joshi, Fei Mi, and Boi Faltings. 2017. Personalization in goal-oriented dialog. arXiv preprint arXiv:1706.07503.
92.d. in Reddit API Terms of Use
$^{10}$ https://github.com/microsoft/DialoGPT
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020. Will i sound like me? improving persona consistency in dialogues through pragmatic self-consciousness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 904-916.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proc. ACL.
Satwik Kottur, Xiaoyu Wang, and Vítor Carvalho. 2017. Exploring personalized neural conversational models. In *IJCAI*, pages 3728-3734.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994-1003.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016c. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019. Personalizing dialogue agents via meta-learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5454-5459.
Bodhisattwa Prasad Majumder, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Julian McAuley. 2020. Like hiking? you probably enjoy nature: Personagrounded dialog with commonsense expansions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9194-9206.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Prasanna Parthasarathi and Joelle Pineau. 2018. Extending neural generative conversational model using external knowledge sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 690-695.
Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 583-593, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Nicolas Schrading, Cecilia Ovesdotter Alm, Ray Ptucha, and Christopher Homan. 2015. An analysis of domestic abuse discourse on reddit. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2577-2583.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586.
I Sutskever, O Vinyals, and QV Le. 2014. Sequence to sequence learning with neural networks. Advances in NIPS.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013. A dataset for research on short-text conversations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 935-945.
Bolin Wei, Shuai Lu, Lili Mou, Hao Zhou, Pascal Poupart, Ge Li, and Zhi Jin. 2019. Why do neural dialog systems generate short and meaningless replies? a comparison between dialog and translation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7290-7294. IEEE.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741.
TH Wen, D Vandyke, N Mrksic, M Gaic, LM Rojas-Barahona, PH Su, S Ultes, and S Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017-Proceedings of Conference, volume 1, pages 438-449.
Yubo Xie, Ekaterina Svikhnushina, and Pearl Pu. 2019. A multi-turn emotionally engaging dialog model. arXiv preprint arXiv:1908.07816.
Jordi Xifra and Francesc Grau. 2010. Nanoblogging pr: The discourse on public relations in twitter. *Public Relations Review*, 36(2):171-174.
Chen Xing, Yu Wu, Wei Wu, Yalou Huang, and Ming Zhou. 2018. Hierarchical recurrent attention network for response generation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563-578.
Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. 2020. Personalized dialogue generation with diversified traits.
Peixiang Zhong, Chen Zhang, Hao Wang, Yong Liu, and Chunyan Miao. 2020. Towards persona-based empathetic conversational models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6556-6566.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In *IJCAI*, pages 4623-4629.
A Conversation Pairs and Response Distribution
We examined the distribution for query-response pairs and the statistic result is shown in the Figure 3. Questions that exceed 100 words and responses that are longer than 40 words are excluded. This led to  $88\%$  of the original pairs.
Figure 3: Message and response length distribution.
Figure 4 illustrates the distribution of the number of responses under same question in our dataset. We found that questions with 2 to 3 responses are the majority.
Figure 4: The average number of replies per question.
B Reliability of API-based User Attribute Information