SlowGuess/NAACL_2021_OCR huggingface dataset:
Yuwei Wu\*, Xuezhe Ma\*, Diyi Yang
\* Shanghai Jiao Tong University, will 8821@sjtu.edu.cn
$\spadesuit$  University of Southern California, xuezhema@usc.edu
$\diamond$  Georgia Institute of Technology, dyang888@gatech.edu
Despite the impressive successes of generation and dialogue systems, how to endow a text generation system with particular personality traits to deliver more personalized responses remains under-investigated. In this work, we look at how to generate personalized responses for questions on Reddit by utilizing personalized user profiles and posting histories. Specifically, we release an open-domain single-turn dialog dataset made up of 1.5M conversation pairs together with 300k profiles of users and related comments. We then propose a memory network to generate personalized responses in dialogue that utilizes a novel mechanism of splitting memories: one for user profile meta attributes and the other for user-generated information like comment histories. Experimental results show the quantitative and qualitative improvements of our simple split memory network model over the state-of-the-art response generation baselines. The dataset and code are available here.
Building human-like conversational systems, in particular chit-chat agents, has been a longstanding goal in language technology communities. Unlike task-oriented dialog agents that focus on completing specific tasks (Wen et al., 2017; Eric et al., 2017; Lei et al., 2018; Lowe et al., 2015), chit-chat agents need to dynamically interact with people, understand the meaning of human conversations (Hovy and Yang, 2021), and thereby make better responses to improve user experience.
Despite the recent successes on building chitchat agents using data-driven approaches (Ritter et al., 2011; Banchs and Li, 2012; Serban et al., 2016; Li et al., 2016c; Parthasarathi and Pineau, 2018), lack of a consistent personality is still one of the common issues. The main reason is that these
The work is mainly done when YW was a visiting student at Georgia Institute of Technology.
Question: Where do you live and what is something you are doing today?
A: I live in Mongolia and I will be making some good sandwiches today.
B: Midwest America, I will be skyping my brothers and going to band practice today.
Question: What's your "go to" when you're sad?
A: I listen to horror stories for some reason.
B: I love to read or listen to sad music.
Respondent Profile:
A: Gender: female; Favorites: sandwich;
Possessions: Russian class; Residence: Mongolia; Asia;
B: Family: brothers; Self-description: guitarist;
Favorite: fakebooks; Residence: America;
Respondent Comment Histories:
A: I often fall asleep while listening to horror stories.
B: Listening to sad music, I know it adds fuel to fire but
the flame will burn out quicker and you'll feel better soon.
Table 1: Example conversation pairs with respondents' profile and posting histories, with related information from profile and histories in blue and red respectively.
models are often trained over conversations spoken by different people, ignoring their personality (Li et al., 2016b; Wei et al., 2019; Zhang et al., 2018). As shown in Table 1, different people responded differently to the same input question due to their diverse background including basic personal information and attitudes towards different things. Thus, it becomes essential to incorporate personalization into the modeling and evaluation of response generation and eventually chit-chat agents.
There have been several personality-related dialogue datasets built for evaluating models' performances in personalized conversations, such as PERSONA-CHAT dataset (Zhang et al., 2018) and Facebook's Reddit dataset (Mazare et al., 2018). The PERSONA-CHAT dataset was collected by intentionally assigning annotators to predefined personas described by a set of sentences instead of their real personality. Such artificially generated conversations cannot adequately represent respondents and their personalities which would lead to
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1956-1970 June 6-11, 2021. ©2021 Association for Computational Linguistics
dataset bias problems. For example, an introvert annotator can hardly imitate the response of a person with sociable personas. Moreover, the number of personas covered by this corpus is limited.
Today's social media platforms such as Reddit and Twitter provide us with good opportunities to build a large scale of collections of naturally occurring conversations (Xifra and Grau, 2010; De Choudhury and De, 2014; Schrading et al., 2015) and also make it possible to provide consistent personalities. For instance, Facebook's Reddit dataset represents each user by a set of sentences chosen from their comment histories heuristically. However, they also acknowledged that these persona sentences might not well represent a general trait of users due to the limitation of their heuristic rules for sentence retrieval (Mazare et al., 2018).
In this work, we introduce a personalized Reddit dataset PER-CHAT, an open-domain response generation dataset consisting of 1.5M conversations and 300k users. PER-CHAT covers finer-grained personal information for users, including discrete user attributes such as gender, residence, self-description and favorites inferred based on users' self-reported messages on Reddit, and contextual information such as their comments (§3). Based on PER-CHAT, we propose a simple generative split memory network to incorporate diverse personal information, with a novel mechanism of splitting memories: one memory representation for user meta attributes (e.g., profile) and the other for user activity information (e.g., comment histories), respectively (§4). Experimental results show that our generative split memory network outperforms state-of-the-art response generation baselines both quantitatively and qualitatively (§5).
Personalized Generation Datasets Much attention has been paid to construct personalized dialog datasets. Built upon the bAbI dialog dataset, Joshi et al. (2017) extended it to include information such as gender, age and dietary preference. This domain-specific dataset was then used to train goal-oriented dialog models for several restaurant reservation tasks. There are also several dialog datasets that focus on chit-chat scenarios, such as PERSONACHAT dataset (Zhang et al., 2018), Reddit dataset (Al-Rfou et al., 2016), Twitter dataset (Li et al., 2016b) and PersonalDialog dataset (Zheng et al., 2020). PERSONA-CHAT (PC) dataset consists of
1k different personas, and annotators are asked to conduct conversations according to assigned personas. The Reddit dataset and Twitter dataset simply use user ID information without any specific user information to indicate personalization. The PersonalDialog dataset (PD) (Zheng et al., 2020), collected from a Chinese social media Weibo, contains three kinds of personality traits ("gender", "location", "age") for each user. On the other hand, Mazare et al. (2018) introduced personalization from Reddit (PCR) by incorporating the persona of each user with a (randomly chosen) subset of his/her posting comments. Zhong et al. (2020) further extended their datasets with annotated empathy information (PEC). In this work, we combine those two different ways of gathering personalization signals of users, i.e., meta profile attributes and users' posting histories, and provide a more comprehensive, large scale personalized dataset derived from natural social conversations.
Personalized Generation Models Current dialog models can be divided into ranking-based models and generation-based models. Ranking-based models (Al-Rfou et al., 2016; Mazare et al., 2018; Zhang et al., 2018) focus more on the task of response selection that is to pick the best response from a pool of random candidates. In contrast, generation-based models attempt to generate response directly from any given input questions. Under personalized dialog settings, Zhang et al. (2018) claimed that ranking-based models performed better than generative models on their personalized dataset, suggesting that building personalized generation models are more challenging.
With the development of recent large scale social media data and the success of sequence to sequence framework (Serban et al., 2016; Shang et al., 2015; Sutskever et al., 2014), several personalized response generation models have been proposed, and we can only mention a few here due to space limits. Li et al. (2016b) introduced the Speaker Model and the Speaker-Addresssee Model that encoded user-id information into an additional vector and fed it into the decoder to capture the identity of the speakers. Kottur et al. (2017) further extended these speaker models into multi-turn conversations. In addition to using user id to capture personal information, Zhang et al. (2018) proposed a profile memory network that utilizes a memory network for encoding persona sentences. To further utilize personal traits, Zheng et al. (2020) proposed
an attention mechanism to incorporate these user-related attributes in the decoding stage. Recently, there are a few works using meta-learning and reinforcement learning to enhance mutual persona perception Madotto et al. (2019); Kim et al. (2020); Majumder et al. (2020). However, few models have taken into account different potential sources of personalization signals such as profile attributes and comments. Our work conducts persona-aware representation learning by combining these two sources. Note that our split memories architecture is similar to Joshi et al. (2017), but differs in tasks and memorizing histories. In our work, we focused on memorizing relevant history comments instead of dialog histories in multi-turn chat settings.
Evaluation Metrics Most response generation models utilize perplexity, BLEU (Papineni et al., 2002) and recently BERTScore (Zhang et al., 2019) and Moverscore (Zhao et al., 2019) for evaluation (Serban et al., 2016; Xing et al., 2018). For evaluating personalization, Zheng et al. (2020) proposed to measure the accuracy of predicting personality traits by firstly training classifiers for different personality traits such as gender and age. However, for certain trait categories such as hobbies and location, it is quite difficult to train a reliable classifier. In terms of evaluating persona consistency between generated sentences and given user comments, Madotto et al. proposed consistency score using NLI models pre-trained on Dialog NLI dataset (Welleck et al., 2019), which is a corpus based on Persona dataset, with NLI annotation between persona description sentences and dialogues utterance. In this paper, we introduce an automatic metric for evaluating persona consistency between user profiles and these generated sentences.
This section describes how we construct an open-domain single-turn dialog dataset with personalization information from Reddit, together with dataset analysis<sup>1</sup>. Specifically, we used r/AskReddit<sup>2</sup>, one of the most active subreddits based on an online subreddit ranking system sorted by number of active users<sup>3</sup>. Users on r/AskReddit are encouraged to write clear and direct questions, and most posted questions are about open-ended discussion on a va
riety of topics, without definite or correct answers or professional knowledge, making r/AskReddit a suitable place to model personalization in open domain dialogue systems.
Data Preprocessing. We collected all submissions under r/AskReddit as questions and their subsequent comments as responses. Each submission and one of its direct comment form a (question, response) pair in our corpus, i.e., single-turn dialogues. Furthermore, we stripped away potential markdown andHtml syntax tokens and replaced all forms of url links, emails, and digits in our corpus with unique tokens "url", "email" and "digit" respectively. We also processed replicated words and punctuation to their standard form via a set of regular expressions, e.g., "coooool" is converted into "cool" and "!!!!" to "!".
Vocabulary and Conversation Pairs. We use a vocabulary of 50,257 entries the same as Dialogpt (Zhang et al., 2020), since they pretrained their models using the full Reddit data. To avoid lengthy questions or responses, we pruned the conversation pairs based on the statistics (see Figure 3 in Appendix A). Questions that exceed 100 words and responses with over 40 words are excluded. In total, there are 1,566,653 conversation pairs.
3.1 Personalization Information
To augment our dataset with personalization information, we collected three sources of user-related information: (1) user IDs which are unique usernames for their Reddit accounts; (2) comment histories, which are all the comments a user has posted on Reddit; (3) user profile attributes such as gender, residence, favorites and etc.. To collect these user-specific information, we first filtered out inactive users — a user who has made less than 100 comments during the recent year. There remain 301,243 users after removing inactive users.
User Comment Histories. Users' comment histories can often signal their personal preferences toward topics or even texting habits as shown in Table 1, thus it is beneficial to collect these histories. We obtained a user's comment histories by querying the Pushshift Reddit  $\mathrm{API}^4$ . Since (1) it is infeasible for models to operate on the scale of thousands of comments and (2) applying persona extraction process rather than randomly picking up comments can improve model's performance
1Similar process can be employed to our raw data to obtain MULTI-TURN dialog datasets. 2https://www.reddit.com/r/AskReddit/ 3https://www.topsubreddits.com/
in personalization suggested by Mazare et al., we designed an information retrieval (IR) system to automatically pick up query-related comment histories for each user. Specifically, We utilized semantic embedding based similarity between each query and a comment to obtain a smaller set of candidates  $\mathbb{M}$ , following similar retrieval mechanisms as Ritter et al. (2011); Wang et al. (2013). That is, given the input question, we retrieve top  $l$  comments that have the highest cosine similarity scores with the query to construct the user's comment histories. The embedding used in IR systems is the averaged contextual embeddings from pretrained BERT-large models(Devlin et al., 2019). Respondent Comment Histories of Table 1 shows some example query-related histories we extracted from a user's comments.
User Profile. The persona extraction process to construct comment histories might lose valuable user's attributes such as their residence, favorites which are also helpful in generating personalized responses. To this end, we further conduct a finer-grained entity extraction mechanism over all of user's past histories. User profile information was viewed as entities extracted from histories using similar methods as the popular Reddit user analysis site SnoopSnoop<sup>5</sup>. Following the categories provided by the site, we first divided user attributes into eight types, including "pets", "family", "residence", "favorites", "partner", "possessions", "gender", "self-description", where 'possessions' refers to personal possessions owned by users such as users' guitars; "favorites" means users' favorite items and people mentioned by the user and "self-description" denotes concepts that users use to describe themselves such as their occupations. We then applied different extraction regular expressions for different categories. For example, we would gather a noun as "favorites" if it is found after "like, love..." in certain comments.
Examples for these attributes are shown in Respondent Profile of Table 1. Unlike some social media platforms such as Weibo, users on Reddit do not provide very specific profile information. Thus, we need to extract these entities based on their histories, and also check the reliability of such profile information. We manually checked whether such extracted user attributes actually corresponded to users' comments via a small corpus study (details
Figure 1: Distribution of users' number of attributes.
Attr	Pets	Family	Residence	Favorsites
Percent(%)	29.1	70.9	39.1	62.7
Attr	Partner	Possession	Gender	Self-description
Percent(%)	39.1	99.7	99.5	82.8
Table 2: Coverage rate for each attribute.
Relevant	Irrelevant	Unsure
Numbers	434	63	3
Percentage	86.8%	12.6%	0.6%
Table 3: Query-response relevance annotation.
Train	Dev	Test
# Queries	439996	5523	5559
# Response	1528218	19224	19211
Table 4: Statistics of train, dev, and test set.
in Appendix B), and found that in over  $85\%$  cases, our entity extracting process is quite reliable for capturing users' basic information.
User Profile Analysis. We conducted in-depth analyses to show the coverage rate of each attribute out of the eight profile attributes in our collected corpus, as described in Table 2. We found that gender and possession have very high coverage rates above  $99\%$ , and other attributes have different coverage rates, ranging from  $29.1\%$  to  $82.8\%$ . Since it is unnecessary that users contain value under every attribute type, we also computed the percentage of users who have the corresponding number of attribute types. Figure 1 showed that most users have around 4 to 7 attributes.
Question-Response Relevance To examine the quality of our constructed corpus, especially the question-response relevance, we randomly sampled 500 question-response pairs from our corpus, and asked for annotators from Amazon Mechanical Turk to rate them. Each pair is judged by three raters on whether a response appropriately responded to the given question. Raters can select from 'Yes', 'No', and 'Unsure' if they are unsure about the relevance. We obtained an intra-class correlation coefficient of 0.63, indicating good
5https://github.com/orionmelt/snoopsnoo
Figure 2: The overall diagram of Generative Split Memory Network.
Dataset	Source	Comments	Profile	Size	Public
PC	Crowd-Sourced	Yes	No	151K	Yes
PCR	Reddit	Yes	No	700M	No
PEC	Reddit	Yes	No	355K	Yes
PD	Weibo	No	Yes	20.83M	No
PER-CHAT	Reddit	Yes	Yes	1.5M	Yes
Table 5: Comparisons between PER-CHAT and related datasets. PC denotes PERSONA-CHAT (Zhang et al., 2018). PCR denotes the persona-based dialog datasets from Reddit (Mazare et al., 2018). PEC denotes persona-based empathetic conversation (Zhong et al., 2020). PD denotes the PersonalDialog dataset (Zheng et al., 2020). The size denotes the number of expanded conversations.
annotation agreement(Cicchetti, 1994). We categorized a pair as relevant if the majority of annotators vote 'Yes'. As summarized in Table 3,  $86.8\%$  of the pairs are found to be relevant, suggesting a reasonable quality of our corpus.
Most questions in our corpus has around with 2 to 3 responses. We randomly sampled  $1\%$  of questions and their corresponding responses as the development set,  $1\%$  as the testing set, and the remaining  $98\%$  as the training set. Table 4 summarizes the detailed statistics.
Comparisons with Related Datasets Table 5 shows the comparisons between our datasets and the related ones. The biggest advantage of our dataset is that it has both comment histories and user profiles while being 5-10 times bigger than any prior publicly available dataset. In our following experiments in section 5, we show the necessity to provide both dimensions of personalized information. In terms of comments, we applied pre-trained
IR system to extract query related comments instead of simply rule-based filtering used in datasets such as PCR and PEC. In terms of user profiles, we provide more diverse categories with eight main types, larger than PD datasets which only contains age, gender and location. By utilizing social media data, our dataset allows for more diverse personality and natural dialog patterns with over 300k users than datasets collected by human (e.g. PC).
4 Generative Split Memory Network
This section presents our generative models for personalized response generation, which generate responses conditioned on given questions and respondents' personal information<sup>6</sup>.
Let a conversation  $C$  be a tuple of Question, Response and respondent (User)  $C \coloneqq (\overline{Q}, R, U)$ . A user  $U = (ID, P, M)$  consists of three sources of information — username (ID), profile attributes  $P = (f_1, f_2, \ldots, f_n)$ , and user's comments  $M = (m^1, m^2, \ldots)$  where  $f_i$  is given as a key-value pair  $f_i = < k_i, v_i >$  and  $M$  is a set of comment histories the user made.
To better incorporate personal information of different dimensions, we propose a generative memory network with split memories for user profile and user comment history, respectively. The intuition lies in that interpersonal meta attributes and comment patterns may influence the respondents' responses differentially. The overall model architecture is shown in Figure 2. Our model is built
$^{6}$ Multi-turn dialogue generation can be considered in our framework by encoding additional contexts in memories.
upon a standard seq2seq model with attention. For a given conversation  $C$ , we first feed the input comments  $M$  through retrieval system to get query-related comment set  $\mathbb{M}$ , and the memory network encoder computes the representations of related comment history. In parallel, profile attributes are added as separate profile memory by another encoder. At each time step, the decoder utilizes the aggregated representations of comment histories and profile attributes to generate the final response.
4.1 User Profile and Comment Histories
For a user  $U$  with the profile  $P$ , we view  $P$  as the user's attribute sequence and employ a shared word embedding in encoder to encode the attribute key sequence as  $e_k = (e_{k_1}, e_{k_2}, \dots, e_{k_n})$  and to encode each entry in attribute value sequence as  $e_v = (e_{v_1}, e_{v_2}, \dots, e_{v_n})$  respectively. The final set of the user profile representation  $\pmb{H_p}$  is defined as  $\{e_{k_1} \odot e_{v_1}, \dots, e_{k_n} \odot e_{v_n}\}$ , which is considered as the profile memory.
For encoding comment sentences, we encoded the retrieved comments  $\mathbb{M}$  for a user  $U$  as individual memory representations in a memory network, similar to Zhang et al. (2018). Instead of applying weight functions to word vectors of each entry, we feed the comments to the RNN encoder to get the set of encoded history memories denoted as  $H_{m}$ .
4.2 Split Memories
We then pass  $H_{p}$  and  $H_{m}$  to a split memory encoder. The memory network separately attends to the encoded split memories with given query vector  $q$  over  $K$  hops as follows:
$$
a _ {p} ^ {k} = \operatorname {S o f t m a x} \left(\boldsymbol {H} _ {\boldsymbol {p}} \cdot W _ {1} \cdot w _ {p} ^ {k}\right) \tag {1}
$$
$$
w _ {p} ^ {k + 1} = \left(a _ {p} ^ {k}\right) ^ {T} \cdot \boldsymbol {H} _ {\boldsymbol {p}} + w _ {p} ^ {k} \tag {2}
$$
$$
a _ {m} ^ {k} = \operatorname {S o f t m a x} \left(\boldsymbol {H} _ {\boldsymbol {m}} \cdot W _ {2} \cdot w _ {m} ^ {k}\right) \tag {3}
$$
$$
w _ {m} ^ {k + 1} = \left(a _ {m} ^ {k}\right) ^ {T} \cdot \boldsymbol {H} _ {\boldsymbol {m}} + w _ {m} ^ {k} \tag {4}
$$
where  $W_{1}, W_{2} \in \mathbb{R}^{d \times d}$  and  $w_{p}^{1} = w_{m}^{1} = q$ . The outputs from both memories  $w_{p}^{K}$  and  $w_{m}^{K}$  are summed to get the representation  $\pmb{O}^{\pmb{K}}$  and is then fed into decoder side.
The memory decoder utilizes the memory network and RNN. The RNN decoder takes as input the previous hidden state and previous target word embedding and generates the hidden state  $h_t$  at the time step  $t$ . The vocabulary distribution  $P_{\text{vocab}}$  for
time step  $t$  is generated as follows:
$$
P _ {v o c a b} = \operatorname {S o f t m a x} \left(W _ {3} \left[ h _ {t}; O ^ {K} \right]\right) \tag {5}
$$
where  $W_{3}\in \mathbb{R}^{|V|\times 2d}$  is a trainable parameter.