SlowGuess/NAACL_2022_OCR huggingface dataset:
Hanxun Zhong $^{1}$ , Zhicheng Dou $^{2,*}$ , Yutao Zhu $^{3}$ , Hongjin Qian $^{2}$ , Ji-Rong Wen $^{2}$
$^{1}$ School of Information, Renmin University of China, Beijing, China  $^{2}$ Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
<sup>3</sup>University of Montreal, Quebec, Canada
{hanxun_zhong,dou,ian}@ruc.edu.cn
{yutaozhu94, jirong.wen}@gmail.com
Personalized dialogue systems explore the problem of generating responses that are consistent with the user's personality, which has raised much attention in recent years. Existing personalized dialogue systems have tried to extract user profiles from dialogue history to guide personalized response generation. Since the dialogue history is usually long and noisy, most existing methods truncate the dialogue history to model the user's personality. Such methods can generate some personalized responses, but a large part of dialogue history is wasted, leading to sub-optimal performance of personalized response generation. In this work, we propose to refine the user dialogue history on a large scale, based on which we can handle more dialogue history and obtain more abundant and accurate persona information. Specifically, we design an MSP model which consists of three personal information refiners and a personalized response generator. With these multi-level refiners, we can sparsely extract the most valuable information (tokens) from the dialogue history and leverage other similar users' data to enhance personalization. Experimental results on two real-world datasets demonstrate the superiority of our model in generating more informative and personalized responses.
Recent years have witnessed great progress in building personalized dialogue systems. In general, previous work explores building a personalized dialogue system via two pathways: (1) directly modeling user personality from predefined persona descriptions or user attribute (Qian et al., 2018; Zhang et al., 2018; Song et al., 2019); and (2) implicitly modeling the user personality from the user's dialogue history (Li et al., 2016c; Ma et al., 2021). The latter is considered superior as the dialogue history is easy to obtain and comprises rich personalized information. In this paper, we follow the
second pathway that automatically learns implicit user profiles from the user's dialogue history to assist in personalized response generation.
It is challenging to model user personality directly from the dialogue history. The main reason is that a user's dialogue history may contain massive historical dialogues, which are too heavy to load in the model and likely to be noisy. A straightforward solution is to truncate the dialogue history, as done by existing work (Ma et al., 2021; Qian et al., 2021a). However, as tremendous information has been wasted, the model's performance is also influenced. On the other hand, we observe that the dialogue history from other users may also be helpful in generating a personalized response for the current user. For example, users with the same interest in "soccer" may talk about similar things on such a topic. This has been overlooked by existing methods. Intuitively, the problem of "data explosion" is even more severe in the latter case (when other similar users' dialogue history is also considered). To alleviate these problems, we propose using a hierarchical refiner structure to sparsely extract the most valuable query-aware persona information from both the current and other similar users' dialogue history. By this means, more dialogue history can be utilized for learning user personality and improving response generation.
Our model is called MSP, which stands for Modeling and Selecting user Personality from the dialogue history for generating personalized responses. Instead of attending to all dialogue history, MSP refines the most valuable historical information that can well portray the user's personality and guide the response generation. Specifically, MSP consists of three personal information refin- ers working at different levels and a personalized response generator. At first, a user refiner is de- signed to select a group of users who have similar interests to the current user. By refining dialogue history at the user level, we can obtain similar data
Human Language Technologies, pages 5808 - 5820
to share information with similar users and avoid mutual interference with other users. Then, a topic refiner filters out the current and similar users' dialogue history that has different topics with the current query at the sentence level. Next, we design a token refiner to extract the most valuable query-aware user profiles from the remaining dialogue history at the token level. Finally, a personalized response generator combines user profiles and the current query to generate responses. Given that there is no explicit supervisory signal guiding the refiner to extract an exemplary user profile, we design a supplementary sentence matching task and a joint training method. The generator will construct a pseudo-label to guide the refiner's extraction.
(1) We design an MSP model to tackle the data noise problem. It can efficiently refine user profiles through dialogue history and generate personalized responses. By this means, our method can capture abundant user profiles while keeping away from noisy data.
(2) We design a refiner structure to extract the query-aware profile at three levels. Similar users' information is taken into account, which can help improve the personality of the response.
(3) We design a joint training method for the refiner and generator. The refiner provides the generator with user profiles to assist in generating responses, while the generator constructs a pseudolabel for the refiner to assist in selecting user profiles.
Personalized Dialogue Generation Open-domain dialogue generation has been extensively studied (Koehn et al., 2003; Vinyals and Le, 2015; Serban et al., 2016; Zhang et al., 2019a,b; Liu et al., 2020; Xiao et al., 2020; Zhu et al., 2020). Recently, personalized dialogue systems have attached more and more attention. Typical methods include: (1) explicitly using predefined persona descriptions or attributes as users' profile to generate personalized responses (Qian et al., 2018; Zhang et al., 2018; Olabiyi et al., 2019; Song et al., 2019); (2) using user ID embeddings to enhance personalized dialogue generation (Li et al., 2016c; Chan et al., 2019); and (3) extracting implicit user profile from users' dialogue history to generate personalized responses (Al-Rfou et al., 2016; Bak and Oh, 2019; Ma et al., 2021). Since manually collecting user
profiles is impractical for large-scale datasets and the user ID embeddings perform badly, in this study, we focus on the last group of methods for personalized response generation.
DHAP (Ma et al., 2021) is the state-of-the-art method in personalized dialogue generation. It uses a transformer-based structure to model the user's dialogue history and extract personal information for response generation. Unfortunately, this model can only handle a limited number of user dialogue histories, wasting a lot of valuable information. Our method has two main differences from DHAP: (1) We propose a refiner structure in our model so that more dialogue history can be handled and the most valuable information can be extracted for improving response generation; (2) With our proposed refiner, we can further incorporate more dialogue history from other users (having similar interests) to facilitate personalized dialogue generation.
Retrieval-guided Natural Language Generation  
Retrieval-based methods can collect relevant information for language generation (Yang et al., 2019). It has been widely applied in many tasks such as text style transfer (Li et al., 2018) and dialogue generation (Wu et al., 2019; Cai et al., 2019). The idea of using a retrieval system to get useful information inspires our study. We use a refiner to automatically extract personal information from dialogue history and guide the personalized generation.
In this section, we first formalize our problem and provide an overview of our proposed MSP model. Then, we describe the details of each component and the model optimization.
Considering a set of users  $\mathcal{U} = \{u_1,\dots ,u_l\}$  , for any user  $u_{i}$  , we have their dialogue history with others  $U^{i} = \{(q_{1}^{i},r_{1}^{i}),\dots ,(q_{t}^{i},r_{t}^{i})\}$  , where  $q_{j}^{i}$  is a query issued by others, while  $r_j^i$  is the corresponding response given by  $u_{i}$  . Our target is to generate a personalized response  $r^i$  for the user  $u_{i}$  to reply a new query  $q$  . As we introduced earlier, the personalized information can be obtained from the dialogue history  $U^{i}$  of the user  $u_{i}$  and other dialogue history  $U^{j}$  from similar users  $u_{j}(j\neq i)$
<sup>1</sup>Here we use the term "query" to denote the utterance given by others. Generally, the query can be either one utterance in single-turn dialogues, or several history utterances in multi-turn dialogues.
Figure 1: The overview structure of the proposed model which consists of four modules: (1) user refiner, (2) topic refiner, (3) token refiner, and (4) generator.
The overview of our MSP model is shown in Figure 1. MSP consists of three personal information refiners working on different levels and a personalized response generator. Specifically, the first refiner is working at the user level. By comparing the dialogue history of the current user  $u_{i}$  with others, MSP can select a group of users having similar interests with  $u_{i}$ . After obtaining a group of similar users, we further refine their dialogue history according to the relevance to the current query's topic. Moreover, we add the last refiner to extract several tokens so that the most fine-grained personal information can be extracted from the relevant utterances. Finally, the query and the extracted tokens are fed into the generator and construct a personalized response.
3.2 User Refiner
The dialogue history of users with similar interests may share much personal information. Therefore, our first target is to select a group of users with similar interests to the current user. We design a user refiner to achieve this. Since the users' interest is usually contained in their dialogues with others, we consider both the queries and responses in the dialogue history to select similar users. Specifically, for the user  $u_{i}$ 's dialogue history  $U^{i}$ , we apply a pre-trained BERT (Devlin et al., 2019) and
represent them by the embedding of [CLS] token:
$$
\mathbf {U} _ {q} ^ {i} = \sum_ {j = 1} ^ {t} \mathrm {B E R T} (q _ {j} ^ {i}), \quad \mathbf {U} _ {r} ^ {i} = \sum_ {j = 1} ^ {t} \mathrm {B E R T} (r _ {j} ^ {i}).
$$
Then, we can select  $k_{u}$  users that have similar interest with the current user  $u_{i}$  as:
$$
u ^ {\text {s i m}} = \operatorname {T o p K} \left(\mathbf {U} ^ {i} \cdot \mathbf {U} ^ {j}, k _ {u}\right), \tag {1}
$$
$$
\mathbf {U} ^ {i} = \left[ \mathbf {U} _ {q} ^ {i}; \mathbf {U} _ {r} ^ {i} \right], \tag {2}
$$
where  $\mathrm{TopK}(\cdot ,\cdot)$  is the top-  $k$  selection operation.
After the user refiner, we can obtain the dialogue history of the similar users  $\{u_{j}\}_{j = 1}^{k_{u}}$ . It is worth noting that, since the number of users is large in the datasets, we choose to use the dot-product to compute the similarity of the users so that the whole process can be implemented by dense retrieval libraries, such as Faiss (Johnson et al., 2021), which is very efficient.
3.3 Topic Refiner
The users' dialogue history often contains many dialogues with others. These dialogues have various topics, which may be irrelevant to the current query. Therefore, we propose a topic refiner to select relevant dialogue history for personalized response generation. Specifically, we use a topic classifier to compute the topic distribution of the current query
$q$  and the queries  $q_{j}^{i}$  in the history dialogues:
$$
t = \operatorname {M L P} (\text {m e a n} (\operatorname {B E R T} (q)), \tag {3}
$$
$$
t _ {j} ^ {i} = \operatorname {M L P} \left(\operatorname {m e a n} \left(\operatorname {B E R T} \left(q _ {j} ^ {i}\right)\right), \quad \right. \tag {4}
$$
where  $t, t_j^i \in \mathbb{R}^{d^t \times 1}$ , and  $d^t$  is the number of topic. Then, we filter out the dialogue history  $< q_j^i, r_j^i>$  that has different topics with the current query, i.e.,  $\max(t_j^i) \neq \max(t)$ .
In the topic refining process, we compare the queries in the history dialogues with the current query to filter out topic-irrelevant dialogues. This process can further reduce the noise and make our model more lightweight. Both the dialogue history of the current user and that of the similar users (obtained in the former step) are refined. In the next step, we will use the responses in these selected history dialogues and extract the most valuable tokens for personalized response generation.
3.4 Token Refiner
After the previous two refiners, we obtain a collection of historical responses. Though we can directly add them into the generation process, our preliminary experiments indicate that they perform poorly. A major reason is the noisy quality of the responses. Indeed, existing studies (Xing et al., 2017; Zhu et al., 2020) have demonstrated the effectiveness of using informative tokens to improve the response generation. Inspired by these studies, we further devise a token refiner to extract the most fine-grained information (tokens) from the historical responses. Specifically, we compute an attention map  $\mathbf{A}$  between the query  $q$  and the historical responses  $r^{\mathrm{sim}}$  and  $r^{\mathrm{cur}}$  (they are from the similar users and the current user respectively) as:
$$
\mathbf {A} = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q K} ^ {T}}{\sqrt {d}}\right), \tag {5}
$$
$$
\mathbf {Q} = \operatorname {T R M} _ {\mathrm {e n c}} (q) \cdot \mathbf {W} _ {Q}, \tag {6}
$$
$$
\mathbf {K} = \operatorname {T R M} _ {\mathrm {e n c}} (r) \cdot \mathbf {W} _ {K}, \tag {7}
$$
where  $\mathrm{TRM_{enc}}(\cdot)$  is a transformer encoder.  $r$  refers to  $r^{\mathrm{sim}}$  or  $r^{\mathrm{cur}}$ , and correspondingly,  $\mathbf{A}$  refers to the similar user matching map  $\mathbf{A}^{\mathrm{sim}}$  and current user matching map  $\mathbf{A}^{\mathrm{cur}}$ .  $\mathbf{W}_Q, \mathbf{W}_K \in \mathbb{R}^{d \times d}$  are parameters, and  $d$  is the dimension of the hidden state. After obtaining the attention matching map  $\mathbf{A}$ , we select tokens to form the similar users' profile and current user's profile according to each
token's attention weight:
$$
\mathbf {c} ^ {\text {s i m}} = \operatorname {T o p K} \left(\operatorname {M a x} \left(\mathbf {A} ^ {\text {s i m}}\right), k _ {p}\right), \tag {8}
$$
$$
\mathbf {c} ^ {\text {p e r}} = \operatorname {T o p K} \left(\operatorname {M a x} \left(\mathbf {A} ^ {\text {c u r}}\right), k _ {p}\right), \tag {9}
$$
where  $k_{p}$  is a hyper-parameter to control the number of profile tokens. Its influence will be investigated in our experiments.
3.5 Generator
We use a transformer decoder as to generate a personalized response by using the similar users' profile  $\mathbf{c}^{\mathrm{sim}}$ , current user's profile  $\mathbf{c}^{\mathrm{cur}}$ , and query information  $\mathbf{q}$  as input. The decoding process can be defined as:
$$
\hat {y} = \operatorname {T R M} _ {\mathrm {d e c}} (\mathbf {x}), \tag {10}
$$
$$
\mathbf {x} = \left[ \mathbf {c} ^ {\text {s i m}}; \mathbf {c} ^ {\text {p e r}}; \mathbf {q} \right], \tag {11}
$$
where  $[\cdot ]$  is the concatenation operation and  $\hat{y}$  is the word generation probability.
3.6 Training and Optimization
The generator is optimized by maximizing the generation probability of the ground-truth  $y$ :
$$
\mathcal {L} ^ {\mathrm {g}} = - y \log \hat {y}. \tag {12}
$$
In practice, we find that the token refiner is hard to train. We speculate the reason is a missing of direct supervision signals. In this case, it is difficult to tell whether the training errors stem from the generation process or the refining process. To tackle this problem, we propose a supplementary sentence matching task to assist the token selection.
Supplementary Sentence Matching Task The core idea of this task is to train the token refiner directly by introducing supervision signals so that it can automatically mine valuable tokens. Specifically, we design a sentence matching task to match the query with the dialogue history. The task's target is to find the history sentences that help generate personalized responses. We consider using the query-history cross attention weight  $\mathbf{A}$  to generate a matching representation and then use this representation to finish the task. In this way, once the matching task has been well-finished, we can use attention map  $\mathbf{A}$  to identify the most informative and valuable parts of a history sentence that are helpful to generate a personalized response.
To achieve our idea, we design a matching process. First, we calculate the matching representations  $\mathbf{H}$  by the cross-attention map  $\mathbf{A}$  and then
apply a CNN with a max-pooling operation to aggregate token information:
$$
\mathbf {S} = \operatorname {M a x p o o l} (\operatorname {C N N} (\mathbf {H})) \tag {13}
$$
$$
\mathbf {H} = \mathbf {A} \cdot \mathbf {V}, \tag {14}
$$
$$
\mathbf {V} = \operatorname {T R M} _ {\mathrm {e n c}} (r) \cdot \mathbf {W} _ {V}. \tag {15}
$$
Next, we flatten S and applies a LSTM to aggregate the sentence information:
$$
\mathbf {h} = \operatorname {L S T M} (\text {F l a t t e n} (\mathbf {S})). \tag {16}
$$
Finally, we use the sentence matching vector  $\mathbf{h}$  to compute the matching score:
$$
\hat {g} = \operatorname {S i g m o i d} (\mathrm {M L P} (\mathbf {h})). \tag {17}
$$
For guiding the sentence matching task, we design a pseudo-label  $g$  to measure the matching goodness of each history sentence. We expect the history with more persona profile information can achieve a higher score. Therefore, we use the difference between the personalized ground-truth  $y$  and a non-personalized generated response  $\hat{y}'$  to measure the persona profile and create the pseudo-label:
$$
g = \left\{ \begin{array}{l l} 1, & g _ {\text {s o f t}} \geq \alpha \\ 0, & g _ {\text {s o f t}} <   \alpha , \end{array} \right. \tag {18}
$$
$$
g _ {\mathrm {s o f t}} = \operatorname {S u m} \left(\operatorname {M a x} \left(\left(y - \hat {y} ^ {\prime}\right) \cdot r\right)\right) / d _ {\mathrm {y}}, \tag {19}
$$
where  $d_{\mathrm{y}}$  is the length of  $y$ , and  $\alpha$  is a threshold. Finally, we minimize the binary cross entropy loss between  $g$  and  $\hat{g}$ :
$$
\mathcal {L} ^ {\mathrm {s}} = g \log \hat {g} + (1 - g) \log (1 - \hat {g}). \tag {20}
$$
Joint Training To facilitate the learning with the above gradient approximation approach, we design a joint training process to train the refiner and generator in turn. Specifically, in each training iteration, we first sample a batch of query  $q$ , response  $y$ , similar and current users' dialogue history  $r^{\mathrm{sim}}$  and  $r^{\mathrm{cur}}$  from dataset  $D$ . Then, we generate a non-personalized response  $\hat{y}^{\prime}$  and create the pseudo-label  $g$  (Eq. 18) through a non-personalized generator. This pseudo-label is used to train the token refiner by optimizing the loss  $\mathcal{L}^{\mathrm{s}}$  (Eq. 20). Further, we sample another batch  $D^{p}$  from  $D$ . After extracting similar user profile  $\mathbf{c}^{\mathrm{sim}}$  and current user profile  $\mathbf{c}^{\mathrm{cur}}$  (Eq. 8, Eq. 9), we generate the personalized response  $\hat{y}$  and update the generator by optimizing the loss  $\mathcal{L}^{\mathrm{g}}$  (Eq. 12). To avoid misleading the generation (by poor profile) at the beginning of the training process, we pre-train the refiner for  $N_{f}$  steps before extracting the profile for the generator. The detailed training process is summarized in Algorithm 1.
Algorithm 1 Joint Training Process
Input:  $M$  dialogue triplets:  $D = \{\langle q_i,y_i,r_i\rangle \}_{i = 1}^M$
Output: A personalized dialogue model
1: Init the refiner and generator module
2: while not converge do
3: Sample  $n_{s}$  dialogue triplets  $D^{q} = \{q_{i},y_{i},r_{i}\}_{i = 1}^{n_{s}}$
4: Get  $\hat{Y}^q = \{\hat{y}_i'\}_{i=1}^{n_s}$  on  $D^q$  from  $p_g(\hat{y}'|q)$
5: Get pseudo-label  $g$  on  $D^q$
6: Train refiner by optimizing  $\mathcal{L}^s$  on  $D^q$
7: if Current Step  $>N_{f}$  then
8: Sample  $n_d$  dialogue triplets  $D^p = \{q_i, y_i, r_i\}_{i=1}^{n_d}$
9: Extract  $C^p = \{c_i\}_{i=1}^{n_d}$  on  $D^p$  from  $p_s(c|q, r)$
10: Train generator by optimizing  $\mathcal{L}^g$  on  $D^p\cup C^p$
11: end if
12: end while
Table 1: Statistics of Reddit and Weibo datasets.
Reddit	Weibo
# Users	78,031	46,973
Avg. history length	72.4	30.8
Avg. # words of query	19.8	22.9
Avg. # words of response	9.1	9.6
# Training samples	5,734,129	1,495,149
# Validation samples	10,000	10,000
# Testing samples	10,000	10,000
To evaluate our model's performance, we conduct experiments on a Chinese Weibo dataset (Qian et al., 2021b) and an English Reddit dataset. Both are collected from open-domain social media platforms. On these platforms, users can post various topics, and other users can respond to them. We compare user-id and timestamps to associate the query with its corresponding response and the current user's dialogue history. As a result, each training sample contains a query, a response, and a sequence of dialogue history. Finally, the dataset is divided into training, validation, and test sets in chronological order. The statistics of the datasets are provided in Table 1.
We compare our proposed model with eight highly correlated and strong baselines. They can be categorized into four groups:
Non-personalized Methods (1) Seq2Seq-Attention (Sutskever et al., 2014) is a vanilla sequence-to-sequence model with attention mechanism (Luong et al., 2015). (2) MMI (Li et al.,
2Weibo: https://www.weibo.com/; Reddit: https://www.reddit.com/
Table 2: Criteria of human annotation.
Readability
3: Fluent and easy to read
2: Grammatically formed
1: Not a complete sentence or hard to read
Informativeness
3: Have clear and specific meaning
2: Contain few informative words
1: Meaningfulness sentence
Personalization
1: Reflect personal information contained in user history
0: Does not resemble any user history
2016a) is based on Seq2seq and use maximum mutual information as an extra loss to improve diversity. (3) DialoGPT (Zhang et al., 2019b) is a variant of GPT-2 (Radford et al., 2019) designed for dialogue generation.
Predefined Profile-based Methods Since there is no persona description in the datasets, we test these methods by using the user's dialogue history as a simulation of predefined persona profiles. (4) GPMN (Zhang et al., 2018) enhances the Seq2seq model with a memory module, which encodes and stores the persona profile as memory representations. (5) PerCVAE (Zhao et al., 2017) encodes predefined personalized sentences as a conditional representation and uses CVAE to generate a personalized response.
User ID-based Methods (6) Speaker (Li et al., 2016c) is based on seq2seq while using user-ID embedding as user representation to facilitate the response generation. (7) Persona WAE (Chan et al., 2019) uses WAE (Wasserstein autoencoder) for response generation. It maps user-ID embeddings to a personalized Gaussian mixture distribution and then samples the personalized vector to guide the response generation.
User Dialogue History-based Methods (8)  
DHAP (Ma et al., 2021) uses history memory to store and construct the dynamic query-aware user profile from dialogue history and then uses a personalized decoder to generate a response. Since this model also learns the user profile directly from the dialogue history, it is the most relevant baseline of our method.
We experiment with multiple sets of hyperparameters to select the best model, and the final hyperparameters are listed as follows: The dimensions