SlowGuess/ACL_2020_OCR huggingface dataset:
Qian Liu\*, Yihong Chen\*, Bei Chen\$, Jian-Guang Lou\$, Zixuan Chen\*, Bin Zhou\+, Dongmei Zhang\$
†School of Computer Science and Engineering, Beihang University, China  
 $\diamond$ UCL Centre for Artificial Intelligence, University College London, United Kingdom  
 $\spadesuit$ School of Computer Science, Fudan University, China  
 $\$ 6$ Microsoft Research, Beijing, China  
†{qian.liu, zhoubin}@buaa.edu.cn;  $\$ 8$ beichen, jlou, dongmeiz}@microsoft.com;  
 $\diamond$ yihong.chen@cs.ucl.ac.uk;  $\spadesuit$ remch183@outlook.com
Despite the continuing efforts to improve the engagingness and consistency of chit-chat dialogue systems, the majority of current work simply focus on mimicking human-like responses, leaving understudied the aspects of modeling understanding between interlocutors. The research in cognitive science, instead, suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose  $\mathcal{P}^2$  BOT, a transmitter-receiver based framework with the aim of explicitly modeling understanding. Specifically,  $\mathcal{P}^2$  BOT incorporates mutual persona perception to enhance the quality of personalized dialogue generation. Experiments on a large public dataset, PERSONA-CHAT, demonstrate the effectiveness of our approach, with a considerable boost over the state-of-the-art baselines across both automatic metrics and human evaluations.
Thanks to the advance in neural models and the accessibility of massive datasets, open-domain dialogue (i.e. chit-chat) systems have made great progress towards mimicking human-like responses. Nevertheless, there still exist some serious challenges in building personalized chatbots that can deliver engaging conversations and gain user trust (Song et al., 2019). For example, current chit-chat systems tend to generate uninformative responses (Li et al., 2016b). Moreover, they are usually lack of coherent personality traits due to the fact that training dialogues actually come from a diverse set of speakers (Zhang et al., 2018b).
Figure 1: A clipped dialogue from PERSONA-CHAT.
Several attempts have been made to alleviate the above issues. Methods like special reward shaping to reduce generic responses (Li et al., 2016b) and representing the speakers with latent variables (Li et al., 2016a) were introduced to improve the engagingness of chit-chat systems. A more straightforward approach, which equips chit-chat systems with predefined personas, was proposed accompanied by a novel dataset, PERSONA-CHAT (Zhang et al., 2018b). Figure 1 shows a clipped dialogue from PERSONA-CHAT. Two interlocutors meet for the first time and are having a conversation in order to get to know each other. What makes PERSONA-CHAT unique is that personas of both interlocutors are explicitly described using several profile sentences, facilitating the training of chatbots with configurable and persistent personalities.
PERSONA-CHAT has fueled a growing interest in developing methods for personalized dialogue
*Work done during an internship at Microsoft Research.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1417-1427 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: The overview of  $\mathcal{P}^2$  BOT (see text).
generation. Mazaré et al. (2018) incorporated additional data from Reddit to train the model. Wolf et al. (2019b) fine-tuned pretrained language model (Radford et al., 2018) to improve the dialogue generation. Although both works demonstrate promising results, they focus more on mimicking the style of human-like responses, leaving understudied the aspects of explicitly modeling understanding between interlocutors. Our work, instead, takes the perspective of understanding modeling.
According to the research in cognitive science, effective communication creates similar activation maps in the brains of both interlocutors (Hasson et al., 2012), suggesting that understanding between interlocutors is an essential signal for a high-quality chit-chat conversation. For instance, in the conversation shown in Figure 1, the two interlocutors foster understanding either by raising personal-related topics, "Seen any good movies lately?", or by revealing their own personas through answering questions, "I don't watch movies more of a writer". The efforts to build understanding keep the conversation flowing.
Taking into account the above, we propose Persona Perception Bot  $(\mathcal{P}^2\mathrm{BOT})$  , explicitly modeling the understanding between interlocutors with a transmitter-receiver framework. Distinguished from traditional methods,  $\mathcal{P}^2$  BOT highlights a novel concept, mutual persona perception, which is better suited to describe the information exchange process that empowers the interlocutors to get to know each other. In order to train  $\mathcal{P}^2$  BOT for personalized dialogue generation, we employ supervised training and self-play fine-tuning piloted by reward signals characterizing mutual persona perception. Experiments on the PERSONACHAT dataset demonstrate the superiority of our approach over the baselines in both automatic met
rics and human evaluations<sup>1</sup>.
2 Methodology Overview
The central idea of  $\mathcal{P}^2$  BOT is to explicitly model understanding between interlocutors and enhance dialogue generation via mutual persona perception. It comprises two components, Transmitter and Receiver, respectively responsible for dialogue generation and mutual persona perception. Figure 2 gives an overview of  $\mathcal{P}^2$  BOT: interlocutor  $\mathcal{A}$  has a persona  $\mathbf{w}^{\mathcal{A}}$ , described with  $L$  profile sentences  $\{w_1^{\mathcal{A}}, \dots, w_L^{\mathcal{A}}\}$ . When she first meets the other interlocutor  $\mathcal{B}$ , they are going to know each other through a  $N$ -turn dialogue  $(x_1^{\mathcal{A}}, x_1^{\mathcal{B}}, \dots, x_N^{\mathcal{A}}, x_N^{\mathcal{B}})$ , where  $x_n^{\mathcal{A}}$  denotes the utterance that  $\mathcal{A}$  says in  $n$ -th turn and  $N$  denotes the number of total turns. Given the entire dialogue history up to  $n$ -th turn  $\mathbf{h}_n^{\mathcal{A}} = (x_1^{\mathcal{A}}, \dots, x_{n-1}^{\mathcal{B}})$ , Transmitter generates  $x_n^{\mathcal{A}}$  according to the distribution  $p(x_n^{\mathcal{A}} | \mathbf{w}^{\mathcal{A}}, \mathbf{h}_n^{\mathcal{A}})$ , and transmits it to  $\mathcal{B}$ . The same process applies to  $\mathcal{B}$ , keeping the conversation flowing.
As the conversation goes on, impressions are gradually built via utterances. For example, when  $\mathcal{A}$  says "I don't watch movies more of a writer," the impression that "A is a writer" is left on  $\mathcal{B}$ 's mind. As mentioned above, a successful conversation helps interlocutors know each other, which means  $\mathcal{B}$ 's impression of  $\mathcal{A}$  should correspond to  $\mathcal{A}$ 's persona and vice versa. Receiver aims to measure the proximity between the built impressions and the actual personas. Specifically, as demonstrated by the dashed black lines in Figure 2, Receiver first projects impressions and personas into a latent space, and then measures the relevance between them based on the impression encoding (e.g.  $\mathbf{H}^{\mathcal{A}}$ ,  $\mathcal{B}$ 's impression on  $\mathcal{A}$ , projected from  $\mathcal{A}$ 's
Our code is available at https://github.com/SivilTaram/Persona-Dialogue-Generation
Figure 3: The overall architecture of Transmitter. "Block" is short for "Transformer Block". Arrows  $\nearrow$  bridge the current block to subsequent blocks of its following layer. Position encoding is to incorporate position information into block by assigning an embedding for each absolute position in the sequence. Here we omit the architecture inside the block, and refer the readers to Vaswani et al. (2017) for more details. [MASK] tokens are ignored in the training objective.
utterances  $\mathbf{x}^{\mathcal{A}}$ ), and persona encoding (e.g.  $\mathbf{W}^{\mathcal{A}}$ , projected from  $\mathcal{A}$ 's persona  $\mathbf{w}^{\mathcal{A}}$ )<sup>2</sup>. The relevance scores serve as mutual persona perception rewards, and are further incorporated into the training of Transmitter. Details of the two components are presented in Section 3 and 4.
3 Transmitter
Following previous work (Li et al., 2016b; Zhang et al., 2018b), we treat dialogue generation as a sequence generation problem. Concretely, we employ the pretraining transformer language model introduced in Radford et al. (2018) (i.e. GPT) to initialize Transmitter. The entire training procedure consists of two steps: (1) Supervised Dialogue Generation. We optimize Transmitter via maximum likelihood estimation (MLE) on the supervised dialogue generation task. (2) Self-play Model Finetuning. We simulate dialogues between two randomly paired interlocutors, encouraging Transmitter to learn a policy that maximizes reward signals via reinforcement learning (RL) (Sutton et al., 1999). The design of the reward function considers both language modeling and our proposed mutual persona perception.
3.1 Supervised Dialogue Generation
As illustrated in Figure 3, Transmitter follows the overall architecture of 12 stacked transformer layers to encode context and generate response. Here, the context contains the persona  $\mathbf{w}^{\mathcal{A}}$ , the dialogue
history  $\mathbf{h}_n^A$ , and several special tokens (e.g. [PS] which indicates the start of persona). Given a training instance  $(\mathbf{w}^A, \mathbf{h}_n^A, x_n^A)$ , the training objective of MLE is to maximize the conditional log-likelihood as:
$$
\mathcal {L} _ {\mathrm {m l e}} = \sum_ {t} \log p _ {\theta} \left(x _ {n, t} ^ {\mathcal {A}} \mid \mathbf {w} ^ {\mathcal {A}}, \mathbf {h} _ {n} ^ {\mathcal {A}}, x _ {n, <   t} ^ {\mathcal {A}}\right), \tag {1}
$$
where  $\theta$  is the parameter of Transmitter.  $x_{n,t}^{\mathcal{A}}$  means the  $t$ -th token in  $x_{n}^{\mathcal{A}}$ , and  $x_{n, < t}^{\mathcal{A}}$  indicates the token sequence before  $t$ -th token. Equation 1, hereafter simplified as  $\log p_{\theta}(x_n^{\mathcal{A}}|\mathbf{w}^{\mathcal{A}},\mathbf{h}_n^{\mathcal{A}})$ , applies to both  $\mathcal{A}$  and  $\mathcal{B}$ , and we mention  $\mathcal{A}$  for the sake of brevity (the same as below).
During inference, beam search is applied to store top-ranked response candidates  $\{\hat{x}_n^A\}$ , and Transmitter subsequently chooses as prediction the one that maximizes the length-normalized score:
$$
x _ {n} ^ {\mathcal {A} ^ {*}} = \underset {\hat {x} _ {n} ^ {\mathcal {A}}} {\arg \max } \frac {\log p _ {\theta} (\hat {x} _ {n} ^ {\mathcal {A}} \mid \mathbf {w} ^ {\mathcal {A}} , \mathbf {h} _ {n} ^ {\mathcal {A}})}{| \hat {x} _ {n} ^ {\mathcal {A}} |}. \tag {2}
$$
Besides the sequence generation task, inspired by Wolf et al. (2019b), we set up an auxiliary task, Next Utterance Prediction. Apart from training Transmitter to generate responses, we also train it to discriminate whether the response is the next utterance of the given context. Concretely, we append a special token [CLS] to the tail of the generated tokens. A classifier is built on top of the token's hidden state in the last transformer layer, as indicated by the red rounded rectangle in Figure 3. In training, for each response, we randomly sample a distractor and train the classifier to give a higher score on the response than the distractor. In inference, the classifier is used to rank response candidates together with Equation 2. Denoting as  $y_{n} = 1$  the signal indicating the generated response  $\hat{x}_n^A$  is predicted as the next utterance, Equation 2 is extended as:
$$
\begin{array}{l} x _ {n} ^ {\mathcal {A} ^ {*}} = \underset {\hat {x} _ {n} ^ {\mathcal {A}}} {\arg \max } \left(\alpha \cdot \frac {\log p _ {\theta} \left(\hat {x} _ {n} ^ {\mathcal {A}} \mid \mathbf {w} ^ {\mathcal {A}} , \mathbf {h} _ {n} ^ {\mathcal {A}}\right)}{\left| \hat {x} _ {n} ^ {\mathcal {A}} \right|}\right) \tag {3} \\ \left. + (1 - \alpha) \cdot \log p _ {\theta} (y _ {n} = 1 | \mathbf {w} ^ {\mathcal {A}}, \mathbf {h} _ {n} ^ {\mathcal {A}}, \hat {x} _ {n} ^ {\mathcal {A}})\right), \\ \end{array}
$$
3.2 Self-play Model Fine-tuning
Although supervised dialogue generation alone can be used to mimic human-like responses, it does not inherently target at understanding. Therefore, we
2We take  $\mathcal{A}$  as an example, and all are similar to  $\mathcal{B}$
Figure 4: The illustration of the self-play procedure. Arrows  $\Rightarrow$  represent the process of dialogue generation driven by Transmitter. Note that  $x_{1}^{A^{*}}$  is directly taken from the dataset as it is difficult to generate high-quality utterances without any dialogue history.
further fine-tune Transmitter using reinforcement learning with the goal of maximizing mutual persona perception. Analogous to Lewis et al. (2017), we apply self-play to simulate the communication between two Transmitters, both of which have been trained as described in Section 3.1.
Specifically, we have the two Transmitters communicate with each other for several turns. One Transmitter serves as a user with the parameters frozen, while the other is a learnable agent. The parameter of the learnable agent,  $\theta$ , is fine-tuned during the self-play. Without loss of generality, in our experiments, we let interlocutor  $\mathcal{A}$ , who starts a conversation, be the user, and correspondingly  $\mathcal{B}$  be the learnable agent.
Here we introduce some necessary formulations for modeling our problem with reinforcement learning. A state contains the persona and the dialogue history. For example, the state for  $\mathcal{B}$  at turn  $n$  is defined as  $s_n^{\mathcal{B}} = \{\mathbf{w}^{\mathcal{B}},\mathbf{h}_n^{\mathcal{B}}\}$ . An action  $a_{n}^{\mathcal{B}}$  is the response to be generated. The action space is infinitely large as the response can be arbitrary long. Taking  $s_n^{\mathcal{B}}$  as input, the parameter  $\theta$  defines a policy  $p_{\theta}(a_n^{\mathcal{B}}|s_n^{\mathcal{B}})$ , through which the learnable agent generates its response.
As illustrated in Figure 4, when it is  $\mathcal{B}$ 's turn to speak,  $\mathcal{B}$  receives  $s_n^{\mathcal{B}}$  and picks  $a_{n}^{\mathcal{B}}$  according to the policy  $p_{\theta}$ . As for  $\mathcal{A}$ , it receives  $s_n^{\mathcal{A}}$  and generates the response  $x_{n}^{\mathcal{A}^{*}}$  to simulate a user.  $\mathcal{A}$  and  $\mathcal{B}$  alternately produce responses till the number of turns exceeds the given limit. Once a complete dialogue is generated, the reward is collected to optimize  $\theta$  using policy gradient (Sutton et al., 1999). Denoting as  $R(a_{n}^{\mathcal{B}})$  the reward  $\mathcal{B}$  gets at turn  $n$  (more details are provided later), we can optimize it by maximizing the following objective:
$$
\mathcal {L} _ {\mathrm {r l}} = \mathbb {E} _ {a _ {n} ^ {\mathcal {B}} \sim p _ {\theta} \left(a _ {n} ^ {\mathcal {B}} \mid s _ {n} ^ {\mathcal {B}}\right)} [ R \left(a _ {n} ^ {\mathcal {B}}\right) ]. \tag {4}
$$
Applying likelihood ratio trick,  $\theta$  is updated by ascending the following gradient:
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {r l}} = \mathbb {E} _ {a _ {n} ^ {\mathcal {B}} \sim p _ {\theta} (a _ {n} ^ {\mathcal {B}} | s _ {n} ^ {\mathcal {B}})} \nabla_ {\theta} \log p _ {\theta} (a _ {n} ^ {\mathcal {B}} | s _ {n} ^ {\mathcal {B}}) R (a _ {n} ^ {\mathcal {B}}). \tag {5}
$$
As aforementioned, the space of action  $a_{n}^{\mathcal{B}}$  is infinite. In practice, REINFORCE algorithm (Williams, 1992) is leveraged to approximate Equation 5 by sampling  $a_{n}^{\mathcal{B}}$  from policy  $p_{\theta}(a_{n}^{\mathcal{B}}|s_{n}^{\mathcal{B}})$ . Furthermore, subtracting a baseline (Weaver and Tao, 2001), here the mean reward of a mini-batch, is applied on  $R(a_{n}^{\mathcal{B}})$  to reduce variance. The agent samples tokens one by one through multinomial sampling over the output distribution of  $\mathcal{B}$ , until the special token [EOS] is sampled or exceeding the maximum allowed decoding step (e.g. 32). Compared to beam search sampling, multinomial sampling provides more diversities.
3.3 Reward Shaping (RS)
As described in Section 1, we believe that a high-quality chit-chat conversation should highlight both human language modeling and mutual persona perception. Bearing this in mind, we design three rewards to address language style, discourse coherence and mutual persona perception respectively.
RS.1 Language Style The generated responses should conform to human language styles, which we believe can be evaluated by a pretrained language model (i.e. GPT). After length normalization, the score for  $a_{n}^{\mathcal{B}}$  is given as:
$$
R _ {1} \left(a _ {n} ^ {\mathcal {B}}\right) = \frac {1}{\left| a _ {n} ^ {\mathcal {B}} \right|} \sum_ {t} \log p _ {\mathrm {l m}} \left(a _ {n, t} ^ {\mathcal {B}} \mid a _ {n, <   t} ^ {\mathcal {B}}\right), \tag {6}
$$
where  $a_{n,t}^{\mathcal{B}}$  and  $a_{n, < t}^{\mathcal{B}}$  have similar denotation as the previously mentioned  $x_{n,t}^{\mathcal{A}}$  and  $x_{n, < t}^{\mathcal{A}}$ .
RS.2 Discourse Coherence The language score is evaluated individually, without considering the discourse coherence. However, a reasonable response should establish links in meaning with context, which is also an important aspect of humanlike responses. To take into account the discourse coherence, we employ the well-trained Next Utterance Predictor (mentioned in Section 3.1). The reward is given by the log probability of  $a_{n}^{\mathcal{B}}$  being the next utterance of  $s_n^{\mathcal{B}}$ :
$$
R _ {2} \left(a _ {n} ^ {\mathcal {B}}\right) = \log p _ {\theta} \left(y _ {n} = 1 \mid a _ {n} ^ {\mathcal {B}}, s _ {n} ^ {\mathcal {B}}\right). \tag {7}
$$
RS.3 Mutual Persona Perception RS.1 and RS.2 only steer the agent training process towards human-like responding. They do not explicitly encourage understanding between interlocutors. Therefore, we meticulously design the reward to characterize mutual persona perception. Contrast from RS.1 and RS.2, mutual persona perception is a long-term goal throughout the whole dialogue, meaning that the effect of current action might only play out some time later. For instance, receiving "what are your hobbies?" from  $\mathcal{B}$ , it is highly likely that  $\mathcal{A}$ 's response is relevant to  $\mathcal{A}$ 's hobbies. This suggests that, not only  $\mathcal{A}$ 's response but also  $\mathcal{B}$ 's initial question contributes to mutual persona perception. Denoting as  $\gamma$  the discount factor indicating how far ahead  $\mathcal{B}$  looks, the reward of mutual persona perception for  $a_{n}^{\mathcal{B}}$  is defined as:
$$
\begin{array}{l} R _ {3} \left(a _ {n} ^ {\mathcal {B}}\right) = r \left(a _ {n} ^ {\mathcal {B}}\right) + \sum_ {k = n + 1} ^ {N} \left(\gamma^ {2 (k - n) - 1} r \left(x _ {k} ^ {\mathcal {A} ^ {*}}\right) \right. \tag {8} \\ \left. + \gamma^ {2 (k - n)} r \left(a _ {k} ^ {\mathcal {B}}\right)\right), \\ \end{array}
$$
where  $r(a_n^{\mathcal{B}})$  is the persona perception score that  $\mathcal{B}$  obtains in  $n$ -th turn, and  $r(x_k^{A^*})$  is defined likewise.  $r(a_n^{\mathcal{B}})$  can be computed using a score function:
$$
r \left(a _ {n} ^ {\mathcal {B}}\right) = \operatorname {s c o r e} \left(a _ {n} ^ {\mathcal {B}}, \mathbf {w} ^ {\mathcal {B}}\right). \tag {9}
$$
In  $\mathcal{P}^2$  BOT, the score function comes from Receiver, which will be elaborated in Section 4. The final reward  $R(a_{n}^{\mathcal{B}})$  for  $a_{n}^{\mathcal{B}}$  is a weighted sum of the rewards listed above:
$$
R = \lambda_ {1} R _ {1} + \lambda_ {2} R _ {2} + \lambda_ {3} R _ {3}, \tag {10}
$$
where  $\lambda_1, \lambda_2$  and  $\lambda_3$  are hyper-parameters.
4 Receiver
Receiver is devised to measure the proximity between the built impressions and the actual personas, implemented by negative sampling. Specifically, in training, we randomly sample a persona distractor  $\mathbf{w}^{\mathcal{Z}}$ . Receiver is trained to identify the real persona  $\mathbf{w}^{\mathcal{A}}$  from  $\{\mathbf{w}^{\mathcal{A}}, \mathbf{w}^{\mathcal{Z}}\}$ . In inference, for each utterance, Receiver is responsible for providing a reasonable relevance score, to model our proposed mutual persona perception. The score subsequently joins the self-play fine-tuning on Transmitter as part of the rewards, as in Equation 8.
Figure 5: The overall architecture of Receiver (see text).
As illustrated in Figure 5, Receiver contains two different encoders for impression and persona respectively. Initialized by BERT (Devlin et al., 2019), both encoders provide deep contextualized representations for each token. Then we average all the representations, yielding a fixed  $d$ -dimensional vector for one sentence. In this way, feeding  $(x_{1}^{\mathcal{A}}, x_{2}^{\mathcal{A}}, \dots, x_{N}^{\mathcal{A}})$  into the impression encoder consecutively, we obtain the impression encoding  $\mathbf{H}^{\mathcal{A}} \in \mathbb{R}^{N \times d}$ . The persona encoding  $\mathbf{W}^{\Delta} \in \mathbb{R}^{L \times d}$  is produced likewise, where  $\Delta \in \{\mathcal{A}, \mathcal{Z}\}$ . The relevance score matrix  $\mathbf{U}^{\Delta}$  is computed via the scaled dot product (Vaswani et al., 2017):
$$
\mathbf {U} ^ {\Delta} = \frac {\mathbf {H} ^ {\mathcal {A}} \left(\mathbf {W} ^ {\Delta}\right) ^ {\top}}{\sqrt {d}}, \in \mathbb {R} ^ {N \times L}. \tag {11}
$$
In essence, Receiver is expected to capture fine-grained correlations between the persona and the dialogue. However, we do not have access to the golden fine-grained correlations. The only thing we know is that, compared with  $\mathbf{W}^{\mathcal{Z}}$ ,  $\mathbf{H}^{\mathcal{A}}$  is more correlated to  $\mathbf{W}^{\mathcal{A}}$ . Since the comparison is at a coarse granularity, we gather  $\mathbf{U}^{\Delta}$  into the cumulative score  $c^{\Delta}$  through an aggregate function  $Agg$ , as shown in Figure 5. To encourage  $c^{\mathcal{A}}$  while at the same time depress  $c^{\mathcal{Z}}$ , we design a marginal loss  $\mathcal{L}_{\mathrm{rec}}$ , which makes  $c^{\mathcal{A}}$  larger than  $c^{\mathcal{Z}}$  by a margin  $m$ . Moreover, considering that an utterance generally relates to zero or one profile,  $L_{1}$  regularization is enforced to make  $\mathbf{U}^{\Delta}$  sparse. Combining all of these, the training loss for Receiver is:
$$
\mathcal {L} _ {\text {r e c}} = \max  \left(0, m + c ^ {\mathcal {Z}} - c ^ {\mathcal {A}}\right) + \beta \cdot | \mathbf {U} ^ {\Delta} | _ {1}, \tag {12}
$$
where  $\beta$  is a hyper-parameter for penalty.
As for  $Agg$ , one straightforward way is to average over all positions of  $\mathbf{U}^{\Delta}$ . However, it maximizes every entry in  $\mathbf{U}^{\Delta}$ , including all those that
Category	Model	Original	Revised
Hits@1(%)↑	ppl↓	F1(%)↑	Hits@1(%)↑	ppl↓	F1(%)↑
Retrieval	KV Profile Memory	54.8	-	14.25	38.1	-	13.65
Dually Interactive Matching	78.8	-	-	70.7	-	-
Generative	Generative Profile Memory	10.2	35.01	16.29	9.9	34.94	15.71
Language Model	-	50.67	16.30	-	51.61	13.59
SEQ2SEQ-ATTN	12.5	35.07	16.82	9.8	39.54	15.52
Pretrain	Lost In Conversation	17.3	-	17.79	16.2	-	16.83
Fintune	Transfertransfo	82.1	17.51	19.09	-	-	-
P2BOT (Our)	81.9 [0.1]	15.12 [0.16]	19.77 [0.08]	68.6 [0.2]	18.89 [0.11]	19.08 [0.07]
Table 1: Automatic evaluation results of different methods on the PERSONA-CHAT dataset. The standard deviation  $[\sigma]$  (across 5 runs) of  $\mathcal{P}^2$  BOT is also reported. All the results were evaluated on the dev set since the test set was not publicly available.
should not be activated (e.g. relevance scores between unrelated profile sentences and utterances), introducing unnecessary noise into the training of Transmitter. To alleviate the problem, we choose to implement  $Agg$  as a controllable weighted function, which summarizes  $\mathbf{U}_{n,:}^{\Delta}$  as:
$$
A g g \left(\mathbf {U} _ {n,:} ^ {\Delta}\right) = \frac {\sum_ {k = 1} ^ {L} \exp \left(\mathbf {U} _ {n , k} ^ {\Delta} / \tau\right) \cdot \mathbf {U} _ {n , k} ^ {\Delta}}{\sum_ {k = 1} ^ {L} \exp \left(\mathbf {U} _ {n , k} ^ {\Delta} / \tau\right)}, \tag {13}
$$