SlowGuess/NAACL_2022_OCR huggingface dataset:
Hongyuan Lu, Wai Lam, Hong Cheng, Helen M. Meng  
The Chinese University of Hong Kong
{hylu, wlam, hcheng, hmmeng}@se.cuhk.edu.hk
Incorporating personas information allows diverse and engaging responses in dialogue response generation. Unfortunately, prior works have primarily focused on self personas and have overlooked the value of partner personas. Moreover, in practical applications, the availability of the gold partner personas is often not the case. This paper attempts to tackle these issues by offering a novel framework that leverages automatic partner personas generation to enhance the succeeding dialogue response generation. Our framework employs reinforcement learning with a dedicatedly designed critic network for reward judgement. Experimental results from automatic and human evaluations indicate that our framework is capable of generating relevant, interesting, coherent and informative partner personas, even compared to the ground truth partner personas. This enhances the succeeding dialogue response generation, which surpasses our competitive baselines that condition on the ground truth partner personas.
Building informative and engaging dialogue agents (Zhang et al., 2020; Roller et al., 2021) is a popular research direction within the area of natural language processing. For the sake of engagement, diverse and consistent responses (Song et al., 2020, 2021) are important factors, and personas information (Zhang et al., 2018) gives rise to both. There are two types of personas, namely self persona and partner persona. The former refers to a self profile consisting of several sentences representing the dialogue agents. Such a persona allows producing consistent responses rather than solely relying on the personas that are randomly learned and embedded in the model parameters (Kim et al., 2020). The latter refers to a profile that represents the users. Leveraging such partner personas has been empirically shown to be helpful for dialogue response selection (Gu et al., 2021).
Unfortunately, the existence of partner personas suffers from the cold start (Schein et al., 2002; Zhang et al., 2014; Li et al., 2021) at the beginning of the conversation. Most of the works, if not all, (Li et al., 2016b; Mazaré et al., 2018; Song et al., 2019; Gu et al., 2019; Zhao et al., 2019; Madotto et al., 2019; Liu et al., 2020; Majumder et al., 2020; Wu et al., 2020a; Song et al., 2020) have been either overlooking partner personas or simply focusing on the impractical situation where partner personas guarantee to exist. In contrast, our work does not suffer from the practical issue when partner personas are missing during inference, and our proposed framework surpasses the baseline that conditions on the ground truth partner personas.
To our knowledge, this is the first attempt to formulate partner personas generation for improved performance on the downstream dialogue response generation. Our work is motivated by the underlying hypothesis that partner personas generation is plausible given the self personas and dialogue context. Automatic and human evaluation results support the hypothesis and indicate that generated personas are even more interesting than the ground truth, which improves the downstream dialogue response generation. This paper thus paves the way to exploit partner personas generation (PPG) for dialogue response generation (DRG).
We propose a novel framework composed of three major components, namely a personas generator, a dialogue response generator and a critic network. The personas generator generates partner personas, which the dialogue response generator conditions on. We employ reinforcement learning with a critic network that propagates the reward back to the generators for joint training.
Prior works have investigated partner persona retrieval (Zhang et al., 2018; Song et al., 2019). The human-constructed ground truth personas serve as the upper bound for such retrieval-based systems, and we argue that the ground truth is not coherent
Human Language Technologies, pages 5200 - 5212
and diverse enough. Interestingly, we observe that the generative counterpart proposed in our framework generates relevant, informative and coherent partner personas, which further improves the succeeding dialogue response generation. It follows another advantage that our framework does not need an external database to retrieve from (Madotto et al., 2020; Xu et al., 2021).
One close work to ours is a multi-task framework for meta-learning (Lee et al., 2021) that uses personas reconstruction as an auxiliary task to improve response consistency. The differences are that theirs does not differentiate between self personas and partner personas, while ours does. Theirs indicates an improvement over personality consistency, while ours report improvements for the overall quality. We conduct an empirical comparison with their model by reconstructing the partner personas. Experimental results indicate that such a multi-task model does not work well in our problem setting. Very recently, Zhou et al. (2021) formulates personas generation as a Seq2Seq task for improved downstream response generation via multi-task learning. In contrast, our work leverages reinforcement learning to jointly train the partner personas generator and the response generator.
Automatic and human evaluation results indicate that our framework can generate partner personas that are more diverse and interesting than the ground truth partner personas and generate more diverse and engaging responses than the baseline conditioned on ground truth partner personas.
2.1 Personalized Dialoggue Generation
Conditioning on personas helps to produce informative and engaging responses. The most well-known multi-turn dialogue dataset conditioned on personal profiles is PERSONCHAT (Zhang et al., 2018), in which two crowdsources converse and find more about each other. The community has proposed many methods to better utilize self personas. Mazaré et al. (2018) employs a pre-training stage based on dedicatedly extracted large-scale persona-based dialogues. Zhao et al. (2019) fuses information in personas and dialogue context into individual contextualized representations by attending to different parts of both. Gu et al. (2019) exploits the interaction between personas, dialogue
context and response to improve retrieval-based dialogue agents. Madotto et al. (2019) leverages meta-learning with several dialogues of the current speakers to enhance response personality. Welleck et al. (2019) releases a dataset for measuring dialogue consistency. Song et al. (2020) employs a multi-stage pipeline to improve response personality by response rewriting. Lee et al. (2021) uses multi-task learning for improved personality consistency in the meta-learning scenario. Gu et al. (2021) employs four different strategies for personas fusing to leverage both self persona. However, most of these works focus on exploiting self personas rather than partner personas, and they assume the existence of the gold partner personas.
2.2 User Profile Extraction
Li et al. (2014) leverages distant supervision to classify the spouse, education and job information from user twitters. Wu et al. (2020b) proposes a two-staged profile extractor that extracts attributes before extracting the underlying relationship. Wang et al. (2021) proposes to categorize the profile extraction task into two different difficulties, namely 'extraction' and 'inference', and they leverage a GPT-based generator to extract user profiles. These works have formulated user profile extraction as a classification task that conditions on an input sentence, and they aim at better profile extraction. In contrast, we propose to formulate personas generation to be conditioned dialogue input to be jointly trained with response generation. While ground truth personas serve as the upper bound for these user profile extractors, we empirically demonstrate that our reinforcement learning algorithm surpasses the response model conditioned on the ground truth partner personas. As supported by our human evaluation, we believe the underlying reason is that our model can leverage pre-trained generators to generate coherent and relevant partner personas.
Reinforcement learning (RL), or specifically, policy gradient methods (Williams, 1992), have been frequently adopted to both task-oriented dialogue agents (Roman Roman et al., 2020; Deng et al., 2021) or open-domain chitchat agents (Li et al., 2016c; Saleh et al., 2020). It can either propagate non-differentiable loss (Cai et al., 2019a) or optimize an expert reward such as ease of answering (Li et al., 2016c). It also adopts a scenario where a user simulator and a dialogue agent interact, and an
1Related resources can be found at https://github. com/HongyuanLuke/PPG.
Figure 1: An example of the inference flow that shows the generated partner personas and the incorporation of partner personas generation into response generation.
Figure 2: The illustrated reinforcement learning strategy that directly backpropagates the response-related rewards from the critic network to the partner personas generator and the dialogue response generator.
expert reward function can be defined to assign the goodness to each response generated (Roman Roman et al., 2020).
We propose a novel framework composed of three major components, namely a partner personas generator, a dialogue response generator and a reinforcement learning component with a critic network. Figure 1 depicts the inference flow of our setting. The input dialogue context with self persona is first fed into the partner personas generator.
The generated partner personas output is then concatenated with the dialogue context and the self personas as the input into the dialogue response generator. In the beginning, we train our partner personas generator and dialogue response generator under supervised learning. In the training stage, we use the ground truth partner personas to train the dialogue response generator, and we replace it with generated partner personas in the inference stage. After the supervised learning stage, the second stage is a reinforcement learning stage which jointly optimizes both partner personas generator and dialogue response generator as depicted in Figure 2 to train the partner personas generator under the reward signal that is relevant to dialogue response generation as well as fine-tuning dialogue response generator trained on the generated partner personas.2 Particularly, we employ a dedicatedly designed critic network that receives generated partner personas and generated dialogue responses as the input and output a reward that measures the relevance between the generated personas and responses and propagates back to the generators.
3.1 Partner Personas Generation (PPG)
A Seq2Seq neural network (Sutskever et al., 2014) is adopted as our partner personas generator for the task of partner personas generation (PPG). The concatenation of dialogue context  $\mathbf{c}$  and self personas  $s$  is fed as an input into the partner personas generator. The personas generator then outputs an approximated partner personas  $\hat{\boldsymbol{p}}$  conditioned on the input that maximises the following likelihood:
$$
P \left(\hat {\boldsymbol {p}} \mid \boldsymbol {s}, \boldsymbol {c}\right) = \prod_ {t = 1} ^ {T} P \left(\hat {p} _ {t} \mid \hat {p} _ {1}, \dots , \hat {p} _ {t - 1}, \boldsymbol {s}, \boldsymbol {c}\right),
$$
where  $T$  represents the length of the generated partner personas and  $\hat{p_t}$  represents the word at the position  $t$  that has been inferred.
For training, the ground truth partner personas  $\pmb{p}$  is used and we train our generator to maximise the likelihood  $P(\pmb{p} \mid s, c)$ . We generate the complete partner personas profiles in an one-off shot for all the dialogue samples.
3.2 Dialogue Response Generation (DRG)
We also adopt a Seq2Seq neural network for the task of dialogue response generation (DRG). During inference, the concatenation of dialogue context
$^{2}$ Section 5.7 presents an ablation study on reinforcement learning that demonstrates the effectiveness of this approach.
$\mathbf{c}$ , self personas  $\pmb{s}$ , and generated partner personas  $\hat{\pmb{p}}$  is fed as an input into the dialogue response generator. The response generator then outputs a dialogue response  $\hat{\pmb{r}}$  conditioned on the input, which maximises the conditional likelihood:  $P(\hat{\pmb{r}}\mid s,\hat{\pmb{p}},\pmb {c})$
For training, the ground truth partner personas  $\pmb{p}$  and the ground truth dialogue responses  $\pmb{r}$  are used.
3.3 Reinforcement Learning (RL)
We employ a critic network to compute the reinforcement learning rewards for our generators. We use a binary classifier as critic by extracting training instances  $(s, r, \mathbb{L} = 1)$ ,  $s^A, r^A, \mathbb{L} = 1$  and  $(s^B, r^B, \mathbb{L} = 1)$ . Then we can derive two negative samples as:  $(s^A, r^B, \mathbb{L} = 0)$  and  $(s^B, r^A, \mathbb{L} = 0)$ . Thereafter, we fine-tune on a binary classifier to be used as our critic in RL on the training partition by minimizing the binary cross-entropy loss:
$$
- \mathbb {L} l o g (P (\mathbb {L} | \boldsymbol {s}, \boldsymbol {r})) - (1 - \mathbb {L}) \log (1 - P (\mathbb {L} | \boldsymbol {s}, \boldsymbol {r})),
$$
where the binary label  $\mathbb{L}$  indicates whether the response is relevant to the personas.
We then use this classifier acting as a critic network that outputs  $\hat{\mathbb{L}}$ , conditioned on the generated partner personas  $\hat{\pmb{p}}$  and generated response  $\hat{\pmb{r}}$ . The predicted binary label  $\hat{\mathbb{L}}$  is then converted to a reward  $\mathbb{R}$ .  $\mathbb{R}$  is a positive reward when  $\hat{\mathbb{L}} = 1$ , and  $\mathbb{R}$  is a negative reward when  $\hat{\mathbb{L}} = 0$ . We empirically set the reward  $\mathbb{R}$  for RL to  $\{1, -1\}$  for both PPG and DRG. We then update our RL agents with the following gradients:
$$
\Delta \theta_ {\mathrm {P P G}} = - \mathbb {R} \nabla_ {\theta_ {\mathrm {P P G}}} \log P (\hat {\boldsymbol {p}} | \boldsymbol {s}, \boldsymbol {c})
$$
for the partner personas generator (PPG), and for the dialogue response generator (DRG):
$$
\Delta \theta_ {\mathrm {D R G}} = - \mathbb {R} \nabla_ {\theta_ {\mathrm {D R G}}} \log P (\hat {\boldsymbol {r}} | \boldsymbol {s}, \hat {\boldsymbol {p}}, \boldsymbol {c})
$$
By formulating a reward that measures the relevance between generated partner personas and generated dialogue response, we are motivated by the following objectives:
- Further fine-tune the partner personas generator to generate personas that benefits the downstream dialogue response generation.
- Further fine-tune the dialogue response generator trained with ground-truth partner personas to adapt to noisy partner personas generated by the partner personas generator.
As mentioned in Section 3.1, the first motivation is that we are generating the complete personas profile. However, some of them can be irrelevant and unhelpful for the next-turn dialogue response generation. It could be challenging for the partner personas generator alone to identify which personas could be helpful. Therefore, we design such a reward to train the personas generator to learn to generate a set of personas that is more helpful for the downstream dialogue response generation. Our second motivation is that the dialogue response generator has not been exposed to the generated partner personas. We would like to fine-tune the response generator to mitigate the potential training-inference discrepancy. Experimental results indicate that our design empirically works well.
The previous work from Cai et al. (2019a) employed critic network for RL loss backpropagation. The major difference is that their critic is trained in an adversarial manner (Li et al., 2018) to pick up the gold response among other negative candidates. Also, their critic network conditions only on the dialogue response but not on the generated skeleton. In contrast, we aim for improved response generation with a classifier conditioning on both the generated personas and the generated response.
For both PPG and DRG, perplexity (PPL) is reported to measure the intrinsic performance with the ground truth output (Roller et al., 2021). We adopt well-known sequence evaluation metrics weighted BLEU (Papineni et al., 2002) and F-measure for ROUGE-L (Lin, 2004) as the extrinsic evaluations. For PPG, we also report Distinct-N with  $N = \{1,2\}$  to measure the response diversity (Li et al., 2016a; Cai et al., 2019b; Gao et al., 2019) with the ratio of distinct unigrams/bigrams against total number of unigrams/bigrams generated.
We conduct experiments on the PERSONACHAT (Zhang et al., 2018), the most well-known multi-turn dialogue dataset conditioned on personas. We follow the train-valid/test split from the PARLAI
3 Our critic reports a test accuracy of about  $75\%$ . We empirically choose to use  $s$  instead of  $p$ . The latter reports a test accuracy of about  $60\%$ . This indicates that it seems people talk more seldomly about their partner during the conversation, but they still do, and it is useful to exploit partner perspas.
Model	PERSONACHAT-ORI	PERSONACHAT-REV
Perplexity↓	BLEU	ROUGE	Perplexity↓	BLEU	ROUGE
E2E w/o Partner Personas	14.70	0.7502	16.87	14.81	0.6772	16.79
E2E w/ Partner Personas in Training	14.76	0.7109	16.56	14.44	0.7083	16.43
E2E w/ Partner Personas in Training and Inference	14.00	0.8105	17.52	14.17	0.6631	16.64
GPT-2	17.53	0.8031	17.31	15.11	0.6095	16.59
TRANSFERTRANSFO (Wolf et al., 2019)	15.55	0.7346	17.02	18.32	0.5534	16.40
PERCVAE (Song et al., 2019)	41.43	0.2400	13.90	42.80	0.2385	13.51
PAML (Madotto et al., 2019)	41.86	0.4448	13.83	-	-	-
MTL w/ Personas Reconstruction (Lee et al., 2021)	232.1	0.0570	9.937	244.1	0.0504	9.916
Ours w/o Reinforcement Learning	13.91	0.8068	17.40	13.88	0.6583	17.01
Ours w/ Reinforcement Learning on PPG&DRG	13.05	1.0862	18.11	13.85	0.7380	17.47
Table 1: Automatic evaluation results on PERSONCHAT-ORI and PERSONCHAT-REV. Perplexity (PPL) attains a better quality with lower values and the remaining metrics attain a better quality with higher values.
platform (Miller et al., 2017) that contains about 65,000/7,800/7,500 instances respectively. Each instance contains about 8 utterances on average and about 4 traits for each of the self and partner personas. We denote the dataset with this original personas as PERSONCHAT-ORI. Later the original personas have been manually scrutinized by rephrasing, generalizing or specializing, which we denote as PERSONCHAT-REV. We apply the same preprocessing operation to both datasets. To train the critic for RL, we collected about 130,000 instances from the train split with equally distributed positive and negative samples.
4.2 Baselines and Comparison Models
E2E w/o Partner Personas This is an end-to-end (E2E) response generator without partner persona.
E2E w/ Partner Persons in Training With partial ground truth partner personas for training only.
E2E w/ Partner Personas in Training and Inference With ground truth partner personas for training and inference. During early experiments, we found that feeding all traits yields lower performance. Retrieving Top-3 relevant partner personas using BM25 (Robertson and Walker, 1994) yields the best performance on the original personas.
GPT-2 This is a comparison model fine-tuned on GPT-2 (Radford et al., 2019). We build the same three E2E systems described above, and the best model is selected, the third one.
TRANSFERTRANSFO A comparison model built with a Transformer-based model pre-trained on gen
eral domain corpus, which is then fine-tuned on PERSONACHAT (Wolf et al., 2019).
PERCVAE This is a comparison model that employs a memory-augmented architecture incorporated with conditional variational autoencoder that exploits persona information (Song et al., 2019).
PAML This is a comparison model that leverages several dialogues collected from the same speaker to enhance response personality via meta-learning (Madotto et al., 2019). As the authors did not conduct experiments on the PERSONACHAT-REV and no preprocessing scripts are provided for the revised personas, we only report the results of their model on the PERSONACHAT-ORI only.
MTL w/ Personas Reconstruction This is a multi-task learning (MTL) comparison model (Lee et al., 2021) trained to maximise the objective:
$$
\alpha \mathcal {L} _ {\mathrm {P P G}} + (1 - \alpha) \mathcal {L} _ {\mathrm {D R G}},
$$
where  $\mathcal{L}_{\mathrm{PPG}}$  represents the auxiliary PPG likelihood, and  $\mathcal{L}_{\mathrm{DRG}}$  represents the DRG likelihood.  $\alpha$  is weight tuned over the validation set, and both tasks condition on dialogue context and self personas and share the same model parameters.
5.1 Dialogue Response Generation Results
We build our baselines, the partner personas generator and the dialogue response generator based on a state-of-the-art pre-trained dialogue model DIALOGPT (Zhang et al., 2020) for parameters
Dialogue Context	PARTNER PERSONAS	DIALOGUE RESPONSE
Gold Partner	Generated Partner	Ours	E2E w/ Full	Human
Hey! Do you like music? I like metallica.	My favourite band is metallica.	My favourite band is metallica.	Hi! I do! I love metallica!	I do.	I like country music mostly.
I know, I grew up in a horse ranch that is so large.	I also enjoy cooking meals with food from our garden.	I like to cook with the food I grow in my garden.	I have a small garden in my home.	I am sorry, I am not very tall.	I wish my childhood was as exciting.
I meditate. It helps with my anger, I can be pretty violent.	I am in the army.	I am an army ranger.	I am a general in the army.	I do not have time for things like that. I am a busy person.	Do you like to work for long hour? Do you have any pet?
Yes, I have 8 dogs, 7 cats and 17 birds.	-	I have a dog and two cats.	I have a dog and a cat.	I have a dog.	That is a lot! I have 3 cats and 2 dogs.
Table 2: Case studies that compare our framework against the baseline with the complete partner personas as well as the human response. We present the preceding partner utterance as dialogue context, and we give the most salient ground truth partner personas (Gold Partner) and generated partner personas (Generated Partner) for clarity.
initialization. More implementation details can be found in Appendix B.
The dialogue response generation results are presented in Table 1. Our framework with reinforcement learning attains the best over all the metrics on both PERSONACHAT-ORI and PERSONACHAT-REV. This supports the usefulness of our framework, which generates reasonable personas and effectively enhances the succeeding dialogue response generation, through the use of RL.
Although TRANSFERTRANSFO attains a better score on the PPL than the fine-tuned GPT-2, GPT-2 have better extrinsic scores than TRANSFERTRANSFO. GPT-2 also has better overall scores than the E2E baselines without the complete partner personas. However, it is surpassed by the E2E baseline with the complete partner personas during training and inference.
The E2E baseline with the complete ground truth partner personas attains better scores on all of the metrics than our remaining baselines. Our framework with RL succeeds the performance of such a competitive baseline for both PERSONCHAT-ORI and PERSONCHAT-REV, indicating our proposed framework's robustness against paraphrasal.
The multi-task learning comparison model (Lee et al., 2021) produces less promising results. Concretely, we postulate that the nature of PPG and DRG largely differs. The textual format of partner personas always initiates with first-person sentence starters, while dialogue responses are more general, ranging from greetings to goodbyes. Therefore, it could be hard to capture both in a single model.
Figure 3: Analysis for the cold start problem with limited dialogue turns available. Note that all of these baselines are fine-tuned on DIALOGPT.
5.2 Cold Start