SlowGuess/ACL_2020_OCR huggingface dataset:
Siqi Bao,\* Huang He,\* Fan Wang, Hua Wu and Haifeng Wang Baidu Inc., China
{baosiqi, hehuang, wangfan04, wu_hua, wanghaifeng}@baidu.com
Pre-training models have been proved effective for a wide range of natural language processing tasks. Inspired by this, we propose a novel dialogue generation pre-training framework to support various kinds of conversations, including chit-chat, knowledge grounded dialogues, and conversational question answering. In this framework, we adopt flexible attention mechanisms to fully leverage the bi-directional context and the uni-directional characteristic of language generation. We also introduce discrete latent variables to tackle the inherent one-to-many mapping problem in response generation. Two reciprocal tasks of response generation and latent act recognition are designed and carried out simultaneously within a shared network. Comprehensive experiments on three publicly available datasets verify the effectiveness and superiority of the proposed framework.
Dialogue generation is a challenging task due to the limited corpus of human conversations, complex background knowledge, and diverse relationships between utterances. Recently, pre-trained large-scale language models, such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019), have achieved prominent success in natural language processing. Such models are usually constructed based on a massive scale of general text corpora, like English Wikipedia or BooksCorpus (Zhu et al., 2015), where distributed representations can be learned automatically from the raw text. With these representations being fine-tuned, breakthroughs have been continuously reported for various downstream tasks, especially those on natural language understanding, such as question answering, natural language inference, and so on.
This pre-training and fine-tuning paradigm also sheds light on the tasks of natural language generation, like dialogue generation. However, the previous study demonstrates that there are some deficiencies in performance while directly fine-tuning BERT on small conversation datasets (Rashkin et al., 2019; Wolf et al., 2019). Possible reasons are three-fold: 1) the underlying linguistic patterns in human conversations can be highly different from those in general text, which suggests a potentially large gap in knowledge or data distribution; 2) the training mode of uni-directional dialogue generation is also distinct from that of bi-directional natural language understating as applied in BERT; 3) unlike most of the general NLP tasks, there exists a one-to-many relationship in dialogue generation, where the dialogue context may correspond to multiple appropriate replies.
In this paper, we propose a new method to tackle the above challenges, aiming to obtain a high-quality pre-training model for dialogue generation. First of all, to reduce the gap between data distributions, large-scale Reddit and Twitter conversations are utilized to further pre-train the generation model (upon the basis of language models pre-trained with general text). Secondly, to mitigate the difference in training mode, a flexible paradigm integrating uni- and bi-directional processing is employed in this work, which is inspired by the latest unified language modeling (Dong et al., 2019). Thirdly, a discrete latent variable is introduced to model the one-to-many relationship among utterances in conversations. Each value of the latent variable corresponds to the particular conversational intent of one response, which is referred as latent speech act.
Distinct with those controllable dialogue generation based on explicit labels (including emotion, keywords, domain codes, and so on) (Huang et al., 2018; Keskar et al., 2019), our latent variable gets
*First two authors contributed equally to this work.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85-96 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
exempted from the restriction of human annotations and can be learned automatically from the corpus in an unsupervised way. In the pre-training of dialogue generation, response generation and latent act recognition are carried out simultaneously within a shared network. Based on the context and latent variable, the generation task tries to maximize the likelihood of the target response. Meanwhile, the recognition task aims to estimate the latent variable w.r.t. the given context and target response. Apparently, the accurate recognition of the latent variable is a crucial factor in boosting the quality of response generation.
We conducted experiments on three different kinds of conversation tasks: chit-chat, knowledge grounded conversation, and conversational question answering. Experimental results verify the effectiveness and superiority of our pre-trained model as compared with the other state-of-the-art methods. Our pre-trained models and source code have been released at GitHub, hoping to facilitate further research progress in dialogue generation.
2 Dialogue Generation Pre-training
Given a piece of context, there exist multiple appropriate responses, leading to diverse conversation flows. It is widely recognized that the capability of modeling one-to-many relationship is crucial for the dialogue generation system (Zhao et al., 2017; Chen et al., 2019). To this end, we propose to encode discrete latent variables into transformer blocks for one-to-many relationship modeling, where two reciprocal tasks of response generation and latent act recognition are collaboratively carried out.
In our model, there are three elements: dialogue context  $c$ , response  $r$  and latent variable  $z$ . The dialogue context  $c$  consists of several history utterances. (For knowledge grounded conversation, it is conventional to concatenate background knowledge into the context as well (Wolf et al., 2019).) The response  $r$  is one piece of appropriate reply towards the given context. The latent variable  $z$  is one  $K$ -way categorical variable  $z \in [1, K]$ , with each value corresponding to a particular latent speech act in the response.
Figure 1: Graphical illustration of response generation (gray lines) and latent act recognition (dashed blue lines).
The probabilistic relationships among these elements are elaborated with the graphical model in Figure 1. Given a context  $c$ , there are multiple latent speech acts which can be taken as response intents (represented by the latent variable  $z$ ). Conditioned on the context and one selected latent speech act, the response is generated as  $p(r|c,z)$  (gray lines). Given a pair of context and response, the underlying latent speech act can be estimated as  $p(z|c,r)$  (dashed blue lines). As such, our pretraining of dialogue generation contains the following two tasks - response generation and latent act recognition.
We propose a unified infrastructure for the joint learning of both tasks, shown as Figure 2. The backbone of our infrastructure is inspired by the transformer blocks in (Dong et al., 2019), which supports both bi-directional encoding and uni-directional decoding flexibly via specific self-attention masks. Both response generation and latent act recognition are carried out under the unified network with shared parameters. Their detailed implementations are described as follows.
Given the context  $c$  and a specific speech act  $z$ , the response generation can be estimated as
$$
p (r | c, z) = \Pi_ {t = 1} ^ {T} p (r _ {t} | c, z, r _ {<   t}), \qquad (1)
$$
where  $T$  is the length of the target response  $r$  and  $r_{<t}$  denotes previously generated words. Since the response generation is a uni-directional decoding process, each token in the response only attends to those before it, shown as dashed orange lines in Figure 2.
The latent act recognition task is included to identify the corresponding value of  $z$  for the given context and the target response in the training data. The latent act recognition shares network parameters with response generation, but has a separate self-attention mask for bi-directional encoding. As shown in Figure 2, with a special mask symbol [M] as input, it keeps collecting information from the context and target response (red lines). In this way,
<https://github.com/PaddlePaddle/ Research/tree/master/NLP/Dialogue-PLATO
Figure 2: Architecture of dialogue generation with discrete latent variable. In self-attention visualization, red and blue lines denote bi-directional attention, and dashed orange lines denote uni-directional attention.
Figure 3: Input representation. The input embedding is the sum of token, role, turn and position embeddings.
the corresponding speech act for the target response can be recognized as  $z \sim p(\mathbf{z} | c, r)$ , where  $p(\mathbf{z} | c, r)$  is the estimated posterior distribution over discrete latent values.
2.2 Input Representation
For multi-turn conversation modeling, elaborate designs have been made on the input representation in this work. For each token, its input embedding is the sum of corresponding token, role, turn and position embeddings. One visual example is shown in Figure 3 and details are described in the following.
- The input is the concatenation of latent variable, dialogue context and response. Following the pre-processing of BERT (Devlin et al., 2019), the input text is tokenized with WordPiece (Wu et al., 2016). A special end-of-utterance [EOU] token is appended to the end of each utterance for separation. Another begin-of-utterance [BOU] token is added at the beginning of the response, whose final hidden state (i.e., output of the last transformer block) is used to predict next token during generation.
- Given that  $z$  is one  $K$ -way categorical variable,
its token embedding  $\mathbf{E}_{[z]}$  is mapped from the latent embedding space  $\mathbf{E}_{\mathbf{z}} \in \mathbb{R}^{K \times D}$ . For the rest tokens in the vocabulary, they are initialized using BERT's WordPiece embeddings.
- Role embeddings are employed to differentiate the characters evolved in the conversation. The role embedding  $\mathbf{E}_A$  is added for the response, as well as dialogue utterances generated by the same character in the context. And role embedding  $\mathbf{E}_B$  is used for the other character. (For knowledge grounded conversation,  $\mathbf{E}_C$  is used as the role embedding of background knowledge.)
- In the interactive conversation, there are multi-turn utterances and we employ relative order in the assignment of turn embeddings. The turn embedding for the response is set to  $\mathbf{E}_{[0]}$ , and the turn embedding of its last utterance is  $\mathbf{E}_{[-1]}$ , and etc. Our utilization of relative turn embeddings instead of absolute ones enables the model to assign turn embedding  $\mathbf{E}_{[0]}$  to the response consistently and makes response generation exempt from the disturbance of its round number within the dialogue.
- Position embeddings are added according to the
token position in each utterance. Note that for the special token of latent variable, its corresponding role, turn and position embeddings are all set to empty.
2.3 Pre-training Objectives
We employ three loss functions in dialogue generation pre-training: negative log-likelihood (NLL) loss, bag-of-words (BOW) loss and response selection (RS) loss. Brief illustration is shown in the last column of Figure 2 and detailed descriptions will be provided in this section.
2.3.1 Response Generation
In our model, the response is generated conditioned on the latent variable and the context. The widely adopted NLL loss is embraced in the pre-training:
$$
\begin{array}{l} \mathcal {L} _ {N L L} = - \mathbb {E} _ {z \sim p (\mathbf {z} | c, r)} \log p (r | c, z) \tag {2} \\ = - \mathbb {E} _ {z \sim p (\mathbf {z} | c, r)} \sum_ {t = 1} ^ {T} \log p (r _ {t} | c, z, r _ {<   t}), \\ \end{array}
$$
where  $z$  is the latent speech act of this training pair  $(c,r)$ , sampled from the probability distribution  $p(\mathbf{z}|c,r)$ . The posterior distribution over latent values is estimated through the task of latent act recognition:
$$
p (\mathbf {z} \mid c, r) = \operatorname {s o f t m a x} \left(W _ {1} h _ {[ M ]} + b _ {1}\right) \in \mathbb {R} ^ {K}, \tag {3}
$$
where  $h_{[M]}\in \mathbb{R}^D$  is the final hidden state of the special mask,  $W_{1}\in \mathbb{R}^{K\times D}$  and  $b_{1}\in \mathbb{R}^{K}$  denote the weight matrices of one fully-connected layer.
Besides the classical NLL loss, the bag-of-words loss (Zhao et al., 2017) is also employed to facilitate the training process of latent discrete variables:
$$
\begin{array}{l} \mathcal {L} _ {B O W} = - \mathbb {E} _ {z \sim p (\mathbf {z} | c, r)} \sum_ {t = 1} ^ {T} \log p \left(r _ {t} \mid c, z\right) \tag {4} \\ = - \mathbb {E} _ {z \sim p (\mathbf {z} | c, r)} \sum_ {t = 1} ^ {T} \log \frac {e ^ {f _ {r t}}}{\sum_ {v \in V} e ^ {f _ {v}}}, \\ \end{array}
$$
where  $V$  refers to the whole vocabulary. The function  $f$  tries to predict the words within the target response in a non-autoregressive way:
$$
f = \operatorname {s o f t m a x} \left(W _ {2} h _ {z} + b _ {2}\right) \in \mathbb {R} ^ {| V |}, \tag {5}
$$
where  $h_{z}$  is the final hidden state of the latent variable and  $|V|$  is the vocabulary size.  $f_{r_t}$  denotes the estimated probability of word  $r_t$ . As compared with NLL loss, the BOW loss discards the order of words and forces the latent variable to capture the global information of the target response.
2.3.2 Response Selection
Response selection helps distinguish whether the response is relevant with the dialogue context and consistent with the background knowledge. Meanwhile, its score can be regarded as an indicator of coherence during inference, helping to select the most coherent one from multiple candidate responses.
Particularly, the training of response selection is carried out together with the bi-directional encoding of latent act recognition. The positive training samples come from the dialogue context and corresponding target response  $(c,r)$ , with label  $l_{r} = 1$ . And the negative samples are created by randomly selecting responses from the corpus  $(c,r^{-})$ , with label  $l_{r^{-}} = 0$ . The binary cross-entropy loss of response selection is defined as follows:
$$
\mathcal {L} _ {R S} = - \log p (l _ {r} = 1 | c, r) - \log p \left(l _ {r ^ {-}} = 0 | c, r ^ {-}\right) \tag {6}
$$
The above probability is estimated through one fully-connected layer, with the final hidden state of the special mask fed as input:
$$
p \left(l _ {r} = 1 \mid c, r\right) = \operatorname {s i g m o i d} \left(W _ {3} h _ {[ M ]} + b _ {3}\right) \tag {7}
$$
To sum up, the total objective of our pre-training model is to minimize the integrated loss:
$$
\mathcal {L} = \mathcal {L} _ {N L L} + \mathcal {L} _ {B O W} + \mathcal {L} _ {R S} \tag {8}
$$
2.4 Pre-training Procedure
Our pre-training model contains 12 transformer blocks, with network parameters initialized using  $\mathrm{BERT}_{\mathrm{BASE}}$ . Large-scale conversation datasets – Twitter (Cho et al., 2014) and Reddit (Zhou et al., 2018; Galley et al., 2019) are employed for pretraining, which results in 8.3 million training samples in total. For each training sample of context and target response  $(c,r)$ , it needs to pass through the network twice to accomplish the tasks of latent act recognition and response generation. And the pre-training steps are summarized as follows:
1) Latent Act Recognition
- Given a pair of context and target response, estimate the posterior distribution  $p(\mathbf{z}|c,r)$
- Randomly select  $r^{-}$  and calculate  $\mathcal{L}_{RS}$
2) Response Generation
- With the sampled latent value  $z \sim p(\mathbf{z} | c, r)$ , calculate  $\mathcal{L}_{NLL}$  and  $\mathcal{L}_{BOW}$
3) Optimization
- Sum up to obtain  $\mathcal{L}$ , and update network parameters with back-propagation
The hyper-parameters used in pre-training are listed as follows. The maximum sequence length of context and response is set to 256 and 50, respectively. The number of transformer blocks in our model  $L$  is 12 and the hidden embedding dimension  $D$  is 768. The batch size is set to 64 and  $K$  is set to 20 for the discrete latent variable. Adam optimizer (Kingma and Ba, 2015) is employed for optimization with a learning rate of 5e-5. The pretraining of dialogue generation was carried out on 8 Nvidia Telsa V100 32G GPU cards for 3.5M steps, taking about two weeks to reach convergence.
2.5 Fine-tuning and Inference
Our pre-trained model is flexible enough to support various kinds of dialogues, including chit-chat, knowledge grounded conversation, conversational question answering, etc. The fine-tuning on small conversation datasets can be carried out by following the training objectives defined in Equation (8). As the fine-tuning process reaches convergence, the response towards the given context can be obtained through the following inference procedure:
1) Candidate Response Generation
- Conditioned on each latent value  $z \in [1, K]$ , generate corresponding candidate response  $r$ .
2) Response Selection
- Calculate the probability for each response  $p(l_r = 1|c,r)$  and select the one with highest coherence value as the final response.
It is worth noting that the above fine-tuning and inference procedures are set up for the dialogue generation without any specific objectives. If there exists a specific objective within the conversation, such as letting both participants know more about each other (Bao et al., 2019), the fine-tuning can proceed to maximize the pre-defined rewards with reinforcement learning (RL). Under such circumstances, our latent discrete variable can be naturally treated as action within RL, and thus the response selection can be straightforwardly solved by selecting the action that results in the maximum reward.
3.1.1 Datasets
To evaluate the performance of our proposed method, comprehensive experiments have been carried out on three publicly available datasets.
- Persona-Chat (Zhang et al., 2018) is a knowledge grounded conversation dataset. It provides
both manually annotated conversations and corresponding persona profiles (background knowledge), where two participants chat naturally and try to get to know each other.
- Daily Dialog (Li et al., 2017) is a chit-chat dataset, which contains high-quality human conversations about daily life.
- DSTC7-AVSD (Alamri et al., 2019), short for Audio Visual Scene-aware Dialog of the DSTC7 challenge, is a conversational question answering dataset. In DSTC7-AVSD, the system need to generate an answer given dialogue context and background knowledge. There are two available options of knowledge utilization: 1) using single-modal information of text only, including video's caption and summary; 2) relying on multi-modal information, including text, audio and visual features. The single-modal option is adopted by our method in the experiments.
The descriptions and statistics of these datasets are summarized in Table 1.
3.1.2 Compared Methods
The following models have been compared in the experiments.
Baseline. Sequence to sequence with attention (Seq2Seq) (Vinyals and Le, 2015) is employed as the baseline for the experiments on Persona-Chat and Daily Dialog. DSTC7-AVSD has provided a baseline system, which is built upon hierarchical recurrent encoders with multi-modal features.
State of the art. Persona-Chat was also utilized in the ConvAI2 challenge (Dinan et al., 2019a), where the team of Lost in Conversation (LIC) (Golovanov et al., 2019) obtains the best performance. LIC is also one transformer based generation method and fine-tuned upon the pre-trained model of GPT (Radford et al., 2018). For the dataset of Daily Dialog, its best results are reported by the recently developed method - iVAEMI (Fang et al., 2019), which generates diverse responses with sample-based latent representation. In DSTC7-AVSD, the team of CMU (Sanabria et al., 2019) obtains the best performance across all the evaluation metrics.
Our method. To better analyze the effects of our latent discrete variable, we also compare to the version without latent variable (Our w/o Latent).2
${}^{2}$  It shares the same training settings as our method with latent variables: network parameters are first initialized with  ${\mathrm{{BERT}}}_{\mathrm{{BASE}}}$  ,and the pre-training is further carried out on Reddit and Twitter. The only difference lies in the incorporation of latent variable.
Dataset	Type	Knowledge	# Train	# Valid	# Test
Persona-Chat	Chit-chat with persona	Persona profiles	8,939 dialogues 131,438 turns	1,000 dialogues 15,602 turns	968 dialogues 15,024 turns
Daily Dialog	Chit-chat	N/A	11,118 dialogues 87,170 turns	1,000 dialogues 8,069 turns	1,000 dialogues 7,740 turns
DSTC7-AVSD	Conversational QA	Video caption & summary	7,659 dialogues 153,180 turns	1,787 dialogues 35,740 turns	1,710 dialogues 13,490 turns
Table 1: Summary of datasets used in the experiments.
Dataset	Model	Automatic Evaluation	Human Evaluation
BLEU-1/2	Distinct-1/2	Knowledge R/P/F1	Fluency	Coherence	Informativeness	Overall
Persona-Chat	Seq2Seq	0.448 / 0.353	0.004 / 0.016	0.004 / 0.016 / 0.006	1.82	0.37	0.85	0.34
LIC	0.405 / 0.320	0.019 / 0.113	0.042 / 0.154 / 0.064	1.95	1.34	1.09	1.29
Our w/o Latent	0.458 / 0.357	0.012 / 0.064	0.085 / 0.263 / 0.125	1.98	1.36	1.04	1.30
Our Method	0.406 / 0.315	0.021 / 0.121	0.142 / 0.461 / 0.211	1.99	1.51	1.70	1.50
Daily Dialog	Seq2Seq	0.336 / 0.268	0.030 / 0.128	-	1.85	0.37	0.44	0.33
iVAEMI	0.309 / 0.249	0.029 / 0.250	-	1.53	0.34	0.59	0.30
Our w/o Latent	0.405 / 0.322	0.046 / 0.246	-	1.91	1.58	1.03	1.44
Our Method	0.397 / 0.311	0.053 / 0.291	-	1.97	1.57	1.23	1.48
Table 2: Experimental results on Persona-Chat and Daily Dialog with automatic and human evaluations, with highest value written in bold.
Dataset	Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGH-L	CIDEr
DSTC7-AVSD	Baseline	0.626	0.485	0.383	0.309	0.215	0.487	0.746
CMU	0.718	0.584	0.478	0.394	0.267	0.563	1.094
Our Method	0.784	0.637	0.525	0.435	0.286	0.596	1.209
Our Method Upper Bound	0.925	0.843	0.767	0.689	0.361	0.731	1.716
Table 3: Experimental results on DSTC7-AVSD with automatic evaluation, with highest value written in bold.
Context	breaking news on the presidential race that could change your vote
Generated Responses	i m not voting for hillary i m voting for trump
i can t believe this is happening
it would be interesting to see the results of this election
trump is a role model that the news media mafia hides
i will be voting for hillary
Context	i plan on walking into the appalachian mountains and never coming back .
Generated Responses	i've been to the appalachian mountains a few times . it's a beautiful place .
i've never been to the appalachian mountains , but i've heard it's nice .
i wouldn't want to live there .
don't worry , you'll come back .
that sounds like a good plan .
Table 4: Examples of response generation with our pre-trained model.
Index	Initialization	Training Configurations	# Fine-tuning Dialogues
Model	Attention	Context Attention	Data	Latent	1k	5k	9k
1.1	BERT	Bi-direction	Bi-direction	-	-	58.091	33.143	26.727
1.2	GPT-2	Uni-direction	Uni-direction	-	-	31.251	25.630	24.638
1.3	GPT-2	Uni-direction	Bi-direction	-	-	25.193	18.225	16.538
2.1	GPT-2	Uni-direction	Bi-direction	Twitter & Reddit	-	16.141	13.981	13.332
2.2	BERT	Bi-direction	Bi-direction	Twitter & Reddit	-	15.836	13.799	13.105
3.1	BERT	Bi-direction	Bi-direction	Twitter & Reddit	✓	15.080	12.936	12.285
Table 5: Perplexity of different pre-trained models on Persona-Chat, with best value written in bold.
Both automatic and human evaluations are employed to assess the performance of compared methods. In automatic evaluation, the following metrics are included:
- BLEU (Chen and Cherry, 2014) measures the n-gram overlap between generated response and the target response.
- Distinct-1/2 (Li et al., 2016) measures the generation diversity, which is defined as the number of distinct uni- or bi-grams divided by the total amount of generated words.
- Knowledge Recall/Precision/F1 (Dinan et al., 2019b) measures the degree of informativeness w.r.t. background knowledge.
- In DSTC7-AVSD, the MSCOCO platform (Chen et al., 2015) is employed for evaluation. It compares the generated response with six ground truth responses, using metrics of BLEU, METEOR, ROUGH-L and CIDEr.