SlowGuess/NAACL_2021_OCR huggingface dataset:
Haolan Zhan $^{1}$ , Hainan Zhang $^{2\dagger}$ , Hongshen Chen $^{2}$ , Zhuoye Ding $^{2}$ , Yongjun Bao $^{2}$  and Yanyan Lan $^{3}$
<sup>1</sup> Institute of Software, Chinese Academy of Sciences, Beijing, China
$^{2}$ Data Science Lab, JD.com, Beijing, China
$^{3}$ Institute for AI Industry Research, Tsinghua University, Beijing, China  
zhanhaolan316@gmail.com, zhanghainan6@jd.com, ac@chenhongshen.com, dingzhuoye@jd.com, lanyanyan@tsinghua.edu.cn
Knowledge data are massive and widespread in the real-world, which can serve as good external sources to enrich conversations. However, in knowledge-grounded conversations, current models still lack the fine-grained control over knowledge selection and integration with dialogues, which finally leads to the knowledge-irrelevant response generation problems: 1) knowledge selection merely relies on the dialogue context, ignoring the inherent knowledge transitions along with conversation flows; 2) the models often over-fit during training, resulting with incoherent response by referring to unrelated tokens from specific knowledge content in the testing phase; 3) although response is generated upon the dialogue history and knowledge, the models often tend to overlook the selected knowledge, and hence generates knowledge-irrelevant response. To address these problems, we proposed to explicitly model the knowledge transition in sequential multi-turn conversations by abstracting knowledge into topic tags. Besides, to fully utilizing the selected knowledge in generative process, we propose pretraining a knowledge-aware response generator to pay more attention on the selected knowledge. In particular, a sequential knowledge transition model equipped with a pretrained knowledge-aware response generator (SKT-KG) formulates the high-level knowledge transition and fully utilizes the limited knowledge data. Experimental results on both structured and unstructured knowledge-grounded dialogue benchmarks indicate that our model achieves better performance over baseline models.
Knowledge-grounded conversations (Long et al., 2017; Liu et al., 2018; Niu et al., 2019; Xu et al., 2020), aiming at improving the informativeness
Do you like Aries actors?
course, I am the Aries!
Knowledge Transition
Chao Wu's astrological sign is also the Aries.
I know him, his blood type is O.
Really? I just know one of his movies Lunar Eclipse.
Figure 1: The example from the DuConv dataset (Wu et al., 2019), shows the knowledge transition in real dialogue.
It tells a story about twin flowers.
{Lunar Eclipse, review, twin flowers}
and specificity of dialogue generation by exploiting external knowledge sources, has attracted much attention as a potential solution to relieve the common response problem (Li et al., 2015; Zhang et al., 2018a; Ren et al., 2020) in dialogue generation, i.e., 'I don't know' and 'What do you mean?'. Typically, knowledge-grounded conversation is decomposed into two sub-processes (Dinan et al., 2018; Wu et al., 2019): knowledge selection (KS) based on dialogue context, and response generation with reference to the selected knowledge. Therefore, to select relevant knowledge and then incorporate it efficiently, is of great significance for multi-turn knowledge-grounded dialogue generation task.
Although external knowledge sources are widespread in the real-world, in fact, current knowledge-grounded conversations still lack the fine-grained control over knowledge selection and integration with dialogues. Most existing works (Liu et al., 2018; Niu et al., 2019) select knowledge according to the given dialogue context (Lian et al., 2019; Kim et al., 2020). However, the sequential transition characteristic of knowledge (also known as knowledge shift) along multiple sequential conversation turns is neglected. As
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5621-5630 June 6-11, 2021. ©2021 Association for Computational Linguistics
shown in Figure 1, two people are talking about an actor from the knowledge "astrological sign" to another knowledge "blood typology", which is a natural transition in human personality chat (Mayo et al., 1978; Miller, 2014). By nature, taking the knowledge sequential transition characteristic into account is of tremendous benefits to the knowledge grounded conversations.
What's more, knowledge-irrelevant response generation problem also hampers the performance of existing models. This is caused by two reasons. The first reason is that current models often over-fit during training, resulting with incoherent response by referring to unrelated tokens from specific knowledge content in testing phase. To resolve this problem, we propose to calculate the knowledge transition probability among different turns on a high-level representation, i.e., knowledge topic tag. With such concise high-level knowledge representation, our model is not limited to conventional structured knowledge-grounded conversation but can be easily adapted to unstructured knowledge-based conversations. For example, in structured triple data, i.e., {obj, relation, content}, we can utilize the "relation" as the high-level topic tag to model the sequential knowledge transition process in conversations. As shown in Figure 1, the topic migrates from the "astrological sign" tag to the "blood typology" tag, and then moves to the "master-piece". In the unstructured dataset like 'Wizard of Wikipedia' (Dinan et al., 2018), we can utilize topic models, such as LDA (Blei et al., 2003), to obtain the knowledge tag for each turn, and then calculate the sequential transition probability among these tags. Since the number of tag categories is limited, it can be well employed to model the knowledge transition.
Moreover, the second reason is that the models often tends to overlook the selected knowledge, and hence generates knowledge-irrelevant response. To address this problem, we propose pre-training a knowledge-aware response generator, aiming at generating a natural sentence based on a given knowledge, in order to make full use of the limited knowledge data. For example in Figure 1, given the triple  $\{\text{Chao Wu, astrological sign, Aries}\}^{\prime}$ , the knowledge-aware generator is optimized to generate a sentence  $\{\text{Chao Wu's astrological sign is Aries}\}$ . Obviously, the generator should also has the ability to generate  $\{\text{Zhiling Lin's astrological sign is Virgo}\}$  while given  $\{\text{Zhiling Lin, astrologi-}$
cal sign, Virgoj'. Actually, the knowledge-aware response generator learns how to generate a natural sentence based on a relation tag rather than the knowledge content. It is like that one student learns grammar rules rather than specific examples while learning a foreign language. Therefore, even with the limited data, the generator can also generate relevant sentences about given knowledge.
In this paper, we propose a sequential knowledge transition model equipped with a pre-trained knowledge-aware response generator (SKT-KG), which can conduct the high-level knowledge transition in conversation and fully use of the limited knowledge data. Specifically, at first, we pre-train a transformer-based response generator based on the knowledge. And then, we utilize a BiLSTM-CRF (Huang et al., 2015) network to model the knowledge transition process, and select the knowledge tag with maximum score and its corresponding knowledge content. Finally, we feed the dialogue utterances and the selected knowledge content together into the pre-trained knowledge-aware response generator to generate final response.
In our experiments, we use two public knowledge-grounded dialogue datasets to evaluate our proposed models, i.e. structured DuConv corpus and unstructured Wizard of Wikipedia (WoW) corpus. The results show that our SKT-KG model has the ability to produce more diverse and suitable responses than traditional knowledge-grounded models. Besides, we conduct an analysis on knowledge selection, and the results show that the SKT-KG model obtains higher ranking measure than baselines, which indicates that the knowledge selected by our model is reasonable.
Recently, dialogue systems have gained more attention in both research community (Vougiouklis et al., 2016; Liu et al., 2018; Zhou et al., 2018; Shen et al., 2019; Shen and Feng, 2020) and industry (Xu et al., 2020; Zhao et al., 2020), because of its practicality in the real application, such as chatbot and customer services (Chen et al., 2020; Liu et al., 2020; Shen et al., 2021; Zhang et al., 2019; Chen et al., 2018; Zhang et al., 2020). With external knowledge sources, dialogue systems can generate more specific and informative response, which has great potential to resolve the common response problem (Zhang et al., 2018b; Ren et al., 2020). The majority of previous works decomposed the
Figure 2: The architecture of SKT-KG model. The left shows the pre-training phase of knowledge-aware response generator with a flexible self-attention mask mechanism. The bottom-right shows the knowledge transition module, which can select the knowledge sequentially along with conversations. And the top-right shows the fine-tuning phase to generate a response based on the selected knowledge and dialogue utterances in history.
knowledge-grounded dialogue generation task into two sub-problems: knowledge selection and response selection.
In knowledge selection, previous works proposed to use the keyword matching (Ghazvininejad et al., 2018; Liu et al., 2018), information retrieval (Young et al., 2018) and entity diffusion (Liu et al., 2018) methods to detect the relevant knowledge based on dialogue context, and finally feed both dialogue utterances and the selected knowledge into generative models. Specifically, Zhou et al. (2018) proposed to employ the graph attention mechanism to encode the retrieved relevant knowledge graph, which can augment the semantic understanding of dialogue context. Lian et al. (2019) proposed to use the prior and posterior distributions over knowledge to facilitate knowledge selection. Although these work are capable to model the relationship between context and knowledge, they still ignored the knowledge transition characteristic, which is important for knowledge selection.
Human dialogue depends on both local information and global information. Peng et al. (2019) also pointed out that natural language understanding requires a coherent understanding of a series of events or actions, not only what events have appeared, but also what is likely to happen next. Therefore, it is critical to obtain the natural and relevant knowledge for the knowledge-grounded dialogue generation. Sun et al. (2020) proposed to recurrently update the knowledge based on conversation history and progressively incorporate it into the history step-by-step. But they only consider the relationship of history to knowledge. However, these models may also suffer from a knowledge sparse problem, due to the low-resource limitation
in reality (Zhao et al., 2020).
In reality, sufficient knowledge-grounded dialogues data are difficult to obtain. To tackle this practical challenge, Su et al. (2020) proposed to augment the dialogue generation with external non-conversational text, which may also introduce much noise. Li et al. (2020) proposed to pre-train the knowledge encoder with unstructured knowledge and fine-tune the model using the limited knowledge-grounded training examples. In our work, we propose to make full use of our training data and model the high-level knowledge transition process, which can resolve the sparse problem in knowledge-grounded dialogue data.
In this section, we propose a novel sequential knowledge transition model with pre-trained knowledge-aware response generator (SKT-KG), as shown in Figure 2. This model contains three major parts: pre-trained knowledge-aware response generator, sequential knowledge transition, and transformer decoder. Specifically, we firstly pretrain a transformer-based knowledge-aware response generator based on the knowledge and its corresponding natural sentence. And then, we utilize a BiLSTM-CRF (Huang et al., 2015) network to model the knowledge transition process, and select knowledge tag with maximum score and its corresponding knowledge content. Finally, we feed the context utterances and this selected knowledge content into the knowledge-aware response generator to fine-tune it. After fine-tuning, response can be generated by given the selected knowledge tag and corresponding content, and history dialogue utterances.
Figure 3: An example for the input representation. In the pre-training phase, we mask the context utterances part. And for the fine-tune response generation phase, we concatenate the selected knowledge tag, the selected knowledge content and history utterances as input.
Firstly, we introduce the data formulation in our model. Given the history knowledge content  $K = \{k^1,\dots ,k^n\}$ , the history context  $C = \{c^{1},\dots ,c^{n}\}$  and the candidate knowledge set for response  $CK = \{ck^{1},\dots ,ck^{m}\}$ , the goal of our model is to select the most relevant and natural knowledge  $ck^{t}\in CK$  based on the sequential  $K$  and  $C$ , and then generate the response  $Y = \{y_{1},\dots ,y_{|L|}\}$  based on the selected knowledge  $ck^{t}$  and context  $C$ . It is worth noting that each history utterance  $c^i$  is related to a history knowledge  $k^i$  and each knowledge  $k^i$  has a knowledge tag  $t^i\in T$ , which is explicit in the structured knowledge, such as 'relation' in triple knowledge as shown in Figure 1, and implicit in the unstructured knowledge, which is abstracted by topic model, i.e., LDA (Blei et al., 2003). Knowledge tag category  $T = \{t^1,\ldots ,t^N\}$  has  $N$  different knowledge tags.
We utilize the classical transformer blocks as the backbone framework. To generate response  $Y$ , the original input is the concatenation of the selected knowledge tag  $s_t$ , the selected knowledge content  $ck^t$  and the history context utterances  $\{c^1, \dots, c^n\}$ . We use three different embedding methods for the original input: Token embedding, Role embedding and Position embedding, as shown in Figure 3. For knowledge content and dialogue utterances, we utilize the word embedding of each token as the token embedding. For knowledge tag, we map each tag to different categories as the token embedding. A special end-of-knowledge [EOK] token is inserted between knowledge and utterance context to mark the border. Another token end-of-utterance [EOU] is added at the end of each history dialog utterance. Role embeddings are employed to differentiate knowledge content and dialogue utterances. The role embedding  $E_K$  is added for the knowledge content, as well as dialogue utterances are represented by role embedding  $E_C$ . Position embeddings are added according to the token position
in each utterance. Note that for the special token of knowledge tag, its corresponding role and position embeddings are both set to zero.
3.2 Pre-trained Knowledge-aware Response Generator
In our pre-trained knowledge-aware response generator, there are two essential phases we should consider: pre-training phase and fine-tuning response generation phase. In the pre-training phase, given the knowledge tag and knowledge content, our generator focuses on generating the relevant sentence, as shown in the left of Figure 2. And in the fine-tuning response generation phase, given the context utterances, the knowledge tag and the selected knowledge content, our generator focuses on generating the natural and relevant response, as shown in the top-right of Figure 2. To unify the pretraining phase and fine-tuning phase, we propose to utilize the flexible self-attention mask mechanism to distinguish the input representation in this two phases, as shown in Figure 3.
In the pre-training phase, we employ a self-attention mask mechanism to the history dialogue utterances, in order to train the knowledge-aware response generator independently. Given the knowledge content  $k^i \in K$ , its knowledge tag  $t^i \in T$  and its corresponding sentence  $c^i = \{x_1^i, \dots, x_N^i\}$ , we choose the negative log-likelihood loss as our training optimization.
$$
\mathcal {L} _ {p r e} (\theta) = - \sum_ {t = 1} ^ {N} \log p (x _ {t} ^ {i} | x _ {<   t} ^ {i}, k ^ {i}, t ^ {i}; \theta),
$$
where  $\theta$  denotes the model parameters and  $x_{< t}^{i}$  denotes the previously generated words.
3.3 Sequential Knowledge Transition
In this section, we will introduce the knowledge selection process, including the utterance encoding and transition modules. To obtain the next knowledge tag, we should consider both the sequential
knowledge tags and the sequential context utterances, as shown in Figure 4.
Utterance Encoding. To conduct the context sequential representation, we use the standard base BERT model with average pooling (Cer et al., 2018) and the BiLSTM to obtain the context sequential representation. Given the context utterances  $C = \{c^1,\dots ,c^n\}$  where  $c^i$  is composed of a group of words  $\{x_1^i,\ldots ,x_N^i\}$ , we utilize a standard BERT model to encode each utterance  $c^i$  as a sentence embedding  $\mathbf{u}_c^i$ . And then, we apply a BiLSTM on these sentence embedding to obtain the context sequential representation:
$$
\begin{array}{l} \mathbf {H} _ {c} ^ {i} = \mathrm {B E R T} _ {b a s e} \{[ x _ {1} ^ {i}, \dots , x _ {N} ^ {i} ] \}, \\ \mathbf {u} _ {c} ^ {i} = \operatorname {a v e r p o o l} \left(\mathbf {H} _ {c} ^ {i}\right), \\ \mathbf {h} _ {c} ^ {i} = \mathrm {B i L S T M} (\mathbf {u} _ {c} ^ {i}, \mathbf {h} _ {c} ^ {i - 1}). \\ \end{array}
$$
Knowledge Transition. We model the knowledge tag transition process with the assistance of Conditional Random Field (CRF) mechanism (Lafferty et al., 2001). We combine a BiLSTM network and a CRF network to form a BiLSTM-CRF model, as shown in Figure 4. This network can efficiently use past input features via a BiLSTM layer and sentence level tag information via a CRF layer. For each BiLSTM cell, it will output the score of each tag. Given a context representation  $\mathbf{h}_c^i$ , the corresponding tag scores is:
$$
s c o r e _ {i + 1} [ t ^ {i + 1} ] = \mathrm {s o f t m a x} (\mathbf {W} _ {1} \mathbf {h} _ {c} ^ {i} + \mathbf {b} _ {1}),
$$
where  $\mathbf{W}_1$  and  $\mathbf{b}_1$  are the training parameters.  $score_{i+1}[t^{i+1}]$  means the output score of knowledge tag  $t^{i+1}$  at the  $(i+1)$ -th step. CRF layer is capable to model the sequential tag relationship by maximizing a global score  $C(t^1, t^2, \ldots, t^n, \theta)$ . This global score is the concatenation of a transition score  $T[i, j]$  and a matrix of score.  $T[i, j]$  is to model the transition probability from i-th tag to j-th for a pair of consecutive steps. The matrix of score is used to record tag transition path along with the context sentences.
$$
C \left(t ^ {1}, t ^ {2}, \dots t ^ {n}, \theta\right) = \sum_ {i = 1} ^ {n} T \left[ t ^ {i}, t ^ {i + 1} \right] + \sum_ {i = 1} ^ {n} s c o r e _ {i + 1} \left[ t ^ {i - 1} \right]
$$
Therefore, our final selected knowledge tag  $s_t$  should be:
$$
s _ {t} = \operatorname {a r g m a x} (C (t ^ {1}, t ^ {2},.. t ^ {n}, \theta)).
$$
Figure 4: Sequential knowledge transition phase.
Once we get the knowledge tag  $s_t$ , we are able to pick out the corresponding knowledge content  $ck_t$  from the candidate knowledge set  $CK$ . If there are multiple knowledge contents with the same tag  $s_t$ , we will apply a coarse-to-fine knowledge matching module to select out the knowledge content with maximum score as  $ck_t$ .
Coarse-to-fine Knowledge Matching. To select out the final knowledge content from multiple candidates with the same knowledge tag, we adopt BM25 (Robertson and Zaragoza, 2009), as the supporting coarse-to-fine matching model. Given a knowledge content and dialogue context pair  $(ck^i, c)$ , the matching model will output a matching score. We will choose the knowledge content with the highest score as the final knowledge content.
Knowledge Transition Loss. In the training phase, we adopt two level knowledge loss to optimize the sequential selection process. Knowledge tag loss  $\mathcal{L}_{\text{tag}}^{kg}(\theta)$  is a log-likelihood loss to minimized the difference between true tag label and prediction tag label. Knowledge content loss  $\mathcal{L}_{\text{cont}}^{kg}(\theta)$  is a cross-entropy loss to minimize the divergence between true knowledge sentence and prediction one. Therefore, the total knowledge transition loss is defined to be:
$$
\mathcal {L} _ {t r a n s} (\theta) = \mathcal {L} _ {t a g} ^ {k g} (\theta) + \mathcal {L} _ {c o n t} ^ {k g} (\theta).
$$
3.4 Fine-tuning and Response Generation
The flexible self-attention mask mechanism enables our pre-trained generator to consider the dialogue history in the response generation phase. Given the generated knowledge tag  $s_t$  and its corresponding knowledge content  $ck_t$ , and the dialogue contexts  $\{c^1, \dots, c^n\}$ , the fine-tuning procedure can be carried out by the following training optimization to generate response  $y = \{y_1, \dots, y_N\}$ , defined as:
$$
\mathcal {L} _ {N L L} (\theta) = - \sum_ {t = 1} ^ {N} \log p (y _ {t} | y _ {<   t}, c k _ {t}, s _ {t}, c ^ {1}, \dots , c ^ {n}; \theta),
$$
The process is shown in the right of the Figure 2. After fine-tuning phase, response can be generated by given selected knowledge tag, corresponding knowledge content, and history dialogue context.
Dataset. We employ two public knowledge-grounded dialogue benchmarks in our experiments. The structured DuConv dataset consists of 29,000 context-response pairs. The corresponding knowledge pool contains 32 different knowledge tags. We randomly divided the corpus into the training, validation and testing set, containing 25,000, 2,000, and 2,000 pairs respectively. The Wizard of Wikipedia (WoW) dataset is conducted with 201,999 dialogues about diverse topics. We randomly split this corpus as 18,430 dialogues for training, 1,948 dialogues for validation and 1933 dialogues for test. The test set is split into two subsets: test seen and test unseen. Test Seen contains 965 dialogues on the topics overlapped with the training set, while test unseen contains 968 dialogues on the topics never seen before in training and validation set.
Baselines. We compare our SKT-KG model with several state-of-the-art models, including (i) Transformer: a fully self-attention mechanism model (Vaswani et al., 2017), (ii) MemNet: The E2E Transformer with memory mechanism (Dinan et al., 2018), which uses a Transformer memory network for knowledge selection and a Transformer decoder for utterance prediction. (iii) PostKS: Posterior Knowledge Selection (Lian et al., 2019), which uses the posterior knowledge distribution as a pseudo-label for knowledge selection. (iv) SLKS: sequential latent knowledge selection model (Kim et al., 2020), which keeps track of prior and posterior distribution over knowledge and sequentially updated considering contexts in previous turns. we also employ some degraded SKT-KG models to investigate the effect of our proposed pre-trained knowledge-aware response generator mechanisms: SKT is the model without pre-trained knowledge-aware response generator, only using the knowledge transition to select the knowledge and then generate the response with transformer decoder.
Parameters Setting. For WoW, we set the vocabulary size to 30,522, as the default setting in BERT<sup>1</sup>.
Dataset	DuConv
Model	BLEU-1 / 2	Dist-1	Dist-2	Avg.	Ext.	Gre.
Transformer	21.39/11.42	4.67	10.36	51.79	32.07	40.62
MemNet(soft)	22.48/19.95	5.26	13.66	57.28	35.06	41.89
PostKS(fusion)	29.76/21.84	5.84	15.52	55.57	39.54	43.72
SLKS	33.93/24.72	8.40	20.06	59.69	40.31	44.79
SKT	35.79/22.36	9.01	21.47	62.57	46.83	50.11
SKT-KG	37.80/26.31	10.57	23.20	65.37	49.63	57.46
Table 1: Automatic evaluation results on DuConv. The metrics Distinct, Average, Extrema, and Greedy are abbreviated as Dist, Avg., Ext., and Gre., respectively. The best results are highlighted with bold.