SlowGuess/ACL_2020_OCR huggingface dataset:
Xiexiong Lin Weiyu Jian Jianshan He Taifeng Wang Wei Chu
Ant Financial Services Group
{xiexiong.lxx,weiyu.jwy,yebai.hjs}@antfin.com
{taifeng.wang,weichu.cw}@alibaba-inc.com
Knowledge-driven conversation approaches have achieved remarkable research attention recently. However, generating an informative response with multiple relevant knowledge without losing fluency and coherence is still one of the main challenges. To address this issue, this paper proposes a method that uses recurrent knowledge interaction among response decoding steps to incorporate appropriate knowledge. Furthermore, we introduce a knowledge copy mechanism using a knowledge-aware pointer network to copy words from external knowledge according to knowledge attention distribution. Our joint neural conversation model which integrates recurrent Knowledge-Interaction and knowledge Copy (KIC) performs well on generating informative responses. Experiments demonstrate that our model with fewer parameters yields significant improvements over competitive baselines on two datasets Wizard-of-Wikipedia(average Bleu +87%; abs.:0.034) and DuConv(average Bleu +20%; abs.:0.047) with different knowledge formats (textual & structured) and different languages (English & Chinese).
Dialogue systems have attracted much research attention in recent years. Various end-to-end neural generative models based on the sequence-to-sequence framework (Sutskever et al., 2014) have been applied to the open-domain conversation and achieved impressive success in generating fluent dialog responses (Shang et al., 2015; Vinyals and Le, 2015; Serban et al., 2016). However, many neural generative approaches from the last few years confined within utterances and responses, suffering from generating uninformative and inappropriate responses. To make responses more meaningful and expressive, several works on the dialogue sys
tem exploiting external knowledge. Knowledge-driven methods focus on generating more informative and meaningful responses via incorporating structured knowledge consists of triplets (Zhu et al., 2017; Zhou et al., 2018; Young et al., 2018; Liu et al., 2018) or unstructured knowledge like documents (Long et al., 2017; Parthasarathi and Pineau, 2018; Ghazvininejad et al., 2018; Ye et al., 2019). Knowledge-based dialogue generation mainly has two methods: a pipeline way that deals with knowledge selection and generation successively (Lian et al., 2019), and a joint way that integrates knowledge selection into the generation process, for example, several works use Memory Network architectures (Sukhbaatar et al., 2015) to integrate the knowledge selection and generation jointly (Dinan et al., 2018; Dodge et al., 2015; Parthasarathi and Pineau, 2018; Madotto et al., 2018; Ghazvininejad et al., 2018). The pipeline approaches separate knowledge selection from generation, resulting in an insufficient fusion between knowledge and generator. When integrating various knowledge, pipeline approaches lack flexibility. The joint method with the memory module usually uses knowledge information statically. The confidence of knowledge attention decreasing at decoding steps, which has the potential to produce inappropriate collocation of knowledge words. To generate informative dialogue response that integrates various relevant knowledge without losing fluency and coherence, this paper presents an effective knowledge-based neural conversation model that enhances the incorporation between knowledge selection and generation to produce more informative and meaningful responses. Our model integrates the knowledge into the generator by using a recurrent knowledge interaction that dynamically updates the attentions of knowledge selection via decoder state and the updated knowledge attention assists in decoding the next state, which
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 41-52 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
maintains the confidence of knowledge attention during the decoding process, it helps the decoder to fetch the latest knowledge information into the current decoding state. The generated words ameliorate the knowledge selection that refines the next word generation, and such repeated interaction between knowledge and generator is verified to be an effective way to integrate multiple knowledge coherently that to generate an informative and meaningful response when knowledge is fully taken account of.
Although recurrent knowledge interaction better solves the problem of selecting appropriate knowledge for generating the informative response, the preferable integration of knowledge into conversation generation still confronts an issue, i.e., it is more likely that the description words from external knowledge generated for the dialog response have a high probability of being an oov(out-of-vocabulary), which is a common challenge in natural language processing. A neural generative model with pointer networks has been shown to have the ability to handle oov problems (Vinyals et al., 2015; Gu et al., 2016). Very few researches on copyable generative models pay attention to handle external knowledge, while in knowledge-driven conversation, the description words from knowledge are usually an important component of dialog response. Thus, we leverage a knowledge-aware pointer network upon recurrent knowledge interactive decoder, which integrates the Seq2seq model and pointer networks containing two pointers that refer to utterance attention distribution and knowledge attention distribution. We show that generating responses using the knowledge copy resolves the oov and the knowledge incompleteness problems.
In summary, our main contributions are: (i) We propose a recurrent knowledge interaction, which chooses knowledge dynamically among decoding steps, integrating multiple knowledge into the response coherently. (ii) We use a knowledge-aware pointer network to do knowledge copy, which solves oov problem and keeps knowledge integrity, especially for long-text knowledge. (iii) The integration of recurrent knowledge interaction and knowledge copy results in more informative, coherent and fluent responses. (iv) Our comprehensive experiments show that our model is general for different knowledge formats (textual & structured) and different languages (English & Chinese). Furthermore, the results significantly outperform
competitive baselines with fewer model parameters.
2 Model Description
Given a dataset  $D = \{(X_{i},Y_{i},K_{i})\}_{i = 1}^{N}$ , where  $N$  is the size of the dataset, a dialog response  $Y = \{y_{1},y_{2},\ldots ,y_{n}\}$  is produced by the conversation history utterance  $X = \{x_{1},x_{2},\ldots ,x_{m}\}$ , using also the relative knowledge set  $K = \{k_{1},k_{2},\ldots ,k_{s}\}$ . Here,  $m$  and  $n$  are the numbers of tokens in the conversation history  $X$  and response  $Y$  respectively, and  $s$  denotes the size of relevant knowledge candidates collection  $K$ . The relevant knowledge candidates collection  $K$  is assumed to be already provided and the size of candidates set is limited. Each relevant knowledge element in candidate collection could be a passage or a triplet, denoted as  $k = \{\kappa_1,\kappa_2,\dots ,\kappa_l\}$ , where  $l$  is the number of the tokens in the knowledge element. As illustrated in Figure 1, the model KIC proposed in this work is based on an architecture involving an encoder-decoder framework (Sutskever et al., 2014) and a pointer network (Vinyals et al., 2015; See et al., 2017). Our model is comprised of four major components: (i) an LSTM based utterance encoder; (ii) a general knowledge encoder suitable for both structural and documental knowledge; (iii) a recurrent knowledge interactive decoder; (iv) a knowledge-aware pointer network.
2.1 Utterance Encoder
The utterance encoder uses a bi-directional LSTM (Schuster and Paliwal, 1997) to encode the utterance inputs by concatenating all tokens in the dialogue history  $X$  and obtain the bi-directional hidden state of each  $x_{i}$  in utterance, denoted as  $H = \{h_1,h_2,\dots ,h_m\}$ . Combining two-directional hidden states, we have the hidden state  $h_t^*$  as
$$
h _ {t} ^ {*} = \left[ \overrightarrow {L S T M} \left(x _ {t}, h _ {t - 1}\right); \overleftarrow {L S T M} \left(x _ {t}, h _ {t + 1}\right) \right]. \tag {1}
$$
2.2 Knowledge Encoder
As illustrated in Model Description, the knowledge input is a collection of multiple knowledge candidates  $K$ . The relevant knowledge  $k_{i}$  can be a passage or a triplet. This paper provides a universal encoding method for both textual and structured knowledge. The relevant knowledge is represented as a sequence of tokens, which are encoded by a transformer encoder (Vaswani et al., 2017), i.e.,  $z_{t} = \text{Transformer}(\kappa_{t})$ . Static attention  $a_{i}^{k}$  is
Figure 1: The architecture of KIC. Here,  $U_{d}^{t}$  is calculated by decode-input and utterance context vector  $C_{u}^{t}$  at current step,  $C_{k}^{t}$  represents the knowledge context vector resulted from dynamic knowledge attention.  $u_{gen}$  and  $k_{gen}$  are two soft switches that control the copy pointer to utterance attention distribution and knowledge attention distribution, respectively.
used to encode knowledge  $Z = \{z_{1},z_{2},\ldots ,z_{l}\}$  to obtain the overall representation  $K^{rep}$  for the relevant knowledge as
$$
a _ {i} ^ {k} = \operatorname {s o f t m a x} \left(V _ {z} ^ {T} \tanh \left(W _ {z} z _ {i}\right)\right) \tag {2}
$$
$$
K ^ {r e p} = \sum_ {i = 1} ^ {l} a _ {i} ^ {k} z _ {i}, \tag {3}
$$
where  $V_{z}^{T}$  and  $W_{z}$  are learnable parameters. So far we have the knowledge representations for the knowledge candidate collection  $C_k^{rep}$ .
2.3 Recurrent Knowledge Interactive Decoder
The decoder is mainly comprised of a single layer LSTM (Hochreiter and Schmidhuber, 1997) to generate dialogue response incorporating the knowledge representations in collection  $C_k^{rep}$ . As shown in Figure 1, in each step t, the decoder updates its state  $s_{t + 1}$  by utilizing the last decode state  $s_t$ , current decode-input  $U_d^t$  and knowledge context  $C_k^t$ . The current decode-input is computed by the embeddings of the previous word  $e(y_{t})$  and utterance context vector  $C_u^t$ . We provide the procedure as
$$
e _ {i} ^ {t} = v _ {e} ^ {T} \tanh  \left(W _ {h} h _ {i} + W _ {s} ^ {u} s _ {t} + b _ {u a}\right) \tag {4}
$$
$$
u ^ {t} = \operatorname {s o f t m a x} \left(e ^ {t}\right) \tag {5}
$$
$$
C _ {u} ^ {t} = \sum_ {i = 1} ^ {m} u _ {i} ^ {t} h _ {i} \tag {6}
$$
$$
U _ {d} ^ {t} = V _ {u} \left[ e (y _ {t}), C _ {u} ^ {t} \right] + b _ {u}, \tag {7}
$$
where  $V_{u}, b_{u}, v_{e}, W_{h}, W_{s}^{u}, b_{ua}$  are learnable parameters.
Instead of modeling knowledge selection independently, or statically incorporating the representation of knowledge into the generator, this paper proposes an interactive method to exploit knowledge in response generation recurrently. The knowledge attention  $d^{t}$  updates as the decoding proceeds to consistently retrieve the information of the knowledge related to the current decoding step so that it helps decode the next state correctly, which writes as
$$
\theta_ {i} ^ {t} = v _ {k} ^ {T} \tanh  \left(W _ {k} K _ {i} ^ {r e p} + W _ {s} ^ {k} s _ {t} + b _ {a k}\right) \tag {8}
$$
$$
d ^ {t} = \operatorname {s o f t m a x} \left(\theta^ {t}\right) \tag {9}
$$
$$
C _ {k} ^ {t} = \sum_ {i} ^ {s} d _ {i} ^ {t} K _ {i} ^ {\text {r e p}}, \tag {10}
$$
where  $v_{k}, W_{k}, W_{s}^{k}, b_{ak}$  are learnable parameters. A knowledge gate  $g^{t}$  is employed to determine how much knowledge and decode-input is used in the generation, which is defined as
$$
g ^ {t} = \operatorname {s i g m o i d} \left(V _ {g} \left[ U _ {d} ^ {t}, C _ {k} ^ {t} \right] + b _ {g}\right), \tag {11}
$$
where  $V_{g}$  and  $b_{g}$  are learnable parameters. As the steps proceed recurrently, the knowledge gate can dynamically update itself as well. Hence, the decoder updates its state as:
$$
s _ {t + 1} = L S T M \left(s _ {t}, \left(g _ {t} U _ {d} ^ {t} + \left(1 - g ^ {t}\right) C _ {k} ^ {t}\right)\right) \tag {12}
$$
2.4 Knowledge-Aware Pointer Networks
Pointer networks using a copy mechanism are widely used in generative models to deal with oov problem. This paper employs a novel knowledge-aware pointer network. Specifically, we expand the scope of the original pointer networks by exploiting the attention distribution of knowledge representation. Besides, the proposed knowledge-aware pointer network shares extended vocabulary between utterance and knowledge that is beneficial to decode oov words. As two pointers respectively refer to the attention distributions of utterance and knowledge, each word generation is determined by the soft switch of utterance  $u_{gen}$  and the soft switch of knowledge  $k_{gen}$ , which are defined as
$$
u _ {g e n} = \sigma \left(w _ {u c} ^ {T} C _ {u} ^ {t} + w _ {u s} ^ {T} s _ {t} + w _ {u} ^ {T} U _ {d} ^ {t} + b _ {u p}\right) \tag {13}
$$
$$
k _ {g e n} = \sigma (w _ {k c} ^ {T} C _ {k} ^ {t} + w _ {k s} ^ {T} s _ {t} + w _ {g} ^ {T} U _ {g} ^ {t} + b _ {k p}), (1 4)
$$
where  $w_{uc}^{T}, w_{us}^{T}, w_{u}^{T}, b_{up}, w_{kc}^{T}, w_{ks}^{T}, w_{g}^{T}, b_{kp}$  are learnable parameters. The  $U_{g}^{t}$  here is defined as
$$
U _ {g} ^ {t} = V _ {g} \left[ e \left(y _ {t}\right), C _ {k} ^ {t} \right] + b _ {g}, \tag {15}
$$
where  $V_{g},b_{g}$  are learnable parameters. Therefore, the final probability of the vocabulary  $w$  is
$$
P _ {f i n a l} (w) = \left(\lambda u _ {g e n} + \mu k _ {g e n}\right) P _ {v} (w) +
$$
$$
\lambda \left(1 - u _ {g e n}\right) \sum_ {i} u _ {i} ^ {t} + \mu \left(1 - k _ {g e n}\right) \sum_ {i} d _ {i} ^ {t}, \tag {16}
$$
$$
P _ {v} (w) = \operatorname {s o f t m a x} \left(V _ {2} \left(V _ {1} \left[ s _ {t}, C _ {u} ^ {t}, C _ {k} ^ {t} \right] + b _ {1}\right) + b _ {2}\right), \tag {17}
$$
where  $V_{1}, V_{2}, b_{1}, b_{2}, \lambda$  and  $\mu$  are learnable parameters under constrain  $\lambda + \mu = 1$ . Note that if the word is an oov word and does not appear in utterance,  $P_{v}(w)$  is zero and we copy words from knowledge instead of dialogue history.
We use two recently released datasets Wizard-of-Wikipedia and DuConv, whose knowledge formats are sentences and triplets respectively.
Wizard-of-Wikipedia (Dinan et al., 2018): an open-domain chit-chat dataset between agent wizard and apprentice. Wizard is a knowledge expert who can access any information retrieval system recalling paragraphs from Wikipedia relevant to the dialogue, which unobserved by the agent apprentice who plays a role as a curious
learner. The dataset contains 22311 dialogues with 201999 turns, 166787/17715/17497 used for train-valid/test, and the test set is split into two subsets, Test Seen(8715) and Test Unseen(8782). Test Seen has 533 overlapping topics with the training set; Test Unseen contains 58 topics never seen before in train or validation. We do not use the ground-truth knowledge information provided in this dataset because the ability of knowledge selection during generation is a crucial part of our model.
DuConv (Wu et al., 2019b): a proactive conversation dataset with 29858 dialogs and 270399 utterances. The model mainly plays the role of a leading player assigned with an explicit goal, a knowledge path comprised of two topics, and is provided with knowledge related to these two topics. The knowledge in this dataset is a format of the triplet(subject, property, object), which totally contains about 144k entities and 45 properties.
3.2 Comparison Approaches
We implement our model both on datasets Wizard-of-Wikipedia and DuConv, and compare our approach with a variety of recently competitive baselines in these datasets, respectively. In Wizard-of-Wikipedia, we compare the approaches as follows:
- Seq2Seq: an attention-based Seq2Seq without access to external knowledge which is widely used in open-domain dialogue. (Vinyals and Le, 2015)
- MemNet(hard/soft): a knowledge grounded generation model, where knowledge candidates are selected with semantic similarity(hard); / knowledge candidates are stored into the memory units for generation (soft). (Ghazvininejad et al., 2018)
- PostKS(concat/fusion): a hard knowledge grounded model with a GRU decoder where knowledge is concatenated (concat); / a soft model use HGFU to incorporated knowledges with a GRU decoder.(Lian et al., 2019)
- KIC: Our joint neural conversation model named knowledge-aware pointer networks and recurrent knowledge interaction hybrid generator.
While in dataset DuConv, a Chinese dialogue dataset with structured knowledge, we compare to the baselines referred in (Wu et al., 2019b)
that consists of retrieval-based models as well as generation-based models.
We adopt an automatic evaluation with several common metrics proposed by (Wu et al., 2019b; Lian et al., 2019) and use their available automatic evaluation tool to calculate the experimental results to keep the same standards. Metrics include Bleu1/2/3, F1, DISTINCT1/2 automatically measure the fluency, coherence, relevance, diversity, etc. Metric F1 evaluates the performance at the character level, which mainly uses in Chinese dataset DuConv. Our method incorporates generation with knowledge via soft fusion that does not select knowledge explicitly, therefore we just measure the results of the whole dialog while not evaluate performances of knowledge selection independently. Besides, we provide 3 annotators to evaluate the results on a human level. The annotators evaluate the quality of dialog response generated on fluency, informativeness, and coherence. The score ranges from 0 to 2 to reflect the fluency, informativeness, and coherence of results from bad to good. For example, of coherence, score 2 means the response with good coherence without illogical expression and continues the dialogue history reasonably; score 1 means the result is acceptable but with a slight flaw; score 0 means the statement of result illogically or the result improper to the dialog context.
3.4 Implement Detail
We implement our model over Tensorflow framework(Abadi et al., 2016). And our implementation of point networks is inspired by the public code provided by (See et al., 2017). The utterance sequence concatenates the tokens of dialog history and separated knowledge. And the utterance encoder has a single-layer bidirectional LSTM structure with 256 hidden states while the response decoder has a single-layer unidirectional LSTM structure with the same dimensional hidden states. And the knowledge encoder has a 2-layer transformer structure. We use a vocabulary of  $50\mathrm{k}$  words with 128 dimensional random initialized embeddings instead of using pre-trained word embeddings. We train our model using Adagrad (Duchi et al., 2011) optimizer with a mini-batch size of 128 and learning rate 0.1 at most  $130\mathrm{k}$  iterations(70k iterations on Wizard-of-Wikipedia) on a GPU-P100 machine. The overall parameters are about 44 mil
lion and the model size is about 175MB, which decreases about  $38\%$  against the overall best baseline PostKS parameters:71 million, model size: 285M)
3.5 Results and Analysis
3.5.1 Automatic Evaluation
As the experimental results on Wizard-of-Wikipedia with automatic evaluation summarized in Table 1, our approach outperforms all competitive baseline referred to recently working (Lian et al., 2019), and achieves significant improvements over most of the automatic metrics both on Seen and Unseen Test sets. The Bleu-1 enhances slightly in Test Seen while improving obviously in Test Unseen. Bleu-2 and Bleu-3 both yield considerable increments not only in Test Seen but in Test Unseen as well, for example, the Bleu-3 improves about  $126\%$  (absolute improvement: 0.043) in Test Seen and about  $234\%$  (absolute improvement: 0.047) in Test Unseen. The superior performance on metrics Bleu means the dialog response generated by model KIC is closer to the ground-truth response and with preferable fluency. As all
Figure 2: Bleu improvements on Wizard-of-Wikipedia.
Bleu metrics are shown in Figure 2, we can find that the improvement of result increasing with the augment of Bleu's grams, which means the dialog response produced via model KIC is more in line with the real distribution of ground-truth response in the phrase level, and the better improvement on higher gram's Bleu reflects the model have preferable readability and fluency. Generally, the ground-truth responses in datasets make up with the expressions from knowledge which conduces to the informativeness of response. As the recurrent knowledge interaction module in model KIC provides a mechanism to interact with the knowledge when decoding words of dialog response step by step. Moreover, the knowledge-aware pointer
Models	Test Seen	Test Unseen
Bleu-1/2/3	DISTINCT-1/2	Bleu-1/2/3	DISTINCT-1/2
Seq2Seq	0.169/0.066/0.032	0.036/0.112	0.150/0.054/0.026	0.020/0.063
MemNet(hard)	0.159/0.062/0.029	0.043/0.138	0.142/0.042/0.015	0.029/0.088
MemNet(soft)	0.168/0.067/0.034	0.037/0.115	0.148/0.048/0.023	0.026/0.081
PostKS(concat)	0.167/0.066/0.032	0.056/0.209	0.144/0.043/0.016	0.040/0.151
PostKS(fusion)	0.172/0.069/0.034	0.056/0.213	0.147/0.046/0.021	0.040/0.156
KIC(ours)	0.173/0.105/0.077	0.138/0.363	0.165/0.095/0.068	0.072/0.174
Table 1: Automatic Evaluation on Wizard-of-Wikipedia. The results of baselines are taken from (Lian et al., 2019).
Models	F1	Bleu-1	Bleu-2	DISTINCT-1	DISTINCT-2	ppl
norm retrieval	34.73	0.291	0.156	0.118	0.373	-
norm Seq2Seq	39.94	0.283	0.186	0.093	0.222	10.96
generation w/o klg.	28.52	0.29	0.154	0.032	0.075	20.3
generation w/ klg.	36.21	0.32	0.169	0.049	0.144	27.3
norm generation	41.84	0.347	0.198	0.057	0.155	24.3
KIC(ours)	44.61	0.377	0.262	0.123	0.308	10.36
network in KIC allows copying words from the expression of knowledge while decoding. Therefore, the dialog response generated by KIC contains relatively complete phrases of knowledge that as knowledge-informativeness as the ground-truth response. In addition, the improvements of metrics Bleu increase from Test Seen to Test Unseen, that is to say, the KIC with an advantage in case of unseen knowledge guided dialogue, which shows that our model is superior to address the dialogues with topics never seen before in train or validation. Besides, the metrics DISTINCT also achieves impressive results and prior than most of the baselines, about average  $77\%$  over the most competitive method PostKS. The metrics DISTINCT mainly reflects the diversity of generated words, whose improvements indicating that the dialogue response produced by KIC could present more information. In addition to experiments on Wizard-of-Wikipedia, we also conduct experiments on DuConv to further verify the effectiveness of our model on structured knowledge incorporated conversation. As the dataset DuConv released most recently that we compare our model to the baselines mentioned in the (Wu et al., 2019b) which are first applied to the DuConv including both retrieval-based and generation-based methods. The results presented in Table 2 show that our model obtains the highest results in most of the metrics with obvious improvement over re
trieval and generation methods. Concretely, the F1, average Bleu, average DISTINCT, and ppl are over the best results of baseline norm generation about  $6.6\%$ ,  $20.5\%$ ,  $115.8\%$ , and  $5.5\%$ . Similar to Wizard-of-Wikipedia, the impressive augments of metrics demonstrate that the model has the capacity of producing appropriate responses with fluency, coherence, and diversity.
Table 2: Automatic Evaluation on DuConv. Here, klg. denotes knowledge and norm stands for normalization on entities with entity types, norm generation is the PostKS in Table1. The results of baselines are taken from (Wu et al., 2019b).
Metrics	Wizard-of-Wikipedia	DuConv
Fluency	1.90	1.97
Coherence	1.50	1.64
Informativeness	1.12	1.62
Table 3: Human Evaluation for the results of KIC.
3.5.2 Human Evaluation