Low-Resource Knowledge-Grounded Dialogue Generation
Authors: Xueliang Zhao
Wei Wu
Chongyang Tao
Can Xu
Dongyan Zhao
Rui Yan
Venue:  (2020)

Abstract: Responding with knowledge has been recognized as an important capability for an intelligent conversational agent. Yet knowledge-grounded dialogues, as training data for learning such a response generation model, are difficult to obtain. Motivated by the challenge in practice, we consider knowledge-grounded dialogue generation under a natural assumption that only limited training examples are available. In such a low-resource setting, we devise a disentangled response decoder in order to isolate parameters that depend on knowledge-grounded dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of ungrounded dialogues and unstructured documents, while the remaining small parameters can be well fitted using the limited training examples. Evaluation results on two benchmarks indicate that with only $1/8$ training data, our model can achieve the state-of-the-art performance and generalize well on out-of-domain knowledge. 

Published as a conference paper at ICLR 2020
LOW-RESOURCE KNOWLEDGE-GROUND DIALOGUE GENERATION
Xueliang Zhao! 1.2 Wei Wu², Chongyang Tao', Can Xu², Dongyan Zhao Rui Yan 12,4. Wangxuan Institute of Computer Technology. Peking University, Beijing, China
2 "Center for Data Science, AAIS, Peking University, Beijing, China 3 Microsoft Corporation, Beijing, China
"Beijing Academy of Artificial Intelligence (BAAI,, Beijing, China {x1. zhao, chongyangtas, zhaody, ruiyanj @pku. (wuwei, caxu @mi wrwsoft.coommmmmmooo..
ruw.adjje...yy edu. on
ABSTRACT
Responding with knowledge has been recognized as important capability for an intelligent conversational agent. Yet knowledge-groune grounded dialogues, as training data for learning such response generation model, are difficult to obtain. Mo- tivated by the challenge practice, we consider knowledge-gon laige--gronnedd dialogue generation under natural assumption that only limited training examples avail- able. In such a low-resource setting, we devise a disentangled response decoder in order to isolate parameters that depend entire generation model. By this means, the major part of the model can be learned from large number of ungrounded dialogues and unstructured documents, while the remaining small parameters can be well fitted using the limited training exam- ples. Evaluation results on two benchmarks indicate that with only 1/8 training data, our model can achieve the state-of state-of-the-an the-art performance and generalize well on oft-oann knowledge.
thowledge-grounden
dialogues from the
INTRODUCTION
Open domain dialogue systems, due to the applications on social chatbots such as Microsoft XiaoIce (Shum et al. 2018) and virtual assistants such as Amazon Alexa (Ram al. 2018), have drawn increasing attention from the research community of natural language processing and artificial intelligence. Thanks to the advances in neural sequence modeling (Vaswani et al.) 2017 Sutskever et al. 2014) and machine learning techniques (Li et al. 2017 2016). such systems now are able to reply with plausible responses regarding to conversation history, and thus allow an agent to have a natural conversation with humans. On the other hand, when people attempt to dive into a specific topic, they may clearly realize the gap between the conversation with a state-of-the-an system and the conversation with humans, the system is only able to awkwardly catch up with the conversation. owing to the lack of knowledge of the subject.
We consider grounding open domain dialogue generation with knowledge which is assumed to be unstructured documents. While documents are abundant on the Web, is difficult to obtain large scale dialogues that naturally grounded on the documents for learning of neural generation model. To overcome the challenge, some recent work (Zhou et al., 20188; Dinan et al. (011)) resorts crowd-sourcing and builds benchmarks with the source of Wikipedia. On the one hand, the datasets pave the way to the recent research on Enowledge-groundd ct al. 2019; Lian al. 2019: Li et al. 2019); on the other hand, we argue that there still a long way to go for application of the existing models in real scenarios. since (1) the models, especially those achieve ffate-of-the-ane performance via sophisticated neural architectures, just overfit to the small training data (e.g.. ~ 18 8k dialogues). An evidence is that when they are applied to documents out of the domain of the training data, their performance drops dramatically, as will be seen in our experiments; and (2) it is difficult to collect enough training data for new domain new language, as human effort is expensive.
response generation/selection (Zhao
Corresponding author; Rui Yan (ruiyan @ suiiyan@pu.eduu.cm)

---

Published as a conference paper at ICLR 2020
As step towards application of chowledge-groundee dialogue generation in real-world systems, explore how to learn a model with as few sowwlege-groundee dialogues as possible, yet the model achieves ofte-of-theearr performance and generalizes well out-of-domain documents. The key idea is to make parameters that rely on disentangling the response decoder, and thus we can learn the major part of the generation model from ungrounded dialogues and plain text that are much easier to acquire. Specifically, the encoder of the generation model consists of two independent components with one for encoding the context and the other for representing the knowledge. The decoder is decomposed into conditionally independent components including a language model, a context processor, and a knowledge processor, and the three components are coordinated by decoding manager that dynamically determines which component is activated for response prediction. The language model predicts the next word of a response based on the prior sub-sequence, and the context processor ensures coherence of the dialogue by attending over the conversation history. Both components, along with the context encoder, are independent with the extra knowledge, and thus can be pre-trained using the ungrounded dialogues. The knowledge encoder has nothing to do with dialogues, and thus can be pre-trained with the plain text. The knowledge processor is responsible for grounding response generation on the document. This part, together with the decoding manager, depends on the wwwwlleeegroondeee dialogues, but the parameters are small in size, and estimation of these parameters just requires few training examples depending on specific domains or tasks, By fixing the pre-traiee pre-trained parameters, we can adapt the model to a new domain with only a little cost.
dialogues small and independent by
We pre-train the language model, the context processor, and the context encoder with a clean version of Reddit data (Dziri al. 2018), pre-train the knowledge encoder using a Wikipedia dump available on ParlAI, and compare our model with baselines that hold satte-of-th---e performance on two benchmarks including the Wizard of Wikipedia (Wizard) (Dinan et al.) 2019) and CMU Document Grounded Conversations (CMU DoG) (Zhou et al., 201 8b). Evaluation results indicate that (1) to achieve the sate-offthhearr performance, our model only needs 1/8 training data (~~ 2.3k dialogues on Wizard and ~ 0.4k dialogues on CMU DoG); (2) Wizard, the model significantly outperforms the baseline models on out-of-dommin documents even though the baselines have leveraged all training data, while our model is only learned with 1 1/16 training data; and (3) the model performs comparably well on in-domain and out-of-domall documents in a low-resource setting.
Contributions in this work are three-folde (1) exploration of knowledge-grounee
dialogue generation under a low-resource setting; (2) proposal of pre-training the knowledge waodgeggounded dialogue generation
model with a disentangled decoder using ungrounded dialogues and documents; and (3) empirical verification of the effectiveness of the model on two benchmarks.
2
APPROACH
We elaborate our approach to learning a response generation model with knowledge- -roondddd dialogues, ungrounded dialogues, and plain text.
2.1 PROBLEM FORMALIZATION
Suppose that we have a dataset Dş = {(U ,r ) where Vi E (1,...,,7) n). D is a document that serves as the background of the dialogue (U$, r,)), = ( .. i.n.) is the context of the dialogue with ui, the j-th utterance, and r is the response regarding to US and D. In addition
S
to Ds, we further assume that there are Dp {D,), and Dc (10) (...g9)) ( 1 document and (Uf: ,r9) a context-response pair, Vi f1, (1....NN N) and Vj E (1....,MT) N > n and M > n. The goal is learn generation model P(r D: 6) (0 denotes the parameters of the model) with D = (Ds U Dp U ] Thus, given new document D with the associated dialogue context U, one can generate a response r following P(r[U, 0)
MM with D."
Our idea is inspired by the observation on the nature of open domain dialogues: despite the fact that a dialogue is based on a document D, words and utterances in the dialogue are not always related to D (e.g., reply just echoing the previous turn), even for the turns from the interlocutor who has access to D, as demonstrated by the examples in (Dinan et al. 2019 Zhou et al. 2018b). Therefore, we postulate that formation of response could be decomposed into three uncorrelated actions: (1) selecting a word according to what has generated to make the sentence linguistically valid (corresponding to language model); (2) selecting word according to the context to make the
2

---

Published as a conference paper at ICLR 2020
P(w
Kroeeedgg Encoder
Hidden States
Decoding
Manager
Contec Procesoor
Coocel cage Pcc0s0
Language Model
Vecabular
Disrrb
MLP
MLP
Layer
Deegonue Decocer
Contect j hooder
<590T>
mad
the
ropal
Figure I: Architecture of the generation model.
dialogue coherent (corresponding to context processor); and (3) selecting word according the extra knowledge to ground the dialogue (corresponding to knowledge processor). The three actions can be independently learned, which becomes the key aiding the small Ds with the large Dp and Dc.
2.2 GENERATION MODEL
Figure illustrates the architecture of the model. The model made up of context encoder, knowledge encoder, decoder, and decoding manager. The major difference lies in the decoding phase which simulates the aforementioned actions by decomposing the decoder into a language model, context processor, and knowledge processor. The three components independent conditioned on the hidden states of the decoder, and are coordinated by the manager.
2.2. ENCODERS
Given dialogue context U (Mi,..., ut), the context encoder concatenates {ui) as (wf...., w." "...., w with w." the i-th word in the sequence, and then exploits a recurrent neural network with gated recurrent units (GRUs) (Chung et al. 2014) to transform the word sequence into sequence of hidden vectors given by
hT h h GRUa, (ei,...., e" e7,),
where e is the embedding of w. initialized with GloVe (Pennington et al. 2014 (h serve as the input of the context processor decoding.
In the meanwhile, given document D (d1...., di...., dm) with de the i-th sentence, the knowledge encoder represents d, as sequence of hidden vectors through a bidirectional GRU (Cho
al. 2014):
Claa), where et, is the embedding of the j-th word in d; initialized using GloVe. (h J=ij=1 are fed to the knowledge processor to ground response prediction on D.
h hi N BiGRUs, (ei,,...., e
iimyj-ta
Different from Transformer Memory Network (Dinan et al. 2019), our model does not perform knowledge selection in the encoding phase (c.g., via attention over (he [=lle= (==,,j=ld) )) but leaves it to
the decoding phase. This could remove the dependency between context encoding and knowledge encoding, and facilitate estimate 6g and 0 with Dy and De respectively.
2.2.2 DISENTANGLED DECODER
The decoder maintains hidden sequence {s:}l Let eç_ be the embedding of the word predicted step 1, then 8; is defined by
8: GRU,, (c, 8t-1);
(3)
where 8o h Based {s.)/= the three components defined follows:
3

---

Published as a conference paper at ICLR 2020
Language Model. The language model predicts word based 8t. For words that do not need the context and the document (e.g.. function words), employing the language model may enhance decoding speed without loss of accuracy. Formally, the generation probability is defined by
P(w]) wf--1) MLPo, (st).
Context Processor. The context processor predicts word by attending over (h#) could be either fetched from the vocabulary or copied from the context U. Let ct be the context vector step t, then oj can be formulated as
Ji-- The word
c = /n or,ch"
(5)
where Ctr.i = exp(er,))E, exp(er,)) denotes the attention distribution and et, = go, (St, ht))
=
JT tanhh Whh + W, + b). The generation probability defined by
P( w(a-)) Pgen Prcaab ( (w: [U, w(a-1) + (1 Pgan
Σ
(6)
In Equation (6). the first term models the correspondence between a context and a response, and for- mulated as (voacb (w=]] w( MLP ([s(;e]]) The second term models the copy mechanism, and Pgen MLPa, e, (Ic); 8G; 1--1)) E [0, 1] a trade-off between the two terms.
The knowledge processor goes through the document D by hierarchical attention mechanism, and predicts word a similar way as Equation (6). Formally, let { )m and
Knowledge Processor.
{ /= VEm be the sentence-level attention distribution and the word-level attention distributions respectively at step t, then Vi {1, .I. m) and Vj E {I,....,aa la), B8, and (t,) calculated
B = exp(ge, ( (8t, /Z.5 PE. = exp(go, (8t, ///))//..
where Z, and zw are normalization factors, and h. represents the average pooling of (h la knowledge vector of that is analogous to cº is then defined by
j=1- A
Bi,h
Finally, the generation probability is formulated as
P, w(a-1) = Pgon Pvocab (w/( D, w + (1 (1-pgen)
Σ
Brass
where Be,i.j = Bi, BYag why is the j-th word of d,, Pvocab D, w' = MLP Ba..e..e ([8,; c. and
P'gan = MLPa ([c); st; 1]1 acts as a trade-off between the common term and the copy term.
2.2.3 DECODING MANAGER
The three components are controlled by the decoding manager with one picked up at each step of response prediction. Then, the probability predict word wa can be formulated
P( D, wf. [P(w] P(W]
P(w] wi (, P(w ID, wf: me.
(10)
In training, to handle the discrete and undifferentiable process, we employ the Gumbel trick (Jang al. 2016) and define म. as
n gumbel softmax(J R311
(11)
where f.) MLP9, (.), gwwbel.sooftteeeeoeeeeeoeooooooe ssoftmaxx))) denotes the Gumbel-Softmax function (Jang et al. 2016), and is the temperature (hyperparametter). n. approaches one-hot vector when - 0.
We start from a high temperature and gradually reduce In test, we discretize ग. as a one-hot vector according to the distribution in Equation (11)
4

---

Published as a conference paper at ICLR 2020
2.3 LEARNING DETAILS
Let us denote (Oot, loc, Jod} as the parameters of word embedding in response prediction correspond- ing to the language model, the context processor, and the knowledge processor respectively. For simplicity, we let foc = ood = 0o- Then {fe; OJ; 0g; Qui gg ) (including parameters of the context encoder, parameters of the hidden states of the decoder, and parameters of the context processor) are estimated with maximum likelihood estimation (MLE) on De = {UU (r)))
To estimate O2 (i.e., parameters of the language model) and Oot, we construct corpus PLM {uf MAMMMM j= with u/ a response or an utterance from context in De, and then learn the parameters with MLE on Dr.M with 0g fixed.
Inspired by Peters et al. (018)), we estimate 0g (i.e., parameters of the knowledge encoder) using a bidirectional language model by minimizing the following loss function on Dp:
(log((waluaa-)) + (og
(12)
The remaining parameters {00;; Prr: 0g: (i.e., parameters of the knowledge processor and param- eters of the decoding manager) are learned with MLE on Ds with all other parameters fixed. Note that parameters of word embedding in the encoders supposed to be included in O, and On.
Remarks. We focus on secument-groundee dialogue generation in this work, but the approach proposed actually provides a recipe for general solution to low-resource knowledge-gree grouuddd dialogue generation in which the knowledge could be structured knowledge base, images, or videos. To do that, one only needs to modify the knowledge encoder and the knowledge processor to make them compatible with the specific type of knowledge, and pre-train the knowledge encoder, if possible, on single-modal modal knowledge data.
3 EXPERIMENTS
We test the proposed model on Wizard of Wikipedia (Wizard) published in Dinan et al. (2019) and CMU Document Grounded Conversations (CMU DoG) published in /hhou al. (2018)).
3.1 DATASETS AND EVALUATION METRICS
Both Wizard and CMU DoG consist of open domain dialogues grounded on wiki articles, and the dialogues are collected from frowd-workens on Amazon Mechanical Turk. In Wizard, the articles cover a wide range of topics (totally 1, 1,56 such as bowling, Gouda cheese, and Arnold Schwarzenegger, etc. Each conversation happens between wizard who has access to knowledge about a specific topic and an apprentice who is just eager to learn from the wizard about the topic. On average, each wizard turn is associated with 60.8 sentences retrieved from the wiki articles and each sentence contains 30.7 words. The data is split data owner. The test set split into two subsets: Test Seen and Test Unseen. Test Seen contains new dialogues with topics appearing in the training set, while topies in Test Unseen never appear in the training set and the validation set, and thus the data allow us to examine the generalization ability of models. The task is to generate response for each wizard turn based on the dialogue history and the retrieved knowledge. As pre-processing, for each wizard turn in the Ruining/validation//e sets, the latest 128 words in the dialogue history are kept as context. The pre-processing strictly follows the procedure in Dinan et al. (2019). and is conducted with the code published on ParlA
training set, validation set, and test set by the
Different from Wizard, CMU_DoC focuses on movie domain (although covering various genres). In addition to wizard & apprentice, the data also contain dialogues between two workers who know the document and try to discuss the content in depth. Fach document consists of 4 sections and these sections are shown to the workers one by one every 3 turns (the first section lasts 6 turns due to initial greetings). On average, each section contains 8.22 sentences and 27.86 words per sentence. The data has been divided into training set, validation set, and test set by the data owner. The task is
ward_of_ __ wi pidiaa
5

---

Published as a conference paper at ICLR 2020
Metrics
PPL
F1
BLEU-1
BLEU-2
BLEU-3
BLEU-4
Average
Extrema
Greedy
Models
TMN (Diaan al. 2019
66.5
15.9
0.184
0.073
0,033
0.017
0.844
0.427
0.658
ITDD a. 2019
17.8
16.2
0.158
0.071
0,040
0.025
0.841
0.425
0.654
FULL DATA
23.0
18.0
0.218
0,115
0,075
0.055
0.835
0.434
0.658
1/2 DATA
25.3
17.5
0.217
0.113
0.073
0.053
0.833
0.431
0.657
1/4 DATA
29.2
16.9
0.212
0.105
0,064
0.044
0.833
0.429
0.658
1/8 DATA
33.5
16.3
0.206
0.098
0.059
0.039
0.832
0.425
0.658
1/16 DATA
38.6
15.7
0.197
0.091
0.052
0.033
0.834
0.428
0.655
Table 1: Evaluation results on Test Seen of Wizard.
to generate response for each turn from worker who has access to the document based on the dialogue history and the associated section as knowledge. Similar to Wizard, the latest 128 words in the dialogue history are kept as a context. More details of the datasets can be found in Appendix [A]
We choose Reddit Conversation Corpus cleaned by Dziri et al. (2018) as Dc. The data contain 15, 120, 136 context-response pairs for training and 830, 777 context-response pairs for validation. On average, each context consists of 3.5 utterances. We use the Wikipedia dump published on ParlA as Dp. The training set and the validation set contain 5, 233, 799 articles and 52, 867 articles respectively with the first paragraph kept for learning. Articles that appear in Wizard and CMU DoG are removed beforehand. For both Wizard and CMU DoG, the vocabulary is made up of top 60, 000 most frequent words appearing in D3 U Dp U Dc with other words regarded as (unk).
Following the common practice in evaluating open domain dialogue generation, we choose perplex- ity (PPL) of the ground-truth response, BLEU (Papineni al. 20022, and BOW Embedding (Liu jet al. 2016) as metrics. Besides, we also follow Dinan et al. (2119) and employ unigram Fl metrie. BLEU and Embedding- based metrics are computed with an NLG evaluation open source available at lated with the code published httpt:ttt////
wwt.:///w.../aa//a//aaaaaaaaaaaa eval, and unigram F1 is calcu- com/ h www.gttttuu..oo
http://atterraa/aa /wtp://oreeeeeeee /ctt///etreeeeeeeeeeeeeeeee cs pyy Besides quantitative evaluation, we also recruit human annotators do qualitative analysis response quality, which is presented in Appendix C
3.2 BASELINES
The following models are selected as baselines:
Transformer Memory Network (TMN). The model proposed by Dinan al. (2019) along with the release of the Wizard data. It is built upon transformer architecture with an external memory hosting the knowledge. We implement the model using the code shared at /ettps://githin
com/
facebookresearchh/ Par 1AI htt.l///mmmtee/roooeeeeeoe
wi zwazzrdddccc of wtw.aaaddf_wiiiieeeeeeeeeeeeeeee
Incremental Transformer with Deliberation Decoder (ITDD). A transformer- model pub- lished very recently on ACL'19 (Li et al. 2019). The encoder incrementally represents multi-turn dialogues and knowledge, and the decoder conducts response decoding in two passes similar to the deliberation network in machine translation. We implement the model using the code shared at https: / thub. com/ 1 wiizekng//TDD
Note that to make the comparison fair, we employ the end-to-end version of TMN without the knowledge regularization in learning. After all, one can include ground-truth signals on knowledge selection in both our model and TMN, and improve the two in the same way, although such signals are not available in most scenarios (e.g., in CMU_DoG).
3.3 EVALUATION RESULTS
To simulate a low-resource scenario, we start from using the full training data as Ds. and gradually reduce the number of training examples by halving the training set. Note that baseline models are learned with the full training sets. Table I and Table evaluation results on Test Seen and Test Unseen of Wizard respectively, and Table reeports evaluation results CMU MoG. Through pre- training 95% parameters with the ungrounded dialogues and the plain text and fixing the parameters afterwards, model holds the state-of-the-arr performance in terms of most metrics on all test sets even when the training sets have been cut to 1/8, and has stable performance on Test Unseen with
attps: http::///ccchaa.ccc////sssssss/s

---

Published as a conference paper at ICLR 2020
Metries
Models
PPL
F1
BLEU-1
BLEU-2
BLEU-3
BLEU-4
Average
Extrema
Greedy
TMN (Dinan et al. 2019
103.6
14.3
0.168
0.057
0.022
0.009
0.839
0,408
0.645
ITDD La ar. 2019
44.8
11.4
0.134
0.047
0.021
0.011
0.826
0,364
0.624
FULL DATA
25.6
16.5
0.207
0.101
0.062
0.043
0.828
0.422
0.628
1/2 DATA
27.7
16.7
0.208
0.103
0.064
0.045
0.827
0.421
0.647
1/4 DATA
32.4
16.2
0.205
0.098
0.050
0.041
0.828
0,423
0.650
1/8 DATA
35.8
16.0
0.201
0.093
0.054
0.035
0.831
0.419
0.653
1/16 DATA
41.0
15.3
0.191
0.087
0.050
0.032
0.832
0.424
0.652
Table 2: Evaluation results Test Unseen of Wizard.
Metrics
Models
PPL
F1
BLEU-1
BLEU-2
BLEU-3
BLEU-
Average
Extrema
Greedy
TMN (Dinan al. 2019
75.2
0.115
0.040
0.016
0.007
0.789
0.399
0.615
(TD al. 2019
26.0
10.4
0.095
0.036
0.017
0.009
0.748
0.390
0.587
FULL DATA
54.4
10.7
0.150
0.057
0.025
0.012
0.809
0.413
0.6333
1/2 DATA
57.0
10.4
0.142
0.052
0.022
0.010
0.808
0.414
0.635
1/4 DATA
61.7
10.5
0.131
0.046
0.019
0.009
0.781
0.402
0.613
1/8 DATA
67.6
10.2
0.121
0.044
0,019
0.009
0.787
0.407
0.622
Table 3: Evaluation results on CMULDoG
Ouneace fnwaaee
Our model (lekkn)
Our model lon
Oa model Aen
Our model bet riedgo
17
Ou model (nboerdgs)
1..7
12
16
16
15
FJLL 1/2
1/4
1/16 RULL 112
144
1710 FJLL
1/2
1/4
171
*ULL
12
16
1/16
Data
(d)
Figure 2: Performance of variants of the proposed model on Wizard. (a) Comparison of parameter fine-tuning and parameter fixing on Test Seen. (b) Comparison of parameter fine tin-ttnnng and parameter fixing on Test Unseen. (c) Results of pre-training ablation on Test Seen. (d) Results of pre-training ablation on Test Unseen.
respect different training sizes. Particularly, the model achieves more significant improvement over the baselines on Test Unseen, and when the training set shrinks, the performance gap on Test Seen and Test Unseen becomes marginal. The results show good generalization ability of the proposed model on out-of-domain knowledge. ITDD achieves low PPL on both Test Seen and CMU DoG, which may stem from overfitting by the two-pass decoder. As an evidence, the model is just comparable with TMN on most metrics except PPL on Test Seen and MMD,, and is worse than our model on Test Unseen even in terms of PPL.
3.4 DISCUSSIONS
In addition to the performance of the model under low-resource settings, are also curious about QI: what if we fine-tune the pre-trained parameters, rather than fixing them, with the training data of the knowledge-grounder dialogues, given that pre-training - fine-tuning has become the fashion in NLP research and engineering? Q2: can we somehow leverage the ungrounded dialogues and the plain text learning of TMN, and in this case, will there be any change in the comparison with our model? and
Q3: what is the impact of pre-training to different components of the proposed model?
Answer to Q1: Figure 2(a) and Figure 2(b) compare our models with fine-tuned parameters and fixed parameters on Test Seen and Test Unseen respectively. Basically, when there are enough training data (e.g., > 1/2), fine-tuning can further improve the model on both in-domain and out-of-domamn knowledge. On the other hand, when the training size is small, which is the assumption of the paper, fine-tuning may cause overfitting and lead to performance drop on the test sets. Test Unseen is more vulnerable than Test Seen, and the smaller the training size is, the bigger the gap is between the model with fixed parameters and the model with fine-tuned parameters. Therefore, in a low-resource setting (e.g., less than 5k training dialogues), it is better to fix the pre-trained parameters and only estimate the remaining 5% parameters with the training data.
Answer to Q2: Normally, is not trivial to lear entangled architecture like TMN with ungrounded dialogues and plain text. However, to make the comparison even more fair, we first pre-train a transformer-base based encoder-decoder with the Reddit data. The encoder is fixed and used for TMN, and
7

---

Published as a conference paper at ICLR 2020
Our mosel sunser??
20.0
0.30
Cormadel (ISSne))
0.25
60
17.5
0.20
8 40
0
0.15
12.5
0:10
10.0
0.05
FULL
1/2
1/4
1/8
1/16
FULL
1/4
0.00
1/4
Training
Training Data
(a) PPI.
(b) F1
(c) BLEU-1
Figure 3: Comparison with pre-trained TMN on Wizard.
the parameters of the decoder is used to initialize the parameters of the decoder of TMN. Then, we pre- train the document representation TMN with the Wikipedia dump. Finally, the knowledge attention in encoding and the decoder are learned (fine-tuned) with the training data of knowledge-groundde dialogues, as knowledge and dialogue contexts are entangled in the two modules. Figure compprres the pre-trained TMN with our model. Even though we have tried our best to make TMN use De and Dp, it is still much worse than our model. The results indicate the importance of disentangling to leveraging ungrounded dialogues and plain text for low-resource knowledeg- grounded dialogue generation.
Answer to Q3: Figure 2(c) and Figure 2(d) show the results of ablation study in terms of pre-training -Im means that Og and au are estimated using Dy together with (0z; 0.r; fgr: 0z). Similarly, -context and -knowledge mean that pre-training is removed from {0.: 0a; 0g; o:: 0y: 0.) and 0g respectively. We can conclude that (1) pre-training is crucial to low-resource knowledge-gree -rouuddeg dialogue generation, since removing any component from pre-training causes performance drop when training data is small; and (2) in terms of impact to performance, Im> contextokkowwleee. on Test Seen, while tnowledge>lm>>oom on Test Unseen.
4 RELATED WORK
Research on end-to-end open domain dialogue generation is encouraged by the success of neural sequeence-t-ssqquee
models on machine translation (Sutskever et al. 2014). On top of the basic
architecture (Shang et al. 2015 Vinyals & Le| (015)), various extensions have been made to tackle the safe response problem (Li et al.
2015 Xing et al. 2017: Zhao et al. 2017 Song et al. 2018 8 Tao et al. 2018 Qiu et al. 2019); to model dialogue history for multi-turn conversation (Serban et al. 2016
2017;; and to learn with advanced machine learning techniques (Li al. 2016 2017). Very recently,
grounding response generation on a specific type of knowledge, such as triples from knowledge
base (Zhou al. 2018a), documents (Ghazvininejad et al., [018 /hhoo et al. (019)). personas (Zhang et al. 2018), and images (Mostafazadeh et al. [[01]]] 7). has emerged as a new fashion in the research of open domain dialogue systems. This work aligns with the trend by considering document-grounded dialogue generation. Our model is built upon state-of-the-a neural generation techniques such as
attention (Bahdanau et al. 2015 Yang et al. 2016) and copying (See et al. 2017 Raghu et al. 2019 Yavuz et al. 2019), but is unique in that components are pre- tre-trained from various sources, thanks to the disentangled design. Thus, rather than testing new architectures on the benchmarks, our main contribution lies in investigation of wewwledge-groundee dialogue generation under a low-resource setting with pre-training techniques, which roots in the requirement from practice.
The idea of "disentangling response decoding" is inspired by the similar research in representation learning that aims to seek a representation axis aligning with the generative factors of data (Bengio
et al. 2013). State-of-the
models are built within the framework of variational auto-encoding
(Kingma & Welling 2013) either under an unsupervised assumption (Higgins et al., 2017 Kim &
Mnih 2018: Chen et al. 2016 2018) or aided by few labels (Narayanaswamy et al. [017]] Locatello ct al. 2019)) In this work, we borrow the concept of "disentangling but apply it to the structure of the decoder of response generation model. The result is few independent components that allow
asynchronous parameter estimation. The work is also encouraged by the recent breakthrough on
pre-training for NLP tasks (Peters al. 2018 Devlin al. 2018 Yang et al. 2019 Liu et al. 2019 Song et al. 2019). We take advantage of disentanglement, and employ pre-training techniques to tackle the low- low-resource challenge the task of knowledge- -roondddd dialogue generation.
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSIONS
We study knowledge gegllldg--gruunddd dialogue generation under low-rewn low-resource setting. To overcome the challenge from insufficient training data, we propose decomposing the response decoder into independent components in which most parameters do not rely on the training data any more and can be estimated from large scale ungrounded dialogues and unstructured documents. Evaluation results on two benchmarks indicate that our model achieves the state-of-the-an performance with only 1/8 training data, and exhibits a good generalization ability on out-of-domain knowledge.
ACKNOWLEDGMENTS
We would like to thank the reviewers for their constructive comments. This work was supported by the National Key Research and Development Program of China (No. 2017YFC0800001) the National Science Foundation of China (NSFC No. 61876 196 and NSFC No. 61672058). Rui Yan was sponsored as the young fellow of Beijing Academy of Artificial Intelligence (BAAT). Rui Yan is the corresponding author.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-182) 2013.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1006.0788 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 181.004805 2018.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of
wikipedia: Knowledge-powered conversational agents. In ICLR, 2019.
Nouha Dziri, Ehsan Kamalloo, Kory W Mathewson, and Osmar Zaiane. Augmenting neural response generation with context-aware topical attention. arXiv preprint 1xXiv:181.01106 2018.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76 (5):378, 1971.
Marjan Ghazvininejad, Chris Brockett, Ming-Wer Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. knowledge-grounded neural conversation model. In AAAI, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5):6, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:161 1.01 144, 2016.

---

Published as a conference paper at ICLR 2020
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2654-2663. 2663, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114,22 2013.
arXiv preprint
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. NAACL, pp. 110-119, 2015.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforce- ment learning for dialogue generation. In EMNLP, pp. 1192-1202, 2016.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky, Adversarial learning for neural dialogue generation. In EMNLP, pp. 2157-2169, 2017.
Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, and Jie Zhou. Incremental transformer with deliberation decoder for document grounded conversations. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 12-21, 2019.
Rongzhong Lian, Min Xie, Fan Wang, Jinhua Peng, and Hua Wu. Learning select knowledge for response generation in dialog systems. arXiv preprint IVXiv:1902.0499
2019. 2019.
Chia-Wer Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2122-2132, 2016.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 19XX::1907.11922 2019.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem, Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258, 2019.
Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios Spithourakis, and Lucy Vanderwende. Image-grounde conversations: Multimodal context for natural question and response generation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 462 472, 2017.
Siddharth Narayanaswamy, Brooks Paige, Wil-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, pp. 5925-5935, 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting association for computational linguistics, pp. 311 311-318. Association for Computational Linguistics, 2002.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, pp. 2227-2237, 2018.
Lisong Qiu, Juntao Li, Wei Bi, Dongyan Zhao, and Rui Yan. Are training samples correlated?
learning to generate dialogue responses with multiple references. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3826 3835-3835, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v (Pt//P19111 1372. URL
https: // www. a web. www.worg/bnthhheoo htw.ettooooooyyyyooooooooo 1ttp:///11111111 9-1372
10

---

Published as a conference paper at ICLR 2020
Dinesh Raghu, Nikhil Gupta, et al. Disentangling language and knowledge in task dialogs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1239-1255, 2019.
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng. Ashish Nagar, et al. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv: 003604, 2018.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, 2017.
Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI, volume 16, pp. 9776-3784, 2016.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI, pp. 3295-3301, 2017.
Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In ACL, pp. 1577-1586, 2015.
Heung- Yeung Shum, Xiaodong He, and Di Li. From cliza to xiaoice: Challenges and opportunities with social chatbots. Frontiers of & EE, 19(1): 2018.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence sequence pre-training for language generation. In International Conference on Machine Learning, pp. 5926-5936, 2019.
Yiping Song, Rui Yan, Cheng-Te Li, Jian- Yun Nie, Ming Zhang, and Dongyan Zhao. An ensemble of retrieval-based and generation-based human-computer conversation systems. In IJCAI, pp. 4382-4388, 2018.
Ily Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 14222, 2014.
Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, and Rui Yan. Get the point of my utterance! learning towards effective responses with multi-head attention mechanism. In IJCAI, pp. 4424, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 19998-008,. 2017.
Oriol Vinyals and Quoe Le. A neural conversational model. arXiv preprint arXiv: 2015.
Chen Xing, Wei Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei- Ying Ma. Topic aware neural response generation. AAAI pp- 3351-3351 2017.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.082577 2019.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 1489, 2016.
Semih Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tuu Deepcopy: Grounded response generation with hierarchical pointer networks. arXiv preprint arXiv: 1908. 10731, 2019.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: have a dog, do you have pets too? arXiv preprint arXiv: 2018.
11

---

Published as a conference paper at ICLR 2020
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. ACL, pp. 654 664, 2017.
Xueliang Zhao, Chongyang Tao, Wei Wu, Can Xu, Dongyan Zhao, and Rui Yan. A document- grounded matching network for response selection in retrieval-based chatbots. In IJCAI, pp. 5443-5449, 2019.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Commonsense knowledge aware conversation generation with graph attention. IJCAI, pp. 4623-4629, 2018a.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. A dataset for document grounded conver- sations. arXiv preprint arXiv: /t////09900755 2018b.
12

---

Published as a conference paper at ICLR 2020
APPENDIX
A DETAILS OF DATASETS
Table reports the statistics of the Wizard data and the DOG data.
Wizard of Wikipedia
CMU
Train
Valid
Test Seen
Test Unseen
Train
Valid
Test
Number of Ulterances
166,78 787
17,715
8,715
8,782
74,717
4,993
13,646
Number of Conversations
18,430
1.948
965
968
3,373
229
619
Number of Dopics/Docuneeet
1,247
599
533
58
30
30
30
Average Turns per Dialogue
9.0
9.1
9.0
9.1
22.2
21.8
22.0
Table 4: Statistics of the two datasets.
B MORE IMPLEMENTATION DETAILS
set the size of word embedding 300, the hidden size of the context encoder, the knowledge encoder, and the decoder as 1024. The context encoder and the decoder have 3 layers respectively. The go, and go., are similarity functions which contain two single-layer feed-forward networks (FFNs) of size 512 with tanh non-linearity. The MLPo, MLP and MLPe, are two-layer FFNs of size 1024 and 300 respectively. The MLP8,, MLPs, and MLPs, single-layer FFNs. All models are learned with Adam (Kingma & Ba 2013) optimizer with 31 = 0.9, 32 = 0.999, and an initial learning rate = 5e 4. We increase the learning rate linearly for the first 5000 training steps and decrease thereafter proportionally of the step number. We set the initial temperature, the minimum temperature, and the anneal rate of gumbel.soffmm fmwx 1.0, 0.6, and 4e 5 respectively. In training, we choose 64 the size of mini-batches, and add dropout to 90., and MLPo,,. a... but do not see much difference. Early stopping on validation is adopted as a regularization strategy. We employ beam search in response decoding with a beam size 5. We add weak supervision to guide the training of the decoding manager where the words that belong to modal verb are forced to be classified as language model.
In both Wizard and
the inverse square root
C HUMAN EVALUATION
Unseen
Metrics
Models
Fluency
Context Coherence
Knowledge Relevance
Kappa
Fluency
Context Coherence
Knowledge Relevance
Kappa
TMN Dinan et
2019
1.26
0.51
0.47
0.60
1.40
0.35
0.46
0.68
ITDD Li atat. 2019
1.69
1.18
1.16
0.70
1.72
0.73
0.71
0.69
1/4 DAIA
1.77
1.54
1.17
0.58
1.75
1.26
1.18
0.57
DATA
1.68
1.44
1.13
0.60
1.73
1.21
0.57
Table 5: Human evaluation results on Wizard.
The goal of human study is to get more insights quality of responses generated by different models from human annotators. To this end, we randomly sample 300 examples from Test Seen and Test Unseen respectively, and recruit 3 well educated native speakers as the annotators. Comparison conducted among TMN, ITDD, our model (with 1/4 training data), and our model (with 1/8 training data). On each test set, for each of the 300 examples, annotator is provided with context, the ground-trui -ruth knowledge, and responses provided by the models under evaluation (the top one response in beam search). Responses each annotator judges the responses from three aspects including fluency, context coherence, and knowledge relevance, and assigns score from {0, 1,2] each of the response on each aspect, in which 0 means bad, 1 means fair, and 2 means good. Each response receives 3 scores on each aspect, and agreement among the annotators are calculated with Fleiss' kappa (Fleiss) 1971). Table 5 shows the average scores on the three aspects. Overall, the proposed model achieves the state-of- the-art art performance in terms of all the three aspects on both Test Seen and Test Unseen when only
pooled and randomly shuffled to hide their sources. Then,
""can" "would", "could", "will", "should", "may"
13

---

Published as a conference paper at ICLR 2020
1/8 training examples are left. All kappa values exceed close to 0.6, indicating substantial agreement among the annotators. The results are consistent with those reported in Table I and Table 2] Our model estimates the decoder with abundant extra resources, and ITDD exploits two-pass decoder. Therefore, both of the two models can provide grammatical and fluent responses, no matter the background knowledge is within the domain of training or out of the domain of training. On the other hand, with the 15M Reddit data in learning of the context processor, our model can make the dialogues more coherent than the baselines, although there little drop on Test Unseen compared to Test Seen. Since the model only obtains limited guidance from training in terms of the connection between the knowledge and the dialogues, how to make the responses relevant to the knowledge is still challenging, although our model has done better job than the baselines.
(Knowledge)
+ the lineup, when first signed to geffen records in 1986, consisted of vocalist and rose, lead guitarist slash, rhythm guttarist iny stradinn, bassist duff ricken, and drummmr steven adler.
guns n' rrses has released six studio alburss, accumulattig soles of more than 200 million records worldwide, including 45 million the united states, making them the yest best-selling artist all time.
guns n debut album, appetite destruction "(190y) reached number the billboard' year after release, strongth of sweet child o' mine". the band single american hard rack band from los angeles, california, formed in 1985-
number the "bllbaard " hot
guns n roses, often abbreviated as ger, the album has sold approximately go million copies workswide, including s8 million units in the united states, making it the best-selling debut album of time the us, as well as the eleventh sest-seling album in the united states.
success debut was followed by the eight-song album gin lies" ! (g888) which reached number two on the billboard 200. usee your illusion "(1992] Idebuted number two and number one billboard "200
twin albums use your illusion"" respectively and have sold combined 35 million coples workdwide, including 14 million units the united states.
[Context]
have you heard the band guns 'roses? they my favorite band ever.
[Groundrruth]
sure the hand was formed iin 198; in los angeles, califoriaa what your favorite album?
[TMN]
they are great band.
[ITD)]
have n't, but do know that they are the most popular band in the us
Our Mudel
me too Ihey 're an american rock band from los angeles and has released six studio albums
Generated by Language model
Generated Context Processof
Generated Knowledge Processor
Table 6: A case from Test Unseen of Wizard.
Table 6 shows an example from Test Unseen, from which we can see that the response from our model (with 1/8 training data) not only smoothly catches the context, but also expands the topic with proper pieces of knowledge (highlighted in red). On the other hand, responses from the baselines just reply to the context but lose the connection with the knowledge, as we have analyzed with the results in Table Moreover, we also visualize the sources of words in the response with colors. Basically, words that have weak or no correlation with the context and the knowledge are generated by the language model, words that connect with the context but have nothing to do with the knowledge are generated by the context processor, and words that are copied from the knowledge are generated by the knowledge processor.
D COMPARISON WITH MASS
20
mode (scon)
0.30
Our maseen
Our model (seon) Our model (unseen) MASS oveen)
18
MASS Iseen)
MASS (unsewn))
MASS ounseen)
0.20
16
0.15
14
0.10
12
0.05
10
FULL
1/2
1/4
1/8
1/16
FULL
1/2
1/4
1/8
1/16
Training Data
Training Data
(a) Fl
(b) BLEU-1
Figure 4: Comparison with MASS on Wizard.
14

---

Published as a conference paper at ICLR 2020
We compare model with MASS (Song al. 2019). pre-training technique that achieves state-of-the-art performance on several language generation tasks such as machine translation, text summarization, and conversational response generation. MASS firstly pre-trains an encoder-decoder architecture with large-scale monolingual data from WMT News Crawl datasets by reconstructing fragment of sentence from the remaining, and then fine-tunes the architecture downstream language generation tasks. We use the code and the model published at httpp::/ /etg://gthtbb com/ mi crosofot / /MASS The original model for souuenncet-ssequence generation. To adapt it to dialogue generation task, we concatenate the knowledge sentences and conversational history as a long context as the input of the encoder.
the
Figure44 shows shows the evaluation results. Note that we do not include PPL as metric like in Figure 3 since MASS performs generation with sub- words, and thus not comparable with our model on PPL.. On both Test Seen and Test Unseen, our model consistently outperforms MASS over all training sizes. The reason might that "mask then predict", which is basically the pre-training strategy exploited by MASS, is not an effective way to leverage the text data for knowledge- grounded dialogue generation, since the task needs more complicated operations such as deep copying. Another reason might be that MASS is designed for the sequence- peeeenc--t--seeeee co-sequence generation task and isn' 't compatible with the knowledge-groune response generation task which has extra knowledge input.
knowledge-rrounded
E ABLATION OVER COMPONENTS
Our model (fxed)
18
Our model (fixn)
Our model (-Im)
Our model (-Im)
Our model (.contaxt)
Our model (-context)
17
Our model (knowedde)
Our model ( knowledge)
17
W
16
16
15
15
14
14
FULL
1/2
1/4
1/8
1/16
FULL
1/2
1/4
1/8
1/16
Training Data
Training Data
(a)
(b)
Figure 5: Ablation study over the three components of the decoder. (a) Results Test Seen. (b)
Results on Test Unseen.
We conduct ablation study over the language model, the context processor, and the knowledge processor by completely dropping any of them from the decoding manager (in both training and test). Figure ((a) and Figure 5(b) report the results Test Seen and Test Unseen respectively. First of all, all the three components are useful, since removing any of them in general will cause performance drop. Second, in terms of importance, knowledge processor>
processor>languaee model. The explanation is that (1) part of the function of the language model may be covered by the context processor and the knowledge processor after it is removed") since both the context processor and the knowledge processor also contain language models, although in the full model, the language model generates 17% words in the responses Test Seen and Test Unseen; (2) the context processor is important (generating 27% words), but not always, since a large proportion of responses in the Wizard data highly depend on the knowledge (e.g., the examples shown in (Dinan et al. )019);; (3) the knowledge processor (generating 56% words) the most important component due to the nature of the Wizard data. The results also remind us that perhaps we can try pre-training the language model with larger and more heterogeneous data such Common Crawl in the future.
15

---

Published as a conference paper at ICLR 2020
Our model (with pre (re-inaining)
Our model (with pre training)
Our model (Wlo pre-training) TMN
18
Our model (wo pre-training)
TMN
16 [
14
14
12
10
FULL
1/2
1/4
1/8
1/16
FULL
1/2
1/4
1/8
1/16
Training Data
Training Data
Figure 6: Comparison with the proposed model without pre-training. (a) Results on Test Seen. (b)
Results on Test Unseen
F COMPARISON WITH NON-PRETRAINING
Figure 6(a) and Figure 6(b) compare two versions of our model on Test Seen and Test Unseen respectively. One version is the model pre-trained using ungrounded dialogues and documents, and the other version is the one trained with knowledge- grounded -egougggg dialogues (i.e., pre-training is performed). Besides, we also include the results of TMN to get more insights. We can see that when there enough training data (e.g., full data), our model without pre-training outperforms both TMN and the pre-trained version on Test Seen. This is because the attention and copying operations can well capture the correlation among the knowledge, the contexts, and the responses in the training data, while in the pre-trained version, only a small proportion of the model can benefit from the training data, and a large proportion may suffer from the gap between the knowledge -ooggg-grrddedd dialogues collected from crowd-sourcing and the ungrounded dialogues and documents collected from the Web. However, when the training size shrinks, which is basically the problem we study in the paper, the performance of our model without pre-training drops dramatically, and becomes even worse than that of TMN on Test Seen when the training size is no more than 1/8. This is because when training model is more prone overfit the small training set than TMN, and thus
data is not enough, results in bad generalization ability. In the low-resource setting, pre-training, especially with the disentangled decoder if consider the results in Figure 3 is an effective approach obtaining good generalization ability on test data. The conclusions are further verified by the comparison on Test Unseen, where non-pre-training worse than pre-training over all training sizes, and non-pre-training quickly drops below TMN when the training data is halved. On Test Unseen, with 1/8 training data, the pre-traiee tre-trained model achieves the performance of the model learned from the full training data without pre-training.
S<Part of is because the language model is pre-trained with monolingual Reddit data, which is different from the context processor and the knowledge processor.
16


Reviews:
[{"review_id": "ryed7RM8tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an encoder-decoder model architecture for knowledge-grounded dialogue generation in a low-resource setting. The model includes two GRU-based encoders which represents knowledge and dialogue context independently from each other. The decoder also has the following three independent components: language model, context processor, and document reader, each of which works as an individual response decoder. While the language model considers the decoder's hidden state only, the context processor and the document reader apply the copy mechanism from dialogue context and knowledge, respectively. Then, the decoding manager generates the final distributions by aggregating the component outputs with Gumbel softmax.\n\nThe main contribution of this work is the disentangled architecture where the following three types of internal components are trained separately on different data from each other. Firstly, context encoder, context processor and language model are trained on un-grounded dialogues with no knowledge. Secondly, knowledge encoder can be built purely on knowledge sources in parallel with the dialogue-based components. Finally, document reader and decoding manager require knowledge-grounded dialogues as training data.\n\nThe experimental results show that the proposed architecture help to resolve the low resource problems of knowledge-grounded dialogue generation. The model achieved state-of-the-art performances on both Wizard of Wikipedia and CMU DoG even with only 1/8 of the training data.\n\nOverall, this is a well written paper with reasonable ideas supported by strong empirical evidences.\n\nPlease find below some questions and minor comments:\n- Would it be possible to evaluate the performance of the model with no grounded dialogues? It would be interesting to see how well the model works only with the un-grounded Reddit and Wikipedia data.\n- Have you thought about plugging in other language models trained on larger amount of texts such as Common Crawl? It seems possible, since the language model itself has no dependency to the context or the knowledge.\n- What do you think of generalizing the architecture for general dialogue generation problems with no knowledge grounding? This disentangled decoder may be applied to the language model trained on the larger amount of texts and the context model on the smaller amount of dialogues.\n- I suggest to provide some actual outputs from the models to show the advantage of the proposed method qualitatively.\n- Could you possibly make the subfigures of Figure 2 and 3 larger? They are now too tiny to see the details.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1em_DSCYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel framework for training a knowledge-grounded dialogue model. They decouple the three main elements of such a model - a language model, a context processor, and a document reader - and as such can pretrain each component separately. They achieve state-of-the-art results on two benchmark datasets, and can additionally obtain near-state-of-the-art results while training on a fraction of the task data.\n\nOverall I think it’s a strong paper with a good set of experiments, baselines, and considers multiple datasets.\n\nDisentangling the model elements is a clever way to allow for more robust pre-training, and indeed yields favorable results. The authors show that just the pre-training aspect is not the root cause of their boost in performance, as a pre-trained baseline model fails to replicate their best results. The contribution is broadly applicable to other areas in which data collation is more difficult; the authors additionally do a good job of pointing out that their knowledge encoder is not limited to text but can also use other knowledge grounding including images, videos, or a knowledge-base. Finally, the authors detail thorough ablation studies for their models.\n\nOne major thing missing from their ablation in Figure 2 is a setting when *no* pretraining is used. That would be much more comparable to the setting used for TMN, since that had *no* pretraining available to it. Alternative, adding pretraining to the baselines would be another good way to do this, which would help disentangle how much the architecture is helpful over the pretraining.\n\nAlthough the authors point out that a major advantage of their architecture is that we can separate the pretraining for each of the components, I would also be interested to see how they find the model doing if a single source of pretraining is used. I.e., only reddit pretrained weights for all 3 components, or only wikipedia etc. I definitely don’t think that holds back this paper, just think it would provide some evidence of the value gained utilizing the disparate sources.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BkgyhNioqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies knowledge-grounded dialogue response generation in the low-resource setting. More precisely, it proposes a disentangled decoder consisting of three components: language model, context processor, and document reader. Disentangled decoder architecture provides a flexibility to train (or pre-train) different components on different data, making it convenient for low-resource setting. Overall, it is a sound idea for low-resource setting with generally positive experimental results, but limited in novelty in terms of the proposed architecture (similar to [1*, 2*]) and disentangling language and knowledge idea (similar to [3*]), lacks comparison with baselines for low-resource setting, and lacks discussion/reference of a few closely related works.\n\nHere are some of my questions and concerns for the paper:\n\nGreat to have some ablations in terms of pre-training to see its effect on different components. However, it would be quite useful to also have ablations over components by completely dropping a component (like LM) while training decoding manager. It would also be useful to see some statistics/discussion on the effect of different components in the actual generation of responses. It might also be useful to include qualitative examples of the generated responses annotated with predictions of different components for each generated word.\n\nIn Figure-2 (c) and (d), some ablation results are reported for when pre-training is removed for each of the three components independently. It is interesting, though, to see that removing pre-training does not hurt (might even improve) the performance (esp. for Test Unseen) much for FULL training data case. Is there a particular reason for this observation? Also, it makes me curious how the proposed model would perform without pre-training any of the components. Would it already outperform the baselines discussed in the paper? If so, are these baselines strong enough (SOTA or close to SOTA) to help draw a meaningful conclusion from comparison with them? For example, how would fine-tuning a pre-trained MASS  [4*] perform and compare as a baseline? Can authors comment on this?\n\nThe proposed approach is very similar in architecture to [1*, 2*, 3*], which are not discussed/referenced in the paper. Except for pre-training, the only difference from [2] is that copying and generation distributions are softly combined into separate distribution each, independently. Inducing a single output distribution is done instead by deciding which source to use by an MLP layer on decoder state as in Eq. 11. So, I think the authors need to better isolate what the core contribution of this paper is: disentangled decoder or pre-training strategy? If it is the proposed disentangled decoder architecture, then authors should compare with similar architectures [2, 3] in the low-resource setting by initializing encoder and decoder from pre-trained weights on the same Reddit corpus. If it is the pre-training strategy, then it should be compared with various pre-training strategies for sequence generation (e.g., [4*]) proposed recently. For example, it would be useful to include a comparison with fine-tuning a pre-trained MASS  [4*] with the same amount of WoW training data (changing from 1/16 to 1/1).\n\nPresentation of the paper can be improved by 1) changing the name of “document reader” (maybe to “knowledge processor” similar to context) as it essentially attends on the document rather than reading, 2) using abstraction in the technical section to help simplify notation and make it more interpretable.\n\n\nREFERENCES:\n[1*] Get To The Point: Summarization with Pointer-Generator Networks, See et al.\n[2*] DeepCopy: Grounded Response Generation with Hierarchical Pointer Networks, Yavuz et al.\n[3*] Disentangling Language and Knowledge in Task-Oriented Dialogs, Raghu et al.\n[4*] MASS: Masked Sequence to Sequence Pre-training for Language Generation, Song et al.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of knowledge-grounded dialogue generation with low resources. The authors propose to disentangle the model into three components that can be trained on separate data, and achieve SOTA on three datasets.

The reviewers agree that this is a well-written paper with a good idea, and strong empirical results, and I happily recommend acceptance.
Decision: Accept (Poster)