Augmenting Transformers with KNN-Based Composite Memory
Authors: Angela Fan
Claire Gardent
Chloe Braud
Antoine Bordes
Venue:  (2020)
Keywords: knn
memory-augmented networks
language generation
dialogue

Abstract: Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialogue modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge from Wikipedia, images, and human-written dialogue utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.

Under review as a conference paper at ICLR 2020
AUGMENTING TRANSFORMERS WITH KNN-BASED COMPOSITE MEMORY
Anonymous authors Paper under double-blind review
ABSTRACT
Various machine learning tasks can benefit from access to external information of different modalities, such text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting generative Transformer neural networks with KNN-based In- formation Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules generative dialogue modeling, challenging task where information must be flexibly retrieved and in- corporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge from Wikipedia, images, and written dialogue utterances, and show that leveraging this re- trieved information improves model performance, measured by automatic and hu- man evaluation.
INTRODUCTION
Machine learning solutions various tasks, such as game-playing or dialogue, are often dependent on external information. This information can take multi-modal forms, including structured knowl- edge bases, free text, and images, and also comes overwhelmingly large quantities. press- ing challenge is to create models that can identify which specific elements of multiple information sources are relevant, and incorporate them into standard architectures on each task.
Previous work has explored incorporating large external memories into neural network layers (We- ston et al. 2014, Sukhhaatar al.] 2015 2019 Lample et al., 2019). Many existing approaches Tocus on using attention over the memory slots, which is computationally intensive and becomes less effective as the the size of the memory grows. In this work, we propose representing multi- fixed encodings and using K Nearest Neighbors search to
plc sources of external information fetch relevant information. KNN search is computationally efficient and scalable, and libraries like faiss (Johnson et al.] 2019) allow KNN to be easily used GPUs and integrated into neural networks. As the external memories are kept fixed, they do not require any training to learn the memories along with the model. We can thus scale more easily larger memories by learning only the KNN-based read operation to identify relevant information from the memory.
Our core contribution proposes an efficient, KNN-based Information Fetching (KIF) module that can access relevant external knowledge, combine knowledge from different sources, and integrate this information into standard sequence sequence architectures. We apply these flexible modules to two dialogue datasets, challenging tasks where generative models can leverage external information to write coherent, on-topic responses. We show that relevant information can be identified from hundreds of thousands of candidates in a multi-modal, performance of generative dialogue models. On both datasets, we achieve state of the art results compared to generative models and match the quality of retrieval models.
setting improve
2 RELATED WORK
Incorporating External Knowledge into Neural Networks, Augmenting neural networks with memory, or longer term components that can be accessed with read and write operations, has been

---

Under review as a conference paper at ICLR 2020
INPUT
ENCODER M
ENCODED INPUT
Jaft)) OR Sadts)
KNN
EXTERNAL INFORMATION
DONGAT
OUTPUT
ENCODED INFORMATION ON
DECODER
KIF modules (orange) fetch relevant information from multi-modal external knowledge sources and incorporate it in standard neural architectures.
Figure 1:
explored in various proposed architectures. For example, Memory Networks (Weston et al. 2014
Sukhbaatar et al. 2015 2019) introduce attention mechanisms over large external memories. Neural cache models ( (Grave et al., 2016) simplify these to access previous memories with a dot product.
Previous work has studied how read and write into these memory architectures (Rae al. [016
Graves et al. 2014 Joulin & Mikolov 2015). Another line of research has focused on computational
scalability for larger external memories. For example, Chandar et al. (0166) propose hierarchical
memory network rather than a flat one Rae et al. (2016) learn sparse operations to read and write. Lample al. (2019) focus on learning memories of up to one million slots and how to efficiently access the slots using product keys. Beyond explicit memory representations, it may be possible to store information implicitly during training time by memorizing common patterns present text (Petroni et al., 2019). We focus on learning to fetch relevant information from multiple explicit external modal knowledge sources and integrate them into one network.
Work has also focused on computationally efficient softmax operations (Mnih & Hinton 2009. Grave et al., 2017; Chen et al.] 2015).. Many approximate softmaxes use KNN-like operations to form clusters, and the softmax operation is constrained by the slow calculation of the exponential. Our use KNN benefits from efficient and scalable libraries such as faiss and nms amslib.
Generative Dialogue. We develop a general architecture for incorporating external information and apply it to the case of generative dialogue models. Previous work in dialogue has leveraged knowledge as necessary information to accomplish the task. For example, airline and restaurant booking tasks often use API calls to access information about reservation times and availability (Bordes et al. 2016). In contrast, our work focuses on how to incorporate unstructured knowledge, such free text found the web. Previous work has employed architectures that attend over the available knowledge and identify relevant pieces of information, which scales poorly with large quantities of information (Dinan al.) 2018 Qin et al. 2019 Lian al. 2019). ). In this work, we replace the use of attention over external information with the output of our KNN module.
On the modeling side, work has explored both generative (Serban et al. 2016a and retrieval based models (Zhang et al., 2018), which identify the best utterance from the training set to return as the dialogue response. This often leverages self-attention or cross-attention mechanisms (Humeau et al. 2019). Further work has explored hybrid models, for example using the output of retrieval model as input for a generative model (Dinan al.) 22018 Weston et al. 2018). We extend these approaches by augmenting generative models with retrieval-like operations based on KNN search, allowing dialogue models to flexibly incorporate various sources of external knowledge.
3 KNN-BASED INFORMATION FETCHING MODULES
Broadly, the KNN-based Information Fetching (KIF) module assumes a model M can access inputs X -t, 22...... ... Tn to produce outputs O 01, ..,..., Ox. ing information, the model will process inputs to make output predictions: M (x,) = ô, However,
In setting without additional support- in many tasks, additional information is present, represented rate into M, we encode each element of X and E to a fixed-size vector representation. This can be accomplished in variety of ways, for example with an encoder neural network.
E {e1, ......, Cm J. To incorpo-
2

---

Under review as a conference paper at ICLR 2020
Then, to make predictions, the model encodes r; and uses K Nearest Neighbors to find the closest related information in E. The representations of the identified nearest neighbors are combined in a weighted sum, where each of the k retrieved neighbors is weighted by its similarity to 2;-
These operations are differentiable, so they can be incorporated into neural networks in a straight- forward way. All elements of the knowledge source E are pre-computed and kept fixed not backpropagate to affect the embeddings of the pre-encoded knowledge. However, this lack of backpropagation can introduce a mismatch between the encoding of E and the model that is train- ing, as the training model has constantly changing representations because the weights are being learned. The model must learn a function to align its representations the external memory. To circumvent this misalignment, we learn a mapping operator fE (x) that maps elements of X into the information representation space E. Concretely, (x(()) multi-layer perceptron with ReLU nonlinearities. From the input elements of X, [z) learns representation of output close to the corresponding projection of X into F. This can be interpreted as learning a read operation on fixed external memory. If there was no change to the encoding of the model compared to the pre-computed knowledge, then the ideal mapping operator would be the identity function. However, as the model changes significantly during the training process, the nonlinear mapping capability of fz) is essential to be able to identify the correct knowledge E from the input X.
do
Thus, model augmented with KIF will incorporate external knowledge in the following manner. First, we find the k nearest elements to the projection of x; in F based on KNN search using inner product, and then the relevant elements encoded by M.
KIF; Mc) e KNearest CicaaresttEE,,er
These elements are weighted by their nearest neighbor scores and summed. This is then concate- nated to the representation of x; and used by M to form the prediction:
M(a ie eeeeigghtssu(KKTt
This is easily extended using multiple modules simultancously. For instance, two sources of information, Ei and Ez. can be combined by identifying the top candidates of each information source. The weighted sum of the KIF output on each information source concatenated with x;.
Finally, different sources of information may not be required for every prediction and some infor- mation sources can be more important than others. To allow the model to make more fine-grained decisions about what information to use from what source, and how much of it, we add gating mechanism using a sigmoid function around each weighted sum of KNN representations. KIF1; and KIF2, denote the KIF module from Equation Drrtiiionppppee applied to E₁ and Eg respectively.
WS1, = We (eeehhhsssu((((11))
(3)
WS2; WeightedSum(KIF2.
(4)
M( (MS1,) ws1;, (WS2.) ws2.]) Óz
(5)
APPLYING KIF TO DIALOGUE TASKS
We describe how apply our method to the task of generative dialogue, a challenging setting where models must autoregressively generate engaging and on-topic responses. We investigate dialogue for two main reasons: first, dialogue agents must be able to consult relevant information to maintain the topic of the conversation. Second, retrieval-based agents have strong performance compared to generative ones, due their ability to copy dialogue utterances from the training set. Using KIP, we can incorporate the benefits of retrieval architectures into generative, knowledge-basse models.
KIF for Generative Dialogue In a dialogue setting, x; represents the text the conversation i. A conversation consists of multiple and--hhd-oorth utterances (or turns). For example, conversation could consist of turns: x; = [ Z2,,, F4,3, Fr,al where Ii,4 is the direct utterance the model should respond to, and the earlier utterances are the conversation context.
Standard generative dialog models use a Transformer neural network as M and want to produce an output o, that is appropriate response to the conversation. However, in many cases, the
3

---

Under review as a conference paper at ICLR 2020
conversation history alone does not include all of the information required to produce appro- priate response. To incorporate knowledge, models often concatenate a knowledge source E such as Wikipedia to 2g, such that M[,,, e1, e cn.) o₁, and use attention modules to identify the most relevant knowledge. However, this approach is computationally intensive when handling large quantities of information. Further, attention mechanisms have been found to operate poorly over long sequences, as the mechanism is blurry and struggles to make fine-grained decisions (Fan et al., 2018 The same is true for hierarchical approaches, which lack scalability.
We augment Transformer sequence-fo-seqquecce (seq2seq) networks with KIF. We experiment on two dialogue tasks, Wizard of Wikipedia (Dinan al. 2018) and Engaging Imagechat (Shuster it al. 2018). We use faiss (Johnson et al. 2019 to perform KNN search efficiently and at scale.
Wizard of Wikipedia The goal of the Wizard of Wikipedia dataset is to train knowledgeable agents that can chat in any domain. The dataset contains 1,365 various topics discussed in 18,430 dialogues in the training set, totalling 166,787 training utterances. The topic is included as the first utterance of the conversation. The dataset includes relevant Wikipedia sentences for each turn of the chat, identified by an information retrieval system.
Our model for Wizard of Wikipedia has access to two sources of external information, E1 and Eg:
E1 is Wikipedia Knowledge provided by the dataset as evidence to support knowledgeable chitchat. The scale of this KNN search is to filter through an average of 34 sentences. The KIF module uses dialogue features to fetch relevant knowledge to condition upon to generate the subsequent utterance.
E2 is Training Ulterances. To incorporate the benefits of retrieval-based dialogue models to the generative setting, we use KIF to identify relevant utterances from the training set and take their responses as input. If many conversations about dogs have already occurred, models should be able to take advantage of these human-written examples to improve their generations. There are around 1 17KK dialogue utterances as inputs to KNN search. This can be interpreted as incorporating the benefits of retrieval models by identifying an utterance with similar structure as the text the model would like generate.
Access to these two sources of knowledge can be seen as as learning template and topic sepa- rately. Sample templates can be identified from the training utterances, and topic-specific informa- tion learned by accessing the Wikipedia knowledge.
To better identify relevant training utterances from the large quantity available, we break down I; into conversation sub-features for a more fine-grained match in the KNN search step. We con- catenate the encoding of the most recent dialogue utterance (e.g. Fi,lust) with the encoding of the dialogue context and the turn number. These are known to be salient conversation features. The most recent dialogue utterance is the direct turn the model is responding to, and the dialogue context may provide additional clues. The turn number is important, as earlier turns are often more generic (e.g. hi, how are you doing today) and later turns are more specific.
Engaging ImageChat The goal of Engaging ImageChat is to create agents capable of chitchat- ting about images, selected from the YFFC100M dataset (Thomee et al. 2015). The dataset con- tains 786,782 dialogues in the training set, each about a unique image, totalling 355,862 utterances. Agents assigned one of 215 personalities (e.g. sweet) increase engagingness. We use Multi- Modal neural network designed to handle both image input and text input. Following (Shuster et al.) (2018), the images are encoded using ResNeXt (Xie et al.) 2017). To extract the final image rep- resentation, we project the 2048-dimensional output of the image encoder to 51 2-dimenssons using a deep multi-layer perceptron with ReLU activation units. The conversation history, which includes the personality, is encoded with a Transformer encoder network. The image and conversation are combined using the Multimodal Sum Combiner proposed in (Shuster al. (2018).
Our model for Engaging Imagechat has access to two sources of external information, Eg and Ey:
E1 is Chat on Similar Images. While there are over 180K different images used in this dataset, many of the images are similar. For example, conversations associated with two pictures of dogs could be relevant to each other. The model is able to fetch from around 160K different images and returns 6 turns of related chat for each image. Fetching from E,
4

---

Under review as a conference paper at ICLR 2020
Model
F1
F1
Model
F1
(Seen)
(Unseen)
Retrieval Trans.*
9.8
Retrieval Trans. MemNer*
15.4
12.4
Generative Trans. MemNet
7.1
2-Stage Generative M
18.9
17.4
+ Reddit TreiTrrai
12.8
Generative Trans. MemNet*
16.9
14.4
Retrieve and Refine
13.6
+ Reddit Pre-Train
17.6
16.3
KIF-Augmented Transformer
14.4
Retrieve and Refine
18.2
17.9
Klf-Augmented Transformer
26.9
23.3
Table 2: Results the Engaging Im-
agechat dataset. denotes results from
Table I: Results on the Wizard of Wikipedia dataset.
Shuster et al. (2018)
denotes results from Dinan et al. 2018
consists of identifying Related Image Chats, or conversations on similar topics (as similar
images are likely to have similar conversations).
E2 Training Utterances. Similar the motivation for the previous dataset, we allow the model to identify training utterances that could be useful for responding in the current conversation. The scale this fetching task is large: 350K dialogue utterances. This could be interpreted as identifying utterances with similar structure to what the model would like to generate, and complementary to the topic-based Related Image Chats.
To identify relevant information from the training utterances, we use the same dialogue features in the KNN search step, with one modification: add the personality provided by the dataset. As utterances from speakers with the same personality are likely to be more related, this feature improves the quality of the fetched information.
5 EXPERIMENTAL SETUP
5.1 IMPLEMENTATION DETAILS
We use parl a (Miller al. 2017) implement our models. We byte-pair encoding (Sen- nnichhh et al. 20155) to represent the text to better handle the rare word problem (Dinan et al. 2018 Fan al. 2017). Our generative Transformer models have 8 encoder layers and 8 decoder layers, with FFN size 2048, embedding dimension 512, and 4 attention heads. We optimize using Adam (Kingma & Ba 2014) and the inverse square root learning schedule (Vaswani et al., 2017) with 10k warmup updates. The initial learning rate is 0.0001 and we optimize for model perplexity. We use a dropout of 0.5 and set gradient clipping to 0.1. We set k for all cases. We pre-train the Trans- former seq2seq model used for both datasets on 250M comments from Reddit. The comments parsed to maintain conversational threads, so the encoder network has been exposed to conversa- tional context at training time. The ResNeXt encoder is pretrained on 3.5 billion images (Mahajan et al., 2018). For both datasets, we model a vocabulary size of 54944 based on the baE-d vocabulary from the Reddit pretraining. We tuned the learning rate and batchsize hyperparameters together. The model size is not tuned, as it was pre-trained with this size and thus kept fixed.
5.2 EVALUATION
Generation
We generate with beam search, setting the beam size to 4 with 3-gram blocking.
Automatic Metrics Following Dinan et al. (018)) we compute FI, a metric of unigram overlap, between the generated utterance and the human-writtee utterance. For generative models, utterances are generated using beam search. For retrieval models, the next utterance is predicted by ranking the entire of training utterances, and the highest scoring utterance is chosen.
In Shuster et al. (2088), retrieval Transformer models report Hits N using fixed candidate set of 99 distractor candidates and true candidate. We compute Fl using their open-sourced model by ranking the entire training set of over 350K utterances.
5

---

Under review as a conference paper at ICLR 2020
WIZARD OF WIKIPEDIA
Preferenca over Betrieval
Proforance over Generative MemNett
ENGAGING IMAGECHAT
Freference ever Retrieval
Preforence over Generative Mamkiet
JNNN Model Preference
807
KENNN Model Preferencee
80
607
60
40-
40
20-
Prefer
More
More
More
Prafer in Long
More
More
More Coherent
Conversation
Figure 2: Human Evaluation. More than 50% indicates the KNN Model is preferred. Stars indicate statistical significance at 0.05
In Wizard Wikipedia, there are two test sets: one set seen topics, or topics that have been seen at training time with new test-time dialogues. The second set is unseen, topics that have not been encountered at all during training time. We evaluate on both of these subsets.
Human Evaluation We follow the setup and use the analysis questions proposed in the Acute-Eval dialogue evaluation system (Li et al. (019). For reproducibility, we adopt this exist- ing evaluation setting that has been applied to several dialogue datasets. collect 100 human-bot dialogues crowdsourcing platform for both datasets. Then, we show pairs of dialogues side by side, and ask the following questions:
Who would you prefer talk for long conversation?
If you had to say one of the speakers interesting and one is boring, who would you say is more interesting?
Which speaker sounds more human?
Which speaker has more coherent responses in the conversation?
If you had to say that one speaker is more knowledgeable and one is more ignorant, who is more knowledgeable? (Wizard of Wikipedia only)
We measure the percentage of time one model was chosen over the other, taking the agreement between three evaluators. To reduce variance, the dialogues that are paired in the evaluation were collected the same topic for Wizard of Wikipedia and collected on the same image for Engaging ImageChat. Each topic and image used is unique and taken from the test set randomly.
5.3 BASELINES
We compare Transformers augmented with KIF the state of the retrieval models published on each dataset, as well as two additional generative baselines that have access to knowledge:
Transformer Memory Networks. To contrast the ability of KIF to existing work, we com- pare models to published Transformer Memory Networks (Dinan et al.) 2018). These models encode each piece external information independently with a Transformer En- coder, and these stored memory slots. To access information in the memory slots, model performs dot-product attention between the memory slots and the dialogue context. In Dinan al. (2018), the knowledge selection from Wikipedia was supervised with either a two-stage model where the first model was trained to predict the right knowledge, or an end-to-end model with an auxiliary loss for knowledge prediction accuracy.
Retrieve and Refine. We implement hybrid model (Weston et al. 2018) that incorporates top retrieval candidates as additional input Generative Transformer MemNets.
All of the generative baselines initialized with the same pre-training Reddit that for our models for fair comparison on modeling quality. For existing published models, we re-trat train the open-sourced generative models with the same pre-training.

---

Under review as a conference paper at ICLR 2020
6 RESULTS
We describe the results of incorporating KIF modules into Transformer networks. We display an ex- ample conversation between a human and our model in Figure and show the top scoring Wikipedia knowledge and Training Utterance fetched by KIF modules.
6.1 KIF IS EFFECTIVE FOR INCORPORATING KNOWLEDGE
Automatic Evaluation. Comparing KIF augmented Transformer networks to published baselines and Retrieve and Refine, we find improved results (see Table [ and Table ( For Wizard of Wikipedia, the improvement in F1 score is almost 10 points. A major contributing factor is the construction of the dataset - as each dialogue turn is grounded in a specific knowledge sentence from Wikipedia, improving the ability to identify the relevant fact strongly improves performance. Contrasting the results from the seen and wnseen test sets, the improvement on unseen is worse it is harder fetch training utterances for unseen topics. Imagechat has no explicit dependency on knowledge. We see a 2 point improvement, indicating that KIF can be generally useful.
Human Evaluation. Results are shown in Figure 2 On both datasets, we find there is large improve- ment over existing generative models (green). Evaluators agree that KlF-augmented Transformers existing retrieval models (blue) is more nuanced. Along the lines of existing work (Zhang et al. 2018 Dinan et al. [208)), we find that retrieval-bai bbsed models score well in human evaluations that ask how human or interesting a dialogue sounds, as they return human- written utterances from the training set.
are generally more coherent and numan-sounding Comparison
surprising result is that KIF-augmented Transformers are voted more human sounding than re- trieval models on Wizard of Wikipedia. This is because the dataset's human utterances long and factual due to the tendency of crowdworkers to copy Wikipedia. Sometimes humans chatting with the retrieval bot would respond uh... that's 's interesting fact? Otherwise, model scores simi- larly to retrieval models, with most of the evaluations not having statistically significant differences.
On Engaging ImageChat, while model has significantly improved it does not beat retrieval based methods in sounding more human or being more interesting. The retrieval baseline directly copies perfectly human-written utterances from the training set, so it is difficult baseline to beat with generative model.
the generative baseline,
6.2 SCALING KIF TO CHALLENGING RETRIEVAL SETTINGS
KIF modules can be used in more realistic and challenging settings for knowledge retrieval that test the scalability of the module. In Figure a),, we compare the Generative Transformer MemNet Baseline with KiF-Augmented Transformers in three settings. The first is the standard Wikipedia sentences provided by the dataset (average 34 sentences). Then, we extend to providing the full Wikipedia article (average 215 sentences) and finally providing multiple Wikipedia articles (av- erage 1035 sentences), identified using the conversation's 's topic. This increasing size of available knowledge could be realistic for settings where it is unclear what information is most relevant, if filtering steps to preprocess the data remove potentially relevant information, or if information syn- thesis from multiple knowledge sources necessary produce high quality generation. As the Wikipedia knowledge becomes more difficult to identify, performance decreases, but still outper- forms the baseline that uses the tetasset-rovided set of 34 sentences.
Comparing the scaling capability of KIF to the standard Generative Transformer MemNet Bascline highlights the advantage of using KNN. The attention- based mechanism used in Dinan et al. (2018) struggles to identify salient information when given increasingly larger quantities of knowledge, unlike the KNN based information fetch. We hypothesize the attention mechanism challenged by the softmax over larger quantity of inputs, as it can be difficult to make sharp distinctions.
6.3 ABLATIONS
Multiple Knowledge Sources. For both Wizard of Wikipedia and Engaging ImageChat, multiple knowledge sources are used - training set utterances to capture the capability of a retrieval-basee model as well as knowledge from Wikipedia or related chats based on image features. The perfor-

---

Under review as a conference paper at ICLR 2020
Model
Test
Model
Valid
Wizard of Wikipedia
Wizard of Wikipedia
Post-Editing Seq2Seq
17.4
Previous Utterance Only
24.6
Retrieval Input Seq2Seq
11.8
+ Dialogue Context
26.4
KIF-Augmented Transformer
18.1
+ Turn Embedding
27.4
Engaging IhageChat
Engaging ImageChat
Post-Editing Seq2Seq
13.1
Previous Utterance Only
13.3
Retrieval Input Seq2Seq
10.2
+
Dialogue Context
14.5
KIF-Augmented Transformer
13.9
+ Turn Embedding + Personality
15.1
Improving generation using KIP on training utterances compared other improve- ments in generative models.
Table 3:
Table 4: Important Features for KNN Search
using KIF. Salient conversation features im- prove performance on both datasets.
(a) SCALING TO WIKI ARTICLE
(b)
IMPACT
(e)
30
30-
26-
Validation
20
Volidation
20-
Validation F1
10
10-
20
Gate Sources Sources
10
15
W/ Gate
Figure 3: Ablations on Wizard of Wikipedia. (a) KIF can scale thousands of relevant sentences (blue) while the baseline model scales poorly (gray) (b) Gating can remove irrelevant information. In the Sources case, one source of external information unrelated. (c) Performance as varies.
mance using fetching the training utterances alone with KIF is shown in Table 3 There is a decrease in performance if only the training utterances can be accessed as an external knowledge source.
We further analyze the importance of accessing training utterances compared to other possible mod- eling techniques to improve Transformer models. contrasss KIF on Training Utterances with two methods of improving generative models: post-editing and using retrieved human utterances as input. Both models are Transformer seq2seq models initialized with our Reddit pre-training where the target the human written utterance. post-editing, the input the predicted utterance from the best generative model. In retrieved human utterances, the input is the utterance returned by the best retrieval model. In both comparisons, we see our model is able to leverage the training set utterances returned by KIF to improve performance.
Multiple Modalities for Fetching, On Engaging ImageChat, we identify conversations similar images and utterances in similar dialogue contexts. We examine the importance of being able to fetch based on the image features access chats from related images, the KIF module can be used for multiple modalities. Table 3 shows results without image-based KIF, which loses 0.5 Fl compared Table 2. The large size the training set allows separate KIF modules be help- ful. KIF using images could be interpreted as identifying topic-based information, and KIF using conversations as identifying templates for the generated response.
Dialogue Features for KNN Performance. The quality of the KNN search is critical the per- formance of KIF modules. As the external knowledge is kept fixed, KIF must be able to align the dialogue context with the knowledge to identify relevant pieces information. In Table we show that matching on more features can improve the quality of the retrieved information. Using only the encoding the immediate previous utterance can improve results on Wizard of Wikipedia by 7 F1 points, but this is further improved by also leveraging the encoding of context (+1.8 F1) and additionally using the dialogue turn number (+1 F1). These features are available in the datasets, and we leverage them improve the relatedness of retrieved knowledge.
8

---

Under review as a conference paper at ICLR 2020
Human: love disney movics! Do you like Disncy?
Model:
love disney movies, I love the fact that they have variety animations and animations that make them
so unique.
Fetched Knowledge: disney consumer products and interactive media the company was formed rendtt of the merger herween disney consumer products and disney interactive
Fetched Training Utterance: love going music shows and dancing and having fun
Human: What's your favoritc disney movic? There are so many good oncs.
Model:
love the incredibles, they are my favorite disney movie.
Fetched Knowledge: disney announced intentions to develop additional superhero films after success of the incredibles
Fetched Training Utterance: kiteboarding sounds very fun to me, but have never done
Human:
omg, me too! second one just came out think. don't watch a sequels though, think the last one saw was toy story.
Model: I remember when toy story was first relcased in 1995.
Fetched Knowledge: talk of a second film following toy story began around a month after the film's s
opening, in december 1995
Fetched Training Utterance: That's 's neat! like so many of her songs!
Human: I've actually never seen toy story before. Kind of emburrassigg...
Model: Oh.. toy story is a classic! It was first released 1999.
Fetched Knowledge: toy story 2 is 1999 american computer-animater comedy film produced by pixar animation studios for walt disney pictures
Fetched Training Utterance: Ohhh see. What do people with color blindness do to cope with the effects?
Conversation between Human and KIF-Augmented Transformer Wizard of Wikipedia. The top-scoring Wikipedia knowledge and training utterances are displayed with generated output.
Figure 4:
Effeet of Gating. We analyze the effect of the gating mechanism used in KIF by evaluating the capability of the gate to identify and focus on salient information. On Wizard of Wikipedia, we concatenate third source of information: dialogue turns from completely different corpus called PersonaChat (Zhang et al., 2018). This dataset looks quite different - short utterances without fac- tual knowledge and should d be easy for the model to identify as distinct from Wizard of Wikipedia. As shown in ( if KIF on PersonaChat is included without gating, it has harmful effect the model includes irrelevant information. When equipped with gating, the model learns the gate to ignore some inputs, and can recover almost the full performance of the model without this irrelvant information source.
Size of Figure 3(c) shows the performance on Wizard of Wikipedia when varying the amount of knowledge from both sources. Generally, being able to access multiple relevant pieces of informa- tion helpful, but too much information can be harmful. This is likely because the weighted sum operation becomes more blurry if too many sentences summed.
7 CONCLUSION
We present KNN-based Information Fetching module that learns to identify relevant information from external knowledge sources by learning a mapping-bane based read operation. KIF modules benefit from the scalability and efficiency of K Nearest Neighbors search, enabling computation with large external memories. We show in the context of two dialogue datasets that relevant knowledge can be identified and incorporated create more engaging, high quality dialogue.
REFERENCES
Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv: 07663, 2016.
Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Ben- gio. Hierarchical memory networks. arXiv preprint 1605.06605.07422 2016.
Welin Chen, David Grangier, and Michael Auli. Strategies for training large vocabulary neural language models. arXiv preprint 1rXiv:1512.000 2015.

---

Under review as a conference paper at ICLR 2020
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: srwwledge-pwweree conversational agents. ICLR, 2018.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv 2017.
preprint
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: 1805:08805.048333 2018.
Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. arXiv preprint arXiv:16/2.04420 2016.
Edouard Grave, Armand Joulin, Moustapha Cissé, Hervé Jégou, al. Efficient softmax approxima- tion for gpus. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1302-1310. JMLR. org. 2017.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv:1410.5001, 2014.
arXiv preprint
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux. and Jason Weston. Real-time inference in multi-sentence tasks with deep pretrained transformers. arXiv preprint arXiv: 1905:/995.00990 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 2019.
Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with back-aaumenterr recurrent nets. In Advances in nearal information processing systems, pp. 190-198, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Guillaume Lample, Alexandre Sablayrolles, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. arXiv preprint arXiv:1907.05243 2019.
Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with
optimized questions and multi-tum comparisons. arXiv preprint arXiv: 190.009909.03887, 2019.
Rongzhong Lian, Min Xie, Fan Wang, Jínhua Peng, and Hua Wu. Learning select knowledge for response generation in dialog systems. arXiv preprint /902.0499 2,119
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 181- 196, 2018.
A. H. Miller, W. Feng, A. Fisch, Lu, Batra, A. Bordes, D. Parikh, and J. Weston. Parlai: A dialog research software platform. arXiv preprint 1rXiv:1705.06476 2017.
Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. In Ad- vances neural information processing systems, pp. 1081-1088, 2009.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 2019.
Lianhui Qin, Michel Galley, Chris Brockett, Xiaodong Liu, Xiang Gao, Bill Dolan, Yejin Choi, and Jianfeng Gao. Conversing by reading: Contentful neural conversation with on-demand machine reading. arXiv preprint arXiv: 1906.009000000 02738, 2019.
Jack Rac, Jonathan Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap. Scaling memory augmented neural networks with sparse reads and writes. In Advances in Neural Information Processing Systems, pp. 3621-3629, 2016.
Rico Sennrich. Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint ar Xiv: 150 550.007999 2015.
10

---

Under review as a conference paper at ICLR 2020
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference Artificial Intelligence, 2016a.
Tulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. Generative deep neural net- works for dialogue: A short review. arXiv preprini (tt::/611,022111 2016b.
Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. Engaging image chat: Modeling personality in grounded dialogue. arXiv preprint / 2018.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, pp. 2440-2448, 2015.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Aug- menting self-attention with persistent memory. arXiv preprint 10X7v:190..01477 2019.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfce100m: The new data in multimedia research. arXiv preprint arXiv:1503.01817, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 09995-9998, 2017.
Jason Weston, Sumit Chopra, and Antoine Bordes.
Memory networks.
arXiv preprint
arXiv: 2014.
Jason Weston, Emily Dinan, and Alexander H Miller. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint arXiv:1808.047700 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Saizheng Zhang, Emily Dinan, Jack Urbanck, Arthur Szlam, Douwe Kiela, and Jason Weston. Per- sonalizing dialogue agents: I have dog, do you have pets too? arXiv preprint arXiv:1801.07244 2018.

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Example: Wizard of Wikipedia
Human: Hey, how are you doing
Fetched Training Utterances: I'm great, thanks for asking. Craving some chocolate. Do you like chocolate?
Hello, how is going? know some trivia about this movie
Human: What are your hobbies?
Fetched Training Utterances: work at an elementary school. hope you find job you love too [...]
I have a hound, we just got her. Although, grew up with Labrador Retrievers.
Human: hi buddy, what do you think about cinematngraphyy?
Fetched Training Utterances: typically, a lens is used to repeatedly focus the light reflected from objects [...]
the modern photographic camera evolved from the camera obscura
Human: Speaking of blue skies, have you seen the 1946 movie staring bing crosby?
Fetched Knowledge: blue skies is a 1946 umerican musical comedy film [...] and starring bing crosby {...]
blue skies the bund has since broken up
Figure 5: Examples of Top-2 Fetched Training Utterances and Fetched Knowledge when responding to human chat from the dataset using a trained Wizard of Wikipedia model. Examples are taken from validation.
Model
Valid F1
Model
Valid F1
KIF with Concatenation 27.4
Wizard of Wikipedia
with Addition with Inner Product
18.2
Transformer Gen MemNet
17.6
20.3
TTrn Number
17.8
Double Last Turn
17.8
Table 5: Comparison of KIF Module Construction on Wizard of Wikipedia. KIF concatenates the fetched knowl- edge, which performs better compared to addition inner product.
Engaging ImageChat
Transformer Generator MemNet
10.4
No Personality Feature Personality in Dialogue History 10.2
9.9
Table Comparison of Features for the Base- line Transformer Generative MemNets on Wizard of Wikipedia and Engaging ImageChat.
A. ABLATIONS
Concatenation of Fetched Knowledge. The knowledge of multiple KIF modules is combined by concatenating the input representation (e.g. the dialogue context). We examine two other alter- natives to incorporating knowledge from multiple sources, specifically addition and inner product. Results shown in T disspla that concatenation outperforms both of these alternatives, and that inner product performs better than addition.
Adding Features Used in KIF Baselines. To improve KNN search of relevant information, the KIF modules use additional features such as the dialogue turn to identify the most useful elements from the fixed memory. We investigate the importance of those features on the dialogue modeling itself by augmenting the baselines with these fetching features. Results are shown in Table
For Wizard of Wikipedia, we add an explicit representation of the turn number and the concatenate the most recent turn again (as the KIF modules these features to fetch knowledge). The addition of these features to the baseline model does not significantly improve performance.
For ImageChat, experiment with representing the personality within the dialogue history as the KIF module uses the personality feature. In our Transformer baseline model following Shuster et al. (2018), the personality is represented as a separate feature. The removal of the personality feature is harmful, but including the personality in the dialogue history as the KIF modules do not as good compared to the separate personality representation learned by Shuster al. (2018).
12

---

Under review as a conference paper at ICLR 2020
WIZARD OF WIKIPEDIA UNSEEN SET
Preforence Gver Ratiioaal
Generative MemNet
WIZARD OF WIKI HIKIPEDIA A VARIANCE
Preference cver flor evai
Proferonce over Generative Memitet
1OW Moce Preferencee
Modell Prefemenc
80
50%
80
20-
More Krowledge
In Long
Convercation Interesting
Humar
Coherent
Knowledge
Figure 6: (left) Human Evaluation on the Unseen Test set of Wizard of Wikipedia. More than 50% indicates the KNN Model is preferred. Stars indicate statistical significance p < 0.05 (right) Variance of Human Evaluation Study on Wizard of Wikipedia. Analysis shows that multi- ple trials of the same experiments relatively stable.
A.2 ADDITIONAL HUMAN EVALUATION
Additional human evaluation results are shown in Figure 6 On the left, the same human evaluation on the Seen
is repeated the Unseen test set, showing similar trends.
On the right, we display an analysis of the variance of our human evaluation. The study was re- peated three different days. There is greater variance the More Human and More Interesting questions, as perhaps different evaluators have different understanding of these aspects. Further, comparison with the Retrieval baseline has less variance compared to Generative models. is pos- sible that the Retrieval model is a bit easier to evaluate given the written text is always copied from human written utterance and is usually devoid of mistakes.
13


Reviews:
[{"review_id": "HJxDDJkX9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses KNN-based retrieval method for extracting relevant information from different sources for the task of generating response in a dialogue task. They utilized different sources of information from Wikipedia, YFCC image set and dialogue utterances. The method is very straight forward and using nearest neighbors from external information sets as auxiliary information and encoding that information via same neural net to produce the encoded external information. The concatenation of encoded input and auxiliary information is concatenated to produce the output in the dialogue via decoder network. The novelty of the proposed method is incremental over Dinan et al. (2018). The KNN-based retrieval module can produce some drift in dialogue from the actual context of the input which can result in irrelevant response in dialogue. Some failure cases can show the quality changes with those drifts in dialogues.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xMxNR2FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- The paper proposes augmenting transformer neural networks with KNN-based information fetching modules that can access relevant external knowledge, combine knowledge from different sources, and integrate the information into seq-to-seq architectures. The authors apply their proposal to generative dialog modeling, and apply it to two dialog datasets.\n\nStrengths:\n- The paper is well-written and well-motivated.\n- The authors evaluate their approach on 2 publicly available datasets and compare it to existing approaches, showing improvements in terms of F1.\nThe authors conduct a human evaluation to compare their approach against other approaches.\n\nWeaknesses:\n- There are some details that are missing from the paper, for example details about the mapping operator, and the specific representations of E_i for the datasets used.\n- Parts of the analysis are rushed (e.g., sections 6.2 and 6.3).\n\nQuestions/Comments:\n- In the human evaluation, is there a difference between the ratings for seen and unseen topics for the Wizard of Wikipedia dataset?\n- Section 6.3 (especially the part on the effect of gating) can be improved with additional information/analysis.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lDPWhucr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n​\nThis paper provides a framework to augment dialogue generation with external data sources using K-Nearest Neighbors in the embedding space. The idea seems simple and intuitive, and the results show improvements over prior work in dialogue generation and retrieval.\n​\n​\n### Strengths\n- The paper shows quantitative improvement over some prior works on dialogue agents (however this needs to be correctly validated).\n- The reviewer appreciates the human study and conversation example provided in the paper to qualitatively evaluate their model.\n- The paper provides ablation studies of the various training tricks for each dataset they have proposed.\n​\n### Weaknesses\n- There are certain changes in the training pipeline that makes the comparison with prior work difficult (Tables 1 and 2), and find the real contribution of data-augmentation caused by KNN-based information fetching (KIF). e.g., In Wizard of Wikipedia experiment, most recent dialogue utterance and turn number are used as salient features. Similarly, the paper utilizes a \"personality\" feature in ImageChat dataset. It seems the results taken from prior papers have different training settings. Could the authors verify that the extra assumptions made for their model are equivalently applied to other generative baselines? If not, the reviewer recommends to provide some experiments with same conditions.\n- Could the authors provide the embedding dimensions and other training details, to assess the contribution of different components of the input's embeddings.\n​\n​\n#### Minor:\n- Is there a sound reason behind using concatenation of input representation and fetched representations? This design choice makes the architecture inflexible to the number of data sources. Another way to combine embeddings - e.g. addition or inner product, can also be tried to see if they provide performance improvements.\n- It is mentioned that attention based modules scale poorly with large sized datasets. If the authors conducted a quantitative evaluation to test this, it would be valuable to add it to the paper.\n- The title and abstract should be limited to generative dialogue modeling, instead of transformers, since the contributions proposed are more suited for this particular application and are quite engineered. Hence, it is not correct to make this claim for transformers in general.\n​\n​\n### Score\nWeak Reject (Leaning towards Accept if appropriate experiements can be provided)", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper augments transformer encoder-decoder networks architecture with k nearest neighbors to fetch knowledge or information related to the previous conversation, and demonstrates improvements through manual and automated evaluation. Reviewers note the fact that the approach is simple and clean and results in significant improvements, however, the approach is incremental over the previous work (including https://arxiv.org/pdf/1708.07863.pdf). Furthermore, although the authors improved the article in the light of reviewer suggestions (i.e., rushed analysis, not so clear descriptions) and some reviewers increased their scores, none of them actually marked the paper as an accept or a strong accept.
Decision: Reject