Title: A Neural Knowledge Language Model
Authors: Sungjin Ahn
Heeyoul Choi
Tanel Parnamaa
Yoshua Bengio
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Current language models have significant limitations in their ability to encode and decode knowledge. This is mainly because they acquire knowledge based on statistical co-occurrences, even if most of the knowledge words are rarely observed named entities. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model. At each time step, the model predicts a fact on which the observed word is to be based. Then, a word is either generated from the vocabulary or copied from the knowledge graph. We train and test the model on a new dataset, WikiFacts. In experiments, we show that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words. In addition, we demonstrate that the sampled descriptions include named entities which were used to be the unknown words in RNN language models.


TL;DR: A neural recurrent language model which can extract knowledge from a knowledge base to generate knowledge related words such as person names, locations, years, etc.

Review discussion:
If the word-level alignments to the KB are created using string matching when building the dataset, how do you resolve ambiguities when the same named entities may occur in multiple facts (e.g. the location of birth and the location of death could both be the same state.)
What exact representation do you use for the fact embeddings? Trans-E produces three embeddings per fact, e1, e2, and r. Are these simply concatenated or is another network used?
How do you resolve ambiguities when the same named entities may occur in multiple facts?
-> In this case, we assigned a random fact out of the facts with same named entities. This is okay because our final goal as a language model is to generate correct words (the location name) even if the underlying relation was not matching.
But, we agree that having correct relations even in this specific case, will definitely provide more useful semantic information. To achieve this, we can also use existing relation prediction methods during the string matching.
Representation of fact
-> Yes, we use concatenation of the three Trans-E embeddings e1,e2, and r.
- This approach looks similar to that of the paper "Neural Generation of Text from Structured Data with Application to the Bibliography Domain" by Lebret et al.(EMNLP 2016). What are the main differences? Is there a plan to benchmark it on the data introduced in there?
- I don't understand why the description of the object entity alone is used as fact description. Isn't the relation type (like Married-To in the example of the paper) very important too?
- Thanks for pointing the paper. We have not been aware of the paper.
Indeed the paper tries to solve a similar problem by a similar approach.
But, we found that there exist significant differences between the EMNLP paper and ours.
First, the model in the EMNLP paper is based on feedforward neural network language model. Due to this constraint, the experiments (and their dataset) only consider the first sentence of each Wikipedia page. Considering the fact that the state-of-the-art language models have been based on recurrent neural networks, the model in the EMNLP paper seems to have a problem in the generation of multiple sentences.
In contrast, our model is based on recurrent neural networks, and our dataset and experiment consider the generation of the first a few paragraphs, not only a single sentence.
Second, they used the infobox as the knowledge and thus the conditioned knowledge contains a rather limited amount of information. In our model, we consider a large number of facts in the freebase knowledge base. This usually contains much more information and thus a more general approach.
Finally, in our model, we learn a binary switch to decide whether to generate from vocabulary or copy from the knowledge base. This makes the computation efficient because we do not need to compute for the part not selected by the switch.
However, in the EMNLP paper, both the vocabulary words and infobox words are all evaluated at every iteration.
Importantly, in our paper, we also show that the traditional perplexity metric can be problematic in such problems and to resolve propose a new metric. We will discuss about the EMNLP paper in our paper.
- We agree that it will be interesting to compare to the dataset but it can also be considered as a simple task because their dataset consists of only a single sentence per topic.
- The relation type is used when the model selects relevant facts.
But after choosing a fact, copying a word is applied only to the words in the object entity because the words in the relations are usually very frequent words which already exist in the vocabulary (liked, 'married' or 'to').
This paper addresses the practical problem of generating rare or unseen words in the context of language modeling. Since language follows a Zipf’s law, most approaches limit the vocabulary (because of computation reasons) and hence rare words are often mapped to a UNK token. Rare words are especially important in context of applications such as question answering. MT etc. This paper proposes a language modeling technique which incorporates facts from knowledge bases (KBs) and thus has the ability to generate (potentially unseen) words from KBs. This paper also releases a dataset by aligning words with Freebase facts and corresponding Wikipedia descriptions.
The model first selects a KB fact based on the previously generated words and facts. Based on the selected fact, it then predicts whether to generate a word based on the vocabulary or to output a symbolic word from the KB. For the latter, the model is trained to predict the position of the word from the fact description.
Overall the paper could use some rewriting especially the notations in section 3. The experiments are well executed and they definitely get good results. The heat maps at the end are very insightful.
Comments
This contributions of this paper would be much stronger if it showed improvements in a practical applications such as Question Answering (although the paper clearly mentions that this technique could be applied to improve QA)
In section 3, it is unclear why the authors refer the entity as a ‘topic'. This makes the text a little confusing since a topic can also be associated with something abstract, but in this case the topic is always a freebase entity.
Is it really necessary to predict a fact at every step before generating a word.
In other words, how many distinct facts on average does the model choose to generate a sentence. Intuitively a natural language sentence would be describe few facts about an entity. If the fact generation step could be avoided (by adding a latent variable which decides if the fact should be generated or not), the model will also be faster.
In equation 2, the model has to make a hard decision to choose the fact. For this to be end to end trained, every word needs to be annotated with a corresponding fact which might not be always a realistic scenario. For e.g., in domains such as social media text.
Learning position embeddings for copying knowledge words seems a little counter-intuitive. Does the sequence of knowledge words follow any particular structure like word O_2 is always the last name (e.g. Obama).
It would also be nice to compare to char-level LM's which inherently solves the unknown token problem.
Thanks for the comments and feedback. We found that the feedbacks are legitimate and believe that it will help improve our paper. Below, we provide some clarification regarding the comments
1. Is it really necessary to predict a fact at every step?
- No. As you pointed, it is computationally wasteful and unrealistic.
In fact, what you suggested is already implemented in our model.
Specifically, the model predicts a fact only when there exists a relevant fact in the knowledge memory. This is implemented by defining a special fact, called Not-A-Fact (NaF), as described in "Section 3.1 Fact Extraction".
Whenever the model predicts NaF, the model skips the remaining fact search procedure and directly goes to the usual word generation from the fixed vocabulary, resulting in faster computation.
2. Every word needs to be annotated with a corresponding fact which might not be always a realistic scenario
- Similarly to the above answer, words which are not relevant to facts (e.g., is, a, the, have, go, etc.) are mapped to the "NaF"
3. Position embedding.
- Based on our investigation into Freebase KB, there indeed exists a significant extent of regularity in the generation. The clearest regularity is that the words in a fact description are generated from position 1 to N increasing one position at a step.
For example, in most cases, to generate a person name, e.g., "Charles Collingwood", the model (knowing that it is time to start generating a person's name) first predicts the first position, and at the next time, given that the first part ("Charles") is already generated, the second part ("Collingwood") is generated. This applies in most cases including movie titles, location names, etc. (please refer the sample generation in Table4). It is very rare for the model to be required to learn to generate arbitrary order like 4->2->1->3.
Even if the required order is somewhat arbitrary, we still obtain significant gain (in perplexity) by reducing the words to consider from the whole vocabulary to a few fact description words.
We again would like to say that all the comments are really legitimate and helpful for us!
Dear Reviewer3,
As pointed by the reviewers, we have made a major revision of our writing on Section 3 as well as including clarification on other commented points. There is no change in the proposed model and the experiment results. We only tried to clarify the exposition of the proposed model and the dataset. We hope you take a look at the revised version (the revision is mostly in Section 3). And, as you find it become clearer and improved, we also hope reviewers to have a chance to appropriately reconsider the rating.
Also, please feel free to let us know if there are other parts to improve further.
The paper proposes an evolution upon traditional Recurrent Language Models to give the capability to deal with unknown words. It is done by pairing the traditional RNNLM with a module operating on a KB and able to copy from KB facts to generate unseen words. It is shown to be efficient and much better than plain RNNLM on a new dataset.
The writing could be improved. The beginning of Section 3 in particular is hard to parse.
There have been similar efforts recently (like "Pointer Sentinel Mixture Models" by Merity et al.) that attempt to overcome limitations of RNNLMs with unknown words;
but they usually do it by adding a mechanism to copy from a longer past history. The proposal of the current paper is different and more interesting to me in that it try to bring knowledge from another source (KB) to the language model. This is harder because one needs to leverage the large scale of the KB to do so. Being able to train that conveniently is nice.
The architecture appears sound, but the writing makes it hard to fully understand completely so I can not give a higher rating.
Other comments:
* How to cope with the dependency on the KB? Freebase is not updated anymore so it is likely that a lot of the new unseen words in the making are not going to be in Freebase.
* What is the performance on standard benchmarks like Penn Tree Bank?
* How long is it to train compare to a standard RNNLM?
* What is the importance of the knowledge context $e$?
* How is initialized the fact embedding $a_{t-1}$ for the first word?
* When a word from a fact description has been chosen as prediction (copied), how is it encoded in the generation history for following predictions if it has no embedding (unknown word)?
In other words, what happens if "Michelle" in the example of Section 3.1 is not in the embedding dictionary, when one wants to predict the next word?
Thanks for the comments. We found that the feedbacks are very legitimate and helpful for us to revise the paper.
First of all, as pointed by the reviewers, we have made a major revision of our writing on Section 3 as well as including clarification on other commented points. There is no change in the proposed model and the experiment results. We only tried to clarify the exposition of the proposed model and the dataset. We would like to ask you to take a look at the revised version (the revision is mostly in Section 3). And, as you find it clearer and improved, we also hope you to have a chance to appropriately reconsider the rating.
Also, please feel free to let us know if there are other parts to improve further.
In the following, we clarified and answered on your comments.
- How to cope with the dependency on the KB?
=> It's true that we relied on Freebase, which was the largest open KB, to make our dataset.
However, the proposed model is not dependent on any Freebase-specific property.
In fact, any knowledge base, where a fact is defined as the triple form and which provides some description on the facts so that we can apply the copy mechanism, can be used to apply our model (this is satisfied in most of the other available KBs). As the alternatives to the Freebase in the short term, we can use many other open knowledge bases such as DBPedia, Wikidata (to which Freebase is migrated), YAGO, and so on. And, in the long run and more generally, the proposed model will become more useful along the advances in the technology of the automatic knowledge extraction.
- Performance on PennTreebank
Applying our model to a corpus of general topics like PennTreebank or the Google one-billion word dataset will be really interesting. It, however, requires more advances in the line of research beyond the scope of our paper. In this work, as one of the first attempting works in this line of work, we focused more on developing a model that can use the provided knowledge. We are in fact working on the on-the-fly topic searching as our next project in order to extend the model to general topics.
- Training time
As we matched the number of parameters of both models, the training times were not significantly different, but the NKLM took slightly longer time as it has a deeper network.
- The Importance of knowledge context.
The role of the knowledge context is similar to the context representation in the attention-based seq2seq models. The source sentence is replaced by the knowledge memory and we used mean-pooling instead of the weighted averaging.
- Initializing the fact for the first word
It is the same as in the standard word-based language models. We start with the first word and fact as usual.
Otherwise, they are generated from the initial hidden states h0 and the random input x0.
- Embedding for the copied word
For copied words, we use position embeddings instead of word embeddings, because we predict the position to copy. As we also find this part was not so clearly explained in the previous version, we elaborated this further in Section 3 of the revised version.
This paper proposes to incorporate knowledge base facts into language modeling, thus at each time step, a word is either generated from the full vocabulary or relevant KB entities.
The authors demonstrate the effectiveness on a new generated dataset WikiFacts which aligns Wikipedia articles with Freebase facts.  The authors also suggest a modified perplexity metric which penalizes the likelihood of unknown words.
At a high level, I do like the motivation of this paper -- named entity words are usually important for downstream tasks, but difficult to learn solely based on statistical co-occurrences. The facts encoded in KB could be a great supply for this.
However, I find it difficult to follow the details of the paper (mainly Section 3) and think the paper writing needs to be much improved.
- I cannot find where  f_{symbkey} / f_{voca} / f_{copy} are defined
- w^v, w^s are confusing.
- e_k seems to be the average of all previous fact embeddings? It is necessary to make it clear enough.
- (h_t, c_t) = f_LSTM(x_{t−1}, h_{t−1})  c_t is not used?
- The notion of “fact embeddings” is also not that clear (I understand that they are taken as the concatenation of relation and entity (object) entities in the end).  For the anchor / “topic-itself” facts, do you learn the embedding for the special relations and use the entity embeddings from TransE?
On generating words from KB entities (fact description), it sounds a bit strange to me to generate a symbol position first.  Most entities are multiple words, and it is necessary to keep that order.
Also it might be helpful to incorporate some prior information, for example, it is common to only mention “Obama” for the entity “Barack Obama”?
Thanks for the comments. We found that the feedbacks are very legitimate and helpful for us to improve the paper.
First of all, as pointed by the reviewers, we have made a major revision of our writing on Section 3 as well as including clarification on other commented points. There is no change in the proposed model and the experiment results, but following your feedback we tried to clarify the exposition of the proposed model and the dataset. We would like to ask you to take a look at the revised version (the revision is mostly in Section 3). And, as you find it clearer and improved, we also hope you to have a chance to appropriately reconsider the rating.
Also, please feel free to let us know if there are other parts to improve further.
Regarding your comment on copying words by position, we agree that in most of the cases the words will be generated one by one in the increasing order.
In fact, based on our investigation on the generated samples (as shown in Table 4), we found that our copy-by-position mechanism learns this tendency very well. The position prediction mechanism, however, can be considered a more general approach which is expected to properly handle other cases as well.
For example, if some knowledge base represents person names by the format of [last_name, first_name] while Wikipedia descriptions follow the opposite order of [first_name last_name], our position prediction will learn this relation and hence generate properly by reversing the generation order from the original fact description. We also thank for the idea of incorporating the prior information in the generation.
Indeed, we believe there are more interesting things in this direction.
Dear Reviewers and AreaChair,
Approaching to the deadline, I'd like to remind the reviewers of the fact that the revised version of the paper (that we uploaded in Dec. 27) has significant improvements with respect to most of the comments pointed by the reviewers, which are mainly in terms of the writing clarity in Section 3.

Meta-review:
role: PC
data: comment: This work introduces a combination of a LM with knowledge based retrieval system. This builds upon the recent trend of incorporating pointers and external information into generation, but includes some novelty, making the paper "different and more interesting". Generally though the reviewers found the clarity of the work to be sufficiently an issue that no one strongly defended its inclusion.
 
 Pros:
 - The reviewers seemed to like the work and particularly the problem space. Issues were mainly on presentation and experiments. 
 
 Mixed:
 - Reviewers were divided on experimental quality. The work does introduce a new dataset, but reviewers would also have liked use on some existing tasks. 
 
 Cons:
 - Clarity and writing issues primarily. All reviewers found details missing and generally struggled with comprehension.
 - Novelty was a question. Impact of work could also be improved by more clearly defining new contributions

Decision: Reject