Task: Language Identification
Input (Spanish, English): This paper addresses the practical problem of generating rare or unseen words in the context of language modeling. Since language follows a Zipf’s law, most approaches limit the vocabulary (because of computation reasons) and hence rare words are often mapped to a UNK token. Rare words are especially important in context of applications such as question answering. MT etc. This paper proposes a language modeling technique which incorporates facts from knowledge bases (KBs) and thus has the ability to generate (potentially unseen) words from KBs. This paper also releases a dataset by aligning words with Freebase facts and corresponding Wikipedia descriptions.  The model first selects a KB fact based on the previously generated words and facts. Based on the selected fact, it then predicts whether to generate a word based on the vocabulary or to output a symbolic word from the KB. For the latter, the model is trained to predict the position of the word from the fact description.  Overall the paper could use some rewriting especially the notations in section 3. The experiments are well executed and they definitely get good results. The heat maps at the end are very insightful.   Comments  This contributions of this paper would be much stronger if it showed improvements in a practical applications such as Question Answering (although the paper clearly mentions that this technique could be applied to improve QA) In section 3, it is unclear why the authors refer the entity as a ‘topic'. This makes the text a little confusing since a topic can also be associated with something abstract, but in this case the topic is always a freebase entity.  Is it really necessary to predict a fact at every step before generating a word. In other words, how many distinct facts on average does the model choose to generate a sentence. Intuitively a natural language sentence would be describe few facts about an entity. If the fact generation step could be avoided (by adding a latent variable which decides if the fact should be generated or not), the model will also be faster. In equation 2, the model has to make a hard decision to choose the fact. For this to be end to end trained, every word needs to be annotated with a corresponding fact which might not be always a realistic scenario. For e.g., in domains such as social media text. Learning position embeddings for copying knowledge words seems a little counter-intuitive. Does the sequence of knowledge words follow any particular structure like word O_2 is always the last name (e.g. Obama). It would also be nice to compare to char-level LM's which inherently solves the unknown token problem.
Output (English): en