Title: LEARNING TO ORGANIZE KNOWLEDGE WITH N-GRAM MACHINES
Authors: Fan Yang
Jiazhong Nie
William W. Cohen
Ni Lao
Venue: ICLR 2018
Keywords: neuro-symbolic reasoning
information extraction
learn to search

Abstract: Deep neural networks (DNNs) had great success on NLP tasks such as language modeling, machine translation and certain question answering (QA) tasks. However, the success is limited at more knowledge intensive tasks such as QA from a big corpus. Existing end-to-end deep QA models (Miller et al., 2016; Weston et al., 2014) need to read the entire text after observing the question, and therefore their complexity in responding a question is linear in the text size. This is prohibitive for practical tasks such as QA from Wikipedia, a novel, or the Web. We propose to solve this scalability issue by using symbolic meaning representations, which can be indexed and retrieved efficiently with complexity that is independent of the text size. More specifically, we use sequence-to-sequence models to encode knowledge symbolically and generate programs to answer questions from the encoded knowledge. We apply our approach, called the N-Gram Machine (NGM), to the bAbI tasks (Weston et al., 2015) and a special version of them (“life-long bAbI”) which has stories of up to 10 million sentences. Our experiments show that NGM can successfully solve both of these tasks accurately and efficiently. Unlike fully differentiable memory models, NGM’s time complexity and answering quality are not affected by the story length. The whole system of NGM is trained end-to-end with REINFORCE (Williams, 1992). To avoid high variance in gradient estimation, which is typical in discrete latent variable models, we use beam search instead of sampling. To tackle the exponentially large search space, we use a stabilized auto-encoding objective and a structure tweak procedure to iteratively reduce and refine the search space.


TL;DR: We propose a framework that learns to encode knowledge symbolically and generate programs to reason about the encoded knowledge.

Review discussion:
The authors propose the N-Gram machine to answer questions over long documents. The model first encodes the document via tuple extraction. An autoencoder objective is used to produce meaningful tuples.
Then, the model generates a program, based on the extracted tuple collection and the question, to find an answer.
I am very disappointed in the authors' choice of evaluation, namely bAbI - a toy, synthetic task long abandoned by the NLP community because of its lack of practicality. If the authors would like to demonstrate question answering on long documents, they have the luxury of choosing amongst several large scale, realistic question answering datasets such as the Stanford Question answering dataset or TriviaQA.
Beyond the problem of evaluation, the model the authors propose does not provide new ideas, and rather merges existing ones. This, in itself, is not a problem.
However, the authors decline to cite many, many important prior work.
For example, the tuple extraction described by the authors has significant prior work in the information retrieval community (e.g. knowledge base population, relation extraction). The idea of generating programs to query over populated knowledge bases, again, has significant related work in semantic parsing and program synthesis. Question answering over (much more complex) probabilistic knowledge graphs have been proposed before as well (in fact I believe Matt Gardner wrote his entire thesis on this topic).
Finally, textual question answering (on realistic datasets) has seen significant breakthroughs in the last few years. Non of these areas, with the exception of semantic parsing, are addressed by the author. With sufficient knowledge of related works from these areas, I find that the authors' proposed method lacks proper evaluation and sufficient novelty.
We thank the reviewer for the insightful feedback.
[lack of sufficient novelty and missing citations]
We disagree with the reviewer and would like to clarify the novelty of our proposed framework. The novelty in our framework is the end-to-end objective function (Equation 2), which learns to construct knowledge storage using down-stream QA tasks as weak supervision. This objective function is different from the ones in the related work mentioned by the reviewer. More specifically, 1) comparing to relation extraction, our method does not use expert-defined schema as supervision; 2) comparing to QA over knowledge graph, our method does not assume knowledge graph is given and instead constructs knowledge storage from text.
About the reading comprehension tasks (e.g., SQUAD), they are not comparable to our work since they do not need to solve the search (from a big corpus) problem.
[lack of evaluation]
We understand the disappointment about evaluation. At this point, we can only defend that this is a theoretic work, which proposes a novel framework, and points to a new direction of how a long lasting problem in search might be solved.
This paper presents the n-gram machine, a model that encodes sentences into simple symbolic representations ("n-grams") which can be queried efficiently. The authors propose a variety of tricks (stabilized autoencoding, structured tweaking) to deal with the huge search space, and they evaluate NGMs on five of the 20 bAbI tasks. I am overall a fan of the general idea of this paper; scaling up to huge inputs is definitely a necessary research direction for QA.
However, I have some concerns about the specific implementation and model discussed here. How much of the proposed approach is specific to getting good results on bAbI (e.g., conditioning the knowledge encoder on only the previous sentence, time stamps in the knowledge tuple, super small RNNs, four simple functions in the n-gram machine, structure tweaking) versus having a general-purpose QA model for natural language? Addressing some of these issues would likely prevent scaling to millions of (real) sentences, as the scalability is reliant on programs being efficiently executed (by simple string matching) against a knowledge storage. The paper is missing a clear analysis of NGM's limitations... the examples of knowledge storage from bAbI in the supplementary material are also underwhelming as the model essentially just has to learn to ignore stopwords since the sentences are so simple. In its current form, I am borderline but leaning towards rejecting this paper.
Other questions:
- is "n-gram" really the most appropriate term to use for the symbolic representation? N-grams are by definition contiguous sequences... The authors may want to consider alternatives.
- why focus only on extractive QA? The evaluations are only conducted on 5 of the 20 bAbI tasks, so  it is hard to draw any conclusions from the results as to the validity of this approach. Can the authors comment on how difficult it will be to add functions to the list in Table 2 to handle the other 15 tasks? Or is NGM strictly for extractive QA?
- beam search is performed on each sentence in the input story to obtain knowledge tuples...
while the answering time may not change (as shown in Figure 4) as the input story grows, the time to encode the story into knowledge tuples certainly grows, which likely necessitates the tiny RNN sizes used in the paper. How long does the encoding time take with 10 million sentences?
- Need more detail on the programmer architecture, is it identical to the one used in Liang et al., 2017?
We thank the reviewer for the insightful feedback.
[“N-gram” might be a misleading term]
We agree that “N-gram” could be misleading, since it commonly means sequences of contiguous words. We are considering other names to use in the future, such as "skip n-gram", or  “engram” https://en.wikipedia.org/wiki/Engram_(neuropsychology).
[why only extractive QA?]
Extractive QA is a family of representative tasks in text understanding. To handle non-extractive QA tasks, we will need to add other functions, which operate on infinite domains (e.g., mathematical operations). The overall model structure should not change, but is beyond the scope of the current result.
[How long does the encoding take with 10 million sentences?]
With our current implementation, scoring 10M sentences would take more than two hours on a single machine without parallelization. A typical commercial search engine uses thousands of machines to encode the meaning of pages (indexing). Even with more complex LSTM structures, scalability is not likely to be an issue for encoding.
[model design overfit the bAbI dataset?]
We agree that the n-gram design and function design have limited expressiveness. We are currently working on more datasets to further understand the balance between model expressiveness and learning difficulty.
The paper presents an interesting framework for bAbI QA.  Essentially, the argument is that when given a very long paragraph, the existing approaches for end-to-end learning becomes very inefficient (linear to the number of the sentences).  The proposed alternative is to encode the knowledge of each sentence symbolically as n-grams, which is thus easy to index.
While the argument makes sense, it is not clear to me why one cannot simply index the original text. The additional encode/decode mechanism seems to introduce unnecessary noise.  The framework does include several components and techniques from latest recent work, which look pretty sophisticated.
However, as the dataset is generated by simulation, with a very small set of vocabulary, the value of the proposed framework in practice remains largely unproven.
Pros:
1. An interesting framework for bAbI QA by encoding sentence to n-grams
Cons:
1. The overall justification is somewhat unclear
2. The approach could be over-engineered for a special, lengthy version of bAbI and it lacks evaluation using real-world data
We thank the reviewer for the insightful feedback.
[why not index the text directly?]
The proposed knowledge encoder is indeed learning to index the text. From an information retrieval perspective, we expect the proposed approach to be a goal-dependent index mechanism, and produces better quality index than traditional indexing approaches. We are not aware of any existing work in this domain.
[sophisticated model but simulated data]
The model architecture is not more sophisticated than a directed probabilistic graphical model with two discrete latent variables, as shown in Figure 1. One might say that the inference procedure is complex, but this is a common challenge shared by many graphical models. The code assist and structure tweak techniques are very similar to conditional sampling (e.g. Gibbs sampling).
Therefore, the proposed learning method is principled, and the choice of dataset does not affect this. We will clarify this in the final version.

Meta-review:
role: PC
data: comment: i am a big fan of this idea, but i agree with the reviewers that evaluating this idea on bAbI (which was originally created from a small set of rules and primitives) discounts quite a bit of what is being claimed here. one of the future directions mentioned by the authors ("investigating whether the proposed n-gram representation is sufficient for natural languages") should have been included even with a negative result, which would've increased the significance significantly.

Decision: Invite to Workshop Track