Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents the n-gram machine, a model that encodes sentences into simple symbolic representations ("n-grams") which can be queried efficiently. The authors propose a variety of tricks (stabilized autoencoding, structured tweaking) to deal with the huge search space, and they evaluate NGMs on five of the 20 bAbI tasks. I am overall a fan of the general idea of this paper; scaling up to huge inputs is definitely a necessary research direction for QA. However, I have some concerns about the specific implementation and model discussed here. How much of the proposed approach is specific to getting good results on bAbI (e.g., conditioning the knowledge encoder on only the previous sentence, time stamps in the knowledge tuple, super small RNNs, four simple functions in the n-gram machine, structure tweaking) versus having a general-purpose QA model for natural language? Addressing some of these issues would likely prevent scaling to millions of (real) sentences, as the scalability is reliant on programs being efficiently executed (by simple string matching) against a knowledge storage. The paper is missing a clear analysis of NGM's limitations... the examples of knowledge storage from bAbI in the supplementary material are also underwhelming as the model essentially just has to learn to ignore stopwords since the sentences are so simple. In its current form, I am borderline but leaning towards rejecting this paper.

Other questions:
- is "n-gram" really the most appropriate term to use for the symbolic representation? N-grams are by definition contiguous sequences... The authors may want to consider alternatives.
- why focus only on extractive QA? The evaluations are only conducted on 5 of the 20 bAbI tasks, so  it is hard to draw any conclusions from the results as to the validity of this approach. Can the authors comment on how difficult it will be to add functions to the list in Table 2 to handle the other 15 tasks? Or is NGM strictly for extractive QA?
- beam search is performed on each sentence in the input story to obtain knowledge tuples... while the answering time may not change (as shown in Figure 4) as the input story grows, the time to encode the story into knowledge tuples certainly grows, which likely necessitates the tiny RNN sizes used in the paper. How long does the encoding time take with 10 million sentences?
- Need more detail on the programmer architecture, is it identical to the one used in Liang et al., 2017?

review_title: review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an interesting framework for bAbI QA.  Essentially, the argument is that when given a very long paragraph, the existing approaches for end-to-end learning becomes very inefficient (linear to the number of the sentences).  The proposed alternative is to encode the knowledge of each sentence symbolically as n-grams, which is thus easy to index.  While the argument makes sense, it is not clear to me why one cannot simply index the original text. The additional encode/decode mechanism seems to introduce unnecessary noise.  The framework does include several components and techniques from latest recent work, which look pretty sophisticated. However, as the dataset is generated by simulation, with a very small set of vocabulary, the value of the proposed framework in practice remains largely unproven.

Pros:
  1. An interesting framework for bAbI QA by encoding sentence to n-grams

Cons:
  1. The overall justification is somewhat unclear
  2. The approach could be over-engineered for a special, lengthy version of bAbI and it lacks evaluation using real-world data

review_title: An interesting, but weird framework for bAbI QA

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose the N-Gram machine to answer questions over long documents. The model first encodes the document via tuple extraction. An autoencoder objective is used to produce meaningful tuples. Then, the model generates a program, based on the extracted tuple collection and the question, to find an answer.

I am very disappointed in the authors' choice of evaluation, namely bAbI - a toy, synthetic task long abandoned by the NLP community because of its lack of practicality. If the authors would like to demonstrate question answering on long documents, they have the luxury of choosing amongst several large scale, realistic question answering datasets such as the Stanford Question answering dataset or TriviaQA.
Beyond the problem of evaluation, the model the authors propose does not provide new ideas, and rather merges existing ones. This, in itself, is not a problem. However, the authors decline to cite many, many important prior work. For example, the tuple extraction described by the authors has significant prior work in the information retrieval community (e.g. knowledge base population, relation extraction). The idea of generating programs to query over populated knowledge bases, again, has significant related work in semantic parsing and program synthesis. Question answering over (much more complex) probabilistic knowledge graphs have been proposed before as well (in fact I believe Matt Gardner wrote his entire thesis on this topic). Finally, textual question answering (on realistic datasets) has seen significant breakthroughs in the last few years. Non of these areas, with the exception of semantic parsing, are addressed by the author. With sufficient knowledge of related works from these areas, I find that the authors' proposed method lacks proper evaluation and sufficient novelty.
review_title: Lacks proper evaluation and sufficient novelty

Meta-review: i am a big fan of this idea, but i agree with the reviewers that evaluating this idea on bAbI (which was originally created from a small set of rules and primitives) discounts quite a bit of what is being claimed here. one of the future directions mentioned by the authors ("investigating whether the proposed n-gram representation is sufficient for natural languages") should have been included even with a negative result, which would've increased the significance significantly.
Decision: Invite to Workshop Track