Title: Learning a Natural Language Interface with Neural Programmer
Authors: Arvind Neelakantan
Quoc V. Le
Martin Abadi
Andrew McCallum
Dario Amodei
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Learning a natural language interface for database tables is a challenging task that involves deep language understanding and multi-step reasoning. The task is often approached by mapping natural language queries to logical forms or programs that provide the desired response when executed on the database. To our knowledge, this paper presents the first weakly supervised, end-to-end neural network model to induce such programs on a real-world dataset. We enhance the objective function of Neural Programmer, a neural network with built-in discrete operations, and apply it on WikiTableQuestions, a natural language question-answering dataset. The model is trained end-to-end with weak supervision of question-answer pairs, and does not require domain-specific grammars, rules, or annotations that are key elements in previous approaches to program induction. The main experimental result in this paper is that a single Neural Programmer model achieves 34.2% accuracy using only 10,000 examples with weak supervision. An ensemble of 15 models, with a trivial combination technique, achieves 37.7% accuracy, which is competitive to the current state-of-the-art accuracy of 37.1% obtained by a traditional natural language semantic parser.

TL;DR: To our knowledge, this paper presents the first weakly supervised, end-to-end neural network model to induce programs on a real-world  dataset.

Review discussion:
Could you please clarify the definition of the lookup answer loss? The notation there is a bit confusing. And could you give more explanations on model ablation? The model is improved a lot by using regularization, but there seems no contents related to the way of carrying out the regularization.
1) At a high level, the lookup loss follows the one described in: https://arxiv.org/pdf/1511.04834v3.pdf . Each entry in the variable "lookup_answer" stores the probability that the corresponding table entry is part of the model prediction. The lookup loss has two terms: the first term computes the loss for entries that are part of the ground-truth answer, while the second term computes it for entries that are not part of the ground-truth answer. We use log loss in both cases.
In this work, the first term additionally handles the ambiguity that arises when the ground-truth answer is simply written down instead of being explicitly marked in the table.
When the ground-truth answer occurs in multiple table entries, we do not know which entry (or entries) in the table is actually responsible for the answer.
So, we pick the entry that has the minimum loss. Intuitively, we encourage the model to select at least one path among the many possible paths to reach the final answer.
2) We exactly follow prior work in applying dropout and weight decay (or L2 regularization). The dropout techniques we use  along with the hyperparameter settings for dropout and weight decay are discussed in the subsection "Training Details" in the "Experiment" section.
We would be happy to answer further questions on the above explanation.
We found a typo in the lookup loss equation (y[i][j] should be g[i][j]). We have fixed it and revised our submission.
-- Definitions of lookup_answer and scalar_answer are not clear.
-- Why run the model for only T=4 time steps? Where did this choice come from?
-- Baselines seem very weak. Did you compare with models which have explicit memory, such as, Memory Networks?
1) The selector module induces two probability distributions at every time step: one over the set of operations and other over the set of columns. Operations that output a scalar modify the scalar_answer variable. In the current version, only the count operation outputs a scalar.
Hence, scalar_answer is set as the output of the count operation multiplied by the probability assigned to that operation by the model.
Similarly, a column of the lookup_answer is updated using the probability assigned to that column and the probability assigned to the print operation.
While the set of operations are different, the definitions of these variables follow prior work: https://arxiv.org/pdf/1511.04834v2.pdf
We made small changes to the notation to make it more clear and revised our submission.
2) We selected T=4 after looking at the programs induced in the baseline paper: http://cs.stanford.edu/~ppasupat/resource/ACL2015-paper.pdf
We experimented with larger values till T=7 but we did not see performance improvements which makes it likely that the dataset does not contain questions that require longer programs.
3) Section 3.3.1 talks about neural network baselines. Memory Networks generate answers that need not be present in the memory.
However in this task, a majority of the questions require the model to pick entries from the table as the final answer. Questions that require an answer that is not in the table are mostly numbers which neural networks are not good at generating. This can be seen in the Seq2Seq experiment discussed in the same section.
Hence, a pointer network model which selects entries from the input is a more suitable model for this task. We modify the pointer network model to select entries only from the table (and not from the question) after reading the question with many latent pondering steps.
So, the baseline discussed in the paper is very similar to the end-to-end memory network model with differences being that the memory is a table and the model selects entries from the table as the final answer instead of generating a new answer. These differences are motivated by the task in hand.
This paper proposes a weakly supervised, end-to-end neural network model to learn a natural language interface for tables. The neural programmer is applied to the WikiTableQuestions, a natural language QA dataset and achieves reasonable accuracy. An ensemble further boosts the performance by combining components built with different configurations, and achieves comparable performance as the traditional natural language semantic parser baseline. Dropout and weight decay seem to play a significant role.
It'll be interesting to see more error analysis and the major reason for the still low accuracy compared to many other NLP tasks. What's the headroom and oracle number with the current approach?
We thank the reviewer for the constructive feedback. If the reviewer has more suggestions for error analysis, we would be happy to perform them.
1) The oracle score is 50.5% which indicates that there is still a lot of room for improvement.
2) We think there are three reasons why the accuracy is lower compared to other NLP tasks:
a) Weak Supervision: most of the standard NLP tasks like parsing, part-of-speech tagging and machine translation have full supervision.
Whereas in this task, the program that needs to be induced is not annotated and the model learns from a weak supervision signal consisting of only the final answer.
b) Small Training set: The dataset considered in this work has only 10k training examples. The datasets for other related language understanding tasks like textual entailment (http://nlp.stanford.edu/projects/snli/) and reading comprehension (https://rajpurkar.github.io/SQuAD-explorer/) have at least an order of magnitude more examples. The accuracy of our model in the training set is 53% while it is only 34% in the development and test set indicating that there is significant overfitting even after employing strong regularization. We think that with more training data, this gap can be reduced.
c) 21% questions not answerable: The paper that introduced this dataset (http://cs.stanford.edu/~ppasupat/resource/ACL2015-paper.pdf) reports that 21% of questions (on a random sample of 200 examples) cannot be answered because of various issues like annotation errors, tables requiring advanced normalization etc.
We have revised our submission incorporating the above analysis.
This paper proposes a weakly supervised, end-to-end neural network model for solving a challenging natural language understanding task.
As an extension of the Neural Programmer, this work aims at overcoming the ambiguities imposed by natural language.
By predefining a set of operations, the model is able to learn the interface between the language reasoning and answer composition using backpropagation.
On the WikiTableQuestions dataset, it is able to achieve a slightly better performance than the traditional semantic parser methods.
Overall, this is a very interesting and promising work as it involves a lot of real-world challenges about natural language understanding.
The intuitions and design of the model are very clear, but the complication makes the paper a bit difficult to read, which means the model is also difficult to be reimplemented. I would expect to see more details about model ablation and it would help us figure out the prominent parts of the model design.
We thank the reviewer for the constructive feedback.
1) As mentioned in the paper, we have open sourced our code and the code is here: https://github.com/tensorflow/models/tree/master/neural_programmer . The results presented in the paper can be reproduced using the code.
2) We have included an ablation study under the title "contribution of different operations", analyzing the contribution of different operations. We revised our submission and request the reviewer to check the latest version of the paper. The paper also discusses the effect of adding regularization to the model. We are currently running experiments to study the effect of anonymizing matched phrases and the addition of boolean features.
Apart from these changes, to the best of our knowledge, the work presented in the paper does not make any significant new model design choices.
For example, the newly proposed training objective is a requirement to apply Neural Programmer on this dataset and not a choice.  If the reviewer has more model ablation studies in mind, we would be happy to include them.
The paper presents an end-to-end neural network model for the problem of designing natural language interfaces for database queries. The proposed approach uses only weak supervision signals to learn the parameters of the model.
Unlike in traditional approaches, where the problem is solved by semantically parsing a natural language query into logical forms and executing those logical forms over the given data base, the proposed approach trains a neural network in an end-to-end manner which goes directly from the natural language query to the final answer obtained by processing the data base. This is achieved by formulating a collection of operations to be performed over the data base as continuous operations, the distributions over which is learnt using the now-standard soft attention mechanisms. The model is validated on the smallish WikiTableQuestions dataset, where the authors show that a single model performs worse than the approach which uses the traditional Semantic Parsing technique.
However an ensemble of 15 models (trained in a variety of ways) results in comparable performance to the state of the art.
I feel that the paper proposes an interesting solution to the hard problem of learning natural language interfaces for data bases. The model is an extension of the previously proposed models of Neelakantan 2016. The experimental section is rather weak though. The authors only show their model work on a single smallish dataset. Would love to see more ablation studies of their model and comparison against fancier version of memnns (i do not buy their initial response to not testing against memory networks).
I do have a few objections though.
-- The details of the model are rather convoluted and the Section 2.1 is not very clearly written.
In particular with the absence of the accompanying code the model will be super hard to replicate. I wish the authors do a better job in explaining the details as to how exactly the discrete operations are modeled, what is the role of the "row selector", the "scalar answer" and the "lookup answer" etc.
-- The authors do a full attention over the entire database. Do they think this approach would scale when the data bases are huge (millions of rows)? Wish they experimented with larger datasets as well.
We thank the reviewer for the constructive feedback.
1) We open-sourced the implementation of our model: https://github.com/tensorflow/models/tree/master/neural_programmer . The results reported in the paper can be reproduced using the provided code.
2) We added an appendix section discussing how the operations are exactly defined and the role of the variables.
3) We agree that since we do full attention the model does not scale easily to huge databases. We have few ideas to scale up the model and leave that for future work. We would also like to point out the fact that public datasets requiring rich semantic parsing on large databases are not currently available.
For example, current semantic parsing datasets on Freebase require much simpler programs to be induced than those considered in this work.
We thank all the reviewers for the constructive feedback. We performed more experiments and added significant new material to the paper.
To summarize:
1) We open-sourced the implementation of our model: https://github.com/tensorflow/models/tree/master/neural_programmer . The results reported in the paper can be reproduced using the provided code.
2) We performed two kinds of model ablation studies (AnonReviewer3 and AnonReviewer4) :
a) The effect of different model design choices (Table 2 and Section 3.4.1).
b) The effect of different built-in operations (Table 4 and Section 3.4.3).
3) We report oracle score and discuss how the model's performance can be potentially improved (Section 3.4.4). (AnonReviewer5)
4) We added an appendix section discussing how the operations are exactly defined and the role of the variables. (AnonReviewer4)
We request the reviewers to check our revised submission, and our separate reply to each reviewer below.

Meta-review:
role: PC
data: comment: The paper applies a previously introduced method (from ICLR '16) to the challenging question answering dataset (wikitables). The results are strong and quite close to the performance obtained by a semantic parser. There reviewers generally agree that this is an interesting and promising direction / results. The application of the neural programmer to this dataset required model modifications which are reasonable though quite straightforward, so, in that respect, the work is incremental. Still, achieving strong results on this moderately sized dataset with an expressive 
 model is far from trivial. Though the approach, as has been discussed, does not directly generalize to QA with large knowledge bases (as well as other end-to-end differentiable methods for the QA task proposed so far), it is an important step forward and the task is already realistic and important.
 
 Pros
 
 + interesting direction
 + strong results on a interesting dataset
 
 Cons
 - incremental, the model is largely the same as in the previous paper
role: Author
data: value: comment: Thanks for taking time to fully understand the paper, its pros and cons, and giving a thorough feedback.

Decision: Accept (Poster)