Learning a Natural Language Interface with Neural Programmer

Learning a natural language interface for database tables is a challenging task that involves deep language understanding and multi-step reasoning. The task is often approached by mapping natural language queries to logical forms or programs that provide the desired response when executed on the database. To our knowledge, this paper presents the first weakly supervised, end-to-end neural network model to induce such programs on a real-world dataset. We enhance the objective function of Neural Programmer, a neural network with built-in discrete operations, and apply it on WikiTableQuestions, a natural language question-answering dataset. The model is trained end-to-end with weak supervision of question-answer pairs, and does not require domain-specific grammars, rules, or annotations that are key elements in previous approaches to program induction. The main experimental result in this paper is that a single Neural Programmer model achieves 34.2% accuracy using only 10,000 examples with weak supervision. An ensemble of 15 models, with a trivial combination technique, achieves 37.7% accuracy, which is competitive to the current state-of-the-art accuracy of 37.1% obtained by a traditional natural language semantic parser. 1 BACKGROUND AND INTRODUCTION Databases are a pervasive way to store and access knowledge. However, it is not straightforward for users to interact with databases since it often requires programming skills and knowledge about database schemas. Overcoming this difficulty by allowing users to communicate with databases via natural language is an active research area. The common approach to this task is by semantic parsing, which is the process of mapping natural language to symbolic representations of meaning. In this context, semantic parsing yields logical forms or programs that provide the desired response when executed on the databases (Zelle & Mooney, 1996). Semantic parsing is a challenging problem that involves deep language understanding and reasoning with discrete operations such as counting and row selection (Liang, 2016). The first learning methods for semantic parsing require expensive annotation of question-program pairs (Zelle & Mooney, 1996; Zettlemoyer & Collins, 2005). This annotation process is no longer necessary in the current state-of-the-art semantic parsers that are trained using only question-answer pairs (Liang et al., 2011; Kwiatkowski et al., 2013; Krishnamurthy & Kollar, 2013; Pasupat & Liang, 2015). However, the performance of these methods still heavily depends on domain-specific grammar or pruning strategies to ease program search. For example, in a recent work on building semantic parsers for various domains, the authors hand-engineer a separate grammar for each domain (Wang et al., 2015). Recently, many neural network models have been developed for program induction (Andreas et al., 2016; Jia & Liang, 2016; Reed & Freitas, 2016; Zaremba et al., 2016; Yin et al., 2015), despite ∗Work done at Google Brain. 1 ar X iv :1 61 1. 08 94 5v 4 [ cs .C L ] 2 M ar 2 01 7 Published as a conference paper at ICLR 2017 Operations Count Select ArgMax ArgMin ... ... > < Print Neural Network What was the total number of goals scored in 2005 Row Selector Scalar Answer Lookup Answer timestep t