Modeling question asking using neural program generation

1 INTRODUCTION . People can ask rich , creative questions to learn efficiently about their environment . Question asking is central to human learning yet it is a tremendous challenge for computational models . There is always an infinite set of possible questions that one can ask , leading to challenges both in representing the space of questions and in searching for the right question to ask . Machine learning has been used to address aspects of this challenge . Traditional methods have used heuristic rules designed by humans ( Heilman & Smith , 2010 ; Chali & Hasan , 2015 ) , which are usually restricted to a specific domain . Recently , neural network approaches have also been proposed , including retrieval methods which select the best question from past experience ( Mostafazadeh et al. , 2016 ) and encoder-decoder frameworks which map visual or linguistic inputs to questions ( Serban et al. , 2016 ; Mostafazadeh et al. , 2016 ; Yuan et al. , 2017 ; Yao et al. , 2018 ) . While effective in some settings , these approaches do not consider settings where the questions are asked about partially unobservable states . Besides , these methods are heavily data-driven , limiting the diversity of generated questions and requiring large training sets for different goals and contexts . There is still a large gap between how people and machines ask questions . Recent work has aimed to narrow this gap by taking inspiration from cognitive science . For instance , Lee et al . ( 2018 ) incorporates aspects of “ theory of mind ” ( Premack & Woodruff , 1978 ) in question asking by simulating potential answers to the questions , but the approach relies on imperfect agents for natural language understanding which may lead to error propagation . Related to our approach , Rothe et al . ( 2017 ) proposed a powerful question-asking framework by modeling questions as symbolic programs , but their algorithm relies on hand-designed program features and requires expensive calculations to ask questions . We use “ neural program generation ” to bridge symbolic program generation and deep neural networks , bringing together some of the best qualities of both approaches . Symbolic programs provide a compositional “ language of thought ” ( Fodor , 1975 ) for creatively synthesizing which questions to ask , allowing the model to construct new ideas based on familiar building blocks . Compared to natural language , programs are precise in their semantics , have clearer internal structure , and require a much smaller vocabulary , making them an attractive representation for question answering systems as well ( Johnson et al. , 2017 ; Yi et al. , 2018 ; Mao et al. , 2019 ) . However , there has been much less work using program synthesis for question asking , which requires searching through infinitely many questions ( where many questions may be informative ) rather than producing a single correct answer to a question . Deep neural networks allow for rapid question-synthesis using encoder-decoder modeling , eliminating the need for the expensive symbolic search and feature evaluations in Rothe et al . ( 2017 ) . Together , the questions can be synthesized quickly and evaluated formally for quality ( e.g . the expected information gain ) , which as we show can be used to train question asking systems using reinforcement learning . In this paper , we develop a neural program generation model for asking questions in an informationsearch game similar to “ Battleship ” used in previous work ( Gureckis & Markant , 2009 ; Rothe et al. , 2017 ; 2018 ) . The model uses a convolutional encoder to represent the game state , and a Transformer decoder ( Vaswani et al. , 2017 ) for generating questions . Building on the work of Rothe et al . ( 2017 ) , the model uses a grammar-enhanced question asking framework , such that questions as programs are formed through derivation using a context free grammar . Importantly , we show that the model can be trained from human demonstrations of good questions using supervised learning , along with a data augmentation procedure that leverages previous work to produce additional human-like questions for training . Our model can also be trained without such demonstrations using reinforcement learning . We evaluate the model on several aspects of human question asking , including reasoning about optimal questions in synthetic scenarios , density estimation based on free-form question asking , and creative generation of genuinely new questions . To summarize , our paper makes three main contributions : 1 ) We propose a neural network for modeling human question-asking behavior , 2 ) We propose a novel reinforcement learning framework for generating creative human-like questions by exploiting the power of programs , and 3 ) We evaluate different properties of our methods extensively through three different experiments . 2 RELATED WORK . Question generation has attracted attention from the machine learning community . Early research mostly explored rule-based methods which strongly depend on human-designed rules ( Heilman & Smith , 2010 ; Chali & Hasan , 2015 ) . Recent methods for question generation adopt deep neural networks , especially using the encoder-decoder framework , and can generate questions without handcrafted rules . These methods are mostly data-driven , which use pattern recognition to map inputs to questions . Researchers have worked on generating questions from different types of inputs such as knowledge base facts ( Serban et al. , 2016 ) , pictures ( Mostafazadeh et al. , 2016 ) , and text for reading comprehension ( Yuan et al. , 2017 ; Yao et al. , 2018 ) . However aspects of human question-asking remain beyond reach , including the goal-directed and flexible qualities that people demonstrate when asking new questions . This issue is partly addressed by some recent papers which draw inspiration from cognitive science . Research from Rothe et al . ( 2017 ) and Lee et al . ( 2018 ) generate questions by sampling from a candidate set based on goal-oriented metrics . This paper extends the work of Rothe et al . ( 2017 ) to overcome the limitation of the candidate set , and generate creative , goal-oriented programs with neural networks . Our work also builds on neural network approaches to program synthesis , which have been applied to many different domains ( Devlin et al. , 2017 ; Sun et al. , 2018 ; Tian et al. , 2019 ) . Those approaches often draw inspiration from computer architecture , using neural networks to simulate stacks , memory , and controllers in differentiable form ( Reed & De Freitas , 2016 ; Graves et al. , 2014 ) . Other models incorporate Deep Reinforcement Learning ( DRL ) to optimize the generated programs in a goal oriented environment , such as generating SQL queries which can correctly perform a specific database processing task ( Zhong et al. , 2018 ) , translating strings in Microsoft Excel sheets ( Devlin et al. , 2017 ) , understanding and constructing 3D scenes ( Liu et al. , 2019 ) and objects ( Tian et al. , 2019 ) . Recent work has also proposed ways to incorporate explicit grammar information into the program synthesis process . Yin & Neubig ( 2017 ) design a special module to capture the grammar information as a prior , which can be used during generation . Some recent papers ( Bunel et al. , 2018 ; Si et al. , 2019 ) encode grammar with neural networks and use DRL to explicitly encourage the generation of semantically correct programs . Our work differs from these in two aspects . First , our goal is to generate informative human-like questions in the new domain instead of simply correct programs . Second , we more deeply integrate grammar information in our framework , which directly generates programs based on the grammar . 3 BATTLESHIP TASK . In this paper , we work with a task used in previous work for studying human information search ( Gureckis & Markant , 2009 ) as well as question asking ( Rothe et al. , 2018 ) . The task is based on an information search game called “ Battleship ” , in which a player aims to resolve the hidden layout of the game board based on the revealed information ( Figure 1 ) . There are three ships with different colors ( blue , red , and purple ) placed on a game board consisting of 6× 6 grid of tiles . Each ship can be either horizontal or vertical , and takes 2 , 3 or 4 tiles long . All tiles are initially turned over ( light grey in Figure 1 ) , and the player can flip one tile at a time to reveal an underlying color ( either a ship color , or dark grey for water ) . The goal of the player is to determine the configuration of the ships ( positions , sizes , orientations ) in the least number of flips . In the modified version of this task studied in previous work ( Rothe et al. , 2017 ; 2018 ) , the player is presented with a partly revealed game board , and is required to ask a natural language question to gain information about the underlying configuration . As shown in Figure 1 , the player can only see the partly revealed board , and might ask questions such as “ How long is the red ship ? ” In this paper , we present this task to our computational models , and ask the models to generate questions about the game board . Rothe et al . ( 2017 ) designed a powerful context free grammar ( CFG ) to describe the questions in the Battleship domain . The grammar represents questions in a LISP program-like format , which consists of a set of primitives ( like numbers , colors , etc . ) and a set of functions over primitives ( like arithmetic operators , comparison operators , and other functions related to the game board ) . Another research by ( Rothe et al. , 2018 ) shows that it captures the full range of questions that people asked in an extensive dataset , mainly because the majority of this grammar is general functions which make it flexible enough . The grammar is able to generate an infinite set of other possible questions beyond collected human questions , capturing key notions of compositionality and computability . Figure 1 provides some examples of produced programs . The full grammar is provided in Appendix C . 4 NEURAL PROGRAM GENERATION FRAMEWORK . The neural network we use includes a Convolutional Neural Network ( CNN ) for encoding the input board , and a Transformer ( Vaswani et al. , 2017 ) decoder for estimating the symbol distribution or selecting actions in different settings described below . The input x ∈ { 0 , 1 } 6×6×5 is a binary representation of the 6x6 game board with five channels , one for each color to be encoded as a one-hot vector in each grid location . A simple CNN maps the input x to the encoder output e ∈ R6×6×M , where M is the length of encoded vectors . Then a Transformer decoder takes e and a sequence of length L as input , and outputs a sequence of vectors yi ∈ RNo , i = 1 · · ·L , where No is the output size . As shown later , the input sequence and output vectors can be interpreted differently in different settings . The model is shown in Figure 2 ( a ) , and details are described in Appendix A . Our model is compatible with both supervised and reinforcement training . Supervised training . In the supervised setting , the goal is to model the distribution of questions present in the training set . Each output yi ∈ RNo is a symbol at position i in the program , where No is the number of different symbols in the grammar . The model is trained with symbol-level cross entropy loss , and can be used to calculate the log-likelihood of a given sequence , or to generate a question symbol-by-symbol from left to right . Generation works as follows . Suppose at step t , a sequence of length t along with the encoded board is presented to the decoder . The model predicts the vector yt which represents the probability of each symbol to be chosen as next . Then we sample a symbol at location t + 1 and execute the decoder again with the new sequence , until an < eos > symbol is generated or maximum length is reached . Sequence-based RL . The framework can be adapted to generate a sequence of symbols without stepwise supervision , such that reward is provided only after the entire question is generated . Grammar-enhanced RL . Finally , the framework can be used with a novel grammar-enhanced RL training procedure . Figure 2 ( b ) illustrates the process of generating a program from the context-free grammar specified in Rothe et al . ( 2017 ) . Beginning from the start symbol “ A ” , at each step a production rule is chosen and applied to one of the non-terminals in the current string . The choice of rule is modeled as a Markov Decision Process , and we solve it with DRL . Each state is a partially derived string passed to the decoder , and we use the first output y1 ∈ RNo to represent the probability of selecting each production rule from all possible No rules . After the rule is applied , the new string is passed back into the decoder , repeating until only terminals are contained in the sequence . We adopt the leftmost derivation here to avoid the ambiguity of parsing order , so at each step the left-most non-terminal will be replaced .

Summary: The paper uses a deep neural network architecture (CNN + Transformer) to model logical translations of questions in the form of programs. The experimental setup uses the "battleship" game scenario, which is an interesting domain for questions because of the inherent partial observability present in the game. The paper is clearly written and well-presented.