Modeling question asking using neural program generation

1 INTRODUCTION . People can ask rich , creative questions to learn efficiently about their environment . Question asking is central to human learning yet it is a tremendous challenge for computational models . There is always an infinite set of possible questions that one can ask , leading to challenges both in representing the space of questions and in searching for the right question to ask . Machine learning has been used to address aspects of this challenge . Traditional methods have used heuristic rules designed by humans ( Heilman & Smith , 2010 ; Chali & Hasan , 2015 ) , which are usually restricted to a specific domain . Recently , neural network approaches have also been proposed , including retrieval methods which select the best question from past experience ( Mostafazadeh et al. , 2016 ) and encoder-decoder frameworks which map visual or linguistic inputs to questions ( Serban et al. , 2016 ; Mostafazadeh et al. , 2016 ; Yuan et al. , 2017 ; Yao et al. , 2018 ) . While effective in some settings , these approaches do not consider settings where the questions are asked about partially unobservable states . Besides , these methods are heavily data-driven , limiting the diversity of generated questions and requiring large training sets for different goals and contexts . There is still a large gap between how people and machines ask questions . Recent work has aimed to narrow this gap by taking inspiration from cognitive science . For instance , Lee et al . ( 2018 ) incorporates aspects of “ theory of mind ” ( Premack & Woodruff , 1978 ) in question asking by simulating potential answers to the questions , but the approach relies on imperfect agents for natural language understanding which may lead to error propagation . Related to our approach , Rothe et al . ( 2017 ) proposed a powerful question-asking framework by modeling questions as symbolic programs , but their algorithm relies on hand-designed program features and requires expensive calculations to ask questions . We use “ neural program generation ” to bridge symbolic program generation and deep neural networks , bringing together some of the best qualities of both approaches . Symbolic programs provide a compositional “ language of thought ” ( Fodor , 1975 ) for creatively synthesizing which questions to ask , allowing the model to construct new ideas based on familiar building blocks . Compared to natural language , programs are precise in their semantics , have clearer internal structure , and require a much smaller vocabulary , making them an attractive representation for question answering systems as well ( Johnson et al. , 2017 ; Yi et al. , 2018 ; Mao et al. , 2019 ) . However , there has been much less work using program synthesis for question asking , which requires searching through infinitely many questions ( where many questions may be informative ) rather than producing a single correct answer to a question . Deep neural networks allow for rapid question-synthesis using encoder-decoder modeling , eliminating the need for the expensive symbolic search and feature evaluations in Rothe et al . ( 2017 ) . Together , the questions can be synthesized quickly and evaluated formally for quality ( e.g . the expected information gain ) , which as we show can be used to train question asking systems using reinforcement learning . In this paper , we develop a neural program generation model for asking questions in an informationsearch game similar to “ Battleship ” used in previous work ( Gureckis & Markant , 2009 ; Rothe et al. , 2017 ; 2018 ) . The model uses a convolutional encoder to represent the game state , and a Transformer decoder ( Vaswani et al. , 2017 ) for generating questions . Building on the work of Rothe et al . ( 2017 ) , the model uses a grammar-enhanced question asking framework , such that questions as programs are formed through derivation using a context free grammar . Importantly , we show that the model can be trained from human demonstrations of good questions using supervised learning , along with a data augmentation procedure that leverages previous work to produce additional human-like questions for training . Our model can also be trained without such demonstrations using reinforcement learning . We evaluate the model on several aspects of human question asking , including reasoning about optimal questions in synthetic scenarios , density estimation based on free-form question asking , and creative generation of genuinely new questions . To summarize , our paper makes three main contributions : 1 ) We propose a neural network for modeling human question-asking behavior , 2 ) We propose a novel reinforcement learning framework for generating creative human-like questions by exploiting the power of programs , and 3 ) We evaluate different properties of our methods extensively through three different experiments . 2 RELATED WORK . Question generation has attracted attention from the machine learning community . Early research mostly explored rule-based methods which strongly depend on human-designed rules ( Heilman & Smith , 2010 ; Chali & Hasan , 2015 ) . Recent methods for question generation adopt deep neural networks , especially using the encoder-decoder framework , and can generate questions without handcrafted rules . These methods are mostly data-driven , which use pattern recognition to map inputs to questions . Researchers have worked on generating questions from different types of inputs such as knowledge base facts ( Serban et al. , 2016 ) , pictures ( Mostafazadeh et al. , 2016 ) , and text for reading comprehension ( Yuan et al. , 2017 ; Yao et al. , 2018 ) . However aspects of human question-asking remain beyond reach , including the goal-directed and flexible qualities that people demonstrate when asking new questions . This issue is partly addressed by some recent papers which draw inspiration from cognitive science . Research from Rothe et al . ( 2017 ) and Lee et al . ( 2018 ) generate questions by sampling from a candidate set based on goal-oriented metrics . This paper extends the work of Rothe et al . ( 2017 ) to overcome the limitation of the candidate set , and generate creative , goal-oriented programs with neural networks . Our work also builds on neural network approaches to program synthesis , which have been applied to many different domains ( Devlin et al. , 2017 ; Sun et al. , 2018 ; Tian et al. , 2019 ) . Those approaches often draw inspiration from computer architecture , using neural networks to simulate stacks , memory , and controllers in differentiable form ( Reed & De Freitas , 2016 ; Graves et al. , 2014 ) . Other models incorporate Deep Reinforcement Learning ( DRL ) to optimize the generated programs in a goal oriented environment , such as generating SQL queries which can correctly perform a specific database processing task ( Zhong et al. , 2018 ) , translating strings in Microsoft Excel sheets ( Devlin et al. , 2017 ) , understanding and constructing 3D scenes ( Liu et al. , 2019 ) and objects ( Tian et al. , 2019 ) . Recent work has also proposed ways to incorporate explicit grammar information into the program synthesis process . Yin & Neubig ( 2017 ) design a special module to capture the grammar information as a prior , which can be used during generation . Some recent papers ( Bunel et al. , 2018 ; Si et al. , 2019 ) encode grammar with neural networks and use DRL to explicitly encourage the generation of semantically correct programs . Our work differs from these in two aspects . First , our goal is to generate informative human-like questions in the new domain instead of simply correct programs . Second , we more deeply integrate grammar information in our framework , which directly generates programs based on the grammar . 3 BATTLESHIP TASK . In this paper , we work with a task used in previous work for studying human information search ( Gureckis & Markant , 2009 ) as well as question asking ( Rothe et al. , 2018 ) . The task is based on an information search game called “ Battleship ” , in which a player aims to resolve the hidden layout of the game board based on the revealed information ( Figure 1 ) . There are three ships with different colors ( blue , red , and purple ) placed on a game board consisting of 6× 6 grid of tiles . Each ship can be either horizontal or vertical , and takes 2 , 3 or 4 tiles long . All tiles are initially turned over ( light grey in Figure 1 ) , and the player can flip one tile at a time to reveal an underlying color ( either a ship color , or dark grey for water ) . The goal of the player is to determine the configuration of the ships ( positions , sizes , orientations ) in the least number of flips . In the modified version of this task studied in previous work ( Rothe et al. , 2017 ; 2018 ) , the player is presented with a partly revealed game board , and is required to ask a natural language question to gain information about the underlying configuration . As shown in Figure 1 , the player can only see the partly revealed board , and might ask questions such as “ How long is the red ship ? ” In this paper , we present this task to our computational models , and ask the models to generate questions about the game board . Rothe et al . ( 2017 ) designed a powerful context free grammar ( CFG ) to describe the questions in the Battleship domain . The grammar represents questions in a LISP program-like format , which consists of a set of primitives ( like numbers , colors , etc . ) and a set of functions over primitives ( like arithmetic operators , comparison operators , and other functions related to the game board ) . Another research by ( Rothe et al. , 2018 ) shows that it captures the full range of questions that people asked in an extensive dataset , mainly because the majority of this grammar is general functions which make it flexible enough . The grammar is able to generate an infinite set of other possible questions beyond collected human questions , capturing key notions of compositionality and computability . Figure 1 provides some examples of produced programs . The full grammar is provided in Appendix C . 4 NEURAL PROGRAM GENERATION FRAMEWORK . The neural network we use includes a Convolutional Neural Network ( CNN ) for encoding the input board , and a Transformer ( Vaswani et al. , 2017 ) decoder for estimating the symbol distribution or selecting actions in different settings described below . The input x ∈ { 0 , 1 } 6×6×5 is a binary representation of the 6x6 game board with five channels , one for each color to be encoded as a one-hot vector in each grid location . A simple CNN maps the input x to the encoder output e ∈ R6×6×M , where M is the length of encoded vectors . Then a Transformer decoder takes e and a sequence of length L as input , and outputs a sequence of vectors yi ∈ RNo , i = 1 · · ·L , where No is the output size . As shown later , the input sequence and output vectors can be interpreted differently in different settings . The model is shown in Figure 2 ( a ) , and details are described in Appendix A . Our model is compatible with both supervised and reinforcement training . Supervised training . In the supervised setting , the goal is to model the distribution of questions present in the training set . Each output yi ∈ RNo is a symbol at position i in the program , where No is the number of different symbols in the grammar . The model is trained with symbol-level cross entropy loss , and can be used to calculate the log-likelihood of a given sequence , or to generate a question symbol-by-symbol from left to right . Generation works as follows . Suppose at step t , a sequence of length t along with the encoded board is presented to the decoder . The model predicts the vector yt which represents the probability of each symbol to be chosen as next . Then we sample a symbol at location t + 1 and execute the decoder again with the new sequence , until an < eos > symbol is generated or maximum length is reached . Sequence-based RL . The framework can be adapted to generate a sequence of symbols without stepwise supervision , such that reward is provided only after the entire question is generated . Grammar-enhanced RL . Finally , the framework can be used with a novel grammar-enhanced RL training procedure . Figure 2 ( b ) illustrates the process of generating a program from the context-free grammar specified in Rothe et al . ( 2017 ) . Beginning from the start symbol “ A ” , at each step a production rule is chosen and applied to one of the non-terminals in the current string . The choice of rule is modeled as a Markov Decision Process , and we solve it with DRL . Each state is a partially derived string passed to the decoder , and we use the first output y1 ∈ RNo to represent the probability of selecting each production rule from all possible No rules . After the rule is applied , the new string is passed back into the decoder , repeating until only terminals are contained in the sequence . We adopt the leftmost derivation here to avoid the ambiguity of parsing order , so at each step the left-most non-terminal will be replaced .

Summary: The authors explore different ways to generate questions about the current state of a “Battleship” game. To do this, they introduce a neural network architecture with a convolutional encoder and a Transformer-based decoder and consider both supervised and reinforcement-learned training approaches. They evaluate the introduced methods in three different ways that demonstrate basic classification and generation abilities of the model.