Modeling question asking using neural program generation
Authors: Ziyun Wang
Brenden M. Lake
Venue:  (2020)
Keywords: question asking
language generation
program induction
reinforcement learning
density estimation
cognitive science

Abstract: People ask questions that are far richer, more informative, and more creative than current AI systems. We propose a neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decoder based deep neural network. From extensive experiments using an information-search game, we show that our method can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings. We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data.

Under review as a conference paper at ICLR 2020
MODELING QUESTION ASKING USING NEURAL PRO- GRAM GENERATION
Anonymous authors Paper under double-bling review
ABSTRACT
People ask questions that far richer, more informative, and more creative than current AI systems. We propose neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decode based deep neural network. From extensive experiments using an information-seann can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings. We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data.
game, we show that our method
INTRODUCTION
People can ask rich, creative questions to learn efficiently about their environment. Question asking is central to human learning yet it is tremendous challenge for computational models. There is always an infinite set of possible questions that one ask, leading to challenges both in representing the space of questions and searching for the right question to ask.
Machine learning has been used to address aspects of this challenge. Traditional methods have used heuristic rules designed by humans (Heilman & Smith 2010; [Chaii & Hasan (2015), which are usually restricted to a specific domain. Recently, neural network approaches have also been proposed, including retrieval methods which select the best question from past experience (Mostafazadeh al. | (2016) and pecoder-decodee frameworks which map visual or linguistic inputs to questions (Serban et al. 2016 Mostafazadeh et al.) 2016 Yuan et al. 2017: Yao et al.) 2018). While effective some settings, these approaches do not consider settings where the questions are asked about partially unobservable states. Besides, these methods are heavily data-driven, limiting the diversity of generated questions and requiring large training sets for different goals and contexts. There still
large gap between how people and machines ask questions.
Recent work has aimed to narrow this gap by taking inspiration from cognitive science, For instance, Lee et al. (2018) incorporates aspects of "theory of mind" (Premack & Woodruff 1978) in question asking by simulating potential answers to the questions, but the approach relies on imperfect agents for natural language understanding which may lead to error propagation. Related our approach, |Rothe jet al. (2017) proposed a powerful question-asking framework by modeling questions as symbolic programs, but their algorithm relies on hand-designed program features and requires expensive calculations to ask questions.
We use "neural program generation" to bridge symbolic program generation and deep neural networks, bringing together some of the best qualities of both approaches. Symbolic programs provide a compositional "language of thought" (Fodor 1975) for creatively synthesizing which questions to ask, allowing the model to construct new Ideas based on familiar building blocks. Compared to natural language, programs are precise in their semanties, have clearer internal structure, and require a much smaller vocabulary, making them an attractive representation for question answering systems as well (Johnson et al., 2017 Yi et al. 2018, Mao et al., 2019). However, there has been much less work using program synthesis for question asking, which requires searching through infinitely many questions (where many questions may be informative) rather than producing a single correct answer to a question. Deep neural networks allow for rapid guestion-synthes modeling, climinating the need for the expensive symbolic search and feature evaluations in Rothe et al. (017). Together, the questions can be synthesized quickly and evaluated formally for quality
using encoder-decoder

---

Under review as a conference paper at ICLR 2020
How long the red ship? (si>e Red)
Is purple ship horizontal?
(=e (or tent Purple) R)
Do all three ships have the same size? (mom (Rap (size 82) (set AllShips)))
groundtruth board
partly revealed board
example questions
Figure I: The Battleship task. Blue, red, and purple tiles are ships, dark gray tiles are water, and light gray tiles are hidden. The agent can see partly revealed board, and should ask question to seek information about the hidden board. Example questions and translated programs are shown on the right. We recommend viewing the figures in color.
(e.g. the expected information gain), which as we show can be used to train question asking systems using reinforcement learning.
In this paper, we develop a neural program generation model for asking questions in an information- search game similar to "Battleship" used in previous work (Gureckis & Markant 2009 Rothe et al., 2017: 2018). The model uses a convolutional encoder to represent the game state, and a Transformer decoder (Vaswani et al. 2017) for generating questions. Building on the work of Rothe al. (2017), the model uses a grammar-enhancee question asking framework, such that questions as programs are formed through derivation using a context free grammar. Importantly, we show that the model can be trained from human demonstrations of good questions using supervised learning, along with data augmentation procedure that leverages previous work to produce additional human-like questions for training. Our model can also be trained without such demonstrations using reinforcement learning. We evaluate the model on several aspects of human question asking, including reasoning about optimal questions in synthetic scenarios, density estimation based on free-form question asking, and creative generation of genuinely new questions.
To summarize, our paper makes three main contributions: 1) We propose neural network for modeling human question-asking behavior, 2) We propose a novel reinforcement learning framework for generating creative human-like questions by exploiting the power of programs, and 3) We evaluate different properties of our methods extensively through three different experiments.
2 RELATED WORK
Question generation has attracted attention from the machine learning community. Early research mostly explored rule-based methods which strongly depend on human- rules (Heilman & Smith 2010 Chali & Hasan 2015). Recent methods for question generation adopt deep neural networks, especially using the encoder-decoder framework, and can generate questions without hand- crafted rules. These methods are mostly data-driven, which use pattern recognition to map inputs to questions. Researchers have worked on generating questions from different types of inputs such as knowledge base facts (Serban et al. 2016), pictures (Mostalazadeh et al.) 2016), and text for reading comprehension (uaan et al. 2017 Yao et al. 2018). However aspects of human question-asking remain beyond reach, including the goal-directed and flexible qualities that people demonstrate when asking new questions. This issue is partly addressed by some recent papers which draw inspiration from cognitive science. Research from Rothe et al.) (2017) and et al.) (2018) generate questions by sampling from candidate set based on goal-oriented metries. This paper extends the work of Rothe et al. (2017) to overcome the limitation of the candidate set, and generate creative, goal-oriented programs with neural networks.
Our work also builds on neural network approaches to program synthesis, which have been applied to many different domains (Devlin et al.) 2017; Sun al. 20188] Tian et al.) (2019). Those approaches often draw inspiration from computer architecture. using neural networks to simulate stacks, memory, and controllers in differentiable form (Reed & De Freitas 2016 Graves et al. 2014). Other models incorporate Deep Reinforcement Learning (DRL) to optimize the generated programs in a goal oriented environment, such as generating SQL queries which can correctly perform a specific database processing task (Zhong et al. 2018), translating strings in Microsoft Excel sheets (Devlin et al. 2017), understanding and constructing 3D scenes (Liu et al., 2019) and objects (Tian et al. 2019). Recent work has also proposed ways to incorporate explicit grammar information into the program synthesis process. (Ynn& & Neubig (2017) design a special module to capture the grammar information prior, which can be used during generation. Some recent papers (Bunel et al. 2018
2

---

Under review as a conference paper at ICLR 2020
(a) Network architecture
(b) Derivation for (> (sice Bllad)
A-B
Layer Softmex
B
5x5 Biter
NNN
Decoder Atlantion Layer
Ca (oNN)
aftte S)
3x3 fitor
(a (rize N
Decoder Actantion Layer
B-Biuc
filtr
for Biue) Nj
N-3
One-hot
encoding
Convolutions
Concat Linear Projection
WE
(size Bluc) 3)
Figure 2: Neural program generation. Figure (a) shows the network architecture. The board is represented as grid of one-shot vectors and is embedded with convolutional neural network. The board embedding and sequence of symbols are inputted to Transformer decoder (Vaswani et al. (011)) to generate output vectors (details in section 4.. PE means positional embeddings, and WE means word embeddings. (h) shows the derivation steps for program (> (size Blue) " using CFG. Non-terminals terminals are shown italic. The production rules used are shown next to each arrow.
shown bold-faced,
Si et al. 2019) encode grammar with neural networks and use DRL to explicitly encourage the generation of semantically correct programs. Our work differs from these two aspects. First, our goal is to generate informative human-like questions in the new domain instead of simply correct programs, Second, we more deeply integrate grammar information in our framework, which directly generates programs based on the grammar.
3 BATTLESHIP TASK
In this paper, we work with task used in previous work for studying human information search (Gureckis & Markant (2099) as well as question asking (Rothe al. 2018). The task based on an information search game called "Battleship", in which a player aims to resolve the hidden layout of the game board based on the revealed information (Figure [ There are three ships with different colors (blue, red, and purple) placed on game board consisting of 6 x 6 grid of tiles. Each ship can be either horizontal or vertical, and takes 2, 3 or 4 tiles long. All tiles are initially turned over (light grey in Figure [ and the player color, or dark grey for water). The goal of the player is to determine the configuration of the ships (positions, sizes, orientations) in the least number of flips.
flip one tile at time to reveal underlying color (either ship
In the modified version of this task studied in previous work (Rothe et al.) 2017 2018). the player is presented with a partly revealed game board, and is required ask natural language question to gain information about the underlying configuration. As shown in Figure | the player can only see the partly revealed board, and might ask questions such "How long the red ship?" In this paper, we present this task to our computational models, and ask the models to generate questions about the game board.
Rothe et al. (2017) designed powerful context free grammar (CFG) to describe the questions in the Battleship domain. The grammar represents questions LISP program-like format, which consists of set of primitives (like numbers, colors, etc.) and set of functions over primitives (like arithmetic operators, comparison operators, and other functions related to the game board). Another research by (Rothe et al. 2018) shows that it captures the full range of questions that people asked in an extensive dataset, mamlyy because the majority of this grammar is general functions which make flexible enough. The grammar is able to generate infinite set of other possible questions beyond collected human questions, capturing key notions of compositionality and computability. Figure provides some examples of produced programs. The full grammar provided in Appendix
4 NEURAL PROGRAM GENERATION FRAMEWORK
The neural network we use includes a Convolutional Neural Network (CNN) for encoding the input board, and Transformer Vaswani et al., 2017) decoder for estimating the symbol distribution or selecting actions in different settings described below. The input r {0, 1 1666x5 is binary representation of the 6x6 game board with five channels, one for each color to be encoded as one-hot vector in each grid location. A simple CNN maps the input x to the encoder output E R6x6xM
3

---

Under review as a conference paper at ICLR 2020
Counting Task
(topleft (coloredTiles X)) Acc: 99.80% (e) Compositionalty Task
(Z. (culoredTiles N))
Doplet (cobraffiles Blne)) (toplell (color-fTiles Purle))
(toplel (cokeedTiles Blue))
(biitonsig (ookeedTilce Blue))
(loplell (cokeadfilo Puplec)
(b)
Missing Tile Task
(- (calor X)X3)
Acc: 95.50%
(lopieft (soloredTTlls Redl) (boltonsigh (coleedfTiles Redj)
(bottomrigh (coforedTilos Parplo))
(color
Known Question Types
Hlldd Our Question Type
Figure 3: Design of the tasks in experiment The goal of task (a) is to find the color which has the least number of visible tiles; the goal of task (b) to find the location and color of the missing tile; (c) is the compositionality task with questions known question types, and another format generated question shown alongside the title of each task, where X, and Z are variables. The accuracy of supervised model for task (a) and (b) are given below each task.
(in dotted box) held out question type. The
where M the length of encoded vectors. Then a Transformer decoder takes and a sequence of length L input, and outputs sequence of vectors y; E IR-N i 1... where N, is the output size. As shown later, the input sequence and output vectors can be interpreted differently in different settings. The model is shown in Figure 2[a), and details are described in Appendix
Our model is compatible with both supervised and reinforcement training.
Supervised training. In the supervised setting, the goal is to model the distribution of questions present in the training set. Each output y: RR. is symbol position i in the program, where No the number of different symbols in the grammar. The model is trained with symbol-level cross entropy loss, and can be used calculate the log-likelihood of given sequence, or to generate question symbol-by-symine
from left to right. Generation works as follows. Suppose at step t, sequence of length along with the encoded board is presented the decoder. The model predicts the vector which represents the probability of each symbol to be chosen as next. Then we sample and execute the decoder again with the new sequence, until an <eos>
symbol location symbol is generated maximum length is reached.
Sequence-based RL. The framework can be adapted to generate a sequence of symbols without stepwise supervision, such that reward is provided only after the entire question is generated.
Grammar-enhanced RL. Finally, the framework be used with novel grammar-enhanced RI. training procedure. Figure 2bb) illustrates the process of generating a program from the context-free grammar specified in Rothe et al. (2017). Beginning from the start symbol "A", at each step production rule is chosen and applied to one of the non-terminals in the current string. The choice of rule modeled as a Markov Decision Process, and we solve with DRL. Each state a partially derived string passed to the decoder, and we use the first output y1 E R N. to represent the probability of selecting each production rule from all possible N,, rules. After the rule applied, the new string is passed back into the decoder, repeating until only terminals are contained in the sequence. We adopt the leftmost derivation here to avoid the ambiguity parsing order, so at each step the left-most non-terminal will be replaced.
5
EXPERIMENTS
5.1
REASONING IN SYNTHETIC SETTINGS
the first experiment, we designed three tasks to evaluate whether the model can learn simple compositional rules and reasoning strategies. These tasks include counting the number of visible ship tiles, locating a missing ship tile, and generalizing both strategies to unseen scenario types. Figure illustrates the three tasks we designed in this experiment by providing examples of each task.
5.1.1 TASK DESCRIPTIONS
Counting task. Models must select the ship color with the least number visible tiles on the board. Each board has unique answer, and models respond by generating program " (topleft
4

---

Under review as a conference paper at ICLR 2020
Table 1: Accuracy (%) the compositionality task using different numbers of training examples from the held out question type.
# of training examples
0
10
50
100
200
400
800
Acc. on held out question type
0,0
2.0
39.0
69.5
81.0
92.0
96.0
Acc. on known question types
96.6
97.3
97.1
96.0
96.3
97.8
96.1
Acc. classify on held out question type
33.0
37.0
49.0
75.5
88.0
94.0
99.5
(coloredTiles X) ) where X is ship color. 4000 examples are used for training, and another 1000 examples are used for testing.
Missing tile task. Models must select the ship that missing tile and identify which tile missing. All ships are completely revealed except one, which is missing exactly one tile. Models respond by generating (== (color Y) X) where x is color and Y is a location on the board. The number of training and test examples are the same as the counting task.
Compositionality task. Models must combine both of the above strategies to find the missing tile of the ship with the least visible tiles. Outputs are produced as (Z (coloredTiles x) ) where X is color and 2 is either topleft or bottomr ight Each board has unique answer.
This task further evaluates compositionality by withholding question types from training. With three values for X and two for 2, there are six possible question types and one is picked as the "held out" type. The other five "known" question types have 800 training examples. For the held out question type, the number of training examples is varied from 0 to 800, to test how much data is needed for generalization. Another 200 new boards of each question type is used for evaluation. More information about the model hyperparameters and training procedures are provided in Appendix B.
5.1.2 RESULTS AND DISCUSSION
We train model in fully supervised fashion. Accuracy for the counting and missing tile tasks is shown in Figure 3 The full neural program generation model shows strong reasoning abilities, achieving high accuracy for both the counting and missing tile tasks, respectively. We also perform ablation analysis of the encoder filters of the model, and provide the results in Appendix D
The results for the compositionality task are summarized in Table When no training data regarding the held out question type is provided, the model cannot generalize to situations systematically different from training data, exactly as pointed out in previous work on the compositional skills of encoder-decoder models (Lake & Baroni (018). However, when the number of additional training data increases, the model quickly incorporates the new question type while maintaining high accuracy on the familiar question tasks. On the last row of Table D we compare our model with another version where the decoder replaced by two linear transformation operations which directly classify the ship type and location (details in Appendix B. This model has 33.0% transfer accuracy compositional scenarios never seen during training. This suggests that the model has the potential to generalize unseen scenarios if the task can be decomposed to subtasks and combined together.
5.2 ESTIMATING THE DISTRIBUTION OF HUMAN QUESTIONS
In this experiment, we examine if the neural network has the capability of capturing the distribution of human questions as a conditioned language model.
5.2.1 DATA AUGMENTATION
To train the model, we need to construct a training set of many paired game boards and questions. Instead of laboriously collecting a large number of real human questions, and translating them into programs by hand, we construct the dataset by sampling from a previous computational model of human question asking (Rothe et al., 2017). More precisely, we randomly generate a large number of game boards and sample K questions given each board. For generating the boards, we first uniformly sample the configuration of three ships, and randomly cover arbitrary number of tiles, with the restriction that at least one ship tile is observable. Next we randomly sample K programs for each board with importance sampling based the cognitive model proposed by Rothe et al. (207)) which models the probability of question under a given context as
p(g: 0) exp(e(q; 0))
5

---

Under review as a conference paper at ICLR 2020
Table 2: Results of the second experiment.
(a) log-liielihood (LL) on two evaluation sets (sampled data and human data) of different ver- sion models.
(b) Log-likelihood (LL) different split of sam- pled evaluation set based on the uncertainty of the board (called low entropy, medium entropy. high entropy respectively). More comparisons are pro-
Model
LL sampled
LL human
vided in Appendix B Table
Full model
.3.997
-7.124
no pretrain
-3.217
-7.280
Model
LL low
LL mid
LL high
LSTM decoder
-3.222
-9.013
Full model
-2.990
-3.190 -3.397
-3,414
MLP encoder
-3.385
-7.475
decoder only
-3.312
-3,494
decoder only
-3.401
-8.434
where e(-) is a parameterized energy function for estimating the likelihood of a question being asked by human, which considers multiple features such as question informativeness, complexity, answer type, etc. is a normalization constant.
We also randomly generate larger set of questions pretrain the decoder component of the model as a "language model" over questions, enabling it to better capture the grammatical structure of possible questions. Details regarding the model hyperparameters, training procedure, and pre-training procedure are provided in Appendix B.2
5.2.2 RESULTS AND DISCUSSION
We evaluate the log- likelihood of reference questions generated by our full model as well as some lesioned variants of the full model, including model without pretraining, model with the Trans- former decoder replaced by an LSTM decoder, a model with the convolutional encoder replaced by a simple MLP encoder, and a model that only has decoder (unconditional language model). Though the method from Rothe et al. (0177) also works on this task, here we cannot compare with their method for two reasons. One is that dataset is constructed using their method, so the likelihood of their method should be an upper bound in our evaluation setting. Additionally, they can only approximate the log-like ligleiool due to an intractable normalizing constant, and thus it difficult to directly compare with our methods.
Two different evaluation sets used, is sampled from the same process on new boards, the other is a small set of questions collected from human annotators. In order to calculate the log-likelihood of human questions, we use translated versions of these questions that were used in previous work (Rothe et al. (2017). and filtered some human questions that score poorly according to the generative model used for training the neural network (Appendix B.2.
A summary of the results is shown in Table 2a The full model performs best on both datasets, suggesting that pretraining, the Transformer decoder, and the convolutional encoder are all important components of the approach. However, we find that the model without an encoder performs reasonably well too, even out-performing the full model with a L.TT--dcooddr on the human- auman-produced questions. This suggests that while contextual information from the board leads to improvements, it is not the most important factor for predicting human questions. To further investigate the role of contextual information and whether not the model can utilize it effectively,
conduct another analysis.
Intuitively, if there is little uncertainty about the locations of the ships, observing the board is critical since there are fewer good questions to ask. To examine this factor, divide the scenarios based on the entropy of the hypothesis space of possible ship locations into low entropy set (bottom 30%), medium entropy set (40% in the middle), and high entropy set (top 30%). We evaluate different models on the split sets of sampled data and report the results in Table 2b When the entropy is high, it is easier to ask generally good question like "how long is the red ship" without information of the board, so the importance of the encoder is reduced. If entropy is low, the models with access to the board has substantially higher likeeiill than the model without encoder. Also, the first experiment (section 5.1 would be impossible without an encoder. Together, this implies that our model can capture important context-sensiticc characteristies of how people ask questions.
5.3 QUESTION GENERATION
In this experiment, we evaluate our reinforcement learning framework proposed in Section 4on its ability of generating novel questions from scratch, without providing large training set.

---

Under review as a conference paper at ICLR 2020
Table 3: Evaluation results of experiment 3. Our grammar enhanced model is compared with supervised trained baseline from experiment 2, sequence generative baseline, and text-based model. The models are compared in terms of average energy value, average expected information gain (EIG) value, the ratio of EIG value greater than 0.9/0, number of unique questions generated, and number of unique novel questions generated (by "novel" we mean questions not presented in the human datasct). The EIG of text-based model is calculated based on the program form of the generated questions.
Model
avg. EIG
EIG>0.9
EIG>0
#unique
#unique novel
text-hased
0.928
62.80%
76.95%
supervised
1.033
51.65%
84.55%
137
9
sequence RL
1.235
58.60%
75.20%
167
52
grammar enhanced RL
1.266
84.75%
91.35%
129
The reward for training the reinforcement agent is calculated based on the energy value of the generated question We transform the energy value to a proper range for reward by -s(g) /100 and clamp it between -1 and 1. The model optimized with the REINFORCE algorithm (Williams) 1992). A baseline for REINFORCE is established simply as the average of the rewards in batch. In order to produce higher-quality questions, we manually tune the informetion-related parameter of the energy function from Rothe et al.] (2017) to make it more information-ssekingg in this experiment. This process is described in Appendix B.2
We compare the models on their ability to generate diverse questions with high expected information gain (EIG), which is defined as the expected reduction in entropy, averaged across all possible answers to question æ.
(I) EdEA, [Ip(h)) - [(p(hld;)))]
(2)
where I(.) is the Shannon entropy. The terms p(h) and p(hidd;;) are the prior and posterior distri- bution of a possible ship configuration ht given question and answer d E An. We compare our program-bat based framework with simple text-based model, which has the same architecture but trained with supervision the text-based question dataset collected by (Rothe al. 2017]. We also compare with the supervised program- based model from the last experiment. Finally, we implement sequence-based based reinforcement learning agent that specifies the program without direct access the grammatical rules. For this alternative RL agent, we find it necessary to pretrain for 500 epochs with stepwise supervision.
5.3.1 RESULTS AND DISCUSSION
The models are evaluated on 2000 randomly sampled boards, and the results are shown in Table 3 Note that any ungrammatical questions are excluded when calculate the number of unique questions. First, when the text- model is evaluated on new contexts, 96.3% of the questions generates were included in the training data. We also find that the average EIG and the ratio of EIG>0 is worse than the supervised model trained on programs. Some of these deficiencies are due to the very limited text-based training data, but using programs instead can help overcome these limitations. With the program- based framework, we can sample new boards and questions to create a much larger dataset with executable program representations. This supervidd training helps to boost performance, especially when combined with grammar-enhanced RL.
From the table, the grammar-enhanced RL model able to generate informative and creative questions. It can be trained from scratch without examples of human questions, and produces many novel questions with high EIG. In contrast, the supervised model rarely produces new questions beyond the training set. The sequence-leve RI model is also comparatively weak at generating novel questions, perhaps because it is also pre-trained on human questions. It also more frequently generates ungrammatical questions.
We also provide examples in Figure show the diversity of questions generated by the grammar enhanced model, and more in the supplementary materials. Figure shows novel questions the model produces, which includes clever questions such as "Where is the bottom right of all the purple and blue tiles?" or "What is the size of the blue ship minus the purple ship?". while it can also sometimes generates meaningless questions such as "Is the blue ship shorter than itself?" Additional examples of generated questions are provided in Appendix B.
7

---

Under review as a conference paper at ICLR 2020
(a) Novel questions
Comparison questions
Where the botton right of parple und blue shijs? lbottoer gnt cunton (estorecTiles turple)
(sotored:i (lus)))
FXG: 1:76
Energy: 5.76
What size blae slip minus pupplc shig!
Our model:
Our model:
furpla33)
Whel is the sise of the bluc ship plus the
Eaerg -6.43
2.00
sise ofe shig?
2col sotilles Redl))
Blue) Comtsiice
What the sise of the bluc ship"
(or ent Bruss
Whet is the sioe the blue ship?
(set5ize tcoloredtiiee Blues)
Is the blue ship Aberier ship?
les Blucks Blue))
What is the size the purple stip?
fxersize protorediss es nllee)
What siro the blse ship?
size Blus 1
Purple?
oolamn the fins pust the blue ship?
Where is list past the slip? lant Is the rad ship panallel to the blue shij! con for Reds corfent Blue))
Energy:
orientation
ship?
(b) Condirioned on question type
Boolean Question:
Nuz horionntal (en II cor feat (lie))
Humon:
Do the red and purple ships touch?
Hlumar:
ships touch? (touch Rnd purplee
the
(touch Parale Red)
Wher does the blue ship start?
Number Question:
How long is the pupple ship!
(topleft tcol bredfilee Blue))
What is minus the sine bfe shiph
(size turple!
Ix ship two tilex long"
Location Quettion:
<+ EsetStiee traforacTiles Blue)))
pen (ortent Illee) RI
blze ship berizental?
Are there any stips in now
(P (+e thas Cambad x (-- (size x) 201
p00)s 9)
Does the blue ship has tiles long?
y?
Composed Question:
ren pstine Tiue]
(aad ALSTsten)3 4)
(not (on teater hatte)33)
Nuz honzontal and pupple ship lenger than (and (ee a Cortest Blue)) D isersize (coloredittes Parste313)
coloe block 4-19
Is par of ship on the 4-67
Lnot (ow tooler 4-11 hator))
(not Cen scoler 6-6) Naters)
Figure 4; Examples of model-generater questions. The natural language translations of the question programs are provided for interpretation. (a) shows three novel questions generated by the grammar enhanced model, (b) shows an example of how the model generates different type of questions by conditioning the input to the decoder, (c) shows questions generated by our model as well as human annotators.
With the grammar enhanced framework, we can also guide the model to ask different types of questions, consistent with the goal-directed nature and flexibility of human question asking. The model can be queried for certain types of questions by providing different start conditions to the model. Instead of starting derivation from the start symbol "A", we can start derivation from intermediate state such as "3" for Boolean questions composition of two Boolean questions. In Figure we show examples where the model is asked to generate four specific types of questions: true/false questions, number questions, location-related questions, and compositional true/false questions. We see that the model can flexibly adapt to new constraints and generate meaningful questions.
more complicated "(ndd B. B) " for
In Figure 4.. we compare the model generated questions with human questions, each randomly- sampled from the model outputs and the human dataset. These examples again demonstrate that our model is able to generate clever and human-like questions. However, we also find that people sometimes generate questions with quantifiers such "any" and "all", which operationalized in program form with lambda functions. These questions are complicated in representation and not favored by our model, showing a current limitation in our model's capacity.
6 CONCLUSION
We introduce a neural program generation framework for question asking task under partially un- observable settings, which is able generate creative human-like questions with human question demonstrations by supervised learning or without demonstrations by grammar-enhanced reinforce- ment learning. Programs provide models with a "machine language of thought" for compositional thinking, and neural networks provide an efficient means of question generation. We demonstrate the effectiveness of our method in extensive experiments covering a range of human question asking abilities.
The current model has important limitations. It cannot generalize to systematically different see- narios, and it sometimes generates meaningless questions. We plan to further explore the model's compositional abilities in future work. Another promising direction is to model question asking and question answering jointly within one framework, which could guide the model to a richer sense of the question semantics. Besides, allowing the agent to iteratively ask questions and try to win the game is another interesting future direction. We would also like to use our framework in dialog systems and open-ended question asking scenarios, allowing such systems to synthesize informative and creative questions.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, Abs/1600.00450, 2016.
Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In International Conference on Learning Representations, 2018.
Yllias Chali and Sadid Hasan. Towards topic-to-question generation. Compulational Linguistics, 41(1):1-20, 2015.
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 990- 998.998.12 JMLR. org, 2017.
Jerry A. Fodor. The Language of Thought. Harvard University Press, 1975.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 1rXiv:1410.5400 2014.
Todd Gureckis and Doug Markant. Active learning strategies in a spatial concept learning game. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31, 2009.
Michael Heilman and Noah A Smith. Good question! statistical ranking for question generation. In Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 609-617. Association for Computational Linguisties, 2010
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reason- ing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2989-2998, 2017.
Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of Learning, 2018.
recurrent networks. In International Conference on Machine
Sang-Was Woo Lee, Youngjoo Heo, and Byoung- Tak Zhang. Answerer in questioner's mind: Information theoretic approach to goal goal-oriented visual dialog. In Advances in neural information processing systems, 2018
Yunchao Liu, Zheng Wu, Daniel Ritchie, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Learning to describe scenes with programs. In International Conference on Learning Representations, 2019.
Jaayua Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference Learning Representations, 2019.
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, C. Lawrence Zitnick, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. In Annual Meeting of the Association for Computational Linguistics, pp. 1802-1813, 2016.
David Premack and Guy Woodruff. Does the chimpanzee have a theory mind? Behavioral and brain sciences, 1(4):515-526, 1978.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In International Conference on Learning Representations, 2016.
Scott Reed and Nando De Freitas. Neural programmer-interpreeerr, In International Conference on Learning Representation, 2016.
Anselm Rothe, Brenden M Lake, and Todd Gureckis. Question asking as program generation. In Advances in Neural Information Processing Systems, pp. 1046-1055, 2017.

---

Under review as a conference paper at ICLR 2020
Anselm Rothe, Brenden M Lake, and Todd M Gureckis. Do people ask good questions? Computa- tional Brain & Behavior, ((1):69-88)) 89, 2018.
Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In Annual Meeting of the Association for Computational Linguisties,, pp. 188-598, 2016.
Xujie Si, Yuan Yang, Hanjun Dai, Mayur Naik, and Song. Learning meta-solver for syntax- guided program synthesis. In International Conference on Learning Representations, 2019.
Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration videos. International Conference on Machine Learning, pp. 4797- 4806, 2018.
Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In International Conference on Learning Representations, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Kaichun Yao, Libo Zhang, Tiejian Luo, Lili Tao, and Yanjun Wu. Teaching machines ask questions. In International Joint Conferences on Artificial Intelligence, pp. 13346-455,, 2018.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in Neural Information Processing Systems, pp. 1031-1042. 2018.
Pengeheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Anmual Meeting of the Association for Computational Linguistics, pp. 440-450, 2017.
Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Sandeep Subra- manian, Saizheng Zhang, and Adam Trischler. Machine comprehension by text-to-text neural question generation. In Workshop on Representation Learning for NLP, 2017.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, ibs//709.00103,, 2018.
10

---

Under review as a conference paper at ICLR 2020
A NETWORK ARCHITECTURE
Encoder. A simple CNN with one layer of filters is used encode the board. Intuitively, many questions are related to specific positions, thus the position information should be recoverable from the encoding. On the other hand, some features of the board a ship is blocked by another ship. In order to capture the position-senn sensitive information as well as the translation-invarian patterns, three convolution operations with different filter sizes (1 1, and 5 x 5) are performed in parallel on the same input. The inputs are padded accordingly to make sure the feature maps have the same width and height. Then three feature maps concatenated together along the dimension of output channels, and passed through linear projection.
such as whether
Formally, the outputs of the convolutions с can be obtained by
c = ReeLU((Conv (I); Convy Convs (x)))
(3)
where Conv, denotes convolution operation on k x A filter, ReLU((-) means applying ReLU activation, and [A; B] means the concatenation of matrices and B. Then E R6x6x3Cout is projected to the encoder output E R6x6xM by matrix W E RRCCou... where Cout is the number of out channels of each convolution, and M is the length encoded vectors.
Decoder. We use the decoder from the Transformer model (Vaswani et al. 2017). With an input sequence of length L. the decoder computes the hidden states through several stacked Decoder Attention Layers. Each layer composed by three sub-layers, self-attention module, an attention over the encoded board, and fully connected feed-forward network. Residual connections are employed around each sub-layer, followed by a layer normalization (Ba et al. 20T6). After N layers of attention modules, final output layer transforms the hidden states to the output vectors y; R N. at every position from to L, where N, is the output size.
Given the output from encoder and the hidden representation h from Decoder Attention Layer n 1, each layer computes the hidden representation as
g" LN(Seff-ATT(h"-))) h=-1)
V LN(ATT(g" e) + gg)
(4)
h" LN(FC(V"") v")
where LN(-) means layer normalization [B et al. and Sclf-ATT(-) are multi-head attention mechanisms, which computes the attention over the output of encoder e, and the attention over the input h" itself, respectively. They are defined as follows
2016 FC(.) is fully connected layer, ATT(-)
ATT(g" e) Multi-ATT(g", e,,e)
(5)
(aI--ATThh"1> Multi-ATT(h" h" hh-))
Multi-ATT(-) is the multi-head attention mechanism described in the paper by Vaswani et al. (2017) which concatenation of multiple standard attention mechanisms with inputs projected using different matrices. A multi-head attention with heads is defined as
Multi-ATT(Q. K, V) W Attention Q W K, WY V);
(6)
Attention ( Q, W K, WXY
where
QKT
Attention( K, V) softmax Vd V
the scaled dot-product attention operation. Q, K, V are a set of vectors called queries, keys, and values, respectively, and dy is the dimension of queries and keys.
After layers, we apply a linear projection and a softmax activation to h to get the output vectors y1....yy.
B EXPERIMENTAL SETTINGS
B. REASONING IN SYNTHETIC SETTINGS
In this experiment, we use Cout 10, = 50 for the model encoder. Each word is embedded with
50 dimension vectors in the decoder. The decoder has 2 layers, each multi-head attention module has
11

---

Under review as a conference paper at ICLR 2020
4 heads, and W.O. W.K, W.Y E R5008 W E R88500 The model is trained for 500 epochs using Adam optimizer with initial learning rate at 0.001 and batch size set as 32.
To further examine the model's ability compositionality task, we evaluate another version of the model which replaces the decoder with two linear transformations to directly predict yi E E c(topleft. bottoms ght). ), and Ve (BBlue, Red, Color). With the hidden representation of the encoder in equation 1, Yi and Ye are calculated as
V/ = softmax(eWi),
ye softmax(eW..)
where Wi E R We E IR [e|x3 and Č is the flattened vector of c.
B.2 ESTIMATING THE DISTRIBUTION OF HUMAN QUESTIONS
In this experiment, the model encoder has the same syper-parameters as in the first experiment. We increase the size of the decoder by setting number of layers to 3, number of heads to 6, and set W.S, W WY E 500x32 W 322x50 The model is also trained for 500 epochs using Adam optimizer with the same initial learning rate at 0.001. In order to better familiarize the model with grammar, also pretrain the decoder for 500 epochs
larger set of question programs. This pretraining corpus is first uniformly sampled from the PCFG grammar defining questions, then we calculate the average energy value of each program on 10, 000 boards, and keep the top 2500 unique questions.
For the evaluation set of human questions, we found that some simple questions become compli- cated in program form. For example, question "How many ship pieces are there in the second column?" ?" will be translated to "+++ (map (lambda y) 2) (not (== (color y) Water) ) (set All_Tiles) ) ) Such complicated programs score very poorly according to the energy function, so they do not appear the training set. As result, the average log-likelihood is extremely skewed by these unseen questions. For a more robust evaluation, we remove the last 10% human questions with low energy values according to
(and (== (colL
Rothe et al. (2017).
B.3 QUESTION GENERATION
The neural model for this experiment has the same prrameparaetters as in the last experiment, and is optimized by REINFORCE (Williams) 1992) algorithm with initial learning rate 0.0001 and batch size 24. To encourage the exploration of the model, we also apply <-greedy strategy with set to 0.5 the beginning, and gradually decreased to 0.01 as training continues. This model is trained for 500 epochs, within each epoch the model passes 10, 000 different boards.
From some preliminary experiments, we find that the models have a strong preference of generating similar programs of relatively low complexity, with the original energy values rewards. Thus, we tune two parameters of the energy model as mentioned in section which are the two parameters corresponding to information secking features (denoted fEIG and fEIG=o in the original paper Rothe et al. (017)). We increase this two parameters from 0.1 until the reinforcement learning models able to generate diverse of questions.
The sequence RL baseline which directly generates sequence with the decoder is trained with MIXER algorithm (Ranzato al. 2016), which is variant of REINFORCE algorithm widely used in sequence generation tasks. MIXER provides a smooth transition from supervised learning to reinforcement learning. This model is pretrained for 500 epochs, and trained with RL for 300 epochs.
C FULL GRAMMAR FOR BATTLESHIP
The grammar is provided in table and
12

---

Under review as a conference paper at ICLR 2020
Table Part of grammatical rules. Rules marked with have a reference the Battleship game board (e.g.. during evaluation the function orient looks up the orientation of a ship on the game board) while all other rules are domain- domain-general (i.e., can be evaluated without access to game board).
Answer types
+ B
Boolean
A
Number
A
Color
A
Orientation Location
A
Boolean
B + True
False
B)
(andl B B)
B - (or B B)
B B B)
N N)
+ == O O)
B > == C C)
B > (= seiN)
True if all elements in set of numbers equal True if any element in of booleans is True True if all elements in set of booleans are True
(any setB)
> (all setB)
> (> N N)
B > (< N
(touch S)*
if the two ships are touching (diagonal does not count)
- (isSubset setL setL)
True if the first set of locations is subset of the second set of locations
Numbers
N +
+
N + (+ N)
N + (+ B B)
N + (++ setN)
+ (++ setB)
Number of True elements in set of booleans
+ (-NN) N
N + (sizc S)*
Size the ship
+ (row
Row number location Column number of location Number of elements in set of locations
+ (col L)
N + (setSize setl.)
Colors
C -SS
Ship color
> Water
C > (color L)
Color at location L
Blue
Red
+ Purple
+
Lambda variable for ships
Orientation
0 II
Horizontal
+
Vertical
O + (orient S)*
Orientation of the ship S
Locations 1-1
Row column
+ 6-6
Lambda variable for locations
(topleft setl.)
The most left of the top location the of locations
(bottomright setl.)
The most right of the most bottom location in the set of locations
13

---

Under review as a conference paper at ICLR 2020
Table 5: Part of the grammatical rules. See text for details.
Mapping
setB - (map ryB setL.)
Map boolean expression onto location set Map boolean expression ship Map a numerical exprexsion onto ship set Map a location expression onto ship set
setB (map fxB setS)
setN - (map fxN setS)
setL + (map fxL setS)
Lambda expressions
fyB ( B)
Boolean expression with location variable Boolean expression with ship variable Numeric expression with ship variable Location expression with ship variable
fxB - (A B)
fxN + (AxN)
fxL (AxL)
Sets
setS + (set AllShips)
All ships
setL + (sct AllTiles)
All locations
setL (coloredTiles C)
All locations with color
setL 7 (setDiffereene setL setL)
Remove second set from first set
setL + (union setL setL)
Combine both sets
setL - (intersection setl. setL.)
Elements that exist in both sets
setL (unique setL.)
Unique elements in set
Table 6: Results of the synthetic reasoning tasks.
(a) Accuracy different models on the count-
(b) Accuracy for selecting the right location and color for the missing tile
ing and missing tile tasks
Model
Counting
Missing tile
Model
Location acc.
Color acc.
Full model
99.80%
95.50% 51.50%
Full model
97.80%
97.60% 96.20%
3x3 conv only
99.30%
3x3 conv only
53.00%
Ixl conv only
98.60%
1.90%
1xl conv only
3.90%
45.50%
D ADDITIONAL RESULTS
We also perform an ablation test on the neural network the first experiment (section 5.1p. Accuracy for the counting and missing tile tasks is summarized in Table 6a for the full model and lesioned variants. The full neural program generation model shows strong reasoning abilitics, achieving an accuracy of 99.88 99% and 95.50% for the counting and missing tile tasks, respectively. The full model compared to weakened variants with only one filter size in the encoder, either "3x3" and "Ix1 conv only," and the performance of the weakened models drop dramatically the missing tile task.
To better understand the role different filter sizes, Table breaks down the errors the missing tile task whether the question can pick the right ship (color acc. and whether it can select the right location (location acc.). The x 3 convolution filters can accurately select the correct color, but often fail to choose the right tile. The model with 1 convolution filters has poor performance for both color and location. In the current architecture, predicting the correct location requires precise information that seems to be lost without filters of different sizes.
For the experiment on estimating the distribution of human questions (Experiment 5.2 Table 7 provides full table of log-likelihood of different models on evaluation set of different uncertainty level.
Here we provide more examples of questions generated by our models in the generation experiment (Experiment 5.3) Figure 5 6 and provides more examples for the same settings shown in figure in the main text. Figure shows generated examples of the rext-based model.
14

---

Under review as a conference paper at ICLR 2020
Table Log-likelihood (LL.) different splits of the sampled evaluations based the uncertainty of the board.
Model
LL on low entropy
LL on mid entropy
LL on high entropy
Full model
-2.990
-3.190
-3.414
no pretrain
-3.015
-3.210
-3,428
LSTM decoder
-3.044
-3.209
33,416
MLP encoder
-3,293
-3.383
-3,477
decoder only
-3.312
-3.397
-3,494
What the size of the blue ships plus the red ships? (setSize (union (coloredTiles Blue)
(coloredTiles Red)))
EIG: 1.92
Energy: -7.38
Is the purple ship parallel to the blue ship? (== (or Purple) (orient Blue))
EIG: 0.99
Energy: -8.14
Which contains the first tile of the blue ship? (rowL (toplert (coloredTiles Blue)))
EIG: 2.50
Energy: -8.99
What the size of blue plus eight? (+ (size Blue))
EIG: 1.58
Energy: -7.74
What the size of blue plus the number the row which contains the beginning of the red ship?
(+ (size Blue) (colL (topleft (coloredTiles Red)))
BIG: 2.46
Energy: -5.11
Figure S: Novel questions generated by the grammar enhanced model.
15

---

Under review as a conference paper at ICLR 2020
Boolean Question:
purple ship horizontal? (== (ortent Purple))
Number Question:
What is the size of the red ship?
(setS1ze (cotoredTicc Red))
Location Question:
Where is the last tile of purple ship?
(bottonr iight (coloredTiles Purple))
Composed Question:
Is blue ship vertical and also parallel with purple ship?
(and (== fent Blue) ientent Purple)) (== V. (orient Blue)))
Boolean Question:
the purple ship longer than the red ship?
(> (size Purple) (size Red))
Number Question:
What is size of purple ship?
(s1ze Purple)
Location Question:
Where is the last tile of the purple ship?
(bottonri ght (coloredT1 Les Purple))
Compused Question:
Is purple ship parallel blue ship and also parallel to itself?
(and === (orient Purple) (ortent Blue))
(orient Blue) (ar ient Blue)))
Boolean Question:
Is red ship horizontal?
(=- H (orfent Red))
Number Question:
What is size of ship?
(setSize (coloredT les Red))
Location Question:
Where is the last tile of blue ship?
(bottonr ight (coloredTiles Blue))
Composed Question:
Is purple ship parallel to itself and also blue ship parallel with red ship? (and (== (orient Purple) (orient Purple))
(orient Blue) (arient Red)))
Figure 6: Generated questions of different types by controlling the start condition.
16

---

Under review as a conference paper at ICLR 2020
Our model:
Our model:
Where is the bottom right of the blue ship?
What is the size of the blue ship? (s1ze Blue) red ship parallel with blue ship?
(bottomr ght (co cored les Blue))
size the ship?
(setS1ze (co loloredTilss Blue))
(== (orfent Red) (orient Blue))
What is the orientation of the blue ship?
What the size of the rod ship plus 5?
(or ient Purple)
( 5 set5ize ( coloredTile Red
Where is the bottom right of the purple ship?
What size the ship?
(bot tonr ight (coloredTile iles Purple))
setS fizee coloredTiles Red
What is the size of the blue ship?
What the orientation of the red ship? ( or ortent Red )
(size Blue)
What is the size of the blue ship? (setS12e (coloredTiles Blue))
Where is the bottom right of the blue ship? bottomr
coloredTiles Blue
Human:
Human:
Are the majority of the ships horizontal or vertical! (map (lambda (== (ortent H))
Is the red ship touching another ship?
(touch Red Blue) (touch Red Purple))
(set AllColars))) 1)
there an item (not (-- (color 6-1) Water))
tile?
Which direction is purple?
(ortent Purple)
How many tiles is the red ship? (size Red)
the ship placed vertically?
(-- (or tent Red) H)
red ship touching another ship? (or (touch Red Blue). (touch Red Purple)) Is orange ship 2 tiles long?
3 tiles is blue ship?
(s1ze Blue)
Does tile B6 a part of the blue ship?
(=- (size Red) 2)
(== (color 6-2) Blue)
any part of red ship left half of the grid?
Where is the end of the blue ship
(-+ (map (lambda (and (< (coll y) (-- (color y) Red))) (set AllTiles))) 8)
(bot Lonr ight (coloredT iles Blue))
Figure 7: Comparisons of questions generated by our model with human questions.
Does the 6th contain pieces of the red ship?
Is the blue ship horizontal vertical?
EIG: 0.91
RIG: 0.00
How miny blocks is the red ship?
Is the square at b2
EIG: 1.52
Ungrammatical.
each ship the same size?
Is any part of the blue ship in the same direction?
EIG: 0.75
Ungrammatical.
Figure 8: Example questions generated by the text-based model.
17


Reviews:
[{"review_id": "SkeBcFvEKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper uses a deep neural network architecture (CNN + Transformer) to model logical translations of questions in the form of programs. The experimental setup uses the \"battleship\" game scenario, which is an interesting domain for questions because of the inherent partial observability present in the game. The paper is clearly written and well-presented.\n\nThe length of the submission is 9 pages in content. The call for papers states that \"Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages.\" Given the major reservation I have regarding the current version (see below), and the need to apply a higher standard, it is hard for me to recommend acceptance in its current form.\n\nThe major reservation I have is that the paper currently sets up the reader to expect the data to contain real, human-generated questions. The abstract mentions \"human question asking\", talks about predicting \"which questions humans are likely to ask in unconstrained settings\"; on p.2 the paper states that \"the model can be trained from human demonstrations of good questions\"; and so on. Hence it came as a surprise to learn on p.6 that collecting human questions and translating them into programs is too laborious, so an automatic system was used to generate the questions instead. This isn't necessarily a killer for the paper, but it does require more justification, and more sign-posting early in the paper.\n\nOn p.3 there is some attempt at justifying the decision to use a simulator, which is that an existing paper has shown that it captures the full range of questions that people ask. I feel that this justification needs expanding.\n\nAnother issue with the paper is that it heavily builds on the existing work by Rothe et al., in fact so much so that it's difficult to fully appreciate the current paper without reading that existing work (which I haven't done, just for full disclosure). In fact, a summary of the current paper would be that it takes the existing work of Rothe et al., which is a rule-based question-generation system, and it \"neuralizes\" it by replacing the rules with a neural architecture.\n\nThat said, I think there is still a lot to commend in this work: the setting is an interesting one for question generation, the motivation for using programs is well-made, and the work appears technically sound.\n\nSome more minor comments\n--\n\nI wonder about the log-likelihood numbers on the synthetic questions. Doesn't this just show that a neural network can effectively reverse-engineer a synthetic grammar? I'm not sure how interesting that is. I also wonder about the decision to filter questions that score poorly according to the generative model used for training the network - doesn't this just add additional bias in favour of the model being evaluated?\n\nA similar comment applies to the decision to exclude ungrammatical questions from the uniqueness metric. I assume \"ungrammatical\" here means ungrammatical according to the pre-defined grammar. But that just makes it easier for the grammar-based system to perform well on this particular metric, no?\n\nI don't think EIG is defined anywhere in the paper.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hye7F-uXtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors explore different ways to generate questions about the current state of a “Battleship” game. To do this, they introduce a neural network architecture with a convolutional encoder and a Transformer-based decoder and consider both supervised and reinforcement-learned training approaches. They evaluate the introduced methods in three different ways that demonstrate basic classification and generation abilities of the model.\n\nWhile the paper is generally well written and clear, I noticed quite a few grammar mistakes and typos. I expect these can be fixed in the final version.\n\n\nIn my opinion this work has several limitations and I don’t think it is a strong enough contribution to be interesting to the ICLR audience. Thus I am leaning towards rejection.\n\n\nWhile the problem of question generation is quite interesting, this work is limited in several ways. Firstly the authors limit the domain of question answering severely by just considering questions about “Battleship” that can be expressed in a lisp-like language. They do not attempt to generate questions that are oriented towards a more complex goal (like winning the game), but instead use three tasks that are quite limited:\n\n- First they show the model works well on very simple toy tasks; in particular the tasks are 1. Identifying the colour with the least visible tiles, 2. Identifying the ship with a missing tile, and 3. Identifying the colour of the ship with both a missing tile and the least number of visible tiles. These are extremely simple classification problems and no real question has to be generated. While this is a good sanity check for the model, it is not an interesting result.\n- In a second task they try to show that the model can capture the distribution of human authored questions. However, they do not use human-authored questions but instead use importance sampling using a heuristic method from Rothe et al. My skepticism of this experiment comes from the fact that  the “decoderless model” that does not take into account the board state performs similar to models that do (and even outperforms the model with LSTM decoder). I am not sure how meaningful this task is.\n- Lastly they show that using REINFORCE with a tweaked reward (a version of the energy function from Rothe et al.) produces new kinds of questions that have not seen before in the training data. This is not surprising and also not very meaningful without either an application or a thorough qualitative analysis of the generated questions.\n\nSo while extending the work of Rothe et al. with neural architectures seems like an interesting endeavour, the tasks the authors use in the paper are severely limited and do not demonstrate interesting behaviour.\n\n\nI believe this is not in scope for changes to this paper, but in general it could be interesting to evaluate the question generation in an end-to-end setup, similar to the reformulation framework used by Nogueira and Cho in “Task-Oriented Query Reformulation with Reinforcement Learning” and Buck et al. in “Ask the Right Questions: Active Question Reformulation with Reinforcement Learning”.\n\nNit: The colours chosen for the diagrams could be improved. In particular purple is quite hard to distinguish from the dark gray.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bye4GluptS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] \nThis paper studies a very interesting problem - if machines can learn to ask the right questions to address or gain crucial information about tasks. I believe this is a very important yet under-explored problem. Specifically, this paper considers a setting where a neural network model is trained to ask questions by predicting a formal program. The network consists of a CNN encoder which encodes a partially observable state and a Transformer decoder that generates a program as a sequence of tokens. The experiments on a Battleship task show some promising results. \n\nSignificance: are the results significant? 3/5\nNovelty: are the problems or approaches novel? 3/5\nEvaluation: are claims well-supported by theoretical analysis or experimental results? 4/5\nClarity: is the paper well-organized and clearly written? 4.5/5\n\n[Strengths]\n\n*motivation*\nThe motivation for investigating machines' ability to generating questions to gain important information is convincing.\n\n*novelty*\nThe idea of utilizing learning models for generating questions and modeling human-generated questions intuitive and convincing. This paper presents an effective way to implement this idea.\n\n*technical contribution*\nLeveraging a context-free grammar (CFG) and reinforcement learning (RL) for question generation seems effective especially when it comes to generating unique and novel questions.\n\n*clarity*\nThe overall writing is clear and the authors utilize figures well to illustrate the ideas. Figure 1 illustrates the Battleship task and Figure 2 presents a clear overview of the proposed framework.\n\n*ablation study*\nAblation studies are comprehensive. The proposed framework consists of multiple components. The provided ablation studies (Table 2 and Table 4) help analyze the effectiveness of each of them.\n\n*experimental results*\n- The presentations of the results are clear. The conclusions are fairly convincing: \n- Experiment 1: learning to generate programmatic questions encourages learning rules for composition\n- Experiment 2: it is possible to capture the human-generated questions distribution as a conditioned language model.\n- Experiment 3: CFG+RL can produce more diverse and novel questions compared to supervised learning or RL.\n\n[Weaknesses]\n\n*novelty & contribution*\nOverall, I do not find enough novelty from any aspects while the overall effort of this paper is appreciated. \n- The problem is not entirely novel as [1-3] have explored asking questions with neural networks learning to generate programmatic questions.\n- The proposed framework leverages a CFG and learns with RL, while the former (program synthesis with a CFG) has been studied in [4-5] and the latter (program synthesis using RL) has been discussed in [4, 6-7].\n- Many setups considered in this paper have been explored in several neural program synthesis papers [8-12], which are neglected from this paper.\n- The conclusions presented in the experiment section are more or less expected (i.e. other works have presented similar results on slightly different problems).\n\n*oversell the motivation*\nI believe the authors oversell the key motivation (i.e. to enable learning agents to ask questions) a little bit. To be more specific, it would be more interesting if an agent is allowed to ask a question, take actions based on the answer, and then ask the next question. However, this paper investigates the case where a significant amount of human-generated questions are given, which could limit the learning in my opinion. Also, it is not clear to me how the setup evaluated in this paper can be extended to a setting that allows an agent to iteratively ask questions.\n\n*experiment setup*\nDo the models learning using RL get the reward based on just the ground truth questions (a sequence of tokens) or the execution results? \n\n*format*\nThe title & reference format looks wrong\n\n*reference*\n[1] Question Asking as Program Generation in NIPS 2017\n[2] Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding in NeurIPS 2018\n[3] The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision in ICLR 2019\n[4] Leveraging grammar and reinforcement learning for neural program synthesis in ICLR 2018\n[5] Learning a Meta-Solver for Syntax-Guided Program Synthesis in ICLR 2019\n[6] Neural Scene De-rendering in CVPR 2017\n[7] Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning\n[8] RobustFill: Neural Program Learning under Noisy I/O in ICML 2017\n[9] Execution-Guided Neural Program Synthesis in ICLR 2019\n[10] Neural Program Synthesis from Diverse Demonstration Videos  in ICML 2018\n[11] Learning to Describe Scenes with Programs in ICLR 2019\n[12] Learning to Infer and Execute 3D Shape Programs in ICLR 2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors explore different ways to generate questions about the current state of a “Battleship” game. Overall the reviewers feel that the problem setting is interesting, and the program generation part is also interesting. However, the proposed approach is evaluated in tangential tasks rather than learning to generate question to achieve the goal. Improving this part is essential to improve the quality of the work. 


Decision: Reject