Venue: NAACL 2019

# Value-based Search in Execution Space for Mapping Instructions to Programs

Dor Muhlgay\({}^{1}\) Jonathan Herzig\({}^{1}\) Jonathan Berant\({}^{1,2}\)

###### Abstract

Training models to map natural language instructions to programs, given target world supervision only, requires searching for good programs at training time. Search is commonly done using beam search in the space of partial programs or program trees, but as the length of the instructions grows finding a good program becomes difficult. In this work, we propose a search algorithm that uses the target world state, known at training time, to train a _critic_ network that predicts the expected reward of every search state. We then score search states on the beam by interpolating their expected reward with the likelihood of programs represented by the search state. Moreover, we search not in the space of programs but in a more compressed state of program executions, augmented with recent entities and actions. On the SCONE dataset, we show that our algorithm dramatically improves performance on all three domains compared to standard beam search and other baselines.

1School of Computer Science, Tel-Aviv University

2Allen Institute for Artificial Intelligence

{dormuhlg@mail,jonathan.herzig@cs,joberant@cs}.tau.ac.il

## 1 Introduction

Training models that can understand natural language instructions and execute them in a real-world environment is of paramount importance for communicating with virtual assistants and robots, and therefore has attracted considerable attention Branavan et al. (2009); Vogel and Jurafsky (2010); Chen and Mooney (2011). A prominent approach is to cast the problem as semantic parsing, where instructions are mapped to a high-level programming language Artzi and Zettlemoyer (2013); Long et al. (2016); Guu et al. (2017). Because annotating programs at scale is impractical, it is desirable to train a model from instructions, an initial world state, and a target world state only, letting the program itself be a latent variable.

Learning from such weak supervision results in a difficult search problem at training time. The model must search for a program that when executed leads to the correct target state. Early work employed lexicons and grammars to constrain the search space Clarke et al. (2010); Liang et al. (2011); Krishnamurthy and Mitchell (2012); Berant et al. (2013); Artzi and Zettlemoyer (2013), but recent success of sequence-to-sequence models Sutskever et al. (2014) shifted most of the burden to learning. Search is often performed simply using beam search, where program tokens are emitted from left-to-right, or program trees are generated top-down Krishnamurthy et al. (2017); Yin and Neubig (2017); Cheng et al. (2017); Rabinovich et al. (2017) or bottom-up Liang et al. (2017); Guu et al. (2017); Goldman et al. (2018). Nevertheless, when instructions are long and complex and reward is sparse, the model may never find enough correct programs, and training will fail.

In this paper, we propose a novel search algorithm for mapping a sequence of natural language instructions to a program, which extends the standard beam-search in two ways. First, we capitalize on the target world state being available at training time and train a _critic_ network that given the language instructions, current world state, and target world state estimates the expected future reward for each search state. In contrast to traditional beam search where states representing partial programs are scored based on their likelihood only, we also consider expected future reward, leading to a more targeted search at training time. Second, rather than search in the space of programs, we search in a more compressed execution space, where each state is defined by a partial program's execution result.

We evaluated our method on the SCONE dataset, which includes three different domains where long sequences of 5 instructions are mapped to programs. We show that while standard beam

search gets stuck in local optima and is unable to discover good programs for many examples, our model is able to bootstrap, improving final performance by 20 points on average. We also perform extensive analysis and show that both value-based search as well as searching in execution space contribute to the final performance. Our code and data are available at [http://gitlab.com/tau-nlp/vbsix-lang2program](http://gitlab.com/tau-nlp/vbsix-lang2program).

## 2 Background

Mapping instructions to programs invariably involves a _context_, such as a database or a robotic environment, in which the program (or logical form) is executed. The goal is to train a model given a training set \(\{(x_{(j)}=(e_{(j)},\mathbf{u}_{(j)}),y_{(j)})\}^{N}_{j=1}\), where \(c\) is the context, \(\mathbf{u}\) is a sequence of natural language instructions, and \(y\) is the target state of the environment after following the instructions, which we refer to as _denotation_. The model is trained to map the instructions \(\mathbf{u}\) to a program \(\mathbf{z}\) such that executing \(\mathbf{z}\) in the context \(c\) results in the denotation \(y\), which we denote by \(\llbracket\mathbf{z}\rrbracket_{c}=y\). Thus, the program \(\mathbf{z}\) is a latent variable we must search for at both training and test time. When the sequence of instructions is long, search becomes hard, particularly in the early stages of training.

Recent work tackled the training problem using variants of reinforcement learning (RL) Suhr and Artzi (2018); Liang et al. (2018) or maximum marginal likelihood (MML) Guu et al. (2017); Goldman et al. (2018). We now briefly describe MML training, which we base our training procedure on, and outperformed RL in past work under comparable conditions Guu et al. (2017).

We denote by \(\pi_{\theta}(\cdot)\) a model, parameterized by \(\theta\), that generates the program \(\mathbf{z}\) token by token from left to right. The model \(\pi_{\theta}\) receives the context \(c\), instructions \(\mathbf{u}\) and previously predicted tokens \(z_{1...t-1}\), and returns a distribution over the next program token \(z_{t}\). The probability of a program prefix is defined to be: \(p_{\theta}(z_{1...t}~{}|~{}\mathbf{u},c)=\prod_{i=1}^{t}\pi_{\theta}(z_{i}~{}| ~{}\mathbf{u},c,z_{1...i-1})\). The model is trained to maximize the MML objective:

\[J_{MML}=\log\prod_{(c,\mathbf{u},y)}p_{\theta}(y~{}|~{}c, \mathbf{u})=\] \[\sum_{(c,\mathbf{u},y)}\log(\sum_{\mathbf{z}}p_{\theta}(\mathbf{ z}~{}|~{}\mathbf{u},c)\cdot R(\mathbf{z})),\]

where \(R(\mathbf{z})=1\) if \(\llbracket\mathbf{z}\rrbracket_{c}=y\), and 0 otherwise (For brevity, we omit \(c\) and \(y\) from \(R(\cdot)\)). Typically, the MML objective is optimized with stochastic gradient ascent, where the gradient for an example \((c,\mathbf{u},y)\) is:

\[\nabla_{\theta}J_{MML}=\sum_{z}q(z)\cdot R(z)\nabla_{\theta}\log p_{\theta}(z |c,\mathbf{u})\]

\[q(z):=\frac{R(z)\cdot p_{\theta}(z|c,\mathbf{u})}{\sum_{\tilde{z}}R(\tilde{z} )\cdot p_{\theta}(\tilde{z}|c,\mathbf{u})}.\]

The search problem arises because it is impossible to enumerate the set of all programs, and thus the sum over programs is approximated by a small set of high probability programs, which have high weights \(q(\cdot)\) that dominate the gradient. Search is commonly done with _beam-search_, an iterative algorithm that builds an approximation of the highest probability programs according to the model. At each time step \(t\), the algorithm constructs a beam \(B_{t}\) of at most \(K\) program prefixes of length \(t\). Given a beam \(B_{t-1}\), \(B_{t}\) is constructed by selecting the \(K\) most likely _continuations_ of prefixes in \(B_{t-1}\) , according to \(p_{\theta}(z_{1..t}|\cdot)\). The algorithm runs \(L\) iterations, and returns all complete programs discovered.

In this paper, our focus is the search problem that arises at training time when training from denotations, i.e., finding programs that execute to the right denotation. Thus, we would like to focus on scenarios where programs are long, and standard beam search fails. We next describe the SCONE dataset, which provides such an opportunity.

Figure 1: Example from the Scene domain in SCONE (3 utterances), where people with different hats and shirts are located in different positions. The example contains (from top to bottom): an initial world (the people’s properties, from left to right: a blue shirt with an orange hat, a blue shirt with a red hat, and a green shirt with a yellow hat), a sequence of natural language instructions, and a target world (the people’s properties, from left to right: blue shirt with a red hat, green shirt with a yellow hat, and an orange shirt).

 

The SCONE datasetLong et al. (2016) presented the SCONE dataset, where a sequence of instructions needs to be mapped to a program consisting of a sequence of executable commands. The dataset has three domains, where each domain includes several objects (such as _people_ or _beakers_), each with different properties (such as _shirt color_ or _chemical color_). SCONE provides a good environment for stress-testing search algorithms because a long sequence of instructions needs to be mapped to a program. Figure 1 shows an example from the Scene domain.

Formally, the context in SCONE is a _world_ specified by a list of positions, where each position may contain an object with certain properties. A formal language is defined to interact with the world. The formal language contains _constants_ (e.g., numbers and colors), _functions_ that allow to query the world and refer to objects and intermediate computations, and _actions_, which are functions that modify the world state. Each command is composed of a single action and several arguments constructed recursively from constants and functions. E.g., the command move(hasHat(yellow), leftOf(HasShirt(blue))), contains the action move, which moves a person to a specified position. The person is computed by hasHat(yellow), which queries the world for the position of the person with a yellow hat, and the target position is computed by leftOf(hasShirt(blue)). We refer to Guu et al. (2017) for a full description of the language.

Our goal is to train a model that given an initial world \(w^{0}\) and a sequence of natural language _utterances_\(\mathbf{u}=(u^{1},\ldots,u^{M})\), will map each utterance \(u^{i}\) to a command \(z^{i}\) such that applying the program \(\mathbf{z}=(z^{1},\ldots,z^{M})\) on \(w^{0}\) will result in the target world, i.e., \(\llbracket\mathbf{z}\rrbracket_{w^{0}}=w^{M}=y\).

## 3 Markov Decision Process Formulation

To present our algorithm, we first formulate the problem as a Markov Decision Process (MDP), which is a tuple \((\mathcal{S},\mathcal{A},R,\delta)\). To define the state set \(\mathcal{S}\), we assume all program prefixes are executable, which can be easily done as we show below. The _execution result_ of a prefix \(\tilde{z}\) in the context \(c\), denoted by \([\![\tilde{z}]\!]_{c}^{ex}\), contains its denotation and additional information stored in the executor. Let \(\mathcal{Z}_{\text{pre}}\) be the set of all valid programs prefixes. The set of states is defined to be \(\mathcal{S}=\{(x,[\![\tilde{z}]\!]_{c}^{ex})\mid\tilde{z}\in\mathcal{Z}_{ \text{pre}}\}\), i.e., the input paired with all possible execution results.

The action set \(\mathcal{A}\) includes all possible program tokens 1, and the transition function \(\delta:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}\) is computed by the executor. Last, the reward \(R(s,a)=1\) iff the action \(a\) ends the program and leads to a state \(\delta(s,a)\) where the denotation is equal to the target \(y\). The model \(\pi_{\theta}(\cdot)\) is a parameterized policy that provides a distribution over the program vocabulary at each time step.

Footnote 1: Decoding is constrained to valid program tokens only.

SCONE as an MDPWe define the partial execution result \([\![\tilde{z}]\!]_{c}^{ex}\) in SCONE, as described by Guu et al. (2017). We assume that SCONE's formal language is written in postfix notations, e.g., the instruction move(hasHat(yellow), leftOf(hasShirt(blue))) is written as vellowhashHat blue ha

Figure 2: An illustration of a state transition in SCONE. \(\pi_{\theta}\) predicted the action token move. Before the transition (left), the command history is empty and the program stack contains the arguments 3 and 1, which were computed in previous steps. Note that the state does not include the partial program that computed those arguments. After transition (right), the executor popped the arguments from the program stack and applied the action Move 3 1: the man in position 3 (a person with a yellow hat and a red shirt) moved to position 1 (to the left of the person with the blue shirt), and the action is added to the execution history with its arguments. Since this action terminated the command, \(\pi_{\theta}\) advanced to the next utterance.

 

With this notation, a partial program can be executed left-to-right by maintaining a _program stack_, \(\psi\). The executor pushes constants (yellow) to \(\psi\), and applies functions (hasHat) by popping their arguments from \(\psi\) and pushing back the computed result. Actions (move) are applied by popping arguments from \(\psi\) and performing the action in the current world.

To handle references to previously executed commands, SCONE's formal language includes functions that provide access to actions and arguments in previous commands. To this end, the executor maintains an _execution history_, \({\bf h}^{i}=(e^{1},\ldots,e^{i})\), a list of executed actions and their arguments. Thus, the execution result of a program prefix is \([\![\tilde{z}]\!]_{w^{0}}^{ex}=(w^{i-1},\psi,{\bf h}^{i-1})\).

We adopt the model from Guu et al. (2017) (architecture details in appendix A): The policy \(\pi_{\theta}\) observes \(\psi\) and \(u^{i}\), the current utterance being parsed, and predicts a token. When the model predicts an action token that terminates a command, the model moves to the next utterance (until all utterances have been processed). The model uses functions to query the world \(w^{i-1}\) and history \({\bf h}^{i-1}\). Thus, each MDP state in SCONE is a pair \(s=(u^{i},[\![\tilde{z}]\!]_{w^{0}}^{ex})\). Figure 2 illustrates a state transition in the Scene domain. Importantly, the state does not store the full program's prefix, and many different prefixes may lead to the same state. Next, we describe a search algorithm for this MDP.

## 4 Searching in Execution Space

Model improvement relies on generating correct programs given a possibly weak model. Standard beam-search explores the space of all program token sequences up to some fixed length. We propose two technical contributions to improve search: (a) We simplify the search problem by searching for correct executions rather than correct programs; (b) We use the target denotation at training time to better estimate partial program scores in the search space. We describe those next.

### Reducing program search to execution search

Program space can be formalized as a directed tree \(T=(\mathcal{V}_{T},\mathcal{E}_{T})\), where vertices \(\mathcal{V}_{T}\) are program prefixes, and labeled edges \(\mathcal{E}_{T}\) represent prefixes' continuations: an edge \(e=(\tilde{z},\hat{z}^{\prime})\) labeled with the token \(a\), represents a continuation of the prefix \(\tilde{z}\) with the token \(a\), which yields the prefix \(\hat{z}^{\prime}\). The root of the graph represents the empty sequence. Similarly, _Execution space_ is a directed graph \(G=(\mathcal{V}_{G},\mathcal{E}_{G})\) induced from the MDP described in Section 3. Vertices \(\mathcal{V}_{G}\) represent MDP states, which express execution results, and labeled edges \(\mathcal{E}_{G}\) represent transitions. An edge \((s_{1},s_{2})\) labeled by token \(a\) means that \(\delta(s_{1},a)=s_{2}\). Since multiple programs have the same execution result, execution space is a compressed representation of program space. Figure 3 shows a few commands in both program and execution space. Execution space is smaller, and so searching in it is easier.

Each path in execution space represents a different program prefix, and the path's final state represents its execution result. Program search can therefore be reduced to execution search: given an example \((c,{\bf u},y)\) and a model \(\pi_{\theta}\), we can use \(\pi_{\theta}\) to explore in execution space, discover _correct terminal states_, i.e. states corresponding to correct full programs, and extract paths leading to those states. As the number of paths may be exponential in the size of the graph, we can use beam-search to extract the most probable correct programs (according to the model) in the discovered graph.

Our approach is similar to the DPD algorithm Pasupat and Liang (2016), where CKYstyle search is performed in denotation space, followed by search in a pruned space of programs. However, DPD was used without learning, and the search was not guided by a trained model, which is a major part of our algorithm.

### Value-based Beam Search in Execution Space

We propose **Val**ue-based **Beam **S**earch in eXecution space (VBS1X), a variant of beam

Figure 3: A set of commands represented in program space (left) and execution space (right). The commands relate to the first world in Figure 2. Since multiple prefixes have the same execution result (e.g., yellow hasHat and redhashirt), the execution space is smaller.

 

search modified for searching in execution space, that scores search states with a value-based network. VBS1X is formally defined in Algorithm 1, which we will refer to throughout this section.

Standard beam search is a breadth-first traversal of the program space tree, where a fixed number of vertices are kept in the beam at every level of the tree. The selection of vertices is done by scoring their corresponding prefixes according to \(p_{\theta}(z_{1...t}~{}|~{}\mathbf{u},c)\). VBS1X applies the same traversal in execution space (lines 10-21). However, since each vertex in the execution space represents an execution result and not a particular prefix, we need to modify the scoring function.

Let \(s\) be a vertex discovered in iteration \(t\) of the search. We propose two scores for ranking \(s\). The first is _the actor score_, the probability of reaching vertex \(s\) after \(t\) iterations2 according to the model \(\pi_{\theta}\). The second and more novel score is the value-based _critic score_, an estimate of the state's expected reward. The AC-Score is the sum of these two scores (lines 23-24).

Footnote 2: We score paths in different iterations independently to avoid bias for shorter paths. An MDP state that appears in multiple iterations will get a different score in each iteration.

The actor score, \(p^{t}_{\theta}(s)\), is the sum of probabilities of all prefixes of length \(t\) that reach \(s\) (rather than the probability of one prefix as in beam search). VBS1X approximates \(p^{t}_{\theta}(s)\) by performing the summation only over paths that reach \(s\) via states in the beam \(B_{t-1}\), which can be done efficiently with a dynamic programming (DP) chart \(P_{t}[s]\) that keeps actor score approximations in each iteration (line 18). This lower-bounds the true \(p^{t}_{\theta}(s)\) since some prefixes of length \(t\) that reach \(s\) might have not been discovered.

Contrary to standard beam-search, we want to score search states also with a critic score \(\mathbb{E}_{p_{\theta}}[R(s)]\), which is the sum of the suffix probabilities that lead from \(s\) to a correct terminal state:

\[\mathbb{E}_{p_{\theta}}[R(s)]=\sum_{\tau(s)}p_{\theta}(\tau(s)~{}|~{}s)\cdot R (\tau(s)),\]

where \(\tau(s)\) are all possible trajectories starting from \(s\) and \(R(\tau(s))\) is the reward observed when taking the trajectory \(\tau(s)\) from \(s\). Enumerating all trajectories \(\tau(s)\) is intractable and so we will approximate \(\mathbb{E}_{p_{\theta}}[R(s)]\) with a trained value network \(V_{\phi}(s,y)\), parameterized by \(\phi\). Importantly, because we are searching at training time, we can condition \(V_{\phi}\) on both the current state \(s\) and target denotation \(y\). At test time we will use \(\pi_{\theta}\) only, which does not need access to \(y\).

Since the value function and DP chart are used for efficient ranking, the asymptotic run-time complexity of VBS1X is the same as standard beam search (\(\mathcal{O}(K\cdot|\mathcal{A}|\cdot L)\)). The beam search in Line 3, where we extract programs from the constructed execution space graph, can be done with a small beam size, since it operates over a small space of correct programs. Thus, its contribution to the algorithm complexity is negligible.

Figure 4 visualizes scoring and pruning with the actor-critic score in iteration \(t\). Vertices in \(B_{t}\) are discovered by expanding vertices in \(B_{t-1}\), and each vertex is ranked by the AC-score. The highlighted vertex score is \(0.6\), a sum of the actor score (\(0.5\)) and the critic score (\(0.1\)). The actor score is the sum of its prefixes (\((0.05+0.55)\cdot 0.7+0.01\cdot\log(1/2)\)). The actor score is

Figure 4: An illustration of scoring and pruning states in step \(t\) of VBS1X. Discovered edges are in full edges, undiscovered edges are dashed, correct terminal states are dashed and in yellow, and states that are kept in the beam are in dark grey. The actor scores of the highlighted vertex (\(0.5\)) and its parents (\(0.6\), \(0.1\)) are in blue, and its critic score (\(0.1\)) is in green.

 

\(0.08=0.5\)) and the critic score is a value network estimate for the sum of probabilities of outgoing trajectories reaching correct terminal states (\(0.02+0.08=0.1\)). Only the top-\(K\) states are kept in the beam (\(K=4\) in the figure).

VBSiX leverages execution space in a number of ways. First, since each vertex in execution space compactly represents multiple prefixes, a beam in VBSiX effectively holds more prefixes than in standard beam search. Second, running beam-search over a graph rather than a tree is less greedy, as the same vertex can surface back even if it fell out of the beam.

The value-based approach has several advantages as well: First, evaluating the probability of outgoing trajectories provides look-ahead that is missing from standard beam search. Second (and most importantly), \(V_{\phi}\) is conditioned on \(y\), which \(\pi_{\theta}\) doesn't have access to, which allows finding correct programs with low model probability, that \(\pi_{\theta}\) can learn from. We note that our two contributions are orthogonal: the critic score can be used in program space, and search in execution space can be done with actor-scores only.

## 5 Training

We train the model \(\pi_{\theta}\) and value network \(V_{\phi}\) jointly (Algorithm 2). \(\pi_{\theta}\) is trained using MML over discovered correct programs [39]. The value network is trained as follows: Given a training example \((c,\mathbf{u},y)\), we generate a set of correct programs \(\mathcal{Z}_{\text{pos}}\) with VB-S1X. The value network needs negative examples, and so for every incorrect terminal state \(z_{\text{neg}}\) found during search with VBSiX we create a single program leading to \(z_{\text{neg}}\). We then construct a set of training examples \(\mathcal{D}_{v}\), where each example \(((s,y),l)\) labels states encountered while generating programs \(z\in\mathcal{Z}\) with the probability mass of correct programs suffixes that extend it, i.e., \(l=\sum_{z_{t}}p_{\theta}(z_{t...|z|})\), where \(z_{t}\) ranges over all \(z\in\mathcal{Z}\) and \(t\in[1\ldots|z|]\) Finally, we train \(V_{\phi}\) to minimize the log-loss objective: \(\sum_{((s,y),l)\in\mathcal{D}_{v}}l\cdot\log V_{\phi}(s,y)+(1-l)(\log(1-V_{ \phi}(s,y)))\,\).

```
1:procedureTrain()
2: Initialize \(\theta\) and \(\phi\) randomly
3:while\(w\) not converged do
4:\((x:(z_{t},a_{t}))>-\) select random example
5:\(\mathcal{Z}_{\text{pos}}\leftarrow\textsc{ProgramSearch}(c,\mathbf{u},\mathbf{ y},\sigma_{\theta},V_{\phi})\)
6:\(\mathcal{Z}_{\text{pos}}\gets\textsc{programIsading in incorrect terminal states}\)
7:\(\mathcal{D}_{v}\leftarrow\textsc{BuildValueExample}((\mathcal{Z}_{\text{ pos}})\cup\mathcal{Z}_{\text{pos}})\), \(c,y)\)
8: Update \(\phi\) using \(T_{v}\), update \(q\) using \((x,\mathcal{Z}_{\text{pos}},y)\)
9:functionBullVALUEExamples(\(\mathcal{Z},c,\mathbf{u},y\))
10:for\(z\in\mathcal{Z}\)do
11:for\(t\in[1\ldots|z|]\)do
12:\(s^{*}\leftarrow[1..|z|]^{*}\)
13:\(L[s]\gets L[s]+p_{\theta}(z_{t...|t|}\mid c,\mathbf{u})\cdot R(z)\)
14:\(\mathcal{D}_{v}\leftarrow\left\{((s,y),L[s])\right\}_{s\in L}\)
15: Return \(\mathcal{D}_{v}\)
```

**Algorithm 2** Actor-Critic Training

Similar to actor score estimation, labeling examples for \(V_{\phi}\) is affected by beam-search errors: the labels lower bound the true expected reward. However, since search is guided by the model, those programs are likely to have low probability. Moreover, estimates from \(V_{\phi}\) are based on multiple examples, compared to probabilities in the DP chart, and are more robust to search errors.

Neural network architecture:We adapt the model proposed by guu2017deep for SCONE. The model receives the current utterance \(u^{i}\) and program stack \(\psi\), and returns a distribution over the next token. Our value network receives the same input, but also the next utterance \(u^{i+1}\), the world state \(w^{i}\) and target world state \(y\), and outputs a scalar. Appendix A provides a full description.

## 6 Experiments

### Experimental setup

We evaluated our method on the three domains of SCONE with the standard accuracy metric, i.e., the proportion of test examples where the predicted program has the correct denotation. We trained with VBSiX, and used standard beam search (\(K=32\)) at test time for programs' generation. Each test example contains 5 utterances, and similar to prior work we reported the model accuracy on all 5 utterances as well as the first 3 utterances. We ran each experiment 6 times with different random seeds and reported the average accuracy and standard deviation.

In contrast to prior work on SCONE [19; 20], where models were trained on all sequences of 1 or 2 utterances, and thus were exposed during training to all gold intermediate states, we trained from longer sequences keeping intermediate states latent. This leads to a harder search problem that was not addressed previously, but makes our results incomparable to previous results 3. In Scene and Tangram, we used the first 4 and 5 utterances as examples. In Alchemy, we used the first utterance and 5 utterances.

 

Training detailsTo warm-start the value network, we trained it for a few thousand steps, and only then start re-ranking with its predictions. Moreover, we gain efficiency by first returning \(K_{0}\)(=128) states with the actor score, and then re-ranking with the actor-critic score, returning \(K\)(=32) states. Last, we use the value network only in the last two utterances of every example since we found it has less effect in earlier utterances where future uncertainty is large. We used the Adam optimizer Kingma and Ba (2014) and fixed GloVe embeddings Pennington et al. (2014) for utterance words.

BaselinesWe evaluated the following training methods (Hyper-parameters are in appendix B):

1. MML: Our main baseline, where search is done with beam search and training with MML. We used randomized beam-search, which adds \(\epsilon\)-greedy exploration to beam search, which was proposed by Guu et al. (2017) and performed better 4.

Footnote 4: We did not include meritocratic updates Guu et al. (2017), since it performed worse in initial experiments.

2. Expert-MML: An alternative way of using the target denotation \(y\) at training time, based on imitation learning Daume et al. (2009); Ross et al. (2011); Berant and Liang (2015), is to train an _expert policy_\(\pi^{\text{expert}}_{\theta}\), which receives \(y\) as input in addition to the parsing state, and trains with the MML objective. Then, our policy \(\pi_{\theta}\) is trained using programs found by \(\pi^{\text{expert}}_{\theta}\). The intuition is that the expert can use \(y\) to find good programs that the policy \(\pi_{\theta}\) can train from.

3. VBSiX: Our proposed training algorithm.

We also evaluated REINFORCE, where MonteCarlo sampling is used as search strategy Williams (1992); Sutton et al. (1999). We followed the implementation of Guu et al. (2017), who performed variance reduction with a constant baseline and added \(\epsilon\)-greedy exploration. We found that REINFORCE fails to discover any correct programs to bootstrap from.

### Results

Table 1 reports test accuracy of VBSiX compared to the baselines. First, VBSiX outperforms all baselines in all cases. MML is the strongest baseline, but even with an increased beam (\(K=64\)), VBSiX (\(K=32\)) surpasses it by a large margin (more than 20 points on average). On top of the improvement in accuracy, in Alchemy and Tangram the standard deviation of VBSiX is lower than the other baselines across the 6 random seeds, showing the robustness of our model.

Expert-MML performs worse than MML in all cases. We hypothesize that using the denotation \(y\) as input to the expert policy \(\pi^{\text{expert}}_{\theta}\) results in many spurious programs, i.e., they are unrelated to the utterance meaning. This is since the expert can learn to perform actions that take it to the target world state while ignoring the utterances completely. Such programs will lead to bad generalization of \(\pi_{\theta}\). Using a critic at training time eliminates this problem, since its scores depend on \(\pi_{\theta}\).

AblationsWe performed ablations to examine the benefit of our two technical contributions (a) execution space (b) value-based search. Table 2 presents accuracy on the validation set when

\begin{table}
\begin{tabular}{l|c|p{28.5pt} p{28.5pt}|p{28.5pt}|p{28.5pt}|p{28.5pt}|p{28.5pt}}  & \multicolumn{2}{c|}{**Scene**} & \multicolumn{2}{c|}{**Alchemy**} & \multicolumn{2}{c}{**Tangram**} \\  & **Beam** & **3 utt** & **5 out** & **3 att** & **5 int** & **3 utt** & **5 utt** \\ \hline MML & 32 & 8.4\(\pm\)(2.0) & 7.2\(\pm\)(1.3) & 41.9\(\pm\)(22.8) & 33.2\(\pm\)(20.1) & 32.5\(\pm\)(20.7) & 16.8\(\pm\)(14.1) \\  & 64 & 15.4\(\pm\)(12.6) & 12.3\(\pm\)(9.6) & 44.6\(\pm\)(23.7) & 36.3\(\pm\)(20.6) & 45.6\(\pm\)(18.0) & 25.8\(\pm\)(12.6) \\ \hline Expert-MML & 32 & 1.8\(\pm\)(1.5) & 1.6\(\pm\)(1.2) & 29.4\(\pm\)(22.7) & 23.1\(\pm\)(18.8) & 2.4\(\pm\)(0.8) & 1.2\(\pm\)(0.5) \\ VBSiX & 32 & **34.2\(\pm\)(27.5)** & **28.2\(\pm\)**(20.7) & **74.5\(\pm\)**(1.1) & **64.8\(\pm\)**(1.5) & **65.0\(\pm\)**(0.8) & **43.0\(\pm\)**(1.3) \\ \end{tabular}
\end{table}
Table 1: Test accuracy and standard deviation of VBSiX compared to MML baselines (top) and our training methods (bottom). We evaluate the same model over the first 3 and 5 utterances in each domain.

\begin{table}
\begin{tabular}{l|l|l|l|l|l|l|l}  & \multicolumn{2}{c|}{**Scene**} & \multicolumn{2}{c|}{**ALchemy**} & \multicolumn{2}{c}{**Tangram**} \\
**Search space** & **Value** & **3 utt** & **5 cutt** & **3 utt** & **5 utt** & **3 utt** & **5 utt** \\ \hline Program Execution & No & 5.5\(\pm\)(\(0.5\)) & 3.8\(\pm\)(\(0.6\)) & 36.4\(\pm\)(\(26.5\)) & 25.4\(\pm\)(\(23.0\)) & 34.4\(\pm\)(\(18.3\)) & 15.6\(\pm\)(\(12.8\)) \\ Execution & No & 7.4\(\pm\)(\(10.4\)) & 4.0\(\pm\)(\(5.8\)) & 41.3\(\pm\)(\(28.6\)) & 28.2\(\pm\)(\(23.27\)) & 33.5\(\pm\)(\(15.5\)) & 12.7\(\pm\)(\(10.4\)) \\ Program & Yes & 7.6\(\pm\)(\(8.3\)) & 3.4\(\pm\)(\(2.9\)) & 78.5\(\pm\)(\(1.0\)) & 72.8\(\pm\)(\(1.3\)) & 66.8\(\pm\)(\(1.5\)) & 42.8\(\pm\)(\(1.9\)) \\ Execution & Yes & **31.0\(\pm\)**(\(24.7\)) & **22.6\(\pm\)**(\(19.6\)) & **81.9\(\pm\)**(\(1.3\)) & **75.2\(\pm\)**(\(2.9\)) & **68.6\(\pm\)**(\(2.0\)) & **44.2\(\pm\)(\(2.1\)) \\ \end{tabular}
\end{table}
Table 2: Validation accuracy when ablating the different components of VBSiX. The first line presents MML, the last line is VBSiX, and the intermediate lines examine execution space and value-based networks separately.

 

each component is used separately, when both of them are used (VBSiX), and when none are used (beam-search). We find that both contributions are important for performance, as the full system achieves the highest accuracy in all domains. In Scene, each component has only a slight advantage over beam-search, and therefore both are required to achieve significant improvement. However, in Alchemy and Tangram most of the gain is due to the value network.

We also directly measured the _hit accuracy_ at training time, i.e., the proportion of training examples where the beam produced by the search algorithm contains a program with the correct denotation, showing the effectiveness of search at training time. In Figure 5, we report train hit accuracy in each training step, averaged across 6 random seeds. The graphs illustrate the performance of each search algorithm in every domain throughout training. The validation accuracy results are correlated with the improvement in train hit-accuracy.

### Analysis

Execution SpaceWe empirically measured two quantities that we expect should reflect the advantage of execution-space search. First, we measured the number of programs stored in the execution space graph compared to beam search, which holds \(K\) programs. Second, we counted the average number of states that are connected to correct terminal states in the discovered graph, but fell out of the beam during the search. The property reflects the gain from running search over a graph structure, where the same vertex can resurface. We preformed the analysis on VBSiX over 5-utterance training examples in all 3 domains. The following table summarizes the results:

We found the measured properties and the contribution of execution space in each domain are correlated, as seen in the ablations. Differences between domains are due to the different complexities of their formal languages. As the formal language becomes more expressive, the execution space is more compressed as each state can be reached in more ways. In particular, the formal language in Scene contains more functions compared to the other domains, and so it benefits the most from execution-space search.

Value NetworkWe analyzed the accuracy of the value network at training time by measuring, for each state, the difference between its expected reward (estimated from the discovered paths) and the value network prediction. Figure 6 shows the average difference in each training step for all encountered states (in blue), and for high reward states only (states with expected reward larger than \(0.7\), in orange). Those metrics are averaged across 6 runs with different random seeds.

The accuracy of the value network improves during training, except when the policy changes substantially, in which case the value network needs to re-evaluate the policy. When the value network converges, the difference between its predictions and the expected reward is \(0.15-0.2\) on average. However, for high reward states the difference is higher (\(\sim 0.3\)). This indicates that the value network has a bias toward lower values, which is expected as most states lead to low rewards. Since VBSiX uses the value network as a beam-search ranker, the value network doesn't need to be exact as long as it assigns higher values to states with higher expected rewards. Further analysis is provided in appendix D.

Figure 5: Training hit accuracy on examples with 5 utterances, comparing VBSiX to baselines with ablated components. The results are averaged over 6 runs with different random seeds (best viewed in color)





## 7 Related Work

Training from denotations has been extensively investigated Kwiatkowski et al. (2013); Pasupat and Liang (2015); Bisk et al. (2016), with a recent emphasis on neural models Neelakantan et al. (2016); Krishnamurthy et al. (2017). Improving beam search has been investigated by proposing specialized objectives Wiseman and Rush (2016), stopping criteria Yang et al. (2018), and using continuous relaxations Goyal et al. (2018).

Bahdanau et al. (2017) and Suhr and Artzi (2018) proposed ways to evaluate intermediate predictions from a sparse reward signal. Bahdanau et al. (2017) used a critic network to estimate expected BLEU in translation, while Suhr and Artzi (2018) used edit-distance between the current world and the goal for SCONE. But, in those works stronger supervision was assumed: Bahdanau et al. (2017) utilized the gold sequences, and Suhr and Artzi (2018) used intermediate worlds states. Moreover, intermediate evaluations were used to compute gradient updates, rather than for guiding search.

Guiding search with both policy and value networks was done in Monte-Carlo Tree Search (MCTS) for tasks with a sparse reward Silver et al. (2017); T. A. and Barber (2017); Shen et al. (2018). In MCTS, value network evaluations are refined with backup updates to improve policy scores. In this work, we gain this advantage by using the target denotation. The use of an actor and a critic is also reminiscent of \(A^{*}\) where states are scored by past cost and an admissible heuristic for future cost Klein and Manning (2003); Pauls and Klein (2009); lee et al. (2016). In semantic parsing, Misra et al. (2018) recently proposed a critic distribution to improve the policy, which is based on prior domain knowledge (that is not learned).

## 8 Conclusions

In this work, we propose a new training algorithm for mapping instructions to programs given denotation supervision only. Our algorithm exploits the denotation at training time to train a critic network used to rank search states on the beam, and performs search in a compact execution space rather than in the space of programs. We evaluated on three different domains from SCONE, and found that it dramatically improves performance compared to strong baselines across all domains.

VBSiX is applicable to any task that supports graph-search exploration. Specifically, for tasks that can be formulated as an MDP with a deterministic transition function, which allow efficient execution of multiple partial trajectories. Those tasks include a wide range of instruction mapping Branavan et al. (2009); Vogel and Jurafsky (2010); Anderson et al. (2018) and semantic parsing tasks Dahl et al. (1994); Iyyer et al. (2017); Yu et al. (2018). Therefore, evaluating VBSiX on other domains is a natural next step for our research.

## Acknowledgments

We thank the anonymous reviewers for their constructive feedback. This work was completed in fulfillment for the M.Sc degree of the first author. This research was partially supported by The Israel Science Foundation grant 942/16, the Blavatnik Computer Science Research Fund, and The Yandex Initiative for Machine Learning.

Figure 6: The difference between the prediction of the value network and the expected reward (estimated from the discovered paths) during training. We report the average difference for all of the states (blue) and for the high reward states only (\(>0.7\), orange). The results are averaged over 6 runs with different random seeds (best viewed in color).





## References

* Anderson et al. (2018) P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sunderhauf, I. Reid, S. Gould, and A. van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In _Computer Vision and Pattern Recognition (CVPR)_.
* Artzi and Zettlemoyer (2013) Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics (TACL)_, 1:49-62.
* Bahdanau et al. (2017) D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio. 2017. An actor-critic algorithm for sequence prediction. In _International Conference on Learning Representations (ICLR)_.
* Berant et al. (2013) J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Berant and Liang (2015) J. Berant and P. Liang. 2015. Imitation learning of agenda-based semantic parsers. _Transactions of the Association for Computational Linguistics (TACL)_, 3:545-558.
* Bisk et al. (2016) Y. Bisk, D. Yuret, and D. Marcu. 2016. Natural language communication with robots. In _North American Association for Computational Linguistics (NAACL)_.
* Branavan et al. (2009) S. Branavan, H. Chen, L. S. Zettlemoyer, and R. Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In _Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP)_, pages 82-90.
* Chen and Mooney (2011) D. L. Chen and R. J. Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In _Association for the Advancement of Artificial Intelligence (AAAI)_, pages 859-865.
* Cheng et al. (2017) J. Cheng, S. Reddy, V. Saraswat, and M. Lapata. 2017. Learning structured natural language representations for semantic parsing. In _Association for Computational Linguistics (ACL)_.
* Clarke et al. (2010) J. Clarke, D. Goldwasser, M. Chang, and D. Roth. 2010. Driving semantic parsing from the world's response. In _Computational Natural Language Learning (CoNLL)_, pages 18-27.
* Dahl et al. (1994) D. A. Dahl, M. Bates, M. Brown, W. Fisher, K. Hunicke-Smith, D. Pallett, C. Pao, A. Rudnicky, and E. Shriberg. 1994. Expanding the scope of the ATIS task: The ATIS-3 corpus. In _Workshop on Human Language Technology_, pages 43-48.
* Daume et al. (2009) H. Daume, J. Langford, and D. Marcu. 2009. Search-based structured prediction. _Machine Learning_, 75:297-325.
* Fried et al. (2018) D. Fried, J. Andreas, and D. Klein. 2018. Unified pragmatic models for generating and following instructions. In _North American Association for Computational Linguistics (NAACL)_.
* Goldman et al. (2018) O. Goldman, V. Latcinnik, U. Naveh, A. Globerson, and J. Berant. 2018. Weakly-supervised semantic parsing with abstract examples. In _Association for Computational Linguistics (ACL)_.
* Goyal et al. (2018) K. Goyal, G. Neubig, C. Dyer, and T. Berg-Kirkpatrick. 2018. A continuous relaxation of beam search for end-to-end training of neural sequence models. In _Association for the Advancement of Artificial Intelligence (AAAI)_.
* Guu et al. (2017) K. Guu, P. Pasupat, E. Z. Liu, and P. Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In _Association for Computational Linguistics (ACL)_.
* Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Wyer et al. (2017) Mohit Wyer, Wen-tau Yih, and Ming-Wei Chang. 2017. Search-based neural structured learning for sequential question answering. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 1821-1831.
* Kingma and Ba (2014) D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Klein and Manning (2003) D. Klein and C. Manning. 2003. A* parsing: Fast exact viterbi parse selection. In _Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL)_.
* Krishnamurthy et al. (2017) J. Krishnamurthy, P. Dasigi, and M. Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Krishnamurthy and Mitchell (2012) J. Krishnamurthy and T. Mitchell. 2012. Weakly supervised training of semantic parsers. In _Empirical Methods in Natural Language Processing (EMNLP/CoNLL)_, pages 754-765.
* Kwiatkowski et al. (2013) T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* lee et al. (2016) K. lee, M. Lewis, and L. Zettlemoyer. 2016. Global neural CCG parsing with optimality guarantees. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* Liang et al. (2017) C. Liang, J. Berant, Q. Le, and K. D. F. N. Lao. 2017. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In _Association for Computational Linguistics (ACL)_

C. Liang, M. Norouzi, J. Berant, Q. Le, and N. Lao (2018)Memory augmented policy optimization for program synthesis with generalization. In Advances in Neural Information Processing Systems (NIPS), Cited by: SS4.
* P. Liang, M. I. Jordan, and D. Klein (2011)Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL), pp. 590-599. Cited by: SS3.
* R. Long, P. Pasupat, and P. Liang (2016)Simpler context-dependent logical forms via model projections. In Association for Computational Linguistics (ACL), Cited by: SS4.
* D. Misra, M. Chang, X. He, and W. Yih (2018)Policy shaping and generalized update equations for semantic parsing from denotations.. arXiv preprint arXiv:1809.01299. Cited by: SS4.
* A. Neelakantan, Q. V. Le, and I. Sutskever (2016)Neural programmer: inducing latent programs with gradient descent. In International Conference on Learning Representations (ICLR), Cited by: SS4.
* P. Pasupat and P. Liang (2015)Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics (ACL), Cited by: SS4.
* P. Pasupat and P. Liang (2016)Inferring logical forms from denotations. In Association for Computational Linguistics (ACL), Cited by: SS4.
* A. Pauls and D. Klein (2009)K-best A* parsing. In Association for Computational Linguistics (ACL), pp. 958-966. Cited by: SS4.
* J. Pennington, R. Socher, and C. D. Manning (2014)GloVe: global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543. Cited by: SS4.
* M. Rabinovich, M. Stern, and D. Klein (2017)Abstract syntax networks for code generation and semantic parsing. In Association for Computational Linguistics (ACL), Cited by: SS4.
* S. Ross, G. Gordon, and A. Bagnell (2011)A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS), Cited by: SS4.
* Y. Shen, J. Chen, P. Huang, Y. Guo, and J. Gao (2018)Reinforcewalk: learning to walk in graph with monte carlo tree search. In International Conference on Learning Representations (ICLR), Cited by: SS4.
* D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L., M. Lai, A. Bolton, et al. (2017)Mastering the game of go without human knowledge. Nature550 (7676), pp. 354-359. External Links: ISSN 0037-87X5, Document, Link Cited by: SS4.
* A. Suhr and Y. Artzi (2018)Situated mapping of sequential instructions to actions with single-step reward observation. In Association for Computational Linguistics (ACL), Cited by: SS4.
* I. Sutskever, O. Vinyals, and Q. V. Le (2014)Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS), pp. 3104-3112. Cited by: SS4.
* R. Sutton, D. McAllester, S. Singh, and Y. Mansour (1999)Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), Cited by: SS4.
* Z. Tian T. A. and D. Barber (2017)Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems30. Cited by: SS4.
* A. Vogel and D. Jurafsky (2010)Learning to follow navigational directions. In Association for Computational Linguistics (ACL), pp. 806-814. Cited by: SS4.
* R. J. Williams (1992)Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning8 (3), pp. 229-256. Cited by: SS4.
* S. Wiseman and A. M. Rush (2016)Sequence-to-sequence learning as beam-search optimization. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS4.
* Y. Yang, L. Huang, and M. Ma (2018)Breaking the beam search curse: a study of (re-) scoring methods and stopping criteria for neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS4.
* P. Yin and G. Neubig (2017)A syntactic neural model for general-purpose code generation. In Association for Computational Linguistics (ACL), pp. 440-450. Cited by: SS4.
* Tao Yu, Rui Zhang, Kai Yang, Michiihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. (2018)Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887. Cited by: SS4.

## Appendix A Neural Network Architecture

We adopt the model \(\pi_{\theta}(\cdot)\) proposed by Guu et al. (2017)The model receives the current utterance \(u^{i}\) and the program stack \(\psi\). A bidirectional LSTM Hochreiter and Schmidhuber (1997) is used to embed \(u_{i}\), while \(\psi\) is embedded by concatenating the embedding of stack elements. The embedded input is then fed to a feed-forward network with attention over the LSTM hidden states, followed by a softmax layer that predicts a program token. Our value network \(V_{\phi}(\cdot)\) shares the input layer of \(\pi_{\theta}(\cdot)\). In addition, it receives the next utterance \(u^{i+1}\), the current world state \(w^{i}\) and the target world state \(w^{M}\). The utterance \(u^{i+1}\) is embedded with an additional BiLSTM, and wor 

states are embedded by concatenating embeddings of SCONE elements. The inputs are concatenated and fed to a feed-forward network, followed by a sigmoid layer that outputs a scalar.

## Appendix B Hyper-parameters

Table 3 contains the hyper-parameter setting for each experiment. Hyper-parameters of REINFORCE and MML were taken from Guu et al. (2017). In all experiments learning rate was 0.001 and mini-batch size was 8. We explicitly define the following hyper-parameters which are not self-explanatory:

1. _Training steps_: The number of training steps taken.

2. _Sample size_: Number of samples drawn from \(p_{\theta}\) in REINFORCE

3. _Baseline_: A constant subtracted from the reward for variance reduction.

4. _Execution beam size_: \(K\) in Algorithm 1.

5. _Program bea size_: Size of beam in line 3 of Algorithm 1.

6. _Value ranking start step_: Step when we start ranking states using the critic score.

7. _Value re-rank size_: Size of beam \(K_{0}\) returned by the actor score before re-ranking with the actor-critic score.

## Appendix C Prior Work

In prior work on SCONE, models were trained on sequences of 1 or 2 utterances, and thus were exposed during training to all gold intermediate states Long et al. (2016); Guu et al. (2017); Suhr and Artzi (2018). Fried et al. (2018) assumed access to the full annotated logical form. In contrast, we trained from longer sequences, keeping the logical form and intermediate states latent. We report the test accuracy as reported by prior work and in this paper, noting that our results are an average of 6 runs, while prior work reports the median of 5 runs.

Naturally, our results are lower compared to prior work that uses much stronger supervision. This is because our setup poses a hard search problem at training time, and also requires overcoming _spuriousness_ - the fact that even incorrect programs sometimes lead to high reward.

## Appendix D Value Network Analysis

We analyzed the ability of the value network to predict expected reward. The reward of a state depends on two properties, (a) _connectivity_: whether there is a trajectory from this state to a correct terminal state, and (b) _model likelihood_: the probability the model assigns to those trajectories. We collected a random set of 120 states in the Scene domain from, where the real expected reward was very high (\(>0.7\)), or very low (\(=0.0\)) and the value network predicted well (less than \(0.2\) deviation) or poorly (more than \(0.5\) deviation). For ease of analysis we only look at states from the final utterance.

To analyze connectivity, we looked at states that cannot reach a correct terminal state with a single action (since states in the last utterance can perform one action only, the expected reward is 0). Those are states where either their current and target world differ in too many ways, or the stack content is not relevant to the differences between the worlds. We find that when there are many differences between the current and target world, the value network correctly estimates low expected reward in \(87.0\%\) of the cases. However, when there is just one mismatch between the current and target world, the value network tends to ignore it and erroneously predicts high reward in 78.9% of the cases.

To analyze whether the value network can predict the success of the trained policy, we consider states from which there is an action that leads to the target world. While it is challenging to fully interpret the value network, we notice that the network predicts a value that is \(>0.5\) in 86.1% of the cases where the number of people in the world is no more than 2, and a value that is \(<0.5\) in

Figure 7: The model proposed by Guu et al. (2017) (top), and our value network (bottom).

 

82.1% of the cases where the number of people in the world is more than 2. This indicates that the value network believes more complex worlds, involving many people, are harder for the policy.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|}  & **Scene** & **Alchemy** & **Tangram** \\  & **3 utt** & **5 utt** & **3 utt** & **5 utt** & **3 utt** & **5 utt** \\ \hline Long et al. (2016) & 23.2 & 14.7 & 56.8 & 52.3 & 64.9 & 27.6 \\ Guu et al. (2017) & 64.8 & 46.2 & 66.9 & 52.9 & 65.8 & 37.1 \\ Suhr and Artzi (2018) & 73.9 & 62.0 & 74.2 & 62.7 & 80.8 & 62.4 \\ \hline Fried et al. (2018) & – & 72.7 & – & 72.0 & – & 69.6 \\ \hline VBSIX & 34.2 & 28.2 & 74.5 & 64.8 & 65.0 & 43.0 \\ \end{tabular}
\end{table}
Table 4: Test accuracy comparison to prior work.