Venue: ACL 2017

# A Syntactic Neural Model for General-Purpose Code Generation

Pengcheng Yin

Language Technologies Institute

Carnegie Mellon University

pcyin@cs.cmu.edu &Graham Neubig

Language Technologies Institute

Carnegie Mellon University

gneubig@cs.cmu.edu

###### Abstract

We consider the problem of parsing natural language descriptions into source code written in a general-purpose programming language like Python. Existing data-driven methods treat this problem as a language generation task without considering the underlying syntax of the target programming language. Informed by previous work in semantic parsing, in this paper we propose a novel neural architecture powered by a grammar model to explicitly capture the target syntax as prior knowledge. Experiments find this an effective way to scale up to generation of complex programs from natural language descriptions, achieving state-of-the-art results that well outperform previous code generation and semantic parsing approaches.

## 1 Introduction

Every programmer has experienced the situation where they know what they want to do, but do not have the ability to turn it into a concrete implementation. For example, a Python programmer may want to "_sort my_list in descending order_," but not be able to come up with the proper syntax sorted(my_list, reverse=True) to realize his intention. To resolve this impasse, it is common for programmers to search the web in natural language (NL), find an answer, and modify it into the desired form Brandt et al. (2009, 2010). However, this is time-consuming, and thus the software engineering literature is ripe with methods to directly generate code from NL descriptions, mostly with hand-engineered methods highly tailored to specific programming languages Balzer (1985); Little and Miller (2009); Gvero and Kuncak (2015).

In parallel, the NLP community has developed methods for data-driven semantic parsing, which attempt to map NL to structured logical forms executable by computers. These logical forms can be general-purpose meaning representations Clark and Curran (2007); Banarescu et al. (2013), formalisms for querying knowledge bases Tang and Mooney (2001); Zettlemoyer and Collins (2005); Berant et al. (2013) and instructions for robots or personal assistants Artzi and Zettlemoyer (2013); Quirk et al. (2015); Misra et al. (2015), among others. While these methods have the advantage of being learnable from data, compared to the programming languages (PLs) in use by programmers, the _domain-specific_ languages targeted by these works have a schema and syntax that is relatively simple.

Recently, Ling et al. (2016) have proposed a data-driven code generation method for high-level, _general-purpose_ PLs like Python and Java. This work treats code generation as a sequence-to-sequence modeling problem, and introduce methods to generate words from character-level models, and copy variable names from input descriptions. However, unlike most work in semantic parsing, it does not consider the fact that code has to be well-defined programs in the target syntax.

In this work, we propose a data-driven syntax-based neural network model tailored for generation of general-purpose PLs like Python. In order to capture the strong underlying syntax of the PL, we define a model that transduces an NL statement into an Abstract Syntax Tree (AST; Fig. 1(a), SS 2) for the target PL. ASTs can be deterministically generated for all well-formed programs using standard parsers provided by the PL, and thus give us a way to obtain syntax information with minimal engineering. Once we generate an AST, we can use deterministic generation tools to convert the AST into surface code. We hypothesize

that such a structured approach has two benefits.

First, we hypothesize that structure can be used to constrain our search space, ensuring generation of well-formed code. To this end, we propose a syntax-driven neural code generation model. The backbone of our approach is a _grammar model_ (SS 3) which formalizes the generation story of a derivation AST into sequential application of _actions_ that either apply production rules (SS 3.1), or emit terminal tokens (SS 3.2). The underlying syntax of the PL is therefore encoded in the grammar model _a priori_ as the set of possible actions. Our approach frees the model from recovering the underlying grammar from limited training data, and instead enables the system to focus on learning the compositionality among existing grammar rules. Xiao et al. (2016) have noted that this imposition of structure on neural models is useful for semantic parsing, and we expect this to be even more important for general-purpose PLs where the syntax trees are larger and more complex.

Second, we hypothesize that structural information helps to model information flow within the neural network, which naturally reflects the recursive structure of PLs. To test this, we extend a standard recurrent neural network (RNN) decoder to allow for additional neural connections which reflect the recursive structure of an AST (SS 4.2). As an example, when expanding the node \(\star\) in Fig. 1(a), we make use of the information from both its parent and left sibling (the dashed rectangle). This enables us to locally pass information of relevant code segments via neural network connections, resulting in more confident predictions.

Experiments (SS 5) on two Python code generation tasks show 11.7% and 9.3% absolute improvements in accuracy against the state-of-the-art system Ling et al. (2016). Our model also gives competitive performance on a standard semantic parsing benchmark1.

Footnote 1: Implementation available at [https://github.com/neulab/NLZcode](https://github.com/neulab/NLZcode)

## 2 The Code Generation Problem

Given an NL description \(x\), our task is to generate the code snippet \(c\) in a modern PL based on the intent of \(x\). We attack this problem by first generating the underlying AST. We define a probabilistic grammar model of generating an AST \(y\) given \(x\): \(p(y|x)\). The best-possible AST \(\hat{y}\) is then given by

\[\hat{y}=\operatorname*{arg\,max}_{y}p(y|x). \tag{1}\]

\(\hat{y}\) is then deterministically converted to the corresponding surface code \(c\).2 While this paper uses examples from Python code, our method is PL-agnostic.

Footnote 2: We use astor library to convert ASTs into Python code.

Before detailing our approach, we first present a brief introduction of the Python AST and its underlying grammar. The Python abstract grammar contains a set of production rules, and an AST is generated by applying several production rules composed of a head node and multiple child nodes. For instance, the first rule in Tab. 1 is used to generate the function call sorted(\(\cdot\)) in Fig. 1(a). It consists of a head node of type Call, and three child nodes of type expr, expr* and keyword*, respectively. Labels of each node are noted within brackets. In an AST, non-terminal nodes sketch the general structure of the target code, while terminal nodes can be categorized into two types: _operation terminals_ and _variable terminals_. Operation terminals correspond to basic arithmetic operations like AddOp.Variable terminal nodes store values for variables and constants of built-in data types3. For instance, all terminal nodes in Fig. 1(a) are variable terminal nodes.

Footnote 3: bool, float, int, str.

## 3 Grammar Model

Before detailing our neural code generation method, we first introduce the grammar model at its core. Our probabilistic grammar model defines the generative story o

\begin{table}
\begin{tabular}{l|l|l} \hline \multicolumn{1}{l|}{**Production Rule**} & **Role** & **Explanation** \\ \hline Call \(\mapsto\) expr[func] expr*[args] keyword*[keywords] & Function Call & \(\rhd\)_func_: the function to be invoked p _args_: arguments \\  & & list p keywords: keyword arguments list \\ If \(\mapsto\) expr[test] stmt*[body] stmtt*[orelse] & If Statement & \(\rhd\)_test_: condition expression \(>\) body_: statements inside the If clause p _oreles_: elif or else statements \\ For \(\mapsto\) expr[target] expr*[iter] stmt*[body] & For Loop & \(\rhd\)_target_: iteration variable p _iter_: enumerable to iterate over p _body_: loop body p _orels_: else statements \\ FunctionDef \(\mapsto\) identifier[name] arguments*[args] & Function Def. & \(\rhd\)_name_: function name p _args_: function arguments \\ smt*[body] & & \(\rhd\)_body_: function body \\ \hline \end{tabular}
\end{table}
Table 1: Example production rules for common Python statements (Python Software Foundation, 2016) 

torize the generation process of an AST into sequential application of _actions_ of two types:

* ApplyRule\([r]\) applies a production rule \(r\) to the current derivation tree;
* GenToken\([v]\) populates a variable terminal node by appending a terminal token \(v\).

Fig. 1 shows the generation process of the target AST in Fig. 1. Each node in Fig. 1 indicates an action. Action nodes are connected by solid arrows which depict the chronological order of the action flow. The generation proceeds in depth-first, left-to-right order (dotted arrows represent parent feeding, explained in SS 4.2.1).

Formally, under our grammar model, the probability of generating an AST \(y\) is factorized as:

\[p(y|x)=\prod_{t=1}^{T}p(a_{t}|x,a_{<t}), \tag{2}\]

where \(a_{t}\) is the action taken at time step \(t\), and \(a_{<t}\) is the sequence of actions before \(t\). We will explain how to compute the action probabilities \(p(a_{t}|\cdot)\) in Eq. (2) in SS 4. Put simply, the generation process begins from a root node at \(t_{0}\), and proceeds by the model choosing ApplyRule actions to generate the overall program structure from a closed set of grammar rules, then at leaves of the tree corresponding to variable terminals, the model switches to GenToken actions to generate variables or constants from the open set. We describe this process in detail below.

### ApplyRule Actions

ApplyRule actions generate program structure, expanding the current node (the _frontier node_ at time step \(t\): \(n_{f_{t}}\)) in a depth-first, left-to-right traversal of the tree. Given a fixed set of production rules, ApplyRule chooses a rule \(r\) from the subset that has a head matching the type of \(n_{f_{t}}\), and uses \(r\) to expand \(n_{f_{t}}\) by appending all child nodes specified by the selected production. As an example, in Fig. 1, the rule Call\(\mapsto\) expr... expands the frontier node Call at time step \(t_{4}\), and its three child nodes expr, expr* and keyword* are added to the derivation.

ApplyRule actions grow the derivation AST by appending nodes. When a variable terminal node (e.g., str) is added to the derivation and becomes the frontier node, the grammar model then switches to GenToken actions to populate the variable terminal with tokens.

**Unary Closure** Sometimes, generating an AST requires applying a chain of unary productions. For instance, it takes three time steps (\(t_{9}-t_{11}\)) to generate the sub-structure expr*\(\mapsto\)expr \(\mapsto\)Name \(\mapsto\)str in Fig. 1. This can be effectively reduced to one step of ApplyRule action by taking the closure of the chain of unary productions and merging them into a single rule: expr*\(\mapsto^{*}\)str. Unary closures reduce the number of actions needed, but would potentially increase the size of the grammar. In our experiments we tested our model both with and without unary closures (SS 5).

### GenToken Actions

Once we reach a frontier node \(n_{f_{t}}\) that corresponds to a variable type (e.g., str), GenToken actions are used to fill this node with values. For general-purpose PLs like Python, variables and constants have values with one or multiple tokens. For in

Figure 1: (a) the Abstract Syntax Tree (AST) for the given example code. Dashed nodes denote terminals. Nodes are labeled with time steps during which they are generated. (b) the action sequence (up to \(t_{14}\)) used to generate the AST in (a)



stance, a node that stores the name of a function (e.g., sorted) has a single token, while a node that denotes a string constant (e.g., an*'hello world') could have multiple tokens. Our model copes with both scenarios by firing GenToken actions at one or more time steps. At each time step, GenToken appends one terminal token to the current frontier variable node. A special </n> token is used to "close" the node. The grammar model then proceeds to the new frontier node.

Terminal tokens can be generated from a predefined vocabulary, or be directly copied from the input NL. This is motivated by the observation that the input description often contains out-of-vocabulary (OOV) variable names or literal values that are directly used in the target code. For instance, in our running example the variable name my_list can be directly copied from the the input at \(t_{12}\). We give implementation details in SS 4.2.2.

## 4 Estimating Action Probabilities

We estimate action probabilities in Eq. (2) using attentional neural encoder-decoder models with an information flow structured by the syntax trees.

### Encoder

For an NL description \(x\) consisting of \(n\) words \(\{w_{i}\}_{i=1}^{n}\), the encoder computes a context sensitive embedding \(\mathbf{h_{i}}\) for each \(w_{i}\) using a bidirectional Long Short-Term Memory (LSTM) network Hochreiter and Schmidhuber (1997), similar to the setting in Bahdanau et al. (2014). See supplementary materials for detailed equations.

### Decoder

The decoder uses an RNN to model the sequential generation process of an AST defined as Eq. (2). Each action step in the grammar model naturally grounds to a time step in the decoder RNN. Therefore, the action sequence in Fig. 1(b) can be interpreted as unrolling RNN time steps, with solid arrows indicating RNN connections. The RNN maintains an internal state to track the generation process (SS 4.2.1), which will then be used to compute action probabilities \(p(a_{t}|x,a_{<t})\) (SS 4.2.2).

#### 4.2.1 Tracking Generation States

Our implementation of the decoder resembles a vanilla LSTM, with additional neural connections (parent feeding, Fig. 1(b)) to reflect the topological structure of an AST. The decoder's internal hidden state at time step \(t\), \(\mathbf{s_{t}}\), is given by:

\[\mathbf{s_{t}}=f_{\mathrm{LSTM}}([\mathbf{a_{t-1}}:\mathbf{c_{t}}:\mathbf{p_{ t}}:\mathbf{n_{f_{t}}}],\mathbf{s_{t-1}}), \tag{3}\]

where \(f_{\mathrm{LSTM}}(\cdot)\) is the LSTM update function. \([:]\) denotes vector concatenation. \(\mathbf{s_{t}}\) will then be used to compute action probabilities \(p(a_{t}|x,a_{<t})\) in Eq. (2). Here, \(\mathbf{a_{t-1}}\) is the embedding of the previous action. \(\mathbf{c_{t}}\) is a context vector retrieved from input encodings \(\{\mathbf{h_{i}}\}\) via soft attention. \(\mathbf{p_{t}}\) is a vector that encodes the information of the parent action. \(\mathbf{n_{f_{t}}}\) denotes the node type embedding of the current frontier node \(n_{f_{t}}\)4. Intuitively, feeding the decoder the information of \(n_{f_{t}}\) helps the model to keep track of the frontier node to expand.

Footnote 4: We maintain an embedding for each node type.

Action Embedding \(\mathbf{a_{t}}\)We maintain two action embedding matrices, \(\mathbf{W}_{R}\) and \(\mathbf{W}_{G}\). Each row in \(\mathbf{W}_{R}\) (\(\mathbf{W}_{G}\)) corresponds to an embedding vector for an action ApplyRule\([r]\) (GenToken\([v]\)).

Context Vector \(\mathbf{c_{t}}\)The decoder RNN uses soft attention to retrieve a context vector \(\mathbf{c_{t}}\) from the input encodings \(\{\mathbf{h_{i}}\}\) pertain to the prediction of the current action. We follow Bahdanau et al. (2014) and use a Deep Neural Network (DNN) with a single hidden layer to compute attention weights.

Parent Feeding \(\mathbf{p_{t}}\)Our decoder RNN uses additional neural connections to directly pass information from parent actions. For instance, when computing \(\mathbf{s_{9}}\), the information from its parent action step \(t_{4}\) will be used. Formally, we define the _parent action step_\(p_{t}\) as the time step at which the frontier node \(n_{f_{t}}\) is generated. As an example, for \(t_{9}\), its parent action step \(p_{9}\) is \(t_{4}\), since \(n_{f_{9}}\) is the node \(\star\), which is generated at \(t_{4}\) by the ApplyRule\([\texttt{Call}\mapsto\ldots]\) action.

We model parent information \(\mathbf{p_{t}}\) from two sources: (1) the hidden state of parent action \(\mathbf{s_{p_{t}}}\), and (2) the embedding of parent action \(\mathbf{a_{p_{t}}}\). \(\mathbf{p_{t}}\) is the concatenation. The parent feeding schema en

Figure 2: Illustration of a decoder time step (\(t=9\))



ables the model to utilize the information of parent code segments to make more confident predictions. Similar approaches of injecting parent information were also explored in the Seq2Tree model in Dong and Lapata (2016)5.

Footnote 4: See [https://github.com/google-learning-model-with-the-international-code-segmenting-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-instance-the-instance-of-instance-the-instance-of-the-instance-of-instance-](https://github.com/google-learning-model-with-the-international-code-segmenting-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-instance-the-instance-of-the-instance-of-instance-the-instance-of-instance-the-instance-of-instance-the-instance-of-the-instance-of-instance-) 

App. Users of the App write simple instructions (e.g., If Instagram.AnyNewPhotoByYou Then Dropbox.AddFileFromURL) with NL descriptions (e.g., "_Autosave your Instagram photos to Dropbox_"). Each statement inside the If or Then clause consists of a channel (e.g., Dropbox) and a function (e.g., AddFileFromURL)7. This simple structure results in much more concise ASTs (7 nodes on average). Because all examples are created by ordinary Apps users, the dataset is highly noisy, with input NL very loosely connected to target ASTs. The authors thus provide a high-quality filtered test set, where each example is verified by at least three annotators. We use this set for evaluation. Also note Iftttt's grammar has more productions (Tab. 2), but this does not imply that its grammar is more complex. This is because for HS and Django terminal tokens are generated by GenToken actions, but for Iftttt, all the code is generated directly by ApplyRule actions.

Footnote 7: Like Beltagy and Quirk (2016), we strip function parameters since they are mostly specific to users.

**Metrics** As is standard in semantic parsing, we measure **accuracy**, the fraction of correctly generated examples. However, because generating an exact match for complex code structures is non-trivial, we follow Ling et al. (2016), and use token-level **BLEU-4** with as a secondary metric, defined as the averaged BLEU scores over all examples.8

Footnote 8: These two metrics are not ideal: accuracy only measures exact match and thus lacks the ability to give credit to semantically correct code that is different from the reference, while it is not clear whether BLEU provides an appropriate proxy for measuring semantics in the code generation task. A more intriguing metric would be directly measuring semantic/functional code equivalence, for which we present a pilot study at the end of this section (cf. Error Analysis). We leave exploring more sophisticated metrics (e.g. based on static code analysis) as future work.

### Setup

**Preprocessing** All input descriptions are tokenized using nltk. We perform simple canonicalization for Django, such as replacing quoted strings in the inputs with place holders. See supplementary materials for details. We extract unary closures whose frequency is larger than a threshold \(k\) (\(k=30\) for HS and \(50\) for Django).

**Configuration** The size of all embeddings is 128, except for node type embeddings, which is 64. The dimensions of RNN states and hidden layers are 256 and 50, respectively. Since our datasets are relatively small for a data-hungry neural model, we impose strong regularization using recurrent dropouts Gal and Ghahramani (2016) for all recurrent networks, together with standard dropout layers added to the inputs and outputs of the decoder RNN. We validate the dropout probability from \(\{0,0.2,0.3,0.4\}\). For decoding, we use a beam size of 15.

#### Results

Evaluation results for Python code generation tasks are listed in Tab. 3. Numbers for our systems are averaged over three runs. We compare primarily with two approaches: (1) Latent Predictor Network (lpn), a state-of-the-art sequence-to-sequence code generation model Ling et al. (2016), and (2) Seq2Tree, a neural semantic parsing model Dong and Lapata (2016). Seq2Tree generates trees one node at a time, and the target grammar is not explicitly modeled a priori, but _implicitly_ learned from data. We test both the original Seq2Tree model released by the authors and our revised one Seq2Tree-UNK) that uses unknown word replacement to handle rare words Luong et al. (2015). For completeness, we also compare with a strong neural machine translation nmt system Neubig (2015) using a standard encoder-decoder architecture with attention and unknown word replacement9, and include numbers from other baselines used in Ling et al. (2016). On the HS dataset, which has relatively large ASTs, we use unary closure for our model and Seq2Tree, and for Django we do not.

Footnote 9: Fornmt, we also attempted to find the best-scoring syntactically correct predictions in the size-5 beam, but this did not yield a significant improvement over the nmt results in Tab. 3.

[TABLE:S

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & \multicolumn{2}{c}{HS} & \multicolumn{2}{c}{Django} \\ \hline  & acc & bleu & acc & bleu \\ Retrieval System\({}^{\dagger}\) & 0.0 & 62.5 & 14.7 & 18.6 \\ Phrasal Statistical MT\({}^{\dagger}\) & 0.0 & 34.1 & 31.5 & 47.6 \\ Hierarchical Statistical MT\({}^{\ddagger}\) & 0.0 & 43.2 & 9.5 & 35.9 \\ \hline nmt & 1.5 & 60.4 & 45.1 & 63.4 \\ Seq2Tree & 1.5 & 53.4 & 28.9 & 44.6 \\ Seq2Tree–UNK & 13.6 & 62.8 & 39.4 & 58.2 \\ lpn\({}^{\dagger}\) & 4.5 & 65.6 & 62.3 & 77.6 \\ \hline Our system & 16.2 & **75.8** & **71.6** & **84.5** \\ \hline Ablation Study – frontier embed. & **16.7** & **75.8** & 70.7 & 83.8 \\ – parent feed. & 10.6 & 75.7 & 71.5 & 84.3 \\ – copy terminals & 3.0 & 65.7 & 32.3 & 61.7 \\ + unary closure & – & 70.3 & 83.3 \\ – unary closure & 10.1 & 74.8 & – \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results on two Python code generation tasks. \({}^{\dagger}\)Results previously reported in Ling et al. (2016).

 



**System Comparison** As in Tab. 3, our model registers 11.7% and 9.3% absolute improvements over lpn in accuracy on HS and Django. This boost in performance strongly indicates the importance of modeling grammar in code generation. For the baselines, we find lpn outperforms nmt and Seq2Tree in most cases. We also note that Seq2Tree achieves a decent accuracy of 13.6% on HS, which is due to the effect of unknown word replacement, since we only achieved 1.5% without it. A closer comparison with Seq2Tree is insightful for understanding the advantage of our syntax-driven approach, since both Seq2Tree and our system output ASTs: (1) Seq2Tree predicts one node each time step, and requires additional "dummy" nodes to mark the boundary of a subtree. The sheer number of nodes in target ASTs makes the prediction process error-prone. In contrast, the ApplyRule actions of our grammar model allows for generating multiple nodes at a single time step. Empirically, we found that in HS, Seq2Tree takes more than 300 time steps on average to generate a target AST, while our model takes only 170 steps. (2) Seq2Tree does not directly use productions in the grammar, which possibly leads to grammatically incorrect ASTs and thus empty code outputs. We observe that the ratio of grammatically incorrect ASTs predicted by Seq2Tree on HS and Django are 21.2% and 10.9%, respectively, while our system guarantees grammaticality.

**Ablation Study** We also ablated our best-performing models to analyze the contribution of each component. "-frontier embed." removes the frontier node embedding \({\bf n}_{f_{t}}\) from the decoder RNN inputs (Eq. (3)). This yields worse results on Django while gives slight improvements in accuracy on HS. This is probably because that the grammar of HS has fewer node types, and thus the RNN is able to keep track of \(n_{f_{t}}\) without depending on its embedding. Next, "-parent feed." removes the parent feeding mechanism. The accuracy drops significantly on HS, with a marginal deterioration on Django. This result is interesting because it suggests that parent feeding is more important when the ASTs are larger, which will be the case when handling more complicated code generation tasks like HS. Finally, removing the pointer network ("-copy terminals") in GenToken actions gives poor results, indicating that it is important to directly copy variable names and values from the input.

The results with and without unary closure demonstrate that, interestingly, it is effective on HS but not on Django. We conjecture that this is because on HS it significantly reduces the number of actions from 173 to 142 (c.f., Tab. 2), with the number of productions in the grammar remaining unchanged. In contrast, Django has a broader domain, and thus unary closure results in more productions in the grammar (237 for Django vs. 100 for HS), increasing sparsity.

**Performance by the size of AST** We further investigate our model's performance w.r.t. the size of the gold-standard ASTs in Figs. 3 and 4. Not surprisingly, the performance drops when the size of the reference ASTs increases. Additionally, on the HS dataset, the BLEU score still remains at around 50 even when the size of ASTs grows to 200, indicating that our proposed syntax-driven approach is robust for long code segments.

**Domain Specific Code Generation** Although this is not the focus of our work, evaluation on Ifttt brings us closer to a standard semantic parsing set

Figure 4: Performance w.r.t reference AST size on HS

\begin{table}
\begin{tabular}{l c c} \hline \hline  & Channel Full Tree \\ \hline
**Classical Methods** & & \\ posclass Quirk et al. (2015) & 81.4 & 71.0 \\ LR Beltagy and Quirk (2016) & 88.8 & **82.5** \\ \hline
**Neural Network Methods** & & \\ nmt & 87.7 & 77.7 \\ NN Beltagy and Quirk (2016) & 88.0 & 74.3 \\ Seq2Tree Dong and Lapata (2016) & 89.7 & 78.4 \\ Doubly-Recurrent NN Alvarez-Melis and Jaakkola (2017) & **90.1** & 78.2 \\ \hline Our system & 90.0 & 82.0 \\ – parent feed. & 89.9 & 81.1 \\ – frontier embed. & **90.1** & 78.7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results on the noise-filtered fittt test set of “>3 agree with gold annotations” (averaged over three runs), our model performs competitively among neural models.

Figure 3: Performance w.r.t reference AST size on Django

ting, which helps to investigate similarities and differences between generation of more complicated general-purpose code and and more limited-domain simpler code. Tab. 4 shows the results, following the evaluation protocol in Beltagy and Quirk (2016) for accuracies at both channel and full parse tree (channel + function) levels. Our full model performs on par with existing neural network-based methods, while outperforming other neural models in full tree accuracy (82.0%). This score is close to the best classical method (LR), which is based on a logistic regression model with rich hand-engineered features (e.g., brown clusters and paraphrase). Also note that the performance between nmt and other neural models is much closer compared with the results in Tab. 3. This suggests that general-purpose code generation is more challenging than the simpler Ifttt setting, and therefore modeling structural information is more helpful.

**Case Studies** We present output examples in Tab. 5. On HS, we observe that most of the time our model gives correct predictions by filling learned code templates from training data with arguments (e.g., cost) copied from input. This is in line with the findings in Ling et al. (2016). However, we do find interesting examples indicating that the model learns to generalize beyond trivial copying. For instance, the first example is one that our model predicted wrong -- it generated code block A instead of the gold B (it also missed a function definition not shown here). However, we find that the block A actually conveys part of the input intent by destroying all, not some, of the minions. Since we are unable to find code block A in the training data, it is clear that the model has learned to generalize to some extent from multiple training card examples with similar semantics or structure.

The next two examples are from Django. The first one shows that the model learns the usage of common API calls (e.g., os.path.join), and how to populate the arguments by copying from inputs. The second example illustrates the difficulty of generating code with complex nested structures like lambda functions, a scenario worth further investigation in future studies. More examples are attached in supplementary materials.

**Error Analysis** To understand the sources of errors and how good our evaluation metric (exact match) is, we randomly sampled and labeled 100 and 50 failed examples (with accuracy=0) from Django and HS, respectively. We found that around 2% of these examples in the two datasets are actually semantically equivalent. These examples include: (1) using different parameter names when defining a function; (2) omitting (or adding) default values of parameters in function calls. While the rarity of such examples suggests that our exact match metric is reasonable, more advanced evaluation metrics based on statistical code analysis are definitely intriguing future work.

For Django, we found that 30% of failed cases were due to errors where the pointer network failed to appropriately copy a variable name into the correct position. 25% were because the generated code only partially implemented the required functionality. 10% and 5% of errors were due to malformed English inputs and pre-processing errors, respectively. The remaining 30% of examples were errors stemming from multiple sources, or errors that could not be easily categorized into the above. For HS, we found that all failed card examples were due to partial implementation errors, such as the one shown in Table 5.

## 6 Related Work

**Code Generation and Analysis** Most works on code generation focus on generating code for domain specific languages (DSLs) Kushman and

\begin{table}
\begin{tabular}{l l}
**input** & _cname> Brawl</name>cost>5</cost><desc>_ \\  & _Destry all minions except one (chosen randomly) \textless{/desec>}_ \\
**pred.** class & **REMSOUTEDY****.** \\  & def __init\text\textgreater{}(self): \\  & super\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasci{}\textasciasci{}\textasciasci{}\textasciasci{}\textasciasci{}\textasciasciasci{}\textasciasciasciasciasci{}\textasci 

Barzilay, 2013; Raza et al., 2015; Manshadi et al., 2013), with neural network-based approaches recently explored Liu et al. (2016); Parisotto et al. (2016); Balog et al. (2016). For general-purpose code generation, besides the general framework of Ling et al. (2016), existing methods often use language and task-specific rules and strategies Lei et al. (2013); Raghothaman et al. (2016). A similar line is to use NL queries for code retrieval Wei et al. (2015); Allamanis et al. (2015). The reverse task of generating NL summaries from source code has also been explored Oda et al. (2015); Iyer et al. (2016). Finally, our work falls into the broad field of probabilistic modeling of source code Maddison and Tarlow (2014); Nguyen et al. (2013). Our approach of factoring an AST using probabilistic models is closely related to Allamanis et al. (2015), which uses a factorized model to measure the semantic relatedness between NL and ASTs for code retrieval, while our model tackles the more challenging generation task.

**Semantic Parsing** Our work is related to the general topic of semantic parsing, which aims to transform NL descriptions into executable logical forms. The target logical forms can be viewed as DSLs. The parsing process is often guided by grammatical formalisms like combinatory categorical grammars Kwiatkowski et al. (2013); Artzi et al. (2015), dependency-based syntax Liang et al. (2011); Pasupat and Liang (2015) or task-specific formalisms Clarke et al. (2010); Yih et al. (2015); Krishnamurthy et al. (2016); Mei et al. (2016). Recently, there are efforts in designing neural network-based semantic parsers Misra andArtzi (2016); Dong and Lapata (2016); Neelakantan et al. (2016); Yin et al. (2016). Several approaches have be proposed to utilize grammar knowledge in a neural parser, such as augmenting the training data by generating examples guided by the grammar Kocisky et al. (2016); Jia and Liang (2016). Liang et al. (2016) used a neural decoder which constrains the space of next valid tokens in the query language for question answering. Finally, the structured prediction approach proposed by Xiao et al. (2016) is closely related to our model in using the underlying grammar as prior knowledge to constrain the generation process of derivation trees, while our method is based on a unified grammar model which jointly captures production rule application and terminal symbol generation, and scales to general purpose code generation tasks.

## 7 Conclusion

This paper proposes a syntax-driven neural code generation approach that generates an abstract syntax tree by sequentially applying actions from a grammar model. Experiments on both code generation and semantic parsing tasks demonstrate the effectiveness of our proposed approach.

### Acknowledgment

We are grateful to Wang Ling for his generous help with lpn and setting up the benchmark. We thank I. Beltagy for providing the Ifttt dataset. We also thank Li Dong for helping with Seq2Tree and insightful discussions.

## References

* Allamanis et al. (2015) Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language. In _Proceedings of ICML_. volume 37.
* Alvarez-Melis and Jaakkola (2017) David Alvarez-Melis and Tommi S. Jaakkola. 2017. Tree-structured decoding with doubly recurrent neural networks. In _Proceedings of ICLR_.
* Artzi et al. (2015) Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015. Broad-coverage CCG semantic parsing with AMR. In _Proceedings of EMNLP_.
* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transaction of ACL_ 1(1).
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _CoRR_ abs/1409.0473.
* Balog et al. (2016) Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. Deepcoder: Learning to write programs. _CoRR_ abs/1611.01989.
* Balzer (1985) Robert Balzer. 1985. A 15 year perspective on automatic programming. _IEEE Trans. Software Eng._ 11(11).
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of LAW-ID@ACL_.
* Beltagy and Quirk (2016) I. Beltagy and Chris Quirk. 2016. Improved semantic parsers for if-then statements. In _Proceedings of ACL_.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _Proceedings of EMNLP_ 

Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R. Klemmer. 2010. Example-centric programming: integrating web search into the development environment. In _Proceedings of CHI_.
* Brandt et al. (2009) Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontheva, and Scott R. Klemmer. 2009. Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In _Proceedings of CHI_.
* Clark and Curran (2007) Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. _Computational Linguistics_ 33(4).
* Clarke et al. (2010) James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In _Proceedings of CoNLL_.
* Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In _Proceedings of ACL_.
* Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In _Proceedings of NIPS_.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of ACL_.
* Gvero and Kuncak (2015) Tihomir Gvero and Viktor Kuncak. 2015. Interactive synthesis using free-form queries. In _Proceedings of ICSE_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_ 9(8).
* Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In _Proceedings of ACL_.
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _Proceedings of ACL_.
* Kocisky et al. (2016) Tomas Kocisky, Gabor Melis, Edward Grefenstette, Chris Dyer, Wang Ling, Phil Blunsom, and Karl Moritz Hermann. 2016. Semantic parsing with semi-supervised sequential autoencoders. In _Proceedings of EMNLP_.
* Krishnamurthy et al. (2016) Jayant Krishnamurthy, Ovyind Tafjord, and Aniruddha Kembhavi. 2016. Semantic parsing to probabilistic programs for situated question answering. In _Proceedings of EMNLP_.
* Kushman and Barzilay (2013) Nate Kushman and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In _Proceedings of NAACL_.
* Kwiatkowski et al. (2013) Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke S. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In _Proceedings of the EMNLP_.
* Lei et al. (2013) Tao Lei, Fan Long, Regina Barzilay, and Martin C. Riandr. 2013. From natural language specifications to program input parsers. In _Proceedings of ACL_.
* Liang et al. (2016) Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. _CoRR_ abs/1611.00020.
* Liang et al. (2011) Percy Liang, Michael I. Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In _Proceedings of ACL_.
* Ling et al. (2016) Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In _Proceedings of ACL_.
* Little and Miller (2009) Greg Little and Robert C. Miller. 2009. Keyword programming in java. _Autom. Softw. Eng._ 16(1).
* Liu et al. (2016) Chang Liu, Xinyun Chen, Eui Chul Richard Shin, Mingcheng Chen, and Dawn Xiaodong Song. 2016. Latent attention for if-then program synthesis. In _Proceedings of NIPS_.
* Luong et al. (2015) Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In _Proceedings of ACL_.
* Maddison and Tarlow (2014) Chris J. Maddison and Daniel Tarlow. 2014. Structured generative models of natural source code. In _Proceedings of ICML_.
* Manshadi et al. (2013) Mehdi Hafezi Manshadi, Daniel Gildea, and James F. Allen. 2013. Integrating programming by example and natural language programming. In _Proceedings of AAAI_.
* Mei et al. (2016) Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In _Proceedings of AAAI_.
* Misra and Artzi (2016) Dipendra K. Misra and Yoav Artzi. 2016. Neural shift-reduce CCG semantic parsing. In _Proceedings of EMNLP_.
* Misra et al. (2015) Dipendra Kumar Misra, Kejia Tao, Percy Liang, and Ashutosh Saxena. 2015. Environment-driven lexicon induction for high-level instructions. In _Proceedings of ACL_.
* Neelakantan et al. (2016) Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In _Proceedings of ICLR_.
* Neubig (2015) Graham Neubig. 2015. lamtram: A toolkit for language and translation modeling using neural networks. [http://www.github.com/neubig/lamtram](http://www.github.com/neubig/lamtram).
* Nguyen et al. (2013) Tong Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2013. A statistical semantic language model for source code. In _Proceedings of ACM SIGSOFT_ 

Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation (T). In _Proceedings of ASE_.
* Parisotto et al. (2016) Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2016. Neuro-symbolic program synthesis. _CoRR_ abs/1611.01855.
* Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In _Proceedings of ACL_.
* Foundation (2016) Python Software Foundation. 2016. Python abstract grammar. [https://docs.python.org/2/library/ast.html](https://docs.python.org/2/library/ast.html).
* Quirk et al. (2015) Chris Quirk, Raymond J. Mooney, and Michel Galley. 2015. Language to code: Learning semantic parsers for if-this-then-that recipes. In _Proceedings of ACL_.
* Raghothaman et al. (2016) Mukund Raghothaman, Yi Wei, and Youssef Hamadi. 2016. SWIM: synthesizing what i mean: code search and idiomatic snippet synthesis. In _Proceedings of ICSE_.
* Raza et al. (2015) Mohammad Raza, Sumit Gulwani, and Natasa Milickrayling. 2015. Compositional program synthesis from natural language and examples. In _Proceedings of IJCAI_.
* Tang and Mooney (2001) Lappoon R. Tang and Raymond J. Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. In _Proceedings of ECML_.
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In _Proceedings of NIPS_.
* Wei et al. (2015) Yi Wei, Nirupama Chandrasekaran, Sumit Gulwani, and Youssef Hamadi. 2015. Building bug developer assistant. Technical report. [https://www.microsoft.com/en-us/research/publication/building-bing-developer-assistant/](https://www.microsoft.com/en-us/research/publication/building-bing-developer-assistant/).
* Xiao et al. (2016) Chunyang Xiao, Marc Dymettman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In _Proceedings of ACL_.
* Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In _Proceedings of ACL_.
* Yin et al. (2016) Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. 2016. Neural enquirer: Learning to query tables in natural language. In _Proceedings of IJCAI_.
* Zettlemoyer and Collins (2005) Luke Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form structured classification with probabilistic categorial grammars. In _Proceedings of UAI_.