Structural Language Models for Any-Code Generation
Authors: Uri Alon
Roy Sadaka
Omer Levy
Eran Yahav
Venue:  (2020)
Keywords: Program Generation
Structural Language Model
SLM
Generative Model
Code Generation

Abstract: We address the problem of Any-Code Generation (AnyGen) - generating code without any restriction on the vocabulary or structure. The state-of-the-art in this problem is the sequence-to-sequence (seq2seq) approach, which treats code as a sequence and does not leverage any structural information. We introduce a new approach to AnyGen that leverages the strict syntax of programming languages to model a code snippet as tree structural language modeling (SLM). SLM estimates the probability of the program's abstract syntax tree (AST) by decomposing it into a product of conditional probabilities over its nodes. We present a neural model that computes these conditional probabilities by considering all AST paths leading to a target node. Unlike previous structural techniques that have severely restricted the kinds of expressions that can be generated, our approach can generate arbitrary expressions in any programming language. Our model significantly outperforms both seq2seq and a variety of existing structured approaches in generating Java and C# code. We make our code, datasets, and models available online.

Under review as a conference paper at ICLR 2020
STRUCTURAL LANGUAGE MODELS FOR CODE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
We address the problem of Any- Any-Code-to- Code Generation (AnyC2C)- generating code given its surrounding code without any restriction on the vocabulary or struc- ture. The state-of-the-art in this problem is the sequence-to-sequeme (seq2seq) approach, which treats code as sequence and does not leverage any structural information. We introduce syntax of programming languages to model a code snippet as a tree structural language modeling (SLM). SLM estimates the probability of the program's 's ab- stract syntax tree (AST) by decomposing it into product of conditional proba- bilities over its nodes. We present a neural model that computes these conditional probabilities by considering all AST paths leading to a target node. Unlike pre- vious structural techniques that have severely restricted the kinds of expressions that can be generated in this task, our approach can generate arbitrary expressions in any programming language. Our model significantly outperforms both seq2seq and variety of existing structured approaches in generating Java and C# code. We make our code, datasets, and models available online.
approach to AnyC2C that leverages the strict
INTRODUCTION
Generating source code requires reasoning over an unbounded number of syntactic structures and potential symbols. Previous approaches have avoided this issue by limiting the generation prob- lem: program synthesis approaches (Manna and Waldinger I97I) are tailored domain-specific languages (Gulwani] (011), semantic parsing approaches focus on highly templated datasets (Ling) jet al.) 10066|Shin al.J 2019) or SQL. (Yu et al., 2018) Dong and Lapata 2018), while other recent approaches generate code general languages like Java and C#, but severely restrict the syntax, vocabulary or nature of the generated expressions (Murali et al. 2017 Brockschmidt et al. 2019a)).
We introduce the task of Any-Code-to-Code Generation (AnyC2C) generating source code in a purrpose programming language without any restriction on its vocabulary or structure. Specifically. focus generating code in context: given program P and some part of the program predict from the rest of the program P-=P)p. The only restriction we place that the target p must have valid subtree within the program's abstract syntax tree (AST). AnyC2C thus generalizes the restricted expression generation task of [Brockschmidt et al. (2019a), where target code contains only primitive types and excludes user-defined functions. Figure shows two such AnyC2C examples.
(seq2seq) model with copy mechanism works better than existing code generation approaches on AnyC2C (see Section 55. it ignores the structural information avail- able from the code's AST. We present new approach that explicitly leverages the strict syntax of programming languages to model code snippets as trees structural language modeling (SLM). SLM estimates the probability of the program's 's AST by decomposing into a product of conditional probabilities over its nodes. We present neural model that computes these conditional probabilities by considering all AST paths leading to a target node. While prior work uses AST paths to read programs (Alon et al. 2019a), we generate code by producing the target AST node-by-node.
While a
We evaluate SLM on Java AnyC2C benchmarks, where our model achieves a new sate--o-thheans exact-match accuracy of 18.04% (previous SOTA: 16.93%). SLMs also outperform existing models on the restricted expression generation task of Brockscmiid et al. (2019a) C# by a wide margin, 37.61 % compared 26.42%. Our ablation study reveals the importance of using AST paths for

---

Under review as a conference paper at ICLR 2020
public static Path() stat2Pathal
int trai lragpaags (this StringSu1 R huild
PileStatus stata)
var bound Puilder.Length
ir (stats -- pull;
if (builder. Length == 0) return 0;
recurn mull;
Ide !- return 0g
Path[] ret
0; For Ivar bound; <= bound; i--)
for (int Gy stats.iengthy ++1) 4 ret(i] stats gettPathh)
if (i 0) break;
10 builder[1] 1- !) break;
return ret?
c++]
return
Figure I: AnyC2C examples from the Java (left) and C# (right) test sets. The highlighted expression in each example is the target p. which we wish generate from the rest of the snippet.
both reading and generating code. Finally, we discuss the theoretical advantages of SLMs, and show how they generalize many previous structural approaches for code generation.
2 CODE GENERATION AS STRUCTURAL LANGUAGE MODELING
We model the task of Any-Code-to-Code Generation (AnyC2C) by computing the probability of program Pr (P), similar to how a language model computes the probability of a natural language sentence. While language models typically assume a sequence as their input, our input is an ab- stract syntax tree Ap. We thus introduce a structural language modeling approach (SLM) for code generation.
We first show a chain-rule decomposition of the tree's probability Pr (Ap) into product of con- ditional node probabilities, and then describe our path-based model for computing the individual conditional probabilities. We explain how to construct a tree from local node predictions, and fi- nally discuss how our approach differs from previous work on production-based tree generation.
Representing Code as Tree A program P is sequence of tokens that is unambiguously equiv- alent to an abstract syntax tree Ap, where each node represents an element the language (e.g. conditions, loops, variable declarations) from a set T. The AST's leaves (terminals) have addi- tional user-defined value v E V. Nonterminal nodes can have a varying number of children nodes.
Decomposing the Probability of Tree Given tree Ap, we first traverse the tree, depth- firtt F induce an ordering over its nodes ag.... 4|AA| E Ap. We can now decompose the probability of tree Pr (Ap) using the chain rule, akin to the standard approach language modeling:
Pr (Ap) = II Pr (arlac))
(1)
where aze are all the nodes that were traversed before de-
In AnyC2C, part of the tree (Ap-) already observed. Therefore, we order the nodes of Ap- before the nodes of the target p, and compute only the conditional probabilities over the nodes in p, essentially conditioning on the observed tree Ap-.
Representing Partial Trees via Paths How when computing Pr (agla In regular language modeling. the structure is linear, and ace is a sequence. One way represent partial tree is linearize it according to the traversal order (Xiao jtt al., 2016); however, this could create artificially long distances between the current node a, and ancestor nodes (e.g., the root ag). Another option to use only the path from the root node a, Rabinovich et al. 2017). but this ignores a lot of contextual information (e.g., sibling nodes).
represent the partial tree composed of acc
We follow Alon et al. (2088) and use the as well as the path R2 originating from the root. We denote the (candidate) node at time t as its (given) parent, which is the currently expanded node, by π (at), and the set of all paths as S,
paths from every leaf to the current node expand,
S = {l ~e π (at) |e E leaves (a<+)) U {ao ~ π (a+))
DDepthffrst ordering is common practice in tree generation Maddison and Tarlow 2014 Raychev et al. 2016 Brockschmidt al. 2019a), but framework also allows for other orderings, theory.
2

---

Under review as a conference paper at ICLR 2020
Meeroo Risi
Rar
Roor
bath
R
math
R
botb
pelfi
pulliy
(a)
(c)
Sahoo
Bod
R
REsur
iber
if(
x
}
Greaker
Name
Name
(f)
(d)
(e)
Figure 2: The expression x model generates the next node (denoted by question mark: @ of pathi, path2 and path using the rool path R. Dashed lines denote AST parent-child relations; solid lines denote AST paths.
is generated given its surrounding code context. At each step, the
where n (at) is the (only) path in the tree between a leaf and the current node to expand (ax), and R2 the current, relative position of (a:) the program (marked as R Figure 2}. Whereas prior
(ax) is the path from the root of the program to æ (ar). which represents
work used whole paths (between two leaf nodes) encode an AST (Alon et al. 2019b a)
model observes partial paths (between a leaf and an intermediate node) and learns to extend them.
Figure 2 iillustrates the traversal order of subtree that represents the expression some the paths used to compute the probability at each step. At each step, the probability of the next node is computed given the paths S, from the root and every given leaf up to the current node to expand. Figure 2(d) shows how after the terminal node Name and its value x are given, pathy originating from this leaf is also used compute the probability of the next nodes.
as well as
Our bash- approach generalizes previous approaches, such as parent feeding YYn and Neubig, 2017), previous action encoding (Yin and Neubig 2017). context nodes (Bielik et al. (0166), and some of the graph-edges of Brockschmidt et al. (0199). See Section g for further discussion.
Generating Trees In sequence generation, the length of the generated quence controlled by generating a single EOS token to stop. When gener- ating trees, we require more sophisticated mechanism control arity and depth. We augment Ap in two ways to allow node-by-node generation.
Groatto
Name
IntExp
X
First, we add special ECSnode node every nonterminal control for arity. Generating this node indicates that the parent node has no more children. Second, we decompose each terminal node n. into sequence of terminal nodes T, by splitting up the node's value into subtokeny based on camel tol tol.werCase, then T, - lower + case EOStok- We end each subtoken sequence with a special ECStok node to control for depth during generation. Figure] shows example of both EOSnode and EOStok in action.
Figure Aug- menting the AST with EOSmode and ROStok nodes
notation (Allamanis al. 2015). For example,
Node Trees vs. Production Trees While we predict a single node at each step. previous work (Iyrr et al.) 2018 Brockschmidt et al. 2019a) predicts grammar production rule. This more direct grammatical representation decomposes the code a way that often forces the model to predict with partial information. For instance, consider the expression str Substring (3). The
3

---

Under review as a conference paper at ICLR 2020
model of Brockschhidd al. (2019a) would first predict the rule Expr->Expr Substring (Expr), and only then expand Expr str and Expr->3; i.c, the model needs to predict the method name (Substring) before the invoking object (str). Further, the Substring method can get either one or two arguments, forcing the model to choose whether to use the one- or two-argument production rule in advance. Node generation, however, allows to predict the presence of a function call and only then to predict its object, method name, and arguments, rather than predicting these a priori.
We note that there exist other approaches that generate arbitrary number of child nodes with production rule-based models. For example, [Rabinovich et al. (2017) used a "horizontal LSTM" to decide whether or not to generate another child; and Yin and Neubig (2018) presented transition system with "Reduce" action.
MODEL ARCHITECTURE
In the previous section, described how generate code given the probabilities Pr (a, (acr), where act is represented by the set of partial AST paths S.. Here, we present a neural model that estimates Pr (a,]S,) S.). We first apply an LSTM-based path encoder represent each path in S, as a vector (Section 3.1). We then contextualize and aggregate the entire set into a single vector (Section 3.2). Finally, predict the target node a; by combining a limited output vocabulary with a syntactic copy mechanism (Section 3.3
3.1 ENCODING AST PATHS
Given a partial AST path (node sequence "i,,..., ny)), our goal to create a vector representation.
We first represent each node using embeddings. A subtoken node is represented by the index of its subtoken w in the embedding matrix Esubtaken; AST nodes are represented as pair n, = (T, k) where 7 is the node type, c.g. IfStatement, and к is the node index among its sibling nodes. We represent node types using learned embedding matrix Etype and the child indices using a learned matrix Findex The node's vector representation is the concatenation of the type and index vectors.
E [Eiyre E Eindex]
Eaubtoken
n; the subtoken w
e (n:)
n; is the AST node (T, k)
We encode the entire path using dii-directional LSTM stack, and take the final states
f (n1..... .i.....,n)) LSTM (e (m.)......c (nx))
Given set of partial paths S (omitting the iterator for simplicity), denote their encodings as H H= (f (nI...., n2)) (nI,..., S}.
Efficient Computation When processing an entire tree, there are large overlaps between paths from different time steps. In particular, paths that originate from the same leaf share the same prefix. We therefore apply the LSTM on the prefix once, and cache the state across suffixes, speeding up both training and inference significantly. An example is shown in Figure in the Appendix.
3.2 AGGREGATING MULTIPLE PATHS
Given the set of paths S leading up to the target node's 's parent T Ta)), our goal is to represent S as vector, in the context of predicting a. To do so, we introduce the aggregation function (H, i). As its input, takes the set of encoded paths H, the encoded root path r, and the child index i of the currently predicted child node a relatively its parent.
We first contextualize the path encodings H using transformer encoder (Vaswani et al. 2017 In
parallel, we apply a non-linear transformation to the encoding of the root path r =f (R), in order inform it that we wish to predict the i-th child of π (a;):
Z Transformer (
F W. ReLU (G.rr
Replacing the LSTMS with transformers yielded similar results in preliminary experiments. Since H is set, do not use positional emheddings.
4

---

Under review as a conference paper at ICLR 2020
In this formulation, the parameter matrix C2 is used when the child index is i, while the parameter matrix Wa is used for every instance.
We then compute attention root-path encoding F as the query, pass the weighted average and the root-path encoding 7 through another tully-connecto layer and denote the resulting vector representation as h:
the set of contextualized path encodings Z with the
a = soltmax (Z.P)
Σ ag* Z3
g ReLU (W, [:
(5)
3.3 PREDICTING WITH SYNTACTIC COPY MECHANNSMM
We can now predict from the representation h. If the target node's parent (aa) is a nonterminal AST node, then a must be an AST node; otherwise, a is a subtoken.
Predicting AST Nodes We predict using softmax over the node type embeddings Etyee
Pr (a|S) softmax (Etyee
(T is nonterminal)
66)
Predicting Subtokens Programs repeatedly refer to previously declared symbols, resulting in highly repetitive usage of identifiers. We therefore add a copy mechanism (Gu et al. 2016) allow our model to predict either entire tokens or subtokens that already exist in the context. As we show in Section copying greatly improves our model's performance. For brevity, we describe how entire tokens are copied, and elaborate on the copy of subtokens in Appendix We score each leaf & using bilinear function (W.) between its path's s encoding H₃ and h. At the same time, w, which is the token associated with f, from a limited vocabulary using the inner product between its representation in the subtoken embedding matrix Esubtoken and h.
the token
Scopy (c) He W
Sgen (w) E
Esubtoken
The scores Scopy and $gen are then summed over different occurrences that correspond to the same symbol, and subsequently normalized via softmax. A key difference from previous work (Gu et al.] 2016 Yin and Neubig (017) is that our copy mechanism uses the syntactic relation between the source and the target (AST path), rather than their sequential relation. Yin et al. (2019) proposed a graph-based based copying mechanism that is capable of copying both tokens and subtrees from the context.
EXPERIMENTAL SETUP
4.1 BENCHMARKS
Any-Code Generation (AnyC2C): Java We take the Java-small dataset of AAln et al. (2019) which contains 11 GitHub projects, broken down to a single method per example, and split to train/dev/test by project reduce code overlap. This dataset was found to contain the least code du- plication by Aflamanns (20185). We create AnyC2C examples by selecting every expression larger than single AST node as the target, using the remainder of the method the context. We remove methods that contain the word "test" in their body or file name, and remove methods longer than 20 lines avoid auto-generated code. To make the task even harder, we remove examples where the target subtree appear as-is in the context. This dataset contains 1.3M/10k/20k train/dev/test exam- ples. The average number of targets for our model is 10.8; for the seq2seq baselines the average is 7.8 targets; if we modeled our targets using production rules, the average would have been 7.9.
Restrict Code Generation (RestrictC2C):: C# Since the restricted expression generation (Re- strictC2C) dataset of [Brockschmidt et al. (2199)) is not publicly available, we consulted with Brockschmidt et al. directly and use the dataset of Allamaniss et al. (2018a). This dataset con- tains 30 GitHub projects broken down one method per example, and use the "unseen projects test" split. To create RestrictC2C examples, we use the code of Brockschmidt et al. (2019a) which filters out examples where the targets contain non-primitive types or user-defined functions. We extract the exact same types limited expressions. This dataset contains 16k/8k/3k train/dev/test examples.
5

---

Under review as a conference paper at ICLR 2020
Detailed statistics of all datasets provided in Appendix A
Metries We follow Brockschmidt et al. (2019a) and report exact match accuracy at and 5. We also introduce new tree @ k metric, which counts prediction as correct if the entire tree structure, ignoring leaf values, is identical to the tree of the ground truth. For example, the expressions x > and y 2 would not count as identical in exact match, but would count "tree-match identical" because both express that an identifier is greater than an integer (NAME > INT). tree k is interesting because allows to tease apart the model's 's syntactic errors from incorrect subtoken predictions.
4.2 BASELINES
We compare our model to a variety of original implementations and adaptations of existing models. We put significant effort perform fair comparison, including adding copy mechanism to the NMT baselines and subtokenization as in our model. We adapt strong baselines from the literature to our task, even if they were designed to different tasks such as NL->oode and code-NN. We re-train all the following baselines on the same datasets as our model.
Neural Machine Translation We use standard autoregressive sequence-to-seqqueee NMT base- lines, in which we subtokenize the given code snippet, replace the target in the source with a special PRED symbol, and train the network to predict the target as sequence of subtokens. Transformer- base+copy (Vaswani et al.) 2017) uses the implementation of OpenNMT (Klein et al.) 2017) with a copy mechanism (Gu et al. 2016). Trunsformer-smull--oo is a d = 512 2-layer bidirectional LSTM
uses dmmadel = 256, di = 1024, and
4 self attention heads per layer. encoder-decoder with attention (Luong et al. 2015).. seq2tree+ follows Aharoni and Goldberg (2017) and learns to generate the linearized, subtokenized target AST, with the same architecture as BiLSTM-LSTM+HC -cTM+
Java-specific Baselines We used the original implementation of Iyer al. 2018), and also their seq2prod baseline which is a re-implementation of Yin and Neubig (2017) 7): these are designed for NL- tasks, in which we feed the (sub)toke code context as the NL input. The model of Iyer et al.] (2018) is designed to get additional input of the available variables and their types, for which we do not feed types. While in theory these models could also be applied other languages, their implementation only supports Java.
C#-specific Baselines We compare our model to GNN-NAG using the original implementation of Brockshmiddr et al. (2019a) which contains additional improvements using ideas from Cvitkovic al. (2019). Bielik et al. (2016) kindly trained and tested their non-neural PHOG model on our C# dataset. We note that PHOG does not have an explicit copy mechanism, and considers only context to the left of the target code, while we consider also context to the right. Extending PHOG to use copying and considering more context could potentially improve its results.
In both Java and C#,, we compare code2seq (Alon et al., 2019a), which is a strong co model and train it to generate the target code as a sequence of subtokens.
4.3 IMPLEMENTATION AND HYPERPARAMETER SETTINGS
Architecture We use embeddings of size 512, layers of LSTMs with 256 units, and 4 transformer layers with 8 attention heads. We kept subtoken vocabulary of size 1,000 to encourage the model to learn to copy; larger vocabularies did not show an improvement. These resulted in a very lightweight model of only 15M trainable parameters, which is close to Trunsformer-small (11.8M parameters). In comparison, the Transformer-aat model had more than 45M trainable parameters.
Training We train the model end-to-end using the cross entropy objective and the Adam optimizer Kingma and Ba 2014), an initial learning rate of 10 decayed by a factor of 0.95 every 20k steps. We bucket examples based the number of predictions in the target subtree (nodes + subtokens EOS symbols), and vary the batch size such that each batch contains about 512 targets. We train the model to prefer copying entire tokens rather than copying subtokens, if possible. We apply dropout of 0.25 in the Transformer layers, and a recurrent dropout of 0.5 in the LSTMs.
Inference We perform beam search with width of and optimize for accuracy@) 1.

---

Under review as a conference paper at ICLR 2020
Model
acc
ace@
@
tree
code2seq (Alon al.
2019a
10.68
15.56
30,46
43.94
Iyer
(0188)
5.94
9.19
25.54
36.75
seq2prod (Yi and Ncubig 2017
8.05
11.82
30.77
41.73
Transformer-small Vaswani of al
2017
+copy
14.23
21.35
31.83
47.40
Transformer-base Vaswan al.
201
pccpy
16.65
24.05
34.68
50.52
BiLSTMM>LSTM, [Luong cl al.
2015)
+copy
16.93
23.17
34,29
49.72
seq2tree (Aharoni Goldberg
2017
+copy
16.81
23.04
38.14
52.36
SLMM (this work)
18.04
24.83
39.10
55.32
Table I: Results on Any-Code-to-Code Generation (AnyC2C) in Java.
Model
acc |
acc @
tree @
tree @
Ablation
acc @ 1
acc @
GNN-ANAG
15.19
27.05
26.48
40.09
Paths
12.95
18.52
code2seq
6.20
10.05
21.97
30.89
Seq-+Path
12.12
17.12
seq2seq4copy
26.42
37.94
34.10
49.23
Paths->Paths
17.63
24.62
seq2trec+copy
22.29
35.86
31.85
48.53
No Root Att
14.43
18.48
PHOG
7.40
12.00
No Copy
10.72
15.70
SLM (this work)
37.61
45.51
51.10
59.82
SLM (original model)
18.04
24.83
Table 2: Results on RestrictC2C in C#.
Table 3: Ablations on AnyC2C in Java.
RESULTS
Any-Code Generation: Java Table shows that our SLM achieves over 1.1% and 0.78% bet- ter acc@1 @c and acc@5 (respectively) over the two strongest baselines. The improvement over Transformer-smie small, which is closer to our model the number parameters, even higher: over 3.8% and 3.4% in acc@ and acc@
In general, the NMT baselines performed better than code-specific baselines. We hypothesize that the reason is that the NMT baselines are more generic, while the code-specific baselines designed for different tasks: seg2prod is designed for tasks which involve generating code given natural language input; [yer et al. (2018) additionally expects all member methods, variables, and their types as input; code2seq is designed to generate sequences rather than code, and does not have copy mechanism. An approximation of code2seq with a copy mechanism is presented in Section 6
Interestingly. the syntactically- innfetically-informedd informed seq2tree baseline achieved the highest tree k among the baselines, while our model achieved higher acc@k and tree@k. This shows that leveraging the syntax can be beneficial in NMT baselines as well.
Restricted Code Generation (RestrictC2C): C# Table shows the results for the RestrictC2C task in C#. where seq2seq+copy is the WILSTM->LSTM+H among the Java baselines. We first observe that the seq2seq+cppy and the seq2tree+copy baselines outperform the RNN-~ACA of [Brockschmidt al. (2019a). who introduced this task. Although Brockschmidt et al. (2019a) did compare to seq2seq baseline, their GNN GNN-WAA model could copy symbols from the context, but their bascline did not. To conduct fair comparison with SLM model, we equip the seq2seq and seq2tree baselines with copy mechanism. Even though the seq2seq copy and the seq2tree+copy baselines perform substantially better than the state of the art in this setting, our SI.M model is able to go beyond, achieving significant gains over all models.
model which performed the best
Examples for predictions made by model and baselines be found in Appendices D and E.
ABLATION STUDY
To understand the importance of the various components and design decisions in our model, we conducted an extensive ablation study the AnyC2C task in Java.
7

---

Under review as a conference paper at ICLR 2020
private void handleTaskFinisste
(TaskFinishedEven) event) {
Task Info task info.tasksmap..0 event w
taskInfo.counter: = event event.get(Counteraal]
taskInfoo.finistiime event-gettPiniahTimme))
taskinfo Teactaats..tttee..cceennttnntoo (ortiig
EaskInto.successfuukee ttw.rraeeeaaeeooottttooootaaaaat
event
True ref:
event. (eent.getTaskId())
event. getattaeeTa ()
(8.8%)
(tree-match)
SLM top-5 candidates:
event. get.
(8.2%)
(tree-match)
gvent.get.aakk
(3.4%)
(tree-match)
event. event (eent.getTaskI())
(3.3%)
(tree-match)
(3.3%)
(exact match)
Figure 4: A Java AnyC2C example from test set along with the predictions of our model. The predictions of the baselines are shown Figure Appendix D
Paths->Seq SSeq follows code2seq (Alon et al. 2019a) and separates the model to an encoder and decoder, where the decoder generates the target subtree a sequence of subtokens. The main difference from code2seq is that Paths->Seq includes copy mechanism, as in our SLM model.
Seq Seq- follows Rabinovich et al. (2077) and separates the model an encoder and decoder (including copy mechanism), where the encoder encodes the context sequence of subtokens
using a BiLSTM, and the decoder uses only the root path and the index of the generated child.
Paths Paths uses separate encoder and decoder which both are AST-path based. These encoder and decoder have different parameters, unlike our SLM model which models the context and the prediction using the same components.
No Root Attention uses max pooling instead of attention in aggregating multiple paths (see Sec- tion 3.2 The index-informed path from the root to the target's parent (R in Figure 2 is concate- nated with the result, instead being used as attention query.
No Copy replaces copy mechanism with much larger vocabulary (25k subtokens instead of 1k).
Table shows the results of these alternatives. The significantly lower results Paths-teq and Seq-> Path show the great benefit of using unified structural language model, instead of separate encoder and decoder components. While this separation between encoders and decoders might be necessary in semantic parsing (Rabinovich al. 2017 Dong and Lapata 2018), NL-xcode (Yin) Fernandes et al. 2019) tasks because of the different modalities of the input and the output, this separation may hurt performance when the output is essentially missing part of the input's AST. As expected, Paths- performs better than code2seq (Table T). as it includes a copy mechanism and code2seq does not.
and Neubig 2017) and code-NL (Alon et al. 2019a]] |
As SLM performs better than Paths-Pathss this ablation shows the importance of joint modeling of the context and the target subtree by parameter tying. Each of Paths and the seq2seq baselines (Table [ performs better than Paths->Seg and Seq->aath; this shows the importance of using the same type of encoder and decoder for the AnyC2C task, rather than combining "an opti- mal encoder" with "an optimal decoder".. Paths-PPal performs better than the seq2seq baselines (Table []. showing the advantage of using paths over textual sequences, even without parameter tying.
No Root Attention degrades acc@) and acc@5 by 3.6% to 6.3%. This shows that dynamically attending to the context paths given the current root path is crucial, even though the root path is necessarily included as a sub-path other paths in the set S, which go through self-attention.
Not using copying mechanism results in degradation of 7.3% to 9.1%. Programs symbols and identifiers repetitively, thus the ability to copy symbols from the context is crucial for this task. For this reason, we included copying mechanism in all NMT baselines in Section
8

---

Under review as a conference paper at ICLR 2020
protected void checkkäpckkdninkkcssssl throws IOException, AccessContro.Exxee:..n Iser@roopthfoomation ugi
otw.r//rrrrroooooooooooooooots/s ()
OserGroupIIffomatioon zkfcUgi =
if (adminAcl, iodm_nnkcl..UUsssAALLLooo.. (ugi) 11
wgg.ggetShorttserWamme eqqulll zkfeUgi. Setruopp.gttshotttssraaaaaee
LOG. L0G.info(()) ("Allowwaaea) RPC access from ugi
"
Server.gettRemoteeeeo ());
return;
String = "Disallowed RPC access from " ugi at Server Server. sew.er.geeeemtteddrrsssssss ()
Not listed in " Drsconfigkeys DFS_ADIN,
LOG.warn (mag); throw new aocessCont (mag)
True ref:
exfeUgi..ettShortteeeelaa ()
XkceUgl.getthhortiseeaaae ()
(11.7%)
(exact match)
SLM top-5 candidates:
DFSConfigKeys.DDF
(4.5%)
:kfcUgi.
()
(2.6%)
(tree-match)
:kfcUgi.
(1.7%)
(uree-match)
ExfcUgi.getUueee
(0.6%)
(tree-matee
Enrively copied tokens marked in brown; unknown copied subtokens marked in blue; in-vocabulary subtokens marked in black; subtokens that both in-vocabulary and copied from context are marked in purple.
Figure 5: A Java AnyC2C example from test set along with the predictions of our model. The predictions of the baselines are shown in Figure 7 in Appendix D
7 QUALITATIVE ANALYSIS
7.1 CORRECT TREE, INCORRECT IDENTIFIER ASSIGNMENT
As shown in Section there is gap between @cc@k k and tree k across all models: when ignoring identifier values and considering only the tree structure, the accuracy significantly higher. Our SLM model performs better than all baselines in acc@k (Table further, our model also shows greater potential for improvement in its tree results which are much higher than the baselines.
We focus on the studying the cases where the tree structure was predicted correctly, but the model A a representative example for
failed to generate the code exactly including names. this case: the ground truth event get.attieetTaakd () was predicted correctly only the fifth candidate: nevertheless, all top-5 candidates are a "tree-match" since all of them express a method which is invoked on an object without arguments, of the form: NAME NAME (). Generating the correct method name get, task, id] is very difficult in this case, since neither get TaskId nor TaskId appear the context and there no apparent hint for them.
7.2 USEFULNESS OF COPY MECHANISM
As shown in Section 6 the ability to copy is crucial for the AnyC2C task, because of the repetitive use of identifiers and symbols in programs. Figure representative example for the necessity of the copy mechanism: generating the ground truth zkfcUgi get only thanks to the copy mechanism, since zkfc is obviously an UNK subtoken which was not observed in the training data.
is feasible
In this case, since both zkfcUgi and getShortUserNand appear in context, both were copied as entire tokens, rather than generated using subtokens. This example also shows how the ability to copy entire tokens ease the generation process by reducing the number of target symbols (our SLM model is able copy and combine single subtokens well).

---

Under review as a conference paper at ICLR 2020
8 RELATED WORK
Generalizing Previous Approaches Our approach frames code generation as predicting the next node in all partial AST paths. This simple framing generalizes most previous work, without hand- crafted special edges and actions:
All models that use information about ancestor nodes only (Rabinovich et al. 2017| Maddison and Tarlow 2014), as well the "Parent Feeding" of and Neubig (2017), are generalized by our model, since all paths that go into node a, pass through its parent, and the path from the root R₂ (Figure 2 is used as the attention query.
The "previous action encoding" of and Neubig (2017) is also special case of our approach, because S, contains the paths starting from the previously expanded leaves of Ap into the currently expanded node (az). such as pathy in Figure 2(c)
The "context node" of PHOG (Bielik et al., 2016) is just one of the peviously-traveese nodes in act. Thus, not only that our model conditions also takes into account the syntactic relation, i.e., the path, between the context and (ax). single leaf, SLMs condition on every leaf in act- Finally, Brockschmidt et al. (2019a) define special graph edges (e.g.. "NextSib" and "Child") to capture relations on the AST. Most of these relations can be expressed as partial AST paths.
leaf this context node as well, model
Moreover, while PHOG conditions
Program Generation Learning to generate programs is one of the oldest problems in machine learning (Waldinger and Lee, 1969) and has been considered by some the "holy grail of computer science" (Pnueli and Rosner Gulwani et al. (017). Typically, the task is to generate a program given some form of imput or context, such as complete formal specifications (Green 198T] or input- output examples (Gulwani] 2011 Devlin et al.
2017 Parisotto et al.) (017). While these approaches work well in some cases, they are bounded to DSLs that prevent them from being applied realistic, general- purpose code. Maddison and Tarlow (214)) Amodooo et al. (2017) generate supposedly general- general-purpose purpose code in a modern programming language, but do not deal with the challenge of set of APIs; they state that their approach is thus intrinsically limited to generate API-heavy programs and is unusable for general, logical programs lacking external calls. Further, their generated programs in an only "Java-like" language. Yin and Neubig (2017). [yer et al. (2018) and Rabinovich et al. (017)) generated general-purpose code well, but for another task of generating code given natural language description. Yin et al. (2019) generated "any code" given an encoded edit that needs to be applied to a given code snippet.
fitting the code to given context. Murali al. (2017) generate code conditioned
Other work used datasets that are either small (Ling et al., 2016), containing highly aligned examples
(Oda et al.) 2015; Chen et al., 2018), limited-purpose languages like SQL Yu et al. 2018), or ceneral-purpose purpose but containing eminently templated programs (Ling et al.) 2016)) Brockschmidi jet al. (2019a) limit their model to generate only expressions of primitive types or arrays these; use closed vocabulary; and ignore expressions containing user-defined functions, because function names are hardcoded their syntax production rules, In this paper, we lift these constraints and allow any, -preeralp generation of code, of all types and containing any names. Our work also related to Habash (2004). who used structural n n-grams over dependency trees for statistical machine translation (SMT).
general-
9 CONCLUSION
We presented novel approach for generating code given surrounding context: computing the prob- ability of an AST using a structural language model. We show that our approach generalizes most previous work in this area, while reaching state-of-the-art performance on of challenging bench- marks. We are eager to see future work advance SLMs further, and apply them to other real-life coding applications as well as other structured-data domains.
REFERENCES
Roee Aharoni and Yoav Goldberg. Towards string-to-tres neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- page) pages 132-140, Vancouver, Canada, July 2017. Association for Computational Linguistics. 0
www. ac web. haabbborg/anthh. hatpallllyyyyyp///p//
10

---

Under review as a conference paper at ICLR 2020
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. arXiv preprint arXiv: 1a111882..06,,, 2018b.
Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting Foundations of Software Engineering, ESEC/FSE 2015, pages 48-49, New York, NY, USA, 2015. ACM ISBN 978- 1-5503-3755-8 doi 2at10.0144/77880557778868 URL hittp://det acm. oro/ 2786805. 2786949
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018a. URL https:
//openreview. www.://et/rmeeemmmmmmm Beedd--JBBEEER ETxR-
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation
for predicting program properties. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, pages 404-419, New York,
NY, USA, 2018. ACM. ISBN 978 5698-55.56666...0 7/doi. acm. org 10. 1145/15/555 923392336662 9192412
http:
Uri Alon, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured rep- resentations of code. International Conference on Learning Representations, 2019a. URL https://<ppnneww/
net net heee//oouumlddHigggo00000
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed rep- resentations code. Proc. ACM Program. Lang., S(POPL):40:1-40:: January 2019b. ISSN 2475-1421. doi 10.1145/5290353
URL /oi awm.iaac..oom wrr.orggg//lo 10.
Matthew Amodio, Swarat Chaudhuri, and Thomas Reps. Neural attribute machines for program generation. arXiv preprint arXiv: 092311, 2017.
Pavol Bielik, Veselin Raychev, and Martin Vechev. PHOG: probabilistic model for code. In Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 2933-2942, 2016. URL httpt:tt/////t //pp:///mmrrr.// orrg
proceedings papers/ /v4 v48 /tt//belliie www.beeekkk..................... html
Mare Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. Generative code modeling with graphs International Conference on Learning Representations, 2019a.
U.RUhttts:://ppeeerr/ net.eeiwwwwweeeeeeeeeeeeeeeeeeee
net fttc:/foummiii (d-.mmmid--ek-KKAAA5
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Advances in Neural Information Processing Systems, pages 2547-2557, 2018.
Milan Cvitkovic, Badal Singh, and Animashree Anandkumar. Open vocabulary learning on source code with graph-structured cache. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 1475- 1485, 2019. http::
mlr www............m..m............p
html
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International Conference Machine Learning, pages 990-998, 2017.
Li Dong and Mirella Lapata. Coarse toefseet--fineeee decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742, 2018
Patrick Fernandes, Miltiadis Allamanis, and Mare Brockschmidt. Structured neural summa- rization. In International Conference on Learning Representations, 2019. URL https:
wtwp:rrwiiew..ee//orummmmmm ersoRRqm
Cordell Green. Application of theorem proving to problem solving. In Readings in Artificial Intel- ligence, pages 202-222. Elsevier, 1981.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. Incorporating copying mechanism in sequenne-to-sequuene learning- arXiv preprint arXiv: 1603. 1603.0633,, 2016.

---

Under review as a conference paper at ICLR 2020
Sumit Gulwani. Automating string processing in spreadsheets using iuput-output examples. In ACM Sigplan Notices, volume 46, pages 317-330. ACM, 2011.
Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, al. Program synthesis. Foundations and Trends@ in Programming Languages, 4(1-2):1-119) 2017.
Nizar Habash. The use of a structural n-gram language model in generation heavy hybrid machine translation. In International Conference on Natural Language Generation, pages 61-69. Springer, 2004.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. arXiv preprint arXiv: 1808, 1808,09588, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6/412.69800 2014.
G. Klein, Y. Kim, Y. Deng. J. Senellart, and A. M. Rush. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints, 2017.
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Fumin Wang, and Andrew Senior. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 599-609, Berlin, Germany, August 2016. Association for Computational Linguistics. Utthttps: / www ac www.www.bccom org/ anthology/ P16 1057
doi 1o.100.8853/v1/11GG6
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1166. p
1412-1421, 2015. UtLhhtp:://aat
Chris Maddison and Daniel Tarlow. Structured generative models of natural source code. In Proceedings of the International Conference on Machine Learning Volume 32, ICML' 14, pages II 11--9911-577..0 JMLR.org. 2014. URL. htRt:/tp://dl acm. wtt/ciiaaiiiooooroooo.ooo
3044805.. 3044965
Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communications of the ACM, 14()):511-15, 1971.
Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. Bayesian sketch learning for pro- gram synthesis. CoRR, abs/1703.05698, 2017. URL http://arxiv..rr/ab/////1
05698
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation (t). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on, pages 574-584. IEEE, 2015.
Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Push- meet Kohli. Neuro Neuro-symbolic program synthesis. ICLR, 2017.
Amir Pnueli and Roni Rosner. On the synthesis of a reactive module. In Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pages 179-190. ACM, 1989.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code genera- tion and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139-1149, Association for Com- putational Linguistics, 2017. 100110.01533///111111100 URL http: / /www wwwwwww.booooo
anthology/F17-1105
Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. Learning programs from noisy data. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL '16, pages 761-774, New York, NY, USA, 2016. ACM. ISBN 978- 478-1-4503-354977 4o0100.145/2836642288888 10. N5.1115528377111 221767787771
URL http::/ doi. acm. org/
12

---

Under review as a conference paper at ICLR 2020
Richard Shin, Miltiadis Allamanis, Marc Brockschmidt, and Oleksandr Polozov. Program synthesis and semantic parsing with learned code idioms. CoRR, abs/1906.10816, 16, 2019. URL http:
10816
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pages 1000-6010, 2017.
Richard Waldinger and Richard CT Lee. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference Artificial intelligence, pages 241-252. Morgan Kaufmann Publishers Inc., 1969.
Chunyang Xiao, Mare Dymetman, and Claire Gardent. Sequence- based structured prediction for se- mantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1341-1350, 2016.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code genera- tion. Proceedings of the 55th Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers), pages 440 450. Association for Computational Linguistics. 2017. 10. 16653/11/117-1041
URL h/w.tt::://w..//. ac web http://wwww...eraaaaaooaoooo///.
Pengcheng Yin and Graham Neubig. Tranx: A transition- based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 7-12, 2018.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L. Gaunt. Learning to represent edits. International Conference on Learning Representations, 2019. URL
httpss://ppenreve/
net http//oooummiddd.e.. JC5F
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
Brussels, Belgium, October-Novembee 2018. Association for Computational Linguistics. 1o1110.1863//1//111 1/25
URL https: www. aclweb. orq/ argganthology/
D18 212-1425
13

---

Under review as a conference paper at ICLR 2020
Table Statistics of our datasets, When not mentioned otherwise, the statistic was measured on the training set.
Java
C##
#projects training
25
#projects - dev
1
#projects test
1
#examples training
1,309,842
16,295
#examples dev
10,000
8,183
#cxamples - test
20,000
3,305
Avg. number paths
27.8
131.1
Avg. source length lines
10.4
57.5
Avg. source length tokens
77.7
264.3
Avg. source length subtokens
100.6
343.6
Avg. target length tokens
5.4
3,9
Avg. target length subtokens
7.8
5.0
Avg. target length tree nodes
3.8
3.9
Avg. target length targets (including subtokens & BOS)
10.8
10.8
A DATA STATISTICS
Table shows some statisties of used datasets. In Java: for the validation set, randomly sampled 10, 000 examples from the raw dev set; for the test set, we randomly sampled 20, 000 examples from the test set.
B CODE GENERATION PSEUDOCODE
Algorithm shows the pseudocode for our depth-first, left-to-right, code generation approach. We keep a stack (line 1) which initialized with an initial node to expand. We loop while the stack is not empty (line 3), and pop (line 4) the a nonterminal, we predict child node (line 7). If the node is a terminal to a subtoken, we predict subtoken from vocabulary or copy (line 7). the predicted child node, whether AST node or subtoken, can be further expanded (line 10), it is pushed back to the stack (line 12).
node expand at each step. If the node expand is
C COPYING SINGLE SUBTOKENS
In addition to scoring the entire token to be copied, we also score each of the subtokens composing it according to their position. For each position we add scoring function Scopy such that Scopy. (E) produces the copying score of the i' i'th subtoken of f, which we denote as fç:
S
8gen (w) +
Σ val(E)=w
Scopy token (0)
Σ
Σ
Scopy: (2)
(8)
val(E.)
Pr (a|S) = softmax (s)
Where Scopy.token is the scoring function of copying the entire token, described Section 3.3
tokeni For example, token of getX is scored entirely using i each of its subtokens, get and x, are scored using Scopys and Scopuz respectively. That is, the model can either copy the entire token, copy only some of its subtokens. This ability is especially useful in generating name like setx, where get appears in the context, and x is any unknown, difeeedfined. subtoken; the model learss to generate set from the vocabulary, and copy only the subtoken X.
D JAVA EXAMPLES
Figures 5-12 contain examples from test for the AnyC2C task in Java, along with the pre- diction our model and some of the baselines. The highlighted expressions are the true references that should be generated. Indentation and line breaks may have been altered for typesetting reasons.
14

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Pseudocode for the code generation algorithm.
Input partial AST A initial node to expand do
Output: subtree
slack <- emptyStackO:: push(slack, an): initTrec(au):
while stack not empty do parent + pop(stack);
+- -eccoddPatss(AA .T, parent);
parent E nonterminals then
a -rredietNode(S, parent);
else if parent E terminals parend subtokens then
4- -rredicSSubooken(S, parent);
10
a is not EOS then
-insertChild(( parent, y); push(stack, a);
13
return T;
ltExpr
Greater
time-4 4
Name
IntExp
time=3
1
X
Figure 6: Efficient computation: partial paths for different time steps share the same prefix, allowing shared computation. In this example, the prefix is the shared path from the leaf (not shown) to Greater, and is much longer than either of the suffixes.
E C# EXAMPLES
Figures 13-20 contain examples from our test set for the RestrictC2C task in C# along with the prediction of our model some of the baselines. The highlighted expressions the true references that should be generated. Indentation and line breaks may have been altered for typesetting reasons.
15

---

Under review as a conference paper at ICLR 2020
private static String getNameserviceId Configuration conf, String addressKey) { String nameserviceld = conf. (DFS WAMESEEEEERVC (nameserviceId != mull)
return nameserviceId;
Collection<Strings nsIds getNameServicoIds (conf);
(1 nsIds.size
return nsIds. (new String [ i
String nnid conf.get return
getSuffixIDs addressKey, null, nnId, LOCAL ADDRESS MATCHER) [0]
Model
Predictions
True ref:
nalds.sizel)
nsIds. nsids.size(
(83.7%)
SLM (this work)
conf.size
(3.0%)
getSuffixIDs (conf). length
(2.5%)
-1
Transformer-base +copy
ns.size()
conf.size ()
-1
BiLSTM--LSTM +copy
Integer.MAX VALUE conf.size()
Seq2tree +copy
StringFool.blase blank
protected void check?pckddiinkccess/l throws I0Exception, Accesconntrol.xxepp..oo userGrouppnnoomation ugi UserGroupIInformation zkfcUgi = oarttroopp.fommtiiooooeeoooooogo () if (adminAcc1.isUserAAlla (ugi)
http:///www..ttt...ooooooorreeee ();
ugi wgg..getShorttssrrlann .quuass sxteedpp..eethhorttsssssmmee
106.1nfo ("Allowed RFC access from + ugi
+ " at Server gereer.getRemoteaddree..
return;
String msg - UDisallower RPC access from ugi Server Server.get www.teeettteedddeeeeeee...... ()
at
listed in DrsCon gkeys DFS_ADMINN
warn (msg);
throw new AccessControLExxpptiio (mag);
Model
Predictions
True ref:
2kfedggi ()
(11.7%)
SLM (this work)
DFScontigKeys.cr3
(4.5%)
(2.6%)
server..gctccnecccdddrcsss
Transformer-bas +copy
serveet.getRcceettttsssrtaaa
serves..gttShorrtttssraaamil
server.gettUseeNaaaeell
BiLSTM-LLSMM +copy
2kfeUggi.geetUsezNWaane
ugiDgii.geUUseeNaamee
dfstonniggkess.dfa.hih
Seq2tree +copy
2kfc.gettSserName
()
zkicUgi. Set.bii.ggetemmtteccreees...
Figure 7: Java examples from our test set along 16th the predictions of our model and the baselines.

---

Under review as a conference paper at ICLR 2020
private C findCounter (T key)
int i key.ordinal if (counters = null)
counters = newCounter (key)
return
(C) counters [i]
Model
Prediction
True ref:
(C) counters [1]
(C) counters [i]
(71.6%)
SLM (this work)
(C) this
(6.3%)
counters[i]
(4.8%)
(C) this
Transformer-base +copy
(C) [i]
(C) counters
this
BiLSTM +LSTM +copy
(C) counters [i]
counters
(C) counters [i]
Seq2tree +copy
counters [i] ordiind
(C) counters.get (1)
private vold handleraakkPinishedwwww (TaskFinishoddvent event) {
TaskInto taskInfo info.taskaMap.ger eweett.geaass..e. );
taskInfo.counter event.getcounterss)]
faskInfo.finishti event evont.getPiaaaahiimo
taskInfo.status TaskStatus wtw.:ussssttttt.e...uueee tostrigg
= event huttssccc.et..tssssttttett.....
Model
Prediction
True
event.get geant.IIeTask
event.getTaskWamee ()
(8.8%)
SLM (this work)
event.getId()
(8.2%)
event.getTaskk ()
(3.4%)
event.gett/lasklnnoll
Transformer-hase +copy
event.get7askIdd()
event.get
event.name
BILSTM-LLSTM +copy
event.type
gvent.get.di
event.get
Seq2tree +copy
event.getPath ()
event.getDeseriptionn
Figure 8: Java examples from test along with the predictions of our model and the baselines.
17

---

Under review as a conference paper at ICLR 2020
static String replacesubstitution
String base, Pattern from, String boolean repeat) {
Matcher match from.matcherrbase))
(repeat)
return match.replaceA11 (to)
else
return match. reachhrrplacePiii (to);
Model
Prediction
True ref:
match. match.replacce roplaccalll (to)
match matsh.tostring(()
(9.0%)
SLM (this work)
match.replacealll (to)
(8.2%)
match.replaccell (to, from)
(6.5%)
(atch.rreplacerirstttt)))
Transformer-ben +copy
reppacee.replaccefiitttt (to)
mitchher.espLaaeeiirsttttt
match.getrirstt/ match.rreplacefissttts
BiLSTM -LSTM +copy
match.repplaceFisttttoo to)
match.rreploccrr1rsstaaeaa))) (base)
Seq2tree +copy
ratch.rreplacceirsttt ratch.rrelaaeeFisst (repeat))
public void responseReceivedd veeereeeeeeee.eeesssseeeeeeeeeee event) (
RequestResult result event seent.getteguesskkeulttt
Date startDate reault.getsta:tuatetll
Date stopDate = result. Sesult..etSttooatee(>>
long elapsed = stopDate.getTimel
SetartDate.getTimm ()
synchronized (this)
Latt.lasttyylatenny clapsed;
if LOG. (0bbbgggaana))ee
int statusCode result. wet.tettttttttee( String etag = result.getRtagy HttpURR.Coneettoon urlConnection
(#ttptRRLConeection) event peeettggee. ()
int contentLength ur..mmeeriio...g..otttttttteeeee
String requestMethod w:..mmeeri.....teeeeeteeeeeeee
long threadid LOG. debug (String. "sel "sed.fhhhtttt1iiineerrrt:: ResponseReceived:
())
threadId=$d, Status=$d, (ms)
STAG=$s, contentIength-skd, requestMetthod=ks",
threadId, statusCode, elapsed, etag, contentLength, requestMethod))
Model
Prediction
True ref:
elapsed !- nu11
(32.1%)
SI.M (this work)
LOG. NoG.isDDbuggnabled
(29.0%)
IL0G.isDebughaalled
(2.4%)
stopDate 1= null
Transformer-ban +copy
hesutt.nasttauusCdee ()
nesstt.nstt () elapsed
result !- mu11 elapsed
BiLSTM--LSTM +copy
result. Seullt.getSttussoodell
event wvent.geeCommeettooooooeee () instenceof #ttppu=Coonectiin
Seq2tree +copy
startDate != null
LOG i.g.iseebugEnbbled/
18
Figure 9: Java examples from our test set along with the predictions of our model and the baselines.

---

Under review as a conference paper at ICLR 2020
private static boolean (anaameeesooleeeednnddddddddddd address) {
String hostname Cedress.getHosttiee ()
string 1p = address @otHostAddresaa) ();
return Thostname equals (ip) || NetUt ils www..ssooooaaaraaaoo.ooooooa.eee (address); ;
Model
Prediction
True ref:
address HetHostName ()
address gmteottameettteeeeeeeeeeeeeeeeee ()
(3.5%)
SLM (this work)
address Addresss.getHostNaaae ()
(2.0%)
meetAddress..ettByyaa. (address, getAddress ())
(0.7%)
address.ggetBosttddeeess ()
Transformer-bal +copy
address..geLLastEEeememtttt () getValue ()
address..eetAdbresss/
acrress.etibsttdddeesss
BiLSTM-+LSTM +copy
address.geetPortii)
address..eetAdbress/
address..gelBotttdddeesss
Seq2tree +copy
address.getPortti)
cadress.getäddrees
private synchronized void initJournals (List<URI> dirs) int miniiunndddnddddtoorraa.ss conf.getInt(
D#sConfigKeys. DFS NAMENODE SAMENODE_EDITSS S
EDITS _DIIS_DIRR___ DIR MINIMUM KEY,
DFSConfigKeys. L_____MMNOO____EE EDITSEEDISS______SSSS_ SEDDIT_DIRR_____________I__ MINIMUM journalšct = now Journalset inhhimumsecdundanttUuurr.... for (URI dirs)
MISIIIIMIRRMMMMMMMMMMMMMM MINIMMMEDEFAULL)))
boolean zequired
httaammesssttee.egteeeiiiiiii http://www....eoooooooeeeeeeeeea aonta (u)
if u. (u.getscheme
StorageDirectorry sd stocagp...eettoorrgggoereeeeee (u);
if (sd !- null) jou
rileeournallaaageeeoooo (conf, sd, storage),
required, sharedEditsDira
www.siirr..oooiins.....ooo...eee contains (u) ;
} else
journalset. add a (u),
zequired, sharodEditsDirs. contains (u));
if (journalSe) 1eett isRmpty )))
e.G.error ("No edits rectories configured!
Model
Prediction
True ref:
g.gettcheme
u. u.get ()
(27.4%)
SLM (this work)
a.getScheme
(13.1%)
u.getVersion
(8.2%)
fournal.Set. LOCAL URI. SCHEME
Transformer-base +copy
u.getName
Boolean.true
u.toString ()
FILSTM>>LSTA +copy
Boolean. true
u.getURI ()
u. u.getScheme ()
Seq2tree +copy
u. g. ()
Figure 10: Java examples from our test set along #9th the predictions of our model and the baselines.

---

Under review as a conference paper at ICLR 2020
static Enunset<FileAttzzittt parse (String s) {
if (s == mull 1I s. () == 0) return Enumšet. anum (f
Enunset<FLleattrbbbte> set noncof (Filcattriburcocclas)) FilcAttribute attributes values
for (char s.toCharArray
int i - 0; for (; i attributes.
!= attributes [i] symboli; i++) 7
(i attribuuts.
it (!set. iatercoonnaisssttttiitttess (il))
set add. else {
throw new Illegallarqueentxxcpppt
("There are more than one "
+ attributes [i] symboll + "1 " s)
else throw Illegllreyumnntxxepppiala""""""" s " is undefined.")
in "
return set;
Model
Prediction
True rel:
s.toCharArray ()
s.toCharArzay ()
(22.4%)
SLM (this work)
attributes [0] value
(18.5%)
attributes fined] llenghh
(4.6%)
s.split
Transformer-base +copy
set.split(" ")
sttributes. ()
attributes.length
BiLSTM-LLTTM +copy
attributes
attributes next
set.toArray()
Seq2tree +copy
t.CCharArrry
set.toCharArray/ ()
public static Path|] stat2Paths (FileStatus stats)
(stats null)
return null;
Path[] ret new Path[stata length];
for (int i 0; i < stats. stats.Length. Length; ++i)
re] stats|il .gttPath
return ret;
Model
Prediction
True ref:
stats[:. get ()
stats (tattta)_.etPath(( .getPath())
(25.2%)
SLM (this work)
new Path (stata [i])
(3.3%)
Path (statslá) charset)
(2.5%)
stats[1] stats [i] (dd..tt)))
Transformer-ban +copy
new Fath (staatsii)
statss []] new Fath
BiLSTM-LLSMM +copy
stats toStringg
stats new (statssats stat (stats (il)
Seq2tree +copy
20
Figure 11: Java examples from our test set along with the predictions of our model and the baselines.

---

Under review as a conference paper at ICLR 2020
void ensurecuuzeentDiiEXiitttt () throws ISException:
for (
[terattor<Storrgspiitttttttt it storage.dirlteratoo () it. att.hasNex(().
StorageDirectory
nex (
File curDir = ad.ggtuuuretttii
it [xuuriieeexists) & lourDir.mkairsll
throw new IcException ("Could not create directory curDir);
Model
Prediction
True ref:
!curDir.exists]
courDir.exists. ()
(29.0%)
SLM (this work)
curbir !- nu11 curDir.exists
(25.8%)
(24.4%)
curDir nul1 !curDir.exists(( curDir.exists() curbir !- null curDir.exists sd != mull curDir null curDir.exists: !curDir.
Transformer-base +copy
BiLSTM-ALSTM +copy
Seq2tree +copy
public static byte[] getxAttr (final Map<?, json, final String name) throws 10sxception if (json =- null) {
return mall;
Map<String, byte[]> xättrs toXAttrs (json) if (xAttrs I= null)
return xAttrs.ge (name)
return mull;
Model
Prediction
True ref:
xAters.get (name)
xAttrs get (name)
(28.2%)
SLM (this work)
xAttrs. get (xAttrs)
(5.8%)
Atttrs.toytearrnn ()
(4.4%)
xAttrs.get (name)
Transformer- base +copy
<Attrs.toBytearsn ()
new byte
xtttrs.getBytes
GiLSTM--LSTM +copy
new bytel0]
XAttrs.toBtterrra ()
xAttrs. get (name)
Seq2tree +copy
xAtirs.
xAters. get (C)
Figure 12: Java examples from our test along with the predictions of our model and the baselines.
21

---

Under review as a conference paper at ICLR 2020
private void setFlag flag) {
long prev)
do {
prev unsafe. www.aa.. wew.ongVoaatile (null, this thiss.sotAccress)
if (prev & flag) != 0)
return;
while
null, this sotsdssst prev, prev flag) )
Model
Prediction
True ref:
(prev & flag)
(prev & flag)
(8.9%)
SLM (this work)
(prev & flagSlot)
(5.4%)
(5.0%)
(prev & flag)
Transformer-base +copy
(prev | flag)
unsafc. conparccmpareTooo (prev)
prev prev +
BiLSTM-ALSTM +copy
prev
unsafe prev flag
(Snntax error)
Seq2tree +copy
(volatile prev unsafe.get ())
(Syntxx error)
(volatile prev wew.fe.gelllnngooolloo. (mn prev))
(Syntax error)
public synchronized void setInput (byte[] b, int off, int len) { if (b == mull) { throw MollPointerBxxcettioont
if (off 0 II len < 0 II off > b.length len)
throw ArayyTabsoOtt0roooddeeeeeseieddd ();
finished false;
if (len > recorpreeeeeDi.rcccfff....aaaain ()) {
this.userBuf b;
userBuffff off; this.userBufLen len;
else { (ByytcBuffer) uncopreseedbrecc.Bu)) .ptt (b, off, len); uacoppresadDreetbufffe uncopprsseeiirrett...p...t.
bytesRead
1en;
Model
Predictions
True ref:
len 0
len < 0
(41.3%)
SLM (this work)
off b. length
(23.4%)
belength (14.1%)
off < 0
Transformer- +copy
0
null
off < 0
BILSTM-LSTM +copy
len 0
null
off 0
Seq2tree +copy
len < 0
off
Figure 13: Java examples from test set along with the predictions of our model and the baselines.
22

---

Under review as a conference paper at ICLR 2020
private int readData (bytel] [] buf, int off, int len) throws IOException { int bytesRead = 0,
while (bytesRead len)
int n wooiiiswwappaaaaaacooopeeeeeeeee
in, buf, off bytesRead len bytesRead); ;
if (n 0) return bytesRead;
bytesRead n;
return Len;
Model
Prediction
True ref:
off bytesRead
bytesRead bytesRead
(35,0%)
SLM (this work)
off bytesRead
(14.1%)
off bytesRead
(9.4%)
off bytesRead
Transformer- +copy
len
len bytesRead
-bytesRead
BiLSTM-LSTM +copy
bytesRead++
bytesRead bytesRcad
compressed bytesRead (Syntax error)
Seq2tree +copy
off + bytesRead
len bytesRead
private Path getPath (int curId, int limitPerDir, Type type) if (curId <= 0)
return basePath;
String name "";
switch (typc) case FILE:
name [IRFF_PRFTXX_CC Integer (curld limitPerDir) .twperrini...ttinngiign ()
break;
case DIRECTORY:
name DIR_IXEFIX + new Integer (curId (lnitPerDir).ootttte ();
break;
Path base getPath ( (curid limitPerDir), limitFerDir, Type. DIRECTORY)
return
new Path (base, name)
Model
Prediction
True ref:
new Path (base, name)
new Path (base, name)
(6.0%)
SLM (this work)
new Path (base, name, limitPerDir)
(2.9%)
Path (base, name, type)
(2.8%)
now Path (basc)
Transformer- +copy
Path (namc)
get.Pat (base)
new Path (base)
BILSTM-LSSM +copy
new File (base)
Patth (base.get(Path()
new Path (base)
Seq2tree +copy
new File (base, name) new Path (base, name)
Figure 14: Java examples from our test set along with the predictions of our model and the baselines.
23

---

Under review as a conference paper at ICLR 2020
private static IBfumerable<<Tken>> Ofßequence this :Bnumerable<Toker> tokens, Token nameToken, TypeDescriptor info)
var nameIndex tokens. IndexOf (t => t.Equals (rameToken) ;
name ndex >=
return info. Next wawpaalleeaaalllaaaaaaaaaa waw.aaaaaeaeerreaaaallltt...a.
=>
info Maw. tmmm MapwalueorDefaultz
=> tokens. Skiens.skipllo maneIndes Take
tokens. Skip InameIndex 1) TakeWhille (v => v. V.Vallue ())),
tokens. Skip [nameInder
Takemhile (v v.IsValue()))
return now [oken { };
Model
Prediction
True ref:
name Index
nameIndex >= 0
(22.6%)
SLM (this work)
Index (19.1%)
nameIndex
(13.9%)
!nameIndex
BiLSTM-:LSTM +copy
LameIndex
nameIndex 0
name Index == 0
GNN NAG [Brockschmidt al. 2019a
nameIndex
name Index
public static IEnumerable<? Group<T>
this I source, int groupSize)
if (groupSize < 1)
throw new ArgomentOutOfllaggeexxcppp (nameof (groupst ze)
TI) group - new [[rroupiize] zel; int groupIndex = 0; foreach (var item in source)
sroup[groupIndex-* item;
if groupIndex groupSize.
yield zeturn group; group = new [[groupSizej; groupIndex 0;
Model
Prediction
True ref:
groupIndex groupSize
group <
(21.4%)
SLM (this work)
groupIndex -1
(10.3%)
groupIndex < groupIndex
(5.3%)
group.IsMhlLOrEmn:yy(
BiLSTM +copy
SatpGGroupplgrooppnddee..
sroup.Endswitthhrrrru
groupIndex == 0
CNN -WAG (Brockschmidt al. 2019a
groupIndex
groupIndex groupSize
Figure 15: C# examples from test set of the RestrictC2C task along with the predictions of model and the baselines.
24

---

Under review as a conference paper at ICLR 2020
internal static void AddLine (StzingBuilder builder, string value, int maximunLength)
(builder null)
throw new ArgumentNu.l (nameof (builder))
(value nul1)
{
throw new ArgumentNu11 Except i on (nameof (val (ralee)) )
if (maximuneength 1)
throw new ArgumeettoouoOPRnggexxxoooooooo (nameof (value)) ;
value malue.Trimg
builder Appeedwhhee (builder. Length Length do
0, Environment (ww.ommmennt..wneeeee NNwWLine);
var wordBuffer 0;
words value. Splitt" '); for (var 0; words Lengtht i++)
if (words [i] Length (naximumLength - wordBuffer)
builder Append (words wordBuffer += words [i] Leigthh,h
if (maximumlength - wordBuffer) > 1 & != words, Length - 1)
builder Append ");
wordBuffer++;
else if (words [i] length >- maximumlenngee 68 wordHoffer 0)
builder Aphend (words 11] Substring maximunLength)) )
wordBuffer maximunlength;
break;
else break;
value value. Value..ubsttrn Substring (Math. (wordBuffer, value. Lelgeh.ength) )
Meallder.AppedWiiie IvalueLLenth 0, Environment NewLine))
while (value. (engh maximunLength) Appedd ;
Model
Prediction
True ref:
value. Trim()
value. Trim()
(16.0%)
SLM (this work)
Value.Substrin (0, maximunLength)
(10.9%)
Replace (maximuneength, maximumlength (10.7%)
maximuneenghh
BiLSTM-;LSTM, +copy
value.Trim
valueLengtht+
value + <UNK>
-NAG
value maximmieenghh
value. Substrin (0, maximamLengt.a)
Figure 16: C# examples from our test set of the RestrictC2C task along with the predictions of our model and the baselines.
25

---

Under review as a conference paper at ICLR 2020
public static stringl] []
TrimStringArray (this IEnumerable<stringg array)
return azray.Select Select (item =>
item. Trim()
ToArray (
Model
Prediction
True ref:
item item. Trim()
item. (rim()
(20.1%)
SLM (this work)
item. TwUppepppuarrraaaaee ()
(3.5%)
ToUpper)
(1.6%)
Stem.Trime
BiLSTM-LSTM tcopy
item.ToTrim()
item.]
(Syntax error)
item + <UNK>
CNN NAG Brockschmide et al. 2019a
item + item
item +
public static string Camel ze (thia string input)
var word Pascalize (input);
return
word, Substring (0, 1)
ToLower ()
word, Substrinng (1)
Model
Prediction
True ref:
word. Substrinn (0, 1)
word. Sord.tubstring (1)
(0, 1)
word, Substrinn (1)
SLM (this work)
word.Trim()
werdData. Substring (1)
word.Substring (1)
word.Substringg(0.
input.Replace ("a", ("
")
<UNK>
BLLTTM>>LSTI +copy
input.Replace (1, (()
input ". input
input. ("6", "*)
input. (0, 1)
word.Cambinewithh (<UNK>) word.Trim()
word. (<UNK>) word + <UNK>
CNN-
word. (input)
word.Replace (<UNK>, <UNK>)
Figure 17: C# examples from our test set of the RestrictC2C task along with the predictions of our model and the baselines.
26

---

Under review as a conference paper at ICLR 2020
public string Truncate value, int longth, string truncationstring, TruncateFrom truncateFron BruncatoFrom.Righe
if (value == null)
return null;
Ivalue.Leagth 0)
return value;
if (truncationstring -- mull) truncationstring Bmrtt:
if (trumationStriing > length)
return truncateFrom == FruncateFrom.RRie Right ?
value..ubsttingg(. length) value. Substtrng (value. (vague.Length Length
length)
var alpaabueerccalllaaactoeeeeeooeee
0;
if "aalue,ToChaaArre Countt (char. IslecterrrDigitt <= length) return value;
if (truncateFrom == TruncateFrom, Leftt
for (var i Value. i i--)
if (char. Ccher.aaattttterrrrsgstaaallllll
if @alpaaaunnrriccllllaaarteeeeeeee truncationStringg Length
length) return truncationString value. value. Substring (1);
for (var i 0; Length truncationstring.leng. i++)
if Calar:IIssttterrooDiccieaaaalall
if (L.paauwnccoolllaaaatoooo..a.. CruncationString Length
= length) return SallurSubstringe (0, 1 1) + truncationstring:
return value;
Model
Prediction
True ref:
a Ibhhhuunrriiclhhaaaatteerroossss
(48.1%)
SLM (this work)
Count++
(5.8%)
indexx+
(1.6%)
i++
BILSTM-ILSMM +copy
truncatient+
alpha--
GNN-NAG
a ab
Figure 18: C# examples from our test set of the RestrietC2C task along with the predictions of our
model and the baselines.
27

---

Under review as a conference paper at ICLR 2020
public static int BinarySearch<TItem, TSearch> this ILlst<TItem> list, TSearch value, FunckTSearch, TItem, int> comparer)
if (list null)
throw new ArgumentNullExooptiioo (ww.eeeeeppoooiiieeotooeeee (" it (comparer null)
throw new ArgumentMulllxxeptiii ("comparer");
lower 0; var upper list. Count 1;
while (Lower <= uppor)
middle lower (upper lower)
var comparisonReste
comparer (value, list [middle]]
if comparisonResult < 0
upper = middle 1;
else if
comparisonResult
lower middle 1; else
return middle;
return lower;
Model
Prediction
True ref:
compariscnResulc <
comparisonResult > 0
comparisonResult <
comparisonResult >
SLM (this work)
compariscnResie >
comparisonkesult <
middle compart Resoll
comparisonResult ==
ToxerResult: -- mi ddle TowerHesult --
lower < lower + "."
STL +copy
lower != middle
lower != middle
compariscnkesult -- 0
comparisonResult
GNN-NAG
compari >
comparisonResult >
comparisonResult <
comparisonResult == middle
Figure 19: C# examples from our test set of the RestrictC2C task along with the predictions of our model and the baselines.
28

---

Under review as a conference paper at ICLR 2020
public override string ToString
reflection to display all the properties that ... have non default values StringBuilder result = new StringBuilder O); var props =
GetTypeInfo () Declareddroperttess:
result.AppendLil (")";
foreach (var prop in props)
if (prop.Narr Name != "Content" && prop. Name != "Subtitle"
prop.Name & prop. Drop.Name Name != "Titie" & prop. Name != "UniqueId")
object value wwtpppoop.pa..pp.ppppooooo GetValue (this);
bool valuelaNu11 va' ue null;
object defaultvalue - Common.GetDeefaae (wt.:/a.oopoppppoooppp..pop Propertrppryy)))p)) bool itfauuttalueelsuii if ((valueIsMull != defaultValueIsNN11) // one is mull when the other isn't
defaultvalue -- null;
11 IvaluelaNo11
&& (value. ToString != ( ()))) // both aren't 't mull, so compare as strings
repend. (prop. Name + prop.GetValue (this));
result. ("e return result. (oString
Model
Prediction
True
IvalueIsNull
!valueIsNull
(52.4%)
SLM (this work)
IdeffuultValueIsNull
(9.0%)
WwaiucIIotull.raaooooEEeEEeE ()
(3.2%)
[GefaultValueIskul]]
BiLSTM-+LSTM +copy
(dcfauultValueIsNulll 1I value)
[defaultValucIsMuul] II dofaultValue)
!valueIsNull
GNN NAG (Brockschmidt
2019a
[defaultValueIsstu1]
!.valueINuull
Figure 20: C# examples from our test set of the RestrictC2C task along with the predictions of our model and the baselines.
29

---

Under review as a conference paper at ICLR 2020
public "rraieerrdde erorderkeepoonse Pl aceerrddrr (string account id, @eaderrrrder11ar classi fl cation,
arOrderDirecttin rectionn,
string symbol, decinal quantity, decinal price 0, decimal stop string optionSymbol ""
TradicrOzderType type = cradderrrderyypo Market,
TradierOrderDuuatta duration = TradierOrrarbbuaatioo.....
//Compose the request: var request new RestRequest "=ccotts//cccontta//ree/ss/
request AddUe ("accountId", accouttid.rost:ing())>
//Add data:
request request.AddParammnne AddParameter (*class", GetBnumbesecripti (ccassi))c)tion))
("synbol", symbol); request Adduratttddeerrreteeer ("duration", request Reguestt.AdParaaeeee. ( ("type", GerEnunDescription (type))
ietEEnubeescriptiin (duration)) )
("quantity", quantity);
request. Addaarameaareeeeer ("side", GetEnumDescript (direction)))
//Add optionals:
if (price request waaeuesst.daaaaaaaaeccca Www.AAddPaaae.... ("price", Math. (price, )));
if (stop 0) aeqqesst.ddPaaaamee. ("stop", Math. (stop, 2));
if optionnSmbol != ""
request AddParamateeeeeee. Proption_sybbbr optionSymbol)
//Set Method: request.Methan Method.FOST;
return Excuttetrradeerrddeeeepppsss (request, TraddierppPReepestypppo...e.r
Model
Prediction
True
optionSymbol != "n
optionSymbol != ".
(5.5%)
SLM (this work)
optionSymbol ""
(4.4%)
optionSymbol. IoNullOrEmpty ()
(1.1%)
!stopSymbcl
BiLSTM->LSTT +copy
stop$ymbel optionSymbol
(stopSymbel & optionSymbol)
(Syntax error)
optionSymbol <UNK>
-NAG
[Brockschmidt
2019a
opt optionSymbel symbol
opt i onSymbol
symbol
Figure 21: C# examples from our test set of the RestrictC2C task along with the predictions of our model and the baselines.
30

---

Under review as a conference paper at ICLR 2020
[Test, TestCasesource orraaaaarreeeeeeeeeeeeaaaaaaaaan public vold oetSborccccttnnecnnnnarrrrgggggg LembaaallhhTeeesrrraattesss parameters)
var source parameters. Data GetSource(
i parameters. tim. false);
var n new Fileinfo (surrofo FPu1 FFullName; var = Leataaaaaaaaeeaatttooiiipppiiiee Giobals..DataFoddeer SarameterssDDaa........ parameters..ataa.Tinee.....
parameters..Resolut.on. parameters. p
var normaliazzipttiePaah new Filetnfo(eipeileeatth .ateih)Fuulllaaa .FullName;
var = - nor.aaedeou..rat........oooo Last
"4", Stw.ingoomarrisnn..rddee Ordinal))
if (indexOffash 0)
{
normalizedSourceat http://esoourrettttc.uutstttiiii indexOfilash)
Aesert..reeRuue Phormalize/in lePath, norma 1 redSourccePath)
Model
Prediction
True ref:
tosmalliiicSuurrePtthi.iiiiiiiid (0, indexOfHash)
Sommalieedoourcaatthssssssiigg (0, indexOfRash)
(28.3%)
SLM (this work)
nermaliieeduuurePtttiisssssgggg (1)
(8.8%)
nermallieddourrccaahRRRnooooo (indexOflHash)
(8.2%)
indexOfHash "<UNK>" indexOfHash > normalized0fHar
BiLSTM-LLSM +copy
indexOfHash 0
normalizedSorceePath:
nozmalize2ipFileePatt
NNGGNAG
normalizedSourcePath + normalizedSourceaatt
normalizedSourceer + normalize2ipF1lefatt + <UNK>
Figure 22: C# examples from test set of the RestrietC2C task along with the predictions of our model and the baselines.
31


Reviews:
[{"review_id": "rJeyiVOptH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a model to address the Any-Code Generation (AnyGen) task, which basically to fill missing code from a given program. The model makes use of partial Abstract Syntax Tree (AST) as input. The model learns representation for partial AST paths and use the learnt representation to generate AST node at masked steps. The conducted experiments show that using AST paths from root and leaves are good for AST node generation, but whether those inputs are robust and sufficient should be further explored.\n \nThere are some restrictions to the method, for example,  the input is only a single function, and the missing expression is not that complex. Nevertheless this work presents a novel method towards code generation. The paper also introduces a new metric to evaluate the prediction accuracy for generated expressions. Writing is clear. Evaluation is fairly comprehensive.\n\nQuestions:\n1. Did the author test the method without the camel notation assumption, i.e. the data contains non-camel notation or mixed notations?\n2. In the Restrict Code Generation test, it seems that the author filters out non-primitive types and user-defined functions. Therefore, does the experiment on Java-small dataset fully show the proposed model’s strength?\n3. Can the author explain why the SLM model fail on Figure 6? Is it because of dividing token into sub tokens?\n4. How big is the token vocabulary? How does the vocab size affect the performance?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJx4Rjq85B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a grammar-based generation approach for \"slot-filling\" style code generation tasks. Given a context AST with opening non-terminal node, the model completes the opening node by predicting a sequence of child nodes, which forms a sub-AST rooted at the original opening node. The proposed model encodes context ASTs using a path-based approach (Alon et al., 2019a), essentially generalizing the previous model of Alon et al., 2019a from a code-to-sequence setting (e.g., generating natural language comments from code) to a \"code-to-code\" setting (i.e., code completion given contextual snippets). \n\nStrong Points:\n\n* The paper is very well written. The idea of formalizing code completion as structured language modeling and extending Alon et al., 2019a for the task is natural and well executed, with strong models and significantly improved results on two code completion benchmarks for both Java and C#.\n\n* The authors attempted to establish comparisons with most existing code generation models.\n\nDetailed Review:\n\n*Technical Contribution* I have a very mixed feeling with this paper, while the model registers high empirical performance, the technical contribution is a bit limited, as detailed below:\n\n    - *Path-based Context Encoding* The most important contribution in this submission is the application of path-based AST encoding model of Alon et al., 2019a to encode context (the given contextual and partially generated ASTs) for code generation. While the path-based encoding scheme is indeed a powerful model that intuitively encapsulates and generalizes over most previous approaches (Section 7), applying the model to a different task without significant task-specific adaptation or in-depth analysis might not sound technically novel. Meanwhile, the core idea of modeling/encoding the information flow in both the given context AST and partially generated programs for opening node expansion has already been explored in Brockschmidt et al. (2019a), albeit using a different encoding approach (GNNs) and in a relatively restricted setting of generating arithmetic expressions.\n\n    - *Node-based Tree Generation Model* Apart from the path-based context encoding model, the node-based generation model presented in Section 2 also seems interesting. However, it might take longer time-steps to generate the node sequence instead of the sequence of production rules (composed of multiple child nodes), which could make optimization and inference more difficult. On the other hand, to control arity, the node-based approach need to inject synthetic \"EOS\" nodes to signal end of generating an argument sequence, while existing production rule-based systems could easily generate arbitrary number of argument nodes using either a transition system (e.g., Yin and Neubig, 2018) or a special neural component to compute end-of-argument-list probability (e.g., Rabinovich et al. (2017)), without using separate production rules of different arity.\n\n    - *Syntactic Copy Mechanism* While the proposed syntactic terminal token copy mechanism (Section 3.3) could be better than the vanilla sequential one, there have already been syntactic copying models capable of copying both terminal tokens and partial ASTs from the context (Yin et al., 2019). \n\n    How to Improve: to better understand the different technical contributions outlined above and their relative impacts, the following ablation studies would be helpful:\n\n        - Importance of Path-based Context Encoding: the Seq→Path ablation in Table 3 alone might not be adequate to demonstrate the importance of path-based encoding of AST contexts for code generation tasks. The authors should compare with the GNN-based context encoding approach in Brockschmidt et al. (2019a) as this is the most relevant work. The original GNN→NAG model cited in Table 2 used a much simpler copying mechanism and a vanilla production-based tree generation model, and therefore not directly comparable with a tuned SLM.\n\n        - Importance of Node-based Tree Generation Model: If possible, the authors might consider swapping their node-based tree generation model with a state-of-the-art production-based approach (e.g., Yin et al., 2019) to demonstrate its effectiveness.\n\n*Claims* The authors claimed in the beginning of the paper that previous program synthesis approaches are either restricted in domains (e.g., DSLs like SQL) or grammars (e.g., restricted grammar of the full language), therefore coining the proposed approach as \"any-code generation\". However, there are indeed code generation systems (some of them cited in this paper) that synthesize open-domain code snippets in general-porpuse programming languages without restriction on vocabulary or grammar. To give a few examples, Iyer et al. (2018) generate open-domain Java class member functions; Rabinovich et al. (2017) and Zhao et al. (2019) predict full Python classes or partial snippets, while Yin et al. (2019) synthesize open-domain short C# code diffs observed in GitHub commits. In fact, the the proposed \"any-code generation\" benchmark is limited to sub-expressions defined within a function, whose scope is more restricted than other benchmarks like CONCODE. \n\n    How to improve: the authors might present more evidence to substantiate the their claim on the novelty of the AnyGen benchmark compared with existing open-domain, general purpose code synthesis benchmarks, or consider revising the claim and the title.\n\nReferences:\n\n* Zhao et al., Neural Networks for Modeling Source Code Edits. 2019\n* Yin et al., Learning to Represent Edits. 2019\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJe4jZK0FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a generative task for programming code where an expression from the program is generated given the rest of the program (minus the expression). This is in line with language modeling for natural language. The proposed method generates the AST corresponding to the program by generating one node at the time for the missing/to-be-generated expression by approximating the probabilities of the generated notes. Again, this is similar to the prediction of words in language models.\n\nThe method takes into account the AST corresponding to the program. However, when representing the program, the structure of the AST is not preserved, instead, the AST is represented by generating several sequential paths by traversing paths between connected nodes in the tree.\n\nIt would be nice if the paper provided some intuition why generating such connecting paths in the tree are relevant for representing the code, specially for nodes that do not have a direct relationship between them (e.g., the nodes are distant enough in the code that the corresponding probabilities of their nodes do not seem/appear related).\n\nThe paper presents results for 2 datasets (comparing with various related work methods). The results for the Java dataset improve state of the art by 1-2%, while the results for the restricted C# dataset show a much more significant improvement (in the order of 10-15% improvement, depending on the metric).\n\nI would have liked to see a qualitative analysis of the results. In particular, I would have liked to understand how the predictions differ between acc and tree metrics. In other words, when the prediction looking at the tree structure is correct and the overall prediction is not, what goes wrong?\n\nIt was not clear to me why or if all the paths between 2 nodes are necessary when encoding the partial AST and predicting the missing nodes. I was not convinced that the ablation studies were relevant. I would have liked to see ablation studies that considered a subset of the paths in the graph.\n\nThe elimination of the methods with more than 20 lines of code seems ad-hoc to me and biases the evaluation with relatively short methods (how many methods were eliminated this way?).\n\nOne thing that I struggle with is understanding how useful the proposed task is and how it can be generalized/used in practice for some relevant higher level task in AI4code.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for code generation based on structured language models.

After viewing the paper, reviews, and author response my assessment is that I basically agree with Reviewer 4. (Now, after revision) This work seems to be (1) a bit incremental over other works such as Brockschmidt et al. (2019), and (2) a bit of a niche topic for ICLR. At the same time it has (3) good engineering effort resulting in good scores, and (4) relatively detailed conceptual comparison with other work in the area. Also, (5) the title of "Structural Language Models for Code Generation" is clearly over-claiming the contribution of the work -- as cited in the paper there are many language models, unconditional or conditional, that have been used in code generation in the past. In order to be accurate, the title would need to be modified to something that more accurately describes the (somewhat limited) contribution of the work.

In general, I found this paper borderline. ICLR, as you know is quite competitive so while this is a reasonably good contribution, I'm not sure whether it checks the box of either high quality or high general interest to warrant acceptance. Because of this, I'm not recommending it for acceptance at this time, but definitely encourage the authors to continue to polish for submission to a different venue (perhaps a domain conference that would be more focused on the underlying task of code generation?)
Decision: Reject