Code Editing from Few Exemplars by Adaptive Multi-Extent Composition
Authors: Peizhao Li
Xuchao Zhang
Ziyu Yao
Wei Cheng
Haifeng Chen
Hongfu Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: code editing
few-shot learning
compositional generalization

Abstract: This paper considers the computer source code editing with few exemplars. The editing exemplar, containing the original and modified support code snippets, showcases a certain editorial style and implies the edit intention for a query code snippet. To achieve this, we propose a machine learning approach to adapt the editorial style derived from few exemplars to a query code snippet. Our learning approach combines edit representations extracted from editing exemplars and compositionally generalizes them to the query code snippet editing via multi-extent similarities ensemble. Specifically, we parse the code snippets using language-specific grammar into abstract syntax trees. We apply the similarities measurement in multiple extents from individual nodes to collective tree representations, and ensemble them through a similarity-ranking error estimator. We evaluate the proposed method on two datasets in C\# and Python languages and respectively show 8.0\% and 10.9\% absolute accuracy improvements compared to baselines.

Under review as a conference paper at ICLR 2022
CODE EDITING FROM FEW EXEMPLARS BY ADAPTIVE MULTI-EXTENT COMPOSITION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper considers the computer source code editing with few exemplars. The editing exemplar, containing the original and modified support code snippets, showcases a certain editorial pattern, and code editing adapts the common pattern derived from few support exemplars query code snippet. In this work, we propose a novel deep learning approach to solve this code editing problem auto- matically. Our learning approach combines edit representations extracted from support exemplars and compositionally generalizes them to the query code snippet editing via multi-extent similarities ensemble. Specifically, we parse the support and query code snippets using language-specinn grammar into abstract syntax trees. We apply the similarities measurement in multiple extents from individual nodes to collective tree representations for query and support sample matching, and ensemble the matching results through similarity-ranking error estimator. We evaluate the proposed method on C# and Python datasets, and show up to 8.6% absolute accuracy improvements compared to non-composition baselines.
INTRODUCTION
In recent years, a surge of interest has been witnessed in applying machine learning techniques to
code editing (Zhao al. 2019 Chen al. 2019 Dinella et al. 2020) Chakraborty et al. 2020 Yasunaga & Liang, (2020). Code editing in software engineering intends to revise the design, structure, functionality, or implementation of existing programming codes into a desirable pattern. To maintain high-quality code projects or merge several repositories, the programmers typically fix one or few code snippets, and expect the same revision automatically applied to other places in demand over the whole project. Automating this process can facilitate a broad range of programming applications such as code migration, refactoring, version update, and bug repair.
The problem of code editing with exemplar(s) aims to adopt the common editorial pattern from given exemplar(s) to a query code snippet. The editorial pattern of an exemplar describes the type of change between two code snippets, and implies the underlying intent of making a kind of edit for specific purpose, c.g., updating some keywords or reformatting redundant brackets. Pink rectangles in Figure
[ provide an example of editorial patterns with the comparisons between previous and edited code snippets. We also present a list of editorial patterns that appear in our dataset in Appendix [ for reader understanding. This problem is conceptually similar to programming by example in software engineering (Menon et al. 20133 Meng et al., 22013), and recently has been studied by Yin et al. p Brody et al. (2020); Yao et al. (2021) in deep learning fashion. Technically, their methods parse the source code as an abstract syntax tree using language- -sseuaaiesscccccccccce grammar, apply one-time or sequential edit actions over the tree, and always keep the modified code snippet satisfying grammatical rules. In particular, all these works focus on performing code editing with exact one editing exemplar, i.e., the one-shot setting.
While the above studies have demonstrated promising results, simply adapting the editorial pat- tern from one single exemplar can lead to poor generalization or even incorrect code editing. Figure [ presents two cases. In Case (a), when a model observes only the Support #1 exem- plar, it may intuitively interpret the code refactoring rule as replacing the conditional expression only for newly-declared variable
("iE (s==null) ").") by a coalesce one ("...??.... ("string s=...") "). Consequently, applying the model to a query input could mistakenly transform the newly-declared variable (*string title-...") rather than its re-assigned correspondence

---

Under review as a conference paper at ICLR 2022
Support n1 string s GetValnell; - null) 6- ";
Support 12 name UpdateName (name); (name null) name "
Support
string sccount Inthlooour/namml:
account Upgrade wew.ecoooooooooonttt (account mill accomat
Previous
Case (a)
string - GetValoe()
name- tipdatekamm/jaam)) "
string account InttAcoont/oame)) account aagrndeecoommaccconnt ?? ",
Query
string title Gettitle(s);;
Solution #1 (incorrec)
tille BeformatTile(itle)
(aitle mulll { title
Support V1
title ReformatTilleitll)
string title GetTitle(ld) ?? mi
Solution #2 (correcd (with Support
string Uitle GetTithefid). title ReformatTidlcttlee
bool x ly false
Support H1
Support +2 bonl x y 7 true
bool x GetValoel) 7 true X5
Support 13
Previous
Case (b)
bool x AR
bool x=y
bool x GetValuel) X
Edited
Solution (correct
-ly 11sss
Query x
Support
Support
Solution (incorrect) (onty with [1) boolxx &&
Figure I: Two illustrative cases of our motivation for code editing learning with few exemplars, rather than with only one. The purple rectangles on the right part show the incorrect and correct solutions without or with multiple exemplars composition, respectively, in both Case (a) and (b). The editing with one exemplar may be dominated by the inductive bias learned from limited samples and thus lead to an infeasible solution. With few support exemplars, the desirable solution can be reached.
Friitee-eeeoraatte (...) .). see Solution #1). However, Case (b) demonstrates a situation where two editing exemplars turn be sufficient for editing model capture the logical trans- formation over code tokens. The foregoing articulates the necessity of leveraging few but not one exemplar to adapt an editorial pattern to new code snippet. The few exemplars help the editing escape from the inductive bias learned from one exemplar, and enhance the generalized capacity for other snippets with various coding contexts.
In light of this, in this paper, we consider the computer source code editing with few exemplars. The editing of a query code snippet triggered by few editing exemplars that containing the original and modified support code snippets. Different from the above one-shot code editing scenario, few exemplars, even within the same editorial pattern, perform divergent coding contexts and edit actions. Therefore, it is not eligible to adapt the edit actions directly from an arbitrary exemplar. For this scenario, the major challenge lies in how to identify and match helpful support exemplars for a query input and compose them to guide the editing.
To address this challenge, we propose a novel approach that combines edit representations extracted from exemplars and compositionally generalizes them to the query code snippet editing via multi- extent similarities ensemble. Specifically, we parse the support/query code snippets using language- specific grammar into abstract syntax trees, where editing actions are executed nodes. Based on this, we model the similarity between the representations of the support and query abstract syntax trees. We notice that among few exemplars, some editing action only happens concurrently some editing action relies on a general coding context. This inspires us to consider multi-extent similarities between the representations of the support and query abstract syntax trees. We design X-softmax function by scaling the importance of nodes in an abstract syntax tree, i.e., means the support/query code snippet denoted by tree representation, while À - co means the support/query code snippet is denoted by one single node representation in the abstract syntax tree. By controlling the intermediate positions between individual nodes and collective tree, we provide multiple intermediate representational query-support matchings, then adaptively ensemble outcomes from these extents through a similarity- rrnkinn error estimator for a robust composition. We use the term multi-extent to describe the scope of query-suppon matching at various levels from nodes to a collective tree representation.
local region, while
We evaluate the algorithmic performance two code editing datasets, one in C# (Yn et al. 2019) and one in Python (collected by us). We implement the proposed multi extti-extent exemplar composition mechanism top of the ofate-of-the-art Graph2Edit model (ao al. 2021). On both datasets, our
2

---

Under review as a conference paper at ICLR 2022
model outperforms baseline methods by 8.0- 10.9% in terms of absolute accuracy. In addition, experimental results show that (1) enabling few support exemplars can dramatically improve code editing; (2) compared with treating all support exemplars equally, precisely capturing the code snippet similarity and compositing support exemplars are crucial for code editing; and (3) the multi-extent approach offers better performance compared to its single-extent counterpart and other existing methods. Our collected PyFixer dataset and model implementation will be released.
PRELIMINARY
In what follows we shall use C- and C+ to represent the abstract syntax trees of the input (previous) and the output (edited) code snippets. We address the problem of code editing learning with few exemplars. Giving set of K support exemplars with an identical editorial pattern {[C C4 k k== the goal is to adapt the editing from support set to a query snippet C and obtain its desirable C7.
Here we introduce one of the tatte-of-the-arte edit learning models, Graph2Edit Yao al. 2021). that builds the one-shot code editing framework with three major components. (I) A tree encoder based on Gated Graph Neural Networks (Li al. 2015) to embed the input abstract syntax tree C-_ into node representations Z E NxD, where N is the number of nodes in the tree and D is the latent feature dimension. The whole tree embedding & can be obtained by graph pooling operation over all node representations; (2) An editing encoder to embed input C_ and corresponding output C.+ into vector ʃa (C_, C+) E IR P to represent the entire L sequential editing actions (four types of actions: add/delete node, add subtree, stop editing) from C__ to C+ as well as the procedural editorial pattern; (3) A decoder for editing action prediction working conditionally on the tree embedding and editing representation fa(C_, C+). Predictions include the operational type, the executive location on abstract syntax tree, and associated action values, e.g., to be replaced by which node. Denote t1 (4,,...,,4 44) the tree embedding history along the sequential editing, and d1: (aI.-..,, at) as the editing action history until step (. Formally, the whole predictions of actions can be represented as conditional likelihood Pr a C+). model Graph2Tree (Yin et al., 2019), where Graph2Fdit derives from, shares the same methodological concept but edits the input in one-pass, instead of sequentially predict and apply the edit actions.
I Pr (a C+ ), 412). ). An earlier
The Graph2Edit framework is trained in a self-reconstruuttion paradigm. The training feeds the tree encoder and the editing encoder with pair of C_ and C4,, and maximizes the likelihood of the whole predictions of actions towards the shortest sequential actions produced by dynamic programming. The training details with the dynamic programming algorithm can be found in Yao et al. (202 The training only sends the vanilla editing encoder with the ground-truth from one exemplar lar, while {(C Cf H
with an identical editorial pattern are hard to collect from a wild code dataset.
Based on the training, Graph2Edit is mainly designed for code editing with only one support expporte It is struggled to directly tackle few supports since few exemplars, even within the same editorial pattern, perform divergent coding context and edit actions. Unfortunately, we argue the one-exemplar case lacks the capacity of compositional generalization on editorial patterns as we discussed within the introductory part. We also empirically evidence this claim in our experimental section. Our method composites fA C%+ from support set based on multi-extent the query edit representation f for decoding purpose which done by the code editing frameworks. In the next section, we elaborate on our proposed adaptive multi-extent similarity ensemble on exemplar composition address the foregoing challenges computer code editing with few exemplars.
matching to form
3 ADAPTIVE MULTI-EXTENT COMPOSITION
We assume that if the input tree of a support exemplar enjoys more similarity with the query input tree, the query snippet is more likely to adopt the support exemplar's edit representation towards correct code editing. Based on this, we address compositional learning from a query-support matching perspective. Deriving from the basis of code editing reference, some editing can be adapted depending on individual node representation and regardless of its neighbor nodes and context. For instance, the removal of redundant bracket should partially be independent of the inside context, or the change of some outdated function calling mode in a previous language version should be invariant to the host object. On the other hand, some editing highly relies on the contextual information
3

---

Under review as a conference paper at ICLR 2022
among several objects and connections therein, leading the editing learning to be determined by collective representation. Continuing with this analysis, we design a multi-extem measurement to perform multiple intermediate query-suppar matching between individual nodes and the collective tree. Specifically, we measure the multi-extent similarities between query and support snippets in the tree embedding space, where we can define the coverage of nodes in tree want to involve for matching (from single node to every node), then ensemble and learn a convex combination of the editing representations from the support set maximize the adaptability for query snippet editing. In the coming subsections, we introduce the query-suppon matching over abstract syntax tree, extend with multi-extent representation in an ensemble mechanism, then illustrate the meta-learning paradigm and inference procedure.
3.1 QUERY- SUPROR MATCHING: FROM COLLECTIVE TREE TO INDIVIDUAL NODE
Instead of treating each node equally, our model softly emphasizes some nodes in the query snippet when meet similar node in support exemplar, and vice versa. Considering query snippet and a set with K support exemplars, let 2%. and 2 denote the n-th node representation in the abstract syntax tree of the query and k-th support snippet, respectively. Let (:: 2 x 2 R be query-support node matching similarity function (e.g., a neural network) with a learnable parameter 0 that takes the node representation from both sides as input. Note that (:.(..) does not have to be non-negative or symmetrical since the measurement across query and support can be deemed as directional. We compute the initial query-suppons node matching activation m4 and m th as follows:
and m m = max { (p
N,
8. = max (ee
ppl (3% 27
}
(p (2),
:=1
where N2 and N₂ are the numbers of nodes in the query and support abstract syntax trees. The maximum operation over the set of one-to-all node emphasizes the matching between individual nodes, and returns a high activation when there least one good match in the counterpart. The activation represents the likelihood that query node appears in a support snippet and vice versa.
To capture the multi-extent matching, we design a X-softmax function by scaling the importance of nodes in an abstract syntax tree, and seek the query-suppor matching at the intermediate position between individual nodes and a collective tree:
exp (Amg.**)
exp (Am (.
9.8% A,A
and
94.9 n,
INI exp (Am?."H)
Eis exp (Am, 2L.9)
9.5% denotes the parameterized node activation from the query snippet to the k-th support snippet, and similar expression can denote as o #2.4 Based on the above normalized activation for individual node representation, the collective tree representation of the query and support snippets can be calculated using a weighted average pooling as follows:
No. Σ
:= K
n. = and 1
SE n.u Am
The representation of the query snippet has the summation of matching values on nodes from all support exemplars. Combining Eq. (2) and (3), an intuitive interpretation on variable raises: a larger X denotes greater domination of matched individual nodes in the final tree representation, i.e., the sharpness of the outcomes after A-softmax normalization. + 0 preserves the final query/support representation as to their initial tree representation, meaning it calculates a more smoothly weighted average of node representations over the entire tree. X + ac represents the final query/support representation approximately with only single node representation at most of the time, where the node is selected upon the maximum activation from Eq. (T) Setting a large value implies a smaller coverage over the tree. This coverage scaling property holds due to the monotonous increase of the first-order gradient of the exponential function. We will tackle the X value selection problem in the next subsection. Currently, we reach the generation of edit representation for query sample editing. and the updated tree representation yields the expression:
K
f8 :
de (t (N
(C(C..CC))
(4)
A=1
4

---

Under review as a conference paper at ICLR 2022
where de (:.)) is similarity measure query and support tree representations and should satisfy the convexity in the above combination. Note that for simplified expression, we do not distinguish the learnable parameter 0 for each module but their parameters are isolated.
3.2 EXLTI-EXTENT COMPOSITION AND ENSEMBLE
The X-softmax in Eq. controls the matching extent from individual node to collective tree. However, it is hard to access the optimal intermediate position, since code editing samples suffer from huge variance. Therefore, a single extent may not be robust enough help the model generalize in diverse editing scenarios. We hereby propose a multi-extent measurement to complement the above weakness by setting À with different continuous values, and ensemble all the results de o (t3 *xx for
{A) to enhance the generalization on complicated code snippets. However, setting À arbitrarily and ensemble all perspectives linearly without selection may corrupt the overall tree representation. and thus deteriorate the model. For robust ensemble and aggregation, we consider assessing the quality of Φo ( 1x,) towards different X; by using a marginal ranking error on edit representation. Since we have the accessible ground-truit C72 during training, we employ the ranking error ex, to explicitly reflect how good (t (x*) ) is under a certain À with respect to the similarity between query and K support exemplars. In the inference phase, due to the missing of C4. we employ (12, 12) x...... 1* >₂) ( IR to prediet
similarity-ranking error estimator éx2 the quality of each λ, where μi is the learnable parameter in Ry, and ensemble the outcomes. Training details of R2 will be elaborated in the next subsection.
To achieve multi-extent composition and ensemble, we first measure the similarity between query and support edit representations via the editing encoder:
(fa(C2, C4), fa("", CCC)) = [IfACC2,C) ffa C
(5)
Then we access the error for each by:
K
(wre) - wr(w) max {0.7
0,, - (0o(12, -00 tx (
CX
(6)
where ((.) is the index mapping for K support exemplars that satisfy WT(1) 2 WT(2) 2... WT(k). ie., the mapping to sort 8; in a descending order, and y the margin set as a hyperparameter. The formulation indicates how well the nearest support exemplars can outperform other support exemplars, and the factor (Wr(1) Wr(x) reveals the error confidence. Note that the above error term depends on the edit model fa. which is only an approximate estimation. Then we extend Eq. by involving multiple extents and the reciprocal of its error for an adaptive ensemble:
K
1
1 =
do (0 (S) C++).
ex
where is constant in denominator for numerical stability. We omit the normalization over do do(t2, 1xtt (/ex, + c) in the equation. In practice, we apply a softmax function on this term to keep the convexity of the combination. Note that in inference ex, will be replaced by êx.-
3.3 LEARNING AND INFERENCE
In the learning phase, we have set of K support exemplars { k and one complete query snippet {C", C² as inputs. Following the Graph2Edit framework (Yao et al. 2021). we obtain the =g and = of the n-th node representation of the query and k-th support snippet and the edit representation fa (C Cft) ) for each support exemplar. - is the number of steps towards the shortest sequential actions from C to C²,, and 01=14 denotes the corresponding action sequence. Let
e = (EA, Ex2 ..., Exy) calculated by Eq. (6) and = (Éx, êxg...., Exx) is its estimation from the estimator Ry. With NA extents, our objective function can be written as:
min Pr (a1.L= 1,2, 19) + ||e||a + DKI((ele))
(8)
5

---

Under review as a conference paper at ICLR 2022
where || 1/1 is the vector 1 norm to sum up the absolute values of all elements, DxL (-11-) is the KL--divergence, S the composition of edit representations in Eq. 77. 1 Σn 2 N₂ is the initial tree representation of C", O and i are the learnable parameters the similarity measurement de. SPg and the estimator Ry We omit the balanced factor for each term for a simplified expression. The above objective function consists of three parts. The first term maximizes the likelihood between the learned convex combination of edit representations and optimal editing actions, while the rest two terms aim minimize the ranking error between the query and support snippets different extent levels and train an estimator to accurately estimate the error for inference usage. The parameters for tree encoder, editing encoder, and decoder are obtained via pre-training stage on another data split following the selfsreconstructioon paradigm from Graph2Edit, while their parameters are not tuned in the meta-learning stage for optimization, since we observe a serve overfitting if doing so.
We follow an episode training mechanism (Snell et al. 2017) that is widely used in few-shot learning to simulate the encounter of new editorial pattern in testing. For each training forward, we sample a fixed number of editorial patterns with K support exemplars and one query snippet per class and backpropagate the loss. In the inference phase, given a set of K support exemplars {(C C+*) )){) and one input query snippet C", we replace ex, with Êx, in Eq. (7 and yields the formulation.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTING
Dataset We include two code editing datasets for performance evaluation. (1) C#Fixer (Yn et al. 2019) is dataset containing 2.878 editing pairs, generated by applying two C# fixing tools Roslyn and Roslynator on six projects. A fixer is building on top of the C# compiler used to perform refactoring and modernization on codes. One fixer is designed for one type of refactoring and 16 fixers are selected in this dataset. A pre-trained model for C#Fixer is trained on GitHubEdits dataset using reff-reconstructtion. The dataset collects 54 C# projects on GitHub which extract the source code before and after the commits from users. We use the data of GitllubEdits only for pre-training and follow the training protocol in Yao et al. (202I) but not for few-shot experiments. (2) Benefiting from the flexibility and generalized capacity of abstract syntax description language, beyond C#,, we collect another dataset PyFixer written in Python 2. We use Python-Future Future a tool that edits the codes over 5,959 projects written in Python to satisfy the compatibility of Python 3. We obtain the code pairs by taking codes before and after applying the tool and inspecting the changes. To simulate the pre-training and few-shot learning as in GitHlubEdits and C#Fixer, we select 12 types of fixers and extract a fixed number of each fixer from the main data to form the dataset for few-shot learning, while the rest are preserved for pre- pre-training. After data cleaning, we have 14,941 code pairs for (pre) training, 1, 1,664 for validation, and 2,319 for few-shot experiments. The two datasets only contain source code but without any author's 's name, contact information, or offensive content.
Baseline We employ modifications on two abstract syntax tree- base code editing frameworks, Graph2Tree (Yin al., 2019) and Graph2Edit (Yao et al.) (02I), to fit them into the few exemplars editing setting and involve the following baselines. Note that all baselines are only different in calculating the similarity measure Φ in Eq. 4 RS (Random Selection): we randomly select one support exemplar from the support set and use its edit representation for standard decoding on query snippet; NN (Nearest Neighbor): we identify the nearest neighbor in the support set based on the distance calculated by graph edit distance (Sanfeliu & Fu 1983 Abu-Aisheh et al.) (2155) over AST trees (GED-NN), and cosine similarity over input tree representation (CS-NN). AE (Average Edit Representation): we infer the edit representation of K support exemplars using editing encoder respectively, and take the mean of these K representations for query decoding. This is equivalent include two other baselines within our proposed
to setting Φe = 1/K in Eq. 4 Moreover, composition framework. GED-Comp (Composition via graph edit distance) employs the graph edit distance metric as dh when comparing the query and the support exemplar among every query- support snippet pair; we take the reciprocal proportion of this metric and form a convex combination over K edit representations. CS-Comp (Composition via cosine similarity) means that we employ
hhttp://gitubb.cm/doonntttttttt com/dotnct/rosyyn
httpp://gituu..co/jss/ppppp comjjoscppittrtiolyynttt
http::/pythonn-fuuue wwwtoooooonuuooooooeoooooeeoooeo www.re..og//ineeaaae www.re.org/nddxx.mm.

---

Under review as a conference paper at ICLR 2022
Table 1: Experimental results of code editing with 5 support exemplars on C#Fixer dataset. RS:
random select; AE: average edit representation; GED: graph edit distance; NN: nearest neighbor; CS: cosine similarity; Comp: composition. Please refer to the experiment protocol for full explanation. The middle line in the table divides the methods into non-composition and composition ones.
Macro Accuracy
Split #1
Split #2
Split 43
Split #4
Split #5
Avg.
Graph2Tree-Re
0.270 1: 0.077
0.360 = 0.011
0,341 1: 0.029
0.243 1: 0.009
0.384 0.17
0.320
Graph2Edit-RS
0.279 1: 0.022
0.423
0.008
0,413 4: 0.01
0.225 4: 0.007
0.396 0.017
0.347
Seah2Eddt-GED-NI
0.282 +: 0022
0.450
= 0.005
0.444
+ 0018
0.260
+ 0.011
0.434
=
0.374
Gruph2Edii-CS-NN
0.291 +: 0.010
0.454
0.010
0.453 + 0.019
0.270 + 0,020
0.442
+: 0.018
0.382
Graph2Tree-AE
0.381 + 0.022
0.363 = 0.008
0.372 + 0011
0.275 0.014
0.349
0.025
0.348
Graph2Edit-AE
0.336 1: 0.025
0.471 = 0.012
0,465 1: 0.022
0.267 :: 0.018
0,402
0.023
0.388
Seaph2Edit-GED--omm
0.363 +: 0.019
0.479
= 0.012
0,487 + 0.21
0.302
+ 0.012
0.457
+: 0.026
0.418
Braph2Hdit-CS--Com
0.387 =: 0.066
0,500
= 0.004
0.501
+ 0.16
0.337
+ 0.008
0.507
+: 0.000
0.447
Ours
0.416 +: 0.015
0.514
0.021
0.522 +: 0,015
0,352 + 0.018
0.539
0.023
0.468
Micro Accuracy
Split #1
Split #2
Split #3
Split #4
Split #5
Avg.
Graph2Tree--B
0.290 0.04
0.490 = 0.008
0.332 + 0.032
0.166 + 0.008
0.619 0.00
0,380
Graph2Edit-Re RS
0.281 0018
0,538 = 0.010
0.417 + 0.002
0.167 = 0.010
0.599 0.000
0.400
Baaph2EdiiGED--N
0.282 +: 0.000
0.559 = 0.003
0.450
+ 0.019
0.183 + 0.006
0.616
0.016
0,418
Graph2Edit-CS-NN
0,289 +: 0010
0.563 0.010
0.459 1: 0.021
0,186 0,010
0.624 +: 0.00
0.424
Graph2Tree-AE
0.391 1: 0.017
0.546 =: 0.006
0.353 +: 0.013
0.177 +: 0.005
0.673
0.010
0.428
Graph2Hdit-Ae
0.343 +: 0.199
0.603 = 0:011
0.460 + omz
0.184 + 0.012
0.590 0.00
0.436
Graph2Edit-GEDD GED-Comp
0.367 :: 0.04
0.607 = 0.014
0.485 : 0.024
0,200
+ 0.008
0.606
0.02
0.453
Gruph2Edii-CS-Comp
0.388 +: 0.010
0.616 0.008
0.500 + 0.018
0.209 + 0.003
0.625 + 0.015
0.467
Ours
0.411 + 0011
0.636 0,030
0.524 + 0.017
0.218 0.011
0.653 0.039
0.488
mean pooling over query and support trees, compute the similarity dog by cosine similarity on tree representations for every query-support snippet pair, normalize, and form a convex combination over the entire available edit representations from support set. Finally, Ours is the proposed model which learns the similarity measure dg as described in Section 3 where we additionally consider an ensemble of multi-extent similarity scores, as shown in Eq. contains numerous C# specific implementations and is currently not applicable for Python, not include this baseline on PyFixer. All the baselines are under MIT license for public usage.
Note that we find Graph2Tree
do
We randomly split fixers for shw-shot experiments into meta- meta-train and meta-test sets. For C#Fixer, the meta- -itt--trinnn set contains 12 types of fixers and the meta-test set has the rest 4 fixers. The meta-train set of PyFixer contains 8 fixers and the meta-test set takes the rest 4. To perform a comprehensive empirical evaluation, we repeat the random split procedure and obtain 5 different splits denoted from Split #1 to Split #5. For meta-training to simulate the low-resource scenario and mitigate the imbalance on the number of samples per fixer, we take 10 samples for each fixer and follow a standard episode training strategy. Every time we construct the support set to train on one query sample, we randomly select the exemplars who are sharing the same fixing type with the query sample in the sampled data. For meta-testing, we iterate through all samples and each of them as a query snippet at time, then randomly sample from the rest of its fixer peer to build its support set. All numerical results are run with 5 different random seeds. Mean and standard deviation are reported. Note that the variance not only stems from the training per se, but also is affected by the random sampling of support set in testing. Since we fix the random seed every time, all methods are guaranteed to encounter identical query-support combinations in the evaluation under the same seed. validation split outside the five evaluative splits.
Protocol
All configurable hyperparameters are tuned We defer the model architecture and configurable hyperparameters to supplementary material. The accuracy per pair is a binary value obtained by whether the model edits C__ meet exact ground-truth Cz. Following Yao et al. (202T), we report both Macro and Micro accuracy since there is a significant fixer imbalance in C#Fixer. K is set and À set to the ensemble over (1.0, 2.0, 4.0, 10.0) by default if no specification is noted. We instantiate (e (in Eq. (I) as neural networks and de (in Eq. 4 as cosine similarity. Experiments are run on single NVIDIA RTX 3090 graphical card. We complement more technical details in Appendix B
4.2 RESULT AND ANALYSIS
Main numerical results of code editing with 5 support exemplars on C#Fixer and PyFixer are presented in Table and Table 2 Comparing to baseline methods, we nearly achieve the best results all splits
7

---

Under review as a conference paper at ICLR 2022
Table 2: Experimental results of code editing with support exemplars on PyFixer dataset. Macro and micro accuracy are identical on this dataset. The middle line in the table divides the method into non-composition and composition method.
Macro/Micro Accuracy
Split #1
Split #2
Split #3
Split #4
Split #5
Avg.
Graph2Hdit-RS
0.408 + 0017
0.381
0.815
0.498 + 0.33
0.293 0.007
0.2006 0.00
0.357
Gruph2Edil-GED GED-NNN
0.366 1: 0019
0,400
0.015
0.535 +: 0011
0.337
0.005
0.229 0.00
0.373
TraphHEditCSS-N
0.446 0017
0.425
0.016
0.541
+: 0.06
0.346
0.012
0.239
0.000
0.399
Graph2Edit-Ae
0.297 1: 0.122
0.347
= 0.008
0.488 + 0.010
0.276 + 0.008
0.194
0.000
0.320
Graph2Edii-GED-Coom
0.337 + 0.014
0,368
0.007
0.509 + 0.122
0.297 + 0.013
0.219
0.012
0,346
Graph2Hddit-S--Comp
0.441 + 0.030
0.409
0:011
0.543
+ 0.011
0.347
+ 0.010
0.276
0.004
0,403
Ours
0.510 1: 0.036
0.429
0.021
0.544
0.022
0.369 0.021
0.294
0.018
0.429
0.55
0.45
0:50
Macro Accuracy
0.40
Macro Accuracy
2:45
method
0:35
Graph2Edn-RS Grapritdnn.
0:40
0:30
0:35
0.25
0:30
support exemplars, where the integer K E [3, along the x-axis. Left is on C#Fixer dataset Split #1, and right is on PyFixer dataset Split #1. An estimated regression model is plotted for every group of data by the seaborn package, showing as the straight line with deviations. Our approach consistently benefit from adding more support exemplars and is capable to find better composition, while the other two baselines do not enjoy that.
Figure 2: Performance of code editing with various K support
of two datasets, across macro and micro accuracy. We outperform the second-best baseline method by bringing an absolute improvement range from 8.0% (Macro accuracy C#Fixer) to 10.9% (PyFixer). With the same base architecture (exclude Graph2Tree), comparing to GED and CS, our improvements show the significance of multi-extent composition learning of edit representations from support set. Quantitative results demonstrate the effectiveness of treating nodes in abstract syntax tree differently and ensemble learning by the meta-learning strategy. We present extra experiments in this subsection as well as in Appendix to further investigate our approach. We attach failure case analysis and a visualization diagram in Appendix and respectively.
More Support Exemplars Figure shows the performance trends when enlarging the size of support set, i.e., offering more resources for code editing. An estimated regression with uncertainty is plotted from K 3 to K 7. It is relatively straightforward that the performance on random selection is not affected by the scale of support set. The average edit representation performs differently on two datasets. Adding more exemplars into support set can help to improve the robustness of the weighted mean representation, or lead it to deviate the representational manifold if high variances exist among exemplars. Our approach consistently benefits from more support exemplars and gradually enlarging the margin towards the two baselines. This demonstrates our model is capable to find a better composition when there are more options, therefore preserving the possibility of capability enhancement when more resources are available.
Investigation of Multi- Ensemble Figure 3 shows the functional analyses on multi-exten measurement and ensemble, where we investigate seven possible options on extent parameter X, four in a single extent, three in a double extent ensemble, and compare to the full model with four levels of extents plotting in red. A single extent in the first four rows underperforms the ensemble when using two four extents. This confirms our methodological hypothesis that a single extent may not be enough to express the similarity between the query-support snippet pair due to the high variances and that ensembling multiple extents helps the model robustness. The full model with four extents also performs better than the double extent ensemble shown by the median line of the box plot. However, adding more extents implies increasing computational costs and hence trade-off.
8

---

Under review as a conference paper at ICLR 2022
10
10
4.0
4.0
1.0 10.0
1,0 10.0
4.0 10.0
4.0 10.0
Eisemble
Eisemble
0.32
0.34
0.20
0.38
0.40
0.42
2.44
0.00
0.40
0.30
0.35
0:40
0.45
0.30
0.55
Macro Accuracy
Macro Accuracy
Figure 3: Experiments on multi-extent parameter À and its ensemble. The y-axis represents the instantiated value of X, where the plus symbol denotes the ensemble of the two extents, and 'Ensemble' includes all these four. Left: C#Fixer dataset Split #1; Right: PyFixer dataset Split #1.
RELATED WORK
Machine Learning for Code Editing Code editing using machine learning has been developed for a long time and is receiving increasing attention in recent years (Menon et al.) 2013 Zhang et al.) 2019 Tian et al.) 2020). Depending on how a code snippet is represented, the research can be roughly summarized into two categories, i.e., performing code editing over code token sequences or abstract syntax tree. In the former category, Yin al. (2019); Chen al. (2019): Yasunaga Liang (2020) directly generate the expected edited code token sequence, while Shin et al. (2018); Vu & Haffari (2018 Dong et al. )2019); Zhao al. (2019) predict the editing operational sequence, which has been demonstrated to bear more sample efficieccy. On the other hand, works in the second category
(Yin et al. [2199] Chakraborty al., 2020 Tarlow et al., 2020) Dinella et al.) 2020; [Boody al.) 2020 Yao et al. 2021) attempt to edit the trees of programs similarly by either directly generating the edited trees or predicting the tree edit operations. The problem addressed this paper is conceptually
related to "programming by example' in software engineering that finds the sharing edit operations
among different exemplars and applies it to query snippet (Menon et al.] 2013) Osera & Zdancewic
F11ddwsidds et al. 2020 [Mng al. 2013). Recently, Yin et al. (2019); Yao et al. (2021)
tackle this problem using deep learning techniques. Nonetheless, both of the two works have focused
on code editing from one shot of exemplar, which may not be sufficient for learning generalizable editing, as we elaborated in Section [ This inspires us to investigate few-shot code editing.
Few-shot Learning Few-shot learning has been developed in many low-resource scenes (Vinyals et al.) 2016 Snell al.] 2017 Finn et al., 2017), and with advance in meta seq2seq learning considering the concept of generalized composition (Gu et al., 2018 Lake 2019 Nye et al. (020).
Our work conceptually relates to the matching principle few-shot learning (Vinyals al. [0166] Snell et al. 2017) but with a different compositional task and specific design for robust multi- extent abstract syntax tree matching. To our best knowledge, we are the first to consider few- shot compositional generalization in code editing, which we believe can inspire development in cewraammiing-ellttte applications like code transfer, refactoring, and migration.
CONCLUSION
In this work, we considered novel problem software engineering: code editing with few exemplars. Based on previous frameworks on code editing using abstract syntax tree present the computer code snippets, we proposed an adaptive multi-extent composition method to perform varying intermediate representations between a collective tree and individual nodes. We leveraged an ensemble approach to gather the query-support matching from multiple extents and delivered a robust composition over support editing representations for the query snippet decoding. Evaluations datasets demonstrated the effectiveness of our method over baselines by a large margin. To develop this framework into a product, consider letting the user collect few exemplars showing their own purpose. The tool requests an interaction but also offers full flexibility to the user, and is more adapted personal development environment.
two code editing

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
We discuss the potential ethical considerations in this section. We do not discover a noteworthy negative societal impact originate from work. One minor concern is the research in machine learning for source code may encourage and facilitate the code data collection from open-source projects. The data may unintentionally involve user privacy like the author's name, organization, or file date, and the collection process may violate the usage license.
REFERENCES
Zeina Abu-A Aiu-Aisheh, Romain Raveaux, Jean-Yves Yves Ramel, and Patrick Martineau. An exact graph edit distance algorithm for solving pattern recognition problems. In Proceedings of the International Conference on Pattern Recognition Applications and Methods, 2015.
Shaked Brody, Uri Alon, and Eran Yahav. A structural model for contextual code changes. Proceed- ings of the ACM on Programming Languages, 2020.
Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray. Codit: Code editing with tree-based neural models. IEEE Transactions on Software Engineering, 2020.
Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: pepuunnce-tosequeene learning for end-to-end program repair. IEEE Transactions Software Engineering, 2019.
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Reprexentations, 2020.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. Editnts: An neural geggramme--nnteprete model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019.
Kasra Ferdowsifard, Allen Ordookhanians, Hila Peleg, Sorin Lerner, and Nadia Polikarpova. Small- step live programming by example. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 614-626, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. Meta-lcarning for low- resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
Brenden M Lake. Compositional generalization through meta sequence-to-sequencc learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2019.
Yujia Li, Daniel Tarlow, Mare Brockschmidr, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint
2015.
Na Meng, Miryung Kim, and Kathryn S McKinley. Lase: locating and applying systematic edits by learning from examples. In 2013 35th International Conference on Software Engineering (ICSE), pp. 502-511. IEEE, 2013.
Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. A machine learning framework for programming by example. In International Conference on Machine Learning, pp. 187-195. PMLR, 2013.
Maxwell I Nye, Armando Solar-Lezama, Joshua B Tenenbaum, and Brenden M Lake. Learning compositional rules via neural program synthesis. arXiv preprint arXiv:2003.05562 2020.
Peter-Michadl Osera and Steve Zdancewic. caand-example-dieee program synthesis. ACM SIGPLAN Notices, 50(6):619-630, 2015.
10

---

Under review as a conference paper at ICLR 2022
Alberto Sanfeliu and King-Sun Fu. A distance measure between attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, 1983.
Richard Shin, Illia Polosukhin, and Dawn Song. Towards specification-direeeeed program repair, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In Advances in Neural Information Processing Systems, 2017.
Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian. Learning fix build emrors with graph2diff neural networks. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, 2020.
Haoye Tian, Kui Liu, Abdoul Kader Kahoré, Anil Koyuneu, Li Li, Jacques Klein, and Tegawendé F Bissyandé. Evaluating representation learning of code changes for predicting patch correctness in program repair. 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), 981 989- IEEE, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. arXiv preprint arXiv: (66660000000 2016.
Thuy Vu and Gholamreza Haffari. Automatic post-editing of machine translation: neural programmer--mierpreeer approach. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, 2018.
Ziyu Yao, Frank F. Xu, Pengcheng Yin, Huan Sun, and Graham Neubig. Learning structural edits via incremental tree transformations. In International Conference on Learning Representations, 2021.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference Machine Learning, 2020.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Learning to represent edits. In International Conference on Learning Representations, 2019.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural source code representation based on abstract syntax tree. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 783-794. IEEE, 2019.
Rui Zhao, David Bieber, Kevin Swersky, and Daniel Tarlow. Neural networks for modeling source code edits, arXiv preprint arXiv: 190.00288. 2019.

---

Under review as a conference paper at ICLR 2022
A EDITING EXAMPLES FOR FIXERS IN PyFixer
We list the 12 kinds of fixer names in PyFixer in Table 3
Table 3: Fixer examples in PyFixer
Fixer 05: lib2to3. fixes. fix filter
C_ chunk filter(lambda x.feature feature, ichunk)
C+ chunk = [x for x in ichunk x.feature = feature]
www.ffiiihhshhh_ee C_ key = matchdicttmmtcciit((( ""een"" if
Fixer 08: lib2to3. fixes. fix has_kas
has "ke""""""tthas_kk"""""""""" else ""
C+ key request.macchhicc (eaecchddiitike"e[]]] if "key" in request matchditt else
Fixer 09: lib2to3. See2tt...ixe..fff fixes.ffx.x.... idioms
C_ return type(self) = type(other) and self. dict_cc________l_.___.. delf._dict__ = other. dict
C+ return isinstance(self, type(other)) and self. selff_diet______________________ other. dict
Fixer 15: lib2to3. fixes. www.ffiffx_xmmmie.e fic_
C_ map(int, for in awargs.valuest. if != reaans[
C+ list(map(int, [a for a in kwargs. kwargs.values() if != k releeeeeleell
Fixer 18: lib2to3. fixes. fix next C- second_gff fiff__cuukk___ = second -ff.nnxxtfe
C+ second_gft Bafnddd_ff_uukkkkkg chunk next(second_gff)
Fixer 23; lib2to3. fixes.ixc.....c fiissiiix_eeee fiw_riww________ i_jinutu
C_ raw www_input("Me Start the server on 0:1 and press enter." unter.".forr foomma(stt(parsedd.ip) str(parsed.p)))
C+ input(" (Start the server on 0:1 and press enter." "nter."..ormat(stii format(stt(panseddi))) str(parsed.p)))
Fixer 34: lib2to3. fixes.fix_ztt zip C_ coneext.updaeeeiiiiiiinnnnsssse nnames, Swargs.gtt('nlling ()))
C+ coneexxuuntaaeelssstpxiinnneeenp _amee, kwargs.get( inlines"", (I))
"raagg.get("inlines,,,
Fixer 34: lib2to3. fixes.fix absolute absooaute_mpon import
C_ from elks import Elks
C+ from .ekks import Elks
Fixer 41: 4bbtto3.ixes.fi fittt.r
standard ftauuressttadardddd _tibrar
C_ import urlparse
C+ import urllib.parse
Fixer 42: lib2to3. fixes.fix future standard library_ urllib
C_ from urllib2 import HTTРEιror, urlopen, Request
C+ from urllib.request import urlopen, Request
Fixer 42: lib2to3. fixes. fix fix fi__print__aee prrinttwithhh__tpttttt with_impon llse__dt__diii_a dir", aopploccttoos.site ltttioonssite_dtt___dd_e ddtaa_drr) C+ print( (dttti"sttddtaaa___a dir', app.llocaioon.siii site sitt_ddta__drr)
C_
Fixer 49: lib2to3. fixes.fix unicode keep_u
C_ username
"."")
C+ username (
B EXPERIMENTAL DETAILS
We collect the PyFixer dataset by separately applying 51 fixers in Python-Future") over 5,959 projects written Python to satisfy the compatibility of Python 3. We obtain the code pairs by inspecting the changed lines and get the source code before and after the refactoring, We select 12 types of fixers and extract 200 samples for each fixer from the main data build the dataset for few-shot learning, while the rest preserved for model pre-training.
12

---

Under review as a conference paper at ICLR 2022
For training, set the batch size to 16, epochs to 15, learning rate to le-4, with gradient accumulation for every 2 optimized steps. The extent parameter À for main experiments are set as an ensemble for 10, 5, 2, and 1. The architectural parameters are set the same as Graph2Edit (Yao et al. 2021). The predictor R is implemented with neural networks in two layers with Leaky ReI.U. The learnable function (o((;)) -) is implemented with one linear layer followed by cosine similarity between the two input terms. The function o(- .) is cosine similarity function on edit representation. Other detailed implementation or configurations are in our source code since it is not possible to exhasutively include every detail with plain text.
The following equation describes the calculation of macro and micro accuracy. Consider set of class C, T. is the number of correctly edited samples in class c, Ne is the total number of samples in class c. These two metrics can be expressed by
Te N:
Mícro accuracy Ecec Te Ecec Ne
Macro accuracy
C
eEC
Note that, in our experiments, an edited code considered "correct" only when it is exactly the same as the ground truth in both syntax and semantics.
The PyFixer are working under Python 3.8, having the abstract syntax tree grammar from hwt.:/sssspyyoooooo/.////rrrrrr low.
We list the choice of splits for the two datasets be-
Table 4: Fixer Splits on C#Fixer
CA2007, IDE0004, RCS1015. RCS1021, RCS1032, RCS1058. RCS1077, RCS1097, RCSI118, RCSI RCS1119, RCS1206
Split #1
RCSI146, RCS1207, RCS1202, RCS1089
Split #2
IDE0004, RCS1015, RCS1032. RCS1058, RCS1077, CA2007, RCS1089, RCS1146, RCS1202, RCS1206, RCS1207, RCS1097
RCS1118, RCS1123, RCS1021, 197
RCS1015, RCS1021, RCS1032, RCS1058, RCS1077, RCS 1097. CA2007, IDE0004, RCS1118, RCS1146, RCS1202, RCS1207
Split #3
RCS1123, RCS1197, RCS1206, RCS1089
RCS1123, RCS1021. RCS1032, RCS1058, RCS1206. RCS1097. CA2007, IDE0004. RCS1118, RCS1146, RCS1202, RCS1207
Split #4
RCS1015, RCS1197, RCS1077, RCS1089
RCS1077, RCS1021, RCS1032, RCS1058, RCS1206. RCS RC91 CA2007, IDE0004, RCS1197, RCS1146, RCS1089, RCS1207
Split #5
RCS1118, RCS1015, RCSI RCS1202
Table 5: Fixer Split on PyFixer
Split
08, 15, 18, 23, 34, 42, 47,
05, 41, 09, 35
Split #2
08, 09, 35, 23, 34, 42, 47, 05
41, 49, 15, 18
Split #3
41, 09, 15, 23, 49, 42, 47, 05
08, 18, 34, 35
Split #4
41, (9, 34, 23, 49, 42, 08, 05
47, 15, 18, 35
Split #5
15, 09, 34, 23, 49, 35. 08, 47
05, 41, 18, 42
We use the fixer's name in Table Table the fixer's index in Fablee the order in the Python-Future package.
C ADDITIONAL EXPERIMENT
We extend our experiments by considering the 'Hit-l-in-S' and 'Hit-5-in-5" cases when editing with one exemplar from the given few exemplars. In the "Hit-l-in-5' case, we separately decode the query code snippet using the K exemplars and obtain K results. We count the result of the query sample as correct if any of these K decoding results match with the ground truth. Similarly, in the 'Worst' case, we count it as correct when all of these K results are correct. Note that both the 'Hit-l-in-5" and 'Hit-5-in-5' cases are not real in practical usage since from the editing system. Giving possible answers as many as it can is not user-oriented and not scalable. The 'Hit-l-in-5' and 'Hit-5-in-5' cases can be understood as the upper and the lower bound of results if we randomly select one exemplar from the support set but they are almost not achievable. We include an experiment using every exemplar in Figure 4
only expect one output
13

---

Under review as a conference paper at ICLR 2022
0.55
0.50
0.6
Matro accuracy
0.43
Micro accursey
0.5
0.40
0.4
0.35
0.30
0.3
0.25
0.20
comp
0.2
non-comp.
comp.
comp.
Spit
Split
Split
Split
Splt
Split
Solit
Figure 4: Macro accuracy C#Fixer dataset using Graph2Edit. 'comp.' is abbreviated for composi- tion and "non-comp." is abbreviated for non-composition. Blue dots represented the average accuracy over split obtained by using one exemplar in the support set, indexing from I to 5, respectively. Or- ange dots is the accuracy achieved by our proposed method. Our compositional method consistently outperforms the results using only one exemplar that suffers from variance in code editing.
Table 6: Graph2Fddi results of Hit-l-in-2 and Hit-5-in-5 with 5 support exemplars on C#Fixer dataset.
Macro Accuracy
Split #1
Split #2
Split 43
Split #4
Split #5
Avg.
Ours
0,416 0015
0.514 a 0.021
0.522
0.015
0.352
0018
0,539 +: 0.023
0.468
Hit-1-in-5
0.462 0006
0.601
+ 0.00
0.605
= 0.006
0.413
0.660 + 0.033
0.548
Hit-5-in-5
0.072 0.019
0.154 +: 0.010
0.126 = 0.020
0.045 +: 0.004
0.111 + 0.013
0.102
Miero Accuracy
Split #1
Split #2
Split #3
Split #4
Split #5
Avg.
Ours
0.411 0011
0.636 +: 0.030
0.524 0.017
0.218
0.653 + 0.039
0.488
Hit-l-in-5
0.443 0.088
0.721 + 0.004
0.616 = 0.006
0,312 4 0.21
0.780 +: 0.007
0.574
Hit-5-in-5
0.074 0.017
0.222 # 0.012
0.126 0.022
0.033 0.003
0,260 + 0.017
0.143
Hit-1-in-5 requests the ground-truth
Table 7: Graph2Edit results of Hit- 1iit- and Hit-5-in-5 with 5 support exemplars on PyFixer dataset.
Macro and miero accuracy identical on this dataset.
Macro/Micro Accuracy
Split #1
Split #2
Split #3
Split #4
Split #5
Avg.
Ours
0.510 + 0036
0.429 i 0021
0.544 200
0.369 = 0.021
0.294 = 0.088
0.429
Hit-I-in-S
0.780 + 0.111
0,580
+: 0.007
0.623 +: 0.011
0,742
= 0.004
0.455
# 0.011
0.636
Hit-5-in-5
0.059 +: 0.099
0.107 0012
0.165 0.009
0.045 0.007
0.013 0.004
0.078
Hit-1-in-5 requests the ground-truth.
Table 8: A failure case from our method on C#Fixer dataset
Support set
C_
ArrsestAen VARO!= null & this.VARO. VARO.AnOO, LITERAL); Utils. ArgeentArgumen((hhiss VARO1.Any = true, LITERAL);
C
C-
bool VAR0= this. VARI!= null && this. VARLAny0:
C
bool VARO= this. VAR1?.Any( = true;
C_
var VARO= Propertto))))))). Where/ VAR2!= null && VAR2.Name 1= null
&& String. EqualsE VAR2. LITERAL, String Comrrinnoomaaiion...o OrdinalgnnorCCass)))) var VARI= VAR Propertics)) Whero/VAR2=>> VAme 1= null
C+
&& String. Fqualss(ARR2.aame, L.TERAI,, String Orwnrriinnnnniiilggnnnnnneeee
C_
var VAR0= VAR AsBummraeee)) DrnnsssOrDDeeaaaaAAaaaaaa V
VnR2.In null
VnR2.Inline... Tag == Syntux Sww.ux www.iiiiiinnnnnnnnnnnaann && VAR2. Inline.
C+
var VARO= Lel.Lannnnecaeeeeeiisnontteeeeee VAR22>> VAR2.Inline Syntaa.InlineeaagSeiiiie
C_
var VARO= VARI htwpeannrrrrreooFFssstooeeeeeee
VAR2.Inline I= null
&&& VAR2. Maliine..ierrllCoooii = LITERAI.);
C+
var VAR0= VARI A.Rummeraeeer A DiniiiOeeeeaeliiiiiiiii
VAR2=>> VAR
VAR2 LITERAL):
Query sample
C
var VAR0= (VAR1 null) && NodeTToe == ExpressiunTypeeNut)
C+
var VARO= Node = ExpressioTType.Noot
Prediction
var VAR0= AVyR = ExpressionType..Nrr
14

---

Under review as a conference paper at ICLR 2022
Previom:
VErVAID-YAAALLTTEAAAL
Prestoms:
Query
Support #1
DrarySaprssien
LITERAL
"nul
Previous: vue VerVAAttollllss Cound /ooles.Cous"" CoouAtVVRRAL.. Counds
Support
Edited: varVARD-Roloo Count (jtles Count" VARL
(onttericaan
Eptractttkerperrre
"VARL"
LITERAL
Telles
Count
Sprtas Tybertperrar
Roles
Figure 5: Visualization of normalized node activation with x= 1.0 on query snippet and two support exemplars. We plot the top six nodes with the highest activation for each tree and show them in decreasing color saturation. Abstract syntax trees are plotted after pruning on insignificant parts.
D FAILURE CASE
We analyze the failure case in this section. A failure case on editing the &&& logical operation is presented in Table X In this case, most of the exemplars showcase the editorial pattern with complicated context, thus it is hard for our algorithm to extract the sharing editing. Meanwhile, two of these exemplars concurrently edit with the attribute 'Any()' and consequently mislead the query code editing.
E VISUAL DIAGRAM FOR NODE ACTIVATION ON QUERY-SUPPORT MATCHING
We visualize the node activation in Figure The value of an activation is normalized within the query or support snippet to show the relative magnitude the value. Matching on individual nodes facilitates query snippet find the valuable reference in Support #1 under BinaryExpression regardless of the disturbing content under "CondiiionlEExpeesiion. Also, there is faint activation on the multiple operators in Support #2 since the objects this operation is under 'Member AcceseeAceesssppesssin.
15


Reviews:
[{"review_id": "quq8sE8nDPk", "reviewer": "Reviewer_L4AR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important task in software engineering. Modifying source code towards a desired editing style could be a common programming activity. The paper proposed a new DL-based model that can outperform two existing models.\n\nThe results are somewhat unsurprising. It is obvious that providing more examples can improve the generalization ability of a machine learning model. Code editing styles can be better learned from more editing exemplars, so the results are expected.\n\nThe related work of this paper is described in a way that is too brief. There has been much work on code similarity and code changes. For example:\nNa Meng, et al., LASE: locating and applying systematic edits by learning from examples. In Proc. ICSE  2013, 502–511.\n\nTian, Haoye, et al. \"Evaluating representation learning of code changes for predicting patch correctness in program repair.\" 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2020.\n\nJ. Zhang, et al., A Novel Neural Source Code Representation based on Abstract Syntax Tree, In Proc. ICSE 2019, May 2019. \n\nNote that the above LASE (Meng et al., 2013) work is a traditional program analysis work that produces systematic edits by learning from multiple examples. The paper can also compare with traditional approaches such as LASE. Currently, it only compared with two deep learning based baseline models.\n\nThe most severe issue of this paper is on the evaluation. The evaluation metrics used in the paper, Macro and Micro accuracy,  are not explained. Therefore, it is difficult to interpret the experimental results. \n\nA human evaluation is needed to evaluate the effectiveness of the tool in practice. It would also be better to discuss some unsuccessful examples of code editing. Also, it is not clear how the syntax correctness is assured by the proposed approach.\n\nTo capture the multi-extent matching, we design a λ -softmax function by scaling the importance of nodes in an abstract syntax tree. The proposed λ-softmax is derived heuristically . The variable λ is not trainable. So it is difficult to know how it can be learned to reflect the extents of node/tree representations. The presentation can be enhanced if the authors provide more theorical analysis of the proposed method with respect to AST representations. \n\nMinor: In page 7:\nbringing a absolute improvement = > bringing an absolute improvement \n\nOur (GDE) => Our (GED)\n", "summary_of_the_paper": "This paper presents a few-shot learning approach to editing source code with a few exemplars. Unlike previous work that learns code editing with only one exemplar, the proposed approach learns the edit representations from a few exemplars. The proposed approach combines edit representations extracted from editing exemplars and compositionally generalizes them to the query code snippet editing via multi-extent similarities ensemble. The proposed approach models the extents of node-tree representations using a λ-softmax with an adaptive composition of multiple extents. Experimental results on two code editing datasets demonstrate some improvement over baseline models.", "summary_of_the_review": "The results are somewhat expected. The evaluation part is unclear. Comparison with more existing work (including traditional approaches) should be performed.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "abWP0DgSdhy", "reviewer": "Reviewer_ZRFf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper aims at generalizing from multiple example edits rather than applying a single given edit to an input program. This is achieved by aggregating the edit representations by matching the input program with the support set of programs from the example edits. The scores for the edit representations are computed by a series of matching steps over the tree representations of the programs. A hyper-parameter $\\lambda$ is used to control the granularity at which the tree fragments are matched. The representations of the input program (query) and a support program are computed by node-level cross-attention. This operation is quadratic in the size of ASTs. I was wondering if the tree structure can be used to reduce this cost.\n\nThe paper conducts experiments on datasets from two languages: C# and Python with Graph2Tree and Graph2Edit as the underlying neural editors. The results show that the multi-extent matching does better than all other choices.\n\nThe paper aggregates edit representations from multiple examples using a number of complex steps. However, the examples come from a single type of fixer and share the same intent. I am not convinced that the paper demonstrates the need to combine multiple edit representations. In particular, it might suffice to use a single, most-suitable examplar. I would therefore like to see single-example baselines that select the nearest neighbor from the support set i) using a discrete measure like tree edit distance over ASTs with terminals and without terminals, and ii) in the vector space. Note that the single example baselines consider in the experiments (Graph2Tree-RS and Graph2Edit-RS) use random selection and all other baselines consider all the examples together. The number of examples is also small in this case. An iterative baseline that takes each of the examples is also required to demonstrate that the combination of multiple examples is indeed better than any of the examples separately. A qualitative discussion of combining multiple edit representations would also be helpful.\n", "summary_of_the_paper": "Neural code editors are a class of neural networks which take vector representations of an edit and an input to produce the output by applying the edit to the input program. The existing approaches take a pair of source code snapshots and embed them into an edit representation. This paper argues that a single pair may not be enough to unambiguously perform the edit and multiple examples can help. The authors present an approach to aggregate edit representations from multiple such pairs. The approach is evaluated on code editing datasets in C# and Python. The aggregation approach gives better results than simpler baselines and ablations of the aggregation method.", "summary_of_the_review": "The paper presents a technique to combine multiple edit representations for neural code editors. The paper is difficult to understand at times. It makes some technical contribution and provides experimental evidence. However, there is a conceptual gap about the need for combining representations from multiple examples. Though at a high-level it is conceivable that multiple examples could help, the paper does not provide evidence that a systematically chosen single example or an iterative approach which goes over all examples and takes the best output would not be sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AE4dt_bOjP", "reviewer": "Reviewer_gFho", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for the problem was well presented, and made sense. A single example edit can lead a model astray, and figuring out which of multiple exemplars to imitate seems like a very useful facility in your approach.\n\nYour query-support matching functions (Section 3.1) have similarities to attention, but some differences as well. For instance, attention is all about alignment, and your similarity functions are essentially trying to align the query graph nodes to the support snippet graph nodes. It would be instructive in the paper to explain the differences and why this is not attention. In general, the formulation of Section 3.1 is not very well motivated and could use some explanation and intuition (e.g., the difference between $m^{q,k_k}$ and $m^{s_k, q}$, and what each captures that the other cannot).\n\nAlso, it would be good to explain how you exactly implement the two phi functions in Section 3.1. I had to guess that they are some feedforward network from what you say, but it's only a guess.\n\nI found the term `extent` highly confusing, especially since it's associated with the $\\lambda$ parameter, which doesn't have a physical interpretation. I wonder if `resolution` might be a better term, or something else? `Extent` has to do with coverage, or area. If you can intuitively demonstrate how different values of $\\lambda$ represent some notion of extent, I'd love to see it.\n\nSomething I didn't quite get from your presentation is what makes a good set of exemplars. You argue that more are better for your approach, which makes intuitive sense, but presumably, it's just as easy to have a bunch of exemplars that happen to modify a declaration as in Support #1 of Case a in your example, as it is to have one of those. Your model would be just as likely to be convinced that the rewrite only applies to declaration statements. Perhaps you were lucky in your random draws from exemplars, in that you ended up with a diverse set that improved the prediction. But why would that be the case in general? In fact, what is the use case for this multi-exemplar edit imitation? Are you expecting a developer to write a few example edits and let the model apply them to diverse \"queries\"? I couldn't quite tell what kind of use case you're anticipating. Will there be noise in the exemplars (i.e., are they always correct?) In your datasets, the exemplars are generated by \"oracle\" fixers, but I'm guessing in practice you wouldn't have those, or else you wouldn't need your tool in the first place.\n\nNevertheless, I found the contribution interesting and different from prior related work.\n\n# Questions\n\nQ1: How many exemplars are enough? Is more always better? Is there some measurable metric of fitness for a set of exemplars, e.g., based on diversity or some other similar metric?\n\nQ2: What's the use-case for this multi-exemplar task? Would you expect an editor to provide this functionality? Do you anticipate a different kind of usage mode for this sort of multi-extent input?\n\nQ3: What's the relationship between your query-support matching formulation and standard notions of attention?\n\n\n# Smaller Issues\n\n1. In equation 1, activations have $g, s_k$ as a superscript. In equation 2, they have $gs_k$. Please be consistent.\n\n2. The descriptions of the \"Ours\" baselines are confusing. Given the complex formulation of your model, it would help if you pointed out which function of which equation you're replacing with the Graph Edit Distance and cosine similarity, respectively, or just gave a precise alternate formulation. \n\n\n\n", "summary_of_the_paper": "This paper tackles the task of applying an edit to code that is similar to some prior edits (think of a linter edit that replaces double quotes with single quotes in Python). Prior work has addressed such a task, by embedding an edit exemplar (before and after snapshots of a file for such an edit), and the before snapshot for a new example, seeking to predict the after snapshot that adapts the edit exemplar to the new before snapshot. However, this work recognizes that generalizing from a single exemplar can push baseline models to overfit the patern modifications of the exemplar.\n\nInstead, this work (a) enables a model to see multiple exemplars, (b) and uses a similarity ranking estimator to weigh different exemplars in a way specific to the \"query\" (the new before snapshot), so as to influence the prediction accordingly. This similarity formulation is parametrized by how much individual good alignment matches (between exemplars and the new \"before\" snapshot) should dominate the overall prediction, versus a broadly good alignment match across multiple parts of the various exemplars. An ensemble approach combines multiple values for this parameter.\n\nOn a number of datasets generated by applying stock C# and Python fixers to GitHub code, this new approach outperforms baselines.\n\n", "summary_of_the_review": "Task targeted is not well motivated (not clear how you apply it if you don't have the fixers, or why you would use it if you have the fixers). But the approach seems novel and interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7bM8CF_Cp12", "reviewer": "Reviewer_4A6k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Source code editing is a fundamental software engineering task, which has recently attracted a lot of attention in the ML community. \nIt is used in various SE applications: code refactoring, and program repair. \n\nThe paper introduces the problem of code editing from few examples and formulates the method to use multiple examples in training and inference. While the task of code editing from few examples is new, it is conceptually similar to programming by example (PBE). Authors might find it useful to mention program synthesis by examples in related work.\n\nThe paper is relatively difficult to read, and some of the terminology is not explained. Is there a precise definition of \"editorial style\" of source code? Program repair is arguably not a style change, and may lead to a different code semantics (see introduction, first paragraph). On the other hand, programming style is a well established concept which is different from editing style, which needs to be made more clear in text. \n\nIt is not clear from the text how to identify helpful support examples for a given dataset for learning stage. Can you describe a procedure/algorithm to select supporting examples?\n\nBaselines: in code editing, alignment plays a crucial role. Does the AE baseline performs average of aligned representations? If not, please consider adding an average with alignment.\n\nMinor:\n“composite them to guide editing” -> “combine them to guide editing” or “compose them to guide editing”\n\nConsider re-writing the abstract to highlight the main contributions and the key methods used.", "summary_of_the_paper": "The paper introduces a method for code editing from few examples. It allows to automatically generalize code edits from few support examples via adaptive multi-extent composition.\n\nThe approach is evaluated on two standard code editing datasets (C# fixer and Py fixer) against the Graph2Edit baselines. Specifically, Graph2Edit model variant applied to a randomly selected support example and an average Graph2Edit representation over all K support examples.\n\nThe method leads to a 8-10% improvement over baselines.", "summary_of_the_review": "The paper introduces a task of code editing from few support examples, and formulates the learning and inference methods to implement the composition method on top of the Graph2Edit method. While the evaluation is sound, and shows an improvement of accuracy as a function of number of support exemplars, one practical challenge in applying this technique would be to identify groups of support examples / identifying the edit intent. Which could lead to problems applying this method at scale or in the product. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes learning to make stylistic code edits (semantics remains similar) based on information from a few exemplars instead of one. The proposed method first parses the code into abstract syntax trees and then use the multi-extent similarity ensemble. This was compared to a Graph2Edit baseline on C# fixer and pyfixer, which are datasets generated by rule-based transcompilers. The proposed method got around 10% accuracy improvement due to a combination of the method and using more than one examplar. 

The reviewers find that any improvement due to more examplars to be expected and suggested that 1) one carefully chosen examplar is enough, and 2) that the need for multiple examplars means more practical difficulties in providing them in an application 3) the targets are all generated by rule-based methods and the benefits may not extend to a realistic case where the edits are not so clear and the reviewers wondered about the application value and the potential need for human evaluations.  The authors argued that it is unexpectedly difficult to expand the base method to multiple examplars and users should be able to provide examplars in an application. The authors further provided additional results that addressed some of the reviewer's concerns but the reviewers did not change their evaluation.

Rejection is recommended based on reviewer consensus.
Decision: Reject