Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method for code editing from few examples. It allows to automatically generalize code edits from few support examples via adaptive multi-extent composition.

The approach is evaluated on two standard code editing datasets (C# fixer and Py fixer) against the Graph2Edit baselines. Specifically, Graph2Edit model variant applied to a randomly selected support example and an average Graph2Edit representation over all K support examples.

The method leads to a 8-10% improvement over baselines.
main_review: Source code editing is a fundamental software engineering task, which has recently attracted a lot of attention in the ML community. 
It is used in various SE applications: code refactoring, and program repair. 

The paper introduces the problem of code editing from few examples and formulates the method to use multiple examples in training and inference. While the task of code editing from few examples is new, it is conceptually similar to programming by example (PBE). Authors might find it useful to mention program synthesis by examples in related work.

The paper is relatively difficult to read, and some of the terminology is not explained. Is there a precise definition of "editorial style" of source code? Program repair is arguably not a style change, and may lead to a different code semantics (see introduction, first paragraph). On the other hand, programming style is a well established concept which is different from editing style, which needs to be made more clear in text. 

It is not clear from the text how to identify helpful support examples for a given dataset for learning stage. Can you describe a procedure/algorithm to select supporting examples?

Baselines: in code editing, alignment plays a crucial role. Does the AE baseline performs average of aligned representations? If not, please consider adding an average with alignment.

Minor:
“composite them to guide editing” -> “combine them to guide editing” or “compose them to guide editing”

Consider re-writing the abstract to highlight the main contributions and the key methods used.
summary_of_the_review: The paper introduces a task of code editing from few support examples, and formulates the learning and inference methods to implement the composition method on top of the Graph2Edit method. While the evaluation is sound, and shows an improvement of accuracy as a function of number of support exemplars, one practical challenge in applying this technique would be to identify groups of support examples / identifying the edit intent. Which could lead to problems applying this method at scale or in the product.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Neural code editors are a class of neural networks which take vector representations of an edit and an input to produce the output by applying the edit to the input program. The existing approaches take a pair of source code snapshots and embed them into an edit representation. This paper argues that a single pair may not be enough to unambiguously perform the edit and multiple examples can help. The authors present an approach to aggregate edit representations from multiple such pairs. The approach is evaluated on code editing datasets in C# and Python. The aggregation approach gives better results than simpler baselines and ablations of the aggregation method.
main_review: The paper aims at generalizing from multiple example edits rather than applying a single given edit to an input program. This is achieved by aggregating the edit representations by matching the input program with the support set of programs from the example edits. The scores for the edit representations are computed by a series of matching steps over the tree representations of the programs. A hyper-parameter $\lambda$ is used to control the granularity at which the tree fragments are matched. The representations of the input program (query) and a support program are computed by node-level cross-attention. This operation is quadratic in the size of ASTs. I was wondering if the tree structure can be used to reduce this cost.

The paper conducts experiments on datasets from two languages: C# and Python with Graph2Tree and Graph2Edit as the underlying neural editors. The results show that the multi-extent matching does better than all other choices.

The paper aggregates edit representations from multiple examples using a number of complex steps. However, the examples come from a single type of fixer and share the same intent. I am not convinced that the paper demonstrates the need to combine multiple edit representations. In particular, it might suffice to use a single, most-suitable examplar. I would therefore like to see single-example baselines that select the nearest neighbor from the support set i) using a discrete measure like tree edit distance over ASTs with terminals and without terminals, and ii) in the vector space. Note that the single example baselines consider in the experiments (Graph2Tree-RS and Graph2Edit-RS) use random selection and all other baselines consider all the examples together. The number of examples is also small in this case. An iterative baseline that takes each of the examples is also required to demonstrate that the combination of multiple examples is indeed better than any of the examples separately. A qualitative discussion of combining multiple edit representations would also be helpful.

summary_of_the_review: The paper presents a technique to combine multiple edit representations for neural code editors. The paper is difficult to understand at times. It makes some technical contribution and provides experimental evidence. However, there is a conceptual gap about the need for combining representations from multiple examples. Though at a high-level it is conceivable that multiple examples could help, the paper does not provide evidence that a systematically chosen single example or an iterative approach which goes over all examples and takes the best output would not be sufficient.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the task of applying an edit to code that is similar to some prior edits (think of a linter edit that replaces double quotes with single quotes in Python). Prior work has addressed such a task, by embedding an edit exemplar (before and after snapshots of a file for such an edit), and the before snapshot for a new example, seeking to predict the after snapshot that adapts the edit exemplar to the new before snapshot. However, this work recognizes that generalizing from a single exemplar can push baseline models to overfit the patern modifications of the exemplar.

Instead, this work (a) enables a model to see multiple exemplars, (b) and uses a similarity ranking estimator to weigh different exemplars in a way specific to the "query" (the new before snapshot), so as to influence the prediction accordingly. This similarity formulation is parametrized by how much individual good alignment matches (between exemplars and the new "before" snapshot) should dominate the overall prediction, versus a broadly good alignment match across multiple parts of the various exemplars. An ensemble approach combines multiple values for this parameter.

On a number of datasets generated by applying stock C# and Python fixers to GitHub code, this new approach outperforms baselines.


main_review: The motivation for the problem was well presented, and made sense. A single example edit can lead a model astray, and figuring out which of multiple exemplars to imitate seems like a very useful facility in your approach.

Your query-support matching functions (Section 3.1) have similarities to attention, but some differences as well. For instance, attention is all about alignment, and your similarity functions are essentially trying to align the query graph nodes to the support snippet graph nodes. It would be instructive in the paper to explain the differences and why this is not attention. In general, the formulation of Section 3.1 is not very well motivated and could use some explanation and intuition (e.g., the difference between $m^{q,k_k}$ and $m^{s_k, q}$, and what each captures that the other cannot).

Also, it would be good to explain how you exactly implement the two phi functions in Section 3.1. I had to guess that they are some feedforward network from what you say, but it's only a guess.

I found the term `extent` highly confusing, especially since it's associated with the $\lambda$ parameter, which doesn't have a physical interpretation. I wonder if `resolution` might be a better term, or something else? `Extent` has to do with coverage, or area. If you can intuitively demonstrate how different values of $\lambda$ represent some notion of extent, I'd love to see it.

Something I didn't quite get from your presentation is what makes a good set of exemplars. You argue that more are better for your approach, which makes intuitive sense, but presumably, it's just as easy to have a bunch of exemplars that happen to modify a declaration as in Support #1 of Case a in your example, as it is to have one of those. Your model would be just as likely to be convinced that the rewrite only applies to declaration statements. Perhaps you were lucky in your random draws from exemplars, in that you ended up with a diverse set that improved the prediction. But why would that be the case in general? In fact, what is the use case for this multi-exemplar edit imitation? Are you expecting a developer to write a few example edits and let the model apply them to diverse "queries"? I couldn't quite tell what kind of use case you're anticipating. Will there be noise in the exemplars (i.e., are they always correct?) In your datasets, the exemplars are generated by "oracle" fixers, but I'm guessing in practice you wouldn't have those, or else you wouldn't need your tool in the first place.

Nevertheless, I found the contribution interesting and different from prior related work.

# Questions

Q1: How many exemplars are enough? Is more always better? Is there some measurable metric of fitness for a set of exemplars, e.g., based on diversity or some other similar metric?

Q2: What's the use-case for this multi-exemplar task? Would you expect an editor to provide this functionality? Do you anticipate a different kind of usage mode for this sort of multi-extent input?

Q3: What's the relationship between your query-support matching formulation and standard notions of attention?


# Smaller Issues

1. In equation 1, activations have $g, s_k$ as a superscript. In equation 2, they have $gs_k$. Please be consistent.

2. The descriptions of the "Ours" baselines are confusing. Given the complex formulation of your model, it would help if you pointed out which function of which equation you're replacing with the Graph Edit Distance and cosine similarity, respectively, or just gave a precise alternate formulation. 




summary_of_the_review: Task targeted is not well motivated (not clear how you apply it if you don't have the fixers, or why you would use it if you have the fixers). But the approach seems novel and interesting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a few-shot learning approach to editing source code with a few exemplars. Unlike previous work that learns code editing with only one exemplar, the proposed approach learns the edit representations from a few exemplars. The proposed approach combines edit representations extracted from editing exemplars and compositionally generalizes them to the query code snippet editing via multi-extent similarities ensemble. The proposed approach models the extents of node-tree representations using a λ-softmax with an adaptive composition of multiple extents. Experimental results on two code editing datasets demonstrate some improvement over baseline models.
main_review: This paper addresses an important task in software engineering. Modifying source code towards a desired editing style could be a common programming activity. The paper proposed a new DL-based model that can outperform two existing models.

The results are somewhat unsurprising. It is obvious that providing more examples can improve the generalization ability of a machine learning model. Code editing styles can be better learned from more editing exemplars, so the results are expected.

The related work of this paper is described in a way that is too brief. There has been much work on code similarity and code changes. For example:
Na Meng, et al., LASE: locating and applying systematic edits by learning from examples. In Proc. ICSE  2013, 502–511.

Tian, Haoye, et al. "Evaluating representation learning of code changes for predicting patch correctness in program repair." 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2020.

J. Zhang, et al., A Novel Neural Source Code Representation based on Abstract Syntax Tree, In Proc. ICSE 2019, May 2019. 

Note that the above LASE (Meng et al., 2013) work is a traditional program analysis work that produces systematic edits by learning from multiple examples. The paper can also compare with traditional approaches such as LASE. Currently, it only compared with two deep learning based baseline models.

The most severe issue of this paper is on the evaluation. The evaluation metrics used in the paper, Macro and Micro accuracy,  are not explained. Therefore, it is difficult to interpret the experimental results. 

A human evaluation is needed to evaluate the effectiveness of the tool in practice. It would also be better to discuss some unsuccessful examples of code editing. Also, it is not clear how the syntax correctness is assured by the proposed approach.

To capture the multi-extent matching, we design a λ -softmax function by scaling the importance of nodes in an abstract syntax tree. The proposed λ-softmax is derived heuristically . The variable λ is not trainable. So it is difficult to know how it can be learned to reflect the extents of node/tree representations. The presentation can be enhanced if the authors provide more theorical analysis of the proposed method with respect to AST representations. 

Minor: In page 7:
bringing a absolute improvement = > bringing an absolute improvement 

Our (GDE) => Our (GED)

summary_of_the_review: The results are somewhat expected. The evaluation part is unclear. Comparison with more existing work (including traditional approaches) should be performed.
review_title: null

Meta-review: This paper proposes learning to make stylistic code edits (semantics remains similar) based on information from a few exemplars instead of one. The proposed method first parses the code into abstract syntax trees and then use the multi-extent similarity ensemble. This was compared to a Graph2Edit baseline on C# fixer and pyfixer, which are datasets generated by rule-based transcompilers. The proposed method got around 10% accuracy improvement due to a combination of the method and using more than one examplar. 

The reviewers find that any improvement due to more examplars to be expected and suggested that 1) one carefully chosen examplar is enough, and 2) that the need for multiple examplars means more practical difficulties in providing them in an application 3) the targets are all generated by rule-based methods and the benefits may not extend to a realistic case where the edits are not so clear and the reviewers wondered about the application value and the potential need for human evaluations.  The authors argued that it is unexpectedly difficult to expand the base method to multiple examplars and users should be able to provide examplars in an application. The authors further provided additional results that addressed some of the reviewer's concerns but the reviewers did not change their evaluation.

Rejection is recommended based on reviewer consensus.
Decision: Reject