Learning Structural Edits via Incremental Tree Transformations

1 INTRODUCTION . Iteratively revising existing data for a certain purpose is ubiquitous . For example , researchers repetitively polish their manuscript until the writing becomes satisfactory ; computer programmers keep editing existing code snippets and fixing bugs until desired programs are produced . Can we properly model such iterative editing processes with neural generative models ? To answer this question , previous works have examined models for editing sequential data such as natural language sentences . Some example use cases include refining results from a first-pass text generation system ( Simard et al. , 2007 ; Xia et al. , 2017 ) , editing retrieved text into desired outputs ( Gu et al. , 2018 ; Guu et al. , 2018 ) , or revising a sequence of source code tokens ( Yin et al. , 2019 ; Chen et al. , 2019 ; Yasunaga & Liang , 2020 ) . These examples make a single editing pass by directly generating the edited sequence . In contrast , there are also works on modeling the incremental edits of sequential data , which predict sequential edit operations ( e.g . keeping , deleting or adding a token ) either in a single pass ( Shin et al. , 2018 ; Vu & Haffari , 2018 ; Malmi et al. , 2019 ; Dong et al. , 2019 ; Stahlberg & Kumar , 2020 ; Iso et al. , 2020 ) or iteratively ( Zhao et al. , 2019 ; Stern et al. , 2019 ; Gu et al. , 2019a ; b ) , or modify a sequence in a non-autoregressive way ( Lee et al. , 2018 ) . However , much interesting data in the world has strong underlying structure such as trees . For example , a syntactic parse can be naturally represented as a tree to indicate the compositional relations among constituents ( e.g . phrases , clauses ) in a sentence . A computer program inherently is also a tree defined by the programming language ’ s syntax . In the case that this underlying structure exists , many edits can be expressed much more naturally and concisely as transformations over the underlying trees than conversions of the tokens themselves . For example , removing a statement from a ∗Work done while interning at CMU . computer program can be easily accomplished by deleting the corresponding tree branch as opposed to deleting tokens one by one . Despite this fact , work on editing tree-structured data has been much more sparse . In addition , it has focused almost entirely on single-pass modification of structured outputs as exemplified by Yin et al . ( 2019 ) ; Chakraborty et al . ( 2020 ) for computer program editing . In this work , we are interested in a generic model for incremental editing of structured data ( “ structural edits ” ) . Particularly , we focus on tree-structured data , taking abstract syntax trees of computer programs as our canonical example . We propose a neural editor that runs iteratively . At each step , the editor generates and applies a tree edit ( e.g . deleting or adding a subtree ) to the partially edited tree , which deterministically transforms the tree into its modified counterpart . Therefore , the entire tree editing process can be formulated as consecutive , incremental tree transformations ( Fig . 1 ) . While recent works ( Tarlow et al. , 2019 ; Dinella et al. , 2020 ; Brody et al. , 2020 ) have also examined models that make changes to trees , our work is distinct from them in that : First , compared with Dinella et al . ( 2020 ) , we studied a different problem of editing tree-structured data particularly triggered by an edit specification ( which implies a certain edit intent such as a code refactoring rule ) . Second , we model structural edits via incremental tree transformations , while Tarlow et al . ( 2019 ) and Brody et al . ( 2020 ) predict a complete edit sequence based on the fixed input tree , without applying the edits or performing any tree transformations incrementally . Although Dinella et al . ( 2020 ) have explored a similar idea , our proposed tree editor is more general owing to the adoption of the Abstract Syntax Description Language ( ASDL ; Wang et al . ( 1997 ) ) . This offers our editor two properties : being language-agnostic and ensuring grammar validity . In contrast , Dinella et al . ( 2020 ) include JavaScript-specific design and employ only ad-hoc grammar checking . Finally , our tree editor supports a comprehensive set of operations such as adding or deleting a tree node and copying a subtree , which can fulfill a broad range of tree editing requirements . These operations are not fully allowed by previous work , e.g. , Brody et al . ( 2020 ) can not add ( or generate ) a new tree node from scratch ; Tarlow et al . ( 2019 ) and Dinella et al . ( 2020 ) do not support subtree copying . We further propose two modeling and training improvements , specifically enabled by and tailored to our incremental editing formalism . First , we propose a new edit encoder for learning to represent the edits to be performed . Unlike existing edit encoders , which compress tree differences at the token level ( Yin et al. , 2019 ; Hoang et al. , 2020 ; Panthaplackel et al. , 2020b ) or jointly encode the initial and the target tree pairs in their surface forms ( Yin et al. , 2019 ) , our proposed edit encoder learns the representation by encoding the sequence of gold tree edit actions . Second , we propose a novel imitation learning ( Ross et al. , 2011 ) method to train our editor to correct its mistakes dynamically , given that it can modify any part of a tree at any time . We evaluate our proposed tree editor on two source code edit datasets ( Yin et al. , 2019 ) . Our experimental results show that , compared with previous approaches that generate the edited program in one pass , our editor can better capture the underlying semantics of the intended edits , which allows it to outperform existing approaches by more than 7 % accuracy in a one-shot evaluation setting . With the proposed edit encoder , our editor significantly improves accuracy over existing state-of-the-art methods on both datasets . We also demonstrate that our editor can become more robust by learning to imitate expert demonstrations dynamically . Our source code is available at https : //github.com/neulab/incremental_tree_edit . 2 PROBLEM FORMULATION . As stated above , our goal is to create a general-purpose editor for tree-structured data . Specifically , we are interested in editing tree structures defined following an underlying grammar that , for every parent node type , delineates the allowable choices of child nodes . Such syntactic tree structures , like syntax trees of sentences or computer programs , are ubiquitous in fields like natural language processing and software engineering . In this paper we formulate editing such tree structures as revising an input tree C− into an output tree C+ according to an edit specification ∆ . As a concrete example , we use editing abstract syntax trees ( ASTs ) of C # programs , as illustrated in Fig . 1 . This figure shows transforming the AST of “ x=list.ElementAt ( i+1 ) ” ( C− ) to the AST of “ x=list [ i+1 ] ” ( C+ ) . In this case , the edit specification ∆ could be interpreted as a refactoring rule that uses the bracket operator [ · ] for accessing elements in a list.1 In practice , the edit specification is learned 1The corresponding Roslyn analyzer in C # can be found at https : //github.com/JosefPihrt/ Roslynator/blob/master/docs/analyzers/RCS1246.md . by an edit encoder f∆ from a pair of input-output examples 〈C ′− , C ′+〉 , and encoded as a real-valued edit representation , i.e . f∆ ( C ′− , C ′ + ) ∈ Rn . The learned edit representation could then be used to modify C− in a similar way as editing C ′− . Onwards we use f∆ as a simplified notation for edit representations . Revising a tree into another typically involves a sequence of incremental edits . For instance , to modify the input tree in the aforementioned example , one may first delete the subtree rooted at the node MethodCall , which corresponds to the code fragment “ list.ElementAt ( i+1 ) ” , and then replace it with an ElementAccess node denoting the bracket operator , etc . We formulate this editing process as a sequential decision making process ( 〈g1 , a1〉 , . . . , 〈gT , aT 〉 ) , where for each tree gt at time step t , the editor executes a tree edit action at , deterministically transforming it into gt+1 . In particular , g1 is the initial input tree C−.2 The process stops at gT when the editor predicts a special Stop action as aT . Denoting g1 : t = ( g1 , ... , gt ) as the tree history and a1 : t = ( a1 , ... , at ) the edit history until step t , then the editing can be framed as the following autoregressive process : p ( a1 : T |f∆ , g1 ) = p ( a1|f∆ , g1 ) p ( a2|f∆ , g1:2 ) · · · p ( aT |f∆ , g1 : T ) = T∏ t=1 p ( at|f∆ , g1 : t ) . ( 1 ) 3 MODEL . We will introduce our neural editor for modeling p ( at|· ) in § 3.1 , followed by the edit representation model f∆ in § 3.2 . 3.1 NEURAL TREE EDITOR . Fig . 1 ( c ) illustrates our editor architecture . At each time step , the editor first encodes the current tree gt and the tree history g1 : t. It then employs a modular decoder to predict a tree edit action at . Next , we will first introduce our tree edit actions and then elaborate the model details . 3.1.1 TREE EDIT ACTIONS . Our editor uses a sequence of editing actions to incrementally modify a tree-structured input . At each time step , the decoder takes an action at to update a partially-edited tree gt . Specifically , an action at consists of an operator ( e.g . an operator that removes a subtree from gt ) with its optional arguments ( e.g . the target subtree to delete ) . Importantly , the space of actions is limited to maintain consistency with the underlying syntax of the language . While a number of syntactic formalisms such as context free grammar ( Chomsky , 1956 ) or tree substitution grammar ( Cohn et al. , 2010 ) exist , in this work we choose the ASDL formalism due to its ability to flexibly handle optional and sequential fields ( interested readers may reference Wang et al . ( 1997 ) and Yin & Neubig ( 2018 ) for details ) . Under this framework , we define four types of operators . 2Notably , the special case of empty initial trees corresponds to code generation from scratch . Thus our formulation applies to both tasks of editing existing trees and generating new ones . Delete operators take a tree node nt as argument and remove nt and its descendants from gt ( e.g . t = 1 in Fig . 1 ( b ) ) . Note that removing arbitrary ( child ) nodes from gt might produce syntactically invalid trees , since under the grammar , a parent node type would always have a fixed set of edge types . For instance , if the node MethodCall and its incoming edge right were to be removed at t = 1 , the resulting AST would be syntactically invalid under C # ’ s grammar , as the node AssignStmt denoting a variable assignment statement is missing a child node representing its right operand . To maintain syntactic correctness ( no missing child nodes for any parent nodes ) , we therefore replace the to-be-deleted node with a pseudo Dummy node as a placeholder . Next , we define an Add operator to add new nodes to gt . The operator first locates a target position by selecting an existing tree node . We consider two cases based on the edge type ( or “ field ” ) of the target position : for single or optional fields that allow at most one child ( e.g . field right in Fig . 1 ( b ) at t = 1 ) , the selected tree node has to be their dummy child node ( e.g . node Dummy at t = 1 ) and the Add operator will then replace the dummy node with the new tree node ; for sequential fields that accept more than one child ( e.g . the field of a “ statement block ” that allows an arbitrary number of statements ) , the selected tree node can be any child node of the field ( including a rightmost dummy node we append to every sequential field ) and the Add operator will then insert the new node before the selected node . We elaborate this mechanism in § A.1 . For our editor , adding a non-terminal node ( e.g . node ElementAccess in Fig . 1 ( b ) at t = 2 ) is equivalent to selecting a production rule to derive its field ( e.g . AssignStmt right−−→ ElementAccess ) . As with Delete actions , to ensure there is no missing child node , we instantiate the set of child nodes with dummy nodes for the newly added node based on the underlying grammar , which leads to nodes Dummy1 and Dummy2 at t = 2 . Add can also be used to populate empty terminal nodes with actual values ( e.g . string token “ list ” at t = 3 ) . This is the same as picking a token from the token vocabulary . Additionally , observing that in many cases , revising a tree can be easily done by copying a subtree from the initial input g1 ( e.g . subtree Expr 7→ i + 1 in Fig . 1 ( a ) ) to a new position in the updated tree gt ( e.g . the right child position of node ElementAccess in Fig . 1 ( b ) at t = 4 ) , we introduce a high-level operator CopySubTree . This operator locates a target position similarly as the Add operator and then copies a complete subtree from g1 to the target position in a single step . Finally , a Stop action is used to terminate the iterative tree editing procedure , after which the remaining dummy nodes will be cleared . We note that our framework has decoupled the language grammar specifications ( handled by ASDL ) from the model architecture ( corresponding to our languageagnostic model implementation ) , and thus can be applied to various languages flexibly .

Summary: The paper presents an approach for predicting edits in programs, by modeling the programs as trees. The approach is mainly an extension of Yin et al. (2019), with the main difference that the model is required to predict only the output **actions**, instead of generating the entire output tree as in Yin et al. (2019). This difference of predicting only output actions is shared with other previous work though.