GRAPHIX: A Pre-trained Graph Edit Model for Automated Program Repair

1 INTRODUCTION . Detecting bugs and code quality issues in programs and fixing them is an important task in software development . Oftentimes , this two-step process is manual and labor-intensive . In order to ease the burden on developers and reduce the development cost , several automated tools are developed and integrated with development environments ( IDEs ) and/or code review systems . There exist many program analysis techniques ( Sadowski et al. , 2015 ) that use hand-written rules to provide linter-style recommendations while others ( Paletov et al. , 2018 ) are able to detect specific bug types ( e.g. , crypto API uses ) but leave them to the developer to fix . In the recent years , advances in deep learning have appealed researchers to move to data-driven approaches ( Vasic et al. , 2018 ) that learn to detect and fix bugs from data . Henceforth , we refer to the problem of automatically detecting and fixing bugs as automated program repair ( Goues et al. , 2019 ) . One main challenge with using machine learning for automated program repair is the lack of large human-labeled datasets . Several works ( Tufano et al. , 2019b ; a ) propose heuristics to extract bug-fix data from code changes . Code changes are represented as ( code before , code after ) pairs mostly at the method level , and typically pertain to feature enhancements , bug fixes and code refactoring . Code changes relating to bug fixes are identified through the presence of keywords such as “ bug ” in commit messages , through the size of changes or both . An advantage of using code change data is that there are plenty of code changes available in source code repositories . Based on code change data , the learning of bug fixes has been taken in three main directions . The first line of work regards the bug-fix learning problem as sequence-based code generation and harnesses the power of sequence-to-sequence models with RNN ( Hata et al. , 2018 ; Chen et al. , 2019 ; Tufano et al. , 2019b ; a ) and more recently with pre-trained Transformer models ( Feng et al. , 2020 ; Guo et al. , 2021 ; Ahmad et al. , 2021 ; Wang et al. , 2021 ; Berabi et al. , 2021 ) . Secondly , instead of generating the fixed code from scratch , Zhao et al . ( 2019 ) ; Chakraborty et al . ( 2020 ) ; Li et al . ( 2020 ) ; Panthaplackel et al . ( 2021 ) extend sequence models and propose to generate edits on the buggy code . Lastly , Yin et al . ( 2018 ) ; Tarlow et al . ( 2019 ) ; Dinella et al . ( 2020 ) ; Yao et al . ( 2021 ) deviate from the token-based representation of code and leverage the hierarchical structure in the abstract syntax tree ( AST ) . The high level idea is to learn a graph edit model to iteratively transform the AST of the buggy code into the AST of the fixed code . The learning is supervised by tree differencing edits via a cross-entropy loss . Despite the benefits of more precise bug localization , shorter tree edits and natural graph representations of programs , existing graph-based models have been less competitive than Transformer-based models thus far . In this paper , we pursue the direction of representing explicit program structures with graphs and make contributions toward source code understanding using machine learning . Specifically , we present GRAPHIX , a pre-trained graph edit model for automated program repair . Inspired by HOPPITY ( Dinella et al. , 2020 ) , we design GRAPHIX as a sequential decision process with a multi-head graph encoder and an autoregressive tree decoder . Conditioned on the graph state and the edit history at each step , the decoder iteratively makes primitive actions ( e.g. , adding a node ) on the graph corresponding to the input AST . Unlike HOPPITY ( Dinella et al. , 2020 ) , we enhance the encoder with multiple graph heads to capture diverse aspects of hierarchical code structures . In addition , we guide the decoder with an underlying Abstract Syntax Description Language ( ASDL ) . Owing to the syntax language , the graph structure and semantic edits , GRAPHIX exhibits strong inductive biases in learning generic fixing and refactoring patterns from code changes . Moreover , we devise a novel pre-training strategy , namely deleted sub-tree reconstruction , that enables GRAPHIX to learn implicit program structures from unlabeled data . Here , the model is pre-trained to reconstruct a randomly deleted sub-tree given other code context , using the same cross-entropy loss as the program repair task . Our pre-training task generalizes the masked language model and denoising objectives ( Devlin et al. , 2019 ; Lewis et al. , 2020 ) in sequence models to the AST representation of source code . Our idea shares the conceptual principle with generative models of code ( Li et al. , 2018 ; Brockschmidt et al. , 2018 ) for expression generation and the structural language model ( Alon et al. , 2020 ) for any-code completion . We evaluate GRAPHIX on both concrete and abstract versions of the Patches in The Wild Java benchmark ( Tufano et al. , 2019b ) . We show that GRAPHIX significantly outperforms edit-based and Transformer models , and the performance is as competitive as recent state-of-the-art pre-trained baselines despite using fewer parameters . Our in-depth analysis of the generated patterns demonstrates the ability of GRAPHIX to learn a wide range of meaningful and generic bug fixing and code refactoring patterns . Compared to HOPPITY , which merely learns short edits such as modifying access keywords ( private to public ) , adding break , or removing redundant type arguments , etc. , GRAPHIX is able to handle longer , more complicated structural and semantic edits ( up to 20 graph edits ) . We showcase a variety of bug-fix examples such as fixing off-by-one errors , possible null pointer exceptions , etc . in our analysis . In summary , we make the following main contributions in this work : 1 . GRAPHIX is a medium-scale graph edit model that can be pre-trained with the deleted sub-tree reconstruction objective for automated program repair . 2 . GRAPHIX significantly outperforms the edit-based models and is able to achieve high top-1 exact match accuracy on the Patches in the Wild benchmark even without pre-training . Using an explicit graph representation on the abstract syntax tree , our model is as competitive as large-scale Transformer models . 3 . Unlike previous work , which only focused on the abstract version of the benchmark , we evaluate GRAPHIX on both abstract and concrete code and show the effectiveness of our model in both settings . This also suggests that the code abstraction may not be necessary . 4 . Our in-depth analysis of the generated fixes demonstrates that GRAPHIX exhibits strong inductive biases in learning to both structural and semantic code patterns , leading many meaningful bug fixes . Despite our focus on Java , our model and the pre-training strategy can be extended further to work with multiple languages for universal source code understanding using a language-agnostic abstract language and parser such as tree-sitter1 . In addition , the syntax tree structure is highly amenable to program dependencies . We hope our work will inspire the community to advance the area further by incorporating more program dependencies , designing efficient graph learning algorithms with large model architectures and datasets . 1https : //tree-sitter.github.io/tree-sitter/ 2 RELATED WORK . Deep learning for program repair Advances in machine learning in the past decade have drastically changed the landscape of research for automated program repair . Inspired by Neural Machine Translation , Hata et al . ( 2018 ) ; Chen et al . ( 2019 ) ; Tufano et al . ( 2019b ; a ) regard code as a sequence of tokens and use sequence-to-sequence models to “ translate ” the buggy code into the fixed code . More recently , pre-trained Transformer-based models such as CodeBERT ( Feng et al. , 2020 ) , GraphCodeBERT ( Guo et al. , 2021 ) , PL-BART ( Ahmad et al. , 2021 ) and most recently CodeT5 ( Wang et al. , 2021 ) achieve state-of-the-art results on “ Patches in the Wild ” Java benchmark ( Tufano et al. , 2019b ) for program repair . A series of edit-based approaches model the code edits made on the buggy code as opposed to the “ translation ” -based counterparts . Zhao et al . ( 2019 ) ; Chakraborty et al . ( 2020 ) ; Li et al . ( 2020 ) ; Panthaplackel et al . ( 2021 ) extend sequence-to-sequence RNN models to predict the sequence of edits given the buggy code . Unlike these works , we build GRAPHIX on the inherent abstract syntax structure of code instead of its token-based representation . Graph neural networks GNNs such as graph convolutional networks ( Kipf & Welling , 2016 ) , graph attention networks ( Veličković et al. , 2018 ) provide a set of powerful tools for learning graph-structured data . Source code with rich graph structures in abstract syntax trees and program dependency graphs naturally fits to the GNN toolbox . Yin et al . ( 2018 ) ; Allamanis et al . ( 2018 ) ; Hellendoorn et al . ( 2019 ) ; Wang et al . ( 2020 ) augment the AST with additional data flow edges between variables and use GNNs for tasks such as variable name prediction . Yin et al . ( 2018 ) ; Tarlow et al . ( 2019 ) ; Dinella et al . ( 2020 ) ; Yasunaga & Liang ( 2020 ) ; Yao et al . ( 2021 ) use graph models based on ASTs for program repair and editing tasks in C , C # and Javascript , separately . Particularly , Brockschmidt et al . ( 2018 ) ; Dinella et al . ( 2020 ) ; Yao et al . ( 2021 ) use graph representations with treebased decoders for AST generation by editing the partial AST one node at a time . Brockschmidt et al . ( 2018 ) ; Yao et al . ( 2021 ) further equip the decoder with an Abstract Syntax Description Language ( ASDL ) to ensure the syntactic correctness of the generated code . Our work advances HOPPITY ( Dinella et al. , 2020 ) and ( Yao et al. , 2021 ) with multi-head graph encoding and pre-training and demonstrates the benefits of program structures on learning bug-fix patterns in Java code . Pre-training Transformer ( Vaswani et al. , 2017 ) and large-scale pre-training ( Radford et al. , 2018 ; Devlin et al. , 2019 ; Lewis et al. , 2020 ) have gained much traction in the broader field of natural language processing ( NLP ) . Sequence-based pre-training with Transformer has also proven to be effective in learning source code representations . In contrast , techniques for pre-training with graphs are less well-developed , especially for learning graph representations of source code . Existing work on graph pre-training are mainly based on attribute and edge prediction ( see ( Hu et al. , 2020 ) and references therein ) . Distinct from these works , our pre-training task is conceptually related to structural generative modeling of source code ( Brockschmidt et al. , 2018 ; Alon et al. , 2020 ) for tree generation , just as pre-training techniques in BERT and BART derived from language modeling and denoising for sequence generation in NLP . However , we focus on pre-training GRAPHIX to learn hierarchical tree structures of code for program repair rather than on generating code .

Summary: This paper presents a model over sequential structural tree edits, used for program repair based on ASTs. The model itself uses a graph encode and decoder to predict the sequential tree edits. The authors also introduce a method for pretraining the model on existing (non program repair) code data: they delete subtrees of arbitrary size from the code, and predict their reconstruction. The resulting model performs comparably to several other state of the art code repair models on the Patches in the Wild Java repair dataset, but with fewer parameters than several of the best-performing pre-trained models (CodeBERT, CodeT5).