GRAPHIX: A Pre-trained Graph Edit Model for Automated Program Repair

1 INTRODUCTION . Detecting bugs and code quality issues in programs and fixing them is an important task in software development . Oftentimes , this two-step process is manual and labor-intensive . In order to ease the burden on developers and reduce the development cost , several automated tools are developed and integrated with development environments ( IDEs ) and/or code review systems . There exist many program analysis techniques ( Sadowski et al. , 2015 ) that use hand-written rules to provide linter-style recommendations while others ( Paletov et al. , 2018 ) are able to detect specific bug types ( e.g. , crypto API uses ) but leave them to the developer to fix . In the recent years , advances in deep learning have appealed researchers to move to data-driven approaches ( Vasic et al. , 2018 ) that learn to detect and fix bugs from data . Henceforth , we refer to the problem of automatically detecting and fixing bugs as automated program repair ( Goues et al. , 2019 ) . One main challenge with using machine learning for automated program repair is the lack of large human-labeled datasets . Several works ( Tufano et al. , 2019b ; a ) propose heuristics to extract bug-fix data from code changes . Code changes are represented as ( code before , code after ) pairs mostly at the method level , and typically pertain to feature enhancements , bug fixes and code refactoring . Code changes relating to bug fixes are identified through the presence of keywords such as “ bug ” in commit messages , through the size of changes or both . An advantage of using code change data is that there are plenty of code changes available in source code repositories . Based on code change data , the learning of bug fixes has been taken in three main directions . The first line of work regards the bug-fix learning problem as sequence-based code generation and harnesses the power of sequence-to-sequence models with RNN ( Hata et al. , 2018 ; Chen et al. , 2019 ; Tufano et al. , 2019b ; a ) and more recently with pre-trained Transformer models ( Feng et al. , 2020 ; Guo et al. , 2021 ; Ahmad et al. , 2021 ; Wang et al. , 2021 ; Berabi et al. , 2021 ) . Secondly , instead of generating the fixed code from scratch , Zhao et al . ( 2019 ) ; Chakraborty et al . ( 2020 ) ; Li et al . ( 2020 ) ; Panthaplackel et al . ( 2021 ) extend sequence models and propose to generate edits on the buggy code . Lastly , Yin et al . ( 2018 ) ; Tarlow et al . ( 2019 ) ; Dinella et al . ( 2020 ) ; Yao et al . ( 2021 ) deviate from the token-based representation of code and leverage the hierarchical structure in the abstract syntax tree ( AST ) . The high level idea is to learn a graph edit model to iteratively transform the AST of the buggy code into the AST of the fixed code . The learning is supervised by tree differencing edits via a cross-entropy loss . Despite the benefits of more precise bug localization , shorter tree edits and natural graph representations of programs , existing graph-based models have been less competitive than Transformer-based models thus far . In this paper , we pursue the direction of representing explicit program structures with graphs and make contributions toward source code understanding using machine learning . Specifically , we present GRAPHIX , a pre-trained graph edit model for automated program repair . Inspired by HOPPITY ( Dinella et al. , 2020 ) , we design GRAPHIX as a sequential decision process with a multi-head graph encoder and an autoregressive tree decoder . Conditioned on the graph state and the edit history at each step , the decoder iteratively makes primitive actions ( e.g. , adding a node ) on the graph corresponding to the input AST . Unlike HOPPITY ( Dinella et al. , 2020 ) , we enhance the encoder with multiple graph heads to capture diverse aspects of hierarchical code structures . In addition , we guide the decoder with an underlying Abstract Syntax Description Language ( ASDL ) . Owing to the syntax language , the graph structure and semantic edits , GRAPHIX exhibits strong inductive biases in learning generic fixing and refactoring patterns from code changes . Moreover , we devise a novel pre-training strategy , namely deleted sub-tree reconstruction , that enables GRAPHIX to learn implicit program structures from unlabeled data . Here , the model is pre-trained to reconstruct a randomly deleted sub-tree given other code context , using the same cross-entropy loss as the program repair task . Our pre-training task generalizes the masked language model and denoising objectives ( Devlin et al. , 2019 ; Lewis et al. , 2020 ) in sequence models to the AST representation of source code . Our idea shares the conceptual principle with generative models of code ( Li et al. , 2018 ; Brockschmidt et al. , 2018 ) for expression generation and the structural language model ( Alon et al. , 2020 ) for any-code completion . We evaluate GRAPHIX on both concrete and abstract versions of the Patches in The Wild Java benchmark ( Tufano et al. , 2019b ) . We show that GRAPHIX significantly outperforms edit-based and Transformer models , and the performance is as competitive as recent state-of-the-art pre-trained baselines despite using fewer parameters . Our in-depth analysis of the generated patterns demonstrates the ability of GRAPHIX to learn a wide range of meaningful and generic bug fixing and code refactoring patterns . Compared to HOPPITY , which merely learns short edits such as modifying access keywords ( private to public ) , adding break , or removing redundant type arguments , etc. , GRAPHIX is able to handle longer , more complicated structural and semantic edits ( up to 20 graph edits ) . We showcase a variety of bug-fix examples such as fixing off-by-one errors , possible null pointer exceptions , etc . in our analysis . In summary , we make the following main contributions in this work : 1 . GRAPHIX is a medium-scale graph edit model that can be pre-trained with the deleted sub-tree reconstruction objective for automated program repair . 2 . GRAPHIX significantly outperforms the edit-based models and is able to achieve high top-1 exact match accuracy on the Patches in the Wild benchmark even without pre-training . Using an explicit graph representation on the abstract syntax tree , our model is as competitive as large-scale Transformer models . 3 . Unlike previous work , which only focused on the abstract version of the benchmark , we evaluate GRAPHIX on both abstract and concrete code and show the effectiveness of our model in both settings . This also suggests that the code abstraction may not be necessary . 4 . Our in-depth analysis of the generated fixes demonstrates that GRAPHIX exhibits strong inductive biases in learning to both structural and semantic code patterns , leading many meaningful bug fixes . Despite our focus on Java , our model and the pre-training strategy can be extended further to work with multiple languages for universal source code understanding using a language-agnostic abstract language and parser such as tree-sitter1 . In addition , the syntax tree structure is highly amenable to program dependencies . We hope our work will inspire the community to advance the area further by incorporating more program dependencies , designing efficient graph learning algorithms with large model architectures and datasets . 1https : //tree-sitter.github.io/tree-sitter/ 2 RELATED WORK . Deep learning for program repair Advances in machine learning in the past decade have drastically changed the landscape of research for automated program repair . Inspired by Neural Machine Translation , Hata et al . ( 2018 ) ; Chen et al . ( 2019 ) ; Tufano et al . ( 2019b ; a ) regard code as a sequence of tokens and use sequence-to-sequence models to “ translate ” the buggy code into the fixed code . More recently , pre-trained Transformer-based models such as CodeBERT ( Feng et al. , 2020 ) , GraphCodeBERT ( Guo et al. , 2021 ) , PL-BART ( Ahmad et al. , 2021 ) and most recently CodeT5 ( Wang et al. , 2021 ) achieve state-of-the-art results on “ Patches in the Wild ” Java benchmark ( Tufano et al. , 2019b ) for program repair . A series of edit-based approaches model the code edits made on the buggy code as opposed to the “ translation ” -based counterparts . Zhao et al . ( 2019 ) ; Chakraborty et al . ( 2020 ) ; Li et al . ( 2020 ) ; Panthaplackel et al . ( 2021 ) extend sequence-to-sequence RNN models to predict the sequence of edits given the buggy code . Unlike these works , we build GRAPHIX on the inherent abstract syntax structure of code instead of its token-based representation . Graph neural networks GNNs such as graph convolutional networks ( Kipf & Welling , 2016 ) , graph attention networks ( Veličković et al. , 2018 ) provide a set of powerful tools for learning graph-structured data . Source code with rich graph structures in abstract syntax trees and program dependency graphs naturally fits to the GNN toolbox . Yin et al . ( 2018 ) ; Allamanis et al . ( 2018 ) ; Hellendoorn et al . ( 2019 ) ; Wang et al . ( 2020 ) augment the AST with additional data flow edges between variables and use GNNs for tasks such as variable name prediction . Yin et al . ( 2018 ) ; Tarlow et al . ( 2019 ) ; Dinella et al . ( 2020 ) ; Yasunaga & Liang ( 2020 ) ; Yao et al . ( 2021 ) use graph models based on ASTs for program repair and editing tasks in C , C # and Javascript , separately . Particularly , Brockschmidt et al . ( 2018 ) ; Dinella et al . ( 2020 ) ; Yao et al . ( 2021 ) use graph representations with treebased decoders for AST generation by editing the partial AST one node at a time . Brockschmidt et al . ( 2018 ) ; Yao et al . ( 2021 ) further equip the decoder with an Abstract Syntax Description Language ( ASDL ) to ensure the syntactic correctness of the generated code . Our work advances HOPPITY ( Dinella et al. , 2020 ) and ( Yao et al. , 2021 ) with multi-head graph encoding and pre-training and demonstrates the benefits of program structures on learning bug-fix patterns in Java code . Pre-training Transformer ( Vaswani et al. , 2017 ) and large-scale pre-training ( Radford et al. , 2018 ; Devlin et al. , 2019 ; Lewis et al. , 2020 ) have gained much traction in the broader field of natural language processing ( NLP ) . Sequence-based pre-training with Transformer has also proven to be effective in learning source code representations . In contrast , techniques for pre-training with graphs are less well-developed , especially for learning graph representations of source code . Existing work on graph pre-training are mainly based on attribute and edge prediction ( see ( Hu et al. , 2020 ) and references therein ) . Distinct from these works , our pre-training task is conceptually related to structural generative modeling of source code ( Brockschmidt et al. , 2018 ; Alon et al. , 2020 ) for tree generation , just as pre-training techniques in BERT and BART derived from language modeling and denoising for sequence generation in NLP . However , we focus on pre-training GRAPHIX to learn hierarchical tree structures of code for program repair rather than on generating code .

Summary: Automated program repair benefits from knowledge of its many properties, which includes its inherent (parse-)tree structure and graphical properties such as data-flow. This work proposes a graph-based encoder coupled with a tree-edit decoder, and optionally pretrained on a tree-based objective comparable to masked language modeling. The resulting model efficiently leverages relatively few parameters to achieve near-SOTA performance on a benchmark compiled from real-world bug fixes.