DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons

The joint task of bug localization and program repair is an integral part of the software development process . In this work we present DeepDebug , an approach to automated debugging using large , pretrained transformers . We begin by training a bug-creation model on reversed commit data for the purpose of generating synthetic bugs . We apply these synthetic bugs toward two ends . First , we directly train a backtranslation model on all functions from 200K repositories . Next , we focus on 10K repositories for which we can execute tests , and create buggy versions of all functions in those repositories that are covered by passing tests . This provides us with rich debugging information such as stack traces and print statements , which we use to finetune our model which was pretrained on raw source code . Finally , we strengthen all our models by expanding the context window beyond the buggy function itself , and adding a skeleton consisting of that function ’ s parent class , imports , signatures , docstrings , and method bodies , in order of priority . On the QuixBugs benchmark , we increase the total number of fixes found by over 50 % , while also decreasing the false positive rate from 35 % to 5 % and decreasing the timeout from six hours to one minute . On our own benchmark of executable tests , our model fixes 68 % of all bugs on its first attempt without using traces , and after adding traces it fixes 75 % on first attempt . 1 INTRODUCTION . The dominant paradigm in automated program repair is the generate-and-validate approach , which our work follows . In this setting we assume the existence of a suite of test functions that identify the existence of a bug . We must then localize the bug and consider candidate fixes until finding a patch that satisfies the test suite . Throughout our experiments , we work with synthetic bugs in which the error has already been localized to a single buggy method . We take as input the buggy function , along with additional context depending on the experiment , such as surrounding context from the function ’ s file and a stack trace that exposes the buggy function . We feed that input to our sequence-to-sequence transformer , which attempts to generate the fixed function in its entirety . In our deployment scenario , we also attempt to localize the bug using the stack trace . At present we apply a simple heuristic based on which lines of the stack trace come from the developer ’ s own code , considering the most recently called lines to be the most suspect . In future we are interested in improving our heuristic using an encoder transformer that reranks methods given the stack trace . 2 RELATED WORK . 2.1 FAULT LOCALIZATION . Curiously , the standard approach to bug localization eschews stack traces and instead uses spectrumbased fault-localization ( SBFL ) . In this approach , lines of code are ranked in order of suspiciousness based on how many failing tests execute them . One example is the DStar formula ( Wong et al. , 2014 ) . Given a statement s that is executed by failed ( s ) failing tests and passed ( s ) passing tests , it computes the suspiciousness score S ( s ) = failed ( s ) e passed ( s ) + ( totalfailed− failed ( s ) ) where e is an exponent like 2 . A notable exception to the SBFL approach to fault localization is the task of build repair , in which the localization information comes from the compiler message rather than a test suite . Gao et al . use crash traces to query Stack Overflow and produce repairing edits ( Gao et al. , 2015 ) . The DeepDelta model was trained on build data from 300 million lines of Java code from Google projects ( Mesbah et al. , 2019 ) . Using a Neural Machine Translation approach , DeepDelta fixes half of all errors resulting from mismatched method signatures or a missing symbol on its first attempt . 2.2 FALSE POSITIVES . A pervasive problem in the field is the fact that tests are incomplete oracles . Many tools produce patches that are merely test-suite adequate more often than they are genuine fixes . For instance , when evaluated on the QuixBugs challenge ( Lin et al. , 2017 ) , a benchmark consisting of one-line bugs in classic algorithms , a set of ten bugpatching tools found solely false positives for nine functions , while only actually finding any genuine fix for seven ( Ye et al. , 2019 ) . Only three models found more genuine fixes than false positives , in all cases exactly one more genuine fix . The neural approach CoCoNuT does somewhat better , finding thirteen genuine fixes along with seven false positives ( Lutellier et al. , 2020 ) . A survey found similarly disappointing results on the larger benchmark Defects4j ( Just et al. , 2014 ) , a collection of real-world Java bugs ( Durieux et al. , 2019 ) . In stark contrast , we observe an excellent false positive rate with our approach . One can also evaluate bug-patching models without executing tests . The Patches in the Wild benchmark frames bug-patching as a sequence-to-sequence problem as in our approach , which enables the use of NLP metrics ( Tufano et al. , 2019 ) . The authors consider an edit to be a bonafide fix only if it is exactly the same as the edit the original developer made when committing it . SequenceR ( Chen et al. , 2018 ) both narrows and expands on the Patches in the Wild dataset by focusing on one-line changes only , while also providing additional context beyond the buggy method by including other methods ’ signatures , similarly to our use of code skeletons . This extended context gives a 15 % relative boost . 2.3 SYNTHETIC DATA . For certain bug-types it is possible to generate millions of synthetic bugs to train on . Devlin et al . ( Devlin et al. , 2017 ) train an RNN on Python to fix incorrect comparison operators , the mistaken use of “ is ” vs. “ is not ” , variable misuse , and forgotten “ self ” accessors . Overall , they achieve 86 % accuracy on synthetic bugs and 41 % on real-life bugs . Kanade et al . ( Kanade et al. , 2019 ) pretrain a BERT-style model “ CuBERT ” on a larger Python dataset and then finetune on a related suite of synthetic bugs , achieving over 90 % accuracy . Backtranslation is a more generic form of data augmentation , in which a model is trained to translate target data to source data ( Edunov et al. , 2018 ) . In our case , we have an abundance of bug-free data mined from GitHub . We train a backtranslation model to create synthetic bugs and augment the training data for our goal task of fixing bugs . Backtranslation is also put to use for the related task of grammatical error correction ( Kiyono et al. , 2019 ) . 2.4 PRETRAINING . Various task-agnostic pretraining approaches like BERT , BART , T5 , and GPT-3 ( Devlin et al. , 2017 ; Lewis et al. , 2020 ; Raffel et al. , 2019 ; Brown et al. , 2020 ) have seen large performance gains on a diverse array of benchmarks . These models are typically pretrained using a denoising objective on a large corpus of text that has been synthetically corrupted using bidirectional or causal masking , as well as more arbitrary noising such as randomly deleting , replacing , or permuting source tokens . Pretraining has also shown large improvements for the task of program repair ( Drain et al. , 2021 ; Lu et al. , 2021 ) . 3 MODEL . We reuse the 406M-parameter sequence-to-sequence transformer with twelve encoder layers and twelve decoder layers which was pretrained as described under section 4.1 . When experimenting with stack traces , we allot 1024 tokens for the code skeleton , and up to 896 tokens for the trace . In order to accommodate this larger context we thus need to expand the transformer ’ s positional embedding matrix . To this end we use axial embeddings , as inspired by reformer ( Kitaev et al. , 2020 ) . In our implementation , we duplicate the first 896 of the pre-existing 1024 positional embeddings , generate a random axial vector , and add that vector to each of the 896 duplicate embeddings . This approach outperformed randomly initialized embeddings in preliminary experiments . 4 DATA . We work with four different training datasets : Raw python code used for pretraining , commit data used for training a neural bug-creator and bug-patcher , methods extracted from the raw code in which we insert both neural bugs so as to train an even stronger bug-patcher , and , finally , methods that pass executable tests . For this last dataset , we also obtain the list of lines executed by each test , and we obtain another bug-patching dataset by again inserting synthetic bugs and rerunning the passing tests , which allows us to finetune a bug-patcher on stack traces , error messages , and print statements . We also experiment with giving our bug-patcher models either only the focal buggy method or also a ‘ skeleton ’ of the entire file that prioritizes data such as function signatures . 4.1 PRETRAINING . We build on the DeepDev transformer platform . We reuse the 406M-parameter DeepDev Python transformer that was warmstarted from FaceBook ’ s BART model and then further pretrained using a spanmasking objective ( Lewis et al. , 2020 ; Clement et al. , 2020 ) . The pretraining data consists of 200,000 public Python repos filtered to have at least five stars . Pretraining took place on a DGX-2 box for three weeks . Note that the DeepDev tokenizer has appended whitespace tokens , such as the four-space and eight-space tokens , which boosts throughput and the effective context length . To minimize any risk of leakage , we consistently restrict to the same validation and test repositories , in particular to those repositories used in CodeSearchNet ( Husain et al. , 2019 ) . 4.2 COMMIT DATA . We traverse the commit history of 100,000 Python repositories that were filtered to have at least ten stars . We further filter to all commits whose message contains the word “ fix '' , roughly one fifth of all commits . Based upon inspecting many examples , it seems that this simple filter is approximately as precise as more restrictive filters that insist on phrases like “ patch bug ” or “ fix error '' . Nevertheless , the data is still extremely noisy . This commit data serves two purposes for us . First , it allows us to train an edit model which is biased towards constructive , bug-fixing edits . We can evaluate such a model directly on bug-fixing , or finetune it on more filtered bug data . Second , we can reverse the input and output and train an edit model biased towards destructive , bug-inducing edits . We can use this model to create neural bugs to greatly augment our training data . This backtranslation approach has already proven useful elsewhere in nlp . Since we are interested in fixing buggy methods , we consider each method edited by each commit . As we are only interested in nontrivial edits , we normalize each method before and after the commit and discard the edits that do not affect the normalized code . To normalize , we strip comments , replace string and numeric literals with placeholders like ‘ STR_LIT ’ , and standardize whitespace . Finally we are left with 1.1 million , nontrivially edited methods . We experiment with giving solely the focal method and its edit as input and output , as well as putting more context into the input , as described in the section on Code Skeletons .

Summary: This manuscript describes DeepDebug a transformer-based mode that performs code repair on Python methods. Specifically, the model is pretrained on 200k Python code repositories and fine-tuned on a number of datasets consisting of bug fix commits and augmentation through synthetic bugs. The model takes Python methods with bugs as the input and outputs potentially bug free versions of the method. The main contribution of the paper, if held up through the questions I ask below, include: - An innovative application of the backtranslation approach on the code repair domain. - Setting a new state-of-the-art (SOTA) performance for code repair on the Python problems from the QuixBugs benchmark, beating previous SOTA (e.g., CoCoNut) not only in the percentage of bugs fixed, but also in using fewer attempts and significantly less latency budget.  While these contributions are exciting and potentially pushing the boundary of deep learning-based code repair, there are some major issues  with how the methodology is described, how the results are presented, and how the implications of the results are discussed. These critique points from me are listed in the "Main Review" section below.