DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons
Authors: Dawn Drain
Colin Clement
Guillermo Serrato Castilla
Neel Sundaresan
Venue: ICLR 2022 Submitted (2022)
Keywords: program repair
bugpatching
backtranslation
transformers

Abstract: The joint task of bug localization and program repair is an integral part of the software development process. In this work we present DeepDebug, an approach to automated debugging using large, pretrained transformers. We begin by training a bug-creation model on reversed commit data for the purpose of generating synthetic bugs. We apply these synthetic bugs toward two ends. First, we directly train a backtranslation model on all functions from 200K repositories. Next, we focus on 10K repositories for which we can execute tests, and create buggy versions of all functions in those repositories that are covered by passing tests. This provides us with rich debugging information such as stack traces and print statements, which we use to finetune our model which was pretrained on raw source code. Finally, we strengthen all our models by expanding the context window beyond the buggy function itself, and adding a skeleton consisting of that function's parent class, imports, signatures, docstrings, and method bodies, in order of priority. On the QuixBugs benchmark, we increase the total number of fixes found by over 50%, while also decreasing the false positive rate from 35% to 5% and decreasing the timeout from six hours to one minute. On our own benchmark of executable tests, our model fixes 68% of all bugs on its first attempt without using traces, and after adding traces it fixes 75% on first attempt.

Under review as a conference paper at ICLR 2022
DEEPDEBUG: FIXING PYTHON BUGS USING STACK TRACES, BACKTRANSLATION, AND CODE SKELETONS
Anonymous authors Paper under double-blind review
ABSTRACT
The joint task of bug localization and program repair is an integral part of the software development process. In this work we present DeepDebug, an approach automated debugging using large, pretrained transformers. We begin by training a bug-create bug-creation model on reversed commit data for the purpose of generating synthetic bugs. We apply these synthetic bugs toward two ends. First, directly train a backtranslation model on all functions from 200K repositories. Next, we focus on 10K repositories for which of all functions in those repositories that are covered by passing tests. This provides with rich debugging information such as stack traces and print statements, which we use to finetune our model which was pretrained on raw source code. Finally, we strengthen all our models by expanding the context window beyond the buggy function itself, and adding a skeleton consisting of that function's parent class, imports, signatures, docstrings, and method bodies, in order of priority. On the QuixBugs benchmark, while also decreasing the false positive rate from 35% to 5% and decreasing the timeout from six hours to one minute. On own benchmark of executable tests, our model fixes 68% of all bugs on its first attempt without using traces, and after adding traces it fixes 75% % first attempt.
execute tests, and create buggy versions
increase the total number of fixes found by over 50%,
INTRODUCTION
The dominant paradigm in automated program repair is the senerate-and-validate approach, which our work follows. In this setting we assume the existence of suite of test functions that identify the existence of bug. We must then localize the bug and consider candidate fixes until finding a patch that satisfies the test suite.
Throughout our experiments, we work with synthetic bugs in which the error has already been localized single buggy method. We take as input the buggy function, along with additional context depending on the experiment, such surrounding context from the function's 's file and stack trace that exposes the buggy function. We feed that input to our sequence-to-sequence transformer, which attempts to generate the fixed function in its entirety.
In our deployment scenario, we also attempt to localize the bug using the stack trace. At present apply simple heuristic based which lines of the stack trace come from the developer's own code, considering the most recently called lines to be the most suspect. In future we are interested in improving our heuristic using an encoder transformer that reranks methods given the stack trace.
2
RELATED WORK
2.1 FAULT LOCALIZATION
Curiously, the standard approach to bug localization eschews stack traces and instead uses spectrum- based fault- localizocai (SBFL). In this approach, lines of code ranked in order of suspiciousness based on how many failing tests execute them. One example is the DStar formula (Wong et al.) 2014). Given statement that is executed by failed(s) failing tests and passed(s) passing tests,

---

one-line changes Under review as a conference paper at ICLR 2022
Figure 1: Our training pipeline. Following best practice take advantage of extensively pretrained transformers, in this case using the same DeepDev-py PyMTS (Clement al. 2020). We first use commit data to train both a baseline bugpatcher model along with a bug-creator model. Our bug-creator models feeds twenty times as much data to DeepDebug (backtrans). Finally we finetune this model on neural bugs injected into functions that have executable tests and produce traces, thus obtaining its final evolution DeepDebug (traces).
model used to finetune
Init Transformer
BART
DeepDev-py
Commit Data Bugpatcher-commit
English Pretraining
Python Pretraining
Training
Reversed
Commit Data Training
Mu
Model
Neural Bug
Finetuning
Training
Bug creator
Neural Bug Training with Traces
Synthetic Data
Creation
DeepDebug (backtrans)
DeepDebug (traces)
computes the suspiciousness
failed(s) S(s) passed(s) + (Lotal failed failed(())
where e is an exponent like 2.
notable exception the SBFL approach to fault localization is the task of build repair, in which the localization information comes from the compiler message rather than test suite. Gao et al. use crash traces to query Stack Overflow and produce repairing edits (Gao et al. 2015). The DeepDe t model was trained on build data from 300 million lines of Java code from Google projects (Mesbah) ctt al. 2019). Using a Neural Machine Translation approach, DeepDel fixes half of all errors resulting from mismatched method signatures missing symbol on its first attempt.
2.2 FALSE POSITIVES
pervasive problem in the field is the fact that tests are incomplete oracles. Many tools produce patches that are merely test-suite adequate more often than they genuine fixes. For instance, when evaluated on the QuixBugs challenge (Lin et al. 2017). a benchmark consisting of one-line bugs in classic algorithms, set of ten bugpatching tools found solely false positives for nine functions, while only actually finding any genuine fix for seven (Ye et al. 2001)). Only three models found more genuine fixes than false positives, in all
exactly one more genuine fix. The neural approach CoCoNuT does somewhat better, finding thirteen genuine fixes along with seven false positives (Lutellier et al. 2020). A survey found similarly disappointing results on the larger benchmark Defects4j (Just et al. 2014), collection of real-world Java bugs (Durieux et al. 2019). In stark contrast, we observe an excellent false positive rate with our approach.
One can also evaluate bug-patching models without executing tests. The Patches in the Wild bench- mark frames bug-patching as a sequence-to-sequeme problem as in our approach, which enables the use of NLP metrics (Tufano al.) 2019) The authors consider an edit to be bonafide fix only if it is exactly the same as the edit the original developer made when committing it. SequenceR (Chen et al. 2018) both narrows and expands the Patches in the Wild dataset by focusing only, while also providing additional context beyond the buggy method by including other methods' signatures, similarly our use of code skeletons. This extended context gives 15% relative boost.
2

---

Under review as a conference paper at ICLR 2022
2.3 SYNTHETIC DATA
For certain bug-types it is possible to generate millions of synthetic bugs to train on. Devlin et al. (Devlin et al. 2017) train an RNN on Python to fix incorrect comparison operators, the mistaken use of "is" vs. "is not", variable misuse, and forgotten "self" accessors. Overall, they achieve 86% accuracy on synthetic bugs and 41% on real-life bugs. Kanade et al. (Kanade et al., 2019) pretrain BERT-style model "CuBERT" synthetic bugs, achieving over 90% accuracy.
larger Python dataset and then finetune related suite of
Backtranslation is target data to source data (Edunov et al. 2018). In our case, we have an abundance of bug-free data mined from GitHub. We train backtranslation model create synthetic bugs and augment the training data for our goal task of fixing bugs. Backtranslation is also put to use for the related task of grammatical error correction (Kiyono et al. 2019).
generic form of data augmentation, in which model is trained translate
2.4 PRETRAINING
Various task lask-agnostic pretraining approaches like BERT, BART, T5, and GPT-3 (Devlin al. 2 Lewis et al.) 2020: Raffel et al. [2019] Brown et al. 2020) have seen large performance gains on a diverse array of benchmarks. These models are typically pretrained using denoising objective a large corpus of text that has been synthetically corrupted using bidirectional or causal masking, well more arbitrary noising such randomly deleting, replacing, permuting source tokens.
Pretraining has also shown large improvements for the task of program repair (Drain et al. 2021 Lu
al. 102T
MODEL
We reuse the 40rM-paramcter sequence-to-sequene transformer with twelve encoder layers and twelve decoder layers which pretrained as described under section 4.1
When experimenting with stack traces, we allot 1024 tokens for the code skeleton, and up to 896 tokens for the trace. In order to accommodate this larger context thus need to expand the transformer's ' positional embedding matrix. To this end we use axial embeddings, as inspired by reformer (Kitaev et al. 2020). In our implementation, we duplicate the first 896 of the pre-existing 1024 positional embeddings, generate a random axial vector, and add that vector to each of the 896 duplicate embeddings. This approach outperformed randomly initialized embeddings in preliminary experiments.
DATA
We work with four different training datasets: Raw python code used for pretraining, commit data used for training a neural bug-creator and bug-patcher, methods extracted from the raw code which we insert both neural bugs so to train an executable tests. For this last dataset, we also obtain the list lines executed by each test, and we obtain another mug-patching dataset by again inserting synthetic bugs and rerunning the passing tests, which allows us to finetune a bug pat-par on stack traces, eтror messages, and print statements. We also experiment with giving bug-patcher models either only the focal buggy method also 'skeleton' of the entire file that prioritizes data such as function signatures.
stronger bug-patcher, and, finally, methods that pass
4.1 PRETRAINING
We build on the DeepDev transformer platform. We reuse the 406M-parameter DeepDev Python transformer that was warmstarted from FaceBook's BART model and then further pretrained using spanmasking objective (Lewis et al. 2020 Clement et al. 2020). The pretraining data consists of 200,000 public Python repos filtered to have least five stars. Pretraining took place a DGX-2 box for three weeks. Note that the DeepDev tokenizer has appended whitespace tokens, such as the four-space and eight-space tokens, which boosts throughput and the effective context length.
3

---

Under review as a conference paper at ICLR 2022
To minimize any risk of leakage, we consistently restrict to the same validation and test repositories, in particular to those repositories used in CodeSearchNet (Husain et al. 2019).
4.2 COMMIT DATA
We traverse the commit history of 100,000 Python repositories that were filtered to have at least stars. We further filter to all commits whose message contains the word "fix", roughly one fifth of all commits. Based upon inspecting many examples, seems that this simple filter approximately as precise as more restrictive filters that insist on phrases like "patch bug" or "fix error". Nevertheless, the data still extremely noisy.
This commit data serves two purposes for us. First, it allows us to train an edit model which is biased towards constructive, bug-fixing edits. We can evaluate such model directly on bug-fixing, or finetune it on more filtered bug data. Second, we can reverse the input and output and train an edit model biased towards destructive, bug-inducing edits. We can use this model to create neural bugs greatly augment our training data. This backtranslation approach has already proven useful elsewhere in nlp.
Since we interested in fixing buggy methods, we consider each method edited by each commit. As we are only interested in nontrivial edits, we normalize each method before and after the commit and discard the edits that do not affect the normalized code. To normalize, strip comments, replace string and numeric literals with placeholders like "STR_LIT" and standardize whitespace. Finally we left with 1.1 million, nontrivially edited methods.
We experiment with giving solely the focal method and its edit as input and output, as well as putting more context into the input, as described in the section on Code Skeletons.
Table I: Cross-entropy loss for training two transformers, one on commit data and the other on reversed commits. The two models are evaluated both forward and backward edits, and with and without code skeletons. The cross- cross-entropy losses are five times better than those normally reported for generating Python code since the editing task is relatively easy (SSvyatkovkkiy et al.) 2020; Clement et al. 2020)). Furthermore, the losses for reversed edits are third lower than for forward edits. The forward model is 6% better at forward edits than the reverse model, and the reverse model is in turn 6% better at reverse edits. Both models perform 2% better with skeletons than with only the focal method. We report the results in base two on the test set.
Forward Edit Model
Backward Edit Model
Forward Edits With Skeletons
.240
.258
Forward Edits, focal method only
.248
.265
Reverse Edits, With Skeletons
.163
.155
Reverse Edits, focal method only
.167
.157
4.3 SYNTHETIC BUGS
4.3.1 NEURAL BUGS
We use the term 'neural bugs' refer to synthetic bugs that are created using neural edit model, such as the one we trained to revert bug-fixing commits. Using neural bugs for data augmentation has many appealing features. Flexible neural models can generate near arbitrary edits that are drawn from the distribution of mistakes developers actually make. For instance, neural edit model might swap get_key with get_value, whereas a simple heuristic approach might make unlikely or random swaps, such switching get_ key with reverse list. Furthermore, this approach is nearly language agnostic, as we can reuse our framework for mining commits, and only need a parser to extract classes and methods along with the constituent parts need form the code skeletons.
We present three examples of neural bugs. In our first example, the bug-creator model replaces kwargs.pop with kwargs.get, used elsewhere in the function. In the second example, our bug- creator model replaces a check for a bucket name starting with self.name with a stricter equality check. In the third example, model deletes break statement.
4

---

Under review as a conference paper at ICLR 2022
Fixed
Buggy
def get_.key (eel,, schoozgp)s
deř gei Ashaarge)
eenyusa "fozee! ( In
veryass 'force" _ee_Lerritteesa
since the ca2l zigeatere foot_nkyy_ can ouc onanged,
51050 signatere _ost__oy_ can noe be shanged,
if Exangs peg ("foree'y Kone]:
se exargs, get
(Mforeel Konel:
heacezs exargs.get htt.ggetttehheeeeeeee l'headers",
headets skargs.get (" (1)
headere (Mforee") 1 True
headers force" Tree
kvazga ("headezs ] headers
kvazgs "hoddera") headers
return super (Backet, seltt.get Key
return (Backe, celfligen key_
+>xNaEq3)
Figure 2: Our bug-creator model replaces kwargs.pop pop with kwargs.get get, as used elsewhere in the function.
Fixed
Buggy
Doer [awil] for bucket
def bucket boto_coar. aat___l_________eeeeaa
if bucket bato_ccan, get_ alt__a11_a buskkttt
buckets (1
for key in bucket. 1aat
for key in bucket.11st dey.dol
1ey.deleete))
bucket.delese dellate
luckaz.dclete
key
keye
for selt. redtireesss tp1 ccnncction '+1)1
tpl. connect relciredis delete
rocio delete Ik)
Figure 3: Our bug-creator model replaces a check for the bucket name starting with self. name with stricter equality check.
We also observe our model injecting errors such as replacing a dot accessor with bracket accessor, replacing comparison operators like ">=" with "<", truncating chained function calls, deleting return lines, wrapping return values in objects such as tuples and dictionaries and conversely forgetting to wrap objects, replacing precise errors like IndexError with different errors like ValueError, intuitively misnaming variables such as self.result instead of self._resull and mistakenly copying by reference instead of copying by value. In particular we observe all heuristic bugs that have previously been reported in the literature.
Nevertheless, we note a few shortcomings of our approach in generating neural bugs. Our models often hallucinate member variables that do not exist, although this is partially remedied by using longer context than the focal method alone. This a minor issue, although using longer context is especially important when training bugpatching, bugs are often otherwise underspecified. Our approach over-represents bugs that arise from partial refactoring, such as changing member variable names or function signatures. Our approach under-represents bugs that are caught before being committed, such as syntactic errors and typos. Our approach is also prone to generating superficial changes, such as trivial changes formatting, local variable names, and practically equivalent edits such as replacing an "is not None" check with "!= None" check.
One glaring shortcoming of our model, although not necessarily our approach, is that our model prematurely generates the end-of-sentence token in around a fifth of all instances during random sampling. Indeed, these cases represent large majority of edits containing syntax errors. This occurs most often during difficult edits, such when inside nested scope such as f(["..., especially when manipulating strings or numbers or when invoking user-defined functions or objects from user-defined classes. We could artificially decrease the probability of generating the end-of-sentened token, although we do not explore this option and instead filter out 90% of bugs that contain syntactic emrrr after they produced.
4.3.2 HEURISTIC BUGS
We use the term 'heuristic bugs' to refer to synthetic bugs that are manually created using simple rules, such deleting line or swapping parameters in function call.
5

---

Under review as a conference paper at ICLR 2022
Fixed
Buggy
def _ww.eratt__aae......... Sat.oarat__AA__sniie Sin.siinnineeeeeee single
def _guneraae__aa SAA_ win.angggeennn (sulf,
sections, section_hoippe, valcer:
secticns, section_hoighs, valuel:
sections:
seccions:
1f value > [s]]
if value >n vections[s]: Sax
bzoak
zotuzn
Figure 4: Our bug-creator model deletes the break, leading take the last small value in sections, instead of the first. Our model also introduces a trivial indentation eπrоr on the penultimate line.
Previous work has typically considered five or fewer types of bugs at a time, such as replacing a binary operator (e.g. using != instead of ==), using the wrong variable, forgetting the 'self." accessor, or deleting code.
4.3.3 RAW METHODS
Since we are interested in data augmentation via synthetic bugs, we can make use of the plenitude of relatively bug-free code on GitHub. In doing so potentially expand dataset of target methods over twenty-fold compared to working solely with functions mined from bug-fixing commits. Furthermore, we can scale up nearly arbitrarily by creating multiple buggy version of each method. For this paper we restrict to 1.3M functions from 10K repositories, near parity with the commit data, and scale 18M bug-fix pairs via backtranslation. We leave further scaling to future work.
4.4 METHODS WITH EXECUTABLE TESTS
There world opportunity for debugging code that can actually be executed, especially if there is an accompanying test to verify the execution was correct. A typical debugging session involves homing on the suspect block of code with the help of the stack trace, inserting print statements and breakpoints in an approximate binary search, modifying and executing snippets of code to gauge one's intuition, searching StackOverflow for interpretations of error messages and examples of API usage, talking to ducks, and incrementally making progress. In contrast, the baseline neural model is impoverished, and can only do the equivalent staring at a piece code for a few seconds before decisively writing the correction
token time.
At a minimum, the generate-and-vaalt validate approach enabled by executable tests allows for improved performance via multiple chances. For instance, in the domain of short Java methods, researchers have consistently seen top-20 accuracy triple that of the top-1 accuracy (Allamanis et al., 2017] Tufano al. 2019). Indeed, truly random edits be sure of passing test suite with sufficiently many tries, although prior work has shown that these edits can aggressively overfit, as tests found in the wild are not perfect oracles and can often be gamed by simply deleting functionality (Qi et
2015).
4.4.1 TRACES
In addition to using tests to triage inaccurate edits, we incorporate information from tests into training three different ways: appending the error message to the buggy method, additionally appending the stack trace, and further providing the values of all local variables at the point of failure, the last of which is conveniently provided by our testing framework Pytest.
4.4.2 COLLECTING PASSING TESTS
To collect executable tests at training scale we start with the 200K repositories used for pretraining and filter 35K repositories containing tests and setup.py requirements.tx file. For each of these repositories we execute Pytest a unique container, ultimately collecting passing tests from 10K repositories.

---

Under review as a conference paper at ICLR 2022
Figure 5: Example output for a sin- gle frame of the stack trace produced by Pytest In general there are sev- eral frames for a given stack trace. From top to bottom we can see: the in- put variables (self, match, and func in this case); the head of the functioo up until the final line executed, denoted with a '>' sign; the error message; the values of all local variables; and fi- nally the file path, line number, and error name. Line breaks and ellipses have been manually added for read- ability. We append the Pytest trace to the buggy code skeleton, and experi- ment with withholding the heads and local variables.
Example Pytest Trace
selr Hobskendpplicatiio Inange "ostCatchsiiPittrre'
natch rw.compila(ar...
tine <function att.aaaiaattssaaaarrrrooo
coofoocs atter_all at 0xit5570a81d20>
match, [unc)
Eilter_tiple self seli, rilter imatch, func)
Actribute@rror: crookerrpplccaiion detect
...
fine = <fuaction betcaccailPltessatrr..eooo
www.alls..attrraaa...... at 0xTE5170a81120>
=e..umplle (b".**)
self
Hobokenbeplicttion (nana- "oacraccaaaacllliittttt
dobug Faisel
babokan/spoliattee 4B: AttributeEeror
4.4.3
TESTING SYNTHETIC BUGS
After filtering to functions with passing executable tests and inserting neural bugs, then rerun the tests to collect Pytest traces and filter out edited functions that still pass their tests and thus are not actually buggy.
4.5
CODE SKELETONS
Much previous work on bug-fixing has used very abbreviated context consisting solely of the buggy focal method. However, many bugs are not detectable or remediable without larger context. In particular, we often need to reference user-defined functions, classes, and global variables, as well as the list of imported libraries.
To that end, we experiment with fleshing out the entire 1024-token window using the sighest-priority lines within the focal file. We place highest priority on the focal buggy method, followed by its class's definition applicable, imports, its class docstring and attributes, globals, other methods' signatures, other methods docstrings, and finally global variables and other methods' bodies. We also single out focal method using the special book-ending comments "# target edit" and "# end".
5 EXPERIMENTS AND RESULTS
We experiment with training on backtranslation data, adding skeletons, and adding the Pytest stack trace.
5.1 BACKTRANSLATION
In our first experiment we compare between training only on forward commit data vs. training on synthetic bugs produced via backtranslation. We evaluate using cross-entropy the holdout data.
Evaluated on holdout forward commit data as shown table 2 DeepDebuug (backtrans) attains up to 10% better loss. Surprisingly, the backtranslation model actually performs worse on reversed commit data, which is consistent with its bias towards constructively fixing buggy functions.
Overall DeepDebug is much stronger than the previous state of the art. In table 3 report model's results on the QuixBugs challenge, a benchmark of forty classic algorithms with small synthetic bugs, with nearly equivalent versions in both Python and Java (Lin et al., 2017). The original QuixBugs challenge was for developers to fix as many bugs as possible given one minute apiece.
We limit our model to generating one hundred patches via random sampling, which is approximately the number that be generated and evaluated over the span of one minute. We improve upon
7

---

Under review as a conference paper at ICLR 2022
Example Skeleton
import paycopg2
isport paycopg2.
fron dpostools. schemes import schemes fron dpostools, utils import dist ionity fron dpostools import api, excepticas
class ObConnection:
Figure 6: Example skeleton using just 400 tokens. Note the inclusion of the entire buggy focal function (with its name highlighted for the reader) including the control tokens '# tar- get edit' and '# end", well as the imported functions. Due to limited space, the skeleton can only fit all signatures from the focal class, along with several more signatures from out- side the focal class. The skeletons use in practice contain more to- kens and thus tend to be fleshier i.e. contain more doestrings and function bodies.
parswerd, hostt"'oosstosst, def 1intt_tteell neinnet
def comcetion(soltt):
class obCurmor:
-iit (ocif, usez, passverd, hest "ocsssitos,,sst',
catapese-'arz 'azt naimmttee'' dbcomnetioo--onne)
def description (scif):
def (sclu 43%, earge, cur_type-Xone);
retchall (cell, cor_type-Nonel:
def fetchans cur_type Nonet:
def
9ry, +arge, out Nor_type-None):
targot cdit
def excoute_ana, caroe_nn__tseeee techhtccconn ary, eargs, out Kope: exff.xxccteeeae lary, s2r22, (xzran__vpooaar__cpp)
zeturn self. Telchiao bysceett_.ypeesuu...
class OposMode:
def _init usez, passvord, bostt "ooallaa,,,,
databaso fark_nimmet''
def acomnnt dietanl la (swlf, address)
def noight detallddeelllll (sel)):
rode height fself, :fference)
def all del eglless (self):
def ousrent delesadeelaaaa [selft:
def payouta address (self, addresz)
def
tawii, addrwss)
def vote__ bydaeesssss
Table 2: Cross-entropy results for training two transformers, one trained on commit data and the other patching neural bugs. As in table the models evaluated on commit data. Note that the neural bugpatcher does comparatively worse on reversed edits vs. forward edits. The neural bugpatcher actually does worse when using skeletons on commit data, presumably due the fact that commits typically edit multiple functions. Finally, the neural bugpatcher outperforms the forward edit model when evaluated on individually edited functions.
Forward Edit Model
DeepDebug (hacktrans)
Forward Edits With Skeletons
.240
.243
Forward Edits, focal method only
2488
.211
Reverse Edits, With Skeletons
.163
.196
Reverse Edits, focal method only
.167
.174
Table 3: Results on the QuixBugs Benchmark. We use timeout of one minute for each bug and perform joint bug-localization and program repair. In contrast, prior works use timeout of several hours and often assume that the buggy line has already been localized.
Suite of Ten non-neural Models
CoCoNNur
Deepnebuu (hacktrans) (ours)
Duricux et al. 2019
Lunellir et al 2020
Number of Bugs Fixed
7:40 (18%)
13/40 (33%)
21/40 (535))
Number False Poxitives
True Posiiive Rule
44%
65%
95%
Number of Verbuim Fixes
13
8

---

Under review as a conference paper at ICLR 2022
the previous state of the art for number of bugs found by over 50% (Lutellier et al. 2020 while simultaneously decreasing the false positive rate from 35% to 5%. Notably all our models generate many duplicate edits, due to the low-perplexity of this task, which suggests room for improved sampling. Note that these results are even more impressive given the hints and timeouts provided to previous models. For instance, CoCoNu'T is explicitly told which line contains the bug and allows six hours to find a patch (Lutellier et al., 2020). Similarly, five of the non-neural tools collectively found 122 test-suite adequate patches for the longest increasing subsequence algorithm (Durieux et al. 2019) suggesting thousands of attempts.
5.2 SKELETONS
In the second experiment we compare between training and evaluating using only the focal function as input vs. giving the entire skeleton as input. The neural bugpatcher sees large 25% loss decrease when using skeletons when evaluated on neural bugs, as shown in table Surprisingly, the neural bugpatcher actually does worse when using skeletons on commit data as reported in table 2 presumably due to the fact that commits typically edit multiple functions.
Table 4: The neural bugpatcher large 25% cross-entropy loss decrease when using skeletons when evaluated on our large test set of neural bugs. These neural bugs are in general not executable.
DeepDebug (backtrans)
Neural Bugpatching, With Skeletons Neural Bugpatching, focal method only
.306
.409
5.3 TRACES
In the third experiment we append the Pytest stack trace to the buggy input. We expand the context window using axial embeddings in order to fit these additional tokens.
We filter a benchmark of 523 neural bugs from 100 repositories in our validation set. We observe impressive performance, and, contrarily to our cross-entropy results, large boost from using traces. The top-10 10 patch success rate for DeepDebug (backtrans) is 90%, and the top- 10 rate for DeepDebug (traces) is 97%. Note that these are highly nontrivial bugs, with a median length of two lines that must be edited to produce a fix. See table
Table 5: Patch success results on benchmark of neural bugs with executable tests.
DeepDebug (backtrans)
DeepDebug (traces)
Top-I Success Rate
357/523 (68%)
393/523 (75%)
Top-10 Success Rate
472/523 (90%)
509/523 (97%)
FUTURE WORK
Having code with executable tests opens up a world of opportunities for algorithmic debugging with machine learning. We are interested in genuinely iterative fixing (in contrast to independently generating edits), moving beyond the assumption that the bug has already been localized to a single method, and expanding to other popular languages. Most importantly we are interested in deploying a command- line interface to our tool.
REFERENCES
Allamanis, M., Brockschmidr, M., and Khademi, M. (2017). Learning represent programs with graphs.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam,
P., Sastry, G., Askell, A., Agarwal, S., Herbert-Vone A., Krueger, G., Henighan, T., Child, R.,

---

Under review as a conference paper at ICLR 2022
Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.. Sutskever, L., and Amodei, D. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1876-1900. Curran Associates, Inc.
Chen, Z.., Kommrusch, S., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., and Monperrus, M. (2018). Sequencer: Sequence-to-sequene learning for end-to-end program repair.
Clement, C., Drain, D., Timcheck, J., Svyatkovskiy, A., and Sundaresan, N. (2020). PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9052 9065, Online. Association for Computational Linguisties.
Devlin, J., Uesato, J., Singh, R., and Kohli, P. (2017). Semantic code repair using neuro-symbolic transformation networks.
Drain, D., Wu, C., Svyatkovskiy, A., and Sundaresan, N. (2021). Generating bug-fixes using pretrained transformers, arXiv preprint S1KVv:2204.078966
Durieux, T., Madeiral, F., Martinez, M., and Abreu, R. (2019). Empirical review of java program repair tools: A large-scale experiment on 2,141 bugs and 23,551 repair attempts. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, page 302-313, New York, NY, USA. Association for Computing Machinery.
Edunov, S., Ott, M., Auli, M., and Grangier, D. (2018). Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Gao, Q., Zhang, H., Wang, J., Xiong. Y., Zhang, L., and Mei, H. (2015). Fixing recurring crash bugs via analyzing ql&a sites. In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 15, page 307-318. IEEE Press.
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M. (2019). Codesearchnet challenge: Evaluating the state semantic code search. arXiv preprint arXiv:1909.099 094366
Just, R., Jalali, D., and Ernst, M. (2014). Defects4j: A database of existing faults enable controlled testing studies for java programs. ISSTA 2014, page 437-440, New York, NY, USA. Association for Computing Machinery.
Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K. (2019). Pre-trained contextual embedding of source code.
Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The efficient transformer.
Kiyono, S., Suzuki, J., Mita, M., Mizumoto, T., and Inui, K. (2019). An empirical study of incorporating pseudo data into grammatical error correction. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Laguage Processing (EMNI.P-IJCNIPP)
Lewis, M., Liu, Y.. Goyal, N., Ghazvininejad, M.. Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020). BART: Denoising sequence-to-sed te-sueece pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 7871-7888 Online. Association for Computational Linguisties.
Lin, D., Koppel, J., Chen, A., and Solar-Lezama, A. (2017). Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge. SPLASH Companion 2017, page 55-56, New York, NY, USA. Association for Computing Machinery.
10

---

Under review as a conference paper at ICLR 2022
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C. B., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. (2021). Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664..
Lutellier, T., Pham, H. V., Pang, L., Li, Y. Wei, M., and Tan, L. (2020). Coconut: Combining context-aware neural translation models using ensemble for program repair. Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020, page 101-114, New York, NY, USA. Association for Computing Machinery.
Mesbah, A., Rice, A., Johnston, E.,, Glorioso, N., and Aftandilian, E. (2019). Deepdelta: Learning to repair compilation errors. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, page 925-936, New York, NY, USA. Association for Computing Machinery.
Qi. Z., Long, F,, Achour, S., and Rinard, M. (2015). An analysis of patch plausibility and correctness for generate-ande Symposium on Software Testing and Analysis, ISSTA 2015, page 24-36, New York. NY, USA. Association for Computing Machinery.
patch generation systems. In Proceedings of the 2015 International
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer.
Svyatkovskiy, A., Deng, K., Fu, S., and Sundaresan, N. (2020). Intellicode compose: Code generation using transformer. Proceedings the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.
Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., and Poshyvanyk, D. (2019). An empirical study on learning bug-fixing patches the wild via neural machine translation. ACM Trans. Softw. Eng, Methodol,, 28(4).
Wong, W. E., Debroy, V., Gao, R., and Li, Y. (2014). The dstar method for effective software fault localization, JEEE Transactions on Reliability, 63(1):290-308
Ye, H., Martinez, M., Durieux, T., and Monperrus, M. (2019). A comprehensive study of automatic program repair on the quixbugs benchmark. In 2019 JEEE Ist International Workshop on Intelligent Bug Fixing (IBF), pages 1-10. IEEE.


Reviews:
[{"review_id": "Z61BaoLoTPZ", "reviewer": "Reviewer_vC4c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Authors trained a model on reversed commit data to generate synthetic bugs.\n- Authors used the bug generation model to generate buggy code for training bug fixing model.\n- Authors also used generated buggy code to produce stack traces with errors. These stack traces were then used to train the bug fix model.\n- Authors used additional context such as import statements, class definitions, docstrings, etc. to further improve the bug fix model.\n- Authors show a significant improvement on QuixBugs benchmark both in total fixes found and decrease of false positives.\n\nWeaknesses:\n- Fine tuning pre-trained models for bug finding and fixing is not novel.\n- Using reversed commit data to generate bugs is not novel. It is possible that the authors' approach for such bug generation is better than what others have done, but this is not demonstrated.\n- It is possible that using DeepDev-py as an intermediate model and then using two phase training without and with traces yields superior results. It is however not evaluated which parts of this process are important/significant and which are not.\n- Limited comparison to other bug fixing tools.\n\nIf paper is accepted or in any other future versions, it would be good if authors compared their work with\n\"Self-Supervised Bug Detection and Repair\", M. Allamanis, H. Jackson-Flux, M. Brockschmidt. NeurIPS 2021\n\nMinor comments:\n- Page 1 Abstract - Possibly change \"timeout\" to \"execution time\"? \n- Page 4 - Replacing literals with placeholders might be removing fixes/bugs. Is this an issue? Please justify/comment.\n- Page 6: Figure 4: is capital S in Sax intentional?\n\n\nNot a critique to authors, but I am looking forward to some standardization of the bug detection/fixing benchmarks. Different authors/groups use different test sets/benchmarks making it hard to compare respective papers.\n", "summary_of_the_paper": "Authors propose couple approaches to train models for automatically debugging Python programs.\nAuthors create bug generation model that generates training set with automatically added bugs. The bugs are used in code repositories with tests to obtain stack traces of errors. These stack traces are used to train debugging models that include stack traces. Authors also propose including context such as import statements, class declarations, and docstrings to improve performance of the debugging model. Authors show significant improvement on QuixBugs benchmark.", "summary_of_the_review": "Authors present a bug fixing approach using tuned transformer models. Authors generate synthetic bugs by using reversed commit data. They additionally train the model on stack traces with errors caused by synthetic bugs. Authors propose using additional textual context to improve bug fixing. All this results in improved bug fixing model. I am uncertain if the contribution is significant enough for accepting the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NuQqEp_Esus", "reviewer": "Reviewer_K9aQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n=========\n1) Extensive Data used in training and learning the buggy function. The neural edit model is quite a refreshing idea.\n2) Good Result on QuixBug and outperform baseline by significant results\n3) Interesting idea of using the skeleton around the function. Employ information on the program (e.g. skeleton) in training and generating the patches. \n \n\nWeakness\n========\n1) I would argue that the novelty in the paper is quite lacking. The model that the authors employed is just a pre-trained model. Although the data collection are extensive, but it does not really show any novelty in patch generation.  \n\n2) I have some questions about using backtranslation on evaluating the model. I am quite confused about the forward edit and backward edit. I don't think it is defined anywhere on the paper. \nFurthermore, what is the point of using cross-entropy in evaluating the model? I do understand it shows that the model learns better using different edits and contexts, but it does not really reflect that it fixes the bugs. It will be better if you evaluate the model on the quix and report their results.   What do you use cross-entropy in evaluating the model?\n \n", "summary_of_the_paper": "This paper proposes an approach to fixing synthetic python bugs by using the backtranslation model. The idea is to employ four different\ntypes of datasets and employ BART to learn from them. They employ backtranslation in evaluating the trained model and outperform the previous baseline of over 50%. They further evaluate their model with test cases and achieve a fixed rate of over 90%", "summary_of_the_review": "Overall, it is my opinion that the data collection are extensive and covers many different scenarios, although it is quite confusing to read it due to many terms and different type of bugs. \nHowever, the novelty in the learning-based auto program repair is quite lacking. Furthermore, the evaluation is not suitable (e.g., uses cross-entropy to evaluate the model,  where you can actually report the difference model performance on quixbug). The paper will be much comprehensible if you employ similar metrics for all experiments and evaluations (e.g., top-1 fixes)\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5YYbQq2ig6n", "reviewer": "Reviewer_Rbe2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ The idea of data augmentation with back translation is interesting although not new.\n+ Code skeleton to augment the code context.\n\n\nCons:\n- QuixBugs benchmark is too small to evaluate the effectiveness of the model, while cross-entropy train losses are not useful in my opinion. \n- No comparison to state-of-the art models of the same kind such as PL-BART, CodeT5, etc.\n- Lack of discussion to recent work on self-supervised bug detection, e.g., Allamanis et al., 2021.\n\nQuestions:\n\n1. Back translation is interesting but when the model is trained on commit data, the neural bug patterns will be at most the same as the ones seen in commit (?), so I don’t know how back translation would be helpful. At least, one experiment is to compare a model trained and evaluated on <before, after> code pair with the same model augmented by neural bugs.\n\n2. Is there a reason why Python is the main focus? Recently, a lot of work (CodeBERT, PL-BART, CodeT5) were evaluated on the Patches of the Wild Java benchmark. Fine-tuning DeepDebug on that dataset seems straightforward to compare to those models. \n\nFor Python benchmarks, maybe PyPI benchmark and the random dataset therein (Allamanis et al., 2021) can be used.\n\n3. You state that “make use of the plenitude of relatively bug-free code on GitHub”. Do you have a rough estimate of the percentage of the Github code that is bug-free? Without good filtering heuristics (e.g., based on code review), I think that augmentation data would be very noisy as well.\n", "summary_of_the_paper": "In the paper, the authors propose DeepDebug, a Transformer-based approach for automated program debugging. DeepDebug is equipped with multiple sources of data such as stack traces and code skeleton, pre-trained and augmented with synthetic bugs.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "39ii48lv5mo", "reviewer": "Reviewer_sVvX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. This manuscript involves a complex set of 4-5 dataset for training and evaluation. The authors did a good job in presenting the datasets and their relations in Figure 1. But I still find a number of holes in the description of the datasets. \n- Section 4.2: How are the 100,000 Python repositories selected from GitHub? Is there any additional filtering beyond minimum # of stats (10)? It seems that on GitHub there should be more than 100,000 Python repositories that meet this criterion. Is the number 100,000 just a round number or is it the exact number? Please provide numbers in as precisely as possible. Is this dataset available for examination by the public or by reviewers anywhere? I would request the dataset to be made available to make it easier to review. I understand the size of the dataset will be large, but it can be open sourced as a list of URLs or pointers to original datasets on GitHub.\n- Section 4.2: The authors did not report the # of commits used to train the forward and backward edit models. Like I asked above, are these commit examples available for examination anywhere? Was the human-written commit messages used in anyway? How were the commits split into the train / dev / test subset? During the training of the forward and backward edit models, what hyperparameters were tuned and how? What if a commit touches multiple Python methods and/or multiple Python files? Were the commit treated as a single example or multiple examples? Without these details, it is hard for an independent research or lab to assess or reproduce the results. \n- Section 4.2: The authors mentioned in the first paragraph of Sect. 4.2 that the data is \"extremely noisy\". But they did not talk about the implication f the noisiness or whether they did anything to mitigate its effect.\n- Section 4.3: The authors mention that the model(s) generate syntax errors, often due to premature EOS tokens. But the authors didn't say how they dealt with these outputs with syntax errors. Did they simply discard them from the evaluation? Were they replaced with additional outputs by the model until a desired number of synthetic bugs without syntax error is reached?\n- Section 4.4.2: Of the 10K repositories, how many individual tests were used? How were the tests split into train / dev / test subsets? Like asked above, is the container (Docker container?) and the tests available in open source for examination and reproduction? \n- Section 4.4.3: How many such synthetic bugs with tests were there?\n- Section 4.5: It seems that including all this additional context can lead to very long inputs, which may not fit into the 1024-token window. What did the author do to handle such out-of-bound situations? Were different context lines ranked by priorities and only the highest-priority ones preserved during the truncation? If so, the authors should give more details for the ranking algorithm. They should also report statistics on the number of different kinds of context lines that were included. Or even better, the code skeleton dataset should be made available for examination and reproduction.\n- Section 4.5: When obtaining the code lines from methods other than the focal method, did the authors use a recursive approach? For example if the focal method uses another method, which in turn uses a third method, were both methods included in the skeleton? Could the method be from a different file in the same GitHub repository? From a PyPI package installed via setup.py? From a built-in Python module? These are important details without which it'd be hard to assess or reproduce the experiments.\n\n2. The concept of \"false positive\" is mentioned in Section 2.2 and Table 3. However, to me the author didn't define false positives clearly. Does \"false positive\" mean fixes that are only test suite adequate, but not genuine fixes? In the context of the QuixBugs benchmark, how did the authors determine whether a fix is a false positive? \n\n3. Questions about Table 3:\n- Why not report results from DeepDebug trained without backtranlation augmentation as an ablation experiment? This would help quantify the benefit of backtranslation.\n- To quantify the hit rate of the DeepDebug model, please report the # of bugs fixes and other metrics with fewer than 100 random samples (e.g., 10, 50). One minute is still very long for online use cases (e.g., imagine this being integrated into a code editor or IDE). It would be informative to report accuracy numbers under smaller # of attempts and shorter latency budgets.\n\n4. Section 5.1: 2nd sentence of the 2nd paragraph: \"Surprisingly, the backtranslation model actually performs worse on reversed\ncommit data, which is consistent with its bias towards constructively fixing buggy functions.\" seems written incorrectly.\n\n5. General questions for discussion: The QuixBugs benchmark that the DeepDebug model of this paper is evaluated on contains mostly self-contained programs that do not interact with other modules or libraries in complex ways. This significantly reduces the difficulty of bug fixing. Real-life programs generally are organized into multiple modules (code files). They interact not only among themselves but also with other libraries (e.g., packages from PyPI) in complex ways. Some discussion should be added to describe how these challenges might or might not be solvable in the future in the DeepDebug approach. ", "summary_of_the_paper": "This manuscript describes DeepDebug a transformer-based mode that performs code repair on Python methods. Specifically, the model is pretrained on 200k Python code repositories and fine-tuned on a number of datasets consisting of bug fix commits and augmentation through synthetic bugs. The model takes Python methods with bugs as the input and outputs potentially bug free versions of the method. The main contribution of the paper, if held up through the questions I ask below, include:\n- An innovative application of the backtranslation approach on the code repair domain.\n- Setting a new state-of-the-art (SOTA) performance for code repair on the Python problems from the QuixBugs benchmark, beating previous SOTA (e.g., CoCoNut) not only in the percentage of bugs fixed, but also in using fewer attempts and significantly less latency budget.\n\nWhile these contributions are exciting and potentially pushing the boundary of deep learning-based code repair, there are some major issues  with how the methodology is described, how the results are presented, and how the implications of the results are discussed. These critique points from me are listed in the \"Main Review\" section below.", "summary_of_the_review": "In summary, this paper presents some potentially state-of-the-art performance on neural code repair. But there are a large number of nontrivial details that are left out from the methods, results and discussion parts of the manuscript, which I respectfully request the authors to address. If the authors can satisfactorily address these questions and concerns, the manuscript will be a strong candidate for publication in this conference in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Metareview:

This paper proposes a transformer-based automated program debugger, called DeepDebug. All reviewers agree that the addressed problem is interesting. However, there was a consensus among reviewers regarding concerns in terms of the novelty and the lack of comparisons with other. 
In general, all reviewers consistently gave a score that is below the acceptance threshold. This paper is of interest to the ICLR audience, but current form is not ready for acceptance.  

Summary Of Reasons To Publish:

-Good Result on QuixBug

-Synthetic bug generation

Summary Of Suggested Revisions

-Comparisons in/with other datasets/tools
Decision: Reject