TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

Berkay Berabi 1 2 Jingxuan He 1 Veselin Raychev 1 2 Martin Vechev 1

Abstract
The problem of ﬁxing errors in programs has attracted substantial interest over the years. The
key challenge for building an effective code ﬁxing tool is to capture a wide range of errors andmeanwhile maintain high accuracy. In this paper, we address this challenge and present a newlearning-based system, called TFix. TFix works
directly on program text and phrases the problem of code ﬁxing as a text-to-text task. In turn,this enables it to leverage a powerful Transformer
based model pre-trained on natural language andﬁne-tuned to generate code ﬁxes (via a large, highquality dataset obtained from GitHub commits).
TFix is not speciﬁc to a particular programming
language or class of defects and, in fact, improved
its precision by simultaneously ﬁne-tuning on 52
different error types reported by a popular static
analyzer. Our evaluation on a massive dataset of
JavaScript programs shows that TFix is practicallyeffective: it is able to synthesize code that ﬁxes
the error in ∼67 percent of cases and signiﬁcantly
outperforms existing learning-based approaches.

1. Introduction
The high complexity and large size of modern code repositories have led to a substantial amount of coding errors, aserious obstacle to programmer productivity. While powerful static analysis tools can detect errors, a signiﬁcantamount of manual effort is still exerted on examining thereports and trying to ﬁx these errors correctly. This situation indicates that a tool capable of automatically ﬁxing
real-world coding errors would be highly desirable.

such as variable misuses (Allamanis et al., 2018) and integer
type errors (Coker & Haﬁz, 2013), have varying root causes.
To correctly ﬁx these, it is necessary that the tool is able
to capture a range of different program behaviors. Indeed,existing learning-based models either only ﬁx speciﬁc kindsof errors (Vasic et al., 2019; Hellendoorn et al., 2020; Pradel& Sen, 2018; Cornu et al., 2015) or become very inaccurate(e.g., 25% accuracy) when extended to handling a morediverse, but still limited, set of errors (Dinella et al., 2020;
Chen et al., 2019; Lutellier et al., 2020).

This work: TFix To address this challenge, we present
a new learning-based tool, called TFix1 , that can accurately synthesize ﬁxes to a wide range of errors coveredby ESLint (esl, 2021), the most popular static analyzer for
JavaScript. TFix formulates the problem of ﬁxing coding
errors as a text-to-text prediction task. That is, given a coding error as text, TFix predicts a new text representing thecode that ﬁxes the error. Such a formulation beneﬁts TFix
in three ways: (a) it allows TFix to capture various error
types in the same text format, (b) frees TFix from the burden of creating a complicated code representation such asgraphs (Allamanis et al., 2018; Dinella et al., 2020), andmost importantly, (c) enables TFix to utilize a powerfulmodel called Text-to-Text Transfer Transformer (T5) (Raffel et al., 2020). T5 has been demonstrated to generalize
across various natural language problems in the text-to-text
format and is thus well-suited for our task.

Department of Computer Science, ETH Zurich, Switzerland 2 Snyk, Switzerland.
Correspondence to: Berkay
Berabi <berkay.berabi@gmail.com>, Jingxuan He <jingxuan.he@inf.ethz.ch>.

Leveraging new types of knowledge transfer TFix exploits
two kinds of knowledge transfer that are not commonly
adopted. First, our T5 model is pre-trained on natural language and then ﬁne-tuned on the programming task of codeﬁxing. This enables knowledge transfer between naturaland programming languages (Feng et al., 2020). Further,
unlike prior works that train models for each individual
error type (Pradel & Sen, 2018; Bader et al., 2019), we
ﬁne-tune various error types together (Dinella et al., 2020;
Tarlow et al., 2020; Yasunaga & Liang, 2020), which enables knowledge transfer between error types and can beseen as a form of multi-task learning (ﬁxing each error typeis an individual task). These two features are key factors for

Proceedings of the 38 th International Conference on Machine
Learning, PMLR 139, 2021. Copyright 2021 by the author(s).

1
The code, trained model, and dataset can be found athttps://github.com/eth-sri/TFix.

An ideal tool should cover a wide range of errors to beneﬁtmost developers and come with high ﬁx accuracy. However,this is difﬁcult to achieve as different types of coding errors,
1

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

TFix to learn a deep understanding of program semantics
and thus achieve high accuracy for ﬁxing coding errors.
Fine-tuning with a large high-quality dataset To ﬁne-tune
T5, a large model with millions of parameters, we extract alarge-scale dataset of ∼100k aligned pairs of coding errors
and ﬁxes from 5.5 million GitHub commits. The extraction process invokes the static analyzer and goes through a
combination of greedy bipartite matching and Myers diffalgorithm (Myers, 1986), resulting in a dataset of signiﬁcantly higher-quality than existing datasets created by injecting artiﬁcial bugs (Vasic et al., 2019; Hellendoorn et al.,2020; Pradel & Sen, 2018) or applying tree transformation
rules (Bader et al., 2019; Dinella et al., 2020).
Evaluation of TFix TFix’s design is not speciﬁc to a particular programming language or error type as it represents
code as pure text. We conduct an extensive evaluation of
TFix on a massive dataset of JavaScript programs to ﬁx 52
error types detected by ESLint. The results demonstratethat TFix is practically effective: it generates code that ﬁxes
the error in 67% of cases. Moreover, we compare TFix to
SequenceR (Chen et al., 2019), CoCoNuT (Lutellier et al.,2020), and Hoppity (Dinella et al., 2020), state-of-the-art
machine learning tools for code error ﬁxing. TFix signiﬁcantly outperforms these tools. Our case studies show that
TFix can generate correct ﬁxes for complex coding errors.

2. Overview of TFix
In this section, we provide an overview of TFix on a motivating example. Figure 1 shows a JavaScript code snippetthat copies properties from one object to another. Besides
the desired properties, the code incorrectly copies the properties that belong to the prototype chain (pro, 2020). One
way to avoid copying the undesired properties is to call thefunction hasOwnProperty, which ensures that the copied
properties are directly in the object instance and are notinherited from the prototype chain.
The defect in Figure 1 can be found by existing error detectors such as ESLint. However, ESLint cannot automaticallyﬁx the defect and leaves the task to the developer. Our proposed tool, TFix, can synthesize the snippet in Figure 2 used
to replace the corresponding code in the input program tocorrectly ﬁxes the error. This is achieved by learning fromﬁxes made by human developers in open-source projects.
Next, we show step-by-step how TFix ﬁxes this error.
The pipeline of TFix is shown in Figure 3. TFix ﬁrst sends
the input program to ESLint, which identiﬁes a set of errors
in the program. For the example in Figure 1, ESLint detects
the defect and yields the following information:line 670 error: guard-for-in . error message: the body ofa for-in should be wrapped in an if statement to filter

if (value != null && fieldData.type != null) {var type = null;for (var typeEntry in types) {var typeNames = types[typeEntry];
if (typeNames.indexOf(fieldData.type) >= 0) {type = typeEntry;

Figure 1. An example code snippet with an error
var type = null;for(var typeEntry in types) {if(!types.hasOwnProperty(typeEntry)) continue;
var typeNames = types[typeEnrty];

Figure 2. The output of TFix which ﬁxes the error

Input code

Error detector:
ESLintTFix code ﬁx:
T5 modelError-free code

Figure 3. TFix’s pipeline for ﬁxing code errors.

unwanted properties from prototype .

ESLint locates the error line (highlighted in blue in Figure 1)
and outputs the error type and the error message .
The way TFix proposes the correct ﬁx is by ﬁrst extracting
the error context consisting of the error line and the twoneighboring lines. Next, TFix represents all information
about the error as a single piece of text:
ﬁx error type error message error line : error context
Then TFix uses this text to query a machine learning model
called Text-to-Text Transfer Transformer (T5) (Raffel et al.,2020) which generates new text representing the code thatﬁxes the error, as shown in Figure 2. The output text is thesame as the error context, except for the inserted if-statement
calling hasOwnProperty, shown in green . Indeed, theerror is ﬁxed after the error context in the original programis replaced by the generated code.
Insights of TFix To generate this non-trivial code, ourT5 model has to understand the semantics of the erroneouscode and correctly apply hasOwnProperty, considering
JavaScript syntax and context information such as local variables. Towards that, we leverage a T5 model pre-trained ona large corpus of NLP tasks and then ﬁne-tune it for the speciﬁc task of ﬁxing coding errors. This stimulates knowledgetransfer between natural language and code: the pre-training
drives the model to comprehend natural language fragments
in code such as identiﬁer names and then the ﬁne-tuning
quickly adapts the model to code ﬁxing.
To enable ﬁne-tuning, we construct a new high-quality
dataset consisting of aligned pairs of errors and ﬁxes from
5.5 million GitHub commits (discussed in Section 3.3) –TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

the largest dataset that captures ﬁxes of coding errors weare aware of. Importantly, ﬁne-tuning is done jointly on
all 52 error types in the dataset, producing a single model.
We experimentally show that this single model outperforms
52 individual models trained for each error type. Indeed,knowledge transfer between error types is a key factor for
TFix’ ability to generate correct ﬁxes.

Effectiveness of TFix We note that although, in hindsight,
the design of TFix may seem simple, it was clearly not theobvious choice. Many recent works apply neural modelsof code for various tasks (Brody et al., 2020; Allamanis
et al., 2020; Wei et al., 2020; Alon et al., 2020) but thepractice of leveraging valuable knowledge from a model
pre-trained with natural language is not yet common (Fenget al., 2020). Further, most models for code ﬁxing only
deal with one speciﬁc error and do not beneﬁt from theknowledge transfer between error types (Hellendoorn et al.,2020; Allamanis et al., 2018; Vasic et al., 2019; Pradel &Sen, 2018). Some works (Dinella et al., 2020; Tarlow et al.,2020; Yasunaga & Liang, 2020) implicitly learn a single
model for multiple types of errors but do not investigate theeffectiveness of this design choice.
We ran SequenceR (Chen et al., 2019), CoCoNuT (Lutellier et al., 2020), and Hoppity (Dinella et al., 2020), threestate-of-the-art learning-based tools for code ﬁxing, on theexample in Figure 1. None could learn the non-trivial behavior of inserting a new statement and thus failed to generatea correct ﬁx. We also evaluated TFix against the three tools
and show that TFix’s accuracy is signiﬁcantly better. Finally, the techniques of TFix are not speciﬁc to a particularlanguage or error type and hence open up the possibility of
future work for other languages and types of errors.

3. The TFix Approach
In this section, we describe the approach behind TFix, andhow it is used to generate code ﬁxes.
3.1. Applying an External Error Detector
The input to TFix is a set of coding errors found by acode analysis tool which we call detector. In our case,detector parses the input program into an abstract syntax
tree (AST), then performs program analysis techniques todetect and report coding errors. Detecting different errors
requires various program analyses. As a simple example,
ﬁnding unused variables requires computing scopes of variables in an AST. TFix is modular as it does not dependon the complex inner logic of detector and can directly
beneﬁt from advances in bug ﬁnding. The inputs to TFix are
the error reports made by detector. Each report consistsof an error type, an error message, and one location.

More formally, given a program p with N lines [li ]N
i=1 ,detector identiﬁes a list of M errors E = [ei ]M
i=1 in p(if any). The code analyzer detector usually supports
multiple error types denoted by the set T . Each error e is atuple (lk , L, t, m) where lk is the k-th line in the program
p reported by detector for introducing the error, L =[lk−w , ..., lk−1 , lk , lk+1 , ..., lk+w ] is the error context, i.e.,lk and the lines surrounding lk with a window size of w,
t ∈ T is the error type, and m is an error message. We set
w = 1 in our evaluation. The input to TFix is the set oferrors E. TFix processes each error in E individually.
Instantiation with ESLint In our work, we instantiate
detector with ESLint – a static analyzer for JavaScript
covering a wide range of errors (esl, 2021). We focusedon ESLint due to its popularity: it is adopted by major
companies (com, 2021) and has 16 million weekly downloads (dow, 2021). ESLint allows conﬁguring the set T
of returned error types. We used its default conﬁguration,
which reports coding errors and best practices, but no formatting issues. ESLint internally includes a manually crafted
capability to ﬁx a limited set of stylistic issues concerning
whitespaces in code. These whitespace formatting issues
are disabled in the default ESLint conﬁguration and were
not included in TFix. There exist tools like ESLint for otherlanguages, such as Pylint for Python (pyl, 2021), which canbe used when extending TFix to other languages.
3.2. Generating Code Fixes with T5
The goal of TFix is to synthesize a ﬁx for a detected error.
We formulate this task as a text-to-text prediction, i.e., given
a code error as text, TFix generates text representing newcode that has the speciﬁc error ﬁxed. Formally, given anerror e = (lk , L, t, m), TFix represents it as a single pieceof text by ﬁlling the following template:text(e) = “ﬁx” t m lk “:” L

where “ﬁx” and “:” are raw strings, and is a space. Then,
TFix queries a text-to-text encoder-decoder model which
outputs L0 as text. L0 is used to replace the lines L in thecode to ﬁx the error e (assuming the ﬁx of e lies in L). Notethat TFix can be further improved by sampling from themodel multiple times until a ﬁx passes detector’s check.
We leave that as future work.
We leverage the Text-to-Text Transfer Transformer(T5) (Raffel et al., 2020) as our text-to-text model because
our formulation is in line with its design. T5 is a genericmodel that uniﬁes different tasks into the text-to-text format
and is pre-trained on NLP tasks. We discuss how to ﬁnetune T5 for the programming task of generating ﬁxes forcode errors in Section 3.3.
To deal with large vocabulary size and out-of-vocabulary

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer(OOV) tokens, our work represents the input and output
code text with Byte Pair Encoding (BPE) (Sennrich et al.,2016), as with (Karampatsis et al., 2020) and the T5 model.
Initialized with single characters, the BPE algorithm builds
a vocabulary of sub-word tokens by iteratively merging themost frequently co-occurring sub-words. BPE is well-suitedfor text-to-text tasks and can generalize better to new words
unseen at training time than copy mechanism (Chen et al.,2019; Gu et al., 2016) and names splitting based on common
naming styles (Lutellier et al., 2020; Allamanis et al., 2018;
Alon et al., 2020).
We note that our formulation allows TFix to capture moreerrors than existing works. Generating an lk0 to replace lk ,as done in (Lutellier et al., 2020; Chen et al., 2019), is notenough for ﬁxing many errors. This is because a correct ﬁx
usually involves modifying other lines in the context of lk .
Hoppity (Dinella et al., 2020) suffers from the classic issueof limited vocabulary (Karampatsis et al., 2020). When thedesired node value does not exist in the vocabulary, Hoppity
can neither express nor generate the desired ﬁx.
3.3. Fine-tuning T5 for Synthesizing Code Fixes
We now discuss our techniques for ﬁne-tuning the pretrained T5 model for the task of generating code ﬁxes.
Fine-tuning objectives We assume a ﬁne-tuning dataset
D = {(e, L0 )} consisting of d pairs of error e ={lk , L, t, m} and its corresponding ﬁx L0 proposed by human developers. The ﬁne-tuning objective is to minimize
the cross-entropy loss:
X X
L(D) =
log p(L0 | text(e))(1)
t0 ∈T (e,L0 )∈D
t=t0

The teacher forcing algorithm (Williams & Zipser, 1989) isused during ﬁne-tuning.
Fine-tuning all error types together Note that our loss
function in Equation (1) sums over all error types in T ,i.e., we ﬁne-tune all error types together, which can also beviewed as a form of multi-task learning that signiﬁcantly
enlarges the dataset and exploits the knowledge transfer between error types. We show in Section 4 that this techniquesigniﬁcantly increases TFix’s accuracy. A future work item
is to ﬁne-tune multiple languages together to obtain a multilingual code ﬁxing model and beneﬁt from the knowledgetransfer between languages (Zügner et al., 2021).
3.4. Obtaining a Fine-tuning Dataset
To obtain the dataset D, we analyze a large volume of commits C = {(p, p0 )} from GitHub where p and p0 are the two
versions of the program before and after the commit, respectively. A challenge in obtaining a high-quality dataset is to

Algorithm 1 Procedure for extracting a ﬁne-tuning dataset.
Input : C = {(p, p0 )}, a dataset of GitHub commits.
Output : D = {(e, L0 )}, a ﬁne-tuning dataset.
1: D = emptyset()
2: for (p, p0 ) in C do
3:
E = detector(p) ; E 0 = detector(p0 )4:
if |E| > |E 0 | then
5:
Eﬁxed = findFixedErrors(E, E 0 )6:for e in Eﬁxed do
7:
L0 = computeFix(e, p, p0 )8:D.add((e, L0 ))9: return clean(D)separate error ﬁx commits from the many non-relevant commits that remove an error by completely deleting code orother means, and to extract the parts of code that correspondto errors and ﬁxes in the error ﬁx commits.
We present the data extraction and cleaning procedure of
TFix in Algorithm 1. The algorithm starts with an empty set
D (Line 1), iterates over the input commits C (Line 2), andruns detector to obtain E and E 0 on the pair of ﬁles p and
p0 in each commit. Then it checks if the number of errors
in E is larger than that in E 0 . If so, it is very likely that p0
ﬁxes some errors in p. Therefore, the commit is considered
to contain error ﬁxes. We note that we found this criterion
for determining an error ﬁx commit to be signiﬁcantly moreaccurate in practice than previous approaches based on keywords in the commit message (Lutellier et al., 2020) or thenumber of tree edits (Dinella et al., 2020) as it leverages theerror detector in the decision process.
TFix then calls the findFixedErrors function to identify
a set of ﬁxed errors Eﬁxed ⊆ E by comparing E and E 0 . To
achieve this, we leverage a bipartite matching algorithm on
the errors in the sets E and E 0 . For each ﬁxed error e ∈Eﬁxed , TFix invokes the computeFix function to extract
the target ﬁx L0 in p0 and to obtain a sample (e, L0 ) to be
added to the dataset D (Line 6 to 8). Finally, clean (Line 9)
removes noisy samples that contain misaligned error ﬁxes.
In Appendix B, we include an illustrative example of running Algorithm 1. Next we discuss findFixedErrors,
computeFix and clean in detail.
Finding ﬁxed errors with bipartite matching The
findFixedErrors function ﬁrst invokes a greedy bipartite

matching procedure between E and E 0 to identify a set oferrors Eunﬁxed ⊆ E that remain unﬁxed after the commit.
To compute the bipartite matching, we iterate all pairs oferrors e = (lk , L, t, m) ∈ E and e0 = (lk0 , L0 , t0 , m0 ) ∈ E 0 ,and accept (e, e0 ) as a match if t = t0 , m = m0 and the normalized Levenshtein edit distance between the lines of codeat lk and lk0 is small (we used a threshold 0.3). Intuitively,

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer
Table 1. The number of samples and the accuracy on exact match (TFix, T5-large-no-pre-train, and T5-large-per-type) for each error type.
Error typeno-new-symbol
no-compare-neg-zero
no-ex-assign
for-direction
no-unsafe-ﬁnallyuse-isnanno-class-assign
no-dupe-class-members
no-func-assign
no-empty-pattern
no-unused-labels
no-duplicate-casegetter-returnno-sparse-arrays
no-const-assign
no-global-assign
no-new-wrappers
no-this-before-superno-unsafe-negation
require-yieldno-extend-native
no-new-object
no-caller
constructor-supervalid-typeof
no-self-assign#SamplesTFix(T5-large)

T5-large-nopre-train

T5-large-pertype

10
13
40
50
63
71
111
117
147
178
187
195
203
237
277
318
360
413
423
429
443
446
449
464
539
610

100.0
0.0
25.0
40.0
42.9
37.5
41.7
8.3
46.7
27.8
52.6
65.0
52.4
25.0
35.7
59.4
27.8
47.6
72.1
72.1
31.1
71.1
20.0
59.6
51.9
34.4

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
5.6
5.3
5.0
33.3
0.0
3.6
9.4
2.8
9.5
7.0
14.0
0.0
6.7
4.4
10.6
7.4
4.9

0.0
0.0
0.0
40.0
28.6
25.0
25.0
8.3
40.0
16.7
15.8
55.0
47.6
20.8
17.9
37.5
19.4
26.2
60.5
39.5
13.3
53.3
20.0
63.8
37.0
37.7

Error type

#SamplesTFix(T5-large)

T5-large-nopre-train

T5-large-pertype

no-extra-bind
no-case-declarations
no-fallthroughno-inner-declarations
no-array-constructor
no-constant-condition
generator-star-spacing
no-extra-boolean-castno-cond-assign
no-process-exit
no-empty
no-dupe-keys
prefer-spreadno-useless-escapeno-console
guard-for-in
no-throw-literal
no-debugger
prefer-rest-params
no-unreachable
no-extra-semino-redeclarecomma-styleno-unused-vars
no-undef
no-invalid-this

684
725
743
831
980
1284
1396
1458
1512
1514
2063
2181
2496
2923
3067
3232
4075
4164
4582
4727
5973
6381
6398
7768
10, 637
16, 217

71.0
58.9
76.0
38.1
86.7
51.2
67.9
54.1
47.4
32.9
27.1
53.4
46.0
35.2
73.6
41.7
72.1
94.5
35.9
63.8
82.6
49.5
46.2
51.9
22.4
37.7

21.7
0.0
4.0
3.6
10.2
12.4
23.6
2.7
11.2
10.5
6.3
1.8
12.8
0.0
4.9
2.8
11.5
15.1
8.9
16.5
23.6
2.3
7.0
1.8
0.9
5.2

55.1
47.9
62.7
23.8
56.1
36.4
61.4
46.6
33.6
20.4
15.5
48.4
34.4
11.9
69.4
28.4
68.9
89.4
22.0
58.4
76.1
45.4
38.8
47.0
16.5
25.2

Sum: 104, 804

Avg: 49.3

Avg: 6.7

Avg: 36.3

this means e and e0 are very likely to be the same error, so
e remains unﬁxed after the commit. After the iteration ﬁnishes, all matched errors in E form the set Eunﬁxed . It could
happen rarely that an error is matched more than once, inwhich case we simply discarded the commit. At the end, theset of ﬁxed errors can be obtained by Eﬁxed = E − Eunﬁxed .

be very complicated. The reasoning is that in the ﬁlteredsamples, the errors are ﬁxed by the commit, but the target ﬁx
cannot be conﬁdently computed to obtain a clean training
sample. It is intriguing to check if TFix can ﬁx even theseerrors that have longer or unaligned ﬁxes, so we used the
unﬁltered errors as an additional test set in our evaluation.

Computing target ﬁx To compute the target ﬁx L0 withcomputeFix, we ﬁrst leverage the Myers diff algorithm(Myers, 1986) to compute a series of edit operations, which
can be used to transform p into p0 . Each edit operation
inserts or deletes a piece of text (up to one line) from theprogram. We apply the edit operations and meanwhile track
how lk is shifted in the program. We locate lk0 at the ﬁnal
position of lk after all edit operations are performed. Finally,
L0 is obtained by taking the context of lk0 .

4. Evaluation

Note that Lk0 may contain a newly added line and become
badly aligned with Lk due to the ﬁxed window size. For
these cases, we apply a heuristic line-based search in thesurrounding of lk and lk0 . As a result, Lk and Lk0 can beextended by few lines.
Cleaning the dataset To obtain a high-quality development dataset for our T5 model, we perform a cleaning stepthat removes potential noisy samples from the dataset D.
We keep only samples where less than six edit operations
are needed to obtain p0 from p. Note that since each edit operation can change an entire line, the samples in D can still

In this section, we present our extensive evaluation of TFix
and show its effectiveness in ﬁxing real-world errors.
4.1. Experimental Setup
We ﬁrst present the details of our experimental setup.
Dataset We ran Algorithm 1 on 5.5 million commits obtained from the top 500k GitHub public repositories basedon the number of stars and extracted a dataset of ﬁxes for 52
error types detected by ESLint. We found that there were
too many samples for some error types, which would makeour dataset heavily skewed. Therefore, we performed random downsampling for those error types. Our ﬁnal dataset
consists of 104, 804 samples. Detailed statistics are shownin Table 1, and the description for each error type can befound in (esl, 2021). To a create train-test split, we randomly
selected 10% of the samples for each error type as the testset (we call it clean test). The remaining was further split
into 90% for ﬁne-tuning and 10% for validation.

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

To measure the dataset quality, we manually inspected
100 samples randomly drawn from the whole dataset andchecked if they are indeed error ﬁxes. We found that 96 ofthem are true error ﬁxes and the true error-ﬁx rates lie in theinterval [90.1, 98.9] with a conﬁdence level of 95%, meaning that our data construction process yielded a high-quality
dataset with a tiny amount of noise.
Besides clean test, we test how TFix generalizes to errors
where the original ﬁx was complex or noisy and therefore
dropped in our data collection procedure. To this end, we
assembled another test set called random test consistingof all ﬁxable errors in the GitHub commits except for theones used for ﬁne-tuning and validation. This is done byconsidering all the errors in D before Line 9 of Algorithm 1and excluding the ones used for ﬁne-tuning and validation.
random test consists of 243, 054 samples.
We made great efforts to remove duplicates in our dataset.
For the GitHub repositories, we removed duplicate ﬁles thatparse to the same abstract syntax tree and further applieda deduplication process on the commit level: for each ﬁle,we detected and discarded the commits that exactly repeat
the changes in other commits. For instance, if there arecommits A, B and C, where C performs the changes of A
and B together, we drop C even if it might contain changes
in other ﬁles. We also ensured that there are no identicalsamples in the ﬁnal dataset.
Metrics We propose two metrics for measuring the accuracy of TFix. Exact match considers a prediction to becorrect if and only if the ﬁx perfectly matches the humanﬁx done in the commit. Note that this metric presents alower bound on TFix’s precision as an error can be ﬁxed in
multiple ways, and TFix may propose a correct ﬁx different
from the human ﬁx. Error removal counts a prediction ascorrect if the error is removed and no new error is introduced
after the erroneous code is replaced by the prediction. The
average accuracy is computed by averaging the accuracy pererror type. Exact match is a more strict metric than error
removal as it requires that not only the error is removed but
also the ﬁx is the same as the one in the real commit.
We report exact match and error removal accuracy for clean
test. For random test, we only report error removal, as we
do not have aligned human ﬁxes for the error inputs.
Hyperparameters and ﬁne-tuning details We chose T5large as our model for TFix and implemented it with thetransformers library (Wolf et al., 2020). In total, the modelhas 770 million parameters. For details on the architecture, please refer to (Raffel et al., 2020). We initialized
the weights with the pre-trained model (t5l, 2021). For
ﬁne-tuning, we used Adam (Kingma & Ba, 2015) with thelearning rate initialized to 10−4 . We set warm-up iterations

Table 2. Accuracy of T5 variations on clean test and random test.
clean test
Model

TFix (T5-large)
T5-large-no-pre-train
T5-large-per-type
T5-baseT5-smallrandom test

Exact
match

Error
removalErrorremoval

49.3
6.7
36.3
48.5
39.2

67.8
48.1
52.0
68.6
67.7

52.5
27.4
34.2
52.5
54.2

to 500 and applied linear learning rate scheduling. TFix was
ﬁne-tuned on 8 GPUs (GeForce RTX 2080 Ti) with a batch
size of 32. The ﬁne-tuning ran for 30 epochs, which took3-4 days, and applied validation after each epoch. We notethat due to constraints on GPU resources, T5-large is thelargest T5 model we could run. We leave it as future work
item to run TFix with larger T5 models. During inference,
we used beam search with a beam size of ﬁve.
Model variants To investigate the effectiveness of ourdesign choices, we investigate the following T5 baselines:
T5-large-no-pre-train: we initialized a T5-large modelwith no natural language pre-training and trained it fromscratch with our dataset.
T5-large-per-type: we performed ﬁne-tuning and testing
separately for each error type.
T5-base & T5-small: we ﬁne-tuned two smaller T5 models.
T5-base and T5-small have 220 million and 60 million
parameters, respectively.
Other important conditions (e.g., dataset, optimizer, etc.)
were held the same for those models.
4.2. Accuracy on Fixing Coding Errors
We present the accuracy results in Table 2. For clean test,
TFix achieves 49.3% accuracy on exact match (accuracy
per error type is shown in Table 1) and 67.8% accuracy onerror removal. For random test, TFix removes the error in
52.5% of the cases. Next, we provide an ablation study of
TFix with each model variant.
Effect of pre-training with natural languages The accuracy of the T5-large-no-pre-train model trails that of the
pre-trained model. The interesting result for this experimentis the signiﬁcantly worse result of 6.7% on exact match.
This demonstrates that the natural language pre-training is
the key for TFix to generate correct ﬁxes, but also points
to an insight that relying only on code commits may cause
the model to learn how to remove errors in ways a humanwould not do it. Such problems are reported also in priorTFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

systems (Bader et al., 2019) where the proposed ﬁx is frequently not accepted by a human.

Table 3. Accuracy on exact match for three compared tools.
Tool

Effect of ﬁne-tuning all error types together We compare TFix and T5-large-per-type to quantify the effect ofﬁne-tuning all error types together. From Table 2, we cansee that TFix achieves a signiﬁcantly (>13%) higher accuracy than T5-large-per-type on all metrics. Table 1 showsthat TFix signiﬁcantly improves upon T5-large-per-type foralmost all error types. These results conﬁrm the importanceof ﬁne-tuning all error types together and the existence ofknowledge transfer between different types of errors. Intuitively, one instance of such knowledge transfer happens
when different types of errors target similar program statements. Fine-tuning all error types together increases the sizeof the samples containing the target statements, helps themodel learn better the semantics of those statements, andthus results in higher accuracy.
Effect of model size We compare TFix with T5-base andT5-small to investigate the effect of model size. For exact
match on clean test, TFix achieves 0.8% higher accuracy
than T5-base and 10.1% higher accuracy than T5-small.
Therefore, model size is an important factor from T5-small
to T5-base but becomes marginal from T5-base to T5-large.
For the error removal metric, none of the three models issigniﬁcantly better than the others. We pick T5-large as
the model for TFix mainly because T5-large achieves thehighest accuracy on exact match, meaning that it is the bestat synthesizing human-like ﬁxes.
4.3. Comparison with State-of-the-art Approaches
We compare TFix with three state-of-the-art approaches: SequenceR (Chen et al., 2019) and CoCoNuT (Lutellier et al.,2020) based on seq2seq learning, and Hoppity (Dinella et al.,2020) based on graph neural networks.
Comparing with SequenceR and CoCoNuT SequenceR is based on an LSTM encoder-decoder with copy
mechanism (Gu et al., 2016). CoCoNuT is based on a
convolutional encoder-decoder with global attention2 .
In the original papers, SequenceR and CoCoNuT use more
restricted datasets requiring that the error and the ﬁx areat the same line. Therefore, we extracted 42, 394 samples
satisfying this requirement from our dataset and split themin the same way as described in Section 4.1, resulting in a
new test set called restricted test. We compare their exact
match accuracy with TFix on both restricted test and clean
test. For each comparison, all three models were trained
2

We implemented the architecture in OpenNMT (Klein et al.,2017) as the authors conﬁrmed that the released source code currently needs an update and ﬁxing (coc, 2021).

restricted test

clean test

SequenceR

23.6

17.9

CoCoNuT

16.4

11.7

TFix

46.3

49.3

on the same dataset. We also gave the same information to
all three models, i.e., the input and output to these models
were all encoded in the way described in Section 3.2. To befair and measure the real learnability of the models, we set
a reasonable beam size of ﬁve to generate one ﬁx per error
and did not perform ensemble learning.
The results are presented in Table 3 showing that TFix
signiﬁcantly outperforms both SequenceR and CoCoNuT.
SequenceR performs much worse than TFix because the
LSTM encoder-decoder model is small and could only capture the behaviors of a limited set of errors. CoCoNuT
performs even worse than SequenceR. The original paper of
CoCoNuT learned ensembles and used a beam size of 1k to
generate 20k ﬁxes per error. The error is considered ﬁxed
as long as any of the 20k ﬁxes passes the error check. We
believe such a large candidate set is more important than thelearnability of the model to make CoCoNuT effective.
Comparing with Hoppity Hoppity represents a buggy
program as a graph and predicts graph edits with graphneural networks to ﬁx the bugs. We compare TFix with
Hoppity showing that TFix signiﬁcantly outperforms Hoppity in dataset quality, expressivity, and accuracy.
Dataset quality We ﬁrst measured the noise level of Hoppity’s dataset in the same way as we did for our dataset atthe start of Section 4.1. We manually inspected 100 samples randomly drawn from Hoppity’s OneDiff dataset. We
chose the OneDiff dataset as it was the main dataset usedin (Dinella et al., 2020). We found that 34 samples were
non-bug changes, including version changes, name changes,
string changes, etc. The true bug-ﬁx rate of the OneDiff
dataset lies in the interval [55.8, 75.2] with a conﬁdence
level of 95%, which is signiﬁcantly lower than ours.
We also checked if our dataset considers more complicatederror ﬁxes than Hoppity’s, i.e., if more graph edits areneeded to represent the samples in our dataset. To achievethis, we converted our dataset to Hoppity’s format. During
the conversion, we retained the relevant information in our
original dataset: we kept the error localization by extracting
the smallest AST subtree containing the error context andadded the error type as the root of the extracted subtrees.
We did not include the error messages because they are long
string values not ﬁt in Hoppity’s vocabulary (see the next

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer
constructor(location, parameterNameAndValues) {this.location = location;
this.parameterNameAndValues = parameterNameAndValues;

constructor(location, parameterNameAndValues) {super(location);this.parameterNameAndValues = parameterNameAndValues;

Figure 4. TFix ﬁxes a no-this-before-super error by converting an assignment into a call to the parent constructor.
break;
case typeof type === ’array’ && typeof type[0] === ’object’:if (isMethod) {break;
case Array.isArray(type) && typeof type[0] === ’object’:if (isMethod) {Figure 5. TFix ﬁxes a valid-typeof error by calling isArray function from the Array prototype.
scout.NumberField.parent.prototype._init.call(this, model);if (!this.decimalFormat instanceof scout.DecimalFormat) {this.decimalFormat = new scout.DecimalFormat(...

scout.NumberField.parent.prototype._init.call(this, model);if (!(this.decimalFormat instanceof scout.DecimalFormat)) {this.decimalFormat = new scout.DecimalFormat(...

Figure 6. TFix ﬁxes a no-unsafe-negation error by adding parantheses to change the order of the operations.

paragraph for more details). On average, >8 graph edits
were needed to represent our error ﬁxes, while Hoppity’s
datasets only contain up to 3 edits.
Given that our dataset contains signiﬁcantly less noise andmore complex error ﬁxes than Hoppity’s, we used ourdataset for quantitatively comparing the expressivity andaccuracy of TFix and Hoppity.
Expressivity For some graph edits, Hoppity predicts node
values from a vocabulary consisting of values from othernodes and values frequently seen in training. Hoppity’s
expressivity is limited when the desired value does not exist
in the vocabulary, i.e., it can neither express nor predict thedesired change (and eventually the desired ﬁx), but puts
a special UNK token. Out of the 36, 361 samples in the
OneDiff test set, 20, 491 samples involved edits with UNKs.
In fact, according to the repository (hop, 2020), Hoppity
deemed a ﬁx as correct when UNKs are predicted for out-ofvocabulary values. Therefore, Hoppity actually considered
an easier learning task than generating a complete ﬁx. We
followed this for Hoppity in our comparison, giving favor
to Hoppity. On the contrary, TFix does not have such alimitation in vocabulary and can express any ﬁx thanks to
the Byte Pair Encoding. After converting our dataset to Hoppity’s format, 1, 300 of the 10, 504 test samples had edits
with UNKs. TFix output correct ﬁxes for 393 of them, while
Hoppity output 61 correct ﬁxes. We note that adapting BPE
to Hoppity would require non-trivial changes of its learning
framework, which we do not consider in our comparison.
Accuracy We trained and tested Hoppity on our converted
dataset (using the same split) with the hyperparameters provided in their paper and code. The exact match accuracy of
Hoppity was only 7.9% (for generating non-complete ﬁxes
with UNKs), signiﬁcantly lower than TFix (49.3% accuracy
for predicting a complete ﬁx). This is non-surprising because even for the much simpler OneDiff dataset, Hoppity
only had 14.2% Top-1 accuracy. If we consider generating complete ﬁxes for Hoppity, its accuracy would be evenlower. Moreover, Hoppity’s accuracy drops signiﬁcantlywith an increasing number of required graph edits due tothe larger search space for the correct edit sequence: for
1 to 10 edits, its accuracy was 47.2%, 26.1%, 7.6%, 9.7%,
15.6%, 4.8%, 10.6%, 0.7%, 1.4%, 0.3%. For >10 edits, its
accuracy was 0%. TFix does not have such a limitation witha text-to-text format.
4.4. Case Studies
We present three case studies to show how TFix ﬁxes
coding errors. More cases can be found in Appendix
A. The ﬁrst case is shown in Figure 4. ESLint ﬁnds
a no-this-before-super error in the code on theleft-hand side because the programmer tried to access alocation ﬁeld of the child object without having initialized its parent class. TFix generates a correct ﬁx on theright, calling the constructor of the parent class where thelocation ﬁeld is initialized with the location variable.
The second case is shown in Figure 5 and is about an incorrect application of JavaScript’s typeof operator captured by
the valid-typeof error type. The programmer intended
to check if the variable type is an array via the typeof
operator. However, for arrays, the typeof operator justreturns ’object’, making the check inconclusive. TFix
successfully synthesizes a ﬁx that calls the correct function
isArrary from the Array prototype to perform the check.
We show the ﬁnal case in Figure 6.
It is abouta very common bug related to operator precedence.
With the if statement, the developer intended tocheck that this.decimalFormat is not an instance of
scout.DecimalFormat. However, the negation is actually performed on this.decimalFormat instead of
the result of the instanceof operation, resulting in a
no-unsafe-negation error. As a result, the if condition is always false. TFix adds parentheses to ensure theoperators are applied in the correct order.
These case studies show that TFix can generate correct ﬁxes

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

even in the presence of non-trivial JavaScript semantics(e.g., inheritance) and can produce complex changes suchas converting an assignment to a function call.

trained with artiﬁcially injected bugs that cannot capture
real bug distribution. Hoppity (Dinella et al., 2020) learns
graph changes to detect and ﬁx bugs. We showed that TFix
is superior to Hoppity in Section 4.3.

5. Related Work

Another line of research focuses on ﬁxing compilation errors for introductory programming assignments. DeepFix (Gupta et al., 2017) and sk p (Pu et al., 2016) leverage a
RNN model to translate programs with compilation errors tocompiled ones. Reinforcement learning (Gupta et al., 2019)and dynamic program embeddings (Wang et al., 2018) areused to improve the ﬁx rate. DrRepair (Yasunaga & Liang,2020) utilizes a graph neural network and a self-supervised
learning paradigm. Instead of compilation errors, TFix ﬁxes
bugs coming from a static analyzer. It is an interesting future research item to extend TFix to ﬁx compilation errors
where accurate error localization is often unavailable.

We now discuss works more closely related to ours.
Automated program repair Automated program repair(APR) has been a classic research topic in software engineering (Gazzola et al., 2019; Monperrus, 2018). Traditional
APR approaches require a speciﬁcation of correctness, usually in the form of a test suite or logic assertions. Given a
program violating the speciﬁcation, APR approaches output a program satisfying the speciﬁcation. The output canbe obtained by applying edit patterns on abstract syntax
trees (Rolim et al., 2018; Hua et al., 2018) or solving symbolic constraints (Mechtaev et al., 2016; Nguyen et al., 2013;
Xuan et al., 2017). Our work is signiﬁcantly different fromthis line of research. APR works typically suffer from overﬁtting issues (Ye et al., 2021) as the speciﬁcation is speciﬁc to individual programs and is often incomplete (Qiet al., 2015; Smith et al., 2015). Learning-based APR techniques (Long & Rinard, 2016) are often limited to small
and manually curated datasets (Sobreira et al., 2018) anddo not lead to signiﬁcant improvements in ﬁx rates. On thecontrary, TFix can generalize across programs by learningfrom a large dataset. Moreover, APR approaches require asigniﬁcant amount of effort on modeling program semantics
while TFix simply represents code as text.
A notable APR work with machine learning that was
hyped as a useful internal tool at Facebook (get, 2018) is
GetaFix (Bader et al., 2019). GetaFix is similar to TFix inthe way it learns from commits removing static analyzer
warnings. While neither the tool nor the training data of
GetaFix are available, their training data is much smaller,
no learning is done across bugs, and no natural language
pre-training is taken. Based on the amount of training data
and their low accuracy for relatively simple bugs, we believethat TFix is much more performant and useful in practice.
Learning-based code error ﬁxing Recent years witnessed an increasing interest in applying learning-based
methods for detecting and ﬁxing bugs. Graph neural networks (Allamanis et al., 2018), LSTM (Vasic et al., 2019),
and Transformer based models (Hellendoorn et al., 2020)
have been used on the task of detecting and ﬁxing VarMisuse errors. DeepBugs detects three speciﬁc types of bugs
for JavaScript by learning code embeddings and a binaryclassiﬁer for each bug type (Pradel & Sen, 2018). Unfortunately, they are fundamentally non-competitive to static
analysis tools for bug detection in terms of both accuracy
and popularity. The main reason is that their models are

Neural models of code Apart from the models for detecting and ﬁxing coding errors, neural models have been
proposed for other tasks, such as code editing (Brody et al.,2020; Yin et al., 2019), type inference (Allamanis et al.,2020; Wei et al., 2020), and code completion (Alon et al.,2020; Brockschmidt et al., 2019). Several works focus
on learning general purpose code embeddings (Alon et al.,2019; Sui et al., 2020; Wang & Su, 2020). All of thesemodels require complicated code representations, e.g., program trees or graphs, and none beneﬁt from pre-training
on natural language. Natural language fragments are usedin (Kanade et al., 2020; Feng et al., 2020) to train generalpurpose code embeddings for code. The authors of (Zügner
et al., 2021) learn a multilingual representation for code.
One possible future research item is to ﬁnd out how theabove models of code can be used in code error ﬁxing.

6. Conclusion
We presented a new learning-based system, called TFix, forautomatically ﬁxing coding errors. The key idea behind
TFix is to formulate the problem of synthesizing ﬁxes forcode errors as a text-to-text prediction. This enables TFix toleverage T5, a Transformer model pre-trained on NLP tasks
and to ﬁne-tune it for the task of generating code ﬁxes, ona high-quality dataset of errors and ﬁxes that we extracted
from millions of GitHub commits. The accuracy of ourT5 model was boosted further by considering various error
types together during the ﬁne-tuning process. Our extensive
evaluation shows that TFix is practically effective: among
the ﬁxes generated by TFix, about half perfectly match
human ﬁxes, and about two-thirds remove the original error.
We believe our work is an important step in transfer learningfor code. It opens up new directions, e.g., directly applying
large pre-trained models from NLP to programming tasks.

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

References
Getaﬁx: How facebook tools learn to ﬁx bugs automatically,
2018. URL https://engineering.fb.com/2
018/11/06/developer-tools/getafix-ho
w-facebook-tools-learn-to-fix-bugs-a
utomatically/.
Github – hoppity, 2020. URL https://github.com/AI-nstein/hoppity.
Copying objects in javascript, 2020. URL https://www.
digitalocean.com/community/tutorials/copying-objects-in-javascript/.
Github issues – coconut, 2021. URL https://github.com/lin-tan/CoCoNut-Artifact/issues/3.
Who’s using eslint?, 2021. URL https://eslint.o
rg/users.
eslint – npm, 2021. URL https://www.npmjs.com/
package/eslint.
List of rules supported by ESLint, 2021. URL https://eslint.org/docs/rules/.
Pylint – code analysis for python, 2021. URL https://www.pylint.org/.
T5-large pre-trained model, 2021. URL https://gith
ub.com/google-research/text-to-texttransfer-transformer.
Allamanis, M., Brockschmidt, M., and Khademi, M. Learning to represent programs with graphs. In ICLR, 2018.
URL https://openreview.net/forum?id=
BJOFETxR-.

Brockschmidt, M., Allamanis, M., Gaunt, A. L., and Polozov, O. Generative code modeling with graphs. In ICLR,
2019. URL https://openreview.net/for
um?id=Bke4KsA5FX.
Brody, S., Alon, U., and Yahav, E. A structural model forcontextual code changes. Proc. ACM Program. Lang., 4(OOPSLA):215:1–215:28, 2020. URL https://doi.
org/10.1145/3428283.
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.N., Poshyvanyk, D., and Monperrus, M. Sequencer:
Sequence-to-sequence learning for end-to-end program
repair. IEEE Transactions on Software Engineering,
2019.
Coker, Z. and Haﬁz, M. Program transformations to ﬁx C
integers. In ICSE, 2013. URL https://doi.org/
10.1109/ICSE.2013.6606625.
Cornu, B., Durieux, T., Seinturier, L., and Monperrus, M.
Npeﬁx: Automatic runtime repair of null pointer exceptions in java. CoRR, abs/1512.07423, 2015. URL
http://arxiv.org/abs/1512.07423.
Dinella, E., Dai, H., Li, Z., Naik, M., Song, L., and Wang,
K. Hoppity: Learning graph transformations to detectand ﬁx bugs in programs. In ICLR, 2020. URL https://openreview.net/forum?id=SJeqs6EFvB.
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong,
M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M.
Codebert: A pre-trained model for programming andnatural languages. In EMNLP Findings, 2020. URL
https://doi.org/10.18653/v1/2020.fin
dings-emnlp.139.

Allamanis, M., Barr, E. T., Ducousso, S., and Gao, Z. Typilus: neural type hints. In PLDI, 2020. URL https://doi.org/10.1145/3385412.3385997.

Gazzola, L., Micucci, D., and Mariani, L. Automatic software repair: A survey. IEEE Trans. Software Eng., 45(1):34–67, 2019. URL https://doi.org/10.1109/
TSE.2017.2755013.

Alon, U., Zilberstein, M., Levy, O., and Yahav, E. code2vec:
learning distributed representations of code. Proc. ACM
Program. Lang., 3(POPL):40:1–40:29, 2019. URL ht
tps://doi.org/10.1145/3290353.

Gu, J., Lu, Z., Li, H., and Li, V. O. K. Incorporating copying
mechanism in sequence-to-sequence learning. In ACL,
2016. URL https://doi.org/10.18653/v1/
p16-1154.

Alon, U., Sadaka, R., Levy, O., and Yahav, E. Structural
language models of code. In ICML, 2020. URL http://proceedings.mlr.press/v119/alon20a.html.

Gupta, R., Pal, S., Kanade, A., and Shevade, S. K. Deepﬁx:
Fixing common C language errors by deep learning. In
AAAI, 2017. URL http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14603.

Bader, J., Scott, A., Pradel, M., and Chandra, S. Getaﬁx:
learning to ﬁx bugs automatically. Proc. ACM Program.
Lang., 3(OOPSLA):159:1–159:27, 2019. URL https://doi.org/10.1145/3360585.

Gupta, R., Kanade, A., and Shevade, S. K. Deep reinforcement learning for programming language correction. In
AAAI, 2019. URL https://doi.org/10.1609/
aaai.v33i01.3301930.

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

Hellendoorn, V. J., Sutton, C., Singh, R., Maniatis, P., andBieber, D. Global relational models of source code. In
ICLR, 2020. URL https://openreview.net/forum?id=B1lnbRNtwr.

Pradel, M. and Sen, K. Deepbugs: a learning approach toname-based bug detection. Proc. ACM Program. Lang.,2(OOPSLA):147:1–147:25, 2018. URL https://doi.org/10.1145/3276517.

Hua, J., Zhang, M., Wang, K., and Khurshid, S. Towards
practical program repair with on-demand candidate generation. In ICSE, 2018. URL https://doi.org/10.1145/3180155.3180245.

Pu, Y., Narasimhan, K., Solar-Lezama, A., and Barzilay, R.
sk p: a neural program corrector for moocs. In Softwarefor Humanity, SPLASH 2016, 2016. URL https://doi.org/10.1145/2984043.2989222.

Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K.
Learning and evaluating contextual embedding of sourcecode. In ICML, 2020. URL http://proceedings.
mlr.press/v119/kanade20a.html.

Qi, Z., Long, F., Achour, S., and Rinard, M. C. An analysis
of patch plausibility and correctness for generate-andvalidate patch generation systems. In ISSTA, 2015. URL
https://doi.org/10.1145/2771783.2771
791.

Karampatsis, R., Babii, H., Robbes, R., Sutton, C., andJanes, A. Big code != big vocabulary: open-vocabularymodels for source code. In ICSE, 2020. URL https://doi.org/10.1145/3377811.3380342.
Kingma, D. P. and Ba, J. Adam: A method for stochastic
optimization. In ICLR, 2015. URL http://arxiv.
org/abs/1412.6980.
Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A. M.
Opennmt: Open-source toolkit for neural machine translation. In ACL System Demonstrations, 2017. URL
https://doi.org/10.18653/v1/P17-4012.
Long, F. and Rinard, M. Automatic patch generation bylearning correct code. In POPL, 2016. URL https://doi.org/10.1145/2837614.2837617.
Lutellier, T., Pham, H. V., Pang, L., Li, Y., Wei, M., and Tan,
L. Coconut: combining context-aware neural translation
models using ensemble for program repair. In ISSTA,
2020. URL https://doi.org/10.1145/3395
363.3397369.
Mechtaev, S., Yi, J., and Roychoudhury, A. Angelix: scalable multiline program patch synthesis via symbolic analysis. In ICSE, 2016. URL https://doi.org/10.1
145/2884781.2884807.
Monperrus, M. Automatic software repair: A bibliography.
ACM Comput. Surv., 51(1):17:1–17:24, 2018. URL ht
tps://doi.org/10.1145/3105906.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring
the limits of transfer learning with a uniﬁed text-to-texttransformer. JMLR, 21:140:1–140:67, 2020. URL http://jmlr.org/papers/v21/20-074.html.
Rolim, R., Soares, G., Gheyi, R., and D’Antoni, L.
Learning quick ﬁxes from code repositories. CoRR,
abs/1803.03806, 2018. URL http://arxiv.org/
abs/1803.03806.
Sennrich, R., Haddow, B., and Birch, A. Neural machinetranslation of rare words with subword units. In ACL,
2016. URL https://doi.org/10.18653/v1/
p16-1162.
Smith, E. K., Barr, E. T., Goues, C. L., and Brun, Y. Is
the cure worse than the disease? overﬁtting in automatedprogram repair. In FSE, 2015. URL https://doi.
org/10.1145/2786805.2786825.
Sobreira, V., Durieux, T., Madeiral, F., Monperrus, M.,
and de Almeida Maia, M. Dissection of a bug dataset:
Anatomy of 395 patches from defects4j. In SANER, 2018.
URL https://doi.org/10.1109/SANER.20
18.8330203.
Sui, Y., Cheng, X., Zhang, G., and Wang, H. Flow2vec:
value-ﬂow-based precise code embedding. Proc. ACM
Program. Lang., 4(OOPSLA):233:1–233:27, 2020. URL
https://doi.org/10.1145/3428301.

Myers, E. W. An O(ND) difference algorithm and its
variations. Algorithmica, 1(2):251–266, 1986. URL
https://doi.org/10.1007/BF01840446.

Tarlow, D., Moitra, S., Rice, A., Chen, Z., Manzagol, P.,
Sutton, C., and Aftandilian, E. Learning to ﬁx build errors
with graph2diff neural networks. In ICSE Workshops,
2020. URL https://doi.org/10.1145/3387
940.3392181.

Nguyen, H. D. T., Qi, D., Roychoudhury, A., and Chandra,
S. Semﬁx: program repair via semantic analysis. In ICSE,
2013. URL https://doi.org/10.1109/ICSE.2013.6606623.

Vasic, M., Kanade, A., Maniatis, P., Bieber, D., and Singh,
R. Neural program repair by jointly learning to localize
and repair. In ICLR, 2019. URL https://openrevi
ew.net/forum?id=ByloJ20qtm.

TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer

Wang, K. and Su, Z. Blended, precise semantic program
embeddings. In PLDI, 2020. URL https://doi.or
g/10.1145/3385412.3385999.
Wang, K., Singh, R., and Su, Z. Dynamic neural program
embeddings for program repair. In ICLR, 2018. URL
https://openreview.net/forum?id=BJuW
rGW0Z.
Wei, J., Goyal, M., Durrett, G., and Dillig, I. Lambdanet:
Probabilistic type inference using graph neural networks.
In ICLR, 2020. URL https://openreview.net/forum?id=Hkx6hANtwH.
Williams, R. J. and Zipser, D. A learning algorithm
for continually running fully recurrent neural networks.
Neural Comput., 1(2):270–280, 1989. URL https://doi.org/10.1162/neco.1989.1.2.270.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C.,
Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,
Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite,
Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M.,
Lhoest, Q., and Rush, A. M. Transformers: State-of-theart natural language processing. In EMNLP Demos, 2020.
URL https://doi.org/10.18653/v1/2020.emnlp-demos.6.
Xuan, J., Martinez, M., Demarco, F., Clement, M., Marcote,
S. R. L., Durieux, T., Berre, D. L., and Monperrus, M.
Nopol: Automatic repair of conditional statement bugs
in java programs. IEEE Trans. Software Eng., 43:34–55,
2017. URL https://doi.org/10.1109/TSE.
2016.2560811.
Yasunaga, M. and Liang, P. Graph-based, self-supervised
program repair from diagnostic feedback. In ICML, 2020.
URL http://proceedings.mlr.press/v119/yasunaga20a.html.
Ye, H., Martinez, M., Durieux, T., and Monperrus, M. A
comprehensive study of automatic program repair on thequixbugs benchmark. J. Syst. Softw., 171:110825, 2021.
URL https://doi.org/10.1016/j.jss.20
20.110825.
Yin, P., Neubig, G., Allamanis, M., Brockschmidt, M., andGaunt, A. L. Learning to represent edits. In ICLR, 2019.
URL https://openreview.net/forum?id=
BJl6AjC5F7.
Zügner, D., Kirschstein, T., Catasta, M., Leskovec, J., andGünnemann, S. Language-agnostic representation learning of source code from structure and context. In ICLR,
2021. URL https://openreview.net/for
um?id=Xh5eMZVONGF.

A. More Case Studies
In this section, we present eight additional case studies of
TFix. TFix exactly matches the human fixes in the commits
in four cases, removes the coding errors but does not perfectly match human fixes in two cases, and fails to generatea correct fix in the remaining two cases.

Exact matches The first case is shown in Figure 7, thedeveloper forgot to write the return keyword in a getter
function, causing a getter-return error in ESLint. As aresult, the function does not return anything even though it
can still run. TFix correctly inserts the missing return.
Figure 8 presents the second case. The developer declaredvariable value to be a constant type with the const keyword but then assigned a new value to it. This resulted
in a no-const-assign error reported by ESLint. TFix
changes const to let, successfully fixing the error. Notethat the fix line is different from the error line in this example, showing the importance of the error context.
In Figure 9, the developer directly compared counter to
NaN, which is a wrong way of checking if a variable is avalid value in JavaScript. The correct way is to use thelibrary function isNaN, as suggested by TFix.
In the fourth case, as shown in Figure 10, the developer
intended to check if this.debug is true but wrongly
wrote an assignment in the if condition, which is acommon programming mistake. Therefore, ESLint reported a no-cond-assign error. TFix fixes it by using
this.debug solely as the condition.
The above four cases show that TFix is capable of generating
human-like fixes for a wide range of errors.

Error removals Now we discuss two cases where TFix
synthesizes fixes that correctly remove the errors but are
different from human fixes. The first case is shown in Figure 11 where the developer used the throw statement on astring literal, causing a no-throw-literal error. Later,
the developer fixed the error by printing the error message tothe console. On the other hand, TFix proposes to construct
and throw an Error instance, which is also correct.
The second case is shown in Figure 12 and the code contains
a guard-for-in error for the same reason as discussed in
Section 2. While the developer fixed the error by converting
for-in to forEach, TFix proposes to add an if-check forthe property. Both fixes remove the error correctly.
The above two cases show that TFix can generate correct
fixes that are syntactically different from but functionallythe same as human fixes.

Failures Next, we describe two cases where TFix could
not generate a correct fix. In the first case, shown in Figure 13, the programmer assigned a value to itself, which
does not make sense from a programming perspective. ESLint denotes such errors with no-self-assign. In thecommit, the variable was assigned to the correct variable,
namely modified . TFix failed to generate a correct fix
due to its fixed windows size, i.e., the variable modified
is not in the error context. Increasing the window size canpotentially resolve this failure.
The second case is shown in Figure 14. The erroneous code
contains an empty catch block, and the developer fixed
it by simply printing the caught error to console. Unfortunately, TFix generates a fix identical to the incorrect codeand is unable to solve the problem. The reason might be thatsimilar samples do not often occur in our fine-tuning dataset.
Enlarging the fine-tuning dataset can help TFix handle moreerrors.

get [Symbol.iterator]() {this.__data__[Symbol.iterator]();}get [Symbol.iterator]() {return this.__data__[Symbol.iterator]();}Figure 7. TFix fixes a getter-return error by inserting the missing return keyword.
const { value =’’ } = e.detail || {};value = this.getValue(newValue);this._trigger(’onChange’, { detail: {value } });let { value =’’ } = e.detail || {};value = this.getValue(newValue);this._trigger(’onChange’, { detail: {value } });Figure 8. TFix fixes a no-const-assign error by changing the variable’s declaration type.
var randomCounter = false;
if (counter == NaN) {counter = Math.floor(Math.random * 10000);

var randomCounter = false;
if (isNaN(counter)) {counter = Math.floor(Math.random * 10000);

Figure 9. TFix fixes a use-isnan error by using the correct function supported by JavaScript.
var signal = JSON.parse(message.content);
if(this.debug = true)
console.log(message);var signal = JSON.parse(message.content)if(this.debug)
console.log(message);

Figure 10. TFix fixes a no-cond-assign error by correcting the condition.

Coding error:

Coding error:

if (destination == null) {throw "destination is invalid";}

applyCss(obj) {for (var key in obj) {this.element.style[key] = obj[key];

Human fix:

Human fix:

if (destination == null) {console.log("destination is invalid");}

applyCss(obj) {Object.keys(obj).forEach(key) => {this.element.style[key] = obj[key];

Output of TFix:

Output of TFix:

if (destination == null) {throw new Error("destination is invalid");}

applyCss(obj) {for (var key in obj) {if (!obj.elements.hasOwnProperty(key)) continue;
this.element.style[key] = obj[key];

Figure 11. TFix correctly fixes a no-throw-literal error by
constructing an Error instance while the human developer prints
the error message to console.

Figure 12. TFix correctly fixes a guard-for-in error by adding
an if-check while the human developer uses forEach.

Coding error:

Coding error:{} catch (e) {modified = modified;}}

Human fix:

Human fix:{}

} catch (e) {log.warn(error);}

Output of TFix:

Output of TFix:{} catch (e) {modified = modified_;this.modified = parseInt(false);}}

Figure 13. TFix generates irrelevant code as a fix.

Figure 14. TFix generates a fix identical to the erroneous code.

B. Running Data Extraction on an Example
In this section, we explain our data extraction pipeline in
Algorithm 1 in more detail by running it on a simple, synthesized commit. The process is similar in principle but more
complicated in detail for real-world commits.
Example commit with error fixing Figure 15 shows our
example commit (p, p0 ). From p to p0 , the commit removes
Line 6, adds Line 7, and adds a new return token at Line 9.
First, TFix runs detector to detect errors on p and p0 (Line
3 of Algorithm 1). For p, detector detects a set of threeerrors E = {e1 , e2 , e3 }:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class HumanPlayer extends Player {constructor(name) {this.name = name;this.health = 100;}
set health (health) { this.health = health; }+get health () { return this.health; }get name() {+return this.name;}damage(x) {if (x < 0) {throw "No negative damage";}this.health -= x;}}

Figure 15. An example commit fixing a getter-return error.

e1 : no-this-before-super at Line 3.
e2 : getter-return at Line 9.
e3 : no-throw-literal at Line 13.
e2 is fixed in the commit with the newly added return
token. Therefore, for p0 , detector identifies two errors
E = {e01 , e02 }:e01 : no-this-before-super at Line 3.
e02 : no-throw-literal at Line 13.
Since |E| > |E 0 | (Line 4 of Algorithm 1), we proceed theextraction procedure as the commit contains an error fix.
Finding fixed errors with bipartite matching Next
TFix calls the findFixedErrors function to identify theset of errors Efixed ⊆ E fixed in the commit (Line 5 ofAlgorithm 1). To achieve this, findFixedErrors first invokes the greedy bipartite matching procedure between E
and E 0 to find the set of unfixed errors Eunfixed ⊆ E. We
iterate all pairs of errors in E and E 0 to see if they are thesame error. Clearly, e1 = e01 and e3 = e02 . Therefore,
Eunfixed = {e1 , e3 } and Efixed = E − Eunfixed = {e2 }.
Computing target fix Then we compute the fix for e2
with the computeFix function (Line 7 of Algorithm 1). We
first leverage the Myers diff algorithm to obtain a series ofthree edit operations:
1. Delete a whole line (Line 6).
2. Insert a new line (Line 7).
3. Insert return at Line 9 after the second tab.
We perform the above edit operations starting from p andtrack how the error line lk of e2 (Line 9) shifts with each
edit operation to obtain the target fix line lk0 . The deletionat Line 6 pulls lk up by one line while the insertion at Line 7
pushes it down again, so we compute that lk0 stays at Line 9.
In the end, we obtain the following error context Lk and fix
context Lk0 as a sample in our dataset:get name() {this.name;}get name() {return this.name;}