Autorepairability: A New Software Quality
Characteristic
Pongpop Lapvikai* , Chaiyong Ragkhitwetsagul* , Morakot Choetkiertikul* , Yoshiki Higo†*

Faculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, Thailand†
Graduate School of Information Science and Technology, Osaka University, Suita, Osaka, Japan

Abstract—Currently, research on automated program repair(in short, APR) is actively being conducted. APR techniques
have been applied to many bugs in open-source software, but
the probability of a successful fix is not very high. The authors
consider that not only should APR techniques be developed,
but software systems should be developed so that bugs can beeasily fixed with APR techniques. In this paper, we propose
autorepairability, a new characteristic of software quality, thatshows how effective automated program repair techniques arefor a specific code fragment, file, or project. We also show anapproach to automatically measure autorepairability from thesource code of a target project, and present experimental results
on 1,282 Java method pairs. The use of autorepairability allows
many studies to be conducted. For example, research on thedevelopment process for developing software systems with highautorepairability and research on refactoring, which transforms
software with low autorepairability into software systems withhigh autorepairability, will be possible.
Index Terms—Automated Program Repair, Program Analysis,
Software Quality Model, Metrics Measurement

I. I NTRODUCTION
Automated program repair (in short, APR) is a technique toremove exposed bugs fully and automatically without humanintervention. APR has become a significant research topic in
the field of software engineering [1], [2]. APR techniques areoften only applied to some bugs in open-source projects by
the researchers who proposed them. Marginean et al. haveapplied their APR technique to Facebook software [3], butsuch examples are rare. Naito et al. claim that it is difficult toapply APR techniques to industrial projects [4].
At the moment, APR techniques can remove only a smallpart of bugs. Thus, if there is an index to show how effective
APR techniques are in their projects, developers can decide
whether they should use APR techniques in the projects or not.
Such an index allows developers to avoid the situation where
APR techniques hardly contribute to removing bugs that occur
in their project despite the cost to them to introduce the APR
techniques to the project.
ISO/IEC 25010 includes a quality model that comprehensively defines the product quality of systems and software. The
product quality model comprises eight quality characteristics:
Functional suitability, performance efficiency, compatibility,
usability, reliability, security, maintainability, and portability.
Maintainability is the characteristic that is related to fixing
bugs, and it is defined as degree of effectiveness and efficiency

with which a product or system can be modified by the intended
maintainers. This means that maintainability is targeted at
intended maintainers and is not specific to APR techniques.
Considering the fact that there are situations where it is easierfor developers to remove bugs but harder for APR techniques
to remove them and vice versa, maintainability is not sufficientto evaluate how well APR techniques work for a softwaresystem. Consequently, another characteristic specific to APRtechniques is desirable for evaluating a software system fromthe viewpoint of its compatibility with APR techniques.
In this paper, we introduce a new software characteristic,
autorepairability, which means how effective APR techniques
are in a specific project. We create an index to show autorepairability. If autorepairability of a target project can be easily
measured, it can be used to make a decision on the introductionof APR techniques.
The following are the main contributions of this paper.
1) We present a new notion in software quality, autorepairability, and show that the ability to measure autorepairability will provide us with some new research
directions.
2) We propose a simple way to measure autorepairability
and present experimental results on a set of programsfrom a previous study [5].
The replication package of our study can be found athttps://zenodo.org/records/10184827.
II. M OTIVATIONAs previously mentioned, autorepairability represents how
effective APR techniques are in a specific project. In order foran automation technology to be widely used, it must be known
in advance whether the automation technology is effective.
We would like to explain this concept using the following
metaphor. Consider a robot vacuum cleaner (in short, RVC) asa widely used automation technology. An RVC is a machinethat moves around automatically to clean. An RVC uses sensors to avoid obstacles and collects dust with brushes. One of
the most significant advantages of RVC is that it automaticallycleans up when you are out of the house. An RVC has a built-in
battery and is cordless. When the amount of charge gets low, it
returns to the charging station. Each manufacturer and model
has its own unique features, such as a floor mapping function,
smartphone connectivity, and resistance to step differences.
However, if there is a lot of furniture and objects on thefloor, an RVC cannot clean thoroughly. Depending on the©2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, orreuse of any copyrighted component of this work in other works.
To be published in the 31st IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2024), Rovaniemi, Finland.

manufacturer and model, there are also disadvantages suchas dust being left in the corners of the room, running into
furniture, not being able to return to the charging station andbeing stuck, and being vulnerable to step differences.
When we decide whether to install an RVC, we consider
whether there are many obstacles in the room. If there are too
many obstacles, we should forgo the installation of an RVC,
and if there are few obstacles, we will buy an RVC.
In this way, it is crucial to accurately predict whether APR
techniques will work well before it is applied. At this moment,
we already know how well bugs in a project can be localized
affect autorepairability. We fill the gap by studying what
elements in a software source code affect its autorepairability.
The authors believe that the widespread use of autorepairability will lead to new research being conducted. Research on measuring autorepairability automatically and research on the development process to develop software withhigh autorepairability will be conducted. Moreover, if wetake into account the fact that some people refrain fromputting things on the floor so that RVC can work better,
we can imagine that developers conduct refactorings on theirprojects so that APR techniques work more effectively. In
this context, autorepairability is used as an indicator to check
whether conducting refactorings are functional or not from theviewpoint of APR techniques.
III. M EASURING AUTOREPAIRABILITY
Herein, we propose a simple technique to measure autorepairability of a given project. The key idea is to generatea large number of artificial bugs for the given project andmeasure how well an APR technique can fix such artificial
bugs. We use mutation testing techniques to generate artificial
bugs [6], [7]. The inputs to this technique include (1) sourcecode SP and test cases TP of a target project P , (2) a mutation
testing technique M U , and (3) an APR tool A.
This technique includes the following three steps to measureautorepairability.
Step-1: This technique generates mutants of source code
SP by using a mutation testing technique M U . Eachmutant µ ∈ M is a source code that is very similar to
the given source code. A large number of techniques thatgenerate mutants have been proposed [6].
• Step-2: The technique applies an APR tool A to eachof the generated mutants that comes with a set of test
cases TP . In this technique, each mutant µ is regarded asincluding a different bug as well as in mutation testing.
The generated fix from A that passes all the tests in TP iscalled a solution s. Set S represents all the solutions that
A can generate over all the mutants in M . If a mutant
passes all test cases in TP , A is not applied to the mutant.
• Step-3: The technique calculates the ratio of the number of generated solutions |S| per the number of mutants |M |, i.e., the AR-ability(SP , TP , M, A) score. The•

Fig. 1: Steps in our experimentcalculation of AR-ability(SP , TP , M U, A) is shown inEquation III.
AR-ability(SP , TP , M U, A) =

|S|
|M |

(1)

By using the AR-ability(SP , TP , M U, A) score, we cancompare the autorepairability of different programs.
IV. M ETHODOLOGY
We perform an experiment to measure the autorepairabilityof Java programs with functional equivalence. The purpose ofthis experiment is to show that even methods with equivalentfunctionality have different autorepairability. In other words,
to show that some implementations are easy to fix bugs with
APR techniques, while others are not. We also aim to reveal
what kinds of program elements affect autorepairability.
Thus, we ask the following two research questions.
• RQ1: What are the autorepairability scores of Javamethods with functional equivalence?
• RQ2: What kinds of program elements affect autorepairability?
Figure 1 shows the overall process of the experiment. We
explain each part in detail below.
A. Dataset
We choose a set of 1,342 functionally equivalent Javamethod pairs from the Higo et al. dataset [5]. Each method
pair shares the same functionality while having distinct code
structures. The dataset also comes with unit test cases toensure their functional equivalence. Thus, the two methods
in each method pair pass the unit tests of each other. An
example of a functionally equivalent method pair is shownin Figure 2. We can see that the compareByteArrays andequals methods similarly check two arrays whether are thesame or not. However, their implementations are different.
B. Experimental Procedure
We adopt the idea of PIT [8] to generate mutants of the
1,342 functionally equivalent Java method pairs. However,mutation testing tools including PIT change bytecode directly
instead of the Java source code. Thus, we developed a toolthat generates mutants of the source code of a given program
based on PIT. Currently, our tool includes the mutators of‘OLD DEFAULT’ of PIT [7] since they are the most basic.
We pick the method equals as an example. An example
mutant from the equals method is displayed in Figure 3 anda list of all the mutants generated from the equals method isboolean compareByteArrays ( byte [] a , int aOffset , byte []b , int bOffset , int length ) {if (( a. length < aOffset + length ) || ( b . length < bOffset+ length )) {return false ;}for ( int i =0; i < length ; i ++) {if (a[ aOffset + i ] != b [ bOffset + i ]) {return false ;}}return true ;}boolean equals ( byte [] a , int i , byte [] b , int j , int n ) {if (a. length < i + n || b . length < j + n )return false ;while (--n >= 0)
if ( a [ i ++] != b [ j ++])
return false ;return true ;}Fig. 2: An example method pair from the Higo et al. dataset [5]boolean __target__ ( byte [] a , int i , byte [] b , int j , int n ) {if (a. length >= i + n || b . length < j + n )return false ;while (--n >= 0) if ( a [ i ++] != b [ j ++])
return false ;return true ;}will be produced in each generation. A hundred seed values
were used to run kGenProg. In this experiment, TP , M U ,and A of AR-ability(SP , TP , M U, A) is common to all the
programs. Thus, we represent autorepairability of them as
AR-ability(SP ) for short.
Moreover, to understand more about what code structure
affects the autorepairability of a program, the first author(who had more than 5 years of programming) performed amanual check on a sample of the method pairs. The selection
criteria for selecting the sample are as follows. First, the valueof the successful rate, i.e., getting a solution, after applying
kGenProg of both program methods must differ by at least
50%. This is to see the differences between the two methods ina pair, which one has higher autorepairability and vice versa.
Second, the number of mutant programs generated for eachprogram method is more than or equal to 10. This is to haveenough examples for us to observe and draw conclusions.
V. R ESULTS
A. Answering RQ1

Fig. 3: A mutant of the equals method shown in Figure 2

shown in Table I. Looking at the mutation ID 1, we can seethat the Negate Conditionals is applied to line number 5 andchanges the code from a.length < i + n to a.length >=i + n. The other operations (ID 2–12) were also applied tocreate 11 more mutants. In this experiment, we regarded themutated code in each mutant are buggy code and had to be
fixed by an APR tool.
Then, we ran kGenProg [9] to generate solutions for eachmutant. kGenProg is an APR tool of the generation-andvalidation strategy as well as jGenProg [10]. The reason
why we used kGenProg is its ease of use. In this experiment, we ran kGenProg with the –max-generation of 10
and –headcount of 10. The former option means how many
generations kGenProg will turn the processing loop. The
latter option means how many variant programs kGenProg

After finishing running kGenProg, we found that it could
generate solutions for only 1,282 pairs. The summary of theresult is shown in Table II. The lowest number of mutants
is 1 while the highest is 100, with an average of 10.04
mutants per method. The lowest number of solutions that
kGenProg could generate for one method is 0 while thehighest is 56, with an average of 3.39 solutions per method.
After we computed the AR-ability(SP ) scores, we found thatthe average AR-ability(SP ) score is 0.45, while the maximum
score is 1.0 and the minimum score is 0.0. The distributions of
the number of mutants and solutions are shown in Figure 4aand the distribution of the AR-ability(SP ) scores is shownin Figure 4b.
TABLE II: Result of running kGenProg on 1,282 pairs
Value

Mutants

SolutionsAR-ability(SP )Min
Max
Average

1
100
10.04

0
56
3.39

0.0
1.0
0.45

TABLE I: The generated mutants of equals method in Figure 2
ID

OperatorLine

Original Code

Mutated Code

1
2
3
4
5
6
7
8
9
10
11
12

NegateConditionals
NegateConditionals
NegateConditionals
NegateConditionals
Increments
Increments
Increments
ConditionalsBoundary
ConditionalsBoundary
ConditionalsBoundary
Math
Math

5
5
6
6
6
6
6
5
5
6
5
5

a.length < i + nb.length < j + n--n >= 0a[i++] != b[j++]
--n
i++
j++
a.length < i + nb.length < j + n--n >= 0
i + n
j + n

a.length >= i + nb.length >= j + n--n < 0a[i++] == b[j++]++ni-j-a.length <= i + nb.length <= j + n--n > 0
i - n
j - n

(b) AR-ability(SP ) scores(a) No. of mutants and solutions

Fig. 4: Distribution of the no. of mutants, solutions, and AR-ability(SP ) scores
TABLE III: Result of the selected pairs
Mutants

Success

FailedSkippedAR-ability(SP )Difference

intersects(int tx, int ty, int tw, ..., int height)
cdRect(int sx,int sy,int sw, ...,int dh)

24
10

4
9

9
0

11
1

0.17
0.90

0.73

calendarMonthToInt(int calendarMonth)decodeMonth(int month)

25
13

2
10

23
3

0
0

0.08
0.77

0.69

toInts(byte[] b,int off,int len)
toIntsVectorized(byte[] b,int off,int len)

14
100

10
4

4
4

0
92

0.71
0.04

0.67

compare(Float f1,Float f2)
compare(Float o1,Float o2)

10
11

8
2

2
9

0
0

0.80
0.18

0.62

Char(int i)
isXMLCharacter(int c)

15
15

11
2

4
13

0
0

0.73
0.13

0.60

bilinear(double x1, double y1, ...,double z22)
bilinear(double x1, double y1, ...,double z22)

39
39

5
28

34
8

0
3

0.13
0.72

0.59

cmod(double re,double im)
hypot(double a,double b)

12
11

3
9

7
2

2
0

0.25
0.82

0.57

digit(int c,int radix)
digit(int c,int radix)

20
20

17
6

1
8

2
6

0.85
0.30

0.55

quickSearch(int[] array,int value)find(int[] a,int find)

11
14

2
10

6
3

3
1

0.18
0.71

0.53

compare(Byte b1,Byte b2)
compare(Byte o1,Byte o2)

10
11

7
2

3
9

0
0

0.70
0.18

0.52

hexit(char c)getIntValue(char c)

17
15

5
12

11
3

1
0

0.29
0.80

0.51

compare(Short s1,Short s2)
compare(Short o1,Short o2)

10
11

6
1

4
10

0
0

0.60
0.09

0.51

compareByteArrays(byte[] a, ...,int length)
equals(byte[] a, ...,int n)

12
12

10
4

2
5

0
3

0.83
0.33

0.50

Method pairs

B. Answering RQ2|Success|method with the highest AR-ability(SP ) score (i.e., |M
utants|in this case) is cdRect (0.90), followed by digit (0.85),
and compareByteArrays (0.83). The method pairs withthe highest difference of AR-ability(SP ) is intersec andcdRect with the difference of 0.73, followed by the pair ofcalendarMonthToInt and decodeMonth (0.69), and the pairof toInts and toIntsVectorized (0.67).

For the manual investigation, Table III shows the 11 selected pairs from the 1,282 pairs that passed our criteria andwere manually checked, sorted descendingly by the differences of the AR-ability(SP ) score. The method with thehighest number of mutants generated is toIntsVectorized(100), followed by the two bilinear methods (39). The

The first author performed the manual investigation bycomparing the original code of the two method pairs andlooking for differences in their code structures. The value inthe column difference is the difference of the AR-ability(SP )scores. We found the following 4 code structures that affectautorepairability and occurred more than once in the 11

To answer RQ1, based on our preliminary study, we
found that, on average, the autorepairability scores of Javamethods in the dataset with functional equivalence [5] is
0.45. Approximately about half of the generated faults (i.e.,mutants) were fixed by the kGenProg APR technique.

investigated pairs. We sorted them based on their numberof occurrences. One method in the investigated pairs cancontain multiple code structures listed below. In each example
provided along with the 4 code structures below, the first codesnippet always has a higher autorepairability score.
Code structure 1: Curly braces surrounding a code
block (7 occurrences) We found that the usage of curly
braces surrounding a code block can affect the autorepairability. For example, one method in a pair uses if((c>=‘0’)&&(c<=‘9’)) { digit=c-‘0’; } while the other method
uses if ((c>=‘0’)&&(c<=‘9’)) digit=c-‘0’;. The former one has ahigher autorepairability score. This means clearly specifying acode block by using curly braces can make automated program
repair succeed easier.
Code structure 2: Usage of ternary operator (3 occurrences) We found that the usage of the ternary operator
can offer a higher autorepairability score. For example, one
method in a pair uses
return s1.shortValue()<s2.shortValue()? -1: s1.shortValue()>s2.shortValue() ? 1: 0;

while the other method uses
if (o1.shortValue()<o2.shortValue()) return -1;
if (o1.shortValue()>o2.shortValue()) return 1;
return 0;

Code structure 3: Combined logical expressions in a conditional statement (2 occurrences) We found that combining
logical expressions in a conditional statement can result in ahigher autorepairability score. For example, one method in apair uses
if (i >= 0x20 && i <= 0xD7FF) return true;

while another method uses
if (c < 0x20) return false; if (c <= 0xD7FF) return true;.

Code structure 4: Type of conditional statements (2occurrences) We found that using switch-case can provide
a higher autorepairability score than if-else. For example,
one method in a pair uses case Calendar.JANUARY: return 1
and another method uses if (calendarMonth == Calendar.JANUARY)
return 1.
To answer RQ2, we found 4 code structures that affectthe autorepairability of Java programs including curly
braces surrounding code block, usage of the ternary operator, multiple conditional statements, type of conditional
statements, and type of loop statement. The finding can leadto the development of automated analysis of code that will be
fixed by APR whether the code contains such code structures
or not to assess their readiness.
VI. T HREATS TO VALIDITY
The manual investigation of the code structures in 11
selected method pairs was performed by only one investigator.
Thus, the result may potentially contain some human errors.
We mitigated the threat by setting up a clear procedure
for the comparison and identification of code structures that
affect autorepairability. The experiment was done on thedataset of functionally equivalent Java methods [5] due tothe aim of comparing and identifying code structures that

affect autorepairability. Thus, it may not be generalized toother open-source or commercial Java projects. Lastly, we only
studied kGenProg in this paper. Thus, the findings may not
be generalized to other APR techniques.
VII. I MPLICATIONS AND C ONCLUSION
In this paper, we introduce a new characteristic of software quality, autorepairability. Autorepairability means how
effective APR techniques are in a specified project. We also
proposed an automated way to measure the autorepairabilityof the source code of a given program by using mutation
testing techniques. We measured autorepairability for a total of
1,282 functional equivalent Java methods and compared theirautorepairability scores. We manually investigated 12 pairs
with high differences in autorepairability scores and observed
code constructs that may affect the autorepairability scores.
Introducing autorepairability to the characteristics of software quality will lead to new research being conducted. For
example, many methodologies of autorepairability measurement will be proposed. autorepairability at other granularity
levels can be done by using high-level tests such as systemtests. Methodologies of the software development process todevelop high autorepairability software systems will be researched. Besides, autorepairability-based refactoring, which
is a refactoring that transforms low autorepairability sourcecode to high autorepairability one, will be researched actively.
VIII. ACKNOWLEDGEMENT
This research was supported by the Faculty of ICT, Mahidol
University, and JSPS KAKENHI Japan (JP20H04166,
JP21K18302, JP21K11829, JP21H04877, JP22H03567,
JP22K11985).
R EFERENCES[1] L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A
survey,” IEEE Transactions on Software Engineering, vol. 45, no. 1, pp.
34–67, 2019.
[2] “program-repair.org,” http://program-repair.org/.
[3] A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao,
A. Mols, and A. Scott, “SapFix: Automated end-to-end repair at scale,”in ICSE’19, 2019, pp. 269–278.
[4] K. Naitou, A. Tanikado, S. Matsumoto, Y. Higo, S. Kusumoto, H. Kirinuki, T. Kurabayashi, and H. Tanno, “Toward introducing automatedprogram repair techniques to industrial software development,” in
ICPC’18, 2018, pp. 332–335.
[5] Y. Higo, S. Matsumoto, S. Kusumoto, and K. Yasuda, “Constructing
dataset of functionally equivalent java methods using automated test
generation techniques,” in MSR ’22, ser. MSR ’22, 2022, p. 682–686.
[6] Y. Jia and M. Harman, “An analysis and survey of the development ofmutation testing,” IEEE Transactions on Software Engineering, vol. 37,
no. 5, pp. 649–678, 2011.
[7] “Available
mutators
andgroupsin
PIT,”
https://pitest.org/quickstart/mutators/.
[8] H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “Pit:
A practical mutation testing tool for java,” in ISSTA ’16, 2016, pp. 449–
452.
[9] Y. Higo, S. Matsumoto, R. Arima, A. Tanikado, K. Naitou, J. Matsumoto, Y. Tomida, and S. Kusumoto, “kGenProg: A High-Performance,
High-Extensibility and High-Portability APR System,” in APSEC ’18,
2018, pp. 697–698.
[10] M. Martinez, T. Durieux, R. Sommerard, J. Xuan, and M. Monperrus,“Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on
the Defects4j Dataset,” Empirical Software Engineering, vol. 22, no. 4,
pp. 1936–1964, 2017.