Empirical Study of Change-Prone and Fault-Prone Method
Prediction Focusing on Comment Ownership
Aji Ery Burhandenny, Takashi Nakano
Graduate School of Science and Engineering
Ehime University
Matsuyama, Ehime, Japan 790–8577
Email: {aji, nakano}@se.cite.ehime-u.ac.jp
Hirohisa Aman and Minoru Kawahara
Center for Information Technlogy
Ehime University
Matsuyama, Ehime, Japan 790–8577
Email: {aman, kawahara}@ehime-u.ac.jp
ABSTRACTThe existence of comments in method bodies is a double-edged sword, at one side
it helps code reviewers to comprehend complex code while at the same time it could
reflect the lack of confidence by the programmer concerning the clearness of their code.
While comments can be useful clues to find problematic code, the effects may be varyfrom person to person. This paper proposes a novel metric “Ownership- Considered
Comment Rate (OCCR)” that is enhanced from conventional comment rate byconsidering the difference among developers. The empirical study collects 75,204 Javamethods from five popular open source software to understand the relationship between
the developer and their code and its contribution to change-prone and/or fault-prone
method prediction. The results confirmed the significant influence of OCCR for
predicting changes and faults in methods.
Keyword: comment; developer; fault-prone; open source software

INTRODUCTION
The vast evolution in open source software (OSS) projects enforces the development
team to be efficient in managing code evolutions and fault-related problems especiallyin code review phase. In order to shorten time in understanding source code,
programmers often rely on the documentation provided with the code (Steidl, Hummel,& Juergens, 2013). One of the most fundamental documents is a comment statement
written in a source file. Comments have been widely known as useful and important
artifacts familiar to programmers (Souza, Anquetil, & Oliviera, 2005). Whilecomments have no impact on the software systems’ behavior and performance at all,they play a significant role in the program comprehension.
Aside from automatically generated comments, naturally written comments are
humanly readable, descriptive and often express programmer’s thought. These
characteristics alongside programmer’s manner become an interesting subject for studyof software engineering due to the programmer-specific characteristic can be apotential factor either to increase or to reduce the chance of finding latent bugs in theirsource code. This phenomenon ultimately triggering the “clean code practice” among
the programmers exclusively when deciding whether to conduct code refactoring totheir source code (Martin, 2008). Indeed, carefully-written (detailed) comments are
said to be “deodorant” to mask code smell (Fowler, 1999); some empirical studies
reported that well-commented programs tend to be more fault-prone in popular OSS
products (Aman, Amasaki, Sasaki, & Kawahara, 2015; Aman et al., 2015).
While the previous work (Aman et al., 2015) empirically showed the value to focus
comments in order to predict fault-prone and/or change-prone programs, they missed
diversity of commenting, i.e., “who wrote those comments.” One programmer may
to write many comments with their code, but another programmer may prefer the code
having no or less comments. Such differences in commenting preference would havesignificant impact on the change-prone and fault-prone program prediction. The key
contribution of this paper is to empirically analyze the impact of considering
ownership” toward a better performance of the change and/or fault prediction. We
introduce a novel software quality prediction method based on the abnormal level ofindividual developer’s comments. Moreover, we evaluate our proposed method withthe primitive metrics that have been commonly used for the prediction model.
The remainder of the paper is organized as follows: Section 2 presents our research
motivation and research question with empirical background supporting our work inthis paper. Then, Sect.3 conducts an empirical analysis with five popular open source
software projects and discusses the results. Section 4 describes previous work related
to this paper. Finally, Sect.5 gives our conclusions and future work.
FAULT PREDICTION FOCUSING ON COMMENTS
Comments and Fault-proneness
Comments are useful artifacts to enhance the readability of the code as mentioned in
Sect.1. However, they are sometimes used to compensate the lack of clearness incomplicated code (Buse & Weiner, 2008). That is to say, comments have an aspect
like a double-edged sword, and they are not always recommended to be written a lot.
The code refactoring practice also warns detailed comments as deodorant for masking
code smell (Fowler, 1999).
In our latest work (Aman et al., 2015), we had managed to prove that comment
existence in Java method can significantly differ faulty methods and non-faulty ones.
Nevertheless, a limitation of our approach was that we did not take into account thehuman factor such as the difference in each developer. For example, let us consider
two developers A and B: developer A prefers to describe many comments in theircode, and developer B does not like to write comments. Suppose developer A’s
average of comment rate (lines of comments per lines of code) and B’s one are 0:2
and 0:01, respectively.
Now, if both of them developed programs with comment rate 0:2, then A’s program
looks at a normal level of commenting but B’s program seems to be at a significantly
abnormal level. Therefore, we can suspicious that the B’s program may be something
wrong, and the development manager should examine the reason why developer B

wrote such many comments this time. Such differences in commenting preference
among developers may have a serious impact on fault-prone and/or change-prone
method prediction studied in the previous work (Aman et al., 2015). This concern is
our research motivation in this paper.
Research Question
Our main research question is: “Is the difference in comment rates among developers
valuable to focus for predicting problematic code?”
Our primary goal is to study whether taking into account the average amount andvariation of comment description per developer will effectively influence early
prediction of code-change and bug fix. In order to answer the research question, we
examine the predictive accuracy of bug fixes and code modifications with and without
considering the abnormality comment rates by developers in the following section.
EMPIRICAL WORKThis section elaborates our experiments and analytical results on data sets from
popular OSS projects to explore our research question regarding the value ofdeveloper’s diversity for early prediction of code-change and bugs fix. In order toquantitatively measure the relationship between per-developer in comment rate and its
worthiness to predict prospective code changes and bug fixes in Java methods, we
perform data collection for the comment rate of each developer, and statisticallyanalyze the trends.
We intended to choose the OSS systems for analysis with the following criteria: 1) To
eliminate commonalities that might be valid only for particular OSS systems, thechosen products should be varied in their size and domains. 2) The selected OSS
should have similar implementation language, therefore the differences between
languages would not affect the results. For our study purpose, Java was the first
choice due its compatibility with our mining tools. 3) Each project should be
relatively popular confirmed by having a reasonable number of releases and currently
active up to nowadays. 4) For each project, all the source code and relevant
information concerning change history and their logs, and other feature requests
should be available in their repository.
Table 1. Surveyed Software
SoftwareFree Mind
Squirrel SQL Client
Hibernate
JabRefeXo Platform
Total

Investigation Period
Feb. 2011 – Sept.2015
June 2001 – Sept.2015
June 2007 – Sept.2015
Oct. 2003 – Sept.2015
Mar.2007 – Sept.2015#Source Files
507
3,855
3,861
585
172
8,980

Based on those criteria, we selected five prominent OSS products which all aremanaged using Git (see Table 1). The following metrics are harvested from thesurveyed products: lines of comments in a method body (inner comments;) thateach developer has created (first edition), lines of comments followed by a method
declaration (documentation comments;), lines of code (), cyclomatic
complexity ( ), and frequency of code changes and bug fixes occurred in themethod after their first release.
Data Collection
To perform our empirical analysis for the above research question, we conducted adata collection in the following procedure:
1. In order to perform our analysis efficiently, we made a local copy (clone) of thetargeted repository.
2. By utilizing JavaMethodExtractor 1 we perform a syntax analysis to extract
methods’ bodies from the source files.
3. Then we examine the change history of each method using diff utility2. We track
the frequency of changes and bug fixes up to the latest version according to theinformation provided in their commitment log. By this way, we are able todistinguish which method has been changed during each commitment. Similar tothe previous work, we identify bug fixes, when bug-fix-related keywords or bug
IDs appeared in commitment log (Sliwerski, Zimmermann, & Zeller, 2005).
4. Next, we decide the owner of each method: we define a method’s owner is whocreated the method. Method owners are identified by their email addresses
appeared in the commitment logs.
5. Finally, we measure the initial version of each method by using the above metrics
including,,and.
Table 2 shows the number of methods and the number of developers who are theowners of the initial version of methods.
Table 2. Number of Methods and Number of Developers Who Created Initial
Version of Methods
Software#Developers#Methods
Free Mind
3
6,975
Squirrel SQL Client
10
29,309Hibernate91
31,824
JabRef
10
5,606
eXo Platform
36
1,490
Total
150
75,204
Preliminary Study: Comment Ownership
Prior to analyze the comments rate, in order to justify whether to consider thedifferences in the developer or not, we need to confirmed that the comment rates vary
among developers for each tested product. We check the comment rates toineach developer for both documentation and inner comment rates (and) using analysis of variance (ANOVA), and confirmed the trend statistically.
12

http://se.cite.ehime-u.ac.jp/tool/https://www.gnu.org/software/diffutils/

Table 3. ANOVA Results (Documentation Comment Rate)
SoftwareDegree of Freedom
Free Mind
2
Squirrel SQL Client
9
Hibernate
90
JabRef
9
eXo Platform
35

p-value
0.008

0.352
0.049

Table 4. ANOVA Results (Inner Comment Rate)
SoftwareDegree of Freedom
Free Mind
2
Squirrel SQL Client
9
Hibernate
90
JabRef
9
eXo Platform
35

p-value
0.016

0.573
0.001

Tables 3 and 4 shows the ANOVA results for documentation comment and the innercomment respectively. Significant developer differences in both documentation andinner comment rate have been confirmed, except for JabRef. JabRef is the only
product that has inner comment rate’s pvalue exceeded a significance level ( );JabRef seems not to be a worthy subject in the following analysis, so we will analyze
the remaining four OSS products. We can say that the difference of developer has an
impact on the amount of comments written on inside and outside of a method (innercomments and documentation comments).
Analysis (1) Ownership-Considered Comment Rate (Comment Rate

) VS Conventional

We have confirmed the differences of comment rates among developers in tested
products. In the rest of this section, we assess the worth of comment rates forpredicting change-prone and/or fault-prone methods. We start our analysis bycomparing two types of comment rates:
1. Conventional comment rates (the lines of comments divided by the lines of code),
2. Proposed comment rates that take into account the difference among developers.
3. We call the first type as Conventional Comment Rates () and the later one“Ownership-Considered Comment Rate ().”
First, we calculate the mean and standard deviation of the comment rate for eachdeveloper. Based on this measurement, we evaluate the amount of comments of eachmethod as follows:
to be the inner comment rate of method , i.e.,, and method’s owner is developer . then we define
following equation:ofby theWhereandare the mean of ’s comment rates and the standard deviation of’s ones, respectively. Whenis a positive value, it indicates the comment
rate is greater than the developer ’s average ( ). Similarly, when its value isnegative, it means that the comment rate is less than their average. We want tonormalize the variation by dividing the standard deviation of d’s comment rate ( ).
Then, we consider
to be an index of abnormality with regard toby
the developer . In this paper, we define
as a novel metric,
Ownership-Considered Inner Comment Rate (
):

We
can
defineasimilarmetricfordocumentationcomment
rate—Ownership-Considered Documentation Comment Rate () — byreplacing “inner comments” with “documentation comments” in the above definition.
From hereafter, we refer bothandas “ Ownership-Considered
Comment Rate () for the sake of convenient.

On the other hand, we can consider another type ofwith using the meanof all developer’s comment rates () and the standard deviation of them (), asfollows:

While
also denotes an abnormality of comment rate for method m, it
does not take into account its ownership. We consider
to be aconventional way to evaluate the comment rate, and compare it with our proposed.
Figure 1 compares the powers of change-prone method predictions by
and theconventional score. Its horizontal axis corresponds to the methods in the decreasingorder of comment rate’s abnormality, and the vertical axis signifies the cumulative
number of code-change events occurred in the methods. That is to say, an earlier
growth of a curve means a better performance in predicting change-prone methods. In
the figure, solid red lines correspond to the results ofand dashed blue lines
signifies the results by using the conventional scores. Similarly, Fig.2 compares thepower of fault-prone method predictions by
and the power of that by theconventional score.
In order to evaluate the effect to take into account the differences in comment rate
among developers, we introduce the following improvement rate:(a)Doc Comment(b)(FreeMind)

Doc Comment(c)(SQuirrel SQLDoc Comment(d)(Hibernate)

Doc Comment(Platform)

Client)(e)

Inner Comment(f)(FreeMind)

Inner Comment(g)(SQuirrel SQL

Inner Comment(h)(Hibernate)

Inner Comment(Platform)

Client)Figure 1. Cumulative number of modifications occurred at methods that are sorted in thedecreasing order of their comment rates’ abnormality.

(a)

Doc Comment(b)(FreeMind)

Doc Comment(c)(SQuirrel SQLDoc Comment(d)(Hibernate)

Doc Comment(Platform)

Client)(e)

cxcxInner(f)

Inner Comment

Comment(SQuirrel SQL(FreeMind)

Client)(g)

Inner Comment(Hibernate)(h)

Inner Comment(Platform)

Figure 2. Cumulative number of bug fixes occurred at methods that are sorted in thedecreasing order of their comment rates’ abnormality.

Whererepresents the area under the curve ofin Figs. 1 and 2, andsignifies the area under the curve using the conventional score in those figures. Sincea larger area means a better performance in predicting change-prone or fault-prone
methods, indexes the improvement of their prediction power by using
taking into account the difference in comment rate among developers.

, i.e.,

Table 5. Improvement Rate with regard to Documentation Comments.
Software#Modifications#Bug Fixes
Free Mind
0.237
0.352
Squirrel SQL Client-0.074-0.074Hibernate
0.057
0.046
eXo Platform-0.199-0.300
Average0.004
0.003
Table 6. Improvement Rate with regard to Inner Comments.
Software#Modifications#Bug Fixes
Free Mind
0.318
0.488
Squirrel SQL Client
0.320
0.296
Hibernate
0.026
0.017
eXo Platform-0.009-0.070
Average
0.131
0.146
Tables 5 and 6 describe the improvement rates with regard to the documentationcomments and inner comments, respectively. Table 5 shows thatis notresourceful for documentation comment (improvement rate
on average). In
contrast, Table 6 justifies a part of our research question that taking into account thedifferences in developer’s comment rate can be useful to an early prediction ofproblematic code—change-prone and fault-prone methods: their improvement rates
are infor both change-prone method prediction and fault-prone method–
prediction. It seems that the differences in developer’s inner comment rate have apromising effect on change-prone and/or fault-prone method prediction rather than the
documentation comment rates. Therefore, we will examine the change-prone andfault-prone prediction powers of ownership-considered inner comment rate bycomparing with the conventional metrics, lines of code () and cyclomatic
complexity ( ) in Sect. 3.4 and Sect. 3.5, respectively.
Analysis (2) OCCR vs Lines of Code (LOC)
In previous analysis, we have confirmed the helpfulness of(for innercomments) for early detection of code changes and bug fixes in Java methods. In thisanalysis, we comparewith, which is a popular metric commonly used
for predicting fault-prone programs. Our procedure of the comparative study is similar
to Analysis (1) in Sect.3.3: we cumulate the numbers of modifications occurred atmethods that are sorted in the decreasing order of, and that are sorted in thedecreasing order of, respectively. We also compute the cumulative numbers of
bug-fixes occurred at methods in those orders.

(a)#Modifications(b)(FreeMind)#Modifications(c)(SQuirrel SQL#Modifications(d)(Hibernate)#Modifications(Platform)

Client)(e)#Bug Fixes(FreeMind)(f)#Bug Fixes(SQuirrel SQL(g)#Bug Fixes(Hibernate)(h)#Bug Fixes(Platform)

Client)Figure 3. Cumulative numbers of modifications and bug fixes occurred at methods that aresorted in the decreasing order of their comment rates abnormality and in the decreasingorder of LOC.

Figure 3 shows the curves of cumulative numbers. In the figure, solid red lines
correspond to the results ofand dashed green lines signifies the results of. Table 7 shows the improvement rates defined in Eq.(4) where a1 and a0 are thearea under the curve ofand that of, respectively.
From Table 7, we can say thatis at almost the same or better performancethanin predicting change-prone methods and fault-prone ones:
improves the prediction power byandon average compared tofor change-prone method prediction and fault-prone method prediction, respectively.
FreeMind showed the best performance with, which improves aroundfor predicting modification occurrences and nearlyfor predicting bug fixes
occurred in the methods. While the bug fix prediction in SQuirreL SQL Client and thecode modification prediction in eXo Platform show negative improvement rates, bothof them are at small levels (and). Therefore, for most of projects,
has a usefulness at the same with or better thanin predictingchange-prone methods and fault-prone methods.

Table 7. Improvement Rates:
SoftwareFree Mind
Squirrel SQL Client
Hibernate
eXo Platform
AverageAnalysis (3)VS.
#Modifications
0.192
0.003
0.060
-0.001
0.027

vs Cyclomatic Complexity (#Bug Fixes
0.391
0.251
0.000
-0.0440.147)

Next, we conduct another comparative study with using cyclomatic complexity ( ).
has also been widely known as a useful metric for predicting fault-prone
programs. Our analysis procedure is similar to the one described in Sect.3.4, where
we replace “” with “ .” Due to limitations of space, we omit a figure of
cumulative numbers of modifications and bug-fixes occurred at methods in thedecreasing order of metric values like Fig.3, and we show only a table ofimprovement rates (see Table 8).
Table 8. Improvement Rates:
SoftwareFree Mind
Squirrel SQL Client
Hibernate
eXo Platform
AverageVS#Modifications
0.259
0.270
0.012
-0.001
0.131#Bug Fixes
0.391
0.251
0.000
-0.0440.147

As shown in Table 8,seems to be more useful thanin predictingchange-prone methods and faultprone ones: the improvement rates are about
andon average for code modification prediction and bug fix prediction,
respectively. While eXo Platform showed small disimprovements (and) similar to Analysis (2) (see Table 7), the remaining products showed thatis at almost the same level withor better thanin the predictions.
Especially, FreeMind and SQuirreL SQL Client showed over
improvements.
Therefore, we can say thatcan be a more promising metric than CyclomaticComplexity.
Threats to Validity
This empirical work used five popular OSS products from different domains and indifferent size for a generality of our empirical results. However, we had to limit our
subjects to Java products because of our data collection tool. While the language
limitation may be our threats to validity, the fundamental concept of commenting
code is common to almost all modern programming languages, so our results would
not lose its generality for other languages.
If there was a specific coding convention in the surveyed OSS projects and they
controlled developers on how to write comments, such a control can be our threats tovalidity. JabRef did not show statistical difference in comment rate among developers,
so it might be controlled by a certain coding rule. However, we excluded JabRef fromour subjects in the main analyses (1)–(3), and the remaining products showed
diversities in comment rates among developers. Therefore, we can say that the
influence of coding convention would not be concern in our empirical results.
RELATED WORK
Recently, there are several studies that investigate the relationship between developer
activities to the fault proneness (Robbes & Rothlisberger, 2013; Rahman & Devanbu,2011). However, all of their work do not consider comment as one of importantfactors in predicting fault-prone programs. Aman et al. (2015) stressed on theimportance of comments and their relationship to the fault-proneness. They conductedempirical analyses to show the worth to focus on Lines of Comments () as auseful metric along withandin analyzing fault-proneness of Java methods.
While their work is our previous work, they missed to take into account the diversity
in commenting manners among developers, so an empirical analysis using“ownership-considered comment rate ()” is our key contribution in this paper.
CONCLUSION AND FUTURE WORKIn this paper, we aimed to investigate the developer differences in commenting theircode, and the influence of such a diversity to the accuracy of change-prone and/or
fault-prone method prediction. In our first analysis, we had confirmed the efficiencyof our metric “Ownership-Considered Comment Rate ()” in detectingchange-prone and fault (bug)-prone methods compare to the conventional comment
rate. Our second and third analyses revealed that(for inner comments)
can be useful metric in predicting change-prone methods and fault-prone methods,
and its usefulness is at almost the same level with or better than conventional popular
metrics,(Lines of Code) and(Cyclomatic Complexity). Therefore, thispaper successfully showed the importance of taking into account the developers’
differences in commenting their code for more accurate change-prone and/or
fault-prone method prediction.
Our future work includes:
1. Examining the evolution of comments by developers through the softwaredevelopment and maintenance;
2. Analyzing the contents of comments with the natural language processing
methods toward a better understanding the relationship between a developer andtheir comments.

ACKNOWLEDGEMENT
This work was supported by JSPS KAKENSHI Grant Number 25330083.

REFERENCES
Aman, H., Amasaki, S., Sasaki, T., & Kawahara, M. 2015. Empirical Analysis of
Change-Proneness in Methods Having Local Variables with Long Names andComments. Proc. 2015 ACM/IEEE Int’l Symp. Empirical Softw. Eng. andMeasurement: 50–53.
Aman, H., Amasaki, S. Sasaki, T., & Kawahara, M. 2015. Lines of Comments as a
Noteworthy Metric for Analyzing Fault-Proneness in Methods. IEICE Trans.
Inf. & Syst. E98-D(12): 2218–2228.
Buse, R.P. & Weiner, W.R. 2008. A metric for software readability. Proc. Int’l Symp.
Softw. Testing and Analysis: 121–130.
Fowler, M. 1999.
Refactoring: Improving the Design of Existing Code.
Addison-Wesley Longman Publishing.
Martin, R.C. 2008. Clean Code: A Handbook of Agile Software Craftsmanship.
Prentice Hall PTR.
Rahman, F., & Devanbu, P. Ownership, Experience and Defects: A Fine-Grained
Study of Authorship. Proc. 2011 33rd Int’l Conf. Softw. Eng: 491–500.
Robbes, R., & Rothlisberger, D. 2013. Using Developer Interaction Data to Compare
Expertise Metrics. Proc. 2013 10th IEEE Working Conf. Mining Softw.
Repositories: 297–300.
Sliwerski, J., Zimmermann, T., & Zeller, A. 2005. When do changes induce fixes?
ACM SIGSOFT Softw. Eng. Notes. 30(4): 1–5.
Souza, S. C. B., Anquetil, N., & Oliveira, K. M. 2005. A Study of the Documentation
Essential to Software Maintenance. Proc. 23rd Annual Int’l Conf. Design ofCommunication: Documenting &Amp; Designing for Pervasive
Information: 68–75.
Steidl, D., Hummel, B., & Juergens, E. 2013. Quality analysis of source code
comments. Proc. 2013 IEEE 21st Int’l Conf. Program Comprehension:
83–92.