Thresholds for Size and Complexity Metrics: A
Case Study from the Perspective of Defect Density
Kazuhiro Yamashita⇤ , Changyun Huang⇤ , Meiyappan Nagappan‡ , Yasutaka Kamei⇤ ,Audris Mockus§ , Ahmed E. Hassan† , and Naoyasu Ubayashi⇤⇤ Kyushu University, Japan; {yamashita, huang}@posl.ait.kyushu-u.ac.jp, {kamei, ubayashi}@ait.kyushu-u.ac.jp† Queen’s University, Canada; ahmed@cs.queensu.ca‡ Rochester Institute of Technology, USA; mei@se.rit.edu§ University of Tennessee-Knoxville, USA; audris@utk.edu

Abstract—Practical guidelines on what code has better quality
are in great demand. For example, it is reasonable to expect
the most complex code to be buggy. Structuring code into
reasonably sized files and classes also appears to be prudent.
Many attempts to determine (or declare) risk thresholds forvarious code metrics have been made. In this paper we want
to examine the applicability of such thresholds. Hence, we
replicate a recently published technique for calculating metricthresholds to determine high-risk files based on code size (LOC
and number of methods), and complexity (cyclomatic complexity
and module interface coupling) using a very large set of open
and closed source projects written primarily in Java. We relate
the threshold-derived risk to (a) the probability that a file would
have a defect, and (b) the defect density of the files in the highrisk group. We find that the probability of a file having a defect ishigher in the very high-risk group with a few exceptions. This isparticularly pronounced when using size thresholds. Surprisingly,
the defect density was uniformly lower in the very high-risk
group of files. Our results suggest that, as expected, less code isassociated with fewer defects. However, the same amount of codein large and complex files was associated with fewer defects thanwhen located in smaller and less complex files. Hence we conclude
that risk thresholds for size and complexity metrics have to be
used with caution if at all. Our findings have immediate practical
implications: the redistribution of Java code into smaller and lesscomplex files may be counterproductive.
Index Terms—Software metrics; Thresholds; Defect models;I. I NTRODUCTION
There has been a considerable amount of research in thefield of software defect prediction, especially in the last
decade. Over 100 papers have been published just from 2000
- 2011 in this area of Empirical Software Engineering (ESE)
research alone [16, 27]. The primary goal of such research is
to be able to provide guidelines to practitioners on what kindof code has better quality. This goal has two purposes: (a)identifying the metrics that are most useful for the practitioner(often usefulness is measured by how accurately the metrics
identify bad code, how easy is it to collect the metrics, andhow actionable the metrics are), and (b) what values of these
metrics indicate good code or bad code.
We focus on the second purpose. In particular, over the last
30 years, many publications investigated “best” values for thevarious software metrics. Among other methods, thresholds
were derived based on experience [8, 23, 25], analysis of themetrics themselves [11, 14], error models [6, 10, 26], cluster
techniques [22, 34], and metric distributions [31, 32].
Almost all of these approaches treat extreme values of file
metrics as detrimental. Defect prediction approaches that use
a linear statistical model, conclude that an extreme value ofthe metrics implies poor quality of code, by virtue of choosing
such a model. For example, ‘larger files will have more defects’ is a common refrain in ESE research [15]. Alternatively
the “Goldilocks Principle” suggests that extreme values of a
metric are a sign of poor quality code [17, 18]. Fenton and
Neil provide a more comprehensive list of research studies
with respect to metrics based defect prediction [12]. Theyfind that the literature has contradictory evidence regarding
the relationship between software defects and software metrics
like size and complexity. It is, thus, unclear if metric thresholds
should be used to identify source code files that are at highrisk.
Consequently, we aim to observe if a consistent relationshipbetween metric thresholds and software quality is present in
OSS and industrial projects. Therefore we conduct a casestudy on three OSS and four industrial projects. The metrics
that we choose to evaluate are size based (Total LOC ina file, and Module interface size in a file), and complexity
based (cyclomatic complexity and module inward coupling).
We evaluate software quality using two criteria - (a) defectproneness (probability of a file having a defect), and (b) defectdensity (the number of defects/LOC).
We replicate the most recently published state-of-the-art
technique (proposed by Alves et al. [3]) to determine thethresholds for these metrics, and found them to be very close
to ones reported earlier. In order to use this approach, weneed a set of projects to calculate the thresholds from. In
their paper Alves et al. [3] use 100 Java and C# systems.
In our paper we use 1,000 OSS Java projects (from a pool ofmore than 20K OSS Java projects) to calculate the thresholds
for the OSS projects and 200+ industrial systems to calculate
the thresholds for the industrial projects. Our results would
be similar with alternative, simpler approaches to set thethresholds, but we prefer to use the state-of-the-art techniqueto ensure that the thresholds are set in the most appropriatemanner. We evaluate software quality using two criteria - (a)defect proneness (probability of a file having a defect), and(b) defect density (the number of defects/LOC). For reasons
listed in Section II-A, we consider Java projects in this casestudy, and thus our findings relates only to Java files.
On replicating the approach proposed by Alves et al. [3]for determining thresholds in our case study, we make thefollowing key observations:• Files identified as very high-risk by size and complexity
thresholds were associated highest fault-pronenes.
• Files identified as very high-risk by size threshold wereseveral times more fault-prone than files identified as very
high-risk by complexity threshold.
• Files identified as very high-risk by size threshold have alower (often significantly lower) defect density than theremaining files.
• In some cases files identified as very low risk by size andcomplexity threshold had a low defect density value as
well. In these cases the files with medium to high risk
size and complexity values had the highest defect density(which is contradictory to the “Goldilocks Rule”).
These findings confirm earlier results that the size benchmarks are associated with high defect-proneness. They also
indicate that the size is a better threshold of defect-pronenessthan complexity. Defect density (in contrast to defectproneness) tends to be the lowest in the files benchmarked
as high-risk, yet we could not confirm earlier suggestions thatthe smallest files always have the highest fault density or the“Goldilocks rule”. In summary, we found that the thresholds
have to be used with caution, if at all.
Paper organization. Follows: Section II describes the datathat we use in our case study. Section III presents the techniquethat we use to determine the thresholds for the metrics.
Section IV presents the design of our experiments. Section V
presents the results of our experiments and the discussion
about the results. In Section VI, we present the related researchon ways to determine the thresholds for metrics, and on theveracity of metric thresholds. Section VII presents the threats
to validity, and finally Section VIII presents our conclusions.
II. DATA S OURCES
In this case study, we analyzed both OSS and industrial Javasoftware projects. We describe the data that we used in ourcase study below. This section is further subdivided into two
subsections - one for describing the projects that were used
to calculate the thresholds, and one for describing the projects
that we used to determine the relationship between thresholds
and defects.
A. Software Projects used for Calculating Thresholds
In the case of the industrial projects we calculated thethresholds from 205 industrial Java software projects also
from Avaya. In the case of the OSS projects we use thedataset collected by Mockus [24]. Mockus created this dataset
from several forges such as git.kernel.org, SourceForge, PostgreSQL, GitHub and so on. This dataset contains more than
125K software projects (in C, C++, C#, Java etc.). We extracted 22,956 Java projects from this dataset. We chose to

only extract Java projects for 2 reasons - (a) A lot of open
source and industry projects are written in Java, and (b) We
wanted to control for language context, as Java may have somespecific effects on file size. The use of a single language
excludes the possibility that our results were affected by
the thresholds potentially having opposite effects in different
languages. We only included those Java projects that had 200
or more files in our case study to exclude very small projects.
Other investigations we have done (beyond the scope of thispaper) suggest that a much higher percent of projects in theforges with fewer than 200 files appear to be not real softwaredevelopment projects. Therefore, we were left with 4,575 Javaprojects in the end.
B. Software Projects with Defect Data
Although we have this large set of projects to calculate thethresholds, we do not have defect data for them. Hence, wecollected defect data in three OSS and four industrial (from atelecommunications company, Avaya) Java software projects.
The three OSS projects are - Eclipse, Mylyn, and Netbeans.
Eclipse and Netbeans are IDEs for writing Java code, while
Mylyn is a plugin for Eclipse used in task management. All
three projects have been used in many software engineeringstudies before [7, 19, 30]. We collected this data for three
versions in each of the OSS projects. We collect three versionsof these OSS projects even though all our metrics (size andcomplexity) are non-process based (product-based), and hence
static in nature. We do this in order to examine if our results
are consistent across multiple versions of the same project.
The industrial projects are from a telecommunications company and are products that are being used regularly. We only
have one version of each of these projects. Therefore, in
total we have 13 versions (3 ⇤ 3 = 9 OSS and 1 ⇤ 4 = 4
industrial) of software projects for which we collected thedefect and metric data. We collected the defect data forthese projects at a file level. Each commit to a file with acommit message that contains an issue ID that exists in theissue tracking system (note that in the industrial projects, weonly consider customer-reported defects) is considered to bea defect that was fixed in that file. This is the same procedureused by Kamei et al. [19]. Note that in this paper, we use
post release defects, which are found after releasing a versionof software. They are reported in the issue tracking system(such as Bugzilla). Post release defects could be anythingfrom a typo to serious security/performance defects or runtime
errors/crashes. Currently in all software engineering research
all post release defects are considered to be harmful since theyaffect the customer.
III. D ERIVING T HRESHOLDS FOR M ETRICS
As explained in the previous section, we collected 4,575
OSS projects and 205 industrial projects for determining thethresholds for the 3 OSS and 4 industrial projects with defectdata. In this section we explain the method that we replicatedto determine the thresholds [3] and apply this technique toall the 4,575 OSS projects to give an overview of the OSSProject$A$
LOC$10K

LOC$
300

LOC$
100

LOC$
500

LOC$
50

File 1File 2File 31

5

8100/10K= 1%

500/10K= 5%

50/10K= 0.5%...

File N

Step 1
Metrics Extraction
Cyclomatic
5Complexity (CC)Step 2.
Weight Ratio Calculation
Weight Ratios300/10K= 3%
Step 3
Entity Aggregation

0File 2 : 1%

File 1 : 3%

CC = 1File 3 : 5%
CC = 5...

1

2

File N : 0.5%
CC = 8

...

5

8

...
CC

Aggregated Weights Project A: (0%, 1%, …, 8%, …, 0.5%, …)Step 4
System Aggregation

Divide it by the number of projects

Norm. ProjectA: (0%, 0.02%, …, 0.1749%, … 0.01%, …)Norm. ProjectB: (0%, 0.07%, …, 0%, …, 0.11%, …)...
Norm. ProjectZ: (0%, 0.07%, …, 0.02%, …, 0.03%, …)Step 5Weight Ratio Aggregation

Sum them for each Metrics
value across all Projects

All projects:(0%, 34%, 49%, 59%, 66%, 71% …)
100%WeightRatio 90%
80%
70%
60%
50%
40%
30%
0

1

2

3

4

5

6

7

8

9

10

11 ...

CC

Step 6
Thresholds Derivation
Thresholds = Metric$
Cycloma<c$Complexity$

70$%$

80%$

90%$

5$

8$

10$

Fig. 1. The six steps in the technique proposed by Alves et al. [3] to derive
the thresholds.

dataset being used. We chose to replicate this technique forcalculating the thresholds for a number of reasons: (a) it
adheres to the scale and distribution of the metrics themselves,(b) the inherent variability present in the various metrics acrossthe various systems are brought out, and (c) the thresholds
derived have an empirical foundation and do not depend on
the experiences of an individual,
Figure 1 illustrates the technique used by Alves et al. which
consists of the following six steps [3]:1) Metrics extraction: We extract the four metrics (lines
of code, module interface size, cyclomatic complexity, andmodule interface coupling) for each file in each of the software
projects under study using the tools, Understand [2], andSonar [1]. For example as shown in Step 1 of Figure 1, say
Project A has files A1 , A2 , ..., An . We extract the four metrics
for all the n files.
2) Weight ratio calculation: We calculate the weight ratioof each file in each system as the ratio of the size of the fileto the total size of the system. For example as shown in Step
2 of Figure 1, if file A1 has 300 LOC, and if the total lines ofcode of project A is 10K LOC, then the weight ratio for file
A1 will be 300/10K = 3%. Thus there is one weight ratio foreach file, in each project.
3) Entity aggregation: In this step, we aggregate the weightratios of each file according to the metric value. For example
as shown in Step 3 of Figure 1, if files A1 , and A3 are theonly files in project A with a cyclomatic complexity of 5, thenwe add the weight ratios corresponding to files A1 and A3 . If
the weight ratios of A1 and A3 are 3% and 5% respectively,
then the entity aggregated value for cyclomatic complexity of
5 is 8% (3 + 5). Therefore we calculate this aggregated value
for every unique value of all the four metrics.
4) System aggregation: In this step, we take the entity
aggregated value of each unique metric value for all thefour metrics from the previous step, and divide each by the
total number of projects (in our case, this is 4,575), to get
the normalized entity aggregation value. Thus the normalized
entity aggregation value in our example for cyclomatic complexity value of 5 for project A is 0.1749%, as shown in
Step 4 of Figure 1. Then we aggregate these values acrossall the projects. Therefore in our example we would add
the normalized aggregation value for a cyclomatic complexity
value of 5 from all 4,575 projects. We would repeat thisfor every unique value of all the four metrics across all the
projects. In the end we have four vectors, one for each metric.
The length of each metric is equal to the number of uniquevalues that the metric can assume across the 4,575 projects.
Thus we have aggregated the metrics across the 4,575 projects.
5) Weight ratio aggregation: In this step, we take the four
vectors generated at the end of the previos step, sort them inascending order. Then we plot a cumulative line chart for eachof the metrics (in percentile scale). This is shown in Step 5 of
Figure 1.
6) Thresholds derivation: In the final step we arrive at thethresholds. As noted by Alves et al. in their paper, we too
determine the value of each of the metrics at the 70%th ,80%th , and 90%th percentiles. We use these values as thethresholds for various risk categories of the metrics. For
example as shown in Step 5 of Figure 1, if the cyclomatic
complexity thresholds are 5, 8, and 10 (corresponding to the70%th , 80%th , and 90%th percentile values), then we classify
all files with cyclomatic complexity, cc < 5 as Low Risk,
5 <= cc < 8 as Medium Risk, 8 <= cc < 10 as High
Risk, cc >= 10 as Very High Risk. Therefore each file will
be assigned to exactly one risk category based on the metricvalue for that file. Note that a particular file could be in the
low risk category from the perspective of one metric, and in ahigh risk category from the perspective of another metric. For
example, file A1 which has a cyclomatic complexity of 5 and300 LOC, will be classified as a medium risk file with respect
to cyclomatic complexity and a very high risk file with respect
to LOC. For further details on this approach for calculating
thresholds we refer the readers to the original paper by Alves etal. [3].

TABLE I
C ORRESPONDENCE BETWEEN THE M ETRICS A NALYZED AND THE
M ETRICS IN THE TOOLS : U NDERSTAND AND S ONAR
MetricGranularity UnderstandLines of FileCountLineCode
Code
ModuleFileCountDeclInstanceMethod
Interface
Size
Cyclomatic Method
CyclomaticStrict
Complexity
ModuleMethod/File CountInput
InwardCoupling

Sonar
ncloc
functionsfunction complexity
accessors

IV. C ASE S TUDY D ESIGNIn the previous sections we have described our data sources,
and the technique we use to derive the thresholds for the
projects, along with how we determine the number of defects
in each file of each version of the three OSS and fourindustrial projects. In this section, we present the study designof our experiments to evaluate the relationship between metricthresholds and software quality. The details of the metrics usedin this study, and the evaluation criteria to quantify software
quality are described below.
A. Metrics that were Analyzed
In this case study we study two size based and two complexity based metrics. We chose these four metrics, since Alves etal. [3], in their original case study, used these four metrics
and derived the thresholds on 100 OSS and industrial projects.
These metrics were extracted from the OSS projects using thetool Understand [2], and from the industrial projects using thetool Sonar [1]. The metrics analyzed and their corresponding
functions in the tools are presented in Table I. Additional
details on these metrics and how they are calculated can befound at the webpages for the corresponding tools [1, 2].
The two size based metrics were collected at the file level,
while the two complexity based metrics was collected at themethod level. Since Sonar, the tool that was used to extract the
metrics for the industrial project, did not collect the Module
Inward Coupling at a method level, we used the number ofaccessors as a substitute which is measured at the file level.
The threat which is caused by using two different tools tomeasure metrics is discussed in Section VII.
Table II shows the statistics of OSS and Avaya projects
with respect to four analyzed metrics. We first calculate
median values of four metrics shown in Table I for each OSS
and Avaya project. Then we calculate minimum, median andmaximum values to lift up project-level values to categorylevel ones (i.e., OSS-level or Industry-level).
B. Software Quality Evaluation Criteria
We use two well known software quality measures in orderto evaluate our goal - defect proneness and defect density [27].
Typically these are defined for a specific file or class. In
our case, the granularity that we examining them in are:risk categories. Note that these risk categories are defined by

the thresholds that we extract for the metrics. We define thesoftware quality evaluation criteria as follows:
1) Defect proneness - the probability of a file in a particularrisk category having a defect. More formally, it is theratio of the number of files in a risk category which
has at least one defect, to the number of files present
in that category. Let’s assume that there are 400 files
that have a cyclomatic complexity value less than thevalue for the 70th percentile (low risk) in our thresholdderivation technique. If 100 of them have at least one
defect, then the defect proneness of the low risk categoryof files from the perspective of cyclomatic complexity
is 100/400 = 0.25.
2) Defect density - the number of defects per LOC for aparticular risk category. More formally, it is the ratioof the sum of the defects in all the files in a particular
category, to the sum of LOC of all the files in the same
category. Considering the above example, if the same
400 files have a combined size of 45,000 LOC, and 50of the 100 files with defects have 2 defects each (whichmakes it 50 ⇤ 1 + 50 ⇤ 2 = 150 defects in total), then thedefect density of files with low risk from the perspectiveof cyclomatic complexity is 150/45, 000 = 0.0033.
Therefore we calculate the defect proneness, and defectdensity of each risk category (four in total), for all the metrics(four in total) in all the case study projects (three OSS and fourindustrial, and 13 in total). As we mentioned in Section II, in
our case study we use post release defects to calculate both ofthese quality measures. In the case of OSS, these defects werereported in the bug repository, and in the case of the industrial
projects, the defects were customer reported.
C. Choosing Representative Projects for Calculating Metrics
Thresholds
One of the assumptions made by Alves et al. is that thethresholds are derived from a set of systems representative ofthe target system to obtain robustness of the derived threshold.
Note that the case study systems that we calculate the quality
measures are not the same as the case study systems that
we use to calculate the thresholds. We calculate the quality
measures for three OSS and four industrial projects. These
projects are the target systems in our case. To make sure thatthe thresholds are calculated for the target systems from a more
representative sample, we chose a subset of 1,000 projects
from the 4,575 Java projects for each of the three versions of
the three OSS projects. By “representative” we mean having
a similar distribution of file sizes. For selection of the mostsimilar projects we use median to characterize that distribution.
More discussion is in Section VII. The selection was done asfollows:
1) For all of the nine versions of OSS projects and 4,575
projects used to determine the thresholds, we calculated
the median LOC in a file.
2) For each of the nine versions of OSS projects, we chose
1,000 projects from the pool of 4,575 projects that had
the closest median LOC in a file.

TABLE IIS TATISTICS OF OSS AND AVAYA P ROJECTS WITH R ESPECT TO F OUR M ETRICS

OSS
Avaya

Lines
Code

of

MIN
2.0
5.0

MED
29.0
41.5

MAX
403
161.0

ModuleInterfaceSize
MIN MED
0.0
3.0
1.0
4.0

CyclomaticComplexity
MAX
26.0
11.0

TABLE IIIM EDIAN OF THE OSS P ROJECT, ALONG WITH THE I NTER Q UARTILE
R ANGE OF THE DIFFERENCE BETWEEN MEDIAN LOC/ FILE OF EACH OF
THE 1,000 OSS PROJECTS CHOSEN TO DERIVE THE THRESHOLD AND THE
MEDIAN LOC/F ILE IN THE CORRESPONDING OSS PROJECT.
Project Name
Eclipse 3.0
Eclipse 3.1
Eclipse 3.2
Mylyn 1.0
Mylyn 2.0
Mylyn 3.0
Netbeans 4.0
Netbeans 5.0
Netbeans 5.5.1

Median
100
105
105
93
89
93
117
119
106

Interquartile Range
8
8
8
6
5
6
11
11
9

In Table III, we present the median LOC/File of each of
the nine OSS projects. These are the target projects for which
we need to calculate the threshold. As stated above, we intend
to choose 1,000 projects that are most representative of thetarget systems in order to calculate the thresholds. Therefore
we calculate the Inter Quartile Range (IQR) of the absolutedifference between the median LOC/File in the 1,000 projects
chosen for deriving the thresholds and the median LOC/File ofthe target project, in order to examine how representative thesample used to calculate the threshold is to the target project.
IQR is a robust measure of spread that is more appropriate
than the variance or standard deviation for the highly skewed
distribution such as file size. For all the target systems the
IQR of the difference between their median LOC/File and the
median LOC/File in the 1,000 projects chosen to calculate thethreshold is at-most 11 LOC. From this low value we cansee that the 1,000 projects chosen from the dataset of 4,575
projects, to derive the metric thresholds, are indeed comparable
in median LOC/File value to each of the OSS projects understudy.
Using the size and complexity metrics extracted from the
1,000 projects for each target project, we calculated a unique
set of thresholds for each of the nine versions of OSS projects.
As for the four industrial projects we used the set of 205
different industrial projects from the same organization toderive the threshold (i.e., we use same threshold to evaluate
four industrial projects). The thresholds for the nine OSS
projects and the thresholds for the industrial projects arepresented in Table IV. From the four tables, we can see thatthe industrial projects have a considerably higher range forthe thresholds of all the metrics. The 70th percentile values
are much lower than the OSS projects, and the 90th percentile
values are much higher than their OSS counterparts.

MIN
1.0
1.0

MED
1.0
6.5

MAX
5.0
32.0

ModuleInwardCoupling
MIN MED
0.0
2.0
0.0
0.0

MAX
6.0
4.0

V. R ESULTS AND D ISCUSSION
In this section, we present the experiment that we carried out
along with the results for those experiments and a discussion
pertaining to it.
A. Experimental Approach
As described in the previous section, we extract the metrics for all the OSS and industrial projects. We derive thethresholds individually for each OSS software version andcollectively for the industrial projects. Using the thresholds foreach metrics, we split the files in each of the three OSS andfour industrial projects (13 versions in total), for which we also
have the defect data, into the four risk categories(low, medium,high and very high). Using this classification and the defectdata, we calculate the defect proneness, and defect density foreach risk category of each metric for the 13 versions.
B. Experimental Results
The defect proneness and defect densities are shown asline plots in Figure 2 and 3. The plot on the left hand sidecorresponds to the defect proneness, and the plot to the righthand side corresponds to the defect density of each system.
Overall, most of the trends among all the systems are similar.
All the industrial projects bear close resemblance to each other.
Additionally there is not much variance among versions ofeach OSS project either. Some observations that we can makeon more careful analysis from Figure 2 and 3 are as follows:(B1) Defect Proneness:
1) In the industrial projects, files in the low risk category have the same defect proneness, irrespective of
the metric that was used to assign the risk, i.e., files
that have few LOC and low complexity are equallydefect prone. However, files in the high risk category
have considerably different defect proneness values. For
example, files that have a very high value for size based
metrics are two to five times more defect prone than files
that have very high values for complexity based metrics.
When we examine the OSS projects, we consistently findthat files are twice as likely to be defect prone for both
low and high values of the size based metrics, than thecorresponding low and high values of the complexity
based metrics.
2) In the OSS projects, in some cases (like Eclipse 3.0,
Eclipse 3.1, and Netbeans 4.0), the highest risk categoryof size based metrics are not related to defect pronenessas expected. The files with very high values for the sizedbased metrics are marginally less defect prone than files

TABLE IV
T HRESHOLDS FOR THE OSS AND I NDUSTRIAL P ROJECTS
80%
498
515
515
480
466
480
545
553
514
543

90%
884
908
908
861
847
861
952
975
911
1,423

(b) Module Interface Size
Project Name
Version 70%
80%
Eclipse3.0
18
26
Eclipse3.1
19
27
Eclipse
3.2
19
27
Mylyn
1.0
18
26
Mylyn
2.0
18
25
Mylyn
3.0
18
26
Netbeans
4.0
20
28
Netbeans
5.0
20
28
Netbeans
5.5.1
19
27
Industrial
19
30

90%
43
44
44
42
42
42
46
47
44
89

(c) Cyclomatic Complexity
Project Name
Version 70%
80%
Eclipse3.0
6
9
Eclipse3.1
6
9
Eclipse
3.2
6
9
Mylyn
1.0
6
9
Mylyn
2.0
6
9
Mylyn
3.0
6
9
Netbeans
4.0
610
Netbeans
5.0
610
Netbeans
5.5.1
69
Industrial
4.3
5.6

90%
17
18
18
17
16
17
18
19
17
10.3

(d) Module Inward Coupling
Project Name Version 70%
80%
Eclipse3.0
6
9
Eclipse3.1
7
9
Eclipse
3.2
7
9
Mylyn
1.0
6
9
Mylyn
2.0
6
9
Mylyn
3.0
6
9
Netbeans
4.0
7
9
Netbeans
5.0
79
Netbeans
5.5.1
7
9
Industrial
3
7

90%
14
14
14
13
13
13
15
15
14
30

Project Name
EclipseEclipseEclipse
Mylyn
Mylyn
Mylyn
Netbeans
Netbeans
Netbeans
Industrial(a) Lines of Code
Version 70%
3.0
335
3.1
346
3.2
346
1.0
321
2.0
313
3.0
321
4.0
365
5.0
369
5.5.1
346
324

with just high values for the same metrics. However,in the industrial projects, this phenomenon is observed
for the cyclomatic complexity instead. Thus in thesecases, we observe that there is no monotone relationshipbetween these metrics and defect proneness.
(B2) Defect Density:
1) In all three version of Eclipse and Mylyn, we canobserve that the relationship between size based metrics
and defect density, is a monotonically decreasing one.
The higher the risk category that a file is in withrespect to size based metrics, the lower is its defectdensity. However, when we look at the complexity based
metrics for these two projects, we observe that there isa inverted-U shaped curve. This indicates that the files
with medium values for the complexity based metrics,
have a higher defect density than files with extreme
complexity values.
2) When looking at Netbeans, we can see that in version 5.0
and 5.5.1, the relationship between the various metrics
and defect density is all over the place. There is no
consistent trend. However, in version 4.0, we can clearly
see that except for cyclomatic complexity, the other three
metrics have a rotated-Z curve. We can see that thedefect density of the files in the very high risk category
are slightly higher than the defect density of files in themedium risk category. However, as observed in Eclipse
and Mylyn, the defect density of the files in the very highrisk category (the highest), is much lower than the defectdensities in both the medium and high risk categories.
3) In the industrial systems, we see that the files with thehighest value for module interface size (accessors in thiscase) are also the files with the highest defect density(except Industrial 110). Also except in Industrial 208,
the files with the largest value for LOC seems to havethe lowest defect density.
As seen in the result of this subsection, the relationshipbetween defect density and metric threshold values are notconsistent. To quantify if the defect density is indeed higher forthe very high risk values of each metric, we model code defects
via a poisson model that includes two predictors (independent
variables): the amount of code in the set of files within a risk
class and an indicator function for the highest risk threshold.
The response (dependent variable) is the number of defects
observed for that set of files. The fitted coefficient at the very
high risk indicator would allow us to determine if the files in
the highest risk categories are indeed more defect prone thanother files. We present the results in Table V. In the table,
z value is estimated coefficient value divided by the valueof standard error and P r(> |z|) shows significance of the z
value.
From the Table V we can see that in almost all the projects
and for all the metrics the z value is negative, indicating thatthe files in the highest risk category for all the metrics haveindeed lower defect density. However, only in the case of LOC
and cyclomatic complexity (and only in OSS projects), arethese results statistically significant. In the rest of the cases,
there is no statistical evidence to show the inverse relationshipbetween the metrics and defect density.
(B3) Summary:
From Subsection V-B, we can see that defect pronenessof files in a category has an almost linear relationship to
the values that correspond to that risk category. Hence thisvalidates the bulk of the existing literature on defect prediction. However, when looking at the observations made in

●

●

High

Very HighLow

MediumRisk Categories

High

Very High

●
●

●

Low

MediumHigh

Risk Categories

0.5●

Very HighLow

MediumRisk Categories

High

Very High

0.000200.000100.00015

●

●

●
●
●

●

●

●

MediumHigh

Very High●

Risk Categories

●

●

●
●

Low

MediumHigh●

Very HighLow

Risk Categories(b) Industrial 113

Risk Categories(c) Industrial 140

0.1●0.0010●0.0005

0.4
0.2

0.3

●

0.0015●●

Defect Density = Sum(Defects)/Sum(TLOC)●

0.00005

0.20
0.10

0.15

Defect Density = Sum(Defects)/Sum(TLOC)

●

●

0.05

4e−04
3e−04
2e−04

●

●
●
●

(a) Industrial 110

●
●

●
●
●
●

Low

Medium●●0.0000●0.0

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

Defect Density = Sum(Defects)/Sum(TLOC)
●

●

1e−04

0.30
0.25
0.20
0.15
0.10

●

0.05

3e−04

●

●

●

0.00000

Medium

●

●

0.00

Low

●

●

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

●

2e−04

Defect Density = Sum(Defects)/Sum(TLOC)

●

●
●

●
●
●

1e−04

0.10

●

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

4e−04

0.25
0.20
0.15

●

0.05

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

●

●

High

Very High●Low

MediumRisk Categories

High

Very High

Risk Categories(d) Industrial 208
Fig. 2. Defect Proneness and Defect Density of each risk category in each case study system
TABLE Vz AND P r(> |z|) OF THE INTERCEPT OF THE POISSON MODEL . VALUES WITH * NEXT TO THEM INDICATE STATISTICAL SIGNIFICANCE WITH p < 0.05

Eclipse 3.0
Eclipse 3.1
Eclipse 3.2
Mylyn 1.0
Mylyn 2.0
Mylyn 3.0
Netbeans 4.0
Netbeans 5.0
Netbeans 5.5.1
Industrial 110
Industrial 113
Industrial 140
Industrial 208

Lines of Codez
P r(> |z|)-9.56
1.19e-21*-15.316.10e-53*-7.0012.53e-12*-2.48
0.013*
-1.20
0.23
-2.33
0.02*
-3.78
0.02e-02*
-2.09
0.04*
-4.10
4.15e-05*
-2.19
0.03*
-1.26
0.21
-0.05e-02
0.99
1.05
0.30

Module Interface Sizez
P r(> |z|)-5.16
2.50e-07*-8.90
5.61e-19*
-0.52
0.60
-0.24
0.81
-0.24
0.81
-4.46
8.23e-06*
-1.56
0.12
-0.43
0.67
-0.26
0.80
-2.88
0.04e-01*
2.33
0.02*
-0.06
0.95
1.93
0.05

Subsection V-B, we can conclude that in almost all cases
the files in the highest risk category have the lowest defectdensities. Additionally in many cases, the files in the lowest
risk category also have low defect densities. This observation
is in contradiction to the “Goldilocks Conjecture”, which statesthat it is ideal for files to not have metric values in either
extremes.
VI. R ELATED W ORK
In this paper, we derive thresholds using the techniqueproposed by Alves et al. [3]. More details about this waspresented in Section III. However, in this section, we present
some of the other research related to thresholds as well assome of the research that questions such thresholds.
A. Techniques for Deriving Thresholds
In software engineering research there have been many
attempts to define thresholds for various metrics. The rationale
for doing so is to provide guidelines for practitioners to

Cyclomatic Complexityz
P r(> |z|)
-7.14
9.05e-13*-9.41
4.96e-21*-8.11
5.02e-16*-4.12
3.67e-05*-2.95
0.03e-01*-2.98
0.03e-01*
0.34
0.73
-1.88
0.06
-2.90
0.04e-01*
-1.32
0.19
0.25
0.81
0.34
0.73
-0.09e-02
0.99

Module Inward Couplingz
P r(> |z|)-3.70
0.02e-02*-7.37
1.72e-13*-6.63
3.40e-11*
-2.10
0.04*
-0.88
0.38
-1.13
0.26
-2.28
0.02*
-0.66
0.51
-1.18
0.24
1.45
0.15
1.91
0.06
-0.05e-02
0.99
2.85
0.04e-01*

identify poor quality code in their software. Such thresholds
can be derived based on theory, experience, or empiricalanalysis of metrics.
Researchers have argued about fault densities from a theoretical perspective [10, 17, 18]. Emam et al. theoreticallyshow that smallest files having the largest defect densities
are a mathematical artifact. Hatton argues that the software
components that “fit best into human short-term memory
cache” will produce the lowest defect densities [17, 18].
Hence, both small and large files will tend to have higher
defect densities.
Some of authors defined thresholds by their experience. McCabe [23] defined cyclomatic complexity and recommended
the threshold for the McCabe complexity metric to be 10
based on his experience. Nejmeh [25] defined a metric called
NPATH, an objective measure of software complexity, whichcounts the number of acyclic execution paths through afunction. Nejmeh defined a threshold of 200 for the NPATH

Very High

High

●
●

●

●

●

MediumHigh

Very HighLow

Risk Categories

MediumHigh

Risk Categories(g) Netbeans 4.0

Very High

●
●

●
●

●

MediumHigh

Very High

MediumHigh

Very High

●

●

●
●
●

Risk Categories

0.00140.0010

Defect Density = Sum(Defects)/Sum(TLOC)

0.0012

Very High

0.6●

0.00350.00300.0025●●●●0.0020●

Defect Density = Sum(Defects)/Sum(TLOC)

0.4
0.3

●

●
●

0.0015

0.5

●

●

●

Low

MediumHigh

Very High●Low

MediumHigh

Very High

Risk Categories

MediumHigh

Very High

0.12(f) Mylyn 3.0

●

Low

High

●

Risk Categories

●

●

Low

Medium0.0010

●

●

●

0.10

●

0.08

0.08
0.06

●

Low

Risk Categories

0.2

●

●

●
●

Very High

●

0.1

●

Low

0.00080.0030

0.4
0.3
0.2

High

●

Risk Categories

0.04

●

●

Low

●

Very High

0.02

4e−04
3e−04

Defect Density = Sum(Defects)/Sum(TLOC)

●

2e−04

●

●

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

5e−04

0.14
0.12
0.10
0.08
0.06
0.04

●

0.02

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)●

Medium(e) Mylyn 2.0

●

●

●

Risk Categories(d) Mylyn 1.0●

0.0025

0.9
0.8
0.7
0.5

0.6

MediumRisk Categories

●

●

●

Low

●

●
●
●

●
●
●

●

Low

Risk Categories(h) Netbeans 5.0

0.00030

Risk Categories

High

●

●

●

(c) Eclipse 3.2
Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

Medium●●●Risk Categories

0.06

Low

Low

0.04

Very High

●

●

●

0.02

High

Very HighProbability of finding a file with a Defect= Num(Files with defects)/Num(Files)

Medium

High

0.004

●

0.003

●

●
●

0.4

●

●

●

Low

Medium●●●●Risk Categories

●

0.3

0.006
0.005

●
●

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

●

●

0.004

0.6
0.5
0.4

●

Defect Density = Sum(Defects)/Sum(TLOC)

0.007

0.7

●

0.3

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

●

●

Low

●

(b) Eclipse 3.1●●

Very High

0.1

0.0035High

Risk Categories●

0.00025

Medium(a) Eclipse 3.0●

0.0020

0.4
0.3
Low

●

●

●

●

0.00020

Very High●

Defect Density = Sum(Defects)/Sum(TLOC)High

Risk Categories

●

●

●

●

●

0.00015

Medium

0.010

Low

●

0.008

Very High

0.006

High

Risk Categories●●

Defect Density = Sum(Defects)/Sum(TLOC)

Medium●●Defect Density = Sum(Defects)/Sum(TLOC)

Low

●

●

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

●
●

●
●

●

0.0015●

Defect Density = Sum(Defects)/Sum(TLOC)●●

0.0010●0.2

0.00200.0015

●

●
●

●
●

0.00008 0.00010 0.00012 0.00014 0.00016 0.00018

●

●

●

0.1

●
●

Defect Density = Sum(Defects)/Sum(TLOC)●

0.0010

0.25
0.15

0.20

●

●

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)

0.0025

0.30

●

●

0.10

Probability of finding a file with a Defect= Num(Files with defects)/Num(Files)●●

MediumHigh

Very HighLow

Risk Categories

MediumHigh

Very High

Risk Categories(i) Netbeans 5.5.1

Fig. 3. (contd.) Defect Proneness and Defect Density of each risk category in each case study system

metric based on his experience [25]. Coleman et al. [8] defined the Maintainability Index (MI) metric and recommended
thresholds of 65 and 85. This implies that when software
components with MI values lower than 65, and higher than 85
are difficult to maintain, and between 65 and 85 are moderately
maintainable.
Some other authors have empirically evaluated thresholds
for the software metrics. For example, Erni et al. [11] defined
thresholds by using statistical techniques. Their thresholds T
are calculated by using mean (µ) and standard deviation ( )as T = µ + or T = µ. This technique was evaluated on
one system which had three version on metrics such as numberof methods in a class, and the lack of cohesion in a method.
Shatnawi et al. [26] derived thresholds of 12 metrics proposedby Chidamber and Kemerer, Li, and Lorenz and Kidd, by
empirically relating the metrics to error severity categories.
These thresholds were derived using the receiver operating
characteristic (ROC) curves.
In this paper, we use the technique proposed by Alves etal. since this is the only technique that takes the statistical
distribution inherently present in the data into account [3].

B. Questioning Thresholds
There has also be a few studies over the last three decadesthat have questioned the veracity of certain thresholds for
metrics, as well as the veracity of the concept of thresholds
themselves. Basili et al. [4], Compton and Withrow [9], andHatton [17, 18], all questioned the well accepted notion that
extensive modularization will reduce defects in the software.
They found evidence to the contrary. Their finding suggested
that small files in a software had a higher defect density than
medium sized files. This new observation is often called the“Goldilocks Conjecture” - files with atypical size are more
likely to have more defects than files of typical (medium)size. Additionally, Bell et al. [5] found that small files (lessthan 65 LOC) had high defect densities and that the expected
number of defects increased as the square root of the size
metric (instead of a linear increase). In our study, we find thatlarge and complex files have lower defect densities, whichis contrary to the findings in the studies above. Additonally
we also find that in some cases, files with smaller size andcomplexity as well has lower defect densities. This is exactly
the opposite of the “Goldilocks Conjecture” and is contrary tothe findings in the above studies.
Fenton and Neil [12], in their study of the literature on
defect prediction approaches, also found that there is research
evidence both supporting and refuting the relationship between
risk thresholds and size and complexity metrics. However, in
their study they do not empirically evaluate the contradictory
behaviour. They rely completely on past research. Although,it is an important first step, such a comparison has a limitation- the study design in each the past research studies could bevery different. Even they comment on the accuracy of thedata collection process in each of the past studies. Therefore
the contradictory results could be due to the different study
designs and not due to the actual relationship between risk
thresholds and metrics. In our paper, we consistently use thesame process (described in Sections III and IV), to analyze
all the datasets. Hence we are able to empirically verify
if risk thresholds have a consistent relationship to size andcomplexity metrics.
Emam et al. questioned if any kind of thresholds forsoftware size was reasonable [10]. They empirically show that
there is no statistical evidence to prove that files beyond acertain threshold for size was indeed more defect prone. In
the following years, the same authors found move evidence
that the size of a class and defect proneness had a power law
relationship [20, 21].
Our approach differs from Emam et al. [10], and Koru etal. [20, 21] in the following ways:• In addition to software size based thresholds, we also
look at complexity based thresholds.
• In addition to examining the relationship to defect proneness we also investigate the relationship to defect density
as well.
• Instead of asking the question: is there a threshold for size
above which files are more defect prone, we ask: are files
benchmarked as high-risk indeed the most defect prone.
• Instead of obtaining a threshold after determining the relationship between quality and size for the whole dataset,
we replicate a latest threshold-setting technique.
In summary, the focus of the past studies have been to
examine the relationship between a metric like size and quality.
Whereas, in our study we focus on determining if thresholds
for metrics can reliably identify highly-defect prone and highdefect-density files.
From the perspective of results and conclusions, Koru etal. [20, 21] in their work found that, the defect pronenessdoes increase with the size of the software component, butthat the rate of this increase slows down (hence having a
sub linear power law relationship). Therefore they indirectly
come to the conclusion that large files have lower defectdensity (indirectly because only defect proneness is examined
and not defect density). We arrive at a similar conclusion as
them. However, we directly relate defect density to size andcomplexity metrics. Hence, this result from our case study
reinforces the conclusions of Emam et al. [10], and Koru etal. [20, 21]. Additionally we find that in some cases, files withsmall values for the four metrics also have low defect density.

It is the files with medium values for the various metrics thathave the highest defect density.
There are other types of thresholds, apart from ones weconsider here. For example, code smells [13] are particularpatterns that are considered to reflect bad design. Sjøberg etal. [29] investigated if smell thresholds for individual files
were related to maintenance effort spent on these files andfound that there was no relatiobnship if the size of file (LOC)
was taken into account. The results we obtained (even though
we did not consider several thresholds simultaneously) appear
to be consistent with that finding: the high size risk factor was
more strongly related to fault-proneness than high complexity
risk factor. Other work by the same group [28], found that only
the overall size of the system was associated with maintenanceeffort: the smallest system (among four systems implementing
equivalent functionality) was the easiest to maintain despite
having much of its code in a single “God” class (one of thecode smells evaluated in that work).
VII. T HREATS TO VALIDITY
External Validity: We evaluated our goal with Javaprojects. Hence, the results may not generalize to projects
in different programming languages. To make the study as
broad as possible, we examine both OSS (3 versions each of
3 projects) and industrial (four) projects. Additionally for thethresholds we examined 4,575 OSS Java projects. Although,the generalizability of this study can further be improvedon repeating it on many more projects, the current size ofthe case study is the typical size in software defect related
literature [27].
Also, in this study we only examine the size and complexity
metrics, and as such all our conclusions are limited to them.
Further studies on other metrics like churn and pre-releasebugs must be conducted before any claims about their relationship to defects can be made. Since there are a large number ofsoftware metrics for which the relationship to defects has beenstudied [27]. Since it is not possible to study all the metrics inthis paper, we chose to start with four of them. We have tried
to provide all the details of our study in as clear and concise amanner as possible, so that it can be replicated easily on otherprojects or just other metrics too.
We used one specific approach to determine the thresholds.
Other thresholds may not yield exactly the same results,
but we found that simply picking the largest or the mostcomplex files yields very similar results. We, however, chose atechnique that is empirically based, and takes into account thestatistical distribution properties of the file metrics. We derive
the thresholds from 1000s of OSS and 205 industrial projects.
Internal Validity: The metrics were extracted from the
projects using Understand [2], and Sonar [1]. Any error inthese tools can greatly affect the values of the thresholds.
However both tools are standard tools and various studies havebeen carried out using these tools. Furthermore, even if theexact thresholds are slightly changed, the main findings thatthe top-risk group is the most fault-prone and that it has thelowest defect density, are unlikely to change.

In addition to errors in tools, the difference in tools is alsoa threat to our results. The difference in tools may affectthe values of metrics and thresholds. The difference in tools
happened only between OSS and industrial projects (i.e., weused Understand to OSS projects and Sonar to industrial
projects). Hence, we believe that we obtained consistent results during each analysis (OSS and industrial projects). The
larger problem here is the difference in organizational culture
between OSS and industrial projects.
The defect data collection technique is at a commit level andtherefore might have both false positives and false negatives.
Also, industrial projects consider only customer-reported defects, while OSS projects also include alpha/beta testing, anddevelopment defects as it was not practical to separate defects
reported only on stable releases for the studied OSS projects.
However, such a commit based approach has been the standard
in defect prediction literature [7, 19, 27, 30].
Construct Validity: All our results rely on the assumptionthat the 70%, 80% and 90% percentiles of the overall codeare good enough for deriving metrics (which is reported by
Alves et al. [3]). However, we find that our results as far as
metrics are concerned, are not that different from what Alveset al. [3] reported.
In Section IV-C, we chose 1,000 projects that had themost similar Median LOC/File to the OSS projects for which
we had defect data, in order to derive the thresholds. The
choice of representative projects based on Java language andthe median of the distribution of project file sizes may haveaffected our results. It may be that by picking similar projects
based on other quantiles or based on the domain, size or othercharacteristics, would have changed the results. Hence theresults should be interpreted within the context: Java projects
whose risk thresholds are derived from other Java projectsof similar median file size. Further research is needed todetermine if the risk thresholds have a consistent relationshipto defect density for other languages or for a specific subset
of Java projects. We used the median LOC/File to identify
the set of projects that could be used to derive the thresholds.
Because we removed the smallest projects from our dataset(containing fewer than 200 files), we believe that choice to bea reasonable compromise between the sample size and project
similarity.
VIII. C ONCLUSION
In summary we derived the thresholds for three OSS
and four industrial projects and assigned the files having aparticular value for a metric to a particular risk category
depending on the threshold values. Using the defect data forthese projects we examined the relationship between the file
metrics in each risk category and the defect proneness anddefect density of the files. We found that typically defectproneness values monotonically increase with the metric values. However, defect density values typically demonstrate atrend that in some cases appears to contradict the “Goldilocks
Conjecture”, and in some other cases just has low values only
for the files in the very high risk categories for size andcomplexity metrics. However, we also found some instances
when files with small size and complexity metrics had low
defect densities. Therefore, although we found some support
for findings in recent literature [20, 21, 33] that smaller files
have higher defects density, we found further evidence thatvery large or complex files have lower defect densities and insome cases even lower defect proneness.
In summary, a) we found support for the approach toidentify defect-prone files via size thresholds. The complexity
thresholds, however, did not work as well, b) we did not findsupport for approaches that would reduce the size of the largest
files by moving that code into other smaller or less complex
files. On the contrary, such approaches may increase the risk,
and therefore be counterproductive. Hence in conclusion, we
found no support for the idea that files with higher defectdensities can be identified via thresholds of basic size andcomplexity metrics. Therefore, any software practitioner whochoses to use such thresholds must proceed with caution. We
believe that as a result of our study, we as software engineeringresearchers, need to examine the underlying reasons behind therelationship between existing metrics and software defects, andneed to explore other metrics that could be more consistent
with respect to defect densities in files.
IX. ACKNOWLEDGEMENTS
This research was partially supported by JSPS KAKENHI
Grant Numbers 15H05306.
R EFERENCES[1] Sonar, http://www.sonarsource.org/, 2015.
[2] Understand, http://www.scitools.com/, 2015.
[3] T. L. Alves, C. Ypma, and J. Visser. Deriving metricthresholds from benchmark data. In Proc. Int’l Conf. on
Software Maintenance (ICSM), pages 1–10, 2010.
[4] V. R. Basili and B. T. Perricone. Software errors andcomplexity: an empirical investigation. ACM Commun.,27(1):42–52, 1984.
[5] R. M. Bell, T. J. Ostrand, and E. J. Weyuker. Looking
for bugs in all the right places. In Proc. Int’l Symposium
on Software Testing and Analysis (ISSTA), pages 61–72,
2006.
[6] S. Benlarbi, K. E. Emam, N. Goel, and S. Rai. Thresholds
for object-oriented measures. In Proc. Int’l Symposium
on Software Reliability Engineering (ISSRE), pages 24–
38, 2000.
[7] T.-H. Chen, S. Thomas, M. Nagappan, and A. Hassan.
Explaining software defects using topic models. In Proc.
Int’l Working Conf. on Mining Software Repositories(MSR), pages 189 –198, 2012.
[8] D. Coleman, B. Lowther, and P. Oman. The applicationof software maintainability models in industrial softwaresystems. J. Syst. Softw., 29(1):3–16, 1995.
[9] B. T. Compton and C. Withrow. Prediction and controlof ada software defects. J. Syst. Softw., 12(3):199–207,
1990.

[10] K. El Emam, S. Benlarbi, N. Goel, W. Melo, H. Lounis,
and S. N. Rai. The optimal class size for object-oriented
software. IEEE Trans. Softw. Eng., 28(5):494–509, 2002.
[11] K. Erni and C. Lewerentz. Applying design-metrics toobject-oriented frameworks. In Proc. Int’l Symposium on
Software Metrics (METRICS), pages 25–26, 1996.
[12] N. E. Fenton and M. Neil. A critique of software defectprediction models. IEEE Trans. Softw. Eng., 25(5):675–
689, Sept. 1999.
[13] M. Fowler. Refactoring. Improving the Design of Existing
Code. Addison-Wesley, 1999.
[14] V. French. Establishing software metric thresholds. In
Proc. Int’l Workshop on Software Measurement (IWSM),
1999.
[15] T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy.
Predicting fault incidence using software change history.
IEEE Trans. Softw. Eng., 26(7):653–661, 2000.
[16] T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell.
A systematic literature review on fault prediction performance in software engineering. IEEE Trans. Softw. Eng.,38(6):1276–1304, Nov. 2012.
[17] L. Hatton. Reexamining the fault density-component size
connection. IEEE Software, 14(2):89–97, 1997.
[18] L. Hatton. Does OO sync with how we think? IEEE
Software, 15(3):46–54, 1998.
[19] Y. Kamei, S. Matsumoto, A. Monden, K. Matsumoto,B. Adams, and A. E. Hassan. Revisiting common bug
prediction findings using effort aware models. In Proc.
Int’l Conf. on Software Maintenance (ICSM), pages 1–
10, 2010.
[20] A. Koru, D. Zhang, K. El Emam, and H. Liu. An
investigation into the functional form of the size-defectrelationship for software modules. IEEE Trans. Softw.
Eng., 35(2):293–304, 2009.
[21] G. Koru, H. Liu, D. Zhang, and K. E. Emam. Testing
the theory of relative defect proneness for closed-source
software. Empirical Softw. Eng., pages 577–598, 2010.
[22] S. Matsumoto, Y. Kamei, A. Monden, and K. Matsumoto. Comparison of outlier detection methods in
fault-proneness models. In Proc. Int’l Symposium on Empirical Software Engineering and Measurement (ESEM),
pages 461–463, 2007.
[23] T. McCabe. A complexity measure. IEEE Trans. Softw.

Eng., SE-2(4):308 – 320, 1976.
[24] A. Mockus. Amassing and indexing a large sample ofversion control systems: Towards the census of public
source code history. In Proc. Int’l Working Conf. on
Mining Software Repositories (MSR), pages 11–20, 2009.
[25] B. A. Nejmeh. Npath: a measure of execution path complexity and its applications. ACM Commun., 31(2):188–
200, 1988.
[26] R. Shatnawi, W. Li, J. Swain, and T. Newman. Finding
software metrics threshold values using ROC curves. J.
Soft. Maint. Evol., 22(1):1–16, 2010.
[27] E. Shihab. An exploration of challenges limiting pragmatic software defect prediction. PhD Thesis, Schoolof Computing, Queen’s University, Kingston, Ontario,
Canada, 2012.
[28] D. I. Sjøberg, B. Anda, and A. Mockus. Questioning software maintenance metrics: a comparative casestudy. In Proc. Int’l Symposium on Empirical Software
Engineering and Measurement (ESEM), pages 107–110,
2012.
[29] D. I. Sjoberg, A. Yamashita, B. Anda, A. Mockus, andT. Dyba. Quantifying the effect of code smells on maintenance effort. IEEE Trans. Softw. Eng., 99(PrePrints):1,
2013.
[30] J. Śliwerski, T. Zimmermann, and A. Zeller. When dochanges induce fixes? In Proc. Int’l Conf. on Mining
Software Repositories (MSR), pages 1–5, 2005.
[31] D. Spinellis. A tale of four kernels. In Proc. Int’l Conf.
on Software Engineering (ICSE), pages 381–390, 2008.
[32] R. Vasa, M. Lumpe, P. Branch, and O. Nierstrasz.
Comparative analysis of evolving software systems using
the gini coefficient. In IEEE Int’l Conf. on Software
Maintenance (ICSM), pages 179 –188, 2009.
[33] E. J. Weyuker, T. J. Ostrand, and R. M. Bell. Do too
many cooks spoil the broth? using the number of developers to enhance defect prediction models. Empirical
Softw. Eng., 13(5):539–559, 2008.
[34] K.-A. Yoon, O.-S. Kwon, and D.-H. Bae. An approach tooutlier detection of software measurement data using the
k-means clustering method. In Proc. Int’l Symposiumon Empirical Software Engineering and Measurement(ESEM), pages 443–445, 2007.