Assessing the Differences of Clone Detection
Methods Used in the Fault-prone Module Prediction
Masateru Tsunoda

Yasutaka Kamei

Atsushi Sawada

Kindai University
Higashiosaka, Japantsunoda@info.kindai.ac.jp

Kyushu University
Fukuoka, Japankamei@ait.kyushu-u.ac.jp

Nanzan University
Nagoya, Japan
sawada@se.nanzan-u.ac.jp

Abstract—We have investigated through several experiments
the differences in the fault-prone module prediction accuracy
caused by the differences in the constituent code clone metrics ofthe prediction model. In the previous studies, they use one ormore code clone metrics as independent variables to build anaccurate prediction model. While they often use the clone
detection method proposed by Kamiya et al. to calculate these
metrics, the effect of the detection method on the prediction
accuracy is not clear. In the experiment, we built prediction
models using a dataset collected from an open source software
project. The result suggests that the prediction accuracy isimproved, when clone metrics derived from the various clone
detection tool are used.
Keywords— Clone metrics; fault-prone module; prediction
performancedetected clones. Since we focus on the effects caused bydifferent clone metrics used in a fault-prone module prediction
model, we will take the differences in method configurations to
be same as the differences in detection methods.
Existing studies use clone metrics [2] calculated using thecode clones resulting from the detection method proposed by
Kamiya et al. [3]. On the other hand, changes in the prediction
accuracy are not clear if we alter the prediction model so that it
uses another method for code clone detection. Also, thechanges in prediction accuracy is not clear when the model
uses various clone metrics which is derived from different
detection methods simultaneously. In this paper, we assess thedifferences in the prediction power through several
experiments in which we use the model comprised of various
clone metrics calculated from various clone detection methods.

I. INTRODUCTION
To enhance software reliability, it is important to find andremove faults in the testing phase. Predicting fault-prone
modules is recognized to be a promising approach to finding
software faults, and many fault-prone module prediction
models are proposed in the software engineering field. Whenbuilding the prediction model, selecting proper independent
variables is important to enhance prediction accuracy.
When selecting independent variables of a fault-prone
prediction model, we investigate each software metric such as
a complexity metric of the source code from a viewpoint of therelevance to software faults. Existing studies suggested thatcode clone related metrics are some of the effective alternatives[4], and hence using the clone metrics (e.g., the ratio of thelines of duplicated code per the lines of code) is expected toenhance the performance of fault prediction models. In theliterature [2], they tried to use clone metrics as independentvariables of the prediction model, to enhance the performanceof the model.
Many code clone detection methods have been proposed[4], since code clones affect software maintainability. Each
resulting set of code clones differs from another if we use
different detection methods or if we use different
configurations of one detection method [5]. Relevant code
clone metric also differs from one to another. For instance, ifwe set a smaller value to the ‘minimum size of detected clone’
parameter, we will get a larger value of the ‘ratio of clones tolines of code’ metric because of the increasing number of

II. EXPERIMENT
Overview: To assess the clone metrics the effect of clone
metrics which are derived from different detection methods, wehave built several different fault-prone module prediction
models using these clone metrics, and compared the prediction
accuracy. We used dataset measured on open source software,
Lucene 2.4.0. The dataset was collected by D’Ambros et al. [1],and it includes complexity metrics and the number of faults
after the release. We measured clone metrics from the source
code, and merged it to the dataset. The number of modules was
691, and the number of fault modules was 64.
Clone detection tools and metrics: We used four clone
detection tools which are widely used [5]: PMD’s Copy/Paste
Detector (CPD), CCFinderX, Simian, and NiCad3. CPD andCCFinderX is token based, Simian is text based, and NiCad3
is parser based approach. We made two configuration settingson each tool as shown in Table I. One is to find small size
TABLE I.
Clone
metricc1c2c3
c4
c5c6c7c8

CLONE DETECTION TOOLS AND METRICS

Tool

Configuration

CCFinderX
CCFinderX
SimianSimianCPD
CPD
NiCad3# of minimum tokens is 50# of minimum tokens is 100# of minimum LOC is 6
# of minimum LOC is 9, and ignore identifiers# of minimum tokens is 100# of minimum tokens is 200, and ignore literals
block clone, and type 2 (renamed) clone
function clone, and type 3-2c (near-missrenamed consistently) clone

NiCad3

TABLE II.

clones, and the other type is to find large size clones.

Metricc1c2c3
c4
c5
c6
c7
c8

Based on the outputs from these tools, we have calculated
the values of clone metrics c1...c8, which are either the numberof duplicated lines of code divided by total number of lines ofcode (total NLOC), or the number of duplicated tokens divided
by total NLOC. Table 2 shows Spearman’s correlation matrixof the clone metrics. There was not very high correlation (Themaximum value was 0.74).
Variety of models: We have built eight prediction models
M1…M8 using each metric c1…c8 respectively. In addition toclone metrics, we used 17 variables as candidates ofindependent variables. They are CK metrics and other object
oriented metrics. We have also built the model M9 which
includes all of c1…c8 metrics. In order to make a comparison
with the effects caused by conventional metrics such as CKmetrics, we have built the model M10 which includes none of
c1…c8 metrics. Existing studies such as [2] which haveadopted clone metrics in prediction models show theimprovement of prediction accuracy is limited. We have built
M9 since we assume when various detection methods areapplied, code clone is characterized from various points ofview.
Building and evaluating prediction models: In faultprone module prediction, a model discriminates whether each
module will have faults or not after software release (i.e., when
the number of faults after the release is more than zero, themodule has faults). To build the models, we used the logistic
regression, which is widely used in fault-prone module
prediction.
Candidates of independent variables were selected based on
Akaike’s Information Criterion (AIC). When correlation
between independent variables was very high, one of thevariables was removed from the candidates. In model M1 ...
M9, clone metrics were not removed from the model. Note thatmetric c6 was removed from M9 because of high correlation.
To evaluate the models, we used the area under the curve(AUC). AUC is the more appropriate criterion for discriminant
methods than other criteria like F1 score. The value range ofAUC is [0, 1], and higher AUC means that prediction accuracyof the method is high. To calculate the evaluation criterion, weapplied 5-fold cross validation (In each subset, the rate of faultprone modules was same), and repeated it two times to increase
the number of evaluations.
Result: The result is shown in Table 3. We statisticallytested the difference from M10, using the Wilcoxon signedrank test at a significance level of 0.05. In the table, boldedrows indicates there was statistical difference. Focusing on M1… M8, M1 and M3 had statistical difference, although thedifference is small (0.03). On configurations of c1 and c2,minimum length of clone is small. So, in this dataset, it may begood that the minimum length of clone is small. So, choosing
proper detection tools and configurations to derive clone
metrics would not be ignorable to improve prediction accuracy.
Compared with M1 … M8, M9 had relatively large
difference from M10 (0.05), and it was statistically different.
So, using the sole detection tool is not very effective, but usingc110.740.470.51
0.52
0.67
0.65
0.34

CORRELATION MATRIX OF THE CLONE METRICS
c2
0.74
1
0.54
0.55
0.41
0.55
0.60
0.30

TABLE III.
Model
M1
M2
M3
M4
M5
M6
M7
M8
M9
M10

c3
0.47
0.54
1
0.62
0.24
0.45
0.48
0.25

c4
0.51
0.55
0.62
1
0.28
0.50
0.52
0.29

c5
0.52
0.41
0.24
0.28
1
0.57
0.48
0.10

c6
0.67
0.55
0.45
0.50
0.57
1
0.62
0.26

c7
0.65
0.60
0.48
0.52
0.48
0.62
10.34

c8
0.34
0.30
0.25
0.29
0.10
0.26
0.34
1

PREDICTION ACCURACY OF THE MODELS

Clone
metrics
c1c2
c3
c4
c5
c6
c7
c8
c1...c8
-AUC
0.69
0.67
0.69
0.66
0.67
0.65
0.67
0.66
0.71
0.66

Difference
from M10
0.03
0.01
0.03
0.01
0.01
0.00
0.01
0.00
0.05
-p-value
0.02
0.24
0.02
0.53
0.08
0.88
0.37
0.75
0.02

various tool and configurations to derive clone metrics would
be effective, compared with using only conventional metrics.
III. CONCLUSIONSIn this paper, we compared the prediction accuracy of faultprone module prediction models, when using clone metrics
derived from the various clone detection tool andconfigurations. The experimental result showed using variousclone metrics would be effective, compared with using only
conventional source metrics such as CK metrics.
ACKNOWLEDGMENT
This research was partially supported by the Japan Ministryof Education, Science, Sports, and Culture [Grant-in-Aid for
Scientific Research (C) (No. 25330090)], and by Nanzan Univ.
Pache Research Subsidy I-A (2015).
REFERENCES
[1]

[2]

[3]

[4]

[5]

M. D’Ambros, M. Lanza, and R. Robbes, “An extensive comparison ofbug prediction approaches,” Proc. international working conference onmining software repositories (MSR), pp.31-41, 2010.
Y. Kamei, H. Sato, A. Monden, S. Kawaguchi, H. Uwano, M. Nagura,
K. Matsumoto, and N. Ubayashi, “An Empirical Study of Fault
Prediction with Code Clone Metrics,” Proc. Joint Conf. of the Int.
Workshop on Software Measurement and Int. Conf. on Software Process
and Product Measurement (IWSM-MENSURA) , pp.55-61, 2011.
T. Kamiya, S., Kusumoto, and K. Inoue, “CCFinder: a multilinguistic
token-based code clone detection system for large scale source code,”IEEE Transactions on Software Engineering, vol.28, no.7, pp.654-670,
2002.
D. Rattan, R. Bhatia, and M. Singh, “Software clone detection: A
systematic review,” Information and Software Technology, vol.55, no.7,pp.1165-1199, 2003.
T. Wang, M. Harman, Y. Jia, and J. Krinke, “Searching for better
configurations: a rigorous approach to clone evaluation,” Proc. joint
meeting of the European software engineering conference and the ACM
SIGSOFT symposium on the foundations of software engineering(ESEC-FSE), pp.455-465, 2013.