A Dataset of Clone References with Gaps
Hiroaki Murakami, Yoshiki Higo, and Shinji Kusumoto
Osaka University
1-5 Yamadaoka, Suita, Osaka, Japan{h-murakm,higo,kusumoto}@ist.osaka-u.ac.jp

ABSTRACTﬁnd code clones in software systems, many clone detectors
have been developed. Since each clone detector has its own
deﬁnition of code clones, diﬀerent code clones are detectedby diﬀerent clone detectors from the same source code [7].
Bellon et al. compared six clone detectors in order toreveal characteristics of them [1]. He made references ofcode clones, and compared accuracies among clone detectors. The references represent locational information of code
clones with ﬁle name and start/end lines. In his benchmark,code clones are categorized into the following three types.

This paper introduces a new dataset of clone references,which is a set of correct clones consisting of their locational
information with their gapped lines. Bellon’s dataset is oneof widely used clone datasets. Bellon’s dataset contains
many clone references, thus the dataset is useful for comparing accuracies among clone detectors. However, Bellon’s
dataset does not have locational information of gapped lines.
Thus, Bellon’s benchmark does not evaluate some Type-3
clones correctly. In order to resolve the problem, we added
locational information of gapped lines to Bellon’s dataset.
The new dataset is available at “http://sdl.ist.osaka-u.
ac.jp/~h-murakm/2014_clone_references_with_gaps/”.
This paper also shows some examples that the new dataset
and Bellon’s dataset yield diﬀerent evaluation results. Moreover, we report an experimental result that compares Bellon’s dataset and the new dataset by using three clone detectors that can detect Type-3 clones. Finally, we conclude
that the new dataset can evaluate Type-3 clones more correctly than Bellon’s dataset.

Type-1 is an exact copy without modiﬁcations (except for
white space and comments).
Type-2 is a syntactically identical copy; only variable, type,or function identiﬁers were changed.
Type-3 is a copy with further modiﬁcations; statements
were changed, added, or removed.
Programmers often make some changes to code fragments
after cloning [5]. Moreover, cloned fragments often evolve
diﬀerently from the original fragments [3]. These facts indicate that there often exists some gaps between the originalcode fragments and pasted fragments. Thus, it is important
to ﬁnd Type-3 clones for software maintenance.
However, Bellon’s Type-3 clones do not have locational
information of gapped lines. In other words, Bellon’s references represent code clones only with information of line
numbers. We do not consider that gapped lines in code
clones should be regarded as code clones. Thus, Bellon’s
benchmark does not evaluate some Type-3 clones correctly.
Therefore, we remade the references of code clones with locational information of gapped lines. Moreover, we conducted
experiments to indicate diﬀerences between Bellon’s references and the new references by using three clone detectorsthat can detect Type-3 clones.
Contributions of this paper are as following.

Categories and Subject Descriptors
D.2.7 [Software Engineering]: Distribution, Maintenance,and Enhancement; E.5 [Data]: Files

General Terms
Design, Measurement

Keywords
Code clone, Software maintenance, Dataset

1. INTRODUCTION
Many software systems have code clones. Code clones are
identical or similar code fragments to one another in sourcecode. Recent research revealed that some code clones havenegative impacts on software maintenance [2][3]. In order to• We added locational information of gapped lines to
Bellon’s references of code clones and made it public.
• It is revealed that our dataset produces diﬀerent results from Bellon’s dataset.

Permission to make digital or hard copies of all or part of this work for personal orclassroomusetois makegranteddigital
withoutorfeehard
provided
thatofcopies
not madeor work
distributed
Permission
copiesall orarepartof thisfor
for
profit ororcommercial
and that without
copies bear
notice andthecopies
full citation
personal
classroomadvantage
use is granted
feethisprovidedthatareon the first page. Copyrights for components of this work owned by others than ACM
not madeor distributed
forwithprofitor iscommercial
and thatcopiesmust
be honored.
Abstracting
creditpermitted. Toadvantage
copy otherwise,or republish,
bear
full citation
onrequires
the firstpriorpage.
To copyotherwise,topostthison noticeservers andor totheredistributeto lists,
specific
permission
and/ortoa
fee.
Requesttopermissions
from Permissions@acm.org.
republish,
post on servers
or to redistribute to lists, requires prior specific2.

BELLON BENCHMARK

Bellon’s benchmark is one of the most famous benchmark
in clone community. He compared six clone detectors fromthe perspective of accuracy and performance. He made thebenchmark with the following steps.

permission
and/orfee. 1, 2014, Hyderabad, India
MSR’14,May
31 –aJune
MSR ’14, May- June1, 2014, Hyderabad, India
Copyright
201431
ACM
978-1-4503-2863-0/14/05...$15.00
Copyright 2014 ACM 978-1-4503-2863-0/14/05 ....$5.00.
http://dx.doi.org/10.1145/2597073.2597133

412

㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌㻌end line 1
fine name 1
start line 1

file name 2

clone typeend line 2
start line 2

set of gapped line 2
set of gapped line 1

eclipse-jdtcore/src/internal/compiler/codegen/CodeStream.java 703 712 eclipse-jdtcore/src/internal/compiler/codegen/CodeStream.java 861 868 3 705, 706, 707 863

…
…

*
*
*

* gapped line703: final public void dstore_2() {704:
countLabels = 0;705:
stackDepth -= 2;706:
if (maxLocals < 4) {707:
maxLocals = 4; }
708:
try {
709:
position++;710:
bCodeStream[classFileOffset++] = OPC_dstore_2;
711:} catch (IndexOutOfBoundsException e) {712:
resizeByteArray(OPC_dstore_2); } }*861: final public void fcmpl() {862:
countLabels = 0;863:
stackDepth--;
864:
try {
865:㻌
position++;866:㻌
bCodeStream[classFileOffset++] = OPC_fcmpl;
867:} catch (IndexOutOfBoundsException e) {868:
resizeByteArray(OPC_fcmpl); } }…

…

Figure 1: An example of the new dataset and correspondent source ﬁlesclone reference

clone candidate* gapped line

…

…

*

172: return ep;
p }173: expr_ty *174: expr_function_new2(e1, e2)
175:
expr_ty*e1;176;*e2; {
expr_ty177:
expr_ty*ep;
178:
expr_function_ty *this;179:
ep = expr_private_new(&method);180:
this = (expr_function_ty *)ep;181:
expr_list_constructor(&this->children);182:
expr_list_append(&this->children, e1);183:
expr_list_append(&this->children, e2);184:
return ep; }

*
*
*

…

…

205: */
206: stmt_ty *207: stmt_assign_new(name, value)208:
expr_list_ty *name;209:
expr_list_ty *value; {210:
stmt_ty*sp;
211:
stmt_assign_ty*this;
212:
trace(("stmt_assign_new()¥n{¥n"/*}*/));
213:
sp = stmt_private_new(&method);214:
this = (stmt_assign_ty *)sp;215:
expr_list_copy_constructor(&this->name, name);216:
expr_list_copy_constructor(&this->value, value);217:
trace(("return %8.8lX;¥n", (long)sp));
218:
trace((/*{*/"}¥n"));219:
return sp; }Figure 2: An example of reference-candidate match with gapped line3.

Step-1: Bellon selected eight software systems as target
systems, and six clone detectors as target detectors.

DATASETWe made a new dataset by adding locational informationof gapped lines to Bellon’s dataset. The new dataset is available at our website1 . Furthermore, we put ﬁle format of ourclone references on the same website. Fig. 1 illustrates anexample of our clone reference and the correspondent sourceﬁles. The left source ﬁle has three gapped lines (lines 705,
706 and 707), and the right one has a single gapped line
(line 863). If our clone references are used in accuracy evaluation, the evaluation can take into account gapped lines incode clones.
On the other hand, Bellon’s clone references represents
code clones with ﬁle name and start/end lines. Thus, Bellon’s benchmark does not evaluate some Type-3 clones correctly. If our clone references are used in accuracy evaluation, ok and good value may be changed from the case of
Bellon’s clone references. Our clone references can evaluate Type-3 clones more correctly than Bellon’s clone references because our clone references introduce information of
gapped lines into evaluation of code clones.
Herein, we explain the case that ok value gets changed.
Fig. 2 shows that there are two clone pairs. One clone

Step-2: He asked the developers of the clone detectors todetect code clones from the software systems. Thenthe developers sent the locational information of thedetected code clones to Bellon.
Step-3: 2% of the code clones sent from the developers wererandomly selected, then he checked each of them manually whether it was actually code clone or not.
In the remainder of this paper, we use the following terms.
Clone candidates: code clones found by clone detectors.
Clone references: code clones judged by human manually.
In the Bellon’s benchmark, ok and good value are deﬁned.
For deﬁnitions of ok and good value, refer to the literature[1]. These two values decide whether every clone candidate
matches any of clone references or not. The ok value means
intuitively the overlapping ratio of a clone candidate and aclone reference. The more ok value increases, the more theoverlapping part of a clone candidate and a clone reference
becomes large. Meanwhile, the good value is much more
restrictive for a candidate-reference match.

1
http://sdl.ist.osaka-u.ac.jp/~h-murakm/2014_
clone_references_with_gaps/

413

600
6

9

10

Bellon's references

6

5

5

0
Scorpio

CDSW

NiCad

(a) okScorpio

0

CDSW

NiCad

7

12

9

our references

●

7
5

5
0

0
Scorpio

CDSW

NiCad

(a) okScorpio●(b) good

●
●
●
●

1,208
1,013

1000
542 542500

254

Scorpio CDSW

Scorpio

CDSW(a) ok

27 27

NiCad

Scorpio

CDSW(b) good

Figure 5: Experimental results for jdtcore
pair is one of clone references, and the other is one of clone
candidates that our clone detector CDSW [6] detected. If okvalue is calculated for this example by using Bellon’s clone
references, ok value is obtained as follows:
ok

=
=

●

NiCad

Scorpio CDSW(b) ant

Type-3 clones. All the clone detectors were used in theirdefault conﬁgurations. We used four Java systems shown in
Table 2 as targets software systems. The Java systems werealso used in Bellon’s experiments.
Fig. 3, Fig. 4, Fig. 5 and Fig. 6 show experimental resultsfor each target software system. Each graph represents thenumber of detected clone references whose ok and good values exceed the threshold. In the case of NiCad, the numberof the detected clone references was hardly changed. CDSW
increased the number of the detected clone references in half
the cases. Scorpio produced interesting results. By using ourclone references for evaluation, the number of the detectedclone references was increased in the case of netbeans. On
the other hand, in the case of the others, the number of thedetected clone references was decreased.
Fig. 7 shows variabilities of gapped lines in clone candidates among the target clone detectors for netbeans andant. NiCad has narrow variability, meanwhile Scorpio and
CDSW have wider one. Fig. 3, Fig. 4 and Fig. 7 reveal that
our clone references has a strong eﬀect on clone detectors

147

0
NiCad

●

Figure 7: Variabilities of gapped lines

our references

100

0

●

402 403

200

422

CDSW

400
300

604

500

Bellon's references

our references●●(a) netbeans
Bellon's references

Scorpio

●

●

NiCad

Figure 4: Experimental results for ant

1500

NiCad

(b) good

●

CDSW

0

NiCad

●

100

0

0

CDSW

50 50

Figure 6: Experimental results for swing

13

10

Scorpio

135 127(a) ok20

20

our references

290 290

200

0

300

Bellon's references
15

350

300

100

200

our references

26

10

494

392 388(b) good

Bellon's references

12 12

593 593200

Figure 3: Experimental results for netbeans

30

Bellon's references
400

400

5

2

0
NiCad

our references

800

10

18 18

20

our references
13

150

30

Bellon's references
15

100

34

30

38

50

our references0

Bellon's references

40

7 7
7 7
min(max( , ), max( , ))
8 11
10 11
0.7

On the other hand, in the case of using our clone references,
ok value is obtained as follows:
6 6
6 6
ok = min(max( , ), max( , ))
7 10
7 10
 0.857
In this way, ok value is changed if our clone references areused. Accordingly, the evaluation scale (e.g. recall, precision
and F -measure) are might be changed.

4. EXPERIMENTThe purpose of this experiment is to reveal diﬀerences of
evaluation results yielded by Bellon’s clone references andthe new clone references. In this experiment, we compared
the number of detected clone references based on a threshold. We used 0.7 as the threshold, which was the same value
used in Bellon’s benchmark.
In this experiment, we chose the clone detectors shown in
Table 1 as targets for the comparison. All of them can detect

Table 1: Clone detectors used in this experiment
Developer Clone detector Detection method
Roy
NiCad [8]text-based
Higo
Scorpio [4]PDG-based
Murakami
CDSW [6]statement-based

Table 2: Target software systems
Name
Language # Files Lines of code
netbeans
Java101
14,360ant
Java
178
34,744
jdtcoreJava
741
147,634
swing
Java
538
204,037414

clone reference

clone candidate

1: package org.eclipse.jdt.internal.compiler.batch;

1: package org.eclipse.jdt.internal.core;

呍
呍
呍

呍
呍

呍
14:
public class BasicCompilationUnit implements ICompilationUnit {15:
protected char[] contents;

9: public class CompilationUnit implements ICompilationUnit {10:
public char[] contents;
呍
呍

*

*

呍
29:
30:

31:
32:
33:
34:
35:
36:
37:
38:
39:
40

* gapped line

try {return Util.getFileCharContent(new File(new String(fileName)),encoding);} catch (IOException e) { }return new char[0]; }public char[] getFileName() {return fileName; }public char[] getMainTypeName() {return mainTypeName; }public char[][] getPackageName() {return null; }public String toString() {return "CompilationUnit[" + new String(fileName) + "]";
} }

呍
呍

*

*

呍
34:
35:

36:
37:
38:
39:
40:
41:
42:
43:
44:
45:

try {
return Util.getFileCharContent(new File(new String(fileName)),this.encoding);} catch (IOException e) { }return new char[0]; }public char[] getFileName() {return fileName; }public char[] getMainTypeName() {return mainTypeName; }public char[][] getPackageName() {return null; }public String toString(){
return "CompilationUnit: "+new String(fileName); } }Figure 8: A clone reference that becomes to be undetected when our clone references are used for evaluation

6.

that detect Type-3 code clones having many gapped lines.
Fig. 8 shows one of examples of clone references that
becomes to be undetected when our clone reference is used
for evaluation. The left source ﬁle has 40 lines of clone
reference (lines 1-40), and 32 lines of clone candidate (lines9-40), and 2 gapped lines (lines 30, 40). The right source ﬁlehas 45 lines of clone reference (lines 1-45), and 32 lines ofclone candidate (lines 14-45), and 2 gapped lines (lines 35,45). In this case, good value was calculated as follows when
Bellon’s clone references were used.
good= min(


7.

32 32
, )
40 45

0.711

= min(

REFERENCES[1] S. Bellon, R. Koschke, G. Antniol, J. Krinke, and
E. Merlo. Comparison and evaluation of clone detection
tools. IEEE Trans. on Software Engineering,31(10):804–818, October 2007.
[2] N. Bettenburg, W. Shang, W. M. Ibrahim, B. Adams,
Y. Zou, and A. E. Hassan. An empirical study on
inconsistent changes to code clones at release level. In
Proc. of the 16th Working Conference on Reverse
Engineering, pages 85–94, October 2009.
[3] N. Göde and R. Koschke. Frequency and risks ofchanges to clones. In Proc. of the 33rd International
Conference on Software Engineering, pages 311–320,
May 2011.
[4] Y. Higo and S. Kusumoto. Code clone detection on
specialized pdgs with heuristics. In Proc. of the 15th
European Conference on Software Maintenance andReengineering, pages 75–84, March 2011.
[5] M. Kim, L. Bergman, T. Lau, and D. Notkin. An
ethnographic study of copy and paste programming
practices in oopl. In Proc. of the 3rd International
Symposium on Empirical Software Engineering, pages
83–92, August 2004.
[6] H. Murakami, K. Hotta, Y. Higo, H. Igaki, andS. Kusumoto. Gapped code clone detection withlightweight source code analysis. In Proc. of the 21stInternational Conference on Program Comprehension,
pages 93–102, May 2013.
[7] C. K. Roy and J. R. Cordy. A survey on software clone
detection research. Technical Report No. 2007-541,
Queen’s University, 2007.
[8] C. K. Roy and J. R. Cordy. Nicad: Accurate detectionof near-miss intentional clones using ﬂexible
pretty-printing and code normalization. In Proc. of the
16th International Conference on Program
Comprehension, pages 172–181, June 2008.

On the other hand, good value is calculated as follows when
our clone references are used.
good

ACKNOWLEDGMENTS

This study was supported by Grants-in-Aid for ScientiﬁcResearch (S) (25220003), Grant-in-Aid for Exploratory Research (24650011) from the Japan Society for the Promotion of Science, and Grant-in-Aid for Young Scientists (A)(24680002) from the Ministry of Education, Culture, Sports,
Science and Technology.

30 30
, )
38 43

0.697

In this case, good value fell below 0.7 when our clone references were used. Thus, these clone candidates did notmatch any of clone references in good value evaluation. In
this way, not all Type-3 clones increase their ok/good values
when our clone references are used.
The experimental result is summarized as follows. All thetarget clone detectors achieved diﬀerent results. Some detectors yielded almost unchanged results, and others madechanges signiﬁcantly. These results rely on the amount ofgapped lines in code clones. Thus, an evaluation using thelocational information of gapped lines will be a new evaluation of Type-3 clones.

5. CONCLUSION
This paper introduces a new dataset containing locational
information of code clones with their gapped lines. Furthermore, we explained the example that the evaluation results
are changed when our dataset was used. Lastly, the experiment proved that our dataset achieved diﬀerent outcomesfrom Bellon’s dataset by using some clone detectors. We
hope that our dataset will diversify an evaluation of Type-3
clones.
In the future, we are going to conduct strictly experiments
using the new dataset. If our dataset is used for evaluating
Type-3 clones, more accurate results would be obtained.

415