2018 25th Asia-Pacific Software Engineering Conference (APSEC)

Multilingual Detection of Code Clones
Using ANTLR Grammar Deﬁnitions
Yuichi Semura∗ , Norihiro Yoshida† , Eunjong Choi‡ and Katsuro Inoue∗∗ Osaka University, Japan, {y-semura, inoue}@ist.osaka-u.ac.jp† Nagoya University, Japan, yoshida@ertl.jp‡ Nara Institute of Science and Technology, Japan, choi@is.naist.jp

replacement. This mechanism saves practitioners from troubles
during the implementation of lexical analyzers. When users
apply CCFinderSW to an additional language, they should
prepare two ﬁles to deﬁne comment rules and reserved wordsof the language. However, in order to prepare a description ﬁleof comment rule deﬁnitions, it is necessary for users to learn
grammars of the description ﬁle. Moreover, it is troublesome
to create a description all reserved words of a target language.
An extension mechanism to save these troubles is required
CCFinderSW.
A parser generator is a programming tool that creates alexer, parser or compiler based on the grammar deﬁnitions of
the target language. Since such grammar deﬁnition ﬁle of aprogramming language has lexical information in the source
code, by choosing what is necessary for code clone detection
from among them and applying it to the tool, it is possible
to create any easy extension mechanism to handle additional
language. Regarding ANTLR, there is a Github repository‘grammars-v4’1 which contains over 150 grammar deﬁnitionﬁles. Therefore, practitioners of a code clone detection tool
can easily get a grammar deﬁnition ﬁle from the repositoryand give it to the tool.
In this paper, we propose an approach to automatically extract lexical information necessary for token-based code clone
detection from grammar deﬁnitions of the parser generator.
Besides, we extended a clone detection tool CCFinderSW thathas a lexical information extractor from grammar deﬁnitionsof a parser generator ANTLR.

Abstract—So far, many tools have been developed for thedetection of code clones in source code. The existing clone
detection tools support only a limited number of programming
languages and do not provide any easy extension mechanism tohandle additional language. However, from our experience in industry/university collaboration, we found that many practitioners
need to analyze source code written in various languages. In thispaper, we propose an approach for the multilingual detection ofcode clones using grammar ﬁles for a parser generator ANTLR.
We extended a clone detection tool CCFinderSW with theproposed approach and then apply the extended CCFinderSW
to ANTLR grammar ﬁles for 43 languages. As a result, the ﬁlesfor 39 out of the 43 languages can be analyzed correctly by theextended CCFinderSW.
Index Terms—code clone, lexical analysis, parser generator,
ANTLR

I. I NTRODUCTION
Programmers often copy and paste code so that they canreuse existing code fragments. This causes code clones (i.e,code fragments that are identical or similar code fragments
to each other). Generally, a code clone is regarded as oneof the factors that hinder software maintainability [1], [2],[3]. For instance, when a cloned code fragment contains abug, a programmer should check all of its cloned fragments
for the same bug. If the cloned fragments contain the same
bugs, they should be modiﬁed for the same bug. To that end,
he/she should know locations of all code clones in the sourcecode. However, it is difﬁcult for developers to recognize allcode clone from large-scale software systems. To alleviate thisproblem, a multitude of code clone detection tools have beendeveloped [4], [5], [6]. For example, Kamiya has developed atoken-based code detection tool CCFinderX [4] that is widelyused in academic research as well as industries [7], [8].
The existing clone detection tools support only a limited
number of programming languages and do not provide any
easy extension mechanism to handle additional language [4],[5], [6]. From our experience in industry/university collaboration, we found that many practitioners need to analyze sourcecode written in various languages [9]. Such an extension
mechanism to handle additional language saves effort and time
for practitioners and tool developers [10].
A clone detection tool CCFinderSW has an extension
mechanism to handle additional language on demand frompractitioners. It enables practitioners to easily change thelexical mechanism for comment elimination and identiﬁer
978-1-7281-1970-0/18/$31.00 ©2018 IEEE
DOI 10.1109/APSEC.2018.00088

II. C LONE D ETECTORS : CCF INDER X AND CCF INDER SW
To introduce the mechanism of a token-based clone detection tool, we brieﬂy explain CCFinderX and CCFinderSW.
They identify not only Type-1 clones [11] (i.e. identical codefragments except for variations in whitespace, layout and comments) but also Type-2 clones [11] (i.e. syntactically identicalfragments except for variations in identiﬁers, literals, types,
whitespace, layout and comments.) by replacing identiﬁers
related to types, variables, and constants with a special token.
Kamiya has developed a token-based code detection tool
CCFinderX [4] that is widely used in academic research
as well as industries. Practitioners can replace its lexical
analyzer with another one depending on a target language.
1 https://github.com/antlr/grammars-v4673

In other words, when we apply CCFinderX to additional
languages, they should implement lexical analyzers. However,this implementation requires users’ plenty of knowledge andtakes more time and effort.
We developed CCFinderSW [9], a code clone detection
tool based on CCFinderX. It contains a mechanism to tokenize
source code written in many languages and to detect code
clones and requires comment rules and reserved words as the
input. This mechanism saves the time and effort of implementation of lexical analyzers of new languages. The code
clone detection process of CCFinderSW is comprised of four
steps, which are lexical analysis, transformation, detection, andformatting. Additionally, lexical analysis is comprised three
detailed steps, which are comment elimination, tokenization,and identiﬁer distinction.
The following subsections describe the detail of lexical
analysis and transformation, and problems of CCFinderSW.
With respect to the detection and formatting, please refer to[9].

TABLE IM AIN TOKENS USED IN ANTLR
Token‘literal’[char set]
.
˜x
x*
x?
x*?
x|y

Description
Match that character or sequence of characters.
Match one of the characters speciﬁed in the characterset.
The dot is a single character wildcard that matches any
single character
Match any single character not in the set described by
x. In this paper, we call this token a NOT operator.
Match zero or more occurrences of x.
Match zero or one occurrences of x.
Match the shortest occurrences of x.
Match either x or y.

D. Problems of CCFinderSW
When users apply CCFinderSW to an additional language,
they should prepare two ﬁles to deﬁne comment rules andreserved words of the language. However, in order to preparea description ﬁle of comment rule deﬁnitions, it is necessary
for users to learn grammars of the description ﬁle. Moreover,
it is troublesome to create a description all reserved words
of a target language. An extension mechanism to save thesetroubles is required CCFinderSW.

A. Comment Elimination
The lexical analysis of CCFinderSW eliminates commentsin the source code according to the type-1 and type-2 code
clone deﬁnition. This tool requires comment rules and reserved
words as the input. The user creates a description ﬁle ofcomment rule deﬁnitions and gives it to CCFinderSW at
runtime. There are ﬁve kinds of conﬁgurable comment rules,
that are line comment, multi-line comment, full line comment,
full multi-line comment and string literals.

III. A PARSER G ENERATOR : ANTLRThis section describes grammar deﬁnitions in ANTLR. The
following example is a grammar deﬁnition ﬁle which represents a grammar of arithmetic operation. In line 1, grammar
Prog deﬁnes the name of this grammar as Prog. From line
2, each line represents a rule that constitutes this grammar
tree. There are two kinds of grammar rules, namely lexer ruleand parser rule. The lexer rules deﬁne tokens that appear insource code. The parser rules deﬁne syntax grammars. In thisexample, INT and WS are lexer rules. prog, expr, term, andfactor are parser rules.

B. Tokenization
After the comments are eliminated from the source code
based on the deﬁned comment styles, each line of the sourcecode is divided into tokens based on a lexical rule. The
following four lexical rules are used for the tokenization. Notethat a rule with a low number has a higher priority.
1) Each character literal or string literal corresponds to one
token, respectively.
2) White spaces and line breaks are delimiters.
3) Each symbol is one token.
4) Other consecutive alphabetic or numeric strings are
identiﬁed as one token.

1
2
3
4
5
6
7

grammar Prog;
prog: expr;
expr: term ((’+’|’-’) term)*;
term: factor ((’*’|’/’) factor)*;factor: INT | ’(’ expr ’)’;
INT: [0-9]+;
WS: [ \t\r\n]+ -> skip;

Listing 1. An example of an ANTLR grammar ﬁleParr publishes documents of the lexical representation usedin ANTLR [12]. Table I shows parts of the documents that areimportant to understand Section IV.

C. Identiﬁer Distinction and Transformation
In the identiﬁer distinction process, the token generated by
the tokenization process is distinguished between an identiﬁer
or a reserved word as preprocessing of transformation. The
user creates a description ﬁle of reserved words and gives it
to CCFinderSW at runtime.
In the transformation process, the token sequence is transformed in order to detect meaningful code clones. In detail, all
the identiﬁers representing variable names and function names
are replaced by the same token. Reserved words are character
strings that are reserved by the programming language andcannot be used for variable and function names.

IV. I MPLEMENTATION OF A L EXICAL I NFORMATIONE XTRACTOR FROM G RAMMAR D EFINITIONIn this section, we propose an approach to automatically extract lexical information necessary for token-based code clone
detection from grammar deﬁnitions of the parser generator.
Besides, we extended a clone detection tool CCFinderSW
that had a lexical information extractor from grammar deﬁnitions of a parser generator ANTLR. We used Java for thisdevelopment.

674ǢŸȖǋOs®ÞĶsǣrɮǼsŘ_s_əsǋǣÞŸŘ
ŷǋÞ¶ÞŘĶəsǋǣÞŸŘ

NŸŎŎsŘǼrĶÞŎÞŘǼÞŸŘǻŸĨsŘÞʊǼÞŸŘĵsɮÞOĶŘĶɴǣÞǣ
Ý_sŘǼÞ¯Þsǋ^ÞǣǼÞŘOǼÞŸŘ

ǢɴŘǼɮ^s¯ÞŘÞǼÞŸŘ

RegEx is suitable for this study. In addition to this, comment
elimination with a RegEx allow CCFinderSW to handle morecomment rules than the ﬁve existing comment rules.
Section IV-A describes the transformation of comment deﬁnitions into a RegEx, Section IV-B describes the transformationof string literal deﬁnitions, and Section IV-C describes thetransformation of reserved words deﬁnitions.

>ĞǆŝĐĂů/ŶĨŽƌŵĂƚŝŽŶǆƚƌĂĐƚŽƌ
NŸŎŎsŘǼǊs¶rɮ Ř_ǢǼǋÞŘ¶ĵÞǼsǋĶǊs¶rɮǊsǣsǋɚs_ɟŸǋ_Ǌs¶rɮ

A. Transformation of comment rule deﬁnitions into a RegEx
This section describes the implementation to extract comment rule and to transform them into a RegEx.
The process of transformation of comment rule into a RegEx
is comprised of the following four steps:
Step A: Choose grammar deﬁnitions of comment rules
from all rules.
Step B: Apply other grammar deﬁnitions to references inchosen deﬁnitions.
Step C: Transform applied deﬁnitions into RegExes inavailable in Java.
Step D: Combine all transformed RegExes into one RegEx.
Details of each step are as follows.
Step A First, the lexical information extractor chooses
grammar deﬁnitions of comment rules. For this process, we set
4 standards so as to identify grammar deﬁnitions of comment
rules based on our investigation. When a deﬁnition applies atleast one of the standards, it is identiﬁed as a comment ruleby this extractor. The standards are listed below.
1) A name of a deﬁnition contains ‘comment’, ‘COMMENT’ and so on.
2) A deﬁnition is linked to a ‘skip’ command.
3) A deﬁnition is linked to a ‘channel(HIDDEN)’ command.
4) A deﬁniton is linked to a ‘channel(X)’ command. Moreover, X contains ‘comment’, ‘COMMENT’, and so on.
The following listing shows four grammar deﬁnitions ofcomment rules which are applied to standards. Also, theycorrespond to a multi-line comment written in C/C++.

ǻŸĨsŘǢsǇȖsŘOs
ǻǋŘǣ¯ŸǋŎǼÞŸŘǻǋŘǣ¯ŸǋŎs_ǻŸĨsŘǢsǇȖsŘOs
NŸ_sNĶŸŘs^sǼsOǼÞŸŘˀ®ŸǋŎǼǼÞŘ¶NĶŸŘs˚ƻÞǋǣˀNĶŸŘs˚ǢsǼǣ

Fig. 1. An overview of the original and the extended versions of CCFinderSW

CCFinderSW detects code clones using lexical information
extracted from grammar deﬁnitions. This section describes theimplementation method of a new module, which is a lexicalinformation extractor, and some of the changes in the existingprocesses.
Figure 1 depicts an overview of extended CCFinderSW.
Areas highlighted in light yellow represents modules that we
newly develop or extend.
A new module extracts lexical information from grammar
deﬁnitions of ANTLR and outputs three Regular Expressions(RegExes) which represent comments, string literals, and reserved words. A RegEx of comments and a RegEx of string
literals are used in the comment elimination process, and a
RegEx of reserved words is used in the identiﬁer distinction
process. Therefore, we needed to extend the two process andenable them to accept RegExes as inputs.
Because of the online repository ‘grammars-v4’, practitioners can easily obtain a grammar deﬁnition and give it to
CCFinderSW. From an overall point of view, practitioners
should give CCFinderSW source ﬁles and grammar deﬁnitionof a target language. Also, they can choose between theexisting and the new process ﬂow.
We explain an overview of the processes in a lexicalinformation extractor. A lexical information extractor contains
a parser for grammar deﬁnitions of ANTLR that generates
syntax trees.
First, it analyzes the tree of grammar deﬁnitions and extracts
lexical information from the tree. Second, it transforms lexicaldeﬁnitions into RegExes of comment rules, string literals,
and reserved words. Finally, it gives the three RegExes tothe comment elimination process and the identiﬁer distinction
process.
The reason for transforming lexical deﬁnitions into a RegEx
is as follows: A RegEx resembles a grammar deﬁnition in
ANTLR, and is often used for search and replacement ofcharacter strings [13]. Therefore, comment elimination with a

1
2
3
4

Comment:’/*’.*?’*/’;
Block1:’/*’.*?’*/’->skip;
Block2:’/*’.*?’*/’->channel(HIDDEN);
Block3:’/*’.*?’*/’->channel(BComment);

Listing 2. Notations for deﬁning comments

Step B Second, the extractor applies recursively othergrammar deﬁnitions to references in chosen deﬁnitions. In
an example of a grammar deﬁnition ﬁle in Section III, the‘term’ deﬁnition refers to the ‘factor’ deﬁnition. In such cases,
the extractor applies a deﬁnition a reference destination to areference source.
Step C Third, the extractor transforms applied deﬁnitions
to RegExes available in Java. As a reason for that, there
is some difference between descriptions of RegExes in Javaand descriptions of grammar deﬁnitions in ANTLR. Thereare 3 patterns of descriptions in ANTLR which the extractor
transforms into RegExes.
The ﬁrst one is a single quotation. In grammar deﬁnitions
in ANTLR, a literal appearing in the source code are enclosed

675

6WHS$%&RPPHQW&67$57 "&(1'&67$57· ·
6WHS%&(1'· ·string literals. The following listing shows notations of string
literals which frequently appear in our investigation.

/&RPPHQW'6/$6+a>?U?Q@ 

1
2'6/$6+··

StringLiteral: QUOTE StringCharacters? QUOTE;STRING : ’string’;

Listing 3. notation for deﬁning string literals%&RPPHQW   "  /&RPPHQW  a>?U?Q@ 

The grammar deﬁnition in line 1 represents a string literalgrammar. The grammar deﬁnition in line 2 represents a
reserved word ‘string’ which appears in the source code, although its name contains ‘STRING’. Considering the notationsof string literal deﬁnitions, we set a standard for extraction ofstring literal deﬁnitions. The standard deﬁnes a string literaldeﬁnition which is not a reserved word deﬁnition and whosename contains ‘string’, ‘STRING’ and so on.
Then, the extractor identiﬁes string literal deﬁnitions according to the standard and transforms them into a RegEx.
The transformation of string literals is the same as comment
rules, therefore detailed explanation is omitted.

6WHS&
? >?V?6@ "?  ">?U?Q@ >?V?6@

6WHS'
? >?V?6@ "? _ ">?U?Q@ >?V?6@

Fig. 2. An example of the transformation of comment rule deﬁnitions into a
RegEx

in single quotations. In a RegEx, this single quotations is
unnecessary and the extractor eliminates them.
The second one is a NOT operator. In grammar deﬁnitions
in ANTLR, a ‘˜x’ matches any single character not in the set
described by x, that is, ‘˜’ has a function of a NOT operator.
In RegExes, there is no expression corresponding to a NOT
operator. Accordingly, the extractor transforms a ‘˜x’ into a
RegEx with the negative lookahead.
The third one is a dot. In grammar deﬁnitions in ANTLR,
a dot is a single character wildcard that matches any single
character. However, in RegEx used in Java, a dot matches any
single character except the newlines. Because of the difference
between two deﬁnitions of a dot, the extractor transforms a ‘.’
into a ‘[\s\S]’.
Step D Finally, the extractor combines all transformed
RegExes into one RegEx. That is, it outputs a logical sumof all generated RegExes with ‘|’. At the end of Step D, thetransformation of comment grammar deﬁnitions into RegExes
is completed.
Figure 2 is an example of the transformation of comment
rule deﬁnitions into a RegEx. In Figure 2, the extractor
chooses two deﬁnitions, which is labeled ‘BCOMMENT’ and‘LCOMMENT’, and transforms them into a RegEx,

C. Transformation of reserved word deﬁnitions into a RegEx
This section describes an approach to extract reserved words
and to transform them into a RegEx. In the same as theothers, we investigated notations of grammar deﬁnitions corresponding to reserved words. The following listing shows two
different notations for deﬁning a reserved word ’while’. Bothof them are frequently appear in our manual investigation.
1
2

WHILE:’while’;
WHILE:[wW][hH][iI][lL][eE];

Listing 4. Two different notations for deﬁning a reserved word ’while’

In line 1 of this example, the deﬁnition which is named‘WHILE’ links a string literal ‘while’. This notation is widelyused in grammar deﬁnitions which we investigate. In line
2 of this example, the deﬁnition is composed of character
sets such as ‘[wW]’. This character set ‘[wW]’ matchesboth uppercase and lowercase letter of ‘w’. Hence, a RegEx‘[wW][hH][iI][lL][eE]’ matches ‘while’ and ‘WHILE’ and‘WhIlE’, and so on. According to two notation of grammar
deﬁnitions, we implement the extraction process of reserved
words. A character set used in ANTLR is also used in a RegEx.
Consequently, it transforms reserved word deﬁnitions into a
RegEx.
Before our investigation, we deﬁned that a reserved word
contains only alphabets. However, since we found some reserved words which contain symbols such as ‘ ’ and ‘@’, we
add an option which designates characters composing reserved
words.
After the extraction process, it transforms deﬁnitions ofreserved words into a RegEx. The transformation of reserved
words is the same as the others, therefore detailed explanationis omitted.

B. Transformation of string literal deﬁnitions into a RegEx
This section describes an approach for extract string literaland to transform them into a RegEx.
A string literal is a type of literal in programming for the
representation of a string object within the source code, respectively. For instance, in Java program, characters enclosed indouble quotes are identiﬁed as a string literal. CCFinderSW
deﬁnes a string literal as one of the conﬁgurable comment
rules. Also in this expansion, to implement similar the comment elimination process, it is necessary to give string literalgrammar to CCFinderSW. Therefore, the lexical information
extractor extracts string literal deﬁnitions and transforms theminto a RegEx. Finally, it gives the RegEx to the comment
elimination process of CCFinderSW.
In order to implement a lexical information extractor, weinvestigated notations of grammar deﬁnitions corresponding to

V. E VALUATIONAs an evaluation, we applied the proposed approach toa collection of ANTLR grammar ﬁles and conﬁrmed theaccuracy of the proposed approach. After that, we extended
CCFinderSW with the proposed approach. And then, we

676

We downloaded source ﬁles which are written in the selected languages from Github repositories and then applied
the extended CCFinderSW to the source ﬁles. After that, we
randomly selected 20 pairs of code clones from the detection
result for each selected language and then manually conﬁrmed
the all of the selected pairs for each language are correctly
pairs of code clones.

applied it to a collection of source ﬁles written in several
different languages and checked the detected code clones.
A. Result of analyzing grammar ﬁles
We investigated how many grammar ﬁles in an ANTLR
grammar repository ‘grammars-v4’2 are correctly analyzed.
This repository contains over 150 grammar deﬁnition ﬁles.
For the evaluation, we selected 43 of 154 languages which
are available in the advanced search3 of the code search engine
at GitHub because impractical languages such as esoteric languages (e.g., Brainf*ck) should be excluded from the eligible
languages. And then, we manually identiﬁed the notations ofcomments, string literals, and reserved words from each of thegrammar deﬁnition ﬁles of the 43 eligible languages. After
that, we applied the proposed approach to the 43 grammar
deﬁnition ﬁles. Finally, we checked whether the extracted
RegExes are correct as comments, string literals and reserved
words of each language based on the result of our manual
identiﬁcation.
The result shows that the proposed approach successfully
extracted notations for comments, string literals, and reserved
words in 38, 36 and 37 out of the 43 eligible languages
respectively. As the total, all three RegExes are correctly
analyzed in 39 out of the 43 eligible languages.
The following listing shows the comment deﬁnitions ina grammar ﬁle ’Lua.g4’. COMMENT deﬁnes a comment
notation of Lua and refers to NESTED STR. NESTED STR
refers to itself recursively. Therefore, comments used in Lua
is unable to be expressed by RegEx. Our method proposed inthis paper is unable to extract comment notations from suchgrammar deﬁnitions.
1
2
3
4
5
6
7
8

VI. S UMMARY
In this paper, we propose an approach to automatically extract lexical information necessary for token-based code clone
detection from grammar deﬁnitions of the parser generator.
Besides, we extend a clone detection tool CCFinderSW thathas a lexical information extractor from grammar deﬁnitions ofa parser generator ANTLR. In an evaluation, we indicate thatthe lexical information extractor in CCFinderSW can extract
most rules of comment, string literals and reserved words in 43
languages. Also, we manually conﬁrmed the all of the selectedpairs for each of seven languages are correctly pairs of code
clones.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Numbers JP25220003, JP18H04094 and JP16K16034.
R EFERENCES[1] B. Lague, D. Proulx, J. Mayrand, E. M. Merlo, and J. Hudepohl,“Assessing the beneﬁts of incorporating function clone detection in adevelopment process,” in Proc. of ICSM, 1997, pp. 314–321.
[2] Z. Li, S. Lu, S. Myagmar, and Y. Zhou, “Cp-miner: ﬁnding copy-paste
and related bugs in large-scale software code,” IEEE Transactions on
Software Engineering, vol. 32, no. 3, pp. 176–192, 2006.
[3] A. Zeller, Why Programs Fail, Second Edition: A Guide to Systematic
Debugging, 2nd ed. Morgan Kaufmann Publishers Inc., 2009.
[4] T. Kamiya, S. Kusumoto, and K. Inoue, “CCFinder: a multilinguistic
token-based code clone detection system for large scale source code,”IEEE Transactions on Software Engineering, vol. 28, no. 7, pp. 654–670,
2002.
[5] L. Jiang, G. Misherghi, Z. Su, and S. Glondu, “Deckard: Scalable andaccurate tree-based detection of code clones,” in Proc. of ICSE, 2007,
pp. 96–105.
[6] H. Sajnani, V. Saini, J. Svajlenko, C. K. Roy, and C. V. Lopes,“Sourcerercc: Scaling code clone detection to big-code,” in Proc. of
ICSE, 2016, pp. 1157–1168.
[7] K. Inoue, Y. Higo, N. Yoshida, E. Choi, S. Kusumoto, K. Kim, W. Park,and E. Lee, “Experience of ﬁnding inconsistently-changed bugs in code
clones of mobile software,” in Proc. of IWSC, 2012, pp. 94–95.
[8] Y. Yamanaka, E. Choi, N. Yoshida, K. Inoue, and T. Sano, “Applying
clone change notiﬁcation system into an industrial development process,”
in Proc. of ICPC, 2013, pp. 199–206.
[9] Y. Semura, N. Yoshida, E. Choi, and K. Inoue, “CCFinderSW: Clone
detection tool with ﬂexible multilingual tokenization,” in Proc. of APSEC
2017, 2017, pp. 654–659.
[10] K. Sakamoto, K. Shimojo, R. Takasawa, H. Washizaki, and Y. Fukazawa,“OCCF: A framework for developing test coverage measurement tools
supporting multiple programming languages,” in Proc. of ICST 2013,2013, pp. 422–430.
[11] C. K. Roy, J. R. Cordy, and R. Koschke, “Comparison and evaluationof code clone detection techniques and tools: A qualitative approach,”
Science of Computer Programming, vol. 74, no. 7, pp. 470–495, 2009.
[12] T.
Parr,“antlr4/index.md
atmasterantlr/antlr4,”
https://github.com/antlr/antlr4/blob/master/doc/index.md.
[13] J. E. Friedl, Mastering regular expressions. O’Reilly Media, 2002.

COMMENT: ’--[’ NESTED_STR ’]’ -> channel(HIDDEN);
fragment
NESTED_STR: ’=’ NESTED_STR ’=’| ’[’ .*? ’]’;

Listing 5. The comment deﬁnitions in the grammar ﬁle ’Lua.g4’

Please note that several of the eligible languages do not haveany keyword/reserved word. The detailed result is available
online4 .
B. Result of the detection of code clones
We used the extended CCFinderSW with the proposed
approach to a collection of source ﬁles written in several
different languages and checked whether the detection result
is correct.
For this evaluation, we selected c, cobol85, cpp14, ecmasript, java9, python3 and visualbasic6 from the 39 correctlyanalyzed languages in Section V-A. Nowadays many of systems which written in each of the seven selected languages
have much legacy code.
2 We collected ANTLR grammar ﬁles from ‘grammars-v4’ repository on
December 14, 2017.
3 https://github.com/search/advanced
4 https://sites.google.com/site/yoshidaatnu/APSEC2018ERAResult.pdf

677