An Experience Report on
Analyzing Industrial Software Systems
Using Code Clone Detection Techniques Norihiro Yoshida (NAIST)
Yoshiki Higo, Shinji Kusumoto, Katsuro Inoue (Osaka University)

Outline
1.
2.
3.
4.

What is a code clone?
Discussions on the harmfulness of codeclone
Importance of sharing industrial experienceswith clone
Industrial application of clone analysis} Analysis tools} Result5.

Summary	What is a code clone?	
} A code fragment that has identical or similar code fragments

to it in source code.

} Introduced in source program by various reasons such asreusing code by `copy-and-paste’

Code Clone

3

Discussions on the harmfulness of code clone(Opponent)	
There have been numerous discussions:} Cloning opponent: “clone should be avoided because

it makes software maintenance difficult.”
Book on programming practice} Research papers that is a little bit less than state-of-the-art}

Bug isfound	

Code Clone

Bugclones
isThe othershould befound	
inspected.	

Discussions on the harmfulness of code clone(Moderate & Proponent)
} Moderate: “Clone is unavoidable when a language

lacks suitable modularization mechanism toeliminate it.”}e.g., it is difficult to merge code clones into a function in thecase that the identifiers are unmatched.

} Clone proponent: “Clones do not often cause bugs, soleave it be.”	
}

According to Rahman’s study [1] of OSS, there is no significant
relationship among locations of bug and clones.	

[1] Rahman, et al., “Clones: What is that Smell?”, MSR 2011

Importance of sharing experienceswith cloning} Which is truth of code clones?

There is no conclusion currently.
} Probably it depends on the context of cloning [2].
}} Sharing experience with cloning is a promising way foreasy identification of harmful and harmless clones.

} Software engineering community has to reportexperience with clone detection and analysis.

[2] Kapser, et al., “"Cloning considered harmful" considered harmful: patterns of cloning insoftware”, Empirical Software Engineering, 2008.

Researches on cloning in industry} Much research have been done on

Automatic clone detection} Analysis of code clones in OSS}

} On the other hand, quantity of reports on cloning inindustry has been lacking.
Ratio of clones to whole source code is higher in industry thanin OSS} Rather than in OSS, clone causes a problem in industry.
}

It is needed to report an industrial experience
with clone analysis.

Overview of industrial case study
Investigated an industrial software in terms of thefollowing points by clone analysis technique.

1.

A.
B.
C.

2.

Is there significant difference in clones between the endsof the unit testing and the combined testing?
Where clones are concentrated in the source code?
What sort of characteristic clones are involved in thesource code?

Interviewed developers for detected clones

Target software project} Japanese governmental project}

Software system for traffic infrastructure} Source code

Approximately 100,000 LOC, and increased by 20 thousands
after the unit test.
} Main language is C/C++}

} Organization

5 vendors, each of which was assigned for a subsystem.
} 1 project manager from a company different from the vendors}

Tools for clone detection & analysis } Clone detection tool : CCFinder [3]}

Detection of lexically-similar code clones based on theidentification of identical token sequences in source code} Code clone analyzer : Gemini [4]Scatter plot} Metrics for extracting clones}

[3] T. Kamiya, et al.: "A multilinguistic token-based code clone detection system forlarge scale source code”, IEEE TSE, 2002.
[4] Y. Ueda, et al.: “Gemini: Maintenance Support Environment Based on Code CloneAnalysis”, METRICS 2002.	

Token-based clone detection tool :CCFinder
Detection of identical token sequencesin source code 	
throws{{ String$$ $$foo(( (( )) ))throws
foofoo()1. staticvoid
throwsRESyntaxException
staticthrows$$RESyntaxException
RESyntaxException
String aa {static void
throws{{ $$ $$]] == a[]new]{{] {$${ "123,400",$$ =$$String[[ []][ Stringnew
String$ { "123,400",
2. [[String
new[]"abc", "orange 100" };"abc","orange100"};org.
apache.
regexp} ;3. org.apache.regexp.RE
pat = new org.apache.regexp.RE("[0-9,]+");. RE
neworg . apache . regexp$$ pat$$ ==new=new
4. int sum = 0;. RE) ; int$$ (( "[0-9,]+"$$== $$00
int
sum$$ sum$$ =5. for(int"[0-9,]+"i = 0; i) < ;a.length;++i);; for$$ i$$i ==<int
for (( int= 0$$0 ;;; i$$i <<6. a$ if. (pat.match(a[i]));++$$ ;; ++$$ ii )) ))if
length; ++++pat$ . length
if ifif(( (($$ pat7. .. matchsum+=
Sample.parseNumber(pat.getParen(0));$$ (( (( $$ aa [[ [[ $$ ii ]] ]] )) )) )) )) $$sum
sum
8. +=
System.out.println("sum"getParen+ sum);
Sample.. getParen(( 00$$ .. $.$. parseNumber(( $$ .. $$ (( ((patparseNumber
pat$$ =+=9. } )) )) ;; System$$ .. .$$. out.. .$$. println
Systemoutprintln"sum==""(( $$ (( "sumstaticgoo$$ void))) ;;;; goo(String}}}} static$$ $$[]
10. static
RESyntaxException {++ sum
static voidvoidgooa)
Stringstatic(( (( throws$$String[[ exp]] ))) =throws$$RE("[0-9,]+");{{ $$ $$ == {{ RERESyntaxException
RE expexp ==throws
new
11. a$$ REREexp=throws
newRESyntaxException
RE("[0-9,]+");
RE$$ (( "[0-9,]+"$$ )) ;; )) $$;; $=sum$$ == 00
int sum
new$int=intsum== 0;
12. new
intsum"[0-9,]+"0;
i$$i ==00$$ ;; i$$i <<forfor (( int(int0;13. ;; forfor(intint$$ii ==0;= ii << a.length;a.length;++i)++i)$$ ;; ++$$ ii )) ))if;++
length; ++++a$$ .. lengthexp
if ifif(( (($$ expifif (exp.match(a[i]))14. . match(exp.match(a[i]))$$ (( (( $$ aa [[ [[ $$ ii ]] ]] )) )) )) )) $$sum
sumsum+=parseNumber(exp.getParen(0));
15. +=
sum+=parseNumber(exp.getParen(0));+=$$ (( ( $$exp .. (. $$expgetParen()) 0)) )( )0 ) )$$$ .. parseNumber+= parseNumber(( . $$ getParen==""""++sum);
16. ;;; System.out.println("sumSystem.out.println("sumsum);$$ .. $.$. out.. .$$. println(( $$ ((++ "sumout$$ ==Systemprintln"sum++sumsum17. }}) ;; }}Source files
Lexical
Lexical analysis
analysis
Token
Token sequence
sequence
Transformation
Transformation
Transformed
Transformed token
token sequence
sequence
Match
Match detection
detection
ClonesClones onon transformed
transformed sequence
sequence
Formatting
Formatting
Clone pairsCode clone analyzer : Gemini

Scatter Plot
D1} Visually shows where code

F1clones are

F3

F4

F2
F3
F4

a b c a b c c c a b d e f a b c c d e f

F1
D2

means that
corresponding two tokens
on the two axes are thesameD1}

F2a b c a b c c c a b d e f a b c c d e f} Both the vertical andhorizontal axes represent
the token sequence ofsource code} The original point is theupper left corner

D2

F1, F2, F3, F4: files
D1, D2 : directories: matched position detected as a practical code clone: matched position detected as a non- interesting code clone

Code clone analyzer : Gemini

Clone/File Metrics} Example of clone metrics

LEN(S): the average length of code fragments (the number of tokens) in clone
set S} clone set : a set of code fragments, in which any pair of the code fragments
is a code clone} NIF(S): the number of source files including any fragments of S}

} Example of file metrics}

ROC(F): the ratio of duplication of file F
if completely duplicated, the value is 1.0} if not duplicated at all, the value is 0.0}

}

NOC(F): the number of code fragments of any clone set in file FAmount of Code Clones in Subsystems CompanyID	

Afterunit testing	Aftercombined testing	# clones	 Duplicatedratio 	# clones	

Duplicatedratio	V	259	34%	

259	

33%	

W	

369	

27%	

379	

26%	

X	

4,483	

55%	4,768	

51%	Y	6,747	

43%	7,628	

46%	Z	2,450	

56%	2,505	

56%	Amount of Code Clones in Subsystems CompanyID	V	W	X	

Afterunit testing	Aftercombined testing	# clones	 Duplicatedratio 	# clones	

Duplicatedratio	

259	

259	

33%	

34%	

369	
27%	
379	
Clones had increased
duringcombinedtesting	 4,768	
4,483	
55%	

26%	
51%	

Y	

6,747	

43%	7,628	

46%	Z	2,450	

56%	2,505	

56%	Scatter Plot (Company Y)

after unit testing	after combined testing	after combined testing	after unit testing	The parts D and E imply the creation ofclones after the unit testing. 	
A	D	E	

B	C	Interview	The developers insist
that they added trusted
library that has used inmany products.	

Scatter Plot (Company Y)

after unit testing	after combined testing	after combined testing	A	D	

after unit testing	The part A treats geographical information ofseveral types of vehicles. The code for thetypes are mostly cloned.	
B	C	

The part B involves statements
for building SQL queries.	
The part C involves initialization andfinalization for a certain feature.	

E	Example of detected clone

Clone metrics-based analysis} Longest clones

A pair of 154 lines clones between the two files} Implications of copy-and-paste and forgetting modification}

Implication/* ...XX.. *//	/* ..XX.. *//	void …XX…()	

void …XX…()	

AAXXBB.cpp	

AAYYBB.cpp	

Implication ofofforgetting
forgetting
modification
tomodification
toYY	
YY			

Example of detected clones

File metrics-based analysis

358 clones	…	} Source file containing the maximum number of clones} Most duplicated pair of source files

96% tokens areduplicated	

Interview	
Developers had expected this duplication since design phase.	

Summary & Future work} Summary	} Discussed the importance of sharing industrial

experiences with clone analysis} Presented industrial application of clone analysis} Many characteristic clones were extracted} According to interviews for some of the extracted clones, thedevelopers expected the existence of clones.
} Future work} Conduct the further analysis for determining

harmful clones or notwhether