2017 IEEE International Conference on Software Maintenance and Evolution

Flattening Code for Metrics Measurement and Analysis
Yoshiki Higo and Shinji Kusumoto
Graduate School of Information Science and Technology, Osaka University,
1-5, Yamadaoka, Suita, Osaka, 565-0871, Japan
Email:{higo, kusumoto}@ist.osaka-u.ac.jp

Abstract—When we measure code metrics or analyze source
code, code normalization is occasionally performed as a preprocessing. Code normalization means removing untargeted program elements, formatting source code, or transforming sourcecode with speciﬁc rules. Code normalization makes measurement
and analysis results more signiﬁcant. Existing code normalization mainly targets program elements not inﬂuencing program
behavior (e.g., code comments and blank lines) or program tokens(e.g., variable names and literals). In this paper, we propose a newcode normalization technique targeting program structure. Our
proposed technique transforms a complex program statement tosimple ones. We call this transformation ﬂattening. By ﬂattening
code, we can obtain source code including only simple program
statements. As applications of the code ﬂattening, we report how
it changes LOC metric and clone detection results.

to absorb their differences. As just described, the targets ofexisting code normalization are peripheral program elements
not inﬂuencing program behavior and program tokens.
In this paper, we propose a new code normalization technique, which is targeting program structure. The proposed
technique transforms complex program statements to simple
ones. In other words, source code including both complex andsimple statements is transformed to the one including only
simple statements. We call this transformation ﬂattening. By
ﬂattening source code, the amount of functionality in every
code line becomes homogenized. Thus, LOC values become
more signiﬁcant. The program structure is also uniformed bycode ﬂattening. Thus, more code fragments having identical/similar behavior are detected as clones.
In this paper, we also report how LOC value and clone
detection results are changed by code ﬂattening. We conﬁrmedthat there were many source ﬁles whose LOC values got
increased greatly and much more clones were detected in
ﬂattened source ﬁles than original ones.
The main contributions of this paper are as follows:• proposing a new code normalization technqiue,• investigating how LOC and clone detection results arechanged by the proposed technqiue, and• investigating whether the proposed technique raises correlations between LOC and fault-prone modules or not.

I. I NTRODUCTION
Program source code is one of the main targets of researchin the ﬁeld of software engineering. There are various purposesof the research targeting source code.
Understanding: a purpose is helping to understand sourcecode. Quantifying size, complexity, and characteristics of source code is a way to objectively grasp
its status [1]. Dependencies among modules such as
method invocation and class inheritance are worth
being visualized for understanding [2], [3].
Reuse: on research for effective or efﬁcient software reuse,
source code is a main target. Identifying frequentlyreused software components [4] and suggesting instructions such as method invocations for half-written
code [5] are active research topics.
Suggestion: pointing out latent issues in source code orsuggesting corrective strategies for them also havebeen studied for many years. Various methodologies
have been proposed for identifying fault-prone modules [6], [7] and suggesting refactorings [8].
Metrics measurement and code clone (in short, clone)
detection are fundamental techniques widely used in researchon program source code. LOC, cyclomatic complexity, andC&K metrics suite [9] are well-known code metrics. There
are many techniques to detect clones on different granularities
such as ﬁle level, method level, and code fragment level [10].
In metrics measurement and clone detection, code normalization is occasionally performed to obtain more signiﬁcantresult. Herein, code normalization means transforming sourcecode with speciﬁc rules. For example, in measuring LOC,
removing code comments and blank lines and formatting
source code are typical code normalization. In detecting
clones, variables and literals are replaced with special tokens
978-1-5386-0992-7/17 $31.00 © 2017 IEEE
DOI 10.1109/ICSME.2017.65

II. R ELATED W ORK
Code normalization is an often-used technqiue in clone
detection [11], [12], [13], [14], [15], [16], [17]. In clone
detection, variables are replaced with special tokens to ignore
their differences. There are two typical ways for variablenormalization; the ﬁrst one is replacing all variables with thesame special token; the other is replacing each variable witha different special token. The latter one is better from theviewpoint of detecting less false positives.
One way to support source code analysis is transforming
source code to another form. Cordy proposed TXL, which
is a framework to support source code analysis and source
transformation tasks [18]. Maletic proposed srcML, which isa framework to present source code in the XML form [19].
By presenting source code in the XML form, existing various
tools for XML documents can be applied.
Techniques to remove goto statements [20], [21] are a kindof structure normalization. Those techniques leverage control
ﬂow graphs. The structure of target source code is transformed
so as not to include goto statements without changing thestructure of the control ﬂow graphs.
494

#,/0  &' (,/1 &   '(,/2 &   65 "&''(,/3   5*!
,0*   5 !
,0+) (,0,
,0, " &  44'!,0),0-)
,0.)
,0/&  &'%5'(,00  5  &'" &'!
,01& 55$+'(,02   &'!
,03
& &',03
&
&'  &''(,1*    5!
,1+  5+!
,1, " &*'!
,1-)
,1.) (,1/   && ''!
,10!
,11)
,12)
,13$+!
,2* )
#


$!
$"
"'%
"'&
"''
$%

%"
%#
"(&
"('
"((




  &     
!"#* )) 
!"$*!"$*   !"#!"$ &&    !"'* 
 
!"(* !"'!"(!"'!"(  & &

Fig. 2. Source code ﬂattened by the proposed techniquesuch as semantic clone detection or graph-based clone detection because of their scalability. In token-based techniques,given source code is transformed to a sequence of tokens andidentical subsequences are detected as clones. In AST-based
techniques, given source code is parsed to generate ASTs.
Sub-trees in the ASTs having identical/similar structures aredetected as clones. Assume that there are two same-behavior
code fragments: one includes some complex statements while
the other consists of only simple statements. In such a case,existing clone detection techniques cannot detect the two codefragment as clones.
In this research, we are trying to transform complex structures in source code to simple ones without changing its
external behavior. The authors call such a transformation
ﬂattening in this research. By ﬂattening source code, complex
statements are gone. Consequently, the amount of functionality
in each line gets homogenized, so that the LOC becomes moresigniﬁcant values. Besides, same-behavior code fragments get
more similar structures by ﬂattening, which makes it easier
to detect them as clones with clone detection techniques. In
other words, code ﬂattening enables a part of semantic clones
to be detected by syntax-based clone detection techniques.
Figure 2 shows source code ﬂattened by our proposed
technique. Its original version is shown in Figure 1. For
example, the 262nd line in the original code was transformed
to the 286–288th lines. We can see that the postﬁx expression
and the method invocation one are extracted from the complex
return-statement.

Fig. 1. Source code including complex statements

III. R ESERCH M OTIVATIONThe source code in Figure 1 includes both simple statements(e.g., the 259th line) and complex statements (e.g., the 269thline). The authors think, the fact that complex statements are
mixed with simple ones have negative inﬂuences on metrics
measurement and source code analysis.
As an example of metrics measurement, we consider measuring LOC. LOC is a metric to present a size of a given sourceﬁle. In LOC measurement, blank lines and/or code commentsin the ﬁles are ignored to obtain more signiﬁcant values as a
size metric.
In using the LOC metric for guessing the amount offunctionality in the source ﬁle or for identifying fault-prone
modules, the fact that complex statements are mixed withsimple ones can be a factor that degrades its performance.
For example, it is not easy to know whether a given 50line code fragment consisting of complex statements includes
more functionality than another given 100-line code fragment
including only simple statements or not. Moreover, if we
use the metric values (50 and 100), techniques of faultprone module identiﬁcation will regard that the l00-line codefragment has a greater probability of including faults than the
50-line one1 .
As an example of source code analysis, we consider detecting clones from source code. Detected clones are used
to identify code fragments including the same faults or to
be targets of code uniﬁcation refactoring. Like this, clone
detection techniques are used to detect code fragments having
identical/similar behavior. Token-based and AST-based clone
detection techniques are more often used than other techniques

IV. P ROPOSED T ECHNIQUE : C ODE F LATTENING
The proposed technique depends on abstract syntax tree (inshort, AST) implementation. Herein, we explain the proposed
technique with Java Development Tools (in short, JDT). JDTis a widely-used Java source code analysis tool. It takes a setof source ﬁles and builds ASTs. The tool has a function toresolve name bindings.
The proposed technique works on ASTs built by JDT. For
each of the target sub-trees in given ASTs, the proposed
technique checks whether it is enough complex to be ﬂattened
or not. If the sub-tree is complex, its portion is removed fromthe original position and a new sub-tree including the portion is
generated in a different position. The new sub-tree is a variable

1 Using the LOC metric for guessing the amount of functionality may be
an old-fasioned example. However, the LOC metric is still widely used. For
example, in mining software repositories. the code churn metric is well used.
Code churn is calculated from the LOC of changed code.

495       

 

        

"         

    
 !!

  

 !!

          
   

             "  "                         !!

       





Fig. 3. AST transformation of the proposed technique

technique. The upper-left tree is the AST sub-tree for the
261st and 262nd lines of Figure 1. Sub-trees are manupulatedwith depth-ﬁrst search and postorder traversal. In this example,
ﬁrstly the node of the postﬁx expression is regarded that it
increases the complexity of the return statement. Thus, a new
VDS for the postﬁx expression is added and the sub-tree ofthe postﬁx expression is replaced with a reference to the newly
added variable $24. Then, the node of the method invocation
expression is also regarded that it increases the complexity of
the return-statement. This sub-tree is manipulated in the same
way as the postﬁx expression.
Figure 4 is another example of ﬂattened code. The left
side is its original code. The conditional predicate of the ifstatement is very complex, so that ﬁve VDSs are created toreduce the complexity of the conditional predicate. Like this,the proposed technique introduces more VDSs as target subtrees are more complex.
In Figure 4, the conditional predicate of the for-statement isalso a target of ﬂattening. In a case of loop condition, a VDS
is added to just before the loop and a substitution statement
is added to the end inside the loop2 . Both the VDS and thesubstitution statement are required to keep program behavior
after code ﬂattening.

declaration statement (in short, VDS) and its right-hand codeis the removed sub-tree. A reference to the new variable isadded to the original position where the portion was removed.
The newly introduced variable has a type that is the same as
its right-hand code. This transformation strategy realizes thatthe source code can be compiled and executed even after it
has been ﬂattened.
The following AST nodes are targets to check whether they
are complex or not:• program statements,• conditional predicates for branchs and loops, and• initializers and updaters of for-statements.
If the above targets include multiple expressions, such
expressions are extracted as new VDSs. Table I shows a list oftarget expressions. The following expressions are not includedin the extraction targets because they are not complex:• variable reference expression,• literal expression, and• null expression.
Figure 3 shows how code is ﬂattened by the proposed

TABLE IE XTRACTION TARGET JDT NODES
ArrayAccess
ArrayInitializer
ClassInstanceCreation
ExpressionMethodReference
InstanceofExpression
PostﬁxExpression
SuperConstructorInvocation

V. E XPERIMENT
We conducted an experiment with our tool3 . In this experiment, we investigate how LOC and clone detection results

ArrayCreation
CastExpression
ConstructorInvocation
InﬁxExpresssion
MethodInvocation
PreﬁxExpression
SuperMethodInvocation

2 In the strict sense, there is no deﬁnition of substitution statement in Java
speciﬁcation. Herein, a substitution statement means an expression statement
including a substitution operator.
3 https://github.com/YoshikiHigo/JCodeFlattener/

496

Flattened source code

Original source code    1   " #""!1      "##"##!      "#  "##$   1     "# "#  &   &    1   " #  '*1'*1      "#  '+1!1'*   ',
1      "#',1  '
1 ',  "#'-1',
  '.

1!'-'.1!'-"'+'.# $"'+'.#$   '/ 1      "#   1'/ "#  &   &
%

(1)

%
 2    "#"  1("# 00#$  "  " ##%(2) ')(1    "# 2')("  1(2')( 00#$ '))1  " #  "'))#')(1    "#')(1

(1) Many VDS are extracted from a very
complex statement.
(2) A VDS and a substitution statement are
extracted from loop condition.

%









Clone Increasing Rate

LOC Increasing Rate

Fig. 4. An example of code frattening with the implemented tool

















 



 



 

























LOC of the original code at project level

LOC of the original code at project level

Fig. 5. Results of LOC Increasing Rate calculation on project level

Fig. 6. Results of Clone Increasing Rate calculation on project level

are affected by code ﬂattening. We leveraged the dataset that
we created and published in our previous research [22]. The
dataset is open to the public in our web site4 . The dataset
includes two packages. Each package includes source code of
84 or 500 projects. Table II includes more detailed numerical information. The dataset includes neither test cases nor
automatically generated code. They were removed manually.
Figure 5 shows how LOC at project level was changed bycode ﬂattening. The LOC Increasing Rate is calculated withthe following formula. LOCori and LOC f la represent LOC ofthe original code and ﬂattened code, respectively.

As shown in this ﬁgure, the dispersion of the LOC Increasing
Rate becomes larger for smaller projects. If original code size
is 25K LOC or less, the code size was increased more than
one-and-a-half times for several projects. On the other hand,for 100K LOC or larger projects, the LOC Increasing Rate
falls roughly within the range of 20% and 35%. The results
imply that we should not use LOC as an indicator of ourdecision makings, especially for small projects.
Figure 6 shows how the number of detected clones isaffected by code ﬂattening. The Clone Increasing Rate is calculated with the following formula. CLONEori and CLONE f larepresent the number of clones detected from the original codeand ﬂattened code, respectively.

IncreasingRateLOC = 100 ∗LOC f la − LOCori
LOCoriIncreasingRateCLONE = 100 ∗4 http://sdl.ist.osaka-u.ac.jp/˜higo/fse2014/

As shown in this ﬁgure, the Clone Increasing Rate varies more
widely in smaller projects. This is the same tendency as LOC
metric. For 25K or smaller projects, 10 times or more clones
were detected in some cases. On the other hand, for mostof 100K or larger projects, the Clone Increasing Rate is lessthan ﬁve. The results show that code ﬂattening allows clone
detection tools to detect much more clones.

TABLE IITARGET PROJECTS IN E XPERIMENT A
Package# projects# ﬁlesTotal LOC

APACHE84
66,724
11,545,556

CLONE f la −CLONEori
CLONEoriUCI
500
60,548
10,073,635

497

Detected as clones

small-size clones, source code will get more suited for metrics
measurement and clone detection.
We are also going to compare clone detection results on ﬂattened code with semantic clone detection results and programdependence-graph-based clone detection results. We consider
code ﬂattening enables syntax-based detecton techniques todetect a part of semantic clones and graph-based clones.

(
367
 ./  $%!$-   . /368-)./369'"
369
'" -* ,,+.&36:
36:  &36;036;
0145<'145< ' -.&     <    '  #-145.&
374146<374
3
146 < '-.&375-)146./375
3-)146./376'"
376
3'" -* ,, +.&377
377
3 &37803379    <'  -.&3<'
37:
3
  <'
 -.&-.
37;-
'
+../
37;3
 -   -* +..
/(4;;
./
4  $%!!$-   . /522-)./5523'"
523
5'" -* ,,+.&524 &552505
  <<'' -.&     <    '  #-.&52819;< '-.&5529-)19;./5
52:'"52:
'" -* ,, +.&52; &5320<'
533    <
'  -.&534  <' -.&535
<
535
    &(ACKNOWLEDGMENT
This work was supported by MEXT/JSPS KAKENHI
25220003 and 17H01725.
R EFERENCES[1] N. E. Fenton and S. L. Pﬂeeger, Software Metrics: A Rigorous andPractical Approach, Third Edition, 3rd ed. Boston, MA, USA: PWS
Publishing Co., 2014.
[2] R. Koschke, “Software Visualization in Software Maintenance, Reverse
Engineering, and Re-engineering: A Research Survey,” Journal of Software Maintenance, vol. 15, no. 2, pp. 87–109, 2003.
[3] M.-A. D. Storey, D. Čubranić, and D. M. German, “On the Use of
Visualization to Support Awareness of Human Activities in Software
Development: A Survey and a Framework,” in Proc. of SOFTVIS, 2005,
pp. 193–202.
[4] M. Acharya, T. Xie, J. Pei, and J. Xu, “Mining API Patterns As Partial
Orders from Source Code: From Usage Scenarios to Speciﬁcations,” inProc. of ESEC/FSE, 2007, pp. 25–34.
[5] V. Raychev, M. Vechev, and E. Yahav, “Code Completion with Statistical
Language Models,” in Proc. of PLDI, 2014, pp. 419–428.
[6] C. Catal and B. Diri, “A Systematic Review of Software Fault Prediction
Studies,” Expert Systems with Applications, vol. 36, no. 4, pp. 7346–
7354, 2009.
[7] V. B. Livshits and M. S. Lam, “Finding Security Vulnerabilities in Java
Applications with Static Analysis,” in Proc. of USENIX, 2005, pp. 18–
18.
[8] T. Mens and T. Tourwé, “A Survey of Software Refactoring,” IEEE
Transactions on Software Engineering, vol. 30, no. 2, 2004.
[9] S. R. Chidamber and C. F. Kemerer, “A Metrics Suite for Object
Oriented Design,” IEEE TSE, vol. 20, no. 6, pp. 476–493, 1994.
[10] D. Rattan, R. Bhatia, and M. Singh, “Software Clone Detection: A
Systematic Review,” Information and Software Technology, vol. 55,
no. 7, pp. 1165–1199, 2013.
[11] H. A. Basit, S. J. Puglisi, W. F. Smyth, A. Turpin, and S. Jarzabek,“Efﬁcient Token Based Clone Detection with Flexible Tokenization,” inProc. of ESEC/FSE, 2007, pp. 513–516.
[12] S. Ducasse, O. Nierstrasz, and M. Rieger, “On the effectiveness of clone
detection by string matching: Research articles,” Journal of Software
Maintenance and Evolution, vol. 18, no. 1, pp. 37–58, 2006.
[13] B. Hummel, E. Juergens, L. Heinemann, and M. Conradt, “ndex-based
Code Clone Detection: Incremental, Distributed, Scalable,” in Proc. of
ICSM, 2010, pp. 1–9.
[14] E. Juergens, F. Deissenboeck, B. Hummel, and S. Wagner, “Do code
clones matter?” in Proc. of ICSE, 2009, pp. 485–495.
[15] E. Juergens, F. Deissenboeck, and B. Hummel, “CloneDetective – A
Workbench for Clone Detection Research,” in Proc. of ICSE, 2009, pp.
603–606.
[16] H. Murakami, K. Hotta, Y. Higo, H. Igaki, and S. Kusumoto, “Folding
Repeated Instructions for Improving Token-Based Code Clone Detection,” in Proc. of SCAM, 2012, pp. 64–73.
[17] C. K. Roy and J. R. Cordy, “NICAD: Accurate Detection of Near-Miss
Intentional Clones Using Flexible Pretty-Printing and Code Normalization,” in Proc. of ICPC, 2008, pp. 172–181.
[18] J. R. Cordy, “The TXL Source Transformation Language,” Science ofComputer Programming, vol. 61, no. 3, pp. 190–210, 2006.
[19] J. I. Maletic, M. L. Collard, and A. Marcus, “Source Code Files As
Structured Documents,” in Proc. of IWPC, 2002, pp. 289–292.
[20] Z. Ammarguellat, “A Control-Flow Normalization Algorithm and Its
Complexity,” IEEE TSE, vol. 18, no. 3, pp. 237–251, 1992.
[21] F. Zhang and E. H. D’Hollander, “Using Hammock Graphs to Structure
Programs,” IEEE TSE, vol. 30, no. 4, pp. 231–245, 2004.
[22] Y. Higo and S. Kusumoto, “How Should We Measure Functional
Sameness from Program Source Code? An Exploratory Study on Java
Methods,” in Proc. of FSE, 2014, pp. 294–305.

Fig. 7. Newly detected clones by ﬂattening code

Figure 7 shows a pair of clones detected from the ﬂattened
code but not detected from the original code. The 150th and151st lines were a single statement in the original code. On theother hand, the 304th and 305th lines are in this form in theoriginal code. The differences prevent the clones from being
detected by the clone detection tool. By ﬂattening the code,
the differences were removed and the code fragments weredetected as clones, especially for small projects.
Code ﬂattening has a side effect in clone detection. Moretrivial clones are detected from ﬂattened code because code
ﬂattening increases the number of tokens, the number ofstatements, and the number of AST nodes. In this experiment,
we do not check ratio of trivial clones in the detection results.
VI. C ONCLUSION
In this paper, we proposed a code ﬂattening technique,which dissolves a complex program statement to multiple
simple ones. By ﬂattening code, the amount of functionality
in every line gets homogenized. Thus, values of code metrics
such as LOC become more signiﬁcant. Besides, more clones
are detected from ﬂattened source code.
The proposed technique has been implemented and it isopen to the public in our web site5 . We applied the tool to
584 open source software in total and found that there weremany source ﬁles whose LOC became much greater by code
ﬂattening. We found that the code ﬂattening allowed much
more clones to be detected. Furthermore, we found that the
LOC of ﬂattened code was more useful than the one of originalcode in the context of fault-prone module identiﬁcation.
The next step of our research is developing methodologies
of refactorings for ﬂattened code because small-size clones
are ocassionaly generated by code ﬂattening. By merging such
5 https://github.com/YoshikiHigo/JCodeFlattener/

498