Empirical Analysis of Coding Standard Violation
Focusing on Its Coverage and ImportanceAji Ery Burhandenny∗† , Hirohisa Aman‡ and Minoru Kawahara‡∗ Graduate School of Science and Engineering, Ehime University, Matsuyama, Ehime 790-8577, Japan† Engineering Faculty, Mulawarman University, Samarinda 75119, East Kalimantan, Indonesia‡ Center for Information Technology, Ehime University, Matsuyama, Ehime 790-8577, Japan

Abstract—Toward an effective utilization of static code analysis
tools, this paper investigates which violation is familiar with moreprogrammers and widely appears in source files (having a highcoverage), and which violation is really related to bugfixes (havinga high importance), for six popular OSS projects. The results
show: 1) the familiar violations tend to differ among projects,
and only 25 violations are common to all surveyed projects; 2)
the trends of their importance vary from project to project.

I. I NTRODUCTION
An efficient performance of code review has become a greatchallenge for developers. To support code review, static codeanalysis tools have been developed. However, in many cases,
these tools have not been widely used by developers: a static
analysis tool tends to produce a huge number of warnings but
most of them are false positive ones, i.e., they are not serious
points to be revised [1], [2]. It is one of key reasons why
developers do not actively use such a tool. Therefore, toward
a better utilization of such tools, there have been studies in
the past (e.g, [3], [4]). While previous studies provide notablefindings and proposals, they have not well argued the real
trends of violations (warnings) made by a static analysis tool.
In this paper, we examine the coding violations from theperspectives of how widely they appear and how importantthey are in terms of bugfixes.
II. C ODING S TANDARD V IOLATION
A. Change of Coding Standard Violations through Commits
By using both the code repository and a static code analysis tool, we can get the change history of coding standard
violations in source files. Concretely, for each source file, weobtain all versions of the file from the repository and checkthem by a static code analysis tool. Then, for each violation
which appeared in a version of the file, we can see who madeit, who cleared it and when they made/cleared it.
Next, we consider the relationship of violations with latentbugs. It is not always true that all coding standard violations
are related to the code quality, especially, the bug-proneness.
The strength of the relationship can be seen from the perspective of the violation change history. If the warning count of aviolation in a source file decreased when a bugfix was madefor the file, the violation seems to be related to the fixed bug.
On the other hand, if the warning count increased through abugfix, the corresponding violation is not related to the bug.

By checking the changes of violations through commits, wecan see which violation is stronger related to bugs.
B. Research Questions and Metrics
By observing changes of coding violations through commits, we can see the following two things: 1) who makes
which violations, and 2) which violations are related to bugs.
When a violation has been made by a certain developer, theviolation seems to depend only on the developer. On the other
hand, when a violation has been made by more developers,
it would be a familiar violation which may appear more
frequently. Since the former type of violation is developerspecific, the data analysis and discussion about such a violation
would not be attractive to many researchers and practitioners.
Hence, we will focus on the latter (more frequently-appearing)violations in this paper, and analyze their importance from theperspective of the bug-proneness. Now we set up the following
research questions (RQs) to clarify the aims of our empiricalanalysis:• RQ1: Which coding violations are familiar with moredevelopers and more important in preventing bugs?
• RQ2: Does the difference of project cause differences in
the familiarity and the importance of a violation?
In order to collect quantitative data for answering the RQ1and RQ2, we define metrics for measuring the familiarity andthe importance of a violation.
1) Violation Coverages: Suppose s source files have beendeveloped and maintained by p developers, and w violationsvi (for i = 1, . . . , w) have appeared in those source files.
We quantify the familiarity of violation vi from two different
perspectives—the file coverage and the developer coverage.
The following metric FC(vi ) is the file coverage of vi , which
expresses how widely vi appears in source files: FC(vi ) =ns (vi )/s, where ns (vi ) is the number of source files whichhave experiences with being warned as violation vi .
The following metric DC(vi ) is the developer coverage ofvi , which signifies how widely vi is linked to developers:DC(vi ) = np (vi )/p, where np (vi ) is the number of developers
who have made or increased vi .
Both FC(vi ) and DC(vi ) range [0, 1].
When a violation has both a high FC value and a highDC value, the violation widely appears in source files and isfamiliar with more developers.

2) Violation Importance: As mentioned above, if the warning count of violation vi decreased through a bugfix, vi seems
to be related to the fixed bug, and it would be an importantviolation for preventing a bug inducing. On the other hand, if
the warning count of vi increased through a bugfix, vi seemsto be less important. By focusing on the difference between
the decreased count and the increased one, we define thefollowing metric IMP(vi ) which presents an importance of vi :(vi )∑wninc (vi ), where ndec (vi )IMP(vi ) = ∑wndecndec(vj ) −
j=1
j=1 ninc (vj )and ninc (vi ) signify the numbers of bugfix commits in which
the warning count of vi decreased and increased, respectively.
Since the total number of the decrease commits would differ
from the total number of the increase commits, we define theabove metric by using the normalized values instead of raw
counts. It is like an evaluation of reactions to a news by using
the number of “thumbs up” and that of “thumbs down” which
we often see at a news Website such as Yahoo.com.
A higher value of IMP(vi ) represents that vi has a stronger
link to a latent bug. Such link data can be an empiricalbasis toward an effective code review with using a static codeanalysis tool.
III. E MPIRICAL A NALYSIS
To answer the above RQs, we collect a lot of actual
data from OSS development projects and analyze the coding
violations which had appeared in the source files. In order toensure the generality and usefulness of our empirical results,
we randomly selected six popular OSS projects from theGitHub: (1) Elasticsearch, (2) Guava, (3) JabRef, (4) JUnit4,(5) React Native and (6) Spring Framework.
We computed FC, DC and IMP values of violations andfocused on violations having relatively high coverage, whose
FC and CD values are greater than their medians. As a result,they differ among projects; Table I shows the number ofviolations categorized by “IMP > 0” and “IMP ≤ 0.”
As a consequence, only 25 violations are common to all six
projects (see Table II). There is only one violation whose IMPvalue is always positive among projects, “UnusedImports.”
On the other hand, there are only two violations whose IMPvalues are always non-positive, “CallSuperInConstructor” and“CommentRequired.” That is to say, even if we focus only oncommonly familiar (higher coverage) violations, their trendsof importance differ from project to project.
The violation corresponding to always positive IMP value
is “UnusedImports.” Since this is based on the notion “avoidTABLE I
N UMBER OF VIOLATIONS HAVING HIGH COVERAGE IN EACH PROJECT.
project
Elasticsearch
Guava
JabRefJUnit4React Native
Spring Framework

number of violations
IMP > 0IMP ≤ 050423347
2126
232534322652

TABLE II
C OMMON VIOLATIONS HAVING HIGHER FC VALUES AND DC VALUES .
# of projects
where IMP > 06543210violations
UnusedImports
DataflowAnomalyAnalysis
LawOfDemeter, AccessorMethodGeneration
LocalVariableCouldBeFinal, AvoidInstantiatingObjectsInLoops,
AvoidCatchingGenericException, AtLeastOneConstructor,
AvoidLiteralsInIfCondition, BeanMembersShouldSerialize,
UseConcurrentHashMap, OnlyOneReturn, CommentSize
ConfusingTernary, EmptyCatchBlock, GodClass,UselessParentheses, DefaultPackage
TooManyMethods, CommentDefaultAccessModifier,
ShortVariable, LongVariable, MethodArgumentCouldBeFinal
CallSuperInConstructor, CommentRequired

unused import statements,” it does not seem to be a directcause of bug. This violation might be cleared by reorganizing
code when a bug is fixed.
While we found commonly-familiar 25 violations, theirimportance vary from project to project as shown in Table
I. Since there are only a few violations which are commonlyimportant or commonly-worthless, it would be better to tune
the priorities of violations to the project, and build a projectspecific evaluation model by using the feedback from itschange history of violations.
IV. C ONCLUSION
We focused on coding standard violations warned by a static
code analysis tool, and proposed to evaluate them from theperspective of the familiarity—a file coverage and a developer
coverage—and the importance—a strength of relationships
with bugfixes. We conducted an empirical analysis of coding
violations appearing in six popular OSS projects. As a result,familiar violations seemed to differ among projects, and only
25 violations were common to all surveyed projects. Moreover,
the trends of their importance varied from project to project.
Therefore, we found that it is better to tune the assessmentsof violations for each project, rather than to build a general
guideline of coding violations.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number 16K00099 and DIKTI Scholarships, Directorate Generaleof higher Education of Indonesia.
R EFERENCES[1] F. Wedyan, D. Alrmuny, and J. M. Bieman, “The effectiveness of automated static analysis tools for fault detection and refactoring prediction,”in Proc. Int’l Conf. Softw. Testing Verification & Validation, Apr. 2009,
pp. 141–150.
[2] S. Panichella, V. Arnaoudova, M. D. Penta, and G. Antoniol, “Wouldstatic analysis tools help developers with code reviews?” in Proc. Int’lConf. Softw. Analysis, Evolution, & Reeng. , Mar. 2015, pp. 161–170.
[3] Q. Hanam, L. Tan, R. Holmes, and P. Lam, “Finding patterns instatic analysis alerts: Improving actionable alert ranking,” in Proc. 11thWorking Conf. Mining Softw. Repositories, May 2014, pp. 152–161.
[4] J. P. Ostberg, S. Wagner, and E. Weilemann, “Does personality influence
the usage of static analysis tools? an explorative experiment,” in Proc.
IEEE/ACM Cooperative & Human Aspects of Softw. Eng., May 2016, pp.
75–81.