SEL@KIT

KYOTO INSTITUTE OF TECHNOLOGYPhoto by Osamu Mizuno

A Fault-Prone Module Detection Using a Spam Filter
KYOTO INSTITUTE OF TECHNOLOGYFault-prone Filtering✦ Fault-prone module detection using a spam filtering

approach[1][2]✦ Uses frequency of terms like spam e-mail filtering✦ Constructs both faulty and non-faulty corpuses from pastmodules✦ Classifies an unknown module using two corpuses[1] O. Mizuno and T. Kikuno, "Training on Errors Experiment to Detect Fault-Prone Software Modules by Spam Filter," In The 6thjoint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations ofSoftware Engineering (ESEC/FSE2007), pp. 405-414, September 2007. (Dubrovnik, Croatia)[2] O. Mizuno, S. Ikami, S. Nakaichi, and T. Kikuno, "Spam Filter Based Approach for Finding Fault-Prone Software Modules," In
Proc. of Fourth International Workshop on Mining Software Repositories (MSR'07), p. 4, May 2007. (Minneapolis, MN, USA)
KYOTO INSTITUTE OF TECHNOLOGYKYOTO INSTITUTE OF TECHNOLOGYHow It Works (Training)
1. Training faulty and non-faulty modules using Tokenizer.
tokensimport, java.foo.Stateful,import, java.foo.AroundInvoke,@Stateful, public, class,EmployeeBean, implements,EmployeeServiceLocal, Employee
ServiceRemote, public,
EmployeeBean, public, void,
doAction, ...

non-faulty
Tokenizer

faultypublic, class, EmployerBean,
implements,ServiceGlobal, ServiceRemote,public, EmployeeBean, public,
void, didAction, ...

non-faulty
corpus
CRM114
Learner

faultycorpusEach corpus holds frequency information of each word
4
KYOTO INSTITUTE OF TECHNOLOGYHow It Works (Prediction)
2. Calculating probability by the Bayesian filter.
unknown

Tokenizer

public, class, public,
EmployeeBean, public, void,
didAction, ...

CRM114
Classifier

non-faulty
corpus

Probability
to be faulty

faultycorpusFault-proneness is determined by the probability and a threshold.

5KYOTO INSTITUTE OF TECHNOLOGYExperiment:

Result of Prediction

Eclipse(random sample)

ActualPredicted
NFP

FP

not
faulty12,2497,093

faulty2,972

16,243

Precision: 0.696 Recall: 0.845

Eclipse BIRT(all modules)

ActualPredicted
NFPFPnotfaulty

70,34916,011

faulty

2,039

7,501

Precision: 0.319 Recall: 0.786

KYOTO INSTITUTE OF TECHNOLOGYThe accuracy improves as thenumber of modules increases.

Apply software modules tofault-prone filter in order ofconstruction and modification.
Only misclassified modules
are trained in corpuses.

Successive Prediction (TOE)

KYOTO INSTITUTE OF TECHNOLOGYFault-proneness filtering with static code analysis
warning messages

non-faulty

faultyunknown

PMD
orCheckStyle

Avoid, variables, with, short,names, like, gd, Local,
variable, ‘, gd, ‘, could, be,declared, finalLocal,variable, ‘, cmp, Basic,‘ ,could, be, declared, final, …

(static codeanalyzer)

non-faulty
corpus
CRM114
Learner

Tokenizer(static codeanalyzer)

PMD
or
CheckStyletokensParameter , ‘, e, ‘, is, not,assigned, and, could, be,declared, final,Parameter, ,‘, isSelected, ‘,
is, not, assigned, and, could,be, declared, final, …

Tokenizer

Parameter, ,‘, isSelected, ‘,is, not, assigned, and, could,be,...

faultycorpus

CRM114
Classifier

Probability
to be faulty

KYOTO INSTITUTE OF TECHNOLOGYResultRecall rises rapidly and becomesstable earlier. But, lower precision
makes overall result worse.

Originalwith PMD

KYOTO INSTITUTE OF TECHNOLOGYPhoto by thenys
www.freeimages.com

Analysis of Identifiers

KYOTO INSTITUTE OF TECHNOLOGYWhat we did
We investigated the frequency of appearance of identifiersin source code modules from the viewpoint of the length ofidentifiers in Eclipse and NetBeans.
We modelled the relationship between the length ofidentifiers and the fault-proneness.
We used the random forest for the modelling.

Data used:
Frequency of identifier’s occurrence for each file grouped by
the length of identifier + faulty status by SZZ
Kimiaki Kawamoto and Osamu Mizuno, "Do Long Identifiers Induce Faults in Software? --- a Repository
Mining Based Investigation ---," In Proc. of 22nd International Symposium on Software Reliability
Engineering (ISSRE2011), Supplemental proceedings, 3-1, November 2011. (Hiroshima, Japan)
KYOTO INSTITUTE OF TECHNOLOGYmodulesdecTabcremodules

Count up for Counteach lengthup for each lengthFig. 1Results of analysis

Fig.chart
1 A
chart of experiments
A flow
offlow
experiments

Table 2

Evaluation of prediction (average of 10 times)

Table 2

Evaluation of prediction (average of 10 times)

EclipseNetbeans

Accuracy
Eclipse
0.888
Netbeans
0.850
Table 3

Table 3

4.

Accuracy
Precision
0.888
0.866
0.850

Precision
Recall
0.866
0.913
0.765

0.765

0.865

Recall F-Measure
F-Measure
0.913
0.889
0.889 0.812
0.865

0.812

Top 10: Mean Decrease Gini (average of 10 times)

Top 10: Mean Decreasetimes)
Eclipse Gini (average of 10Netbeans

Rank Length Mean decrease Gini Length Mean decrease Gini
Eclipse7Netbeans
17767.47
37505.82
Rank Length Meandecrease
Gini6501.14
Length Mean4decrease 6537.75
Gini
24Lengths of3,1737505.82 6505.30
3 7767.47
86146.32
104, and 7 have24
44 6501.14
35643.99
76537.75 6187.07
5 6146.32
135378.26
66505.30 6109.80
810
impact for3
6 5643.9965073.66
115342.15
4376187.07
fault7 5378.26
94883.3996109.80 5176.19
5136854664.90
85156.06
proneness.
665073.66
115342.15
9114200.25
144900.95
7995176.19 4653.10
10 4883.39
144140.85
13854664.9085156.06

4.1

Firs
RQ
thamoRQ

4.2

NexRQandEcl7) oKYOTO INSTITUTE OF TECHNOLOGY[FYI] The longest identifiers
In Eclipse: 237 characters, appears 26 times
WorkingDirectoryStatusHandler_Eclipse_is_not_able_to_set_the_working_directory_
specified_by_the_program_being_launched_as_the_current_runtime_does_not_sup
port_working_directories__nContinue_launch_without_setting_the_working_director
y__2

In Netbeans: 210 characters, appears 76 times
getCustomResourceOrExternalJndiResourceOrJdbcResourceOrMailResourceOrPer
sistenceManagerFactoryResourceOrAdminObjectResourceOrConnectorResourceOr
ResourceAdapterConfigOrJdbcConnectionPoolOrConnectorConnectionPool

KYOTO INSTITUTE OF TECHNOLOGYDo code review activities
become more productivein Gerrit-based projects
as a result of evolution?

Mining code review repositories

Photo by Paul Barker(www.freeimages.com)

14

Data collection tool (Developed)
Integrated crawling and mining tool [1].
Developed by Junwei Liang (a master course student)[1] JunWei Liang and Osamu Mizuno, "Analyzing Involvements of Reviewers Through Mining aCode Review Repository," In Proc. of the Joint Conference of the 21th International Workshop on
Software Measurement and the 6th International Conference on Software Process and ProductMeasurement (IWSM/MENSURA2011), pp. 126-132, November 2011. (Nara, Japan)

KYOTO INSTITUTE OF TECHNOLOGYRatio of activities in review process
1.00

0.75
Activity

countCreate
Inline Comment0.50

LGTM/LabelMessage
0.25

0.00

(Rietv(Riet(Ger(Gerrrit) Qvit) Aneld)eld)t
ChroGWT
droidmium16

Project

Ratio of Labelactivities increases inGerrit-based projects.

(Rietveld) Chromium

number of LGTM is
proportional to thediscussion time.
✦ In Gerrit, +2 labeledissues quickly finish
discussion.

Discussion time (hour)✦ In Rietveld, the

1e+03

1e+01

1e-01

0LGT M
Ni

123>401

Number of LGTM messages(Gerrit) Android

23>4+1+2(Gerrit) Qt

1e+03

Discussion time (hour)

Discussion time andapproval(Rietveld) GWT

1e+01

1e-01

17
Photo by omznmax
Li-2-10+1+2-2Max label value-10