Life beyond CMMI Level 5:
The Empirical Approach to Software
Engineering
Dr. Katsuro Inoue
Prof. Michael Barker
9/5/2006 (v9)2006/9/12

NASSCOM 2006

Introductions
Who are we?
Who are you?
What is EASE?
Plans for the day

NASSCOM Quality Summit~2~Who are we?
Dr. Katsuro Inoue, Osaka University
co-leader of the EASE project
Professor Mike Barker, Nara Institute of Science andTechnology
Researcher, EASE project

NASSCOM Quality Summit~3~Some Rules for the Day
First, informal meeting. Please ask questions!
Second, each of you should make a “take aways” page. This is just asingle page of paper with whatever key points you would like toremember. NASSCOM has suggested that if you will provide your“take aways” page, they will collate and then provide a copy of theconsolidated page to everyone.

NASSCOM Quality Summit~4~Who are you?
Please pair up
Introduce yourself to your partner.
What is your name?
Where are you from?
Why are you interested in this tutorial?

Now, introduce the rest of us to your partner.

NASSCOM Quality Summit~5~Who are you?
What kind of work do you do?
Quality/Process improvement
Software engineering
Project management
Upper management
Consultants
Other?

Are you a software consumer? Producer?
Industry? Academic? Government?

NASSCOM Quality Summit~6~Who are you?
(specialization)
1
0.9
0.8
0.7
Quality/PI
Software Engineers
Project Managers
Upper Management
Consultants
Others

0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Categories~7~

Who are you?
1
0.9
0.8
0.7
0.6

Consumer
Producer

0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Consumer/Producer~8~Who are you?
1
0.9
0.8
0.7
Industy
Academia
Government
Other

0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Field
~9~

Who are you?
Which of these are you interested in?
CMMI
Empirical Methods
Data collection and measurement
Data analysis
Data feedback
Interaction of quality and empirical methods
Japan

NASSCOM Quality Summit~10~Who are you?
(Interests)
1

CMMI

0.9

Empirical Methods

0.8
0.7

Data collection andmeasurement
Data analysis

0.6
0.5
0.4

Data feedback

0.3
0.2

Interaction of quality
and empirical methods
Japan

0.1
0

Interest

NASSCOM Quality Summit

Problems~11~What is EASE?
Sit back and relax a moment, this is our advertisement ☺

NASSCOM Quality Summit~12~The EASE Project

2006/9/12

NASSCOM 2006

What is the EASE project?
Empirical Approach to Software Engineering
One of the leading projects of the Ministry of Education, Culture, Sports, Scienceand Technology (MEXT).
5 year project starting in 2003.
Budget: $2 million US / year.
Project leader: Koji Torii, NAIST
Sub-leader:
Katsuro Inoue, Osaka University
Kenichi Matsumoto, NAIST

http://www.empirical.jp/English/

NASSCOM Quality Summit~14~

The purpose of the EASE project

Achievement of software development technology based on quantitative data

Construction of a quantitative data collection system

Result 1: Making of EPM open source
Construction of a system that supports development based on analyzed
data

Result 2: EPM application experience
Result 3: Coordinated cooperation with SEC
Spread and promotion of software development technology based on quantitative
data to industry sites

Result 4: Activation of the industrial world (e.g.
Toshiba, Unisys Japan, Fuji Film)

NASSCOM Quality Summit~15~

Empirical activities in EASE
Data collection in real time, e.g.

configuration management history
issue tracking history
e-mail communication history
Data
Collection

Analysis with software tools, e.g.

metrics measurement
project categorization
collaborative filtering
software component retrieval

Data
observations and rules
Analysisexperiences and instances in previous projects

Feedback to stakeholders for improvement, e.g.

NASSCOM Quality Summit~16~

Feedback

The EASE roadmap

Social impact
Effectiveness

Data collection

SoftwareDevelopment With
High Reliability andProductivity

Upstream
Product Data
Production Data
Data analysis
Project
Plan Data
SuggestionsFeedback Data
Sharing
Estimation
Communication
Alternatives
Data
Exception Analysis
Industry Level Sharing
Related Cases
Quality Data
Characterization
Analysis
Sharing Between Organizations
Results
Downstream
Product Data
Sharing Between Projects
Collected data
Sharing Between Developers

2003/4

NASSCOM Quality Summit

2004/4

2005/4

2006/4~17~

2007/4

2008/3

EASE and SEC
Software Engineering Center (SEC) Japan is a project under METI
Strong cooperation
SEC (IPA) has issued an RFP to develop a commercial quality toolset using theexisting EASE EPM and collaborative filtering. This three year project has thefollowing stages:

Develop easy-to-use distribution kit of measurement tools
Practical usage in ten trial projects
Propose service business using measurement database

NASSCOM Quality Summit~18~

The EASE Empirical Tool

SEC Benchmark Database(aka 1000 projects DB)

EPMplus measures
CF Project Selection
CF Estimation

CF Selection
Function
Language
Dev. Type
Dev. Cost
Points# of Staffs

Selected Similar Projects

Current Project EPMplus

NASSCOM Quality Summit

2000Prediction:
7 ? (MV)59.99

Project X

Java

New

Project A

Java

New ? (MV) 8

Project B

Java ? (MV) 2500

1070

Project C ? (MV)
Maintenance
5000

20

250

50

Current Project Estimated by CF~19~

In-process
In-process Project
Project Measurement
Measurement andand Feedback
Feedback
A new research framework for applying empirical software engineering methods in industrial practiceand accomplishments in using it.
The selected target : a governmentally funded software development project involving multiple vendors.
In-process project data measurement in real time.
Data sharing with industry and academia (I & A).
Data analysis, and feedback to the project members.
*) EPM: Empirical Project monitor

Feedback to in-process Software Development Project
RDB
EPM*
Development
Environment

XML Standard Data Format
Configuration
Management
SystemProgramDevelopment

Analyzer

Bug
Management

B

(CVS)
Source Code

Bug
ReportA

Bug Tracking
System(GNATS)

Code Clone
Analyzer

Mailing-list
Management
SystemMail
e-MailC

類似度の分布Similarity
Distribution

度数

number
100
90
80
70
60
50
40
30
20
10
0

Review
Report Sheet

Review

0.0～0.1

Interview
Check-list
Analyzer

Q&A Check-list

Task
Management

Integration

Fig. In-process Project Measurement and Feedback Structure

scope

5
4
3
2
1
0

Motivation
Technology
Organization

Time
Cost
Quality

BasicOperation
Customer
Procurements

NASSCOM Quality Summit

0.1～0.2

0.2～0.3

0.3～0.4

0.4～0.5

0.5～0.6

0.6～0.7

0.7～0.8

0.8～0.9

類似度の値域

Similarity

SEC Benchmark Database(over 1000projects)

Benchmark Data Report
Q&A Interview
to PM

0.9～1.0

Collaborating
Filtering
tool

~20~

Human
Resource
Risk

Communication

Today’s Agenda
Time

Topic

9:00-10:00

Introductions

10:15-11:15

Collecting empirical data

11:30-12:30

Analyzing empirical data

12:30-1:30

Lunch

1:30-2:30

Distributing the results

2:45-3:45

Using empirical methods for quality improvement

4:00-5:00

Next steps, summary and conclusions

NASSCOM Quality Summit~21~

BREAK!

2006/9/12

NASSCOM 2006

Collecting Empirical Data

2006/9/12

NASSCOM 2006

Collecting Empirical Data
1.
2.
3.
4.

Why collect data? The Empirical answer
GQM Planning
An Exercise
Group Data

NASSCOM Quality Summit~24~

Why do we collect data?
How are you going to use it?
Empirical approach use experimentation or intervention
model to measure effects.

NASSCOM Quality Summit~25~

Empirical Methodologies
Why do we need empirical evidence?
How can we get it?

NASSCOM Quality Summit~26~

Empirical Methodologies
Why Empirical Approach?
Software engineering practical
Criteria for success include quality such as:
Accuracy
Appropriateness
Functionality
Reliability
Usability
Efficiency
Maintainability
Portability
Timeliness
Cost effectiveness
Customer satisfaction

Human variation of production
Environment variation
Wide variety of products
Hard to establish guiding principles
Tendency to base practice on experience, hearsay, and general folklore and myth

Dawson, Bones, Oates, Brereton, Azuma, and Jackson (2004).
NASSCOM Quality Summit~27~

Empirical Methodologies
To be engineering, we need:
Observations of working practices
Theories and hypotheses
Testing to validate

Dawson, Bones, Oates, Brereton, Azuma, and Jackson (2004).

NASSCOM Quality Summit~28~

Two Views of The World
Facts and laws must be universallytrueLargely rely on controlled
experiments, isolating independentand dependent variables to establish
cause and effect
Replication
Positivism

It depends on the context
Understand phenomena throughmeanings and values that people
assignExplore and explain how all thefactors in study are related andinterdependent
Unique cases
Interpretivism

Dawson, Bones, Oates, Brereton, Azuma, and Jackson (2004).

NASSCOM Quality Summit~29~Instant Poll!
1
0.9
0.8
0.7
0.6
Universal Laws?
Context Rules?

0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Which View?
~30~

A Pyramid of Empirical Research Types

Positivism
Controlled
Experiments
Experimentswith students
Surveys,multiple case studies
Case studies, action research,
ethnography
Anecdotes, story telling, diaries

Interpretivism
NASSCOM Quality Summit

Dawson, Bones, Oates, Brereton, Azuma, and Jackson (2004).

~31~

Framework For Mixed Method Research
Approach
Sequential

PurposeParallel

Independent

Triangulation(corroboration)
Complementarity(elaboration)
Development

Initiation

Expansion
Petter and Gallivan. (2004).

NASSCOM Quality Summit~32~

A Typology of Research Methods
Researcher's Assumption About Results

Topic ofStudy

Contextual (Kairotic)

Universal (Chronotic)

Concept (Idea)
Study of Concepts

Research Review
Conceptual Book Review

Philosophical Study
Simulation

Objects
Study of
Objects

UniqueObjects

History

Case Study
Post Hoc Analysis

Class or Set of
Objects

SurveyExperimentMeta-analysis

Counelis (2000).
NASSCOM Quality Summit~33~

What Evidence Do We Need?
Aim

Evidence

StudiesDevelop understanding of
nature of SE practice

Natural data

Field or case study
Interviews, think-aloud

Exploring how practice might
be improved

Management evidence
Practitioner evidence

Quantitative experiments
Case studies

Evaluation of effects ofintroducing improvement into
practice

Baseline intervention measure Field or case studies
qualitative

Segal (2004).

NASSCOM Quality Summit~34~

Four General Categories
Scientific method: theory, hypothesis and experimentation
Engineering method: develop and test a solution to ahypothesis, test and improve
Empirical method: statistical method used to validate
hypothesis
Analytic method: formal theory, deductions compared with
empirical observations

Zelkowitz and Wallace (1998)

NASSCOM Quality Summit~35~

Aspects of data collection
Replication: can we do it again?
Local control: can we control the treatment?
Influence: does the experimental design assumed passive
objects or active objects
Temporal properties: is the data collection historical or
current?

Zelkowitz and Wallace (1998)

NASSCOM Quality Summit~36~

SE Validation Methods
Category

Validation method

Observational

Project monitoring
Case study
Assertion
Field study

Historical

Literature search
LegacyLessons learned
Static analysis

Controlled

ReplicatedSynthetic
Dynamic analysis
Simulation
Zelkowitz and Wallace (1998)

NASSCOM Quality Summit~37~

Six Topic Areas
1.
2.
3.
4.
5.
6.

Experimental context
Experimental design
Conduct of the experiment in data collection
AnalysisPresentation of results
Interpretation of results

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~38~

Experimental Context
1.
2.
3.
4.

Be sure to specify as much of the industrial context as possible. In particular,
clearly define the entities, attributes, and measures that are capturing thecontextual information.
If a specific hypothesis is being tested, state it clearly prior to performing thestudy and discuss the theory from which it is derived, so that its implications areapparent.
If the research is exploratory, state clearly prior to data analysis what questions
the investigation is intended to address and how it will address them.
Describe research that is similar to, or has a bearing on, the current research andhow current work relates to it.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~39~Experimental Design (1)1. Identify the population from which the subjects and objects are drawn.
2. Define the process by which the subjects and objects were selected.
3. Define the process by which subjects and objects are assigned to treatments.
4. Restrict yourself to simple study designs or at least two designs that are fully
analyzed in the statistical literature.
5. Define the experimental unit.
6. For formal experiments, perform a pre-experiment or precalculation to identify or
estimate the minimum required sample size.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~40~Experimental Design (2)7.
8.

Use appropriate levels of blinding.
If you cannot avoid evaluating your own work, and make explicit any vested
interests (including your sources of support) and report what you have done tominimize bias.
9. Avoid the use of controls unless you are sure the control situation can be
unambiguously defined.
10. Fully define all treatments (interventions).
11. Justify the choice of outcome measures in terms of their relevance to theobjectives of the empirical study.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~41~

Conducting the Experiment and Data Collection
1.
2.
3.
4.
5.
6.

Define all software measures fully, including the entity, attribute, unit and counting rules
For subjective measures, present a measure of interrater agreement, such as the kappa
statistic or the intraclass correlation coefficient for continuous measures.
Describe any quality control method used to ensure completeness and accuracy of datacollection.
For surveys, monitor and report the response rate and discuss their representativeness of
the responses and the impact of nonresponses.
For observational studies and experiments, record data about subjects who drop out fromthe studies.
For observational studies and experiments, record data about other performancemeasures that may be affected by the treatment, even if they are not the main focus of
the study.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~42~Analysis
1. Specify any procedures used to control for multiple
testing.
2. Consider using blind analysis.
3. Perform sensitivity analysis.
4. Ensure that the data do not violate the assumptionsof the tests used on them.
5. Apply appropriate quality control procedures toverify your results.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~43~

Presentation of Results
1.
2.
3.
4.
5.
6.

Describe or cite a reference for all statistical procedures used.
Report the statistical package used.
Present quantitative results as well as significance levels. Quantitative results
should show the magnitude of the effects and the confidence limits.
Present the raw data whenever possible. Otherwise, confirm that they areavailable for confidential review by the reviewers and independent auditors.
Provide appropriate descriptive statistics.
Make appropriate use of graphics.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~44~Interpretation of Results
1. Define the population to which inferential statistics andpredictive models apply.
2. Differentiate between statistical significance andpractical importance.
3. Define the type of study.
4. Specify any limitations of the study.

Kitchenham, Pfleeger, Pickard, Jones, Hoaglin, El Emam, and Rosenberg, (2002).

NASSCOM Quality Summit~45~

On-line Surveys
Types
Survey types
Mail surveys
Street surveys
Telephone surveys
Electronic surveys

Descriptive or retrospective survey: state-of-the-art overview. E.g.,
which tools, which reasons, what satisfaction?
Explorative claims: to discover opinions or relationships in new areas.
E.g. evaluate demand for a product or service.

Punter, Ciolkowski, Freimut, and John. (2003).

NASSCOM Quality Summit~46~

On-Line Surveys:
Validity
Compared to experiments and case studies, little control
over variables. Low internal validity.
However, a large number of people makes results easier togeneralize. High external validity.

Punter, Ciolkowski, Freimut, and John. (2003).

NASSCOM Quality Summit~47~

On-line Surveys:
Advantages and Problems
Easy for participants: follow link, fill-in form, simple
adjustments
Easy for researcher: data already electronic, response rateeasy to manage the
Problem: lack of response from individuals who are not
comfortable with technology

Punter, Ciolkowski, Freimut, and John. (2003).

NASSCOM Quality Summit~48~

On-Line Surveys: Process
Activity

PurposeSurvey issues

Study definition

Determine goal

Study design

Turn goals into questions
and select respondents

Questionnaire design, define target
population and sampling procedure,
address validity

Implementation

Make design executable

Check completeness andunderstandability, define
distribution

Execution

Collect and process data

Monitor responses

AnalysisInterpret the data

Verify data entry

Packaging

Report results

Statistics and graphics
Punter, Ciolkowski, Freimut, and John. (2003).

NASSCOM Quality Summit~49~

On-Line Surveys:
Sampling
Sampling: probability sampling or convenience sampling
Sampling: personalized or self-recruited

Punter, Ciolkowski, Freimut, and John. (2003).

NASSCOM Quality Summit~50~

On-Line Surveys:
Development Guidelines
Motivation: why should respondents participate? Access to analysis report
Number of questions: restrict questions to the topic!
Type of questions: closed questions are easy to analyze, open questions with shortanswers provide important background information
Layout: easy to read, minimize scrolling, provide indication of progress
Order of questions: maintain motivation. Interesting questions first. Avoid personalquestions.
Provide specific instructions. Avoid mandatory questions. Provide ways for user toquit and restore.
Ensure anonymity.

Punter, Ciolkowski, Freimut, and John. (2003).

NASSCOM Quality Summit~51~

Case Studies
A powerful and flexible empirical method
Primarily exploratory investigations that attempt to understand and explain aphenomenon or construct a theory
Generally observational or descriptive
A popular way to understand, explain, or demonstrate capabilities of a newtechnique, method, tool, process, technology or organizational structure

Perry, Sim, and Easterbrook (2004).
NASSCOM Quality Summit~52~

Case Studies
Defined method for
Posing research questions
Collecting data
Analyzing the data
Presenting results

NOT
Example or case history
Experience report
Quasi-experimental design with n=1

Perry, Sim, and Easterbrook (2004).
NASSCOM Quality Summit~53~

Validity
External validity: how well do the conclusions apply toother people, in other places, at other times?
Internal validity: are there other possible causes or
explanations?
Construct validity how well do the measurements reflect
the theories, ideas, or models?

O’Brien, Buckley, and Exton (2005).
NASSCOM Quality Summit~54~

Validity:
Did something really happen?
Internal Validity: Did something happen?
History: did anything else happen between first and second measurement?
Maturation: did the object of study change?
Testing: did the first measurement change the second one? (Learning, sensitivity)
Instrumentation: did the measurement instrument or observers change?
Statistical Regression: if groups were selected based on extreme scores, they will tend to move to mean without
treatment
Selection: were the treatment and control groups formed in non-random ways?
Experimental mortality: were dropped cases connected to the study?
Interaction Effects: are there interaction effects that hide the treatment?

External Validity: Can we generalize to similar situations?
Reactive or interaction effects: does the treatment or testing hide real effects (aka Hawthorne effect or placebo effect)
Interaction of selection and treatment: are the effects due to selection?
Reactive effects of experimental arrangement: is the experimental environment so different from real world thatgeneralization is not possible?
Multiple treatment interference: multiple treatments of same cases is not the same.

Campbell and Stanley research design summary, found at http://www.csupomona.edu/~tom/lib/senior_projects/research_design_summary.pdf

NASSCOM Quality Summit~55~

How do you collect data?
What kind of data?
Surveys, questionnaires, interviews, etc.
EPM, Hackystat, PSP/TSP

NASSCOM Quality Summit~56~

EPM: the Empirical Project Monitor
An application supporting empirical software engineering
EPM automatically collects development data accumulated
in development tools through everyday development
activities
Configuration management system: CVS
Issue tracking systems: GNATS
Mailing list managers: Mailman, Majordomo, FML

NASSCOM Quality Summit~57~

Implementation of the Empirical Project Monitor (EPM)
Co-existing tools

Code
Code clone
clone
detection
detection

Component
Component
searchsearch

Product data archive (CVS format)

Source Share GUI

Managers

Plug-in

Metrics
Metrics
measurement
measurement

Logical
Logical Coupling
Coupling

Collaborative
Collaborative
filteringfiltering

Process data archive (XML format)
FormatTranslator

FormatTranslator

FormatTranslator

FormatTranslator

Versioning(CVS)

Mailing(Mailman)

Issue tracking(GNATS)

Other tool
dataCore EPM
Project x
Project y
Project z

Developers
NASSCOM Quality Summit

Co-existing tools

...

~58~

Automated data collection in EPM
Reduces the reporting burden on developers

without additional work for developers
Reduces the project information delay

data available in real time
Avoids mistakes and estimation errors

uses real (quantitative) data

NASSCOM Quality Summit~59~

EPM’s GUI

NASSCOM Quality Summit~60~

An example of output
EPM can put data collected by CVS, Mailman, and GNATS
together into one graph.
Cumulative number of mails
exchanged among developers

Time stamp of programcode check-in to CVSTime stamp of
issue occurrence

NASSCOM Quality Summit

Time stamp of
issue fixing~61~

How do you decide what to measure?

2006/9/12

NASSCOM 2006

GQM Approach
Goal
Questions/Model
Metrics

NASSCOM Quality Summit

~63~

The
The GQM
GQM Paradigm:
Paradigm: An
An Example
ExampleGoal

Question

Analyze CVS and GNATS data for file change patterns, for the
purpose of evaluation, with respect to requirements instability,
poor design, or poor product quality, from the point of view ofthe project manager, in the context of the particular project in
the company

The number oflines changed(LCC)?
Change frequencyof each file (CVS)

Metric
NASSCOM Quality Summit

The number of developers
making changes to files(ONR)?

File change level(FCM)?

The reason forthe change?

Size of file (CVS)

Number of developers
making changes to files(CVS)

Change motive: Bug or specification
change (GNATS)

The changed number oflines in each file (CVS)~64~

The GQM Template
object of study: a process, product or any other experience model
purpose: to characterize (what is it?), evaluate (is it good?), predict (can I
estimate something in the future?), control (can I manipulate events?), improve(can I improve events?)
Focus: what aspects of the object of study are of interest
point of view: from what perspective; who needs this information
context: what is the environment where the measurement will be taken

NASSCOM Quality Summit~65~

The GQM Template
What do we want to measure?

Why do we want to measureit?

What aspect are we interested
in?

Who wants to know?

Where are we going to work?

NASSCOM Quality Summit~66~Team Exercise
Pick one objective or goal
Write the GQM template statement
Develop some questions or model to meet that goal
Design metrics
Describe activities to be measured
Consider how to integrate measurements into activities –what measurement tools could you use?

NASSCOM Quality Summit~67~

CMMI Levels
1.
2.
3.
4.
5.

Initial: Competent people and heroics
Repeatable: basic project management
Defined: process standardization
Managed: quantitative management
Optimizing: continuous process improvement

NASSCOM Quality Summit~68~

CMMI Levels
1
0.9
0.8
0.7

InitialRepeatable
DefinedManagedOptimizing

0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

CMMI Levels~69~Data Collections
1
0.9
0.8
Individual
Team
Project
Department
Company
Consortium
Industry

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Levels~70~

Break!

2006/9/12

NASSCOM 2006

Analyzing Empirical Data

2006/9/12

NASSCOM 2006

Analysis targets
Collection, analysis, and search engine ofsoftware products (SPARS)Manager
supportEstimation ofrequirements for reredesign caused byconnections in thesystem structure

Making similarity
visibleClass and utility
evaluation of information
related to use of
methods

Code clone analysis
Collaborative filtering
Logical coupling
The project delay risk
detection model

The project management
models for trouble
evaluation
Estimate of effort tocorrect based on filescale, number ofaccumulated defects,
and defect typeEvaluation ofclone distribution
Developer
supportEvaluation ofcomponent ranking
ranking
Expert identification
Grasp of the situation

NASSCOM Quality Summit

GQM(Goal/Question/Metric)Model

Distinction ofmodules (file) withhigh defect rates

Abnormal detection~73~

ID candidates forrefactoring

Forecast

AdviceAnalyzing Empirical Data
Analysis of potential project delay risks
Logical coupling
Defect correction analysis
Collaborative filtering
Extracting patterns through association rule mining
Analysis in your hands?

NASSCOM Quality Summit~74~

Risk Detection Preventing Project Delay
Detect risks of project delay ( e.g. Unstable Requirements,
Incomplete Designs, Low Quality Program or Inappropriate
Resource Planning ) by monitoring the program change history

NASSCOM Quality Summit~75~

Risk Detection Preventing Project Delay(2)Give the alert for project managers depending on a certain threshold
E.g. change frequency weekly for each module

A: Number of updates weekly

Remarkable changes

y = 0.2763x + 0.6401
R2 = 0.5492

Approximately 30% of changes
include a certain amount ofline deletion normally ->“more than 30%” implies
that some problems happened

B

A

NASSCOM Quality Summit~76~

Logical Coupling Analysis
Discover implicit knowledge for system maintenance to reduce mistakes or lack ofneeded changes

Relationships among files (modules) frequently changed at the sametime

NASSCOM Quality Summit~77~Complication analysis by logical coupling
Logical Coupling

A logical relationship between software modules where changes in a part of one module requirechanges in a part of another module.
By knowing logical coupling, the cause of a module’s complexity becomes clear, providing hints about module
structure and where restructuring may be needed.

File
Bug correction andfunction change

NASSCOM Quality Summit~78~

Change relation
simultaneously

Logical Coupling Analysis(1)Analyze change history from Configuration Management System using
association rule mining

Configuration
Management
SystemMake
Transaction
Data*

Transaction
Data

File
ChangePatterns

Association
Rule
Mining**Trans.
NumberDate of Update
AuthorFile1 File2 File3 File4 File5 File6 …
No.
of Files
1 2000/2/26 9:28:42 tomoko
21 FileA FileB FileC FileD FileE FileF
2 2000/2/27 9:42:12 tomoko
263 FileG FileH FileB FileI
FileJ FileK
3 2000/2/27 11:03:35 noriko
4 FileK FileN FileM FileD
Count ofof FileP FileQ FileR Number
4 2000/2/28 1:30:28 kohei
5 FileF No.
FileOchanged Support
File 1of files
5 2000/2/28 1:36:06 noriko
2 FileP Patterns
FileQ
together
6 2000/2/28 1:37:14 tomoko
2 FileF FileS1
9 0.00438
4 FileL
7 2000/2/28 1:38:07 kohei
1 FileT2
9 0.00438
4 FileF…
3
9 0.00438
4 FileC*Thomas Zimmermann, Peter4
9 0.00438
4 FileC
Weißgerber: ”Preprocessing CVS Data for Fine5
9 0.00438
4 FileA
Grained Analysis”, Proc. International Workshop
6
10 0.00486
3 FileNon Mining Software Repositories (MSR), Edinburgh,
7
9 0.00438
3 FileM
Scotland, UK, May 2004
・・・・・・・

File 2File 3File 4

FileM
FileG
FileJ
FileD
FileB
FileO
FileN

FileN
FileH
FileE
FileE
FileC
FileP
FileP

FileP
FileI
FileK
FileK
FileK

**Apriori Algorithm：R. Agrawal, R. Srikant: ”Fast algorithm for mining association rules”, Proc. 20th Very Large Data
NASSCOM
Quality Summit pp.487-499. Morgan Kaufmann, 1994
~79~
Bases
Conference(VLDB),

Logical Coupling Analysis(2)Analyze each pattern by confidence

Association Rule：X (Antecedent) ⇒ Y (Consequent)Confidence (X⇒Y) ： # of Update (X∪Y) / # of Update (X)Confidence# of Update(Antecedent)

0.8
1
1
1

# of Update
Antecedent(Consequent)
FileA.cpp
54 FileB.h
FileC.cpp
FileA.cpp
44 FileB.h
FileD.h
FileA.cpp
44 FileC.cpp
FileD.h
FileB.h44 FileC.cpp
FileD.hConfidence

Consequent-> FileD.h-> FileC.cpp-> FileB.h

Was “FileY.cpp” copied from
“FileX.cpp”?
Or does “FileY.cpp” strongly
depend on the “FileX.cpp”?

-> FileA.cpp# of Update(Antecedent)

0.5
1
NASSCOM Quality Summit

Has “FileD.h” misseda change?

~80~# of Update
Antecedent
Consequent(Consequent)8
4 FileX.cpp-> FileY.cpp
4
4 FileY.cpp-> FileX.cpp

Factor Analysis of Defect Correction Effort
Improve software process taking into account return on investment

NASSCOM Quality Summit~81~Factor Analysis of Defect Correction Effort(1)E.g.) Defect Correction Effort classified by “Reason why the defect was not detected in thepreferable phase”

Log (Defect Correction Effort)

Lack of Test Cases

These factors seem toinfluence the correction effort->Do any relations to anotherfactors ( e.g. detected phase )exist?

Defects dependingon the environment

Misjudged test results

NASSCOM Quality Summit~82~Factor Analysis of Defect Correction Effort(2)Log (Defect Correction Effort)E.g.) Relation Analysis between “Reason why the defect was notdetected in the preferable phase” and “Defect detected phase” using
Parallel Coordinate Plot

Integration Test
Lack of Test Cases

Unit Test

Misjudged
test results
Defects dependingon the environment

System Test
Detected Phase
NASSCOM Quality Summit~83~

Reasons

Estimation Using Collaborative Filtering: Robust for
Missing Values

2006/9/12

NASSCOM 2006

Collaborative Filtering
Robust estimation method with missing data
Applicable to estimating various attributes of project/system from similar project/system profiles

FocusedOutcome
RepresenQ&M
Collaborative
AdoptedtativeResources

App. A

9

9

9

7

7.5
(target)App. B8

7

8?
(missing)

8App. C?
(missing)

8

8

8

7

App. D

7

6?
(missing)

9

6

NASSCOM Quality Summit~85~

BackgroundFor development planning, practitioners can use estimation methods with historicalprojects’ data, but…

Historical data usually contain many Missing Values.
Briand, L., Basili, V., and Thomas, W. “A Pattern Recognition Approach for Software Engineering
Data Analysis,” IEEE Trans. on Software Eng., vol.18, no.11, pp.931-942 (1992)

Missing values reduce the accuracy of estimation.
Kromrey, J., and Hines, C., “Nonrandomly Missing Data in Multiple Regression: An Empirical
Comparison of Common Missing-Data Treatments,” Educational and Psychological Measurement,
vo.54, no.3, pp.573-593 (1994)

Horse?
Cow?
NASSCOM Quality Summit

~86~

?

Goal and Approach
Goal: to establish an estimation method which treats historical projects’ datacontaining many missing values.
Approach: use Collaborative Filtering (CF).

A technique for estimating users’ favorite items with data containing
many missing values (e.g. Amazon.com)

NASSCOM Quality Summit~87~Estimation of Customer X’s
favorite (rating on book 5).
Step 1: Similarity Computation
Calculating the similarity between Customer X (current estimation target)and other customers.
Selecting top-k similar customers (e.g. k = 2).
Step 2: Prediction
Calculating a prediction of customer X’s rating on book5 using the similarcustomers’ rating on book 5.

Customer X
Similarity: Customer+1.00
A
Similarity: Customer+0.97
B
Similarity:Customer-0.97
C
NASSCOM Quality SummitBook 1

Book 2

Book 3

Book 41 (hate)1 (hate)2 (unlike)2 (unlike)2 (unlike)4 (like)? (unread)? (unread)4 (like)

4.51
5 (love) Prediction:? (unread)5 (love)
5 (love)
5 (love)
4 (like)
1 (hate)1 (hate)~88~? (unread)4 (like)
2 (unlike)

Book 5

Estimation of Project X’s
Development Cost (required man-month)

Step 1: Similarity Computation
Calculating the similarity between Project X (current estimation target) andother past projects.
Selecting top-k similar projects (e.g. k = 2).
Step 2: Prediction
Calculating a prediction of project X’s cost using the similar projects’ cost.

Language Dev. Type
Project X
Similarity: +1.00
Project A
Similarity: +1.00
Project B
Similarity: 0.00
Project C
NASSCOM Quality Summit

Java
Java
Java

New
New
? (MV)? (MV)

Maintenance~89~

Function
Points2000? (MV)

2500
5000# of Staffs Dev. Cost
7
8
10
20

Prediction:? (MV)59.99

50
70
250

Experimental Evaluation of Estimation Accuracy[1]

Methodology: Cross-Validation of Cost Estimation
Data Size: 10 variables and about 140 projects
Missing Value Ratio: from 0 to 50% (by 10%)
Relative Error
6
5

4

4

3

3

2

2

[1] Kakimoto, T., Tsunoda, M., Ohsugi, N., Monden, A., Matsumoto, K., “Experimental Evaluation
1
for Robustness of CF-Based Effort Estimation,”
In Proc. of FOSE2004, pp.73-84 (2004) (Japanese)

10

MV Ratio (%)

5

Stepwise Multiple Regression

MV Ratio (%)

Collaborative Filtering

Relative Error
6

0

10

NASSCOM Quality Summit

20

30

40

50

~90~

0

0

10

20

30

40

50

Association rule mining
for software project data

2006/9/12

NASSCOM 2006

Association rule mining- Motivation A retail sales analysis example:“if A is bought from 12:00~14:00 then B is bought in 80% of the cases,
too.” is obtained from POS histories.
From 12:00~14:00, retailer can locate goods B next to goods A.
A software project analysis example would be…
From project development data,“if the development type is enhancement and code developers areoutsourced, then the testing phase is longer in 70% of the cases.” isobtained.
When doing enhancement development and coding phase is outsourced,
the testing phase should be estimated longer than usual.

NASSCOM Quality Summit~92~Input: project dataset
Project dataset (project descriptive data) has
rows correspond to individual project
columns correspond to data field
EffortCodingRatio

Development
Type

Organization Type

Resource
Level

0001

New
Development

Banking

A

80

230

200 ・・・

0002

Enhancement

Construction

B

120

200

360 ・・・

0003

Enhancement

PublicAdministration

B

60

260

400 ・・・

・・・

・・・

・・・

・・・

NASSCOM Quality Summit~93~

EffortSpecify
Ratio

・・・

・・・

Effort
Test
Ratio

・・・

ID

・・・

・・・

Output: extracted rules
Extracted rule is in the form “if A then B”.
A, B: categorical valueex) if (development type = enhancement) and(effort specify ratio = small) then (effort test = large)

Rules can be unified using “or”.
ex) if (organizational type = banking) then (effort test ratio= medium or large)

NASSCOM Quality Summit~94~

NEEDLE: Case Study

Extracted rules from 37 real system integration projects datawith 40 data fields.
E.g. if known customer and known industry / business andwithout middleware for specific industry / business then ratio ofstaff-month in design phase is 1 or 2 or 3. (1: lowest, 9: highest).
37.8%(support)

known: having experience to develop with
NASSCOM Quality Summit~95~Current EASE Research Topics
Developer Role Categorization
Correspondence Analysis
Personal EPM

NASSCOM Quality Summit~96~Analysis in your hands
In teams
Select a process or project that you could collect data from (oralready do)
How do you analyze this data? Which of the following terms help:
CompareContrast
Abstract
Model
Hypothesis
Context
Relationships

NASSCOM Quality Summit~97~

Which area needs analysis most?
Requirements elicitation
DesignDevelopment
Integration
Testing
Operations
Maintenance

NASSCOM Quality Summit~98~Which area most needs analysis?
1
0.9
0.8
Requirements
DesignDevelopment
Integration
Testing
Operations
Maintenance

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Lifecycle~99~

Which Analysis Method Is Most Useful for You?
Potential Delay Risks
Logical coupling
Defect correction
Collaborative filtering
Association Rule Mining

NASSCOM Quality Summit~100~Which Analysis Method is Most Useful for You?
1
0.9
Potential Delay Risks

0.8
0.7

Logical coupling

0.6

Defect correction

0.5
0.4

Collaborative filtering

0.3
0.2

Association Rule
Mining

0.1
0
NASSCOM Quality Summit

Methods~101~

Break!

2006/9/12

NASSCOM 2006

Distributing the Results

2006/9/12

NASSCOM 2006

Distributing the Results
Feedback through reports, training, simulations, plans, etc.
One approach: the Experience Factory
Replayer/Simulator
Workshops, tutorials, training

NASSCOM Quality Summit~104~

Simulations

2006/9/12

NASSCOM 2006

SIMSE
Most documented software failures can be attributed tosoftware engineering process breakdowns
The root cause of this is lack of practice with issues
surrounding the software engineering process

Navarro and van der Hoek (2004).

NASSCOM Quality Summit~106~

SIMSE
Components
Teams of people
Large-scale projects
Critical decision-making
Personnel issues
Multiple stakeholders
Budgets
Planning
Random, unexpected events

Navarro and van der Hoek (2004).

NASSCOM Quality Summit~107~

SIMSE
Tradeoffs: faithfulness to reality, level of detail, usability, teaching
objectives, and fun factors
Guidelines
SimSE should illustrate both specific lessons and overarchingpractices of the software process
SimSE should support the instructor in specifying the lessons he or she
wishes to teach
SimSE should provide a student with clear feedback concerning theirdecisions
SimSE should be easy to learn, enjoyable, and comparatively quick

Navarro and van der Hoek (2004).

NASSCOM Quality Summit~108~

SIMSEWhy is simulation successful?
Simulation allow students to gain variable hands-on experience of theprocess being simulated without monetary costs or harmful effects ofreal world experience
Simulations can be repeated, allowing experimentation with different
approaches
Relative ease of configuration allows educator to introduce a wide
variety of unknown situations
Simulation can be run at faster pace, allowing students to practiceprocess many more times than feasible in real world

Navarro and van der Hoek (2004).

NASSCOM Quality Summit~109~

Simulation Games
Problems and programmers, an educational software engineering
card gameTwo-person game
Balance budget and reliability

Concept, programmer, and problem cards
Concept: decision regarding approach
Programmer: skill, personality, salary
Problem: various project problems

Pick a project card to start game: complexity, length, quality, budget

Navarro, Baker, and van der Hoek (2004).

NASSCOM Quality Summit~110~Models in Software Engineering
Models have three properties:
Mapping: there is an original object or phenomenon
represented by the model
Reduction: the model does not represent all properties, butdoes represent somePragmatic: for some purposes, the model can replace theoriginal, so it is useful.

Mapping: mapped, removed, and added attributes

Ludewig (2003).

NASSCOM Quality Summit~111~Models in Software Engineering
Descriptive: mirrors existing originalPrescriptive: used to create original
Transient: descriptive, but then modified to guide changes
to original

Ludewig (2003).

NASSCOM Quality Summit~112~Models in Software Engineering
Purpose:Documentation
Concise descriptions
Minutes, protocols, logs
Metrics
Instructions
Exploratory models
Educational models and games
Formal or mathematical models

Ludewig (2003).
NASSCOM Quality Summit~113~

Simulation Based Project Management Training
Why a simulator? Knowledge and skills. Simulator provides experience in manydifferent scenarios without high risk and expense. Provides hands-on, active
experience to build effective skills.
Areas which can benefit from simulation:
Cost assessment
Practicing metric collection
Building consensus and communication skills
Requirements management
Project management
Training
Process improvement
Risk management
Acquisition management

Collofello (2000).
NASSCOM Quality Summit~114~

Simulation Based Project Management Training
The simulator allows various user inputs, both initially and during a run, such as:
Planned completion time
Project complexity
Time allocations
Communication overhead
Staffing, including various levels of expertise

Various output monitors, such as
Current staff load
Elapsed man hours and days
Remaining hours
Schedule pressure gauge
Exhaustion rate gauge
Percent completion
Earned Value outputs

Collofello (2000).

NASSCOM Quality Summit~115~

Simulation Based Project Management Training
Kinds of exercisesLife Cycle model comparison: simulate using waterfall and sequential
incremental development, with varying inspection effectiveness, projectcomplexity, and staffing levels
Risk management, with various contingency plans and personnel losses
Software inspections: effect of varying inspections on test time and projectcompletion
Critical path scheduling: vary the skill levels of staff on the critical path andassess impact
Overall planning and tracking: Plan and execute a project from inception tocompletion, adjusting to events.

Collofello (2000).
NASSCOM Quality Summit~116~Simulation in SE Training
SESAM: Software Engineering Simulation by AnimatedModels
Focus on motivation: it is hard for students to imagine
project management failure because the projects theynormally experience are small. SESAM provides anopportunity to experience large project without real risk or
expense.

Drappa and Ludewig (2000).

NASSCOM Quality Summit~117~

Simulation in SE Training
During game, restricted information like real project
manager receives.
After game, significant extra information available.

Drappa and Ludewig (2000).

NASSCOM Quality Summit~118~

Simulation in SE Training
Observations
Students do not like planning.
Students make the same mistakes again and again.
Students do not reflect on why they fail.

Drappa and Ludewig (2000).

NASSCOM Quality Summit~119~

Project Replayer

An Investigation Tool to Revisit Processes of Past
Project
Keita Goto*, Noriko Hanakawa**, and Hajimu Iida** Graduate School of Information Science, NAIST, Japan** Faculty of Management Information, Hannan Univ., Japan

2006/9/12

NASSCOM 2006
http://sdlab.naist.jp/

A Problem in Recent Software Development (1/2)

It is difficult to accumulate knowledge andexperiences in organizations.
BecauseSoftware life cycle is getting shorter (e.g. software ofmobile phone ).
Team members frequently change.

NASSCOM Quality Summit~121~

A Problem in Recent Software Development (2/2)
The same mistakes are repeated in an organization.
Past project

New project

A mistake was made.

The same kind of themistake was made again.
One year after

By a developer

By a new developer

Knowledge
Developer learned- Why this mistake
happened.
- How to avoid such
mistakes.
NASSCOM Quality Summit

The knowledge was
not transferred.

~122~-Short development periods- High fluidity of developers

Goal and Approach
Goal
Establishing knowledge feedback cycle from past projects to
new projects.

Approach
We propose a Knowledge Feedback Cycle (KFC) framework.
The following 3 tools play very important roles in the KFC
framework.
Empirical Project Monitor (EPM) : automatically collects software
project data. (EPM was made by EASE project Japan)
Project Replayer : replays history of a past project for researchers
to help extracting knowledge.
Project Simulator: reuses the extracted knowledge to provide
estimations for new projects.

NASSCOM Quality Summit~123~Outline- A Problem in Recent Software Development- Goal and Approach- Knowledge Feedback Cycle (KFC)- Project Replayer (one of 3 key tools in the KFC)
Features of Project Replayer
Preliminary experiment
Results and discussion- Conclusions and Future WorkNASSCOM Quality Summit~124~Knowledge Feedback Cycle (KFC) Overview

Developer

Researcher

Project
Execution

Project
Analysis

Project Data

Tacit knowledge

NASSCOM Quality Summit~125~Knowledge Feedback Cycle (KFC) Overview

Developer

Researcher

Project
Execution
Code management log
Bug tracking log
E-mailing log

Project
Analysis

EPM(Empirical Project
Monitor)Tacit knowledge

NASSCOM Quality Summit~126~

Project Data
SummaryKnowledge Feedback Cycle (KFC) Overview

Training/Planning

Project
Simulator

Developer

Explicit knowledge
Simulation
Model

Researcher

Project
Execution
Code management log
Bug tracking log
E-mailing log

EPM(Empirical Project
Monitor)Tacit knowledge

NASSCOM Quality Summit~127~

Project Data
Summary

Project
Analysis&
Building
Simulation
ModelKnowledge Feedback Cycle (KFC) Overview

Training/Planning

Developer

Explicit knowledge

Project
Simulator

Project
Replayer

Simulation
Model

Dynamic
View

Researcher

Project
Execution
Code management log
Bug tracking log
E-mailing log

EPM(Empirical Project
Monitor)

Static View

Tacit knowledge

NASSCOM Quality Summit

Project Data
Summary~128~

Project
Analysis&
Building
Simulation
ModelProject Replayer

Postmortem
Evaluations
Developer

Project
ReplayerSupport two roles in KFC

Project behavior& detail
Researcher

Project Data

Developers can revisit their past projects for postmortem
evaluations
Researchers can deeply understand and analyze dynamic
behavior of the projects

NASSCOM Quality Summit~129~

Showing Project data in a Dynamic Way
Project Replayer reorganizes collected data by EPM, sorted in time order.

Time series

EPM

CVS logs

Delete a fileGnats logs

Report a bug

Mailman logs

Send a mail

NASSCOM Quality Summit

Commit a file
Report a bug
Send a mail~130~

Send a mail

Showing Project data in a Dynamic Way
Project Replayer reorganizes collected data by EPM, sorted in time order.

EventsTime series

EPM

CVS logs

Delete a fileGnats logs

Report a bug

Mailman logs

Send a mail

NASSCOM Quality Summit

Commit a file
Report a bug
Send a mail~131~

Send a mail

Showing Project data in a Dynamic Way
Project Replayer shows events graphically and simultaneously.

Delete a file
Report a bug
Send a mailPlay position
Time series

EPM

CVS logs

Delete a fileGnats logs

Report a bug

Mailman logs

Send a mail

NASSCOM Quality Summit

Commit a file
Report a bug
Send a mail~132~

Send a mail

Showing Project data in a Dynamic Way
Project Replayer shows events graphically and simultaneously.

Report a bug
Send a mailPlay position
Time series

EPM

CVS logs

Delete a fileGnats logs

Report a bug

Mailman logs

Send a mail

NASSCOM Quality Summit

Commit a file
Report a bug
Send a mail~133~

Send a mail

Views provided by Replayer
Event List View shows all events
listed in time order.

Graph View shows graphof project progress data.

File View shows
current status of eachsource file in project

Developer View shows
current status of eachmember in projectTime control
slider
NASSCOM Quality Summit~134~Preliminary Experiment with Replayer in KFC

Aims of the experiment1.Confirming benefits of developers and researchers with replaying.
2.Confirming whether tacit knowledge can be transformed to explicit
knowledge.
Explicit knowledge
Simulation/
Estimation
Model

Project
Execution
Developer

Project
Replayer

Code management log
Bug tracking log
E-mailing log

Tacit knowledge
NASSCOM Quality Summit~135~

Researcher

Project
Analysis&
Building
Hypothesis

Setting of preliminary experiment

Target project
Application domain : Typing game
Developers : 6 graduate school students
Development period : 24 days
Program code : 9,578 lines in C++

Subjects of experiment
3 subjects as developers (who were project members)
1 subject as researcher (was not a project member)

NASSCOM Quality Summit~136~Procedure of the experiment

The experiment was carried out after the target project was finished. Project data wasautomatically collected using EPM.
Phase1 : Making hypotheses and questions
The researcher analyzes the project using Project Replayer. The researcher makes
hypotheses and questions.

Phase2 : Answering the questions
The developers find the answers for the questions using Project Replayer.

Phase3 : Modifying the hypotheses
The researcher modifies the hypotheses in Phase1 according to the answers.
If required, the researcher performs additional analysis using Project Replayer.

Phase1
H1 Q1
The researcher
NASSCOM Quality Summit

Phase2Phase3
A1

The developers
~137~

H1’
The researcher

Results - In Phase1 (1/2)

At first, the researcher made hypotheses using Project Replayer(H1) “Modules developed at the final stage of the project have low quality.”(H2) “If CVS’s event behavior does not match to bug reports and e-mail data, theproject is in confusion, and resulting software has low quality.”

H1 :Developers did not start making somemodules until a few days before deadline.
The researcher expected that these modules
have low quality. Therefore, the researcher
established H1.

H2

H2 :For the last three days, total lines of code did
not change. That is, the development has
somehow stopped before the deadline.
However, the bug reports and e-mail events were
occurring during this period. It looked likeinconsistency.
Therefore, the researcher expected that the
project fallen into confusion and the inconsistency
was produced. So the researcher established H2.

H1
NASSCOM Quality Summit~138~

Results - In Phase1 (2/2)
The researcher made some questions about the development data.
(Q3) “How was the quality of the module made at the final stage of a project?”(This question related hypothesis, H1)(Q4) “Why was not CVS renewed during the last three days?” (This question
related hypothesis, H2)

We only show the question directly related to 2 hypothesis.

Q4

Q3
NASSCOM Quality Summit~139~

Results - In Phase2
The developers answered to the questions.
(Q3) “How was the quality of the module made at the final stage of a project.”(A3) “Most of them have good quality except one module.”(Q4) “Why was not CVS renewed during the last three days?”(A4) “Because the last three days were maintenance phase.”Those answers were derived with help of Project Replayer.

A3 :File view helped to know the name offiles developed in the final stage of the
project
A4

A3
NASSCOM Quality Summit

A4 :Event list view helped developers torecall that the last three day weremaintenance stage after the actual dead
line.
~140~

Results - In Phase3 (1/2)

H1 was not clearly supported by the developers’ answers(H1) “Modules developed at the end stage of the project have low
quality.”(A3) “Most of them have good quality except one module.”

H2 was just withdrawn according to the answers.
Additional analysis was performed using Project
Replayer to refine H1.

NASSCOM Quality Summit~141~

Results - In Phase3 (2/2)
H1 was changed to the following(H1’) “Modules developed at the final stage of the project have
low quality if the developers have little experience ofdeveloping similar functions”

This fact was
found by seeing
Developer view
and File view.

Inexperienced
developer was
assigned to thelow quality
modules

H1’

H1
NASSCOM Quality Summit~142~

Final results of Preliminary Experiment
Project Replayer was useful for researchers.
To initially build hypothesis.
To refine hypothesis.

Project Replayer was also useful for developers.
To recall detail of past project in answering questions.

H1’ may be regarded as an explicit knowledge derived from the past
project.

NASSCOM Quality Summit~143~

Conclusions and Future work
KFC framework was proposed
Prototype of Project Replayer for KFC was developed
Preliminary experiment with Project Replayer was carried out
Project Replayer was useful for both developers and researchers

Future work
Further evaluation and validation of Project Replayer
Development of Project Simulator (another tool in KFC)

NASSCOM Quality Summit~144~Diffusion of Innovations
Relative Advantage: How much benefit do we get?
Compatibility: How well does it fit with what we alreadydo?
Complexity: How hard is it to do?
Observability: Will anyone notice?
Trailability: Can we try it out without risk?

NASSCOM Quality Summit~145~

Acceptance of New Methods
1. Usefulness: will it help me?
2. Voluntariness: do I have a choice?
3. Compatibility: how well does it fit with values, needs, and pastexperiences?
4. Subjective norm: do important others think I should use this?
5. Ease of use (or complexity): how hard is it to use?
6. Relative advantage: how much better is it?
7. Result demonstrability: can it show real advantages?
8. Image: Does it make me look better?
9. Visibility: can others see what I’m doing?
10. Career Consequences: what is the long-term payoff for using this?

Riemenschneider, Hardgrave, and Davis (2002).

NASSCOM Quality Summit~146~

An Exercise
In teams
Estimate the effectiveness of reports, simulations, training
in your organization
Estimate the effort needed (project size) to create reports,
simulations, training in your organization

NASSCOM Quality Summit~147~

Which do you think would be most effective in yourorganization?
Reports
Simulations
Training
Policy Directive
Which do you actually use?

NASSCOM Quality Summit~148~

Which is most effective?
1
0.9
0.8
0.7
Reports
Simulations
Training
Policy

0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Approach~149~

Which is most effective?
1
0.9
0.8
0.7
Reports
Simulations
Training
Policy

0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Use
~150~

Which factor is most important for you?
Relative Advantage: How much benefit do we get?
Compatibility: How well does it fit with what we alreadydo?
Complexity: How hard is it to do?
Observability: Will anyone notice?
Trailability: Can we try it out without risk?

NASSCOM Quality Summit~151~

Which Is Most Important for You?
1
0.9
0.8
0.7
Relative Advantage
Compatibility
Complexity
Observability
Trailability

0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Factors~152~

Acceptance of New Methods
1. Usefulness: will it help me?
2. Voluntariness: do I have a choice?
3. Compatibility: how well does it fit with values, needs, and pastexperiences?
4. Subjective norm: do important others think I should use this?
5. Ease of use (or complexity): how hard is it to use?
6. Relative advantage: how much better is it?
7. Result demonstrability: can it show real advantages?
8. Image: Does it make me look better?
9. Visibility: can others see what I’m doing?
10. Career Consequences: what is the long-term payoff for using this?

Riemenschneider, Hardgrave, and Davis (2002).

NASSCOM Quality Summit~153~

Acceptance of New Methods
1
0.9

usefulness
voluntarinesscompatibility
subjective norm
ease-of-use
relative advantage
result demonstrability
image
visibility
career consequences

0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Factor~154~

Break!

2006/9/12

NASSCOM 2006

Using Empirical Methods for quality
Improvement

2006/9/12

NASSCOM 2006

Putting It All Together
First, an example of an empirical tool called CCFinder that
we have developed for gathering, analyzing and providing
results
Second, we will consider how you can use empiricalmethods in your quality improvement

NASSCOM Quality Summit~157~

Code Clone Analysis and Application
Katsuro Inoue
Osaka University

2006/9/12

NASSCOM 2006

Talk Structure
Clone Detection
CCFinder and Associate Tools
Applications
Summary of Code Clone Analysis and Application

NASSCOM Quality Summit~159~

Clone Detection

2006/9/12

NASSCOM 2006

What is A Code Clone?
A code fragment that has
identical or similar code fragments
code clone
in source code
copy-and-paste
Code clones are introduced
in source code for various reasonscode reuse by `copy-and-paste’
stereotyped function
ex. file open, DB connect, …
intentional iteration
performance enhancement
It makes software maintenance more difficult
If we modify a code clone with many similar code fragments, it isnecessary to consider whether or not we have to modify each of them.
We are likely to overlook some!

NASSCOM Quality Summit~161~Simple Example

AFG::AFG(JaObject* obj) {objname = “afg";
object = obj;}
AFG::~AFG() {for(unsigned int i = 0; i < children.size(); i++)if(children[i] != NULL)
delete children[i];...
for(unsigned int i = 0;i < nodes.size(); i++)if(nodes[i] != NULL)
delete nodes[i];}NASSCOM Quality Summit~162~

Definition of Code Clone

No single or generic definition of code clone
So far, several methods of code clone detection have been proposed,and each of them has its own definition of a code clone

Various detection methods
1.
2.
3.
4.
5.

Line-based comparison
AST (Abstract Syntax Tree) based comparison
PDG (Program Dependency Graph) based comparison
Metrics comparison
Token-based comparison

NASSCOM Quality Summit~163~

CCFinder and Associate Tools

2006/9/12

NASSCOM 2006

Clone Pair and Clone Set
Clone Paira pair of identical or similar code fragments

Clone Set
a set of identical or similar fragments

C1
C2
C3

C4
C5

Clone Pair

Clone Set

(C1, C2){C1, C2, C4}(C1, C4){C3, C5}(C2, C4)(C3, C5)NASSCOM Quality Summit~165~

Our Code Clone Research
Develop tools
Detection tool: CCFinder
Visualization tool: Gemini
Refactoring support tool: Aries
Change support tool: Libra

Deliver our tools to domestic or overseas organizations/individuals
More than 100 companies use our tools!

Promote academic-industrial collaboration
Organize code clone seminars
Manage mailing-lists

NASSCOM Quality Summit~166~

Detection
Detection tool:
tool:

Development of CCFinder
Developed by industry requirement
Maintenance of a huge system
More than 10M LOC, more than 20 years old
Maintenance of code clones by hand had been performed, but ...

Token-base clone detection tool CCFinder
Normalization of name space
Parameterization of user-defined names
Removal of table initialization
Identification of module delimiter
Suffix-tree algorithm

CCFinder can analyze a system in the scale of millions of lines in 530 min.

NASSCOM Quality Summit~167~

Detection
Detection tool:
tool:

CCFinder Detection Process
Source files

1.1.static
throwsstatic( ) ) throws
throws{ String a {{staticvoid
throwsRESyntaxException
static void
$void
$foo foo()( foo()$RESyntaxException{RESyntaxException$ $Lexical
Lexicalanalysis
analysis
Lexicalanalysis]] = a[]new
String[] String$a[]$=[new{] {$ $[]
2.2. [[String
String=new
String[]{{"123,400","123,400","abc","abc","orange"orange100"100"};};} ;3.3. org.apache.regexp.RE
patorg.apache.regexp.RE
pat==newneworg.apache.regexp.RE("[0-9,]+");
org.apache.regexp.RE("[0-9,]+");
RE
pat
$ sum
$ ==new
Token
Tokensequence
sequence
Tokensequence
4.4. int
int
sum
=0;0;
RE
) ; int$ ( "[0-9,]+"$$ sum$ == $0
5.5. forfor(int(intii==0;0;ii<<a.length;a.length;++i)++i);; forfor ( int
$ $i = 0$ ; $i <6.6. a if
if(pat.match(a[i]))(pat.match(a[i]))Transformation
Transformation
Transformation; ++$ i ) )if if( ($ pat$ .. length$ ; ++
7.7. .. match
sum
sum+=
+=Sample.parseNumber(pat.getParen(0));
Sample.parseNumber(pat.getParen(0));$ ( ( $ a [ [ $ i ] ] ) ) ) ) $ sum
8.8. +=""++sum);
System.out.println("sumsum);
Sample( 0+=System.out.println("sum$ . $. parseNumber( $ . $ ( ( pat$ .==getParen
Transformed
Transformedtoken
tokensequence
sequence
Transformed
token
sequence
9.9.}})) )) ; System$ . $. out. $. println( $ ( "sum = "++staticsum
static void
( $throws
String$ void))void;; goo(String}}goo(String
static$ $goo10.
[][]( a)a)10.static
throwsRESyntaxException
RESyntaxException{{[[ exp]exp) =throws
RESyntaxException$RE("[0-9,]+");{ $ $ = { RE exp =Match
Matchdetection
detectionMatchdetection
11.
11. a$ RE
RE
=new
new
RE("[0-9,]+");
newRE("[0-9,]+"
)
;
int
sum
=
0
new
$
$
)
;
$
$
=
$
12.
12. int
intsumsum==0;0;;; for= 0$ ; $i <for ( int
13.
ii<<a.length;
13. forfor(int(int$ ii=$i=0;0;
a.length;++i)++i)ClonesClonesonontransformed
transformedsequence
sequence
Clonesontransformed
sequence$ ; ++
a$ .. length; ++$ i ) )if if( ($ exp
14.
ifif(exp.match(a[i]))14. .. match(exp.match(a[i]))$ ( ( $ a [ [ $ i ] ] ) ) ) ) $ sum
15.
sum
+=
15.+=
sum
+=parseNumber(exp.getParen(0));
parseNumber(exp.getParen(0));+= $$ . parseNumber$ ( $ . ( $exp( . $ getParen) ) ( 0 ) )Formatting
Formatting
Formatting
16.
=
sum);
16. ;; System.out.println("sumSystem.out.println("sumsum);$ . $. out. $. println( $ (+ "sum$ =="""+++sum
System17.
17.})}) ;; }Clone pairs

NASSCOM Quality Summit~168~

Visualization
Visualization Tool:
Tool:

Gemini Outline

Visualize code clones
detected by CCFinder
CCFinder outputs thedetection result to a text file

Provide interactive analysesof code clones
Scatter Plot
Clone metrics
File metrics

Filter out unimportant code
clones

NASSCOM Quality Summit~169~

Visualization
Visualization tool:
tool:

Gemini Scatter Plot
D1Visually show where code clones
areBoth the vertical and horizontal axes
represent the token sequence ofsource code

F2

F3

F4

a b c a b c c c a b d e f a b c c d e f

F2

Dot means corresponding two
tokens on the two axes are the same

F3

Symmetric to main
diagonal (show only lower
left)
D2

F4

a b c a b c c c a b d e f a b c c d e f

F1
D1

The original point is theupper left corner

F1

D2

F1, F2, F3, F4 : files
D1, D2 : directories: matched position detected as a practical code clone: matched position detected as a non-interesting code clone
NASSCOM Quality Summit~170~

Visualization
Visualization tool:
tool:

Gemini Clone and File Metrics

Metrics are used to quantitatively characterize entities
Clone metrics
LEN(S): the average length of code fragments (the number of tokens) inclone set S
POP(S): the number of code fragments in S
NIF(S): the number of source files including any fragments of S
RNR(S): the ratio of non-repeated code sequence in S
File metrics
ROC(F): the ratio of duplication of file F
if completely duplicated, the value is 1.0
if not duplicated at all, the value is 0,0
NOC(F): the number of code fragments of any clone set in file F
NOF(F): the number of files sharing any code clones with file F

NASSCOM Quality Summit~171~

Refactoring Support System: Aries (1)Structural code clones are regarded as a target for refactoring
1.
2.
3.

Detect clone pairs by CCFinder
Transform the detected clone pairs into clone sets
Extract structural parts as structural code clones from the detectedclone sets

What is a structural code clone ?
example: Java language
Declaration: class declaration, interface declaration
Method: method body, constructor, static initializer
statement: do, for, if, switch, synchronized, try, while

NASSCOM Quality Summit~172~

Code clones which
CCFinder detects

fragment 1
609:
610:
611:
612:
613:
614:
615:
616:
617:
618:
619:
620:
621:
622:
623:
624:
625:
626:
627:
628:

Code clones which
Aries extracts

fragment 2

reset();
623:}
grammar = g;624:// Lookup make-switch threshold in the grammar
625://
generic
Lookupoptions
bitset-test threshold in the gramif (grammar.hasOption("codeGenMakeSwitchThreshold")){626:
if (grammar.hasOption("codeGenBitsetT
try {
627:
try {
makeSwitchThreshold = grammar.getIntegerOption("codeGenMakeSwitchThre
628:
bitsetTestThreshold = gramma//System.out.println("setting codeGenMakeSwitchThresholdto " + makeSwitchT
629://System.out.println("settingco} catch (NumberFormatException e)630:{} catch (NumberFormatException e
tool.error(
631:
tool.error("option 'codeGenMakeSwitchThreshold'632:
must be "option
an integer",'codeGenBitsetTe
grammar.getClassName(),633:
grammar.getClassName(grammar.getOption("codeGenMakeSwitchThreshold").getLine()
634:
grammar.getOption("code
);
635:
);
}
636:
}
}
637:
}
638:
// Lookup bitset-test threshold in the grammar
options
639: generic// Lookup
debug code-gen in the gramma
if (grammar.hasOption("codeGenBitsetTestThreshold")){640:
if (grammar.hasOption("codeGenDebug"
try {
641:
Token t = grammar.getOption("codebitsetTestThreshold = grammar.getIntegerOption("codeGenBitsetTestThreshold642:
if (t.getText().equals("true")) {NASSCOM Quality Summit~173~

Refactoring Support System: Aries (2)The following refactoring patterns[1][2] can be used to remove code sets including
structural code clones
Extract Class,
Extract Method,
Extract Super Class,Form Template Method,
Move Method,
Parameterize Method,
Pull Up Constructor,
Pull Up Method,For each clone set, Aries suggests which refactoring pattern is applicable by using
metrics.

[1]: M. Fowler: Refactoring: Improving the Design of Existing Code, Addison-Wesley, 1999.
[2]: http://www.refactoring.com/, 2004.
NASSCOM Quality Summit~174~

NASSCOM Quality Summit~175~

Change Support System: Libra
Input a code fragment

NASSCOM Quality Summit~176~

Change Support System: Libra (2)Find clones between the input and target

NASSCOM Quality Summit~177~

Applications

2006/9/12

NASSCOM 2006

Academic-industrial
Academic-industrial collaboration：
collaboration：

Code Clone Seminar

We have periodically organized code clone seminars from Dec
2002
The seminar is a good place to exchange views with industrial
peopleSeminar overview
Tool demonstration
Lecture of how to use
code clone information
Case study ofcompanies using
our tools

NASSCOM Quality Summit~179~Summary of Code Clone Analysis andApplication

2006/9/12

NASSCOM 2006

Conclusion
We have developed Code clone analysis tools
Detection tool: CCFinder
Visualization tool: Gemini
Refactoring support tool: Aries
Debug support tool: Libra

We have promoted academic-industrial collaboration
Organize code clone seminars
Manage mailing lists

We have applied our tools to various projects

NASSCOM Quality Summit~181~

This is YOUR presentation!
What is the lifecycle of quality improvement?
What are the areas of quality improvement?
What are specific tools or interventions that quality
improvement recommends?

NASSCOM Quality Summit~182~

How do empirical methods relate to quality
improvement?
Relative Advantage: How much benefit do we get?
Compatibility: How well does it fit with what we alreadydo?
Complexity: How hard is it to do?
Observability: Will anyone notice?
Trailability: Can we try it out without risk?

NASSCOM Quality Summit~183~

Team Exercise
As a team, consider those questions:
How would you describe quality improvement?
How could you use empirical methods for quality
improvement?
What kind of plan would show what you intend to do withquality improvement and identify where you will use
empirical methods in that process?

NASSCOM Quality Summit~184~

Large Group
Plan-Do-Check-Act is often given as an approach toquality improvement.
Do you use empirical methods in this cycle? Do you collect
data? Do you analyze it? Do you provide feedback?

NASSCOM Quality Summit

~185~

Plan
1
0.9
0.8
0.7
0.6

Collect Data
Analyze
Feedback

0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Empirical
~186~

Do
1
0.9
0.8
0.7
0.6

Collect Data
Analyze
Feedback

0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Empirical~187~

Check
1
0.9
0.8
0.7
0.6

Collect Data
Analyze
Feedback

0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Empirical
~188~

Act
1
0.9
0.8
0.7
0.6

Collect Data
Analyze
Feedback

0.5
0.4
0.3
0.2
0.1
0
NASSCOM Quality Summit

Empirical~189~

Break!

2006/9/12

NASSCOM 2006

Summary, Next Steps, and Conclusions

2006/9/12

NASSCOM 2006

Review the day
Introductions
Data collection: GQM
Data analysis: collaborative filtering
Feedback: simulations and training
Quality Improvement and Empirical Methods: Measure,
Analyze, and Feedback

NASSCOM Quality Summit~192~

Team Exercise
As a team, list and rank top five points from the day foryou

NASSCOM Quality Summit~193~Quick Collection
Please tell me your top point that has not been mentioned
yet!

NASSCOM Quality Summit~194~

Large group
Let them think individually
Do quick collect as sample

NASSCOM Quality Summit~195~

Next Steps?
These are your ideas. I’d suggest repeating the process we just used
to collect key points.
One possibility: we would like to create and host a conference tofoster India-Japanese research exchange. This would be intended forshort papers and demos. We are considering hosting the initial
meeting in Japan, perhaps with a technology tour.

NASSCOM Quality Summit~196~

Your Questions?
If you want to send us questions later, send them tombarker@computer.org

NASSCOM Quality Summit~197~Instant Vote!
1
0.9
0.8
0.7
0.6
0.5

Data Collection
Data Analysis
Feedback
Quality and Empirical

0.4
0.3
0.2
0.1
0

Did you learnsomething?

NASSCOM Quality Summit

Useful?

~198~

Another Instant Vote!

10
9
8
7
Data Collection
Data Analysis
Feedback
Quality and Empirical
Overall

6
5
4
3
2
1
0

NASSCOM Quality Summit

Were you a good audience?

~199~

Thank you all!
どうもありがと(Remember to turn in your takeaway sheets!)

2006/9/12

NASSCOM 2006