# Presentation: 817086

## Software Metrics Selection

*A **tutorial prepared for NASA personnel by *
*the NASA Software Working Group*
*and*
*the Fraunhofer Center for Empirical Software Engineering*
*College Park, MD*

*This work was supported in part by NASA Grant NAG5-12556*

## Audience

- NASA personnel who:
- Manage or lead software development and/or maintenance efforts
- Want to be able to choose appropriate metrics that will deliver useful information without unnecessary effort
- Make management and technical decisions about NASA software
- (S)EPGs (or similar groups) who are interested in helping software projects use measurement effectively

## Purpose

- Explain how our measurement methodology works
- Describe the background work already done
- Teach how to apply the project-specific part of the measurement methodology
- Give lots of examples
- Answer lots of questions
- Transfer the ability to effectively and efficiently select appropriate metrics for any software effort

## Outline

- Motivation and Examples
- Overview of the FC-MD Measurement Methodology, as tailored for NASA
- Metrics selection infrastructure
- Project-level metrics selection process
- Q&A

## Why Software Metrics?

- Management based on objective indicators that reflect actual project performance
- Decision-making based on quantitative evidence
- Basis for future improvements and baselining

## Measurement for Management

- Measurement is a tool for *rational* management, for example:
**Project level:**
- Plan, track and control** **project development and evolution
- e.g., what should happen, is it happening?
**Product level:**
- Evaluate the quality of the product
- e.g.,  what is the reliability of the product after delivery?
**Organizational level:**
- Create a shared memory - baselines/models of current practices
- e.g., how much will a new project cost?** **

## Measurement for Decision Making

- ... and for *rational* decision making, for example:
**Technology Adoption:**
- Develop a rationale for adopting/refining techniques
- e.g., what techniques will improve schedule performance?
- Assess the impact of techniques
- e.g., does functional testing minimize certain error classes?
-  **Process Improvement:**
- Determine strengths and weaknesses of the current process and product
- e.g., what parts of the process could be streamlined to reduce overall project cost?

## Measurement Examples*

- Example 1
- Schedule slippage caused an early replan
- Actuals close to plan until recently
- Not getting the upturn needed at this point in the project

- Next step – look at design progress by subsystem – is this an across-the-board problem or is it isolated?

** All examples shown are adapted from “Practical Software and Systems Measurement” from the US Department of Defense and the US Army, Version 4.0b, October 2000*

## Measurement Examples

- Example 2
- Incremental development
- Plan 3 represents actuals for builds 1 and 2
- Chart makes it obvious how much has been pushed to Build 3

- Next step – look at effort and schedule plan for build 3 – is it realistic?  Monitor build 3 progress closely.

## Measurement Examples

- Example 3
- Used when funding comes in chunks (e.g. contractor)
- Shows that risky period (Mar 99) was survived because of much lower than expected costs

- Next step – will costs deferred in Mar 99 “catch up” at end of project?

## Measurement Examples

- Example 4
- Shows growth in requirements
- Shows numbers of requirements added, changed, deleted
- Shows a fair amount of activity early on, but is settling down

- Next step – superimpose with project phases – green line should be very close to 0 by start of design

## Measurement Examples

- Example 4 (next step)
- Start of design should not be impacted

- SRR

## Measurement Examples

- Example 5
- Shows open, new and closed defects
- Indicates a significant, but decreasing, backlog of open defects
- Given sufficient time, all defects should be closed

- Next step – project “open” line to make sure all defects will be closed before planned delivery date

## Measurement Examples

- Example 6
- Compares current project to a historical baseline
- Indicates a lower-than-expected defect density
- Better system, or lower defect detection?

- Next step – examine early defect detection activities (e.g. inspections), as compared to past projects

## Measurement Examples

- Example 7
- Shows testing is way behind schedule
- Level of testing effort somewhat below plan
- Most of problem is quality of code

- Next step – replan; increase testing resources and plan for major rework

## Measurement Examples

- Example 8
- Use of historical data to create a baseline model showing relationship between size and effort
- Shows where baseline is sparse and thus, the model is less reliable

- Used to estimate effort for a new project when size has been estimated

## Measurement Examples

- Example 9
- MTBF – mean time between failures
- Often part of RMA requirements
- Use of historical data to create a baseline model showing expected MTBF for different types of systems

- Used to set targets for new projects or check reasonableness of proposals

## Measurement Example: Readiness for Delivery

- For many decisions, a variety of factors must be taken into account
- In deciding a system’s readiness for delivery, one must look at trends in:
- Test completion
- Open problems
- Quality thresholds
- Frequency of failures
- No single chart will show everything
- Criteria for readiness must be established upfront between managers and customers

## Readiness for delivery: Test Progress

- Shows number of successfully tested requirements
- Testing proceeding close to plan
- Almost 80% of requirements successfully tested

- From this point of view, things look good for an on-time delivery

## Readiness for delivery: Problem Report Status

- Open problems at all priorities are going down
- A few high priority problems still remaining
- How many lower-priority problems can remain in the system?

- An on-time delivery still looks likely, as long as the remaining high-priority problems can be resolved, and the numbers of other open problems are within limits

## Readiness for delivery: Staff Level

- The plan was for testing staff to ramp up, then quickly come down as testing was progressing and completing
- However, the planned reduction in testing staff has not happened yet.

- Staffing level indicates that more than enough resources have been applied to testing
- Supports argument that testing is nearing completion

## Readiness for delivery: System Availability

- Readiness for delivery: System Availability

- Shows the availability of the system during acceptance test
- Contract requirement threshold also shown
- Availability showing nice improvement
- Not a concern at this point

- From the point of view of required availability, there are no indications that delivery will be delayed.

*Threshold** **= 99%*

## Readiness for delivery: CPU Utilization

- Shows the CPU utilization of the system, measured against the contract requirement for a 50% reserve
- Not clear if the threshold can be achieved by delivery

- Tradeoff between complying with the CPU utilization requirement and delivering on time may have to be negotiated with the customer

## Readiness for Delivery: Analysis

- Indicators show that on-time delivery is likely
- Things to watch closely until delivery date:
- CPU utilization
- Open high priority problem reports
- Be prepared to negotiate CPU utilization with customer
- Deciding what is “good enough” is difficult and complex
- Metrics will not dictate criteria, but will indicate if criteria are being met or not, and why

## The FC-MD/NASA Measurement Methodology

- Objectives:
- Allow an organization to set up organization-specific mechanisms that enable software projects flexible choices and guidance in choosing software metrics
- Allow a software project to choose the most efficient and effective set of metrics for that project, while complying with higher-level organizational requirements

## The FC-MD/NASA Measurement Methodology (cont.)

- Involves activities at two levels
- Organization-wide activities
- Carried out by measurement experts and managers at MD, Center, and Program level
- Sets up the framework (infrastructure) within which projects can choose their metrics
- Project-specific activities
- Carried out for each project by the software lead
- Choose, plan, and carry out measurement activities

## The FC-MD/NASA Measurement Methodology: Basic Approach

- Develop a set of measurement scenarios
- Each describes a hypothetical project in terms of its measurement goals and other project characteristics
- Each is associated with a set of metrics that both satisfies the requirements of the organization, as well as addresses the project’s goals and is reasonable given the project characteristics
- Each project chooses the scenarios that describe it
- The scenarios provide a set of metrics that the project can adopt in its measurement plan

## The FC-MD/NASA Measurement Methodology

**Notes:**
Software Measurement Service
Method Support
Why these methods
1. Verify Commitment
PROFES: Commitment support and guidance
2. Identify Goals
BSC & SPC: measurements linked to higher level organizational goals
PSM: Stresses project issues (goals) as starting point for lower level measurement
PROFES: Links product quality goals explicitly to software process (Profes is about product quality based SPI methodology)
3. Characterize Projects
QIP: Overall guidance
GQM: provides a mean to characterize
MIKKO: Guides explicitly measurement process characterization
4. Develop Goal-Metric Scenarios
-
5. Plan Implementation
GQM: Measurement plan guidance
MIKKO: Utilization plan guidance
6. Implement and Measure
GQM: provides best available guidance for this step
7. Analyze 
GQM: strong in analysis (GQM feedback sessions)
8. Package
EF: provides overall structure

## Phase 1: Establish or Verify Commitment

- This phase is performed at the Agency level by measurement experts (internal and external)
- Establish or verify managerial commitment
- Clarify organization’s and projects goals and context
- Achieve mutual understanding of the context of the work
- Make sure everyone’s on board
- Main Results:
- A detailed proposal or a contract of the work if not done earlier
- Commitment = $$$
- A work plan for further activities
- Evaluation criteria

## Phase 2: Identify Goals

- Again, this is an Agency-level activity performed by measurement experts
- Identify, understand and make explicit the goals applicable to projects across the Agency
- Identify statements and plans (such as the NPR) that set goals for the organization and for projects
- Explicitly link and prioritize the identified goals.
- Refine into measurement goals
- Build a hierarchy of goals

## Phase 3: Characterize Projects 

- Agency-level activity performed by measurement experts, using data and information from representative projects
- Define typical project types using measurement data from projects and information about project characteristics
- Use existing data wherever possible, augmented by interviews
- Attempt to determine which project characteristics influence measurement needs and practices.
- Identify, analyze and name typical classes of projects the organization has

## Phase 4: Develop Goal-Metric Scenarios  

- Scenarios are built by measurement experts for use across the Agency
- Scenarios packaged into a “Goal-Metric Matrix”
- A row for each measurement goal
- A column for each project type
- In each cell, a set of metrics appropriate for a project of that project type (column) trying to satisfy that goal (row) – a “scenario”
- Based on:
- Measurement data from existing projects of different project types
- Experience base (internal experience, general knowledge, literature etc.)
- Expertise of FC-MD and NASA measurement experts
- Matrix is augmented with guidance on data collection and analysis procedures

## Main Result After Phase 4:The Goal-Metric Matrix (an excerpt)

## Phase 5: Plan Implementation

- At the project level (software leads)
- Using Goal-Metric Matrix, choose metrics to be used
- Incorporate metrics information into project plans
- Plan data collection and analysis procedures and needed infrastructure
- At the organizational level (measurement leads)
- Put infrastructure in place for receiving and utilizing measurement data from projects
- Conduct pilot projects
- Facilitate adoption of measurement approach by projects

## Phase 6: Implement and Measure 

- Just do it!
- At the project level
- Develop software
- Collect and analyze data according to plan
- Gather feedback on what’s working and what’s not
- At the organizational level
- Receive project-level data and analyze for status and trends
- Gather feedback on usefulness and effectiveness of measurement plans
- Feedback loops are crucial!

## Phase 7: Analyze 

- Post-mortem analysis of measurement approach
- Both project and organization need to look at what worked and what didn’t
- Evaluate the Goal-Metric Matrix and its use
- Right project types?
- Right goals?
- Right metrics?
- Seek improvements in either the Matrix or other measurement plans
- Gather lessons learned
- Contribute data to further development of models and baselines

## Phase 8: Package

- Agency-level analysis of project experience (performed by measurement experts)
- Generalize feedback and lessons learned
- Package experiences with context information
- Update Goal-Metric Matrix with experience packages (advices, graphs, lessons learned...)
- Update baselines and models and package them for use by future projects
- Disseminate experiences and results achieved via several forums e.g. meetings, bulletin boards etc.

## What’s been done so far...

- Phases 1 through 4 of the measurement methodology have been carried out by the SWG and FC-MD experts, with the help of information from projects across NASA
- NASA-specific results – “infrastructure”:
- Project types
- Measurement goals
- Goal-metric matrix
- These results make it possible for individual NASA software projects to start applying the method

## What comes next:Project-level metrics selection process

- Overview:
- Determining project type
- Eliciting measurement goals
- Refining resulting list of metrics
- Defining collection and analysis procedures
- Doing it
- Feedback loop

## Project-level metrics selection process:Determining project type

- Project type VI reserved for projects with:
- Largely inexperienced management or technical staff
- New or innovative technologies
- Brand-new product
- Project type VII reserved for maintenance projects
- No significant new functionality being developed

## Project-level metrics selection process:Determining project type (cont.)

- Otherwise, project type is determined by:
- Project size – small/medium or large/huge – take into account system size, number of personnel, level of effort, duration
- Maturity of development organization – low (CMM/I 1) or high (CMM/I 3) – rough estimate if CMM assessment not available
- Development work is to be done in-house or contracted out (in the case of contractors working in-house alongside civil servants, counts as in-house)

## Project-level metrics selection process:Determining project type (cont.)

| Project Type | Size | Maturity | In-house? |
| --- | --- | --- | --- |
| I | Small/medium | Low | no |
| II | Small/medium | Low | yes |
| III | N/A | High | no |
| IV | Large/huge | Low | no |
| V | Large/huge | N/A | yes |

## Project-level metrics selection process:Determining project type (cont.)

*A potential input screen for determining project type:*

*Step 1: Determine project type*

- Choose the options that best describe your project

- Project size:		small/medium		large/huge
- Project maturity		low (e.g. CMM 1)		high (e.g. CMM 3)
- Contractor role:		contracted out		in-house

- Submit

- Cancel

## Project-level metrics selection process:Project measurement goals

- Many goals are determined by the NPR and MD/Center/Program requirements
- Additionally determined by the software class (A through H)
- Other goals are specific to a project and can be derived from:
- Risk analysis
- Project constraints and assumptions
- Use of new technologies
- Product acceptance criteria
- Problem areas on previous similar projects

## Whether or not a project must comply with NPR metrics requirements depends on software class
For remainder of presentation, we will assume that the project must comply with the NPR
NPR for Software includes 5 measurement areas:
Software Progress
Software Functionality
Software Quality
Requirements Volatility
Software Product Characteristics
A project may also have other areas of interest

- For remainder of presentation, we will assume that the project must comply with the NPR
- NPR for Software includes 5 measurement areas:
- Software Progress
- Software Functionality
- Software Quality
- Requirements Volatility
- Software Product Characteristics
- A project may also have other areas of interest

- Project-level metrics selection process:Project measurement goals (cont.)

## A project must report in all 5 measurement areas, but a project may choose which metrics they will collect and why
A project’s measurement goals drive the choice of specific metrics in each area
Within each measurement area, possible goals range from complying with the NPR to using the data to control and manage the project

- A project’s measurement goals drive the choice of specific metrics in each area
- Within each measurement area, possible goals range from complying with the NPR to using the data to control and manage the project

- Project-level metrics selection process:Project measurement goals (cont.)

## Possible project measurement goals related to Software Quality:
Comply with NPR Software Quality Measures
Characterize the number of defects in the software product
Track the defects in the software product over time
Evaluate the defects in software product as compared to a baseline
Monitor the number of open severe defects 
Track the progress of the test process
Track the progress and results of the inspection/review process
Track the project risks and associated measures
A project can choose one or more of these goals that best reflect the software lead’s objectives in collecting software quality data

- Comply with NPR Software Quality Measures
- Characterize the number of defects in the software product
- Track the defects in the software product over time
- Evaluate the defects in software product as compared to a baseline
- Monitor the number of open severe defects
- Track the progress of the test process
- Track the progress and results of the inspection/review process
- Track the project risks and associated measures
- A project can choose one or more of these goals that best reflect the software lead’s objectives in collecting software quality data

- Project-level metrics selection process: Project measurement goals example:Software Quality

## Apart from the NPR software measurement areas, there are a number of other concerns or issues that a software lead may wish to include in their list of measurement goals, e.g.
Tracking rework
Tracking reuse 
Tracking reusability of developed software artifacts
Complying with CMMI measurement requirements
Assessing the architectural design
Tracking costs related to SPI efforts and initiatives
The software lead can choose from a list of goals in this area, or create new ones for the project

- Tracking rework
- Tracking reuse
- Tracking reusability of developed software artifacts
- Complying with CMMI measurement requirements
- Assessing the architectural design
- Tracking costs related to SPI efforts and initiatives
- The software lead can choose from a list of goals in this area, or create new ones for the project

- Project-level metrics selection process:Project measurement goals (cont.)

## Project-level metrics selection process:Project measurement goals (cont.)

*A potential input screen for choosing goals:*

*Step 2: Choose measurement goals*

- Select all the measurement areas in which you want to choose goals

- Software Progress		Requirements Volatility
- Software Functionality	Software Product Characteristics
- Software Quality		Other

- Submit

- Cancel

## Project-level metrics selection process:Project measurement goals (cont.)

*A potential input screen for choosing goals:*

*Step 2: Choose measurement goals -> Software Quality*

- Select all relevant  measurement goals in the area of Software Quality

- Submit

- Cancel

- Comply with NPR Software Quality Measures
- Characterize the number of defects in the software product
- Track the defects in the software product over time
- Evaluate the defects in software product as compared to a baseline
- Monitor the number of open severe defects
- Track the progress of the test process
- Track the progress and results of the inspection/review process
- Track the project risks and associated measures

## The software lead must compile a list of project measurement goals, based on:
The lists of goals for each NPR-required measurement area
The list of other possible project-level issues
Other concerns that the software lead wants to address via measurement
The result is the project’s list of measurement goals, which is then used in the next step of the measurement methodology...

- The lists of goals for each NPR-required measurement area
- The list of other possible project-level issues
- Other concerns that the software lead wants to address via measurement
- The result is the project’s list of measurement goals, which is then used in the next step of the measurement methodology...

- Project-level metrics selection process:Project measurement goals (cont.)

## Project-level metrics selection process:Creating the list of metrics

- Identify relevant cells in the goal-metric matrix
- Column determined by project type
- Row determined by measurement goals
- Initial metrics list includes all metrics and indicators listed in relevant cells
- Remove redundancy in list
- Get the biggest “bang” for each metric
- Most metrics can help satisfy multiple goals

## Project-level metrics selection process:Defining collection and analysis procedures

- Metrics information must be incorporated into project plan
- Data collection and analysis procedures must be specified in operational detail
- Procedures should take into account:
- Existing procedures
- Development process and milestones
- Available tools

## Project-level metrics selection process:Defining collection and analysis procedures (cont.)

- Data collection procedures should include, for each data item:
- Who will provide it
- When it will be provided
- The instrument used to record it (how)
- Where it will be stored
- Data analysis procedures should include, for each indicator:
- Who will create it
- To whom it will be reported
- The procedure by which it will be created
- How it is to be interpreted (including possible action items)

## Project-level metrics selection process:Doing it!

- Develop software according to plan
- Follow data collection and analysis procedures
- Plan can be modified, but all changes must be documented and disseminated
- To document lessons learned
- To ensure consistency among everyone collecting and analyzing data

## Project-level metrics selection process:Feedback loop

- Part of project post-mortem
- Reflect on the appropriateness of the project type and the measurement goals
- Reflect on how well the goals were met by the metrics and indicators
- Review changes made in the plan
- Gather lessons learned
- Recommend changes to goal-metric matrix

## Summary

- Background on software measurement and examples
- A software measurement methodology
- Infrastructure work already completed for NASA
- Process for software leads to plan and carry out a project-level measurement program
- Determining project type
- Choosing measurement goals
- Developing the measurement plan
- Implementing the plan and feeding back the experience

## Questions?

- Thank you!!
- Carolyn Seaman
*cseaman@fc-md.umd.edu*
- Rose Pajerski
*rpajerski@fc-md.umd.edu*