# Presentation: 417590

## Goddard Process Improvement Project
February- 14, 2005

**February- 14, 2005**

**“****10” Elements of**** Improved Software Development **

## Basis for the Observations

**Information is based on experiences accumulated over a period of 10 to 15 years**
- Primarily based on software development organizations ranging in size from less than 200 to over 2000 professionals (includes some experiences from NASA/GSFC, but primarily form CSC and CSC clients)
- Information includes results of empirical studies, surveys, historical data, interviews, and my own general observations.
- Also includes results of CMM SCE’s and other formal process appraisals.
- Most of these observations are further developed/explained in other reports, papers, and briefings given on particular topics
**In addition to the information of software projects (both successful as well as problematic projects), much of the information is based on efforts of ‘process improvement programs’**
- Approximately 12 major business units contributed information which represents many thousands of staff years of software activity.
**This information is actively used by CSC programs to help formulate process improvement activities for both in-house efforts as well as in support of clients.**** **
**Included are items that the process improvement team can produce as well as items that the PI team can bring to projects**

## References

**‘****Measuring Impacts of Software Process Maturity in a Production Environment’ (McGarry, ‘Metrics ‘98’ November 1998)**
**‘****The Discipline of Process: The Transformation of Software Development’ (Adler, McGarry, Binney, Irion-Talbot, USC, December, 2000 and MISQ –to be published Summer 2005)**
**‘****What’s a Level 5?’ (McGarry, SEL Workshop- December 2001)**
**‘****Software Capability Evaluation (SCE) Final Reports’ (SEAS (1991, 1996, 1997, 1998) ), (CIV (1999, 2001)**
** ‘****Attaining CMM Level 5’ (IEEE Software, McGarry, Decker; Spring 2002...)**
**‘****Eight Key Management Practices for a Diverse Environment’ (CSC Tech and Business Solutions, Laura Cosentino, et. al. 2003)**
**‘****Experiences in Attaining Process Maturity’ (McGarry, Briefing to JPL staff, October, 2002- and other dates)**
**‘****Paradigms of Process Improvement’ (McGarry, Basili, et.al.- Briefings –derived from ‘*****The Experience Factory; How to Build and Run One***’ (Basili- McGarry)
**‘****The Software Engineering Laboratory’ (October 1994, SEI Report-Award for Process Achievement)**
**‘****Data Management and CDX Data Architecture’ (Desantis, Decker, et.al Briefing to EPA, Jan.,2005)**
**‘****7 Guiding Principles of Measurement’  (McGarry, Decker, et.al. first delivered to SEL workshop early 1990’s, multiple derivatives now exist)**

## “10” Elements to Successful Process Improvement

**Engage projects**
**Measure; products and performance **
**Apply Earned Value**
**Empower SQA**
**Establish Process Baseline**
- Process Infrastructure
**Conduct Internal Process Audits**
**PAC**
**Separation of Concerns **

## Engage ‘Projects’ 

***1***

- Process (improvement) team must continually support ‘projects’ as partners
**Writing processes, analyzing compliance, refining structure can be serious distractions;  there is limited value until put into use**
**Target to allocate up to 40% of PI effort in ‘deployment’-**
**Requires capable, trained process engineers**
- Use Shepherds/consultants for projects** **
**Rated as the 2nd most beneficial approach (project feedback) from CMM L5 organization **
**Process experts (from PEO/ QAO) acted as consultant to specific project**
**Internal audits (QAO) used as tutoring and shepherding**
**Required 40% of overall process resources**
- Conduct ‘Deployment’ seminars/meetings
**Rated as the #1 most beneficial approach**
**Combined concepts of “tutorials”, sharing (of project experiences) and project status toward reaching some gate (e.g. preparing for SCE, or SCAMPI)**
**‘****All’ managers invited**
**Significant effort put into preparing ‘relevant’ material- had to be worthwhile**
- Bring concepts of ‘10 to 12’ required/useful activities to projects

**Probably the most critical concept required for success**

## Cost* Distribution for Process

**Cost* Distribution for Process**

- * Includes cost of developing processes, deploying, measuring, training, maintaining (packaging), developing infrastructure, process improvement.  Does not include cost of project ops doing CM, QA, Planning, etc.  It does include their cost for participating in studies, training, audit participation).  Cost based on time: July 1994 through November 1998

*For level 5 organization of 800 persons, over 4 years:*
*We learned that deployment had the value.*

## Measure and report to management and projects 
Report basic trends (e.g. cost, defects, cycle time, estimation)- early
Promotes strong interest in all levels of management
Tracked progress toward objective goals
Helped improvement program recast goals and activities
Included reports of failures or counterintuitive results
Periodic surveys (e.g. process value, awareness) generate wide interest
Retrieve historical data (cost, dates, defects)
At first, ignored existing, historical data (incomplete, old, no QA)
Later found to have wealth of information- with a little work
Helped calibrate measurement program (what was useful vs. not)
Enabled accelerated reporting of trends
Establish a single focus for the collection, archive, reporting
Average .5 -.75 FTE for 15 ‘projects’
Projects impact is minimal
PI team typically carries out analysis of technology and process
e.g Is there a measurable impact of process maturity?

- Report basic trends (e.g. cost, defects, cycle time, estimation)- early
- Promotes strong interest in all levels of management
- Tracked progress toward objective goals
- Helped improvement program recast goals and activities
- Included reports of failures or counterintuitive results
- Periodic surveys (e.g. process value, awareness) generate wide interest
**Retrieve historical data (cost, dates, defects)**
- At first, ignored existing, historical data (incomplete, old, no QA)
- Later found to have wealth of information- with a little work
- Helped calibrate measurement program (what was useful vs. not)
- Enabled accelerated reporting of trends
**Establish a single focus for the collection, archive, reporting**
- Average .5 -.75 FTE for 15 ‘projects’
- Projects impact is minimal
**PI team typically carries out analysis of technology and process**
- e.g Is there a measurable impact of process maturity?

**Measure products and performance***

***2***

- * Briefing describing 7 guiding principles of measurement expands this concept

## Approach to Measuring and Analyzing Trends

**Approach to Measuring and Analyzing Trends**

**Each project that is active in a particular year is included in the year’s average**
**As projects reach completion, their data is added to the analysis (adding information to preceding years)**
**Each trend uses the****same analysis technique**

## Measuring process – Are We Using Process?

**P**rocess
**A**ssessment
**F**orm

- A very quick look at a project’s process use

## Apply ‘Earned Value’ Concepts

***3.***

**Probably the most effective measurement tool we have**
- Rated as one of top 5 reasons for sustained performance
- ‘Required’ on all projects
- Developed internal training for all managers
- Supported by organization tool (Performance Measurement System (PMS) at CSC)
- Addresses: Planning, measurement, reviews, tracking, etc
**‘****Point Counting’ is excellent variation **
**Used as the instrument of review each month by senior manager**
**At CSC, combining EV with organization infrastructure led to decrease of ‘Red Flag’ tasks; 17% to 5% (1996-2001)**
**Enabled planning, tracking, control and infrastructure (reviews, reporting, QA role,...)**
**Used as evidence for CMM(I) assessments**
- Assessors repeatedly expressed value in accomplishing spirit of CMM

## Example of EV (Development Points)

**Typical development tracking**
**Each of the ‘widgets’ is given a point scheme**
- Unit design – 4 pts
- Unit code – 3 pts
- Unit test – 3 pts
**Developer reports completion of each activity regularly (weekly or monthly)**
- Assign responsibility for collection (e.g., QA)
**Reports/plots analyzed by software manager**

**Example – One activity stage**

## Empower SQA

***4***

**Historically, Software Quality Assurance has been ineffective and misused in many environments**
- Generally is major problem area identified by assessments (CMM, ISO, CMMI)
- SQA role in process assurance is often ill-defined
- Limited responsibilities allocated with limited expectations
- Often the organizational structure causes impediments to effective application
**Quality Assurance should be a critical element of process improvement:**
- Realize the value and invest effort to capitalize on the potential
- Successful organizations report SQA as vital to their accomplishments
- Integrated as element of the improvement program
- Redefined historical roles to accommodate process improvement initiative
- Ignored organizational boundaries and clearly identified QA’s process role.
**Role of QA must be clarified for consistent support**
- Allocate reasonable resources
- Clarify/stipulate specific responsibilities
**Sample responsibilities from successful organizations:**
- Verify that agreed processes are known, applied and of value.
- Verify that each deliverable product complies with established form and format.
- Insure all project personnel are aware of their role to assure quality
- E.g. peer reviews, technical reviews, unit testing,...

## Establish Process Baseline( w/support infrastructure)

***5***

**Structure of the written ‘Processes”***
- Typically includes policies, procedures, methods, and support (handbooks)
- There is unintentional overlap of policies and procedures
- Has not caused significant difficulty
- Most value (to Projects) is from the Policies (according to project personnel)
- Value of more detailed methods less apparent
- Selective application of Standards and Procedures
** ****Support infrastructure has been enabling (necessary) attribute**
- Process **Database architecture** is a critical element of success
- Easily accessible, logically organized, controlled (capitalize on paste experiences)
**Tools** (PAL, PAC, PPAF- examples of successful support)
**Support structure**
- QA roles (Internal audits, PAC,consulting)
- Management reviews (PPAF, Internal Audits)
**Deployment**
- Sustained reinforcement of what is required seemed most valuable
- Rationale of ‘why’ of limited value (in the written processes)
**Policies driven by project and management needs (not by benchmarks)**
- Although some adjustments have been made to attain compliance
- Listen (and observe) to projects, but do not produce by Committee

## Example Process Assets LibraryExample from CSC

**Host to all key process assets**
- Documents, reports, lessons, trends,...
**Assures visibility by senior managers and all**
**Instrument for sharing across projects**
**Used to synthesize multiple project activities**

## Conduct Internal Process Audits*

***6***

**Conducted by SQA using staff from SQA, PI as well as projects**
- Each project 2 times per year
- Reported at management reviews
- Results, actions
- Audit of agreed processes (and Product form and format)
- More extensive version of the PPAF reporting
**Enables the propagation of key organization requirements**
- Establish specific criteria required of projects
**Senior Managers rated this as one of top 5 reasons SEAS attained and sustained high performance levels (L5)**
- Gives them confidence key practices are in-place
**SQA and projects agree on process**
- Provided structure for role of SQA, Process, Projects, and management

*** We have developed a training package for conducting ‘Process Assessments’**

## Value of Process Assessments*

**Helps projects improve their software**
- For modest investment, can identify hi-leverage processes/approaches and bring value to the project
**Engages projects**
- Promotes theme of partnership between projects and improvement organization
- Supports the deployment thrust of process improvement
- Raises awareness of software processes and improvement  organization
**Supports the goal of process compliance**
- Step in preparing for formal assessments
- Helps project staff become more aware of organization structure and defined processes

- * Based on experiences from multiple programs at CSC

*Assessments (in any of the forms) rated as one of the top 3 essential ingredients for successful process improvement by CSC managers*

## Process Assessments Cost/Effort(Based on experiences from CSC)

**Total Cost**
- Assessment team
- Typically runs from 6 staff hours to 50 staff hours
- Assessment team ranges in size from 1 to 3 persons
- Project impact
- From 10 to 40 staff hours
- Preparation, gathering artifacts, interviews, debriefing
- Projects range in size from 5 to 60 persons
**Assessment Effort**
- Individual interviews limited to 1 hour, typically 3 to 8 practitioners are interviewed
- Typical time allocation (rough estimate)
- 25% of effort on interviews only (no artifacts)
- 60% interviews w/ artifacts
- 15% evidence and analysis only (no project staff)

## Measuring Process - Trends

- Process use varies over time

## (Adopt Concept of) “PAC”*

***7***

**PAC is nothing more than an agreement between project and QA as to what processes will be used on this project**
- There are checklists, forms and steps that formalize this agreement
- Adds a discipline that encourages project to identify specific processes to be applied
- Forms a partnership between project and SQA
**Agreement is defined at start of project and forms a contract between project and management**
- This agreement becomes the basis for internal audits carried out by the QA role
**Subsequent audits are carried out at key milestones ( or they can be carried out at based on some timeframe)**

*** Process Assurance Cycle**

## Process Assurance Cycle (PAC)

**Periodic audits verify**
**Process Approach Report (PAR) is approved by Project Manager and SQA**
**Project processes are documented**
**Evidence exists that processes are being used**
**Team informed of specific processes in use **
**Non-compliance is reported to senior management**

***Establish agreed project processes and deploy to project team – PAC is the key.***

## “Separation of Concerns”*

***8***

**Project organization focus and priority is to deliver the product using packaged reusable experiences**
- Uses assets supplied by ‘Process Engineering’ organizatione.g. models, lessons, processes, tools
- No need to develop expertise in any external process models (e.g. CMMI)
**Process engineering organization focus and priority is to support project development**
- Analyze experience drawn from people, documents, and measurement
- Synthesize and package that experience into process models and measures
- Supply the experience to various projects as needed
**Hide details of benchmark process requirements from developers**
- Training/deployment should focus completely on organizations process baseline
- Not on detail of CMM(I), ISO and standards details
- Do not deploy multiple forms of required processes (Policies, CMMI, NPR,ISO)
**Measure success of projects by ability to produce****end-product (not by process expertise)**
**Projects should focus on producing good ‘products’, not on learning CMMI Process Areas or ISO Elements**
- Do not expect or require technical staff to be experts in ‘benchmarks (CMMI, ISO, etc.)

## Estimated Effort Required

**Experience shows that a successful organization typically expends 1.% to 3% of resources on process engineering plus task overhead **
- Based on Organization:
- Size from 150 to 1500 persons
- Total staff considered in the scope of defined processes
- Tasks
- Define, develop and implement processes
- Define and operate improvement program
- Operate measurement program
- Coordinate external benchmark application (CMMI, ISO,..)
**For organization initiating new process (improvement) program**
- May require the 3% to 4% for initial organizing, planning, etc.
**For organization at higher maturity levels**
- May require the 1% to 1.5%
**Task personnel overhead runs 1 to 3 hrs/wk**