# Presentation: 489826

## Lessons learned from 25 years of process improvement:

- Victor Basili
- University of Maryland and Fraunhofer Center Maryland
- Frank McGarry
- Computer Sciences Corporation
- Rose Pajerski
- Fraunhofer Center Maryland
- Marvin Zelkowitz
- University of Maryland and Fraunhofer Center Maryland

- Nontechnical issues in process improvement

## Talk based upon work at NASA/SEL

- 25 years of research in software process improvement
- Collaboration among NASA/GSFC, University of Maryland, and Computer Sciences Corporation
- First awardee of Computer Society– SEI Process Achievement award in 1994
- Published over 250 technical papers
- A leader in an empirical-based software engineering research approach based upon measurement, evaluation, and feedback

- But, alas, the SEL no longer exists

## Purpose of talk

- Use of SEL history as a legacy to software process improvement
- Not meant to show you which techniques you should use in your environment
- Process improvement is a local activity
- What succeeds for NASA space shuttle may not be appropriate for your toaster
- But, based upon 25 years of experience:
- What have we learned to do right?
- What did we do wrong?
- What could we do different next time?
- What advice can we offer to new process improvement activities?

## Outline of talk

- Brief history of SEL
- Describe SEL activities through 3 major phases
- 1976-1983: Emphasis on Data collection
- 1984-1994: Emphasis on a learning organization
- 1995-2001: Change
- What are our lessons learned from 25 years of activity?
- Summarize 13 attributes that are crucial for a successful process improvement activity

## Creation of the SEL

- Discussions in April 1976 between University of Maryland and NASA/GSFC flight dynamics branch for a research program in software engineering
- Initial grant began August 1976
- Initial goals: “to analyze the software development process and the software produced in order to understand the development process, the software product itself, the effect of various ‘improvements’ on the process with respect to the methodology, and to develop quantitative measures that correlate well with intuitive notions of good software” [Technical report, 1977]

- These goals did not change much during next 25 years

## Original objectives

- Understand:
- the underlying principles involved in software and the development process,
- the factors that affect the development of software and their interrelationships,
- the characteristics of classes of problems and products

## SEL studies

## SEL publications

## So What happened?

- Reorganization of NASA/GSFC in mid-1990s
- Changing environment: “Faster, better, cheaper”
- New management structure that didn’t understand the SEL
- Lack of organizational support and changing funding structure
- So SEL ceased to exist at the end of 2001

- What did we learn over these 25 years?

## Phase 1  1976-83: Data collection

- Goal
- Data collection as a means to evaluate development
- Concept:
- Data collection forms developed
- NASA collect data and University process data into PDP 11/45
- Reality:
- A naïve view of data collection
- Development contractor suspicious of forms
- University not oriented toward industrial schedules

## Data collection process

- Computer Sciences Corporation took over processing of forms
*Lesson 1: Data collection requires a rigorous process and professional staff.*
- Data we wanted and data we could obtain from programmers often differed:
- Ask for more, collected data is less precise
- Ask for less, collected data answers fewer questions
*Lesson 2: You must compromise in asking for only as much information as is feasible to obtain*

## Conformance to data collection

- Needed staff training repeatedly. Lack of training caused reliability of data to suffer.
*Lesson 3: Staff training in data collection is a never-ending vigil. *
*Periodic “all hands” meetings to give feedback on research*
*Needed rapid feedback on data collected*
*But during crisis periods (e.g., deadlines) data collection suffered.*
*Lesson 4: As important as data collection is, it still takes second place to deadlines. *

## Research approach

- Early activities centered on understanding the SEL environment: resource models (e.g., costs and schedules) and defect models. Baselines of technology:
*screening* experiments (understand what’s going on now) – baselining current GSFC projects,
*semi-controlled* experiments – comparing different methodologies on different related projects,
*controlled* experiments – replicating the same project using different methods at GSFC and UMD.

## Baseline examples 

- Baseline examples** **

- *Data from 11 Flight Dynamics projects (mid 1980s)

- TEST

- 30%

- DESIGN

- 23%

- CODE

- 21%

- OTHER

- 26%

- 85% code writing

- 15% code reading

- Effort Distribution

- Source Code Growth Rate

- 93M11U1.007

- PROJECT PHASE

- 0

- 20

- 40

- 60

- 80

- 100

- DESIGN

- CODE

- SYSTEM

- TEST

- ACCEPTANCE

- TEST

## Three part management structure

- CSC brought into management structure early (1978)
- NASA needs drove SEL research. NASA managed the appropriate spacecraft software and ensured that the necessary data was collected.
- The University was needed to oversee the research under study, provide research hypotheses to test, and analyze the data.
- CSC provided process conformance and data collection.
- Common management of SEL research and Flight Dynamics software development crucial for success.

## Lessons learned from phase 1:

*Lesson 5: Establishing a baseline of an organization’s products, processes, and goals is critical to any improvement program.*
*Lesson 6: The accuracy of the measurement data will always be suspect, but you have to learn to live with it and understand its limitations.*
*Lesson 7: There will always be tension between the need to rapidly feed back information to developers and the need to devote sufficient time to do an analysis of the collected data. *

## Phase 2  1984-94: A learning organization

- Goals:
- liaison with the software engineering research community to identify potential technologies to solve specific development problems at NASA
- evolve the SEL's empirical software engineering research program integrated within the SEL's flight dynamics development activities,
- build an experience base to provide insights for both future NASA missions and other software development groups outside of NASA.

## Activities during this period

- After 8 years we believed we knew how to collect data and the limitations inherent in the process
- Developed models of the improvement process:
- Quality Improvement Process (QIP)
- Goals Questions Metrics (GQM)
- Experience Factory (EF) (around 1990)
- Looked at specific technologies: Ada, cleanroom, IV&V, Object oriented technology, ...
- Develop a learning organization that would understand its development processes, evaluate potential improvement technologies, and feed back that information to developers

## Roles

- NASA management: operational control of the SEL.
- CSC: primary developer of flight dynamics software.
- University of Maryland: the research focus.
- Close to GSFC: daily interaction with developers; University a part of development; No “ivory tower” syndrome
- Use of other research grants and students to prototype and test technologies before use at NASA
*Lesson 8: There is a symbiotic relationship between research and practice in software engineering, and both activities gain from the interaction.**** ***
*Lesson 9: Close proximity of researcher to developer aids both*

## SEL research

- Relatively homogeneous environment
- Flight dynamics software (AGSS, dynamic simulators)
- ~10 staff, 100-200 staff months of effort
- FORTRAN development
- View many developments as replicated case studies
- In vitro experiments tested at university
- In vivo experiments on NASA missions

## Sample results

| Attribute

 | Change:
1987-1991 | Change:
1991-1995 |
| --- | --- | --- |
| Reduced cost | 55%
 | 42% |
| Decreased development
 defect rates | 75%
 | 37%
 |
| Improved reuse | 300%
 | 8%
 |

- Many other results during this period

## Outreach

- Engage NASA personnel:
- Large technical report series (Manager’s handbook, Recommended approach to software development, General Object Oriented Design [GOOD], ...)
- Annual Software Engineering Workshop:
- From 28 in August 1976 to over 600 in 1992
- Feedback, meetings, monthly reports

## Issues during phase 2

- Flight Dynamics management direct participant in SEL research
*Lesson 10: Having a shared commitment over research and development is vital for success.*
- But often slow to feed back information to developers
- Not often clear what research goals were
- Did not do enough to extend reach beyond flight dynamics branch
- Seeds for later problems

## 1994 ...

- First IEEE-SEI Software Process Achievement award

- But change was imminent

- Research funding at a high rate

## Phase 3  1995-2001: Change

- A NASA-wide reorganization to occur
- Amidst the reorganization rumors, senior managers retired (including Flight dynamics branch head and division head)
- The mantra of "faster, better, cheaper"
- Using COTS was a revolutionary change to the usual custom development
- SEL was losing its contact with projects; an increasing preoccupation on decreasing delivery schedules and costs by project managers
- The SEL became less the driver and more the observer, less proactive and more reactive

## 1997 Reorganization

- Decoupled Project management with SEL research
- SEL in Information Systems Center (ISC)
- Project managers in other centers
- ISC leadership unsure of role of SEL
- New policy of empowerment
- Project manager had control over the processes
- Cutting overhead costs drove major ISC decisions
- COTS development grew; SEL viewed as a custom software organization
- The SEL began to lose its ability to monitor and analyze projects.
*Lesson 11: Having upper management support is important for continued success. *
*Lesson 12: The organization trying to improve their process has to own the improvement process.*

## Internal SEL organization

- Structure
- After Flight Dynamics Branch Head retired, SEL director and branch director were separate
- After 1997 reorganization, no SEL director for a period of time
- No internal NASA staff to work on SEL issues
- Outside director hired in 2000, but minimal budget and no staff
- Research
- Continued to work on important NASA issues: COTS development and defect reduction

## Finally, 2001

- Funding by Mission operations office terminated (They fund missions, not research)
- Management claimed SEL had value, but no one able or willing to do something about it
- Funding cut in 2001 by 60% from 2000; 100% in 2002
*Lesson 13: It is difficult to make a hardware organization aware of the importance of software engineering to their mission. *

## Lessons learned from phase 3

- There was no owner of the development process. Each project could have its own structure. The SEL no longer had a central model to build on.
- There was a loss of commitment on NASA/GSFC's part that process improvement was important. There was no vision on what was needed.
- There was significant restructuring at NASA/GSFC that the SEL could not keep up with.
- On the SEL's part, unsuccessful salesmanship in selling vision to NASA project managers.

## Summary

- We have no doubts about the SEL’s value to the evolution of the flight dynamics division or to contributions in the area of software process and product improvement
- But doing good research is only one issue; you have to establish an environment where the research and improvement can be conducted
- The 13 lessons previously mentioned can be grouped into 4 broad areas ...

## Need for collecting project data

- Lesson 1: Data collection requires a rigorous process and professional staff.
- Lesson 2: You must compromise in asking for only as much information as is feasible to obtain.
- Lesson 5: Establishing a baseline of an organization’s products, processes, and goals is critical to any improvement program.
- Lesson 6: The accuracy of the measurement data will always be suspect, but you have to learn to live with it and understand its limitations.

## Need for management buy-in to the process

- Lesson 10: Having a shared commitment over research and development is vital for success.
- Lesson 11: Having upper management support is important for continued success.
- Lesson 12: The organization trying to improve their process has to own the improvement process.
- Lesson 13: It is difficult to make an engineering organization aware of the importance of software engineering to their mission.

## Need for a focused research agenda

- Lesson 8: There is a symbiotic relationship between research and practice in software engineering and both activities gain from the interaction.*** ***
- Lesson 9. Close proximity of researcher to developer aids both.

## Need for continued staff support

- Lesson 3: Staff training in data collection is a never-ending vigil.
- Lesson 4: As important as data collection is, it still takes second place to deadlines.
- Lesson 7: There will always be tension between the need to rapidly feed back information to developers and the need to devote sufficient time to do an analysis of the collected data.

## Conclusions

- We had a successful 25 year run in process and product improvement activities, although we are obviously sorry to see it end
- We developed several improvement approaches that have been adopted and adapted elsewhere
- We have moved on to an activity called CeBASE (Center for Empirically-based Software Engineering) with the University of Southern California (www.cebase.org) to extend the Experience Factory approach

## That’s all folks!