GAO-01-1015R Survey of NASA’s Lessons Learned Process

United States General Accounting Office

Washington, DC 20548

September 5, 2001

The Honorable Dana Rohrabacher

Chairman, Subcommittee on Space

 and Aeronautics

Committee on Science

House of Representatives

The Honorable Bart Gordon

Ranking Minority Member, Subcommittee

 on Space and Aeronautics

Committee on Science

House of Representatives

Subject: Survey of NASA’s Lessons Learned Process

You asked us to assess whether the National Aeronautics and Space Administration

(NASA) has adequate mechanisms in place to ensure that past lessons learned from

mission failures are being applied. As part of your request, you asked that we provide

the results of our survey regarding the collection and sharing of lessons learned by

NASA program and project managers. This letter provides the results. We plan to

issue a more detailed report on the effectiveness of NASA’s lessons learned process

early next calendar year.

As part of our assessment of NASA’s effectiveness in applying lessons learned, we

surveyed all of NASA’s program and project managers to obtain their perspectives on

(1) the collection, access, and use of lessons learned, (2) the strengths and limitations

of current NASA’s lessons learned processes, procedures, and systems, including the

Lessons Learned Information System (LLIS), (3) the challenges or barriers to the

sharing of lessons learned, and (4) suggested areas of improvement. This

represented 192 managers overseeing about 240 programs and projects. On August 6

and 8, 2001, we briefed your staff on the results of our survey and this report

summarizes those results. The briefing slides are included as enclosure I.

We conducted our work from September 2000 through July 2001 in accordance with

generally accepted government auditing standards. Enclosure II describes our

objectives, scope and methodology.
Page 2 GAO-01-1015R Survey of NASA’s Lessons Learned Process

BACKGROUND

NASA’s procedures and guidelines require that program and project managers review

and apply lessons learned from the past throughout a program’s or project’s life cycle

and to document and submit any significant lessons learned in a timely manner.

NASA defines a lesson learned as “knowledge or understanding gained by experience.

The experience may be positive, as in a successful test or mission, or negative, as in a

mishap or failure. A lesson must be significant in that it has a real or assumed impact

on operations; valid in that it is factually correct; and applicable in that it identifies a

specific design, process, or decision that reduces or limits the potential for failures

and mishaps, or reinforces a positive result.” The failure of the Mars Polar Lander

and Climate Orbiter spacecraft in 1999 raised concern that lessons learned from past

mishaps and programs were not being effectively applied towards future mission

success.

Lessons learned systems are used by many military, commercial and government

organizations to capture, store, disseminate, and share knowledge gained from past

experiences. NASA’s principal mechanism for collecting and sharing lessons learned

from programs, projects, and missions agency wide is the LLIS. The goal of LLIS is to

ensure that NASA does not have to keep “relearning” the lessons of the past. NASA

also shares lessons learned through revisions to its policies and guidance. Further,

lessons learned from a mishap or operational event are captured in procedure and

process documents.

SURVEY RESULTS

Our survey highlighted fundamental weaknesses in the collection and sharing of

lessons learned in NASA by program and project managers as well as in the agency’s

LLIS. While some lessons learning does take place, our survey identified that lessons

are not routinely identified, collected, or shared by program and project managers. In

addition, many respondents indicated that they are dissatisfied with NASA’s lessons

learned processes and systems. Respondents also identified challenges or barriers to

the sharing of lessons learned as well as areas of improvement. Specifically,

respondents indicated the following.

• Limited Sharing of Lessons Learned Agency-wide

Lessons learned are of little value unless they are collected and shared with others.

To determine the extent that lessons are being collected and shared within NASA, we

asked program and project managers how knowledgeable they were of lessons

generated by their own programs and centers as well as by other centers. Program

and project managers responded that they are very or generally knowledgeable of

lessons generated by their own programs and centers but are not very knowledgeable

of lessons generated by programs at other centers. This fact is supported by survey

results that show managers primarily identify lessons through program or centerbased activities such as project reviews or informal discussions with colleagues.

LLIS was not identified as a primary source for lessons learning.
Page 3 GAO-01-1015R Survey of NASA’s Lessons Learned Process

• LLIS Is Not a Primary Mechanism for Collecting and Sharing Lessons

Learned

Our survey identified several reasons why LLIS is not the primary mechanism used by

program and project managers for collecting and sharing of lessons. One reason is a

lack of awareness about the system. About one-third of program and project

managers responded that they first learned of LLIS from our questionnaire or only

became aware of the system within the last 6 months. This level of awareness also

corresponds with survey results that show limited use of LLIS by program and project

managers within the last 2 years.

• Dissatisfaction with Lessons Learned Processes and Systems

Our survey also determined that more managers are very or generally dissatisfied

with NASA’s lessons learned processes and systems, including LLIS, than are

satisfied. This level of dissatisfaction may stem from the fact that 58 percent of

managers stated that current processes and systems do not allow them to retrieve the

right lessons at the right time. Another contributing factor to this level of

dissatisfaction could be because program and project managers do not find existing

lessons helpful: 53 percent of managers said they found lessons learned useful less

than 25 percent of the time.

• Barriers Exist to Lessons Learning

If lessons learning is to succeed it is important that an organizational culture exist

that fosters the value of sharing knowledge based on others’ experiences. Our survey

determined that many managers do not believe that implementing lessons learned

significantly improves mission effectiveness. One manager stated that “Until we can

adopt a culture that admits frankly to what really worked and didn’t work, I find

many of these tools to be suspect.”

Another critical barrier highlighted by the survey is the lack of time available for

lessons learned. One manager noted that “ It is time consuming to agree on correct

lessons learned for a project, accurately describe the issue...in a way that is helpful

to a project.” Finally, many managers noted in their narrative responses that there is

reluctance to share negative lessons for fear that they might not be deemed to be

good project managers. According to some respondents this appears to stem from a

culture that sees lessons learned as negative, i.e., an admission of failure.

• Suggested Improvements to Lessons Learning

While program and project managers identified a number of ways for improving

lessons learning within NASA, mentoring was their first choice. One manager noted

that the most effective lessons learned process results from the passing down of

lessons from experienced people to those less experienced. Other suggestions for

improvement included “storytelling” or “post mortems.” A manager stated that story

telling would be a quick way to get the message distributed and allow persons to ask

questions. Another notable suggestion was that senior management be accountable
Page 4 GAO-01-1015R Survey of NASA’s Lessons Learned Process

for the infusion or engineering of lessons into a project with some kind of a

matrix/metric to validate that the lessons was actually incorporated.

Managers also suggested a number of improvements regarding LLIS. Suggested

improvements included (1) inclusion of more positive lessons learned, (2) an

improved search capability, (3) addition of a frequently asked question section, and

(4) addition of information to allow one to determine whether a lesson is applicable

to a specific project. Another suggestion was that lessons be distributed once a

month and an alert used to notify managers about important lessons.

AGENCY COMMENTS

NASA generally agreed with our draft report. Our efforts to assess the effectiveness

of the NASA’s lessons learning processes and procedures is continuing and we plan

to issue a final report on the results of our work early in the next calendar year.

NASA’s comments in their entirety appear in enclosure III.

We are sending copies of this letter to the NASA Administrator and interested

congressional committees. The letter is also available on GAO's home page at

http://www.gao.gov. If you or your staff have any questions regarding the briefing or

this letter, please contact me at (202) 512-4841, or John Oppenheim, Assistant

Director at (202) 512-3111. Key contributors to this assignment were James A. Elgas,

Carl M. Ramirez, Jose A. Ramos, Lorene S. Sarne, and Diana Dinkelacker.

Sincerely yours,

Allen Li

Director

Acquisition and Sourcing Management

Enclosures - 3
ENCLOSURE I ENCLOSURE I

Page 5 GAO-01-1015R Survey of NASA’s Lessons Learned Process

Slides from the Briefing on the Results

of NASA’s Lessons Learned Survey

NASA Lessons Learned

Survey Results

August 6, 2001

Briefing to the

House Science

Committee, Subcommittee

on Space and Aeronautics

1

Purpose of Survey

• Obtain views of National Aeronautics and Space

Administration (NASA) program and project managers

regarding:

• Collection, sharing, and application of lessons learned

• Strengths and limitations of NASA's lessons learned

processes, procedures, and systems, including NASA's

Lessons Learned Information System (LLIS)

• Challenges or barriers to the sharing of lessons learned

• Suggested areas of improvement
ENCLOSURE I ENCLOSURE I

Page 6 GAO-01-1015R Survey of NASA’s Lessons Learned Process

2

Survey Scope

• Conducted an online survey of 192 program/project managers

in June/July 2001

• Covered all Enterprise areas:

• Earth Science

• Human Exploration and Development of Space

• Aerospace Technology

• Space Science

• Biological & Physical Research

• Identified 244 programs and/or projects

3

Survey Instrument Development

• Obtained relevant information from interviews with agency officials

involved in program and project management

• Reviewed lessons learned related documents and guidance

• Analyzed literature on lessons learned and knowledge

management
ENCLOSURE I ENCLOSURE I

Page 7 GAO-01-1015R Survey of NASA’s Lessons Learned Process

4

Survey Methodology

• NASA Headquarters and others reviewed and commented on the

proposed contents of the questions

• Conducted a pretest of the survey questions with program and

project managers at Goddard Space Flight Center to gauge clarity,

and length of time of survey administration

• Names/addresses of NASA program and project managers were

provided by NASA at our request

• We believe that our survey results are generalizable to the

population of NASA program and project managers

• Practical difficulties of conducting any survey, however, may

introduce errors into estimates made from surveys

5

NASA’s Definition of Lessons Learned

 “Knowledge or understanding gained by experience. The

experience may be positive, as in a successful test or mission, or

negative, as in a mishap or failure. A lesson must be significant in

that it has real or assumed impact on operations; valid in that it is

factually and technically correct; and applicable in that it identifies a

specific design, process, or decision that reduces or limits the

potential for failures and mishaps, or reinforces a positive result.”
ENCLOSURE I ENCLOSURE I

Page 8 GAO-01-1015R Survey of NASA’s Lessons Learned Process

6

NASA Processes, Procedures, and Systems for

Lessons Learning

• NASA’s Program and Project Management Processes

and Requirements (NPG 7120.5A) states that program

and project managers:

• should review and apply lessons learned from the

past throughout the program/project life cycle, where

appropriate

• shall document and submit any significant lessons

learned in a timely manner

• NASA’s Procedures and Guidelines for mishap

investigating (NPG 8621.1) requires the identification

and submission of lessons learned

7

NASA Has Established Processes, Procedures,

and Systems for Lessons Learning

• NASA’s Lessons Learned Information System (LLIS) is

an on-line, automated data base system, was

established to facilitate the capture and sharing of

lessons learned agency-wide

• NASA disseminates lessons through periodic updates

to policies and guidance

• Lessons learned through mishaps or other operational

events are incorporated into procedure and process

documents
ENCLOSURE I ENCLOSURE I

Page 9 GAO-01-1015R Survey of NASA’s Lessons Learned Process

8

Characteristics of Survey Respondents

• Sixty (60) percent response rate

Project Managers

58%

Program Managers

19%

No Answer

3%

Other

20%

Biological & Physical 

Research

6%

No answer

6%

Earth Science

18%

Space Science

33%

Human Exploration & 

Development of 

Space

9%

Aerospace 

Technology

28%

Respondent Demographics Respondents by Enterprise Area

9

Summary of Survey Results

The survey identified fundamental weaknesses and

improvements in the collection and sharing of lessons

learned:

• Limited sharing of lessons learned agency-wide

• LLIS is not the primary mechanism for collecting and

sharing lessons learned

• There are barriers to lessons learning

• Dissatisfaction with lessons learned processes and

procedures

• Suggested improvements to strengthen lessons learned

processes and procedures
ENCLOSURE I ENCLOSURE I

Page 10 GAO-01-1015R Survey of NASA’s Lessons Learned Process

10

89.6

46.1

51.3

14.8

74.8

7.8 7.8

0 %

20%

40%

60%

80%

100%

O w n P ro g ra m P ro g ra m s an d P ro je cts in

Center

Pro g ram s an d P ro jects in

o th er C en ters

No Answer

N ot Ve ry

K now le dge a ble

Som ewhat

K now le dge a ble

V e ry K now le dga ble

Limited Sharing of Lessons Learned

Agency-wide

• Program/project managers are knowledgeable of lessons

learned in their own programs/centers, but are less

knowledgeable of lessons at other centers

11

Limited Sharing of Lessons Learned

Agency-wide

• No systematic method for alerting program and project

managers of important lessons

• Sharing of lessons mostly occurs during project review and

on an informal basis
ENCLOSURE I ENCLOSURE I

Page 11 GAO-01-1015R Survey of NASA’s Lessons Learned Process

12

LLIS is Not the Primary Mechanism for

Collecting and Sharing Lessons Learned

• Principal sources for identifying lessons learned:

• 1st Source: Systems and Engineering Reviews

• 2nd Source: Program/Project Briefings

• 3rd Source: Informal Discussions with Colleagues

• Twenty-seven (27) percent of program and project managers

were not aware of LLIS before the survey

• Only 17.5 percent of the respondents have applied a lessons

learned identified through LLIS

• In the past 2 years only 23 percent of program and project

managers have submitted a lessons learned to LLIS

13

None

43%

1-2 times

12%

6-9 times

2%

3-5 times

6%

Do not know

30%

10 or more times

3%

No answer

4%

LLIS is Not the Primary Mechanism for

Collecting and Sharing Lessons Learned

• In the last two years 43 percent of program and project

managers have not submitted a lesson to LLIS
ENCLOSURE I ENCLOSURE I

Page 12 GAO-01-1015R Survey of NASA’s Lessons Learned Process

14

LLIS is Not the Primary Mechanism for

Collecting and Sharing Lessons Learned

No answer

4%

10 or m ore tim es

5%

Do not know

29%

 3-5 times

16%

6-9 tim es

7%

1-2 times

17%

None

22%

• LLIS is seldom used by programs and projects to look for

lessons over the past two years

15

Barriers Exist to Lessons Learning

• Thirty-nine (39) percent of program/project managers

believe that implementing lessons learned contributes only

some or little to improving mission effectiveness

• Managers believe there is too little time for individuals to

share lessons learned:

 “It is time consuming to agree on correct lessons learned for a

project, accurately describe the issue and the lesson in a way

that is helpful to a project”

• Managers are reluctant to share failures with others:

 “[There is an] unwillingness...to share information or air dirty laundry. If you made a mistake you might not be deemed to

be a good project manager.”
ENCLOSURE I ENCLOSURE I

Page 13 GAO-01-1015R Survey of NASA’s Lessons Learned Process

16

Dissatisfaction with Lessons Learned

Processes and Systems

• More respondents are very or generally dissatisfied with

NASA’s processes and systems than are satisfied

• Fifty-eight (58) percent stated that processes and systems

infrequently allows them to retrieve the right lessons at the

right time

• Managers indicate that not enough emphasis is being placed

on reporting positive or successful experiences

17

Suggestions Provided by Managers to

Improve Lessons Learning at NASA

• Mentoring:

“I believe ‘lessons learned’ [equals] experience.

Experience is gained on the job, and much of it is

passed along informally through mentoring. This

means there must be a base of experienced people

who must be allowed to mentor the newer

employees!”

• Expanded practice of “storytelling” or “post mortems”
ENCLOSURE I ENCLOSURE I

Page 14 GAO-01-1015R Survey of NASA’s Lessons Learned Process

18

Suggestions Provided by Managers to

Improve Lessons Learning at NASA

• Develop a capability to disseminate or notify users about

important lessons learned:

“Centers should transmit public (Center-wide) emails

containing important or prominent lessons learned on

a regular basis”

19

Suggestions Provided by Managers to

Improve Lessons Learning at NASA

• Better integration of lessons learned into program/project

planning and implementation procedures

• Allow more time for the lessons learned process:

“If it [lessons learned] is important then make it

important. Give it the processes, procedures,

resources, and particularly [the] time, to make it a

positive experience. However, merely making it

another requirement to an already overwhelming work

schedule will doom it to the failure it presently may be

experiencing”
ENCLOSURE I ENCLOSURE I

Page 15 GAO-01-1015R Survey of NASA’s Lessons Learned Process

20

Recognition of the Importance of

Lessons Learned

 “I am encouraged by the focus and attention the lessons learned

process is getting. I really hope that the follow-through is

sustained...rhetoric is cheap and gets the ‘motherhood and apple

pie’ salute. The real action of process development and

improvement [is done] by actually engineering the lessons into the

processes...[but this] is costly and hard. A lot of the work is about

culture change to accept a new model that contrasts to ‘the way

we've always done it.’"

Project Manager
ENCLOSURE II ENCLOSURE II

Page 16 GAO-01-1015R Survey of NASA’s Lessons Learned Process

Survey Objectives, Scope and Methodology

To obtain the views of NASA program and project managers on lessons learned

processes and systems at NASA we conducted a survey in June and July 2001,

using a self-administered electronic questionnaire posted on the World Wide Web.

The survey contained four groups of questions on (1) how NASA personnel

collect, access and use lessons learned; (2) the strengths and weaknesses of

current NASA lessons learned processes, procedures and systems, including the

Lessons Learned Information System; (3) potential challenges or barriers to

sharing Lessons Learned within NASA; and (4) ways to improve the lessons

learned process or system at NASA.

In designing the questionnaire, we interviewed NASA officials, program and

project managers, as well as other government and industry officials who had

insight into lessons learned processes and systems. We also obtained and

reviewed NASA documents and guidance pertinent to lessons learned. To further

guide the development of appropriate questions, we reviewed current literature

on lessons learned and knowledge management. To validate the content and

structure of the questionnaire, we submitted it to officials at NASA Headquarters

and others for review and incorporated their comments where appropriate. To

verify the clarity, length of time of administration, and suitability of the questions,

we also pretested the questionnaire with selected program and project managers

at Goddard Space Flight Center.

NASA officials provided us with a list of the e-mail addresses of 199 personnel

that they indicated were representative of all NASA program and project

managers as of April 2001. On June 5, 2001, we sent e-mail messages to each

person in this survey population notifying them of the survey and asking them to

complete the questionnaire. Subsequently, we discovered one additional NASA

manager who was eligible for our survey, and removed 7 managers who were

ineligible for the survey because they were not project or program managers at

the time of the survey. We also eliminated one duplicate listing, resulting in a final

survey population of 192.

Over the following several weeks until closing the survey on July 13, 2001, we

received a total of 115 useable responses, for an overall response rate of 60

percent. All of the responses had been submitted using the web questionnaire,

although one participant had asked to submit a paper version of the survey, due to

problems in accessing the web survey. Of the 77 non-respondents, 9 provided

partial questionnaire responses but had not indicated that they were finished with

the questionnaire.

While we believe that our survey results are generalizable to the population of

NASA program and project managers as described above, the practical difficulties

of conducting any survey may introduce errors into estimates made from surveys.
ENCLOSURE II ENCLOSURE II

Page 17 GAO-01-1015R Survey of NASA’s Lessons Learned Process

Although we administered questionnaires to all known members of the

population, and thus our results are not subject to sampling error, nonresponse to

the entire survey or individual questions can introduce a similar type of variability

or bias into our results to the extent that those not responding differ from those

who do respond in how they would have answered our survey questions. In

addition, population coverage errors can occur if some members of the

population are excluded from the survey. Measurement errors can arise from

how questions are interpreted by respondents and mistakes made by respondents.

Data processing errors can arise during the handling or analysis of responses. We

took steps in the design, data collection and analysis phases of our survey to

minimize such errors, such as pretesting questionnaires before the survey,

following up with those not reachable at original e-mail addresses or otherwise

not immediately responding, and checking for errors in computer programming

used to analyze survey results. In addition, the distribution of respondents across

NASA Enterprise areas generally reflected the actual distribution of the entire

population, which was consistent with our belief that nonresponse error was not

significant.
ENCLOSURE III ENCLOSURE III

Page 18 GAO-01-1015R Survey of NASA’s Lessons Learned Process

Comments From the National Aeronautics

and Space Administration

(707565)