MARCH 6, 2012

AUDIT REPORT

REPORT NO. IG-12-012 (ASSIGNMENT NO. A-11-010-00)

OFFICE OF AUDITS

REVIEW OF NASA’S LESSONS LEARNED 

INFORMATION SYSTEM 

OFFICE OF INSPECTOR GENERAL

National Aeronautics and 

Space Administration

 
REPORT NO. IG-12-012 

Final report released by:

Paul K. Martin

Inspector General

Acronyms 

APPEL Academy of Program/Project and Engineering Leadership 

FY Fiscal Year

GAO Government Accountability Office 

JPL Jet Propulsion Laboratory 

LLIS Lessons Learned Information System

NEN NASA Engineering Network

NPR NASA Procedural Requirements
MARCH 6, 2012

REPORT NO. IG-12-012 

OVERVIEW

REVIEW OF NASA’S LESSONS LEARNED 

INFORMATION SYSTEM 

The Issue

Successful organizations develop systems to share information from past successes and 

failures as part of their knowledge management practices. NASA defines these “lessons 

learned” as knowledge or understanding gained by experience. This experience may be 

positive, as in a successful test or mission, or negative, as in a mishap or failure. Sharing 

lessons learned can reduce risk, improve efficiency, promote validated processes, and

improve performance in ongoing and future NASA projects. 

Since 1994, NASA’s principal mechanism for collecting and sharing lessons learned 

from Agency programs and projects has been an online, automated database called the 

Lessons Learned Information System (LLIS).1

 The information in LLIS is drawn from 

individuals, directorates, programs, projects, and any supporting organizations and 

personnel across NASA, including engineering, technical, science, operations, 

administrative, procurement, management, safety, maintenance, training, flight and 

ground-based systems, facilities, medical, and other activities. LLIS is one component of 

NASA’s larger knowledge management and sharing system, which includes the online 

NASA Engineering Network (NEN), NASA’s Academy of Program/Project and 

Engineering Leadership (APPEL) training, ASK Magazine, the Masters Forum, and the 

annual Project Management Challenge seminar.2

NASA’s policy on lessons learned requires the collection, validation, assessment, and 

codification of lessons learned submitted by individuals, NASA directorates, programs 

 

1 The public can access LLIS at http://llis.nasa.gov/llis/search/home.jsp (accessed March 5, 2012).

2 NASA launched the NEN website (https://nen.nasa.gov/web/nen/home) in 2005 to bring together the 

Agency’s knowledge management and sharing system components, including LLIS. NEN provides 

NASA personnel a portal to access, create, and share lessons learned; interact with subject-matter experts 

and practitioners; search many NASA repositories of interest; and find tools and information resources. 

NEN’s suite of information retrieval and knowledge sharing tools has the capability of searching for 

lessons learned across the Agency’s multiple repositories, including LLIS. APPEL provides training to 

meet the learning and development objectives of the NASA program and project management and 

engineering communities. ASK Magazine is designed for program/project managers and engineers to 

share expertise and lessons learned with fellow practitioners. In the Masters Forum, participants share 

best practices and lessons learned with NASA employees and contractors. The Project Management 

Challenge seminar is an annual conference that examines current management trends and provides a 

forum for sharing lessons learned. In February 2012, the Office of the Chief Engineer announced that the 

Project Management Challenge conference would be discontinued in favor of other options, such as 

virtual seminars. 
OVERVIEW

ii REPORT NO. IG-12-012 

and projects, and any supporting organizations and personnel.3

The Office of the Chief Engineer has primary responsibility for oversight of NASA’s 

knowledge management and sharing system, including LLIS. A 2002 Government 

Accountability Office (GAO) report identified weaknesses in NASA’s lessons learned 

and knowledge management processes.

 To this end, LLIS is

designed to be searchable and available across the Agency to the broadest extent possible. 

The usefulness and value of LLIS is contingent on managers and engineers routinely 

consulting and submitting information to the system. 

4

The NASA Office of Inspector General initiated this audit to examine the extent to which 

Agency managers use LLIS and how the system fits within NASA’s overall knowledge 

management strategy. Details of our audit’s scope and methodology are in Appendix A.

 Specifically, GAO found that NASA did not 

routinely identify, collect, or share lessons learned by its program and project managers. 

GAO also concluded that NASA lacked strong management commitment to knowledge 

sharing and needed a well-defined business plan for implementing a knowledge 

management and sharing system that encourages and facilitates easier access to 

information. The report contained seven recommendations, including a suggestion to 

enhance LLIS and develop a knowledge management plan to articulate the relationship 

between lessons learned and the Agency’s overall knowledge management plan. NASA 

concurred with the recommendations and subsequently enhanced LLIS to accept 

submissions in multiple formats and improved the ability to search the system through 

the NEN website. However, NASA has yet to formalize a comprehensive knowledge 

management plan that articulates the role LLIS should play in the Agency’s plan for 

knowledge management. 

Results

NASA’s project managers do not routinely use LLIS to search for lessons identified by 

other projects or routinely contribute new information to LLIS. We found NASA’s 

policies regarding the input of lessons learned into LLIS have weakened over 

time; inconsistent policy direction and implementation for the Agency’s overall lessons 

learned program; disparate levels of funding for LLIS activities across NASA Centers; 

and deficient monitoring of critical Center-based LLIS activities. In addition, we found 

the Chief Engineer’s overall strategy for knowledge management, lessons learned, and 

LLIS is not well defined. Consequently, LLIS has been marginalized in favor of other 

NASA knowledge sharing system components and is of diminishing and questionable 

value. 

 

3 NASA Procedural Requirements (NPR) 7120.6, “Lessons Learned Process,” March 22, 2005.

4 GAO, “NASA: Better Mechanisms Needed for Sharing Lessons Learned” (GAO-02-195, January 30, 

2002).
OVERVIEW

REPORT NO. IG-12-012 iii

Managers Are Not Using LLIS

We found that NASA program and project managers rarely consult or contribute to LLIS

even though they are directed to by NASA requirements and guidance. In fact, input to 

LLIS by most Centers has been minimal for several years. Specifically, other than the Jet 

Propulsion Laboratory (JPL), no NASA Center consistently contributed information to 

LLIS for the 6-year period from 2005 through 2010. For example, the Glenn Research 

Center and the Johnson Space Center contributed an average of one lesson per year 

compared to the nearly 12 per year contributed by JPL (see Table 1, page 9). 

We surveyed project managers for 28 of NASA’s 32 science and space flight projects 

launched between January 2005 and May 2011 to examine the extent to which they used 

and contributed to LLIS.5

The project managers we surveyed offered a variety of reasons for not using or 

contributing to LLIS, including their belief that LLIS is outdated, is not user friendly, and 

does not contain information relevant to their projects. They also cited competing 

demands on their time in managing their respective projects. Taken together, the lack of 

consistent input and usage has led to the marginalization of LLIS as a useful tool for 

project managers. 

 We found that these managers did not consistently search 

LLIS for lessons identified by other programs and projects that could potentially reduce 

risk to their projects. Specifically, only 16 of the 28 (57 percent) project managers 

indicated that they used LLIS during the project acquisition life cycle, with the extent of 

usage varying widely by project. We also found that the project managers did not 

routinely contribute to LLIS, with only 12 of the 28 project managers (43 percent) 

contributing lessons learned from their projects to LLIS. 

Policy Requirements Weakened Over Time. Over time NASA has relaxed its internal 

requirements regarding the level and timing of project managers’ use of and contributions 

to LLIS. For example, program and project management policies issued in 2002 and 

2005 required managers to provide lessons learned for input to LLIS “throughout the 

project lifecycle, for example, at major milestones.”6 In contrast, NASA’s current policy, 

in effect since 2007, does not explicitly require the use of LLIS and does not require 

project managers to identify or archive lessons learned until project conclusion or 

closeout.7

Inconsistent Policy Direction and Implementation. NASA published NASA 

Procedural Requirements (NPR) 7120.6, “Lessons Learned Process,” in 2005 to facilitate 

the capture of knowledge from individuals, projects, and programs. The NPR directs the 

 

 

5 The other four projects were excluded because the project manager was not available, they were 

demonstration projects, or another agency was responsible for the project’s development.

6 NPR 7120.5B, “NASA Program and Project Management Processes and Requirements,” November 21, 

2002. NPR 7120.5C, “NASA Program and Project Management Processes and Requirements,” 

March 22, 2005.

7 NPR 7120.5D, “NASA Space Flight Program and Project Management Requirements,” March 6, 2007.
OVERVIEW

iv REPORT NO. IG-12-012 

establishment of Center-level Lessons Learned Committees and describes the multiplestep process of inputting information to LLIS that involves project, Center, and 

Headquarters personnel. However, we found inconsistent implementation of these

requirements across the Centers. For example, our survey of the lessons learned 

representatives at each NASA Center found that 8 of 10 Centers had not fully complied 

with the policy requirements when implementing their lessons learned processes and 4 

Centers did not even have a Center-level Lessons Learned Committee. Further, we found 

that 6 of 10 Centers did not cross-reference lessons learned to their management or

engineering standards, thereby further limiting the effectiveness of lessons learned for the 

Centers’ programs and projects. 

In addition to the formal guidance provided in the NPR, the Chief Engineer and Chief of 

Safety and Mission Assurance issued a letter in February 2009 encouraging active 

participation by NASA senior leaders in institutionalizing and sharing lessons learned 

across the Agency (see Appendix B). In contrast to the formal policy, the letter

encouraged NASA leaders to convene workshops to discuss and capture lessons learned 

immediately after completing individual elements of a project’s work while memories 

were fresh, rather than waiting until the end of the project’s mission. However, 

according to the Chief Engineer the letter did not result in a measureable improvement of 

the Centers’ use of the LLIS process to institutionalize lessons learned. 

Disparate Funding Support. We found disparate funding levels for LLIS activities at 

the Centers. The Chief Engineer provides most of the funding to support each Center’s 

lessons learned process.

8

 Total LLIS funding for fiscal year (FY) 2011 across NASA 

was $782,000, with funding at the Centers ranging from a low of $21,785 at the Ames 

Research Center to a high of $305,095 at the Kennedy Space Center. At 8 of the 10 

Centers, funding is only a fraction of the amount needed to support one full-time 

employee. However, we found little evidence of correlation between funding levels and 

the number of lessons learned that Centers contributed to LLIS. For example, JPL 

received $70,000 between FYs 2008 and 2010, while Glenn received approximately 

$470,000. Nevertheless, over those 3 years, JPL contributed 35 lessons learned to 

Glenn’s 5.9

Lack of Monitoring. Since the issuance of NPR 7120.6 in March 2005, the Chief 

Engineer has completed an Agency-wide assessment of the lessons learned process just 

once, in 2010. We found that assessment to be inadequate because it did not review and 

evaluate whether the Center-level Lessons Learned Committees administered and 

oversaw the lessons learned process or whether the Committees promoted use of lessons 

learned throughout a project’s life cycle. 

 

8 Only the Dryden Flight Research Center and JPL provided supplemental funding to support their LLIS 

process.

9 The Chief Engineer’s Advanced Planning and Analysis Division Manager stated that the funds provided 

to Kennedy also support lessons learned managerial functions at Headquarters and funding to Glenn 

supports Agency-wide knowledge sharing activities in addition to the Center Manager function.
OVERVIEW

REPORT NO. IG-12-012 v 

NASA Lacks a Comprehensive Strategy for Knowledge Management 

As noted above, LLIS is one component of NASA’s knowledge management strategy. 

However, we found that NASA’s strategy and plans for lessons learned and LLIS are not 

well defined. The Chief Engineer acknowledged that the lessons learned process and 

LLIS have not received the attention needed to enable them to function as originally 

designed and that his efforts to energize the NASA workforce in this effort, including the 

joint letter in 2009 and the 2010 assessment, have resulted in little measureable 

improvement in Centers’ use of the process or system. Finally, although he indicated that

the NEN website will be a key element of any future strategic knowledge management 

plan, he conceded that how LLIS will fit into this future strategy has not been 

determined.

Conclusion 

LLIS is underutilized and has been marginalized in favor of other NASA knowledge 

management tools such as Ask Magazine and the annual Project Management Challenge 

seminar. Users told us they found LLIS outdated, not user friendly, and generally 

unhelpful, and the Chief Engineer acknowledged that the system is not operating as 

originally designed. Although we believe that capturing and making available lessons 

learned is an important component of any knowledge management system, we found that, 

as currently structured, LLIS is not an effective tool for doing so. Accordingly, we 

question whether the three quarters of a million dollars NASA spends annually on LLIS 

activities constitutes a prudent investment. 

Management Action

We recommended that the Chief Engineer develop and implement a cohesive strategic 

plan for knowledge management and sharing, particularly with respect to lessons learned. 

As part of this plan, we recommended that he determine if or how LLIS fits into this plan

and revise the applicable Agency policies accordingly, including: 

• NPR 7120.5D – To establish program/project management requirements that 

align with NASA’s strategic vision for institutional knowledge management and 

sharing. 

• NPR 7120.6 – To align Center and program/project management practices and 

improve the collection and dissemination of lessons learned Agency-wide by 

establishing requirements that can be supported by available resources. 

In response to a draft of this report, the Chief Engineer concurred with our 

recommendations and stated that his office, in accordance with a recommendation from 

NASA’s Aerospace Safety Advisory Panel, will assign an Agency Chief Knowledge 

Officer to develop and implement a strategic plan for knowledge management, including 
OVERVIEW

vi REPORT NO. IG-12-012 

LLIS.10

We find the Chief Engineer’s proposed actions responsive to the intent of our 

recommendations and therefore will close the recommendations upon completion and 

verification of the proposed actions. However, because we continue to question the 

viability of LLIS in its current configuration, we urge the Chief Engineer to further 

consider LLIS’s role in NASA’s larger knowledge management system as part of the 

implementation of his proposed corrective actions. Management’s comments are 

reprinted in Appendix C. 

 The Chief Engineer also noted that LLIS is but one tool that NASA uses to 

capture knowledge and that he considers it a worthwhile investment. Finally, he stated 

that his office will consider what changes are needed to NPR 7120.6 and incorporate any 

changes in other engineering and program and project management requirements 

documents. 

 

10 NASA established the Aerospace Safety Advisory Panel in 1968 to evaluate safety performance and 

advise the Agency on ways to improve that performance. 
MARCH 6, 2012

REPORT NO. IG-12-012 

CONTENTS

INTRODUCTION

Background _________________________________________ 1 

Objectives __________________________________________ 7 

RESULTS

The Lessons Learned Information System Is Underutilized ____ 8 

APPENDIX A 

Scope and Methodology_______________________________ 19

Review of Internal Controls ____________________________ 20

Prior Coverage ______________________________________ 21

APPENDIX B 

Lessons Learned/Knowledge Sharing Letter _______________ 22

APPENDIX C 

Management Comments ______________________________ 26

APPENDIX D 

Report Distribution___________________________________ 29

MARCH 6, 2012

REPORT NO. IG-12-012 1 

INTRODUCTION

Background

NASA describes knowledge management as “the methods an organization uses to retain 

and pass along the knowledge produced by its workers, including libraries, archives, 

in-house magazines, and mentoring.” Effective knowledge management activities foster 

an environment in which the right information gets to the right people at the right time in 

ways that measurably improve performance. 

Over the years, NASA has identified three primary goals for its knowledge management

systems and processes: 

1. Sustain NASA’s knowledge across missions and generations; 

2. Help people find, organize, and share NASA’s existing knowledge; and

3. Increase collaboration and facilitate knowledge creation and sharing of lessons 

learned.

Lessons learned preserve institutional knowledge and communicate experiences that can 

potentially reduce risk and improve efficiency and performance. NASA defines a lesson 

learned as knowledge or understanding gained by experience. The experience may be 

positive, as in a successful test or mission, or negative, as in a mishap or failure. These 

lessons may come from a variety of mission areas, including engineering, technical, 

science, operations, procurement, management, safety, and ground-based systems. The 

collection and dissemination of lessons learned is intended to contribute to the continuous 

improvement of program/project management and engineering, as well as to the 

prevention or reoccurrence of known problems.11

NASA policy requires that lessons learned be assessable and searchable across the 

Agency to the broadest extent possible.

 

12

Historically, NASA’s principal mechanism for collecting and sharing lessons learned 

from programs, projects, and missions Agency-wide has been the Lessons Learned 

 To that end, NASA requires the capturing of 

lessons learned, establishment of Lessons Learned Committees at the Center level, and 

infusion of lessons learned into Headquarters and Center documentation and training so 

managers are better able to make informed decisions, avoid repeating costly mistakes, 

and successfully repeat positive achievements. 

 

11 NASA Policy Directive 7120.4D, “NASA Engineering and Program/Project Management,” March 16, 

2010.

12 NASA Procedural Requirements (NPR) 7120.6, “Lessons Learned Process,” March 22, 2005.
INTRODUCTION

2 REPORT NO. IG-12-012 

Information System (LLIS). First established as a paper-based system in 1992 and 

operating as an automated web-based system since 1994, LLIS is intended to facilitate 

the capture and sharing of lessons learned across all NASA Centers.

13

• The NASA Engineering Network (NEN), a website launched in 2005 to bring 

together the Agency’s knowledge management and sharing system components 

(

 The primary 

purpose of LLIS is to provide a searchable repository of information that allows NASA 

managers to learn from past activities. LLIS is part of NASA’s wider knowledge 

management and sharing system, which includes: 

https://nen.nasa.gov/web/nen/home). NEN provides NASA personnel the 

capability to access, create, and share lessons learned; interact with subject matter 

experts and practitioners; search many NASA repositories of interest; and find 

tools and information resources. NEN’s suite of information retrieval and 

knowledge sharing tools has the capability of searching for lessons learned across 

the Agency’s multiple repositories, including LLIS. 

• The Academy of Program/Project and Engineering Leadership (APPEL), provides 

training to meet the learning and development objectives of the NASA program 

and project management and engineering communities. 

• ASK Magazine, designed for program and project managers and engineers to 

share expertise and lessons learned with fellow practitioners. Published four 

times yearly, the magazine includes articles about meeting the technical and 

managerial demands of complex projects, as well as insights into organizational 

knowledge, learning, collaboration, performance measurement and evaluation, 

and scheduling. 

• The Masters Forum, a program designed to share best practices and lessons 

learned. The forums are interactive 3-day events where project managers and 

engineers learn from fellow practitioners who are invited to share their 

experiences on high-profile projects, including the Space Shuttle, International 

Space Station, and Constellation programs. 

• The Project Management Challenge seminar, designed to examine current 

management trends and provide a forum for knowledge sharing and exchanging 

lessons learned on project management issues. Approximately 20 percent of 

NASA managers attend the annual Project Management Challenge seminar.

14

 

13 The public can access LLIS at http://llis.nasa.gov/llis/search/home.jsp (accessed March 5, 2012).

14 In February 2012, the Office of the Chief Engineer announced that the seminar would be discontinued. 
INTRODUCTION

REPORT NO. IG-12-012 3 

Development of NASA’s Lessons Learned Policy

In December 2000, the Chief Engineer issued a report identifying the need to improve 

communication across the Agency, including improving knowledge management 

practices.

15

The following year, the Government Accountability Office (GAO) surveyed all NASA 

program and project managers and found fundamental weaknesses in the collection and 

sharing of lessons learned Agency-wide.

 The report noted several initiatives underway to improve the sharing of 

knowledge but emphasized that more needed to be done to promote the continuous 

capture, dissemination, and use of Agency knowledge. The report stated that the primary 

mechanism for promoting knowledge management should be an improved lessons 

learned tool. 

16 Specifically, GAO found that lessons learned 

were not routinely identified, collected, or shared by program and project managers. In 

addition, many respondents indicated that they were dissatisfied with NASA’s lessons 

learned processes and LLIS. A 2002 GAO report made a number of recommendations to 

strengthen the Agency’s lessons learned processes and systems, including enhancing 

LLIS, implementing a plan that articulates the relationship between lessons learned and 

NASA’s overall knowledge management objectives, and designating an Agency-wide 

lessons learned manager.17

In response to the GAO report, NASA chartered a NASA Knowledge Management 

Team, consisting of 38 employees from Headquarters and the Centers, to review NASA’s 

knowledge management activities. One of the early pilots implemented under the Team 

was a redesign of LLIS to better infuse lessons learned into day-to-day project activities. 

Moreover, in April 2002, the Team issued in draft a “Strategic Plan for Knowledge 

Management,” which addressed GAO’s recommendations to strengthen the Agency’s 

lessons learned processes and systems. The Team’s plan stated that knowledge 

management applications are the key to helping bring the right information to the right 

people at the right time. The focus of the redesign was an attempt to deliver key 

information to project members at the time they are making a critical decision. However, 

this draft strategic plan was not finalized or published. 

 

Current Lessons Learned Policy

In March 2005, NASA Procedural Requirements (NPR) 7120.6 established the 

requirements for the collection, validation, assessment, and codification of lessons 

learned submitted by individuals, NASA Mission Directorates, programs and projects, 

 

15 NASA, “Enhancing Mission Success – A Framework for the Future,” December 21, 2000, 

http://history.nasa.gov/niat.pdf (accessed March 5, 2012).

16 GAO, “Survey of NASA’s Lessons Learned Process” (GAO-01-1015R, September 5, 2001). Available 

at http://www.gao.gov/new.items/d011015r.pdf (accessed March 5, 2012).

17 GAO, “NASA: Better Mechanisms Needed for Sharing Lessons Learned” (GAO-02-195, January 30, 

2002). Available at http://www.gao.gov/new.items/d02195.pdf (accessed March 5, 2012).
INTRODUCTION

4 REPORT NO. IG-12-012 

and any supporting organizations and personnel. The NPR assigns to the Office of the 

Chief Engineer primary oversight of the NASA lessons learned process, to include 

developing and maintaining LLIS. In addition, the NPR requires the establishment of 

Lessons Learned Committees at both the Headquarters and Center level, a Curator who 

uploads approved lessons learned to LLIS, and “closed-loop infusion of lessons learned 

recommendations into Center and HQ [Headquarters] documentation and training.” 

The Center-level Lessons Learned Committees are headed by Center Data Managers 

(Center Managers) and are expected to administer the lessons learned process at each 

Center as well as support the Headquarters-level Lessons Learned Steering Committee. 

The Center-level Committees are the primary mechanism for collecting, assessing, 

validating, and documenting lessons learned into Center-level documentation and 

training. The Committees are responsible for maintaining the Center-specific lessons 

learned process; administering and overseeing the process of transforming a lesson into a 

complete formatted lesson for Headquarters review and input to LLIS; coordinating the 

transfer of lessons learned into the Center’s corrective action system; and promoting the 

use of lessons learned during the life cycle of program or projects at their Centers. 

The Center-level Committees either identify lessons learned on their own or receive them 

from a submitter or from the NEN website. The Committees validate the lesson with a 

subject matter expert if required, develop a recommendation, and refine the knowledge 

captured into a lesson for Headquarters review. Prior to submitting the lesson to 

Headquarters, Center Managers coordinate export control, patent, legal, and public affairs 

clearance of the information. 

The Headquarters-level Steering Committee is chaired by the Headquarters Data 

Manager (HQ Manager) and is composed of a group of Headquarters and Center 

representatives, including the Center Managers, who provide guidance and develop 

policies related to sharing and using NASA’s lessons learned. The Steering Committee is 

expected to facilitate sharing of lessons learned activities across Centers. The HQ 

Manager reviews lessons learned submitted by Center Managers and Headquarters 

personnel and is responsible for processing lessons learned submitted by the Centers; 

coordinating the transfer of the lessons learned recommendations requiring Headquarters 

action to the Headquarters corrective action system; coordinating trend analysis of 

lessons learned; and maintaining lessons learned process metrics. 

The Curator uploads lessons approved by the HQ Manager to LLIS. In addition, the 

Curator is responsible for ensuring LLIS is maintained in accordance with Federal 

records and Agency policy; providing usage metrics information to the Center-level 

Committees and the Steering Committee; providing helpdesk functions for the LLIS 

website; and providing a final quality assurance function on lessons released by the HQ 

Manager.
INTRODUCTION

REPORT NO. IG-12-012 5 

The figure below illustrates how information flows into LLIS at NASA. 

Lessons Learned Input Process

Source: NPR 7120.6.

Additional Policies Governing Lessons Learned 

NASA Policy Directive 7120.4D, “NASA Engineering and Program/Project Management 

Policy,” March 16, 2010, states that it is NASA’s policy to collect, validate, and 

disseminate lessons to improve outcomes and prevent recurrence of known problems. It 

also states that lessons learned will be incorporated into Agency directives, standards, and 
INTRODUCTION

6 REPORT NO. IG-12-012 

rules. Furthermore, it makes the Chief Engineer responsible for facilitating, promoting, 

and implementing the use of lessons learned and charges program and project managers 

with collecting, documenting, and submitting lessons learned in accordance with NPR 

7120.6. 

NPR 7120.5D, “NASA Space Flight Program and Project Management Requirements,” 

March 6, 2007, requires program and project managers to document how they will 

conduct operations using lessons learned and how they plan to capture lessons learned in 

accordance with NPR 7120.6.18

NPR 7123.1A, “NASA Systems Engineering Processes and Requirements,” March 26, 

2007, established Agency-wide requirements for performing, supporting, and evaluating 

systems engineering. The processes described in this document were to build upon and 

apply best practices and lessons learned from NASA, other Government agencies, and 

industry. However, the NPR does not provide instructions regarding how to disposition 

lessons learned and contains no reference to LLIS and the process described in NPR 

7120.6. 

 The policy also states that the project manager and 

project team shall complete their analysis and archiving of lessons learned at the end of 

each project’s mission. However, the NPR contains no description of how managers are 

to implement the lessons learned process. 

In February 2009, the Chief Engineer and the Chief of Safety and Mission Assurance 

issued a letter to all Centers and Mission Directorates describing LLIS enhancements and 

requesting active participation by NASA’s workforce to institutionalize lessons learned 

and share them across the Agency (see Appendix B). This informal policy guidance 

stated that at the conclusion of every significant development, when projects, 

instruments, subsystems, and other elements are completed, team leadership should 

convene a workshop to reflect on and capture lessons learned. The guidance further 

states that the effort to capture lessons learned should not wait until after a launch or 

closeout, but rather should take place immediately after a team has finished its major 

effort while memories are still fresh.

Observations and Recommendations of the Aerospace Safety 

Advisory Panel 

The Aerospace Safety Advisory Panel, established in 1968 to evaluate NASA’s safety 

performance and advise the Agency on ways to improve that performance, has 

consistently emphasized the importance of knowledge sharing and capturing lessons 

learned. In its two most recent annual reports, the Panel highlighted the value of quickly 

capturing lessons learned in a format that is easily accessible, enables storage for 

 

18 NASA Memorandum 7120-81, “NASA Space Flight Program and Project Management Requirements,” 

was issued on September 22, 2009, as an interim directive while NPR 7120.5D was being revised. 

Requirements of NPR 7120.5D discussed in this report were retained in the interim directive.
INTRODUCTION

REPORT NO. IG-12-012 7 

long-term retrievability, and is searchable by personnel inside and outside the Agency.19 

It also pointed out that while individual Center and programmatic efforts were excellent, 

more needed to be done to establish an Agency-wide process or tool to enable easy 

access to information resources. Consequently, in its Annual Report for 2010, the Panel 

recommended that NASA capture and document the lessons learned from the 

Constellation Program.20

Objectives

 In response, NASA documented those lessons learned in a twovolume submission to LLIS. Furthermore, in its Annual Report for 2011, issued on 

January 25, 2012, the Panel recommended that NASA establish a Chief Knowledge 

Officer to integrate the capturing of lessons learned Agency-wide and consider creating 

Chief Knowledge Officer positions at the Centers and within the Mission Directorates. 

NASA is in the process of developing a course of action to address these

recommendations. 

The objective of our audit was to examine NASA’s use of LLIS and how the system fits 

within NASA’s overall knowledge management strategy. We also reviewed internal 

controls related to the objective. See Appendix A for details of the audit’s scope and 

methodology, our review of internal controls, and a list of prior coverage. 

 

19 The Aerospace Safety Advisory Panel annual reports can be found at http://oiir.hq.nasa.gov/asap/

(accessed March 5, 2012).

20 The Constellation Program was to develop the transportation and infrastructure necessary to enable a 

sustained and affordable human program to explore the Moon in preparation for further deep space 

exploration missions. The Program was canceled in February 2010.
RESULTS

8 REPORT NO. IG-12-012 

THE LESSONS LEARNED INFORMATION 

SYSTEM IS UNDERUTILIZED 

We found that NASA project managers do not routinely use LLIS to search for 

lessons identified by other programs and projects or contribute information to LLIS. 

Project managers we surveyed said the system is not user friendly and that the 

information it contains is outdated and generally unhelpful. We also found that 

Agency policy requirements regarding when and how to input information into LLIS 

have been relaxed over time, policy direction has been inconsistent, LLIS-related 

funding was disparate across the Centers, and monitoring of the essential Centerlevel LLIS process was lacking. In addition, we found that the Chief Engineer’s 

strategy for knowledge management, lessons learned, and LLIS is not well defined 

and that his goal for LLIS was not well understood across the Centers and Mission 

Directorates. Consequently, LLIS has been marginalized in favor of other 

knowledge sharing initiatives. 

Managers Are Not Consistently Using or Contributing to LLIS

We found that NASA program and project managers did not use or contribute to LLIS in 

accordance with established requirements and guidance. In fact, LLIS input by most 

Centers has been minimal for the past several years. Specifically, from 2005 through 

2010 only JPL consistently contributed information to LLIS (see Table 1).
RESULTS

REPORT NO. IG-12-012 9 

Table 1. LLIS Input (by Facility and Year)

Facility 2010 2009 2008 2007 2006 2005 Total 

ARC 0 0 0 3 0 34 37 

DFRC 0 0 0 0 1 20 21 

GRC 0 2 3 0 1 0 6 

GSFC 0 1 1 0 0 5 7 

HQ 0 1 1 2 0 0 4 

JPL 10 15 10 12 10 14 71 

JSC 0 1 0 0 5 0 6 

KSC 80a 2 1 1 0 1 85 

LaRC 0 1 0 0 2 8 11 

MSFC 3 0 4 3 1 4 15 

SSC 1 0 0 0 0 1 2 

 Total 94 23 20 21 20 87 265 

a 78 of the 80 lessons learned submitted by Kennedy in 2010 related to the Space Shuttle Program. 

ARC: Ames Research Center

DFRC: Dryden Flight Research Center 

GRC: Glenn Research Center 

GSFC: Goddard Space Flight Center

HQ: NASA Headquarters

JPL: Jet Propulsion Laboratory

JSC: Johnson Space Center 

KSC: Kennedy Space Center 

LaRC: Langley Research Center 

MSFC: Marshall Space Flight Center

SCC: Stennis Space Center 

We surveyed 28 managers of space and science projects launched between January 2005 

and May 2011 to determine the extent to which they used LLIS. We found that many of 

these managers did not routinely search LLIS for lessons that could potentially reduce or 

limit risk to their projects. We also found that many managers did not contribute 

information to the system. 

Contributions to LLIS. The managers we spoke with provided a variety of reasons for 

not contributing information to LLIS. 

• “The lessons learned during the development and launch . . . were applied to the 

next satellite in the series, which is how the project routinely operated. Since 

these were specific items that only applied to . . . satellites developed by the 

project and would not be useful for any other project, they were not entered into 

LLIS.”

• “We did not contribute lessons learned to LLIS during the project life cycle, 

primarily because we delivered unique one-of-a-kind technology and components 
RESULTS

10 REPORT NO. IG-12-012 

for the . . . missions. Rather, the process and maturation efforts for the delivered 

hardware were documented in publications and technical reports available from 

the NASA Scientific and Technical Information System.”

• “Not to the LLIS, but [we] did through participation on the PM [Project 

Management] Challenge, multiple Road to Mission Success presentations, and 

the . . . Masters Forum. The main reason for not entering data directly into the 

NASA system is that it is just not an effective tool.”

In addition, managers were generally unaware of NASA’s policy requirements for 

contributing to LLIS. Of the 28 project managers we surveyed, only 10 were familiar 

with the requirement in NPR 7120.5D to contribute lessons to LLIS.21

The project managers provided varying reasons for their noncompliance with NASA’s 

lessons learned policy, including 

 Moreover, of 

these 10, some said they simply had not complied with the policy. 

• no explicit statement in NPR 7120.5D requiring them to capture lessons learned 

by inputting information into LLIS; 

• NPR 7120.5D requires periodic reviews and additional scrutiny from various 

external review boards that meet the intent of the lessons learned policy; 

• inadequate resources necessary to implement the requirements; and 

• lessons learned reporting is a low priority, time-consuming process. 

One project manager offered the following observation about the roles and 

responsibilities of project managers in regard to contributing information to LLIS:

No, it was not clear what the Project’s specific obligations were beyond ensuring the 

previous lessons learned were taken into account and the general obligation to capture 

them in some way at completion. The primary reason that lessons learned are not 

formally well captured is that during the execution of a flight project, capturing and 

documenting lessons learned is essentially unfunded scope for an effort that is almost 

always cost- and schedule-challenged already. Once a mission is launched, it is 

quickly disbanded both for budgetary reasons, as well as to reassign critical human 

resources to ongoing Projects.

Use of LLIS. Twelve of the project managers we surveyed said they did not use LLIS as 

a project management tool at all. Of the 16 project managers who stated they had used 

LLIS, the extent of use varied widely by project. Moreover, these managers provided a 

range of opinions about the value of LLIS, with only 6 describing LLIS as useful. These 

6 managers reported that they periodically searched LLIS for relevant lessons and 

 

21 The 11 JPL managers we surveyed were focused primarily on JPL’s internal lessons learned process, 

which applies to all JPL projects. 
RESULTS

REPORT NO. IG-12-012 11

incorporated lessons into their project management and engineering practices. The other 

10 managers said LLIS did not contribute to their projects’ success or provide any 

information that assisted in managing their projects. 

Project managers provided a variety of reasons for not using LLIS, including a lack of 

clear policy criteria requiring they do so and their belief that the system is outdated, is not 

user friendly, and does not contain information relevant to their projects. Some managers 

said that most of the information in the system did not apply to their projects because 

their projects were unique or the information was drawn from projects “vastly different” 

from their own. Others reported they found it difficult to find information in the system 

relevant to the specific issues they faced. One project manager stated, “[We] didn’t find 

anything useful, but we were able to say that we had checked this box.” 

Other Factors Contributing to the Marginalization of LLIS 

Changes to NASA’s program and project management policies over time, inconsistent 

policy direction relating to the capture of lessons learned, uneven funding, and a lack of 

monitoring and oversight have contributed to the underutilization and marginalization 

of LLIS. Specifically, the relevant policies have become less specific regarding LLIS

over time; funding provided to most of the Centers by Headquarters for their lessons 

learned processes has been minimal; and the Center-based process for recommending and 

reviewing potential lessons learned has not been adequately monitored. 

Historical Changes to Lessons Learned Requirements. Over time, NASA policy 

regarding when managers should capture and contribute lessons learned to LLIS has 

changed. Specifically, the program and project management policies issued in 2002 and 

2005 required project managers to capture and contribute lessons learned to LLIS 

throughout a project’s life cycle. However, the current policy only requires these actions 

at project closeout. 

2002 Policy. NPR 7120.5B stated that “at each major milestone” program and 

project managers shall report the extent to which they applied lessons learned. It also 

stated that the managers shall contribute to the body of knowledge all significant lessons 

learned at the end of formulation and implementation and as deemed appropriate. The 

policy also required that program and project managers report the extent to which they 

contributed lessons learned. Moreover, each program and project manager was required 

to apply significant lessons learned from past experiences at the beginning and during the 

conduct of each subprocess. 

 2005 Policy. NPR 7120.5C also stated that the project manager shall provide the 

Office of the Chief Engineer with input to LLIS in the form of captured experiences and 

lessons learned by the project team “throughout the project lifecycle, for example, at 

major milestones.” It also required that the program manager evaluate lessons learned 
RESULTS

12 REPORT NO. IG-12-012 

from existing and previously executed programs and projects to identify applicable 

lessons for use in program planning and execution. 

Current Policy. In contrast to earlier policy requirements, NPR 7120.5D, issued

March 6, 2007, provides no requirement that project managers use LLIS throughout a 

project’s life cycle.

22

Inconsistent Policy Direction and Implementation. In contrast to the formal policy 

embodied in NPR 7120.5D, NASA’s Chief Engineer has urged managers to document 

lessons learned throughout a project’s life cycle. In his 2009 joint letter with the Chief of 

Safety and Mission Assurance (see Appendix B) he stated: 

 Instead, it states that the project manager and project team shall 

complete analysis and archiving of lessons learned “during project decommissioning, or 

closeout.” Because most projects are executed over many years and may go through a 

number of management changes, this change diminishes the timeliness, usefulness, and 

number of documented lessons that are likely to be captured in LLIS. For example, if the 

original project manager failed to capture lessons learned from the project’s formulation 

phase, it is unlikely that a new project manager at closeout will identify those lessons, 

much less be able to capture the information for submission to LLIS. 

At the conclusion of every significant development under your leadership – when 

projects, instruments, subsystems, elements completes its work – the team involved 

should convene a sharing workshop to reflect on and capture its lessons learned. This 

shouldn’t wait until after launch or closeout; it should take place immediately after 

the team has finished its major effort, while teams are still assembled and while 

memories are still fresh. 

However, the Chief Engineer acknowledged that this letter did not result in a measureable 

improvement of the Centers’ use of the LLIS process to institutionalize lessons learned. 

In addition, we also found that implementation of lessons learned process requirements 

across the Centers was inconsistent. We surveyed the lessons learned representatives at 

each NASA Center and found that only two, the Glenn Research Center and JPL, have 

fully complied with NASA policy in the implementation of their respective lessons 

learned processes. Specifically, when we examined the respective Centers’ processes 

against the requirements of NPR 7120.6, we found significant deficiencies. Per the NPR, 

Center-level Committees are the primary mechanism for collecting, assessing, validating, 

and documenting lessons into Center-level documentation and training. However, we 

found that 5 of the 10 NASA Centers did not use a Center-level Committee to develop 

potential lessons learned for Headquarters review and input to LLIS; 5 of 10 Centers did

not use a Center-level Committee to coordinate transfer of lessons to the their Center’s 

corrective action system; 6 of 10 Centers did not use a Center-level Committee to 

promote the use of lessons learned throughout program and project life cycles; and 6 of 

10 Centers did not cross-reference lessons to their engineering standards. Collectively, 

 

22 NASA Memorandum 7120-81, “NASA Space Flight Program and Project Management Requirements,” 

was issued on September 22, 2009, as an interim directive while NPR 7120.5D was being revised. 

Requirements of NPR 7120.5D discussed in this report were retained in the interim directive.
RESULTS

REPORT NO. IG-12-012 13

these process deficiencies limit the potential for new lessons learned to improve current 

and future NASA programs and projects. Table 2 shows the process deficiencies by 

Center. 

Table 2. NPR 7120.6 Process Deficiencies by Center

Center

Center Committee 

Does Not

Administer and 

Oversee Lessons 

Learned Process

Center Committee 

Does Not Coordinate 

between LLIS and 

Center’s Corrective 

Action System 

Center Committee 

Does Not Promote 

Use of Lessons 

Learned throughout 

Project’s Life Cycle

Lessons Learned 

and Engineering 

Standards Not 

CrossReferenced

ARC X 

DFRC X X X X 

GRC

GSFC X X X 

JPL

JSC X X 

KSC X 

LaRC X X X X 

MSFC X X X 

SSC X X X X 

ARC: Ames Research Center

DFRC: Dryden Flight Research Center 

GRC: Glenn Research Center 

GSFC: Goddard Space Flight Center

JPL: Jet Propulsion Laboratory

JSC: Johnson Space Center 

KSC: Kennedy Space Center 

LaRC: Langley Research Center 

MSFC: Marshall Space Flight Center

SSC: Stennis Space Center 

Disparate Funding for LLIS Support. The Chief Engineer’s Office provides most of 

the funding to support the LLIS process at each of the Centers.23 Total LLIS funding for 

fiscal year (FY) 2011 was $782,000, with funding per Center ranging from $21,785 at the 

Ames Research Center to $305,095 at the Kennedy Space Center. However, we found 

little correlation between the funding levels and the number of lessons learned 

contributed to LLIS. For example, JPL received a total of $70,000 for FYs 2008 through 

2010 while Glenn received approximately $470,000 for LLIS activities.

24

 

23 Only the Dryden Research Center and JPL provided supplemental funding to support their LLIS process.

 Over that 

3-year period, JPL contributed 35 lessons learned compared to Glenn’s 5 contributions. 

Table 3 shows the funding provided by Headquarters for the LLIS process at each NASA 

Center from FY 2006 through FY 2011. 

24 The Chief Engineer’s Advanced Planning and Analysis Division Manager stated that the funds provided 

to Kennedy also support the HQ Manager function and funding to Glenn supports Agency-wide 

knowledge sharing activities in addition to the Center Manager function.
RESULTS

14 REPORT NO. IG-12-012 

Table 3. LLIS Funding by Center

Center FY 2011 FY 2010 FY 2009 FY 2008 FY 2007 FY 2006 

ARC $ 21,785 $ 38,622 $ 28,066 $ 48,693 $ 54,854 $ 67,153 

GRC 169,493 184,673 156,036 129,510 24,546 24,077 

LaRC 38,780 15,952 34,818 36,342 27,747 23,194 

DFRC 63,000 56,861 24,932 2,550 4,682 15,720 

GSFC 42,331 52,453 18,372 3,784 57,405 357,564 

JPL 34,000 30,000 20,000 20,000 35,000 35,000 

MSFC 34,000 34,000 14,588 3,631 46,854 24,563 

SSC 23,000 16,726 18,731 49,699 38,076 48,306 

JSC 50,516 51,858 51,596 48,843 39,904 1,879 

KSC 305,095 210,349 143,944 145,534 217,000 39,027 

 Total $782,000 $691,494 $511,083 $488,586 $546,068 $636,483 

ARC: Ames Research Center

DFRC: Dryden Flight Research Center 

GRC: Glenn Research Center 

GSFC: Goddard Space Flight Center

JPL: Jet Propulsion Laboratory

JSC: Johnson Space Center 

KSC: Kennedy Space Center 

LaRC: Langley Research Center 

MSFC: Marshall Space Flight Center

SSC: Stennis Space Center 

Lack of Monitoring. The NASA lessons learned process implemented in March 2005 

requires periodic reviews to assess compliance with NPR 7120.6. However, since 2005, 

the Chief Engineer has completed only one such Agency-wide assessment, in 2010, 

which we found inadequate. Specifically, the review failed to assess and evaluate all of 

the essential elements of the lessons learned process such as whether the Center-level

Lessons Learned Committees administered and oversaw the lessons learned process and 

whether the Committees promoted use of lessons learned throughout project’s life cycle 

(see Table 2 on page 13 for essential elements).

In our judgment, the inconsistent policy directives and the disparate amounts provided to 

the Centers to support the LLIS process, combined with the Centers’ inconsistent 

implementation of policy requirements, has contributed to the reduced number of lessons 

learned added to LLIS, undercuts NASA’s commitment to building a robust lessons 

learned system, and reduces the use and significance of LLIS in favor of other 

components of knowledge sharing, such as the annual Project Management Challenge 

seminar and Ask Magazine. 

Lack of a Formal Strategy for Knowledge Management 

NASA’s overall strategy for knowledge management, lessons learned, and how LLIS fits 

in is not well defined. We asked the Chief Engineer about the Agency’s 2002 draft 
RESULTS

REPORT NO. IG-12-012 15

“Strategic Plan for Knowledge Management.” He conceded that the draft plan was never 

formalized. He also acknowledged that the lessons learned process and LLIS have not 

received the attention needed to enable them to function as originally designed and that 

his efforts to energize the NASA workforce in this effort, including the joint letter in 

2009 and the 2010 assessment, have resulted in little measureable improvement in the 

Centers’ use of the process or system. Finally, although he indicated that the NEN 

website will be a key element of any future strategic knowledge management plan, he 

conceded that how LLIS will fit into this overall strategy has not been determined. 

Conclusion

Following the failures of the Mars Climate Orbiter and Mars Polar Lander in the late 

1990s, the Office of the Chief Engineer was tasked with developing a plan for 

implementing the resulting mishap investigation boards’ recommendations. The Office’s 

report, released in 2000, made the following observations relating to lessons learned.25

The continuous capture and application of project knowledge and lessons learned 

must become a core business process within the Agency’s program and project 

management environment. Regular input into NASA’s knowledge bases, such as the 

lessons learned database, should be emphasized. Programs and projects should 

implement a “document-as-you-go” philosophy, promoting continuous knowledge 

capture for the benefit of current and future missions. More importantly, program and 

project managers must regularly utilize the knowledge management tools to apply 

previous lessons learned to their own projects. The Agency can provide help for 

individuals to understand, learn from, and apply the lessons of others to their own 

work as part of a daily routine.

As of January 2012, the Agency has not met those goals. In fact, NASA’s Aerospace 

Safety Advisory Panel recently stated in its 2011 Annual Report that in spite of excellent 

examples of individual and specific programmatic efforts to facilitate knowledge sharing, 

these efforts do not ensure the identification and capture of critical knowledge or provide 

for an Agency-wide single process or tool for locating and accessing all information 

resources. 

Specifically, we found that LLIS is underutilized and has been marginalized in favor of 

other knowledge management tools such as Ask Magazine and the annual Project 

Management Challenge seminar. Users told us they found LLIS outdated, not user 

friendly, and generally unhelpful, and the Chief Engineer acknowledged that the system 

is not operating as originally designed. Although we believe that capturing and making 

available lessons learned is an important component of any knowledge management 

system, we found that, as currently structured, LLIS is not an effective tool for doing so. 

Consequently, we question whether the three quarters of a million dollars NASA spends 

annually on LLIS activities constitutes a prudent investment. 

 

25 NASA, “Enhancing Mission Success – A Framework for the Future,” December 21, 2000, 

http://history.nasa.gov/niat.pdf (accessed March 5, 2012).
RESULTS

16 REPORT NO. IG-12-012 

Recommendations, Management’s Response, and Evaluation of 

Management’s Response

Given the importance of knowledge management and sharing and to align the mechanisms 

used to capture and share lessons learned with available resources, we made the following 

recommendations. 

Recommendation 1. We recommended that the Chief Engineer develop and implement a 

comprehensive strategy for institutional knowledge management that includes collection and 

dissemination of lessons learned. 

Management’s Response. The Chief Engineer concurred, stating that in accordance 

with a recommendation from the Aerospace Safety Advisory Panel, a staff member from 

the Office of Chief Engineer will be assigned the role of Agency Chief Knowledge 

Officer and will be responsible for developing and implementing a comprehensive 

strategy for institutional knowledge management that will include collection and 

dissemination of lessons learned. The anticipated completion date is March 2013. 

Evaluation of Management’s Response. Management’s proposed actions are 

responsive; therefore, the recommendation is resolved and will be closed upon 

completion and verification of the proposed actions. 

Recommendation 2. We recommended that the Chief Engineer determine if LLIS is worth 

the cost of continued operation, support, and sustainment. 

Management’s Response. The Chief Engineer concurred, stating that although it is 

difficult to quantify the actual benefits of capturing and broadly sharing lessons learned, 

NASA recognizes the value of the process. He further stated that based on monthly LLIS 

utilization metrics, he believes the $750,000 his office spends on LLIS activities is a 

worthwhile investment. 

Evaluation of Management’s Response. We consider management’s actions 

responsive to the intent of the recommendation, and the recommendation is therefore 

resolved. Nevertheless, in light of our findings regarding the utilization level of LLIS by 

project managers, we remain skeptical about the value of continuing to invest in LLIS in 

the absence of significant actions to improve the current system. Accordingly, we urge 

the Chief Engineer to further consider LLIS’s role in NASA’s larger knowledge 

management system as part of the implementation of the Agency’s proposed corrective 

actions to our other recommendations. We will keep this recommendation open until the 

Chief Engineer and the Chief Knowledge Officer have had the opportunity to make this 

assessment and implement corrective actions. 

Recommendation 3. We recommended that the Chief Engineer revise NPR 7120.5D to 

establish program and project management requirements that align with NASA’s strategic 

vision for institutional knowledge management and sharing. 
RESULTS

REPORT NO. IG-12-012 17

Management’s Response. The Chief Engineer concurred, stating that when revisions to 

NPR 7120.6 are complete his office will consider whether changes are required in other 

engineering and program and project management requirements documents. 

Evaluation of Management’s Response. Management’s proposed actions are 

responsive to the intent of the recommendation; therefore, the recommendation is 

resolved and will be closed upon completion and verification of the proposed actions. 

Recommendation 4. We recommended that the Chief Engineer revise NPR 7120.6 to align 

Center and program/project management practices and improve the collection and 

dissemination of lessons learned Agency-wide by establishing requirements that can be 

supported by available resources.

Management’s Response. The Chief Engineer concurred, stating that his office hosted a 

Knowledge Services Retreat on February 26–29, 2012, with a focus on developing a 

strategy to change NPR 7120.6 by March 2013. 

Evaluation of Management’s Response. Management’s proposed actions are 

responsive; therefore, the recommendation is resolved and will be closed upon 

completion and verification of the proposed actions. 

APPENDIXES

REPORT NO. IG-12-012 19

APPENDIX A 

Scope and Methodology

We performed this audit from March 2011 through January 2012 in accordance with 

generally accepted government auditing standards. Those standards require that we plan 

and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable 

basis for our findings and conclusions based on our audit objectives. We believe that the 

evidence obtained provides a reasonable basis for our findings and conclusions based on 

our audit objectives. 

We reviewed lessons learned requirements contained in several NASA policy documents, 

including the following: 

• NPR 7120.5D, “NASA Space Flight Program and Project Management 

Requirements,” March 6, 2007, and NASA Memorandum 7120-81, “NASA 

Space Flight Program and Project Management Requirements,” September 22, 

2009, which was issued as an interim directive while NPR 7120.5D was being 

revised.

• NPR 7120.6, “Lessons Learned Process,” March 22, 2005. 

• NPR 7120.7, “NASA Information Technology and Institutional Infrastructure 

Program and Project Management Requirements,” November 3, 2008. 

• NPR 7120.8, “NASA Research and Technology Program and Project 

Management Requirements (w/change 1 dated 11/24/10).” 

• NPR 7123.1A, “NASA Systems Engineering Processes and Requirements

w/Change 1 (11/04/09).” 

• NPR 7150.2A, “NASA Software Engineering Requirements,” November 19, 

2009. 

• NPR 8621.1, “NASA Procedural Requirements for Mishap and Close Call 

Reporting, Investigating, and Recordkeeping w/Change 5 (03/15/2010).” 

To determine whether NASA had established processes for oversight, support, and 

facilitation of LLIS and whether those processes were functioning in accordance with the 

applicable NASA policies, we interviewed NASA and contractor personnel responsible 

for the collection, validation, assessment, and codification of lessons learned submitted 

by individuals, Mission Directorates, and programs and projects. Additionally, we 

surveyed the lessons learned process representatives at each Center for compliance with 
APPENDIX A 

20 REPORT NO. IG-12-012 

NPR 7120.6. Further, we reviewed all lessons inputted to LLIS during the period of 

review to determine the timeliness of the Headquarters review process.

To determine use of and contributions to LLIS, we surveyed all 32 NASA flight projects 

launched from January 2005 to May 2011. We received and reviewed responses from 28 

flight project managers for 26 Science Mission Directorate projects and 2 Space 

Operations Mission Directorate projects. We did not receive responses from 

four projects: two were small-scale technology demonstration projects; the responsibility 

for the development of one project was with another agency; and the project manager for 

the fourth project was no longer with NASA. 

To determine whether NASA had taken appropriate actions in response to GAO’s 2002 

report, we interviewed personnel responsible for addressing GAO’s recommendations 

who were still with the Agency and reviewed source documents to verify that corrective 

actions were taken.

To determine how NASA’s system and process compare with other significant entities’ 

approaches to obtaining, documenting, and using lessons, we contacted and interviewed 

representatives of several entities from the public and private sectors. 

Use of Computer-Processed Data. We obtained computer-processed data from LLIS 

during the performance of this audit. We verified data obtained with corroborating 

testimony from NASA personnel and source documentation and found no reason to doubt 

the reliability and validity of LLIS data.

Review of Internal Controls 

We evaluated the Chief Engineer’s internal monitoring controls for compliance with the 

NPR 7120.6 process. We interviewed Government personnel at NASA Headquarters 

with oversight responsibilities specified in the NASA policy. We interviewed 

Government and contractor personnel located at the Centers with functional 

responsibilities defined in NASA policy. We conducted a compliance survey focused on 

the implementation of the lessons learned process at each Center. In our opinion, the 

combination of the Steering Committee meetings, NEN-generated process statistics, and 

“OCE [Office of the Chief Engineer] Requirement Compliance Surveys of Centers” 

appear collectively to be an adequate process-monitoring control for Headquarters and 

the Centers. 
APPENDIX A 

REPORT NO. IG-12-012 21

Prior Coverage

No reports concerning LLIS have been issued by the NASA Office of Inspector General 

or GAO during the last 5 years. The following GAO reports, however, are of particular 

relevance to the subject of this report. Unrestricted reports can be accessed over the 

Internet http://www.gao.gov. 

“NASA: Better Mechanisms Needed for Sharing Lessons Learned” (GAO-02-195, 

January 30, 2002) 

“Survey of NASA’s Lessons Learned Process” (GAO-01-1015R, September 5, 2001) 
APPENDIX B

22 REPORT NO. IG-12-012 

LESSONS LEARNED/KNOWLEDGE 

SHARING LETTER
APPENDIX B

REPORT NO. IG-12-012 23
APPENDIX B

24 REPORT NO. IG-12-012 
APPENDIX B

REPORT NO. IG-12-012 25
APPENDIX C 

26 REPORT NO. IG-12-012 

MANAGEMENT COMMENTS
APPENDIX C 

REPORT NO. IG-12-012 27
APPENDIX C 

28 REPORT NO. IG-12-012 
APPENDIX D 

REPORT NO. IG-12-012 29

REPORT DISTRIBUTION

National Aeronautics and Space Administration

Administrator

Deputy Administrator

Chief of Staff

Chief Engineer 

NASA Advisory Council’s Audit, Finance, and Analysis Committee 

NASA’s Aerospace Safety Advisory Panel 

Director, Ames Research Center

Director, Dryden Flight Research Center

Director, Glenn Research Center

Director, Goddard Space Flight Center 

Director, Jet Propulsion Laboratory 

Director, Johnson Space Center 

Director, Kennedy Space Center

Director, Langley Research Center

Director, Marshall Space Flight Center

Director, Stennis Space Center

Non-NASA Organizations and Individuals

Office of Management and Budget

Deputy Associate Director, Energy and Science Division

Branch Chief, Science and Space Programs Branch

Government Accountability Office 

Director, NASA Financial Management, Office of Financial Management and 

Assurance 

Director, NASA Issues, Office of Acquisition and Sourcing Management 

Congressional Committees and Subcommittees, Chairman and 

Ranking Member

Senate Committee on Appropriations 

Subcommittee on Commerce, Justice, Science, and Related Agencies

Senate Committee on Commerce, Science, and Transportation

Subcommittee on Science and Space

Senate Committee on Homeland Security and Governmental Affairs
APPENDIX D 

30 REPORT NO. IG-12-012 

Congressional Committees and Subcommittees, Chairman and 

Ranking Member (continued)

House Committee on Appropriations 

Subcommittee on Commerce, Justice, Science, and Related Agencies

House Committee on Oversight and Government Reform 

Subcommittee on Government Organization, Efficiency, and Financial Management

House Committee on Science, Space, and Technology 

Subcommittee on Investigations and Oversight 

Subcommittee on Space and Aeronautics
REPORT NO. IG-12-012 31

Major Contributors to the Report: 

Raymond Tolomeo, Director, Science and Aeronautics Research Directorate

Tony Lawson, Project Manager

Scott Collins, Senior Auditor 

Greg Lokey, Management Analyst

Keren West, Management Analyst
MARCH 6, 2012

REPORT No. IG-12-012 

OFFICE OF AUDITS

OFFICE OF INSPECTOR GENERAL

ADDITIONAL COPIES

Visit http://oig.nasa.gov/audits/reports/FY12/ to obtain additional copies of this report, or contact the 

Assistant Inspector General for Audits at 202-358-1232. 

COMMENTS ON THIS REPORT

In order to help us improve the quality of our products, if you wish to comment on the quality or 

usefulness of this report, please send your comments to Mr. Laurence Hawkins, Audit Operations and 

Quality Assurance Director, at Laurence.B.Hawkins@nasa.gov or call 202-358-1543. 

SUGGESTIONS FOR FUTURE AUDITS

To suggest ideas for or to request future audits, contact the Assistant Inspector General for Audits. 

Ideas and requests can also be mailed to:

Assistant Inspector General for Audits 

NASA Headquarters

Washington, DC 20546-0001 

NASA HOTLINE

To report fraud, waste, abuse, or mismanagement, contact the NASA OIG Hotline at 800-424-9183 or 

800-535-8134 (TDD). You may also write to the NASA Inspector General, P.O. Box 23089, L’Enfant 

Plaza Station, Washington, DC 20026, or use http://oig.nasa.gov/hotline.html#form. The identity of 

each writer and caller can be kept confidential, upon request, to the extent permitted by law.