NASA: Implementing a Knowledge-Based Acquisition Framework Could Lead to Better Investment Decisions and Project Outcomes
Published: 2005-12-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Aeronautics and Space Administration (NASA) plans to spend over $100 billion on capabilities and technologies to achieve the initial goals of the President's 2004 Vision for Space Exploration. In the past, NASA has had difficulty meeting cost, schedule, and performance objectives for some of its projects because it failed to adequately define project requirements and quantify resources. NASA will be further challenged by a constrained federal budget and a shrinking experienced NASA workforce. To help face these challenges and manage projects with greater efficiency and accountability, NASA recently updated its program and project management policy and is developing an agencywide systems engineering policy. GAO has issued a series of reports on the importance of obtaining critical information and knowledge at key junctures in major system acquisitions to help meet cost and schedule objectives. This report (1) evaluates whether NASA's policy supports a knowledge-based acquisition approach and (2) describes how NASA centers are implementing the agency's acquisition policies and guidance.
section_title: What GAO Found
paragraphs: While NASA's revised policy for developing flight systems and ground support projects incorporates some of the best practices used by successful developers, it lacks certain key criteria and major decision reviews that support a knowledge-based acquisition framework. For example, NASA's policy requires projects to conduct a major decision review before moving from formulation to implementation. Further, before moving from formulation to implementation, projects must validate requirements and develop realistic cost and schedule estimates, human capital plans, a preliminary design, and a technology plan--key elements for matching needs to resources. However, NASA's policies do not require projects to demonstrate technologies at high levels of maturity before program start. By not establishing a minimum threshold for technology maturity, NASA increases the risk that design changes will be required later in development, when such changes are typically more costly to make. In addition, although NASA's policy does require project managers to establish a continuum of technical and management reviews, it does not specify what these reviews should be, nor does it require major decision reviews at other key points in a product's development. Acquiring knowledge at key junctures will become increasingly important as NASA proceeds to implement elements of the Vision. Without a major decision review at key milestones to ensure that the appropriate level of knowledge has been achieved to proceed to the next phase, the risk of cost and schedule overruns, as well as performance shortfalls, increases. NASA centers have varying approaches for implementing the agency's policies and guidance. Some centers have established product development criteria that are similar to the criteria used in a knowledge-based acquisition, while other centers have not. As a result, each center reports a different level and type of knowledge about a project at key decision points. Centers also rely on project managers and systems engineers to employ good project management and systems engineering practices. However, given the loss of experienced project managers and the decline of in-house systems engineering and technical capabilities, that reliance could be problematic. These situations make it difficult for decision makers to evaluate projects on the same basis and make sound investment decisions and tradeoffs based on those evaluations. A standardized, knowledge-based approach would prepare NASA to face competing budgetary priorities and better position the agency to make difficult decisions regarding the investment in and termination of projects.

section_title: Letter
paragraphs: The National Aeronautics and Space Administration (NASA) plans to  spend over $100 billion to develop new capabilities and technologies  critical to supporting the initial goals outlined in the President’s   2004 Vision for Space Exploration (see fig. 1 for a conceptual drawing of  NASA’s proposed crew vehicle). The Vision, which NASA has  characterized as bold, includes plans to explore the moon, Mars, and  beyond. Despite many successes, such as landing the Pathfinder and  Exploration Rovers on Mars, NASA has had difficulty carrying out a  number of other missions, such as the X-33, a technology demonstrator for  future reusable launch vehicles, because the agency was overly optimistic  in what could be achieved within available resources. NASA’s failure to  define requirements adequately and quantify the resources needed to meet  those requirements resulted in some projects costing more, taking longer,  and achieving less than originally planned. In addition to these project  management challenges, a constrained federal budget and a shrinking  experienced project manager and technical workforce, well-versed in  systems engineering and architecture design, will present further  challenges to NASA in the years ahead.
To meet these challenges, NASA recently updated its program and project  management policy and is developing an agencywide systems engineering  policy. NASA expects that these new polices will help the agency manage  its projects with greater efficiency, responsibility, and accountability. We  have issued a series of reports on the importance of obtaining critical  information and knowledge at key junctures in major system acquisitions  before additional investments are made. This work has shown that  following a knowledge-based approach helps developers meet cost and  schedule objectives in developing new and more sophisticated products— the kinds of results that NASA seeks.
Given the major endeavor that NASA is about to undertake, the agency’s  history of cost and schedule overruns and technical problems, and the  significant challenges the agency currently faces and will likely face, you  asked us to (1) evaluate whether NASA’s policies support an acquisition  approach consistent with best practices identified in GAO’s work on  system acquisitions and (2) describe how NASA-wide acquisition policies  are implemented at the various NASA centers.
To conduct our work, we reviewed and analyzed NASA-wide program and  project management policies as they relate to flight systems and ground  support projects and systems engineering guidance and compared the  policies and guidance with criteria contained in GAO’s best practices work  on systems acquisition, including space systems. We also interviewed  NASA headquarters officials from the Office of the Chief Engineer  responsible for the policy and guidance. In addition, we reviewed NASA  center-specific program and project management policies and systems  engineering policies, as they relate to flight systems and ground support  projects, and interviewed officials responsible for implementing those  policies. We focused primarily on Goddard Space Flight Center (GSFC),  the Jet Propulsion Laboratory (JPL), Johnson Space Center (JSC), and  Marshall Space Flight Center (MSFC), which manage the majority of  NASA’s flight systems and ground support projects. Complete details of  our scope and methodology can be found in appendix I. We performed our  work from May 2005 to November 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: Over the past decade, NASA has experienced significant problems with  several of its projects, which GAO and others have reported on (see table 1  for some examples of such problems). In addition, in 2002 we reported on  several sources of failures in NASA programs, including underestimating  complexity and technology maturity, and inadequate review and systems  engineering processes. Further, we reported that the sources of these  problems were not new and that NASA failed to consistently apply lessons  previously learned. The failures identified in our 2002 report were in part  the result of the “faster, better, cheaper” approach to managing its major  acquisitions, which NASA adopted in the 1990s.
These problems, along with others, highlighted the need for the agency to  reevaluate its approach to cost and schedule estimating, risk assessments,  technology development, project reviews, and systems engineering. In  1998, NASA adopted a new program and project management policy,  which was revised in 2002. The policy provided significant flexibility,  including allowing tailoring and projects to opt out of requirements at the  discretion of the project manager. In March 2005—following a series of  internal and external assessments of NASA that showed that the agency  faced significant problems with project management—NASA again revised  its program and project management policy, which includes policy  requirements for the development of flight systems and ground support  projects. According to NASA officials, further changes to the policy are  anticipated in light of new agency leadership.
The March 2005 policy document, NASA Procedural Requirement (NPR)  7120.5C, differs from previous versions of the policy in that it delineates  requirements based on four NASA investment areas: Basic and Applied  Research, Advanced Technology Development, Flight Systems and Ground   Support, and Institutional Infrastructure. The policy also reinstitutes a life  cycle phased approach to product development and institutes a project  categorization scheme, based on project cost and priority, which denotes  the oversight authorities and the level of detail that is needed to support  project planning documents. See figure 2 below.
NASA has also attempted to address some of its cost-estimating  weaknesses by instituting a cost analysis data requirement (CADRe) and  establishing thresholds for the use of earned value management (EVM).  Another major change to the policy is the establishment of the  Independent Technical Authority (ITA). According to the policy, the  purpose of ITA is to establish sound technical requirements and decisions  for safe and reliable system operations separate from the project  management and reporting chain. Finally, agency officials told us that  while the requirements in previous versions of the policy were easy to  tailor, projects now must document compliance with the requirements in a  “compliance matrix” and must request and have approved any deviations  and/or waivers to requirements.
Although NPR 7120.5C contains some mandatory requirements with  regard to systems engineering, according to agency officials, NASA has  never had an agencywide systems engineering policy to inform the  development of flight systems and ground support projects. Since 1995, in  the absence of a policy on systems engineering, project managers and  systems engineers have relied on information contained in NASA’s  Systems Engineering Handbook to guide their systems engineering  approach on projects. Project managers and systems engineers, however,  are not required to follow the handbook. Recognizing the need for a more  structured and rigorous approach to systems engineering agencywide,  NASA’s Office of the Chief Engineer is currently leading the effort to  develop a systems engineering policy. The policy is in draft form.
subsections: section_title: Knowledge-Based Acquisition Framework
paragraphs: Over the last several years, we have undertaken a body of work on how  leading developers in industry and government use a knowledge-based  approach to deliver high quality products on time and within budget. A  knowledge-based approach to product development efforts enables  developers to be reasonably certain, at critical junctures or “knowledge  points” in the acquisition life cycle, that their products are more likely to  meet established cost, schedule, and performance baselines and, therefore  provides them with information needed to make sound investment  decisions. See figure 3 for a depiction of a knowledge-based acquisition  life cycle.
Knowledge point 1 (KP1): Resources and needs match. Knowledge  point 1 occurs when a sound business case is made for the product— that is, a match is made between the customer’s requirements and the  product developer’s available resources in terms of knowledge, time,  workforce, and money. To determine available resources, successful  developers rely on current and valid information from predecessor  projects, new technologies that have demonstrated a high level of  maturity, system engineering data, and experienced people. Successful  developers also communicate extensively with customers to match  their wants and needs with available resources and with the developers  ability to manufacture an appropriate product.
Knowledge point 2 (KP2): Product design is stable. Knowledge   point 2 occurs when a developer determines that a product’s design is  stable—that is, it will meet customer requirements and cost and  schedule targets. A best practice is to achieve design stability at the  product’s critical design review (CDR), usually held midway through  development. Completion of at least 90 percent of engineering  drawings at the CDR provides tangible evidence to decision makers  that the design is stable.
Knowledge point 3 (KP3): Production processes are mature. This  level of knowledge is achieved when it has been demonstrated that the  product can be manufactured within cost, schedule, and quality targets.
A best practice is to ensure that all key manufacturing processes are in  statistical control— that is, they are repeatable, sustainable, and  capable of consistently producing parts within the product’s quality  tolerances and standards—at the start of production. It is important  that the product’s reliability be demonstrated before production begins,  as investments can increase significantly if defective parts need to be  repaired or reworked.
If the knowledge attained at each juncture does not confirm the business  case on which the initial investment was originally justified, the project  should not go forward and additional resources should not be committed.  Product development efforts that do not follow a knowledge-based  approach can be frequently characterized by poor cost, schedule, and  performance outcomes.
section_title: NASA’s Revised Policy Does Not Fully Support a Knowledge- Based Approach to Acquisitions
paragraphs: NASA’s revised acquisition policy for developing flight and ground support  systems incorporates some of the elements of a knowledge-based  acquisition approach. However, it lacks specific key criteria and decision  reviews necessary to fully support such an approach. NASA’s policy  defines a phased life cycle approach and requires a major decision review  to move from project formulation to implementation. The policy  requirements for this review address many of the key elements necessary  to match needs to resources, such as requirements to establish project  baselines. However, the policy does not require that projects demonstrate  technologies at high levels of maturity before launching a project and  investing a large amount of resources. In addition, NASA’s policy does not  require any further major decision reviews following the formulation  phase of the project. Major decision reviews in the implementation phase  based on specific evaluation criteria at the final design and fabrication,  assembly, and testing milestones—critical decision points in any product  development—could help NASA ensure that sufficient knowledge has  been gained to warrant moving forward in the development process.
subsections: section_title: NASA’s Policy Establishes a Phased Life Cycle
paragraphs: The life cycle for all NASA projects is divided into two major phases— formulation and implementation. Because flight systems and ground  support projects are particularly complex and have long life cycles, NASA  has further divided the formulation and implementation phases for these  projects to allow managers to assess management and engineering  progress (see fig. 4).
Flight systems and ground support projects must successfully complete  two major decision reviews: a Preliminary-Non Advocate Review   (Pre-NAR) between Phases A and B and a Non-Advocate Review (NAR)  between Phases B and C. At these reviews, the Governing Program  Management Committee (GPMC) evaluates the cost, schedule, safety, and  technical content of the project to ensure that the project is meeting  commitments specified in key management documents. Following each  of these reviews, the GPMC recommends to the appropriate decision  authority whether the project should be authorized to proceed.
As with KP1 in a knowledge-based acquisition life cycle, the NAR marks  the official project approval point in the life cycle. After approval at the  NAR, projects are included as part of implementation reviews of their  parent program. These implementation reviews are conducted biennially  and are not tied to any design or production milestones. After entering the  implementation phase, the GPMC is notified if cost or schedule  performance exceeds the baselines established at the NAR by 10 percent  or when key performance criteria are not met. Exceeding the NAR  baselines can result in the GPMC conducting a termination review to  determine whether or not to continue the project.
section_title: NASA’s Policy Lacks Requirements for Maturing Technologies
paragraphs: To help ensure project requirements do not outstrip resources, leading  developers obtain the right knowledge about a new product’s technology,  design, and production at the right time. NASA’s policy emphasizes many  elements needed at the NAR (or KP1) to match needs to resources, such as  validating requirements, developing realistic cost and schedule estimates  and human capital plans, and establishing a preliminary design. The policy  does not, however, require projects to demonstrate technologies at high  levels of maturity before launching a project. Table 2 compares KP1  criteria and NASA’s policy criteria.
While NASA requires projects to develop plans that describe how  technologies will be matured and to provide alternative development  strategies for technologies that do not mature as expected, it does not  establish a minimum threshold for technology maturity. Consequently,  projects can enter the implementation phase with immature technologies  and embark on a risky path of having to build technology, design, and  production knowledge concurrently. Our best practices work has shown  that maturing technologies during the preliminary design phase and before  entering product development is a key element of matching needs to  resources and that there is a direct relationship between the maturity of  technologies and the risk of cost and schedule growth. Allowing  technology development to carry over into the product development phase  increases the risk that significant problems will be discovered late in  development. Addressing such problems at this stage may require  extensive retrofitting and redesign as well as retesting, which can  jeopardize performance and result in more time and money to fix. This  approach also makes it more difficult for projects to demonstrate the same  level of design stability in later phases of implementation since technology  and design activities will be done concurrently.
Technology readiness levels (TRL)—a concept developed by NASA—can  be used to gauge the maturity of individual technologies. The higher the  TRL, the more the technology has been proven and the lower the risk of  performance problems and cost and schedule overruns (see fig. 5).   TRL 7—demonstrating a technology as a fully integrated prototype in an  operational environment—is the level of maturity preferred by product  developers to minimize risks when entering product development.
Successful developers will not commit to undertaking product  development, and more importantly investing resources, unless they have  high confidence that they have achieved a match between what the  customer wants and what the project can deliver. Technologies that are  not mature continue to be developed in an environment that is focused  solely on technology development. Once matured, these technologies can  be transitioned to projects. This puts developers in a better position to  succeed because they can focus on integrating the technologies and  testing and proving the product design.
section_title: NASA’s Policy Does Not Support Informed Design and Production Decisions
paragraphs: Our prior work has shown that successful developers establish specific  criteria to ensure that requisite knowledge has been attained before  moving forward from final design into the latter stages of development.  Before making significant increases in investments to fabricate, assemble,  and test the product, these developers conduct a decision review to  determine if the design is stable and performs as expected and the project  is ready to enter the next phase. To make this determination and reach  KP2, successful developers use specific, knowledge-based standards and  criteria. (See app. II for more information on the specific knowledge-based  standards and criteria successful developers use to judge readiness to  proceed beyond detailed design activities.)
Successful developers also demand proof that manufacturing processes  are in control and product reliability goals are attained before committing  to production. To determine whether they have achieved this knowledge  point, KP3, successful developers conduct another mandatory decision  review in which they use specific, knowledge-based standards and criteria  to determine if the product can be produced within cost, schedule, and  quality targets. (See app. II for more information on specific knowledge- based standards and criteria used by successful developers to judge  readiness to enter into production.)
Contrary to these best practices, the NAR at the end of the preliminary  design phase—KP1—is the last major decision review in the NASA project  life cycle. (See fig. 6.) Although NPR 7120.5C requires that projects  document in the project plan a continuum of technical and management  reviews, such as a PDR and CDR, it does not require any specific reviews.  In addition, NASA’s March 2005 policy does not require a NAR-type  decision review to ensure a project has obtained the knowledge needed to  proceed beyond the final design phase into the fabrication, assembly, and  test phase, which serves as both the demonstration and production phase  of the NASA life cycle. According to NASA officials, projects conduct a  CDR at the end of the final design phase to ensure adequate information is  available about product design and producibility before entering the  fabrication, assembly, and test phase. The CDR, however, is a technical  review—not a major decision review like the NAR. Furthermore, the  policy does not establish criteria as to what constitutes successful  completion of a CDR.
NASA’s policy also does not require a major decision review before  beginning manufacturing. (See fig. 6 above.) Therefore, the transition from  final design to fabrication, assembly, and test often marks a de facto  production decision. According to NASA officials, the agency rarely enters  a formal production phase due to the small quantities of space systems  that they build. However, due to the high cost of failure associated with  NASA projects and the costs and risks involved in repairing a system   in-orbit, a major decision review at production that assesses product  reliability is essential even for these limited production systems. In  addition, although NASA’s production quantities are typically low, in some  instances NASA does produce larger quantities of a system or subsystem,  such as the external tanks for the Space Shuttle. Furthermore, NASA’s  plans indicate that the agency may be increasing production for elements  of their future systems. For example, NASA’s Exploration Systems  Architecture Study indicates that NASA plans to build several new crew  exploration vehicles, with disposable elements, such as the lunar lander,  solid rocket boosters, and space shuttle main engines that will require  higher numbers of production runs.
Rather than establish specific criteria by which all projects are judged,  NASA’s policy requires that projects manage to baselines and plans  established in key management documents and approved at the project  NAR. The baselines and plans serve as their primary tools for measuring  project progress and as the primary basis for judgment at project reviews.  While the plans may include some information that addresses   knowledge-based criteria for design and production, the instructions for  preparing them leaves the establishment of thresholds and success criteria  to the discretion of the project manager. For example, NASA policy  requires that projects include, as part of the project plan, a Verification  and Validation Sub Plan that describes the project’s approach to verifying  and validating hardware and software as part of the project plan. The  policy, however, includes no instruction as to what constitutes a sufficient  approach to testing. In other words, there are no requirements concerning  the fidelity of test articles or the realism of the test environment. Similarly,  NASA’s policy requires that projects include, as part of the project plan, a  Systems Engineering Sub Plan that describes the project’s approach to  systems engineering and the technical standards that are applicable,  including metrics that verify the processes. The policy, however, does not  identify the types of metrics appropriate to verify the process or establish  any threshold criteria.
The absence of major decision reviews, along with specific criteria in the  fabrication, assembly, and test phase, could result in concurrent design  and manufacturing activities, a practice our past work has found increases  risk in acquisition programs. Furthermore, lacking a major decision  review to ensure that projects have gained the appropriate levels of  knowledge at KP2 and KP3, NASA decision makers cannot be provided a  high level of certainty that the project will meet cost, schedule, and  performance requirements and have no assurance that the provisions of  the key management documents required by NPR 7120.5C are being  executed after the NAR.
section_title: Lack of NASA-Wide Project Management Criteria May Result in Investment Decisions Based on Inconsistent Information
paragraphs: NASA centers have varying approaches to implementing project  management policies and systems engineering guidance for flight systems  and ground support projects. Some centers use criteria at key decision  points that are similar to the criteria required to ensure a knowledge-based  approach is followed, while others lack such criteria. As a result, each  center reports a different level and type of knowledge about a project at  key decision points. Centers also rely on project managers and systems  engineers to employ good project management and systems engineering  practices. However, given the loss of experienced project managers and  the decline of in-house systems engineering and technical capabilities  agencywide, that reliance could be problematic. These situations make it  difficult for NASA decision makers to evaluate center projects on a  common foundation of knowledge and make sound investment decisions  and tradeoffs based on those evaluations. A standardized, knowledge- based approach would prepare NASA to face competing budgetary  priorities and make difficult decisions regarding the investment in and  termination of projects.
subsections: section_title: Individual Centers Tailor Implementation of Agencywide Project Management Policies and Systems Engineering Guidance
paragraphs: While NASA centers are given discretion about how they implement  agencywide policies, they are expected to have procedures and guidelines  in place for implementing those policies. Some centers have developed  center-specific policies and criteria for implementing NASA’s project  management policies and system engineering guidance, while others have  not. Centers also rely on project managers and systems engineers to  implement the requirements of NPR 7120.5C and to use NASA’s Systems  Engineering Handbook as guidance for good systems engineering  practices. Some NASA centers have also developed criteria in their  policies that are similar to the criteria used to ensure a knowledge-based  approach is followed; other centers lack such criteria.
Because of their varying policies and criteria, each center requires a  different level of knowledge at the same point in a project’s development  cycle. For example, GSFC requires its projects to mature technologies to  TRL 6 by the preliminary design review—before entering the  implementation phase. On the other hand, JPL, MSFC, and JSC policies do  not require projects to mature technology to a particular level before  entering implementation, leaving the determination of needed technology  maturity up to the project manager.
Requirements for assessing design maturity also vary across the centers.  While most of the center policies require a CDR to enter “Phase D— Fabrication, Assembly, and Test”—the criteria used to assess projects at  this point vary. For example, both GSFC and MSFC require projects to  have completed a percentage of design drawings at this review. MSFC  requires that 90 percent of design drawings to be complete by CDR, which  is consistent with best practices. While GSFC establishes a minimum  threshold of drawings to be complete by CDR—greater than 80 percent— neither JSC nor JPL establish a minimum percentage drawing requirement.  Instead, JSC requires the design be complete and drawings ready to begin  production, and JPL requires that the design be mature and provide  confidence in the integrity of the flight system design.
Almost none of the center policies include a requirement to assess the  maturity of production processes. According to NASA officials, due to the  low quantities of systems generally produced by the agency, most of the  center policies do not require a review before beginning manufacturing.  Only JSC requires a Production Readiness Review to ensure that  production plans, facilities, and personnel are in place and ready to begin  production.  However, the criteria do not specify quantifiable thresholds to  measure production readiness at the review. JPL, MSFC, and GSFC do not  have policies that outline requirements for such a review.
In addition to individual policy requirements, centers may rely on project  managers and systems engineers to employ good project management and  systems engineering practices. For example, an experienced project  manager at JPL told us that although JPL policy does not require a  particular TRL at PDR to enter implementation, he required that all  technologies for his project be around a TRL 6 in order to separate  technology development from systems development. Reliance on project  managers to implement good practices, however, could be problematic  given the diminishing number of experienced project managers available  to lead projects and the decline of in-house systems engineering and  technical capabilities agencywide caused by increasing retirements and  outsourcing.
section_title: Informed Investment Decisions Require Consistent Knowledge
paragraphs: In a knowledge-based process, the achievement of each successive  knowledge point builds on the preceding one, giving decision makers the  information they need, when they need it, to make decisions about  whether to invest significant additional funds to move forward with  product development. Our work has shown that successful product  development efforts are marked by adherence to a disciplined process that  establishes and uses common and consistent criteria for decision making  at these key points. With varying project management and systems  engineering criteria, NASA centers’ technical reviews, such as PDR,  provide different levels of knowledge to support NASA’s major decision  reviews, such as the NAR, which NASA uses to support its major  investment decisions for flight systems and ground support projects.
In the near future, NASA will need to determine the resources necessary to  develop the systems and supporting technologies to achieve the  President’s Vision for Space Exploration and structure its investment  strategy accordingly. Initial implementation of the Vision as explained in  NASA’s Exploration Systems Architecture Study calls for completing the  International Space Station, developing a new crew exploration vehicle,  and returning to the moon no later than 2020. NASA estimates that it will  cost approximately $104 billion over the next 13 years to accomplish these  initial goals. These priorities, along with NASA’s other missions, will be  competing within NASA for funding. It will likely be difficult for NASA  managers to agree on which projects to invest in and which projects to  terminate. The NASA Administrator has acknowledged that NASA faces  difficult choices about its missions in the future—for example, between  human space flight, science, and aeronautics missions.
Using consistent criteria to evaluate all NASA projects would help ensure  that the same level and type of knowledge is available about individual  projects at key decision points. Analogous information about all flight  systems and ground support projects would allow decision makers to  make apples-to-apples comparisons across projects and make investment  decisions, and trade-offs, based upon these comparisons. Further, policies  with consistent criteria can provide inexperienced project managers and  systems engineers with the necessary guidance to implement good project  management and systems engineering practices and ensure that the right  knowledge is available for decision makers.
NASA officials within the Chief Engineer’s Office acknowledged the need  for more consistency in criteria across the various NASA centers in order  to successfully achieve NASA’s Vision for Space Exploration. Some NASA  centers, however, are resistant to standardized criteria because they feel it  could be overly prescriptive. These centers indicated that because of the  unique nature of the work done at each of the 10 NASA centers, it would  be unrealistic to hold every project to the same criteria. Nonetheless, our  work has shown that the process used by successful product developers to  develop leading-edge technology and products does not differ based upon  the type of product being developed. Further, these developers adhere to a  disciplined process that establishes and uses common and consistent  criteria for decision making—regardless of the type of product or  technology being developed. Using consistent criteria can allow NASA  decision makers to assess the likely return on competing investment  priorities and to reevaluate alternatives and make investment decisions  across projects to increase the likelihood of attaining the strategic goals of  the agency.
section_title: Conclusions
paragraphs: Several of NASA’s major acquisitions have been marked by cost, schedule,  and performance problems. Yet the challenges NASA faces in the future  are likely to far exceed those it has faced in the past. The complex  technical requirements associated with fulfilling the President’s Vision,  the fiscal constraints under which NASA will be required to operate, the  diminished number of experienced project managers and systems  engineers, and the potential for increased production make following a  knowledge-based approach for flight systems and ground support projects  all the more critical. While NASA has made improvements to its policies  governing project management, the lack of major decision reviews beyond  the initial project approval gate leaves decision makers with little  knowledge about the progress of the agency’s projects. Further, without a  standard set of criteria to measure projects at crucial phases in the  development life cycle, NASA cannot be assured that its decisions will  result in the best possible return on its investments. Since NASA is  currently in the process of revising its program and project management  policies and developing an agencywide systems engineering policy, we  believe this presents a unique opportunity for the agency to correct some  of the problems identified during our review.
section_title: Recommendations for Executive Action
paragraphs: In order to close the gaps between NASA’s current acquisition  environment and best practices on knowledge-based acquisition, we  recommend that NASA take steps to ensure NASA projects follow a  knowledge-based approach for product development. Specifically, we  recommend that the NASA Administrator direct the Office of the Chief  Engineer to take the following two actions:   In drafting its systems engineering policy, incorporate requirements in  the policy for flight systems and ground support projects to capture  specific product knowledge by key junctures in project development.  The demonstration of this knowledge should be used as exit criteria for  decision making at the following key junctures:    Before projects are approved to transition from formulation to  implementation, the policy should require that projects  demonstrate that key technologies have reached a high maturity  level.
Before projects are approved to transition from final design to  fabrication, assembly, and test, the policy should require that  projects demonstrate that the design is stable.
Before projects are approved to transition into production, the  policy should require projects to demonstrate that the design  can be manufactured within cost, schedule, and quality targets.
Revise NPR 7120.5C to institute additional major decision reviews  following the NAR for flight systems and ground support projects,  which result in recommendations to the appropriate decision authority.  These reviews should be tied to the key junctures during project  development mentioned above in order to increase the likelihood that  cost, schedule, and performance requirements of the project will be  met.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to NASA for review and comment.  In  written comments, NASA indicated that it agreed with our  recommendations and outlined specific actions that the agency plans to  take to address them.
The actions that NASA plans to take to address our recommendations are  a positive step toward achieving successful project outcomes and ensuring  that decision makers are appropriately investing the agency’s resources.  We are pleased to hear that many knowledge-based practices identified in  our report are currently being practiced agencywide in the management  and development NASA systems. The addition of such practices to NASA’s  policies will only strengthen their use agencywide and ensure that these  practices continue to be utilized by less experienced project managers and  systems engineers as the more experienced workforce retires. The  effectiveness of such practices, however, will be limited if project officials  are not held accountable for demonstrating a high level of knowledge,  consistent with the success criteria that NASA plans to require in its  policies, at key junctures in development. It is critical that project officials  not only have a high level of knowledge about a project at key junctures,  but also that this information is used by decision makers to make  decisions on whether to invest additional resources and allow a project to  proceed through the development life cycle.
NASA’s comments are reprinted in appendix III. NASA also provided  technical comments, which we addressed throughout the report as  appropriate.
As agreed with your offices, unless you announce its contents earlier, we  will not distribute this report further until 30 days from its date. At that  time, we will send copies to NASA’s Administrator and interested  congressional committees. We will make copies available to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If your or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or lia@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are acknowledged in  appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which NASA’s current policies support an  acquisition approach consistent with best practices identified in GAO’s  work on system acquisitions, we reviewed and analyzed NASA-wide  program and project management policies and systems engineering  guidance. Our review and analysis of the policy focused on requirements  for flight systems and ground support projects. We interviewed NASA  Headquarters officials from the Office of the Chief Engineer who are  responsible for the policy and guidance. We compared NASA’s policy on  program and project management with criteria contained in GAO best  practices work on systems acquisition and space system acquisitions. We  concentrated on whether the policy provides a framework for a  knowledge-based process and the criteria necessary to carry out this  intent.
To determine how NASA-wide acquisition policies are implemented across  the various NASA centers, we reviewed NASA center-specific program and  project management policies and systems engineering policies and  interviewed officials responsible for implementing those policies. While  we interviewed officials from all centers, we focused on the centers that  manage the majority of NASA’s flight systems and ground support  projects—GSFC, JPL, JSC, and MSFC. This approach included site visits  with GSFC, JPL, JSC, and MSFC and teleconferences with the remaining  centers. We compared examples of the centers’ implementation of the  policies and specific criteria included in these policies with our best  practices work on systems acquisition.
We performed our work from May 2005 to November 2005 in accordance  with generally accepted government auditing standards.
section_title: Appendix II: Activities That Enable the Capture of Design and Manufacturing Knowledge
section_title: Appendix III: Comments from the National Aeronautics and Space Administration
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, James L. Morrison, Assistant  Director; Valerie Colaiaco, Tom Gordon, Alison Heafitz, Shelby S. Oakley,  Ron Schwenn, Karen Sloan, and John S. Warren, Jr., made key  contributions to this report.
section_title: Related GAO Products
paragraphs: Space Acquisitions: Stronger Development Practices and Investment  Planning Need to Address Continuing Problems. GAO-05-891T.  Washington, D.C.: July 12, 2005.
Defense Acquisitions: Incentives and Pressures That Drive Problems  Affecting Satellite and Related Acquisitions. GAO-05-570R.   Washington, D.C.: June 23, 2005.
Defense Acquisitions: Space-Based Radar Effort Needs Additional  Knowledge before Starting Development. GAO-04-759. Washington, D.C.:  July 23, 2004.
Defense Acquisitions: Risks Posed by DOD’s New Space Systems  Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004.
Space Acquisitions: Committing Prematurely to the Transformational  Satellite Program Elevates Risks for Poor Cost, Schedule, and  Performance Outcomes. GAO-04-71R. Washington, D.C.:   December 4, 2003.
Defense Acquisitions: Improvements Needed in Space Systems  Acquisition Policy to Optimize Growing Investment in Space.   GAO-04-253T. Washington, D.C.: November 18, 2003.
Defense Acquisitions: Despite Restructuring, SBIRS High Program  Remains at Risk of Cost and Schedule Overruns. GAO-04-48.   Washington, D.C.: October 31, 2003.
Defense Acquisitions: Improvements Needed in Space Systems  Acquisition Management Policy. GAO-03-1073. Washington, D.C.:  September 15, 2003.
Military Space Operations: Common Problems and Their Effects on  Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.:   June 2, 2003.
Military Space Operations: Planning, Funding, and Acquisition  Challenges Facing Efforts to Strengthen Space Control. GAO-02-738.  Washington, D.C.: September 23, 2002.
Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data  Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002.
Defense Acquisitions: Space-Based Infrared System-Low at Risk of  Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February  28, 2001.
subsections: section_title: Best Practices Reports
paragraphs: Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-05-301. Washington, D.C.: March 31, 2005.
Defense Acquisitions: Stronger Management Practices Are Needed to  Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393.  Washington, D.C.: March 1, 2004.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-04-248. Washington, D.C.: March 31, 2004.
Defense Acquisitions: DOD’s Revised Policy Emphasizes Best Practices,  but More Controls Are Needed. GAO-04-53. Washington, D.C.:   November 10, 2003.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-03-476. Washington, D.C.: May 15, 2003.
Best Practices: Setting Requirements Differently Could Reduce Weapon  Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.:   February 11, 2003.
Best Practices: Capturing Design and Manufacturing Knowledge Early  Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.:   July 15, 2002.
Defense Acquisitions: DOD Faces Challenges in Implementing Best  Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.
Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:   March 8, 2001.
Best Practices: A More Constructive Test Approach Is Key to Better  Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.:   July 31, 2000.
Defense Acquisition: Employing Best Practices Can Shape Better  Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April  26, 2000.
Best Practices: DOD Training Can Do More to Help Weapon System  Program Implement Best Practices. GAO/NSIAD-99-206. Washington,  D.C.: August 16, 1999.
Best Practices: Better Management of Technology Development Can  Improve Weapon System Outcomes. GAO/NSIAD-99-162.   Washington, D.C.: July 30, 1999.
Defense Acquisitions: Best Commercial Practices Can Improve Program  Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.
Defense Acquisition: Improved Program Outcomes Are Possible.   GAO/T-NSIAD-98-123. Washington, D.C.: March 18, 1998.
Best Practices: Successful Application to Weapon Acquisition Requires  Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:  February 24, 1998.
Major Acquisitions: Significant Changes Underway in DOD’s Earned  Value Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5,  1997.
Best Practices: Commercial Quality Assurance Practices Offer  Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.:   August 26, 1996.