NASA: Assessments of Selected Large-Scale Projects
Released: 2010-02-01T00:00:00

Letter

A Sound Business Case Underpins Successful Acquisition Outcomes

Many of NASA’s projects are one-time articles, meaning that there is little opportunity to apply knowledge gained to the production of a second, third, or future increments of spacecraft. In addition, NASA often partners with other domestic partners and other space-faring countries, including several European nations, Japan, and Argentina. These partnerships go a long way to foster international cooperation in space, but they also subject NASA projects to added risk such as when partners do not meet their obligations or run into technical obstacles they cannot easily overcome. While space development programs are complex and difﬁ cult by nature, and most are one-time efforts, the nature of its work should not preclude NASA from achieving what it promises when requesting and receiving funds. We have reported that NASA would beneﬁ t from a more disciplined approach to its acquisitions.

The development and execution of a knowledge-based business case for these projects can provide early recognition of challenges, allow managers to take corrective action, and place needed and justiﬁ able projects in a better position to succeed. Our studies of best practice organizations show the risks inherent in NASA’s work can be mitigated by developing a solid, executable business case before committing resources to a new product development. In its simplest form, this is evidence that (1) the customer’s needs are valid and can best be met with the chosen concept, and (2) the chosen concept can be developed and produced within existing resources— that is, proven technologies, design knowledge, adequate funding, and adequate time to deliver the product when needed. A program should not go forward into product development unless a sound business case can be made. If the business case measures up, the organization commits to the development of the product, including making the ﬁ nancial investment. Our best practice work has shown that developing business cases based on matching requirements to resources before program start leads to more predictable program outcomes—that is, programs are more likely to be successfully completed within cost and schedule estimates and deliver anticipated system performance.

At the heart of a business case is a knowledge-based approach to product development that is a best practice among leading commercial ﬁ rms. Those ﬁ rms have created an environment and adopted practices that put their program managers in a good position to succeed in meeting expectations. A knowledge-based approach requires that managers demonstrate high levels of knowledge as the program proceeds from technology development to system development and, ﬁ nally, production. In essence, knowledge supplants risk over time. This building of knowledge can be described over the course of a program, as follows: When a project begins development, the customer’s needs should match the developer’s available resources—mature technologies, time, and funding. An indication of this match is the demonstrated maturity of the technologies needed to meet customer needs—referred to as critical technologies. If the project is relying on heritage—or pre-existing— technology, that technology must be in appropriate form, ﬁ t, and function to address the customer’s needs within available resources. The project will normally enter development after completing the preliminary design review, at which time a business case should be in hand.

Then, about midway through the product’s development, its design should be stable and demonstrate it is capable of meeting performance requirements. The critical design review takes place at that point in time because it generally signiﬁ es when the program is ready to start building production-representative prototypes. If design stability is not achieved, but a product development continues, costly re-designs to address changes to project requirements and unforeseen challenges can occur. By the critical design review, design should be stable and capable of meeting performance requirements.

Finally, by the time of the production decision, the product must be shown to be producible within cost, schedule, and quality targets and have demonstrated its reliability, and the design must demonstrate that it performs as needed through realistic system-level testing. Lack of testing increases the possibility that project managers will not have information that could help avoid costly system failures in late stages of development or during system operations.

Our best practices work has identiﬁ ed numerous other actions that can be taken to increase the likelihood that a program can be successfully executed once that business case is established. These include ensuring cost estimates are complete, accurate, and updated regularly and holding suppliers accountable through such activities as regular supplier audits and performance evaluations of quality and delivery. Moreover, we have recommended using metrics and controls throughout the life cycle to gauge when the requisite level of knowledge has been attained and when to direct decision makers to consider criteria before advancing a program to the next level and making additional investments.

The consequence of proceeding with system development without establishing and adhering to a sound business case is substantial. GAO and others have reported that NASA has experienced cost and schedule growth in several of its projects over the past decade, resulting from problems that include failing to adequately identify requirements and underestimating complexity and technology maturity. We have found that the need to meet schedule is one of the main reasons why programs cannot execute as planned. Short cuts, such as developing technology while design work and construction are already underway, and delaying or reducing tests, are taken to meet schedule. Ultimately, when a schedule is set that cannot accommodate the work that needs to be done, costs go up and capability is delayed. Delaying the delivery of these capabilities can also have a ripple effect throughout NASA projects as staff must then stay on a given project longer than intended, thus increasing the project’s costs, and crippling other projects that had counted on using newly available staff to move forward.

NASA Continues Efforts to Improve Its Acquisitions

In 2005, we reported that NASA’s acquisition policies did not conform to best practices for product development because those policies lacked major decision reviews at several key points in the project life-cycle that would allow decision makers to make informed decisions about whether a project should be authorized to proceed in the development life cycle. Based in part on our recommendations, NASA issued a revised policy in March 2007 that institutes several key decision points (KDP) in the development life cycle for space ﬂ ight programs and projects. At each KDP, a decision authority is responsible for authorizing the transition to the next life-cycle phase for the project. In addition, NASA’s acquisition policies also require that technologies be sufﬁ ciently mature at the preliminary design review before the project enters implementation, that the design is appropriate to support proceeding with full-scale fabrication, assembly, integrating and test at the critical design review, and that the system can be fabricated within cost, schedule, and performance speciﬁ cations. These changes brought the policy more in line with best practices for product development. A more detailed discussion of NASA’s acquisition policy and how it relates to best practices is provided in appendix III of this report.

Further, in response to GAO’s designation of NASA acquisition management as a high risk area, NASA developed a corrective action plan to improve the effectiveness of NASA’s program/project management. The approach focuses on how best to ensure the mitigation of potential issues in acquisition decisions and better monitor contractor performance. The plan identiﬁ es ﬁ ve areas for improvement—program/project management, cost reporting processes, cost estimating and analysis, standard business processes, and management of ﬁ nancial management systems—each of which contains targets and goals to measure improvement. As part of this initiative, NASA has taken a positive step to improve management oversight of project cost, schedule, and technical performance with the establishment of a baseline performance review reporting to NASA’s senior management. Through monthly reviews, NASA intends to highlight projects that are predicted to exceed internal NASA cost and/or schedule baselines, which are set lower than cost and schedule baselines submitted to Congress, so the agency can take preemptive actions to minimize the projects’ potential cost overruns or schedule delays. During our data collection efforts, we reviewed several projects’ monthly and quarterly status reports, which gave us insight into their status, risks, and issues. While this reporting structure might enable management to be aware of the issues projects are facing, it is too early to tell if the monthly reviews are having the intended impact of enabling NASA management to take preemptive cost saving actions, such as delaying a design review or canceling a project.

As a part of the continuing effort to improve its acquisition processes, NASA has begun a new initiative—Joint Cost and Schedule Conﬁ dence Levels (JCL)—to help programs and projects with management, cost and schedule estimating, and maintenance of adequate levels of reserves. Under this new policy, cost, schedule, and risk are combined into a complete picture to help inform management of the likelihood of a project’s success. Utilizing JCL, each project will receive a cost estimate with a corresponding conﬁ dence level—the percentage probability representing the likelihood of success at the speciﬁ ed funding level. NASA believes the application of this policy will help reduce the cost and schedule growth in its portfolio and improve transparency, and increase the probabilities of meeting those expectations. NASA’s goal is for all projects that have entered the implementation phase to have a JCL established by spring 2010.

While these efforts are positive steps, it is too early to assess their impact and they will be limited if project ofﬁ cials are not held accountable for demonstrating that elements of a knowledge-based business case are demonstrated at key junctures in development. For projects to have better outcomes not only must they demonstrate a high level of knowledge at key junctures, but decision makers must also use this information to determine whether and how best a project should proceed through the development life cycle. If done successfully, these measures should enable NASA to foster the expansion of a business-oriented culture, reduce persistent cost growth and schedule delays, and maximize investment dollars.

Our Observations

We assessed 19 large-scale NASA projects in this review. Four of these projects were in the formulation phase where cost and schedule baselines have yet to be established, while 15 had entered implementation. Nine of the 15 projects experienced signiﬁ cant cost and/or schedule growth from their project baselines, while ﬁ ve of the remaining projects had just entered implementation and their cost and schedule baselines were established in ﬁ scal year 2009. NASA provided cost and schedule data for 14 of the 15 projects in the implementation phase of the project life cycle. Despite being in implementation, NASA did not provide cost or schedule data for the Magnetospheric Multiscale (MMS) project. NASA will not formally release its baseline cost and schedule estimates for this project until the ﬁ scal year 2011 budget submission to Congress, and late in our review process agency ofﬁ cials notiﬁ ed us that they will not provide project estimates to GAO until that time. NASA also did not provide formal cost and schedule information for the projects in formulation, citing that those estimates were still preliminary. See ﬁ gure 1 for a summary of these projects.

Based on our analysis, development costs for projects in our review increased by an average of over 13 percent from their baseline cost estimates—including one project that increased by over 68 percent— and an average delay of almost 11 months to their launch dates. These averages were signiﬁ cantly higher when the four projects that just entered implementation are excluded. Speciﬁ cally, there are 10 projects of analytical interest because (1) they are in the implementation phase, and (2) their baselines are old enough to begin to track variances. Most of these 10 projects have experienced signiﬁ cant cost and/or schedule growth, often both. These projects had an average development cost growth of 18.7 percent—or almost $121.1 million—and schedule growth of over 15 months, and a total increase in development cost of over $1.2 billion. Over half of this total increase in development cost—or $706.6 million—occurred in the last year. These cost growth and schedule delays have all occurred within the last 3 years, and a number of these projects had experienced considerable cost growth before baselines were established in response to the 2005 statutory reporting requirement. See table 1 below for the cost and schedule growth of the NASA projects in the implementation phase.

Despite having baselines established in ﬁ scal year 2008, two projects have sought reauthorization from Congress because of development cost growth in excess of 30 percent. Congress reauthorized the Glory project in ﬁ scal year 2009, and new cost and schedule baselines were established after the project experienced a 53 percent cost growth and 6-month launch delay from original baseline estimates. The Glory project has since breached its revised schedule baseline by 16 months and exceeded its development cost baseline by over 14 percent—for a total development cost growth of over 75 percent in just 2 years. Project ofﬁ cials also indicated that recent technical problems could cause additional cost growth. Similarly, the Mars Science Laboratory project is currently seeking reauthorization from Congress after experiencing development cost in excess of 30 percent.

Project Challenges

All six factors we assessed can lead to project cost and schedule growth: technology maturity, design stability, contractor performance, development partner performance, funding issues, and launch manifest issues. These factors—characterized as project challenges—were evident in the projects that had reached the implementation phase of the project life cycle, but many of them began in the formulation phase. We did not speciﬁ cally correlate individual project challenges with speciﬁ c cost and/or schedule changes in each project. The degree to which each speciﬁ c challenge contributed to cost and schedule growth varied across the projects in this review and we did not assign any speciﬁ c challenge as a primary factor for cost and/or schedule growth. Table 2 depicts the extent to which each of the six challenges occurred for each of the 19 projects we reviewed.

Technology Maturity was by far the most prevalent challenge, affecting 15 of the 19 projects. When combined with design instability—another metric related to technical difﬁ culty—17 projects were affected. A discussion of each challenge follows.

Technology Maturity

Our past work on systems acquisition has shown that beginning an acquisition program before requirements and available resources are matched can result in a product that fails to perform as expected, costs more, or takes longer to develop. We have found that these problems are largely rooted in the failure to match customer’s needs with the developer’s resources—technical knowledge, timing, and funding—when starting product development. In other words, commitments were made to deliver capability without knowing whether the technologies needed could really work as intended. Time and costs were consistently underestimated, and problems that surfaced early cascaded throughout development and magniﬁ ed the risks facing the program. Our best practices work has shown that a technology readiness level (TRL) of 6— demonstrating a technology as a fully integrated prototype in a relevant environment—is the level of maturity needed to minimize risks for space systems entering product development. NASA’s acquisition policy states that a TRL of 6 is desirable prior to integrating a new technology. Technology maturity is a fundamental element of a sound business case, and the absence is a marker for subsequent problems, especially in design.

Similarly, our work has shown that the use of heritage technology—proven components that are being modiﬁ ed to meet new requirements—can also cause problems when the items are not sufﬁ ciently matured to meet form, ﬁ t, and function standards by the preliminary design review (PDR). NASA states in its Systems Engineering Handbook that particular attention must be given to heritage systems because they are often used in architectures and environments different from those in which they were designed to operate. Although NASA distinguishes critical technologies from heritage technologies, our best practices work has found critical technologies to be those that are required for the project to successfully meet customer requirements, regardless of whether or not they are based on existing or heritage technology. Therefore, whether technologies are labeled as “critical” or “heritage,” if they are important to the development of the spacecraft or instrument—enabling it to move forward in the development process—they should be matured by PDR.

Of the 14 projects for which we received data and that had entered the implementation phase, four entered this phase without ﬁ rst maturing all their critical technologies, and 10 encountered challenges in integrating or modifying heritage technologies. Additionally, two projects in formulation— Ares I and Orion—also encountered challenges with critical or heritage technologies. These projects did not build in the necessary resources for technology modiﬁ cation. For instance, the recent cost and schedule growth in the Mars Science Laboratory (MSL) highlights the problems that can be realized when a project proceeds past the formulation phase with immature technologies. MSL reported seven critical technologies were not mature at the time of its preliminary design review, and over a year later two of these technologies were still immature at the critical design review; however, the project moved forward into the implementation phase with established cost and schedule baselines and the lack of technology maturity contributed to an unstable design. In part as a result of immature technologies and an unstable design, MSL delayed its launch date by 25 months, and development costs have grown by more than $660 million. In November 2008, the GRAIL project also moved beyond its PDR with an immature heritage technology—the reaction wheel assembly. This technology has been ﬂ own on other NASA missions, but the project team must modify it for GRAIL by integrating electronics into the assembly.

NASA acknowledges in its Systems Engineering Handbook that modiﬁ cation of heritage systems is a frequently overlooked area in technology development and that there is a tendency on the part of project management to overestimate the maturity and applicability of heritage technology. NASA recognizes that as a result of not placing enough emphasis on the development of heritage technologies, key steps in the development process are not given appropriate attention, and critical aspects of systems engineering are overlooked.

Design Stability

The importance of establishing a stable design at a project’s critical design review (CDR) is also critical. The CDR provides assurance that the design is mature and will meet performance requirements. An unstable design can result in costly re-engineering and re-work efforts, design changes, and schedule slippage. Quantitative measures employed at CDR, such as percentage of engineering drawings, can provide evidence that the design is stable and “freeze” it to minimize changes in the future. Our work has shown that release of at least 90 percent of engineering drawings at the CDR provides evidence that the design is stable. Though NASA’s acquisition policy does not specify how a project should achieve design stability by CDR, NASA’s Systems Engineering Handbook adheres to this metric of 90 percent drawings released by the CDR.

Eight projects in our assessment have already held their CDR and were able to provide us with the number of engineering drawings completed and released. None of these 8 projects met the 90 percent standard for design stability at CDR; however, NASA believes that some of these projects had stable designs and pointed to other activities that occurred prior to CDR as evidence. Nevertheless, the percentage of engineering drawings released at CDR by these 8 projects averaged less than 40 percent, and more than three-fourths of these projects had signiﬁ cant cost and/or schedule growth from their established baselines after their CDR when their design was supposed to be stable. Although all the cost and schedule growth for these projects cannot be directly attributed to a lack of design stability, we believe that this was a contributing factor.

Discussions with project ofﬁ cials showed the metric was used inconsistently to gauge design stability. For example, Goddard Space Flight Center requires greater than 80 percent drawings released at CDR, yet we were told by several project ofﬁ cials that the rule of thumb for NASA projects is between 70 and 90 percent drawings released at CDR. However, there was no consensus among the ofﬁ cials. For example, one project manager from Goddard Space Flight Center told us the project is planning to have 70 percent of the drawings released at CDR; a project manager from the Jet Propulsion Laboratory cited that having 85 to 90 percent of the drawings released is what he prefers at CDR; otherwise, he does not consider the project design to be complete. Goddard’s Chief Engineer said that, as a member of a design review board, he will generally question projects that have less than 95 percent of engineering drawings released, especially if the project is using heritage technologies. Ofﬁ cials added that at CDR it is more important to have drawings completed that relate to critical technologies than those related to integration activities.

In addition to released drawings, NASA often relies on subject matter experts in the design review process and other methods to assure that a project has a stable design. Some projects indicated that completing engineering models, which are preproduction prototypes, and holding sub-system level CDR’s for instruments and components helped to assess design stability, at least in part. Ofﬁ cials for these projects indicated that use of engineering models helps decrease risk of ﬂ ight unit development; projects that did not use engineering models indicated they might have caught problems earlier had they used them. For example, at CDR the Mars Science Laboratory’s engineering models were incomplete and could have been a cause of concern. Mars Science Laboratory project ofﬁ cials were aware that avionics were an issue at CDR, but were unaware of future problems with other project components, such as the actuators. Project ofﬁ cials told us that if the engineering models for all subsystems had been completed at CDR, many of the later problems would have been caught and mitigated earlier in the process, thereby avoiding schedule delays. However, these project ofﬁ cials added that engineering models are expensive to employ and not all projects have the available funding required to utilize them.

Contractor Performance

NASA relies heavily on the work of its contractors. Ofﬁ cials at ﬁ ve of the projects we reviewed indicated that the contractors for their projects had trouble moving their work forward after experiencing technical and design problems with hardware that disrupted development progress. Since about 85 percent of NASA’s annual budget is spent by its contractors, the performance of these contractors is instrumental to the success of the projects.

Shifts in the industrial base and a lack of expertise at the contractors affected performance. For example, project ofﬁ cials for the SOFIA project reported that the contractor for the aircraft modiﬁ cation was bought and sold several times during the development process. Project ofﬁ cials further reported that the contractor had limited experience with this type of work and did not fully understand the statement of work. Consequently, the contractor had difﬁ culty completing this work, which led to signiﬁ cant cost overruns. While project ofﬁ cials told us that issues with that contractor have since been resolved, this year another SOFIA contractor that is responsible for developing hardware and software has performed poorly, which ofﬁ cials attribute to a recent buyout of the company. In addition, agency ofﬁ cials said that NASA is a low priority for the contractor, and the project is ﬁ nding it difﬁ cult to exert pressure to ensure better performance. Project ofﬁ cials told us that they currently have three people at the contractor’s site as a permanent presence. They added that if the contract were to be cancelled due to poor performance, this work would be brought in-house and would result in a one year delay. In addition, the Glory project has struggled for several years to develop a key instrument. The Glory project manager cited management inefﬁ ciencies with the instrument’s contractor including senior leadership changes, a loss of core competencies because of a plant closure, and a lack of proper decision authority. The contractor agreed that the plant closure and the need to re-staff were major project challenges.

Development Partner Performance

Six projects in our review encountered challenges with their development partners. In these cases the development partner could not meet their commitments to the project within the planned schedules. For example, NASA collaborated with the European Space Agency (ESA) on the Herschel space observatory. NASA delivered its two instruments to ESA in a timely manner, but ESA encountered difﬁ culties developing its instruments, and the result was a 14-month delay in Herschel’s schedule. Because of this delay, NASA incurred an estimated $39 million in cost growth because of the need to fund component developers for a longer period of time than originally planned. We found that of the projects that are currently in implementation and have experienced cost and/or schedule growth, those with international or domestic partners experienced more than one-and-a- half times as much schedule growth on average as those with no partner. Table 3 below shows the average schedule growth for projects with partners as compared to those without partners.

Funding Issues

During the course of our review, we identiﬁ ed six projects in the implementation phase, as well as three projects still in formulation, that had experienced issues related to the project’s funding because of issues such as agency-directed funding cuts early in the project life-cycle and projects whose budgets do not match the work expected to be accomplished. For example, NASA management cut $35 million from the Kepler project’s ﬁ scal year 2005 budget—a cut amounting to one-half of the project’s budget for the year. Contractor ofﬁ cials told us that this forced the shutdown of signiﬁ cant work, interrupted the overall ﬂ ow and scheduling for staff and production, and required a renegotiation of contracts. This funding instability, according to a NASA project ofﬁ cial, contributed to an overall 20-month delay in the project’s schedule and about $169 million in cost growth.

The funding instability for Kepler affected more than that one project. The WISE project had to extend the formulation phase since funding was unavailable at the time of the conﬁ rmation review in November 2005.

According to NASA and contractor ofﬁ cials, the WISE project experienced funding cuts when NASA took money from that project to offset increased costs for the Kepler project. As a result of the extended formulation phase, the WISE project manager told us that development costs increased and the launch readiness date slipped 11 months. This is an example of how, when problems arise, one project can become the bill-payer for another project, making it difﬁ cult to manage the portfolio and make investment decisions.

We also identiﬁ ed several projects where, according to NASA ofﬁ cials, the projected budget was inadequate to perform work in certain ﬁ scal years. For example, the Constellation program’s poorly phased funding plan has diminished both the Ares I and Orion projects’ ability to deal with technical challenges. NASA initiated the Constellation program relying on the accumulation of a large rolling budget reserve in ﬁ scal years 2006 and 2007 to fund program activities in ﬁ scal years 2008 through 2010. Thereafter, NASA anticipated that the retirement of the space shuttle program in 2010 would free funding for the Constellation program. The program’s risk management system identiﬁ ed this strategy as high risk, warning that shortfalls could occur in ﬁ scal years 2009 through 2012. According to the Constellation program manager, the program’s current funding shortfalls have reduced the ﬂ exibility to resolve technical challenges. In addition, the James Webb Space Telescope project had to delay its scheduled launch date by one year in part because of poor phasing of the project’s funding plan.

Launch Manifest Issues

We identiﬁ ed four projects in our assessment that are experiencing launch delays or other launch manifest-related challenges. By their nature, launch delays can contribute signiﬁ cantly to cost and schedule growth, as months of delay can translate into millions of dollars in cost increases. For example, the Solar Dynamics Observatory (SDO) project missed its scheduled launch date in August 2008 because of test scheduling and spacecraft parts problems. This delay resulted in the SDO project moving to the end of the manifest for the Atlas V launch vehicles on the East coast, causing an 18-month launch delay and $50 million cost increase. While the primary reason for the cost growth is that the SDO project could not meet its original schedule for launch, the project is incurring additional costs to maintain project staff longer than originally planned as they await their turn in the launch queue. According to SDO ofﬁ cials, this has also affected stafﬁ ng at Goddard Space Flight Center since these personnel were scheduled to move to other projects. Furthermore, launch delays of one project can potentially impact the launch manifest for other projects. The 25-month delay of the Mars Science Laboratory project has the potential to cause disruptions for other projects on the launch manifest in late 2011, including those outside of NASA, since planetary missions—those missions that must launch in a certain window because of planetary alignments— receive launch priority to take advantage of optimal launch windows.

Some NASA projects are also experiencing launch manifest-related challenges. For example, the Gravity Recovery and Interior Laboratory project is monitoring the availability of trained launch personnel as that mission is the last to launch on the Delta II vehicle. United Launch Alliance ofﬁ cials told us that they are taking active steps, such as cross-utilizing the Delta II personnel with other launch vehicles, to ensure that trained launch personnel are available for all the remaining Delta II launches. In addition, the recent failure of the Taurus XL launch vehicle during the launch of the Orbiting Carbon Observatory has the potential to delay the Glory mission if the Taurus XL is not cleared for use before Glory has corrected its technical problems.

Project Assessments

The 2-page assessments of the projects we reviewed provide a proﬁ le of each project and describe the challenges we identiﬁ ed. On the ﬁ rst page, the project proﬁ le presents a general description of the mission objectives for each of the projects; a picture of the spacecraft or aircraft; a schedule timeline identifying key dates for the project; a table identifying programmatic and launch information; and a table showing the baseline year cost and schedule estimates and the most current available cost and schedule data; a table showing the challenges relevant to the project; and a project status narrative. On the second page of the assessment, we provide an analysis of the project challenges and the extent to which each project faces cost, schedule, or performance risk because of these challenges. In addition, NASA project ofﬁ ces were provided an opportunity to review drafts of the assessments prior to their inclusion in the ﬁ nal product, and the projects provided both technical corrections and more general comments. We integrated the technical corrections as appropriate and characterized the general comments below the detailed project discussion. See ﬁ gure 2 below for an illustration of the layout of each two-page assessment.

Agency Comments and Our Evaluation

We provided a draft of this report to NASA for review and comment. In its written response, NASA agrees with our ﬁ ndings and states that it will strive to address the challenges that lead to cost and schedule growth in its projects. NASA agrees that GAO’s cost and schedule growth ﬁ gures reﬂ ect what the agency has experienced since the baselines were established in response to the 2005 statutory reporting requirements. Importantly, NASA has begun to provide more data regarding cost growth prior to these baselines, and we look forward to working with NASA to increase transparency into cost and schedule information of large-scale projects even further in the future.

NASA noted that its projects are high-risk and one-of-a-kind development efforts that do not lend themselves to all the practices of a “business case” approach that we outlined since essential attributes of NASA’s project development differ from those of a commercial or production industry. We agree, however NASA could still beneﬁ t from a more disciplined approach to its acquisitions whereby decisions are based upon high levels of knowledge. Currently, inherent risks are being exacerbated due to projects moving forward with immature technologies and unstable designs and difﬁ culties working with contractors and international partners, leading to cost and schedule increase which make it hard for the agency to manage its portfolio and make informed investment decisions.

NASA’s comments are reprinted in appendix I. NASA also provided technical comments, which we addressed throughout the report as appropriate and where sufﬁ cient evidence was provided to support signiﬁ cant changes.

We will send copies of the report to NASA’s Administrator and interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Ofﬁ ces of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix II: Objectives, Scope, and Methodology

Our objectives were to report on the status and challenges faced by NASA systems with life-cycle costs of $250 million or more and to discuss broader trends faced by the agency in its management of system acquisitions.

In conducting our work, we evaluated performance and identiﬁ ed challenges for each of 19 major projects. We summarized our assessments of each individual project in two components—a project proﬁ le and a detailed discussion of project challenges. We did not validate the data provided by the National Aeronautics and Space Administration (NASA). However, we took appropriate steps to address data reliability. Speciﬁ cally, we conﬁ rmed the accuracy of NASA-generated data with multiple sources within NASA and, in some cases, with external sources. Additionally, we corroborated data provided to us with published documentation. We determined that the data provided by NASA project ofﬁ ces were sufﬁ ciently reliable for our engagement purposes.

We developed a standardized data collection instrument (DCI) that was completed by each project ofﬁ ce. Through the DCI, we gathered basic information about projects as well as current and projected development activities for those projects. The cost and schedule data estimates that NASA provided were the most recent updates as of October 2009; performance data that NASA provided were the most recent updates as of November 2009. At the time we collected the data, 4 of the 19 projects were in the formulation phase and 15 were in the implementation phase. NASA only provided cost and schedule data for 14 projects in implementation. Despite being in the implementation phase, NASA did not provide cost or schedule data for the Magnetospheric Multiscale (MMS) project. To further understand performance issues, we talked with ofﬁ cials from most project ofﬁ ces and NASA’s Ofﬁ ce of Program Analysis and Evaluation (PA&E).

The results collected from each project ofﬁ ce, Mission Directorate, and PA&E were summarized in a 2-page report format providing a project overview; key cost, contract, and schedule data; and a discussion of the challenges associated with the deviation from relevant indicators from best practice standards. The aggregate measures and averages calculated were analyzed for meaningful relationships, e.g. relationship between cost growth and schedule slippage and knowledge maturity attained both at critical milestones and through the various stages of the project life cycle. We identiﬁ ed cost and/or schedule growth as signiﬁ cant where, in either case, a project’s cost and/or its schedule exceeded the baselines that trigger reporting to the Congress.

To supplement our analysis, we relied on GAO’s work over the past years examining acquisition issues across multiple agencies. These reports cover such issues as contracting, program management, acquisition policy, and estimating cost. GAO also has an extensive body of work related to challenges NASA has faced with speciﬁ c system acquisitions, ﬁ nancial management, and cost estimating. This work provided the context and basis for large parts of the general observations we made about the projects we reviewed. Additionally, the discussions with the individual NASA projects helped us identify further challenges faced by the projects. Together, the past work and additional discussions contributed to our development of a short list of challenges discussed for each project. The challenges we identiﬁ ed and discussed do not represent an exhaustive or exclusive list. They are subject to change and evolution as GAO continues this annual assessment in future years.

Our work was performed primarily at NASA headquarters in Washington, D.C. In addition, we visited NASA’s Marshall Space Flight Center in Huntsville, Alabama; Dryden Flight Research Center at Edwards Air Force Base in California; and Goddard Space Flight Center in Greenbelt, Maryland, to discuss individual projects. We also met with representatives from NASA’s Jet Propulsion Lab in Pasadena, California and a provider of NASA launch services, the United Launch Alliance.

Data Limitations

NASA only provided speciﬁ c cost and schedule estimates for 14 of the 19 projects in our review. For one project, the Magnetospheric Multiscale project, NASA will not formally release its baseline cost and schedule estimates until the ﬁ scal year 2011 budget submission to Congress, and late in our review process agency ofﬁ cials notiﬁ ed us that they will not provide project estimates to GAO until that time. For three of the projects that had not yet entered implementation, NASA provided internal preliminary estimated total (life-cycle) cost ranges and associated schedules, from key decision point B (KDP-B), solely for informational purposes. NASA formally establishes cost and schedule baselines, committing itself to cost and schedule targets for a project with a speciﬁ c and aligned set of planned mission objectives at key decision point C (KDP-C), which follows a non- advocate review (NAR) and preliminary design review (PDR). KDP-C reﬂ ects the life-cycle point where NASA approves a project to leave the formulation phase and enter into the implementation phase. NASA explained that preliminary estimates are generated for internal planning and ﬁ scal year budgeting purposes at KDP-B, which occurs mid-stream in the formulation phase, and hence, are not considered a formal commitment by the agency on cost and schedule for the mission deliverables. NASA ofﬁ cials contend that because of changes that occur to a project’s scope and technologies between KDP-B and KDP-C, estimates of project cost and schedule can change signiﬁ cantly heading toward KDP-C. Finally, NASA did not provide data for the Global Precipitation Measurement mission because NASA ofﬁ cials said it did not have a requirement for a KDP-B review, because it was authorized to be formulated prior to the requirements of NPR 7120.5D being in place.

Project Proﬁ le Information on Each Individual Two-Page Assessment

This section of the 2-page assessment outlines the essentials of the project, its cost and schedule performance, and its status. Project essentials reﬂ ect pertinent information about each project, including, where applicable, the major contractors and partners involved in the project. These organizations have primary responsibility over a major segment of the project or, in some cases, the entire project.

Project performance is depicted according to cost and schedule changes in the various stages of the project life cycle. To assess the cost and schedule changes of each project we obtained data directly from NASA PA&E and from NASA’s Integrated Budget and Performance documents. For systems in implementation, we compared the latest available information with baseline cost and schedule estimates set for each project in the ﬁ scal year 2007, 2008, or 2010 budget request.

All cost information is presented in nominal “then year” dollars for consistency with budget data. Baseline costs are adjusted to reﬂ ect the cost accounting structure in NASA’s ﬁ scal year 2009 budget estimates. For the ﬁ scal year 2009 budget request, NASA changed its accounting practices from full-cost accounting to reporting only direct costs at the project level. The schedule assessment is based on acquisition cycle time, which is deﬁ ned as the number of months between the project start, or formulation start, and projected or actual launch date. Formulation start generally refers to the initiation of a project; NASA refers to project start as key decision point A, or the beginning of the formulation phase. The preliminary design review typically occurs during the end of the formulation phase, followed by a conﬁ rmation review, referred to as key decision point C, which allows the project to move into the implementation phase. The critical design review is held during the ﬁ nal design period of implementation and demonstrates that the maturity of the design is appropriate to support proceeding with full scale fabrication, assembly, integration, and test. Launch readiness is determined through a launch readiness review that veriﬁ es that the launch system and spacecraft/ payloads are ready for launch. The implementation phase includes the operations of the mission and concludes with project disposal.

We assessed the extent to which NASA projects exceeded their cost and schedule baselines. To do this, we compared the project baseline cost and schedule estimates with the current cost and schedule data reported by the project ofﬁ ce in October 2009.

Project Challenges Discussion on Each Individual Two-Page Assessment

To assess the project challenges for each project, we submitted a data collection instrument to each project ofﬁ ce. We also held interviews with representatives from most of the projects to discuss the information on the data collection instrument. These discussions led to identiﬁ cation of further challenges faced by NASA projects. These challenges were largely apparent in the projects that had entered the implementation phase. We then reviewed pertinent project documentation, such as the project plan, schedule, risk assessments, and major project reviews.

To assess technology maturity, we asked project ofﬁ cials to assess the technology readiness levels (TRL) of each of the project’s critical technologies at various stages of project development. Originally developed by NASA, TRLs are measured on a scale of one to nine, beginning with paper studies of a technology’s feasibility and culminating with a technology fully integrated into a completed product. (See appendix IV for the deﬁ nitions of technology readiness levels.) In most cases, we did not validate the project ofﬁ ces’ selection of critical technologies or the determination of the demonstrated level of maturity. However, we sought to clarify the technology readiness levels in those cases where the information provided raised concerns, such as where a critical technology was reported as immature late in the project development cycle. Additionally, we asked project ofﬁ cials to explain the environments in which technologies were tested.

Our best practices work has shown that a technology readiness level of 6— demonstrating a technology as a fully integrated prototype in a relevant environment—is the level of maturity needed to minimize risks for space systems entering product development. In our assessment, the technologies that have reached technology readiness level 6 are referred to as fully mature because of the difﬁ culty of achieving technology readiness level 7, which is demonstrating maturity in an operational environment—space. Projects with critical technologies that did not achieve maturity by the preliminary design review were assessed as having a technology maturity project challenge. We did not assess technology maturity for those projects which had not yet reached the preliminary design review at the time of this assessment.

To assess the complexity of heritage technology, we asked project ofﬁ cials to assess the TRL of each of the project’s heritage technologies at various stages of project development. We also interviewed project ofﬁ cials about the use of heritage technologies in their projects. We asked them what heritage technologies were being used, what effort was needed to modify the form, ﬁ t, and function of the technology for use in the new system, whether the project encountered any problems in modifying the technology, and whether the project considered the heritage technology as a risk to the project. Heritage technologies were not considered critical technologies by several of the projects we reviewed. Based on our interviews, review of data from the data collection instruments, and previous GAO work on space systems, we determined whether complexity of heritage technology was a challenge for a particular project.

To assess design stability, we asked project ofﬁ cials to provide the percentage of engineering drawings completed or projected for completion by the preliminary and critical design reviews and as of our current assessment. In most cases, we did not verify or validate the percentage of engineering drawings provided by the project ofﬁ ce. However, we collected the project ofﬁ ces’ rationale for cases where it appeared that only a small number of drawings were completed by the time of the design reviews or where the project ofﬁ ce reported signiﬁ cant growth in the number of drawings released after CDR. In accordance with GAO’s best practices, projects were assessed as having achieved design stability if they had released at least 90 percent of projected drawings by the critical design review. Projects that had not met this metric were determined to have a design stability project challenge. Though some projects used other methods to assess design stability, such as computer and engineering models and analyses, we did not analyze the use of these other methods and therefore could not assess the design stability of those projects. We could not assess design stability for those projects that had not yet reached the critical design review at the time of this assessment.

To assess whether projects encountered challenges with contractor performance, we interviewed project ofﬁ cials about their interaction and experience with contractors. We also relied on interviews we held in 2008 with contractor representatives from Orbital Sciences Corporation, Ball Aerospace and Technologies Corporation, and Raytheon Space Systems about their experiences contracting with NASA. We were informed about contractor performance problems pertaining to their workforce, the supplier base, and technical and corporate experience. We also discussed the use of contract fees with NASA and contractor’s representatives. We assessed a project as having this challenge if these contractor performance problems—as conﬁ rmed by NASA and, where possible, the project contractor—caused the project to experience a cost overrun, schedule delay, or decrease in mission capability. For projects that did not have a major contractor, we considered this challenge inapplicable to the project.

To assess whether projects encountered challenges with development partner performance, we interviewed NASA project ofﬁ cials about their interaction with international or domestic partners during project development. Development partner performance was considered a challenge for the project if project ofﬁ cials indicated that domestic or foreign partners were experiencing problems with project development that impacted the cost, schedule, or performance of the project for NASA. These challenges were speciﬁ c to the partner organization or caused by a contractor to that partner organization. For projects that did not have an international or domestic development partner, we considered this challenge not applicable to the project.

To assess whether projects encountered challenges with funding, we interviewed ofﬁ cials from NASA’s Program Analysis and Evaluation Division, NASA project ofﬁ cials, and project contractors about the stability of funding throughout the project life-cycle. Funding stability was considered a challenge if ofﬁ cials indicated that project funding had been interrupted or delayed resulting in an impact to the cost, schedule, or performance of the project, or if project ofﬁ cials indicated that the project budgets do not have sufﬁ cient funding in certain years based on the work expected to be accomplished. We corroborated the funding changes and reasons with budget documents when available.

To assess whether projects encountered challenges with their launch manifests, we interviewed NASA Launch Services ofﬁ cials and ofﬁ cials from one of NASA’s contracted providers for launch services about project launch scheduling, launch windows, and projects that missed their opportunities. Launch manifest was considered a challenge if, after establishing a ﬁ rm launch date, a project had difﬁ culty rescheduling its launch date because it was not ready, if the project could be affected by another project slipping its launch, or if there were launch vehicle ﬂ eet issues. Projects that have not yet entered into the implementation phase have not yet set a ﬁ rm launch date and were therefore not assessed.

In addition, NASA received an appropriation from the American Recovery and Reinvestment Act of 2009 (ARRA). NASA provided a record of projects involved in our review that received ARRA funds.

The individual project ofﬁ ces were given an opportunity to comment on and provide technical clariﬁ cations to the 2-page assessments prior to their inclusion in the ﬁ nal product.

We conducted this performance audit from April 2009 to February 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufﬁ cient, appropriate evidence to provide a reasonable basis for our ﬁ ndings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our ﬁ ndings and conclusions based on our audit objectives.

Appendix III: NASA Life Cycle For Flight Systems Compared to a Knowledge-Based Approach

GAO has previously conducted work on NASA’s acquisition policy for space- ﬂ ight systems, and in particular, on its alignment with a knowledge-based approach to system acquisitions. The ﬁ gure below depicts this alignment.

As the ﬁ gure shows, NASA’s policy deﬁ nes a project life cycle in two phases—the formulation and implementation phases, which are further divided into incremental pieces: Phase A through Phase F. Project formulation consists of Phases A and B, during which time the projects develop and deﬁ ne the project requirements and cost/schedule basis and design for implementation, including an acquisition strategy. During the end of the formulation phase, leading up to the preliminary design review (PDR) and non-advocate review (NAR), the project team completes its preliminary design and technology development. NASA Interim Directive NM 7120-81 for NASA Procedural Requirements 7120.5D, NASA Space Flight Program and Project Management Requirements, specify that the project complete development of mission-critical or enabling technology, as needed, with demonstrated evidence of required technology qualiﬁ cation (i.e., component and/or breadboard validation in the relevant environment) documented in a technology readiness assessment report. The project must also develop, document, and maintain a project management baseline that includes the integrated master schedule and baseline life-cycle cost estimate. Implementing these requirements brings the project closer to ensuring that resources and needs match, but it is not fully consistent with knowledge point 1 of the knowledge-based acquisition life-cycle. Our best practices show that demonstrating technology maturity at this point in the system life cycle should include a system or subsystem model or prototype demonstration in a relevant environment, not only component validation. As written, NASA’s policy does not require full technology maturity before a project enters the implementation phase.

After project conﬁ rmation, the project begins implementation, consisting of phases C, D, E, and F. During phases C and D, the project performs ﬁ nal design and fabrication as well as testing of components and system assembly, integration, test, and launch. Phases E and F consist of operations and sustainment and project closeout. A second design review, the critical design review (CDR), is held during the implementation phase toward the end of phase C. The purpose of the CDR is to demonstrate that the maturity of the design is appropriate to support proceeding with full scale fabrication, assembly, integration, and test. Though this review is not a formal decision review, its requirements for a mature design and ability to meet mission performance requirements within the identiﬁ ed cost and schedule constraints are similar to knowledge expected at knowledge point 2 of the knowledge-based acquisition life-cycle. Furthermore, after CDR, the project must be approved at KDP D before continuing into the next phase.

The NASA acquisition life-cycle lacks a major decision review at knowledge point 3 to demonstrate that production processes are mature. According to NASA ofﬁ cials, the agency rarely enters a formal production phase due to the small quantities of space systems that they build.

Appendix IV: Technology Readiness Levels

None (paper studies and analysis)

Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies.

None (paper studies and analysis)

Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

Analytical studies and demonstration of nonscale individual components (pieces of subsystem).

Basic technological components are integrated to establish that the pieces will work together. This is relatively “low ﬁ delity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory.

Low ﬁ delity breadboard. Integration of nonscale components to show pieces will work together. Not fully functional or form or ﬁ t but representative of technically feasible approach suitable for ﬂ ight articles.

Fidelity of breadboard technology increases signiﬁ cantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high ﬁ delity” laboratory integration of components.

High ﬁ delity breadboard. Functionally equivalent but not necessarily form and/or ﬁ t (size weight, materials, etc). Should be approaching appropriate scale. May include integration of several components with reasonably realistic support elements/subsystems to demonstrate functionality.

Lab demonstrating functionality but not form and ﬁ t. May include ﬂ ight demonstrating breadboard in surrogate aircraft. Technology ready for detailed design studies.

Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high ﬁ delity laboratory environment or in simulated realistic environment.

Prototype.

Should be very close to form, ﬁ t and function. Probably includes the integration of many new components and realistic supporting elements/subsystems if needed to demonstrate full functionality of the subsystem.

High-ﬁ delity lab demonstration or limited/restricted ﬂ ight demonstration for a relevant environment. Integration of technology is well deﬁ ned.

Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in a realistic environment, such as in an aircraft, vehicle or space. Examples include testing the prototype in a test bed aircraft.

Prototype. Should be form, ﬁ t and function integrated with other key supporting elements/subsystems to demonstrate full functionality of subsystem.

Flight demonstration in representative realistic environment such as ﬂ ying test bed or demonstrator aircraft.

Technology is well substantiated with test data.

Technology has been proven to work in its ﬁ nal form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design speciﬁ cations.

Actual application of the technology in its ﬁ nal form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug ﬁ xing” aspects of true system development. Examples include using the system under operational mission conditions.

Appendix V: NASA Projects Receiving Additional Funding

There are 7 NASA projects in our review, including all of those in formulation, that are receiving money from the American Recovery and Reinvestment Act (ARRA) of 2009. See table 4 below for the NASA projects in our review receiving this funding and the intended use of those funds.

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Jim Morrison, Assistant Director; Jessica M. Berkholtz; Greg Campbell; Richard A. Cederholm; Kristine R. Hassinger; Jeff R. Jensen; Kenneth E. Patton; Brian A. Tittle; and Letisha T. Watson made key contributions to this report.