NASA: Sound Management and Oversight Key to Addressing Crew Exploration Vehicle Project Risks
Released: 2006-09-28T00:00:00

Letter

Background

Despite many successes in the exploration of space, such as landing the Pathfinder and Exploration Rovers on Mars, NASA has had difficulty bringing a number of projects to completion, including several efforts to build a second generation reusable human spaceflight vehicle to replace the space shuttle. NASA has attempted several costly endeavors, such as the National Aero-Space Plane, the X-33 and X-34, and the Space Launch Initiative. While these endeavors have helped to advance scientific and technical knowledge, none have completed their objective of fielding a new reusable space vehicle. We estimate that these unsuccessful development efforts have cost approximately $4.8 billion since the 1980s. The high cost of these unsuccessful efforts and the potential costs of implementing the Vision make it important that NASA achieve success in its new exploration program beginning with the CEV project.

Our past work has shown that developing a sound business case, based on matching requirements to available and reasonably expected resources before committing to a new product development effort, reduces risk and increases the likelihood of success. High levels of knowledge should be demonstrated before managers make significant program commitments, specifically: (1) At program start, the customer’s needs should match the developer’s available resources in terms of availability of mature technologies, time, human capital, and funding; (2) Midway through development, the product’s design should be stable and demonstrate that it is capable of meeting performance requirements; (3) By the time of the production decision, the product must be shown to be producible within cost, schedule, and quality targets, and have demonstrated its reliability. Our work has shown that programs that have not attained the level of knowledge needed to support a sound business case have been plagued by cost overruns, schedule delays, decreased capability, and overall poor performance. With regard to NASA, we have reported that in some cases the agency’s failure to define requirements adequately and develop realistic cost estimates—two key elements of a business case—resulted in projects costing more, taking longer, and achieving less than originally planned.

Firm Cost Estimates Cannot Be Developed at This Time

Although NASA is continuing to refine its exploration architecture cost estimates, the agency cannot at this time provide a firm estimate of what it will take to implement the architecture. The absence of firm cost estimates is mainly due to the fact that the program is in the early stages of its life cycle. NASA conducted a cost risk analysis of its preliminary estimates through fiscal year 2011. On the basis of this analysis and through the addition of programmatic reserves (20 percent on all development and 10 percent on all production costs), NASA is 65 percent confident that the actual cost of the program will either meet or be less than its estimate of $31.2 billion through fiscal year 2011. For cost estimates beyond 2011, when most of the cost risk for implementing the architecture will be realized, NASA has not applied a confidence level distinction. Since NASA released its preliminary estimates, the agency has continued to make architecture changes and refine its estimates in an effort to establish a program that will be sustainable within projected resources. While changes to the program are appropriate at this stage when concepts are still being developed, they leave the agency in the position of being unable to firmly identify program requirements and needed resources. NASA plans to commit to a firm cost estimate for the Constellation program at the preliminary design review in 2008, when requirements, design, and schedule will all be baselined. It is at this point where we advocate program commitments should be made on the basis of the knowledge secured.

Expected Budget and Competing Demands Will Challenge Architecture Implementation

NASA will be challenged to implement the ESAS recommended architecture within its projected budget, particularly in the longer-term. As we reported in July 2006, there are years when NASA has projected insufficient funding to implement the architecture with some yearly shortfalls exceeding $1 billion; while in other years the funding available exceeds needed resources. Per NASA’s approach, it plans to use almost $1 billion in appropriated funds from fiscal years 2006 and 2007 in order to address the short-term funding shortfalls. NASA, using a “go as you can afford to pay” approach, maintains that in the short-term the architecture could be implemented within the projected available budgets through fiscal year 2011 when funding is considered cumulatively. However, despite initial surpluses, the long-term sustainability of the program is questionable given the long-term funding outlook for the program. NASA’s preliminary projections show multibillion-dollar shortfalls for its Exploration Systems Mission Directorate in all fiscal years from 2014 to 2020, with an overall deficit through 2025 in excess of $18 billion. According to NASA officials, the agency will have to keep the program compelling for both Congress and potential international partners, in terms of the activities that will be conducted as part of the lunar program, in order for the program to be sustainable over the long run.

NASA is attempting to address funding shortfalls within the Constellation program by redirecting funds to that program from other Exploration Systems Mission Directorate activities to provide a significant surplus in fiscal years 2006 and 2007 to cover projected shortfalls beginning in fiscal year 2009. Several Research and Technology programs and missions were discontinued, descoped, or deferred and that funding was shifted to the Constellation Program to accelerate development of the CEV and CLV. In addition, the Constellation program has requested more funds than required for its projects in several early years to cover shortfalls in later years. NASA officials stated the identified budget phasing problem could worsen given the changes that were made to the exploration architecture following issuance of the study. For example, while life cycle costs may be lower in the long run, acceleration of development for the five segment Reusable Solid Rocket Booster and J-2x engine will likely add to the near- term development costs, where the funding is already constrained. NASA has yet to provide cost estimates associated with program changes.

NASA must also contend with competing budgetary demands within the agency as implementation of the exploration program continues. NASA’s estimates beyond 2010 are based upon a surplus of well over $1 billion in fiscal year 2011 due to the retirement of the space shuttle fleet in 2010. However, NASA officials said the costs for retiring the space shuttle and transitioning to the new program are not fully understood; thus, the expected surplus could be less than anticipated. This year, NASA plans to spend over 39 percent of its annual budget for space shuttle and International Space Station (ISS) operations—dollars that will continue to be obligated each year as NASA completes construction of the ISS by the end of fiscal year 2010. This does not include the resources necessary to develop ISS crew rotation or logistics servicing support capabilities for the ISS during the period between when the space shuttle program retires and the CEV makes its first mission to the ISS. While, generally, the budget for the space shuttle is scheduled to decrease as the program moves closer to retirement, a question mark remains concerning the dollars required to retire the space shuttle fleet as well as transition portions of the infrastructure and workforce to support implementation of the exploration architecture. In addition, there is support within Congress and the scientific community to restore money to the Science Mission Directorate that was transferred to the space shuttle program to ensure its viability through its planned retirement in 2010. Such a change could have an impact on future exploration funding.

Lack of Sound Business Case Puts CEV Acquisition at Risk

In July 2006, we reported that NASA’s acquisition strategy for the CEV placed the project at risk of significant cost overruns, schedule delays, and performance shortfalls because it committed the government to a long- term contract before establishing a sound business case. We found that the CEV contract, as structured, committed the government to pay for design, development, production and sustainment upon contract award—with a period of performance through at least 2014 with the possibility of extending through 2019.

Our report highlighted that NASA had yet to develop key elements of a sound business case, including well-defined requirements, mature technology, a preliminary design, and firm cost estimates that would support such a long-term commitment. Without such knowledge, NASA cannot predict with any confidence how much the program will cost, what technologies will or will not be available to meet performance expectations, and when the vehicle will be ready for use. NASA has acknowledged that it will not have these elements in place until the project’s preliminary design review scheduled for fiscal year 2008. As a result, we recommended that the NASA Administrator modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a long-term contractual obligation prior to establishing a sound business case at the project’s preliminary design review. In response to our recommendation, NASA disagreed and stated that it had the appropriate level of knowledge to proceed with its current acquisition strategy. NASA also indicated that knowledge from the contractor is required in order to develop a validated set of requirements and, therefore, it was important to get the contractor on to the project as soon as possible. In addition, according to NASA officials, selection of a contractor for the CEV would enable the agency to work with the contractor to attain knowledge about the project’s required resources and, therefore, be better able to produce firm estimates of project cost. In our report, we highlighted that this is the type of information that should be obtained prior to committing to a long-term contract. To our knowledge, NASA did not explore the possibility of utilizing the contractor, through a shorter-term contract, to conduct work needed to develop valid requirements and establish higher-fidelity cost estimates—a far less risky and costly strategy.

Subsequent to our report, NASA did, however, take steps to address some of the concerns we raised. Specifically, NASA modified its acquisition strategy for the CEV and made the production and sustainment schedules of the contract—known as Schedules B and C—contract options that the agency will decide whether to exercise after project’s critical design review in 2009. Therefore, NASA will only be liable for the minimum quantities under Schedules B and C when and if it chooses to exercise those options. These changes to the acquisition strategy lessen the government’s financial obligation at this early stage. Table 1 outlines the information related to the CEV acquisition strategy found in the request for proposal and changes that were made to that strategy prior to contract award. While we view these changes as in line with our recommendation and as a positive step to address some of the risks we raised in our report, NASA still has no assurance that the project will have the elements of a sound business case in place at the preliminary design review. Therefore, NASA’s commitment to efforts beyond the project’s preliminary design review—even when this commitment is limited to design, development, test and evaluation activities (DDT&E)—is a risky approach. It is at this point that NASA should (a) have the increased knowledge necessary to develop a sound business case that includes high-fidelity, engineering- based estimates of life cycle cost for the CEV project, (b) be in a better position to commit the government to a long-term effort, and (c) have more certainty in advising Congress on required resources.

Sound Management and Oversight Key to Addressing CEV Project Risks

Sound project management and oversight will be key to addressing risks that remain for the CEV project as it proceeds with its acquisition approach. To help mitigate these risks, NASA should have in place the markers necessary to help decision makers monitor the CEV project and ensure that is following a knowledge based approach to its development. However, in our 2005 report that assessed NASA’s acquisition policies, we found that NASA’s policies lacked major decision reviews beyond the initial project approval gate and a standard set of criteria with which to measure projects at crucial phases in the development life cycle—key markers for monitoring such progress. In our review of the individual center policies, we found that the Johnson Space Center project management policy, which is the policy that the CEV project will be required to follow, also lacked such key criteria. We concluded that without such requirements in place, decision makers have little knowledge about the progress of the agency’s projects and, therefore, cannot be assured that they are making informed decisions about whether continued investment in a program or project is warranted.

We recommended that NASA incorporate requirements in its new systems engineering policy to capture specific product knowledge at key junctures in project development. The demonstration of such knowledge could then be used as exit criteria for decision making at the following key junctures: Before projects are approved to transition in to implementation, we suggested that projects be required to demonstrate that key technologies have reached a high maturity level.

Before projects are approved to transition from final design to fabrication, assembly, and test, we suggested that projects be required to demonstrate that the design is stable.

Before projects are approved to transition to production, we suggested that projects be required to demonstrate that the design can be manufactured within cost, schedule, and quality targets.

In addition, we recommended that NASA institute additional major decision reviews that are tied to these key junctures to allow decision makers to reassess the project based upon demonstrated knowledge.

While NASA concurred with our recommendations, the agency has yet to take significant actions to implement them. With regard to our first recommendation, NASA stated that the agency would establish requirements for success at the key junctures mentioned above. NASA planned to include these requirements in the systems engineering policy it issued in March 2006. Unfortunately, NASA did not include these criteria as requirements in the new policy, but included them in an appendix to the policy as recommended best practices criteria. In response to our second recommendation, NASA stated it would revise its program and project management policy for flight systems and ground support projects, due to be completed in fall 2006. In the revised policy, NASA indicated that it would require the results of the critical design review and, for projects that enter a large-scale production phase, the results of the production readiness review to be reported to the appropriate decision authority in a timely manner so that a decision about whether to proceed with the project can be made. NASA has yet to issue its revised policy; therefore, it remains to be seen as to whether the CEV project decision authorities will have the opportunity to reassess and make decisions about the project using the markers recommended above after the project has initially been approved. Briefings that we have recently received indicate that NASA plans to implement our recommendation in the revised policy.

The risks that NASA has accepted by moving ahead with awarding the contract for DDT&E for CEV could be mitigated by implementing our recommendations as it earlier agreed. Doing so would provide both NASA and Congress with markers of the project’s progress at key points. For example, at the preliminary design review, decision makers would be able to assess the status of the project by using the marker of technology maturity. In addition, at the critical design review, the agency could assess the status of the project using design stability (i.e., a high percentage of engineering drawings completed). If NASA has not demonstrated technology maturity at the preliminary design review or design stability at the critical design review, decision makers would have an indication that the project will likely be headed for trouble. Without such knowledge, NASA cannot be confident that its decisions about continued investments in projects are based upon the appropriate knowledge. Furthermore, NASA’s oversight committees could also use the information when debating the agency’s yearly budget and authorizing funds not only for the CEV project, but also for making choices among NASA’s many competing programs. If provided this type of information from NASA about its key projects, Congress will be in a better position to make informed decisions about how to invest the nation’s limited discretionary funds.

NASA’s ability to address a number of long-standing financial management challenges could also impact management of NASA’s key projects. The lack of reliable, day-to-day information continues to threaten NASA’s ability to manage its programs, oversee its contractors, and effectively allocate its budget across numerous projects and programs. To its credit, NASA has recognized the need to enhance the capabilities and improve the functioning of its core financial management system, however, progress has been slow. NASA contract management has been on GAO’s high-risk list since 1990 because of such concerns.

Conclusions

In conclusion, implementing the Vision over the coming decades will require hundreds of billions of dollars and a sustained commitment from multiple administrations and Congresses. The realistic identification of the resources needed to achieve the agency’s short-term goals would provide support for such a sustained commitment over the long term. With a range of federal commitments binding the fiscal future of the United States, competition for resources within the federal government will only increase over the next several decades. Consequently, it is incumbent upon NASA to ensure that it is wisely investing its existing resources. As NASA proceeds with its acquisition strategy for the CEV project and other key projects, it will be essential that the agency ensure that the investment decisions it is making are sound and based upon high levels of knowledge. NASA should require that the progress of its projects are evaluated and reevaluated using knowledge based criteria, thereby improving the quality of decisions that will be made about which program warrant further investment. Furthermore, it will be critical that NASA’s financial management organization delivers the kind of analysis and forward- looking information needed to effectively manage its programs and projects. Clear, strong executive leadership will be needed to ensure that these actions are carried out. Given the nation’s fiscal challenges and those that exist within NASA, the availability of significant additional resources is unlikely. NASA has the opportunity to establish a firm foundation for its entire exploration program by ensuring that the level of knowledge necessary to allow decision makers to make informed decisions about where continued investment is justified. Doing so will enhance confidence in the agency’s ability to finally deliver a replacement vehicle for future human space flight.

Mr. Chairman, this concludes my prepared statement. I would be pleased to respond to any questions that you or other Members of the Committee may have.

GAO Contact and Staff Acknowledgements

For further information regarding this testimony, please contact Allen Li at (202) 512-4841 or lia@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. GAO staff who made key contributions to this testimony include Greg Campbell, Richard Cederholm, Hillary Loeffler, James L. Morrison, Jeffrey M. Niblack, and Shelby S. Oakley.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.