NASA: Ares I and Orion Project Risks and Key Indicators to Measure Progress
Released: 2008-04-03T00:00:00

Letter

Background

In September 2005, NASA outlined an initial framework for implementing the President’s Vision for Space Exploration in its Exploration Systems Architecture Study. NASA is now implementing the recommendations from this study within the Constellation Program, which includes three major development projects—the Ares I Crew Launch Vehicle, the Orion Crew Exploration Vehicle, and the Ares V Cargo Launch Vehicle as shown in figure 1.

To reduce cost and minimize risk in developing these projects, NASA planned to maximize the use of heritage systems and technology. Since 2005, however, NASA has made changes to the basic architecture for the Ares I and Orion designs that have resulted in the diminished use of heritage systems. This is due to the ability to achieve greater cost savings with alternate technology and the inability to recreate heritage technology. For example, the initial design was predicated on using the main engines and the solid rocket boosters from the Space Shuttle Program. However, NASA is no longer using the Space Shuttle Main Engines because greater long-term cost savings are anticipated through the use of the J-2X engine. In another example, NASA increased the number of segments on the Ares I first-stage reusable solid rocket booster from four to five to increase commonality between the Ares I and Ares V, and eliminate the need to develop, modify, and certify both a four-segment reusable solid rocket booster and an expendable Space Shuttle main engine for the Ares I. Finally, according to the Orion program executive the Orion project originally intended to use the heat shield from the Apollo program as a fallback technology for the Orion thermal protection system, but was unable to recreate the Apollo material.

NASA has authorized the Ares I and Orion projects to proceed with awarding development contracts. In April 2006, NASA awarded a $1.8 billion contract for design, development, test, and evaluation of the Ares I first stage to Alliant Techsystems. NASA also awarded a $1.2 billion contract for design, development, test, and evaluation of the Ares I upper stage engine—the J-2X—to Pratt and Whitney Rocketdyne in June 2006. NASA is developing the upper stage and the upper stage instrument unit, which contains the control systems and avionics for the Ares I, in-house. However, NASA awarded a $514.7 million contract for design support and production of the Ares I upper stage to the Boeing Company in August 2007. In August 2006, NASA awarded Lockheed Martin a $3.9 billion contract to design, test, and build the Orion crew exploration vehicle. According to NASA, the contract was modified in April 2007, namely by adding 2 years to the design phase and two test flights of Orion's launch abort system and by deleting the production of an cargo variant for the International Space Station. NASA indicates that these changes increased the contract value to $4.3 billion. Federal procurement data shows that an additional modification has been signed which increased the value of the contract by an additional $59 million.

NASA has completed or is in the process of completing key reviews on both the Ares I and Orion projects. NASA has completed the system requirements review for each project and is in the midst of finalizing the system definition reviews. At the systems requirements review, NASA establishes a requirements baseline that serves as the basis for ongoing design analysis work and systems testing. Systems definition reviews focus on emerging designs for all transportation elements and compare the predicted performance of each element against the currently baselined requirements. Figure 2 shows the timeline for Ares I and Orion critical reviews.

NASA is using its Web-based Integrated Risk Management Application to help monitor and mitigate the risks with the Ares I and Orion development efforts and for the overall Constellation Program. The risk management application identifies and documents risks, categorizes risks—as high, medium, and low based on both the likelihood of an undesirable event as well as the consequences of that event to the project—and tracks performance against mitigation plans. For the Ares I project, the application is tracking 101 risks, 36 of which are considered high-risk areas. For the Orion project, NASA is tracking 193 risks, including 71 high- risk areas. NASA is developing and implementing plans to mitigate some of these risks.

Requirements Setting is a Primary Challenge for Both the Ares I and Orion Projects

Although project level requirements were baselined at both systems requirements reviews, continued uncertainty about the systems’ requirements have led to considerable unknowns as to whether NASA’s plans for the Ares I and Orion vehicles can be executed within schedule goals, as well as what these efforts will ultimately cost. Such uncertainty has created knowledge gaps that are affecting many aspects of both projects. Because the Orion vehicle is the payload that the Ares I must deliver to orbit, changes in the Orion design, especially those that affect weight, directly affect Ares I lift requirements. Likewise, the lift capacity of the Ares I drives the Orion design. Both the Orion and Ares I vehicles have a history of weight and mass growth, and NASA is still defining the mass, loads, and weight requirements for both vehicles. According to agency officials, continuing weight growth led NASA to rebaseline the Orion vehicle design in fall 2007. This process involved “scrubbing” the Orion Vehicle to establish a zero-based design capable of meeting minimal mission requirements but not safe for human flight. Beginning with the zero-based design NASA first added back the systems necessary to ensure crew safety and then conducted a series of engineering trade-offs to determine what other systems should be included to maximize the probability of mission success while minimizing the system’s weight. As a result of these trade-offs, NASA modified the requirement for nominal landing on land to nominal landing in water, thereby gaining 1500 lbs of trade space in the Orion design.

NASA recognizes that continued weight growth and requirements instability are key risks facing the Orion project and that continued instability in the Orion design is a risk facing the Ares I project. The Ares I and Orion projects are working on these issues but have not yet finalized requirements or design. Our previous work on systems acquisition work shows that the preliminary design phase is an appropriate place to conduct systems engineering to support requirement and resource trade- off decisions. For the Ares I project, this phase is scheduled to be completed in August 2008, whereas for the Orion project, it is September 2008—leaving NASA only 4 and 5 months respectively to close gaps in requirements knowledge. NASA will be challenged to close such gaps, given that it is still defining requirements at a relatively high level and much work remains to be done at the lower levels. Moreover, given the complexity of the Orion and Ares I efforts and their interdependencies, as long as requirements are in flux, it will be extremely difficult to establish firm cost estimates and schedule baselines.

Technology and Hardware Gaps Along with Requirements Uncertainty are Increasing Risk

Currently, nearly every major segment of Ares I and Orion faces knowledge gaps in the development of required hardware and technology and many are being affected by uncertainty in requirements. For example, computer modeling is showing that thrust oscillation within the first stage of the Ares I could cause excessive vibration throughout the Ares I and Orion. Resolving this issue could require redesigns to both the Ares I and Orion vehicles that could ultimately impact cost, schedule, and performance. Furthermore, the addition of a fifth segment to the Ares I first stage has the potential to impact qualification efforts for the first stage and could result in costly requalification and redesign efforts. Additionally, the J-2X engine represents a new engine development effort that, both NASA and Pratt and Whitney Rocketdyne recognize, is likely to experience failures during development. Addressing these failures is likely to lead to design changes that could impact the project's cost and schedule. With regard to the Orion project, there is currently no industry capability for producing a thermal protection system of the size required by the Orion. NASA has yet to develop a solution for this gap, and given the size of the vehicle and the tight development schedule, a feasible thermal protection system may not be available for initial operational capability to the space station. The table 1 describes these and other examples of knowledge gaps in the development of the Ares I and Orion vehicles.

Constellation Cost Estimates Are Preliminary Due to Requirements Uncertainty

NASA’s preliminary cost estimates for the Constellation Program are likely to change when requirements are better defined. NASA will establish a preliminary estimate of life cycle costs for the Ares I and Orion in support of each project’s system definition review. A formal baseline of cost, however, is not expected until the projects’ preliminary design reviews are completed. NASA is working under a self-imposed deadline to deliver the new launch vehicles no later than 2015 in order to minimize the gap in human spaceflight between the Space Shuttle’s retirement in 2010 and the availability of new transportation vehicles. The Constellation Program’s budget request maintains a confidence level of 65 percent (i.e., NASA is 65 percent certain that the actual cost of the program will either meet or be less than the estimate) for program estimates based upon a 2015 initial operational capability. Internally, however, the Ares I and Orion projects are working toward an earlier initial operational capability (2013), but at a reduced budget confidence level—33 percent. However, NASA cannot reliably estimate the money needed to complete technology development, design, and production for the Ares I and Orion projects until requirements are fully understood.

NASA has identified the potential for a life cycle cost increase as a risk for the Orion program. According to NASA’s risk database, given the historical cost overruns of past NASA systems and the known level of uncertainty in the current Orion requirements, there is a possibility that Orion's life cycle cost estimate may increase over time. NASA acknowledges that such increases are often caused by the unknown impacts of decisions made during development. One factor currently contributing to cost increases is the addition of new requirements. NASA is working to formulate the best life cycle cost estimate possible during development, is identifying and monitoring costs threats, and is implementing management tools all aimed at addressing this risk.

Schedule Pressures Add Additional Risks for Ares I and Orion

There are considerable schedule pressures facing both the Ares I and Orion projects. These are largely rooted in NASA’s desire to minimize the gap between the retirement of the space shuttle and availability of the new vehicles. Because of this scheduling goal, NASA is planning to conduct many interdependent development activities concurrently—meaning if one activity should slip in schedule, it could have cascading effects on other activities. Moreover, some aspects of the program are already experiencing scheduling delays due to the fact that high-level requirements are still being defined.

Ares I

The development schedule for the J-2X is aggressive, allowing less than 7 years from development start to first flight, and highly concurrent. Due to the tight schedule and long-lead nature of engine development, the J-2X project was required to start out earlier in its development than the other elements on the Ares I vehicle. This approach has introduced a high degree of concurrency between the setting of overall Ares I requirements and the development of the J-2X design and hardware. Consequently, the engine development is out of sync with the first stage and upper stage in the flow-down and decomposition of requirements, an approach our past work has shown to be fraught with risk. NASA acknowledges that the engine development is proceeding with an accepted risk that future requirements changes may affect the engine design and that the engine may not complete development as scheduled in December 2012. The J-2X development effort represents a critical path for the Ares I project. Subsequently, delays in the J-2X schedule for design, development, test, and evaluation would have a ripple effect of cost and schedule impacts throughout the entire Ares I project.

The schedule for the first stage also presents a potential issue for the entire Ares I project. Specifically, the critical design review for the first stage is out of sync with the Ares I project-level critical design review. NASA has scheduled two critical design reviews for the first stage. The first critical design review is scheduled for November 2009, 5 months before the Ares I project critical design review. At this point, however, the project will not have fully tested the first stage development motors. The second critical design review, in December 2010, occurs after additional testing of developmental motors is conducted. By conducting the Ares I critical design review before the first stage project critical design review, the project could prematurely begin full-scale test and integration activities a full 9 months before the first stage design has demonstrated maturity. If problems are found in the first stage design during the later testing, implementing solutions could result in costly rework and redesign and delay the overall project schedule.

Orion

Cost and schedule reporting on the Orion project indicates that the Orion project’s efforts to mature requirements and design and to resolve weight issues is placing pressure on the Orion schedule. Specifically, activities aimed at assessing alternate designs to reduce overall vehicle mass, rework to tooling concepts, and late requirements definition have contributed to the project falling behind schedule. Further, the Orion risk system indicates that schedule delays associated with testing may occur. The current Orion design has high predicted vibration and acoustic levels. Historically, components designed and qualified for uncertain vibration and acoustic environments have resulted in some failures and required subsequent redesign and retest. Failures during qualification testing of Orion components may lead to schedule delays associated with redesigning components.

NASA’s Administrator has publicly stated that if Congress provided the Agency an additional $2 billion that NASA could accelerate the Constellation program’s initial operational capability date to 2013. We believe that this assessment is highly optimistic. The development schedule for the J-2X engine, the critical path for the Ares I development, is already recognized as aggressive, allowing less than 7 years for development. The development of the Space Shuttle Main engine by comparison took 9 years. Further, NASA anticipates that the J-2X engine is likely to require 29 rework cycles to correct problems identified during testing. Given the linear nature of a traditional test-analyze-fix-test cycle, even large funding increases offer no guarantee of program acceleration, particularly when the current schedule is already compressed and existing NASA test facilities are already maximized.

Test Facilities for Ares I and Orion Insufficient

According to NASA, at this time, existing test facilities are insufficient to adequately test the Ares I and Orion systems. Existing altitude test facilities are insufficient to test the J-2X engine in a relevant environment. To address this issue, NASA is in the process of constructing a new altitude test facility at Stennis Space Center for the J-2X. Also, current facilities are inadequate to replicate the Orion vibration and acoustic environment. Further, Pratt and Whitney Rocketdyne—the J-2 X upper stage engine contractor—indicated that existing test stands that could support J-2X testing will be tied up supporting the Space Shuttle program until 2010. NASA has taken steps to mitigate J-2X risks by increasing the amount of component-level testing, procuring additional development hardware and test facilities, and working to make a third test stand available to the contractor earlier than originally planned. NASA has compensated for this schedule pressure on the Ares I project by adding funds for testing and other critical activities. But it is not certain that added resources will enable NASA to deliver the Ares I when expected.

With respect to Orion’s thermal protection system, facilities available from the Apollo era for testing large-scale heat shields no longer exist. Therefore, NASA must rely on two facilities that fall short in providing the necessary capability and scheduling to test ablative materials needed for Orion. Additionally, NASA has no scheduled test to demonstrate the thermal protection system needed for lunar missions. NASA is exploring other options, including adding a lunar return flight test and building a new improved test facility. Due to the scheduled first lunar flight, any issues identified during such testing would need to be addressed in the time between the flight test and the first flight.

Oversight Based on Best Practices and Key Indicators Important for Program Success

NASA is poised to invest a significant amount of resources to implement the Vision over the long term and specifically to develop the Ares I and Orion projects over the next several years. Accordingly, you asked us to articulate indicators that Congress could use to assess progress. Our prior work has shown that investment decisions of this magnitude need to be based on an established and executable business case and that there are several key indicators that Congress could be informed of to assess progress throughout development. These include areas commonly underestimated in space programs, such as weight growth and software complexity, as well as indicators used by best practice organizations to assess readiness to move forward in the development cycle. Space programs which we have studied in detail in the past have tended to underestimate cost in some of these areas.

Our previous work on government-funded space systems has shown that weight growth is often not anticipated even though it is among the highest drivers of cost growth for space systems. Weight growth can affect the hardware needed to support a system, and, in the case of launch vehicles, the power or thrust required for the system. As the weight of a particular system increases, the power or thrust required for that system will also increase. This could result in the need to develop additional power or thrust capability to lift the system, leading to additional costs, or to stripping down the vehicle to accommodate current power or thrust capability. For example, NASA went through the process to zero-base the design for the Orion to address weight concerns. Continual monitoring of system weight and required power/thrust, as well as margins or reserves for additional growth, can provide decision makers with an indicator of whether cost increases can be anticipated.

The complexity of software development on a system, often denoted by the number of lines of code on a system, can also be used as an indicator to monitor whether a program will meet cost and schedule goals. In our work on software development best practices, we have reported that the Department of Defense has attributed significant cost and schedule overruns on software-intensive systems to developing and delivering software. Generally, the greater the number of lines of code, the more complicated the system development. Changes to the amount of code needed to be produced can indicate potential cost and schedule problems. Decision makers can monitor this indicator by continually asking for information on the estimated amount of code needed on a system and inquiring about any increases in need and their impact on cost and schedule.

There are other areas, such as the use of heritage systems and industrial base capability that are commonly underestimated in space programs as well. However, weight increases and software growth are more quantifiable and thus useful for oversight purposes.

Indicators that Can be Used to Assess Knowledge Gap at Key Junctures Additionally, since the mid-1990s, GAO has studied the best practices of leading commercial companies. On the basis of this information, and taking into account the differences between commercial product development and major federal acquisitions, we have outlined a best practices product development model—known as a knowledge-based approach to system development. This type of approach calls for investment decisions to be made on the basis of specific, measurable levels of knowledge at critical junctures before investing more money and proceeding with development.

Importantly, our work has shown the most leveraged decision point is matching the customer’s needs with the developer’s resources (time, dollars, technology, people, etc.) because it sets the stage for the eventual outcome—desirable or problematic. The match is ultimately achieved in every development program, but in successful development programs, it occurs before product development is formally initiated (usually the preliminary design review). If the knowledge attained at this and other critical junctures does not confirm the business case on which the acquisition was originally justified, the best practice organizations we have studied do not allow the program to go forward.

We have highlighted the three critical junctures at which developers must have knowledge to make large investment decisions—the preliminary design review, the critical design review, and the production review—and the numerous key indicators that can be used to increase the chances of successful outcomes.

In assessing the Orion and Ares programs, the Congress and NASA decision-makers can use these indicators in order to reliably gauge whether there is a sufficient business case for allowing the programs to proceed forward.

Preliminary design review: Before product development is started, a match must be made between the customers’ needs and the available resources—technical and engineering knowledge, time, and funding. To provide oversight at this juncture, NASA could provide Congress with information to verify that the following have indicators been met: All critical technologies are demonstrated to a high level of technology maturity, that is demonstrated that they can perform in a realistic or, more preferably, operational environment. A technology readiness level 6 or 7 would indicate that this has been achieved. One approach to ensure that technology readiness is reliably assessed is to use independent testing; Project requirements are defined and informed by the systems Cost and schedule estimates established for the project are based on knowledge from the preliminary design using systems engineering tools; Additional resources are in place, including needed workforce, and a decision review is conducted following completion of the preliminary design review.

A critical enabler for success in this phase of development is performance and requirements flexibility. Customers and product developers both need to be open to reducing expectations, deferring them to future programs, or to investing more resources up front to eliminate gaps between resources and expectations. In successful programs we have studied, requirements were flexible until a decision was made to commit to product development because both customers and developers wanted to limit cycle time. This makes it acceptable to reduce, eliminate, or defer some customer wants so that the product’s requirements could be matched with the resources available to deliver the product within the desired cycle time.

Critical design review: A product’s design must demonstrate its ability to meet performance requirements and be stable about midway through development. To provide oversight at this juncture, NASA could provide Congress with information to verify that the following indicators have been met: At least 90 percent of engineering drawings are complete; All subsystem and system design reviews have been completed; The design meets requirements demonstrated through modeling, simulation, or prototypes; Stakeholders’ concurrence that drawings are complete and producible Failure modes and effects analysis have been completed; Key system characteristics are identified; Critical manufacturing processes are identified; Reliability targets are established and a growth plan based on demonstrated reliability rates of components and subsystems is developed; and A decision review is conducted following the completion of the critical design review.

Production Review: The developer must show that the product can be manufactured within cost, schedule, and quality targets and is demonstrated to be reliable before production begins. To provide oversight at this juncture, NASA could provide Congress with information to verify that the following indicators have been met: Manufacturing processes have been demonstrated; Production representative prototypes have been built; Production representative prototypes have been tested and have Production representative prototypes have been demonstrated in an operational environment through testing; Statistical process control data have been collected; Critical processes have been demonstrated to be capable and that they are in statistical control; A decision review is conducted following completion of the production readiness review.

Over the past 2 years, we have recommended that NASA incorporate a knowledge-based approach in its policies and take steps to implement this type of approach in its programs and projects. NASA has incorporated some knowledge-based concepts into its acquisition policies. For example, NASA now requires a decision review between each major phase of the acquisition life cycle and has established general entrance and success criteria for the decision reviews. In addition, we have reported that this type of approach is being embraced by the Ares I project.

Concluding Observations

In conclusion, the President’s Vision for Space Exploration is an ambitious effort, not just because there will be technical and design challenges to building systems needed to achieve the Vision’s goals, but because there are limited resources within which this can be accomplished. Moreover, the long-term nature of the Vision means that commitments for funding and to the goals of the Vision will need to be sustained across presidential administrations and changes in congressional leadership. For these reasons, it is exceedingly important that the right decisions are made early on and that decision-makers have the right knowledge going forward so that they can make informed investment decisions.

In looking at the first major investments, the Ares I and Orion projects, it is important to recognize that they are risky endeavors, largely due to their complexity, scope, and interdependencies. It is also important to recognize that the desire to minimize the gap in human space flight adds considerable risk, since it could limit NASA's ability to study emerging problems and pursue alternative ways of addressing them. For these reasons, as well as the magnitude of investment at stake, it is imperative that NASA be realistic and open about the progress it is making and to be willing to make changes to the architecture and design if technical problems can not be solved without overly compromising performance. Additionally, Congress needs to be well-informed about the extent to which knowledge gaps remain and what tradeoffs or additional resources are needed to close those gaps and to support changes if they are determined to be necessary. The upcoming preliminary design review milestones represent perhaps the most critical juncture where these assessments can take place and where hard decisions can be made as to whether the programs should proceed forward. It may well be the last opportunity to make significant adjustments before billions of dollars are spent and long term commitments become solidified.

Mr. Chairman, this concludes my prepared statement. I would be pleased to answer any questions that you may have at this time.

GAO Contacts and Staff Acknowledgements

For further questions about this statement, please contact Cristina T. Chaplain at (202) 512-4841. Individuals making key contributions to this statement include James L. Morrison, Meredith A. Kimmitt, Lily Chin, Neil Feldman, Rachel Girshick, Shelby S. Oakley, and John S. Warren, Jr.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.