NASA: Ares I and Orion Project Risks and Key Indicators to Measure Progress
Published: 2008-04-03T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Aeronautics and Space Administration (NASA) is in the midst of two new development efforts as part of the Constellation Program--the Ares I Crew Launch Vehicle and the Orion Crew Exploration Vehicle. These projects are critical to the success of the overall program, which will return humans to spaceflight after Space Shuttle retirement in 2010. To reduce the gap in human spaceflight, NASA plans to launch Ares I and Orion in 2015--5 years after the Shuttle's retirement. GAO has issued a number of reports and testimonies that touch on various aspects of NASA's Constellation Program, particularly the development efforts underway for the Orion and Ares I projects. These reports and testimonies have questioned the affordability and overall acquisition strategy for each project. NASA has revised the Orion acquisition strategy and delayed the Ares I preliminary design review based on GAO's recommendations in these reports. In addition, GAO continues to monitor these projects on an ongoing basis at the request of members of Congress. Based on this work, GAO was asked to testify on the types of challenges that NASA faces in developing the Ares I and Orion vehicles and identify the key indicators that decision makers could use to assess risks associated with common trouble spots in development. The information in this testimony is based on work completed in accordance with generally accepted government auditing standards.
section_title: What GAO Found
paragraphs: NASA is currently working toward preliminary design reviews for the Ares I and Orion vehicles. While this is a phase for discovery and risk reduction, there are considerable unknowns as to whether NASA's plans for these vehicles can be executed within schedule goals and what these efforts will ultimately cost. This is primarily because NASA is still in the process of defining many performance requirements. Such uncertainties could affect the mass, loads, and weight requirements for the vehicles. NASA is aiming to complete this process in 2008, but it will be challenged to do so given the level of knowledge that still needs to be attained. The challenges NASA is facing pose risks to the successful outcome of the projects. For example: both vehicles have a history of weight issues; excessive vibration during launch threatens system design; Uncertainty about how flight characteristics will be impacted by a fifth segment added to the Ares I launch vehicle; Ares I upper stage essentially requires development of a new engine; no industry capability currently exists for producing the kind of heat shields that the Orion will need for protecting the crew exploration vehicle when it reenters Earth's atmosphere; and existing test facilities are insufficient for testing Ares I's new engine, for replicating the engine's vibration and acoustic environment, and for testing the thermal protection system for the Orion vehicle. All these unknowns, as well as others, leave NASA in the position of being unable to provide firm cost estimates for the projects at this point. Meanwhile, tight deadlines are putting additional pressure on both the Ares I and Orion projects. Future requirements changes raise risks that both projects could experience cost and schedule problems. GAO's past work on space systems acquisition and the practices of leading developers identifies best practices that can provide decision makers with insight into the progress of development at key junctures, facilitate Congressional oversight, and support informed decision making. This work has also identified common red flags throughout development, which decision makers need to keep in mind when assessing the projects. They include: Key indicators: Weight growth is often among the highest drivers of cost growth. Unanticipated software complexity, often indicated by increases in the number of lines of code, can portend cost and schedule growth. Key junctures: The preliminary design review, critical design review, and production review are key junctures that involve numerous steps and help focus the agency on realistic accomplishments within reachable goals. A disciplined approach aligned with key indicators can provide the knowledge needed to make informed investment decisions at each review.

section_title: Letter
paragraphs: I am pleased to be here today to discuss challenges that the National  Aeronautics and Space Administration (NASA) faces in developing the  systems to achieve its goals for the President’s Vision for Space  Exploration. We have been focusing our work primarily on the Ares I  Crew Launch Vehicle and the Orion Crew Exploration Vehicle, as they are  among the first major efforts conducted as part of NASA’s Constellation  Program to support implementation of the Vision and represent a  substantial investment for NASA. Over $7 billion in contracts has already  been awarded—and nearly $230 billion is estimated to be ultimately spent  over the next two decades. Moreover, NASA is under pressure to develop  the vehicles quickly, as the Space Shuttle’s retirement in 2010 means that  there could be at least a 5-year gap in our nation’s ability to send humans  to space.
In summary, NASA is currently working toward preliminary design  reviews for the vehicles—a milestone that successful development  organizations use to make hard decisions about whether a program should  proceed with development. While this is a phase for discovery and risk  reduction, there are considerable unknowns as to whether NASA’s plans  for the Ares I and Orion vehicles can be executed within schedule goals, as  well as what these efforts will ultimately cost. In fact, we do not know yet  whether the architecture and design solutions selected by NASA will work  as intended. This is primarily because NASA is still in the process of  defining both of the projects’ performance requirements and some of these  uncertainties could affect the mass, loads, and weight requirements for the  vehicles. It is also working through significant technical risks, such as  oscillation within the first stage of the Ares I vehicle, which computer  modeling indicates could cause unacceptable structural vibrations.
NASA is aiming to complete preliminary design reviews for the Ares I and  Orion this year, scheduled for August 2008 Ares I and September 2008  respectively, but it will be challenged in doing so given the level of  knowledge that still needs to be attained. In addition, to minimize the gap  in human spaceflight caused by the shuttle’s retirement, there is a high  degree of concurrency within the projects. Our prior work has shown that  concurrent development, especially when new technologies are involved,  increases the risk that significant problems will be discovered as the  systems’ designs are integrated that could result in cost and schedule  delays. NASA’s schedule leaves little room for the unexpected. If  something goes wrong with the development of the Ares I or the Orion, the  entire Constellation Program could be thrown off course and the return to  human spaceflight delayed.
NASA recognizes the risks involved with its approach and has taken steps  to mitigate some of these risks.  It is important that, in mitigating risks,  NASA continually assess the viability of its plans for the Ares I and Orion.  The current state of play requires that NASA remain open to the possibility  that it may need to revisit decisions on its architecture and design as these  vehicles are expected to be in use for decades to come and decisions made  now will have long-term consequences.
Moreover, with additional significant investment decisions still ahead, it is  important that agency decision makers and Congress maintain clear  insight into the progress the projects are making as well as any potential  problems. This type of oversight is important, not just for the Ares I and  Orion vehicles, but for the entire future exploration effort—since  resources available to fund the Vision are constrained, as competition for  resources increases within the federal government over the next several  decades. In this regard, our work has identified specific markers that can  be used to (1) assess NASA’s progress in closing critical knowledge gaps  and (2) identify issues that could result in cost growth, schedule delays, or  decreased performance. In other words, they can be used to assess  whether there is a viable business case for pressing forward with the  projects.
We have issued a number of reports and testimonies that touch on various  aspects of NASA’s Constellation Program and in particular the  development efforts underway for the Orion and Ares I projects. These  reports and testimonies have questioned the affordability and overall  acquisition strategy for each project. In July 2006 we recommended that  NASA modify the Orion Crew Vehicle acquisition strategy to ensure the  agency did not commit itself to a long-term contractual obligation prior to  establishing a sound business case.  Although initially NASA disagreed  with our recommendation, the agency subsequently revised its acquisition  strategy to address some of the concerns we raised. In October 2007 we  recommended that NASA develop a sound business case supported by  firm requirements, mature technologies, a preliminary design, a realistic  cost estimate, and sufficient funding and time—before proceeding beyond  preliminary design review.  NASA concurred with this recommendation  and subsequently slipped the Ares I preliminary design review from July  2008 to August 2008.
My statement today is based on these products, as well as updated  information based on our continual monitoring of the projects at the  request of members of Congress.  To conduct these reviews, we analyzed  relevant project documentation, prior GAO reports, NASA documents, and  contractor information; interviewed program and project officials; and  reviewed NASA’s risk management system for the Constellation Program.  Based on this work, my statement will specifically address the challenges  that NASA faces developing the Ares I and Orion vehicles with regard to  requirements definition, technology and hardware gaps, cost and schedule  estimates, and facilities needs. Further, I will provide key indicators that  decision makers could use to assess risks as the two development efforts  move forward. We conducted this performance audit from October 2007  through April 2008 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives.  We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In September 2005, NASA outlined an initial framework for implementing  the President’s Vision for Space Exploration in its Exploration Systems  Architecture Study.  NASA is now implementing the recommendations  from this study within the Constellation Program, which includes three  major development projects—the Ares I Crew Launch Vehicle, the Orion  Crew Exploration Vehicle, and the Ares V Cargo Launch Vehicle as shown  in figure 1.
To reduce cost and minimize risk in developing these projects, NASA  planned to maximize the use of heritage systems and technology. Since  2005, however, NASA has made changes to the basic architecture for the  Ares I and Orion designs that have resulted in the diminished use of  heritage systems. This is due to the ability to achieve greater cost savings  with alternate technology and the inability to recreate heritage technology.  For example, the initial design was predicated on using the main engines  and the solid rocket boosters from the Space Shuttle Program. However,  NASA is no longer using the Space Shuttle Main Engines because greater  long-term cost savings are anticipated through the use of the J-2X engine.  In another example, NASA increased the number of segments on the Ares I  first-stage reusable solid rocket booster from four to five to increase  commonality between the Ares I and Ares V, and eliminate the need to  develop, modify, and certify both a four-segment reusable solid rocket  booster and an expendable Space Shuttle main engine for the Ares I.  Finally, according to the Orion program executive the Orion project  originally intended to use the heat shield from the Apollo program as a  fallback technology for the Orion thermal protection system, but was  unable to recreate the Apollo material.
NASA has authorized the Ares I and Orion projects to proceed with  awarding development contracts.  In April 2006, NASA awarded a $1.8  billion contract for design, development, test, and evaluation of the Ares I  first stage to Alliant Techsystems. NASA also awarded a $1.2 billion  contract for design, development, test, and evaluation of the Ares I upper  stage engine—the J-2X—to Pratt and Whitney Rocketdyne in June 2006.  NASA is developing the upper stage and the upper stage instrument unit,  which contains the control systems and avionics for the Ares I, in-house.  However, NASA awarded a $514.7 million contract for design support and  production of the Ares I upper stage to the Boeing Company in August  2007. In August 2006, NASA awarded Lockheed Martin a $3.9 billion  contract to design, test, and build the Orion crew exploration vehicle.  According to NASA, the contract was modified in April 2007, namely by  adding 2 years to the design phase and two test flights of Orion's launch  abort system and by deleting the production of an cargo variant for the  International Space Station. NASA indicates that these changes increased  the contract value to $4.3 billion. Federal procurement data shows that an  additional modification has been signed which increased the value of the  contract by an additional $59 million.
NASA has completed or is in the process of completing key reviews on  both the Ares I and Orion projects. NASA has completed the system  requirements review for each project and is in the midst of finalizing the  system definition reviews. At the systems requirements review, NASA  establishes a requirements baseline that serves as the basis for ongoing  design analysis work and systems testing.  Systems definition reviews  focus on emerging designs for all transportation elements and compare the  predicted performance of each element against the currently baselined  requirements. Figure 2 shows the timeline for Ares I and Orion critical  reviews.
NASA is using its Web-based Integrated Risk Management Application to  help monitor and mitigate the risks with the Ares I and Orion development  efforts and for the overall Constellation Program. The risk management  application identifies and documents risks, categorizes risks—as high,  medium, and low based on both the likelihood of an undesirable event as  well as the consequences of that event to the project—and tracks  performance against mitigation plans. For the Ares I project, the  application is tracking 101 risks, 36 of which are considered high-risk  areas. For the Orion project, NASA is tracking 193 risks, including 71 high- risk areas. NASA is developing and implementing plans to mitigate some  of these risks.
section_title: Requirements Setting is a Primary Challenge for Both the Ares I and Orion Projects
paragraphs: Although project level requirements were baselined at both systems  requirements reviews, continued uncertainty about the systems’  requirements have led to considerable unknowns as to whether NASA’s  plans for the Ares I and Orion vehicles can be executed within schedule  goals, as well as what these efforts will ultimately cost. Such uncertainty  has created knowledge gaps that are affecting many aspects of both  projects. Because the Orion vehicle is the payload that the Ares I must  deliver to orbit, changes in the Orion design, especially those that affect  weight, directly affect Ares I lift requirements.  Likewise, the lift capacity  of the Ares I drives the Orion design. Both the Orion and Ares I vehicles  have a history of weight and mass growth, and NASA is still defining the  mass, loads, and weight requirements for both vehicles.  According to  agency officials, continuing weight growth led NASA to rebaseline the  Orion vehicle design in fall 2007.  This process involved “scrubbing” the  Orion Vehicle to establish a zero-based design capable of meeting minimal  mission requirements but not safe for human flight. Beginning with the  zero-based design NASA first added back the systems necessary to ensure  crew safety and  then conducted a series of engineering trade-offs to  determine what other systems should be included to maximize the  probability of mission success while minimizing the system’s weight.  As a  result of these trade-offs, NASA modified the requirement for nominal  landing on  land to nominal landing in water, thereby gaining 1500 lbs of  trade space in the Orion design.
NASA recognizes that continued weight growth and requirements  instability are key risks facing the Orion project and that continued  instability in the Orion design is a risk facing the Ares I project. The Ares I  and Orion projects are working on these issues but have not yet finalized  requirements or design. Our previous work on systems acquisition work  shows that the preliminary design phase is an appropriate place to  conduct systems engineering to support requirement and resource trade- off decisions. For the Ares I project, this phase is scheduled to be  completed in August 2008, whereas for the Orion project, it is September  2008—leaving NASA only 4 and 5 months respectively to close gaps in  requirements knowledge. NASA will be challenged to close such gaps,  given that it is still defining requirements at a relatively high level and  much work remains to be done at the lower levels. Moreover, given the  complexity of the Orion and Ares I efforts and their interdependencies, as  long as requirements are in flux, it will be extremely difficult to establish  firm cost estimates and schedule baselines.
section_title: Technology and Hardware Gaps Along with Requirements Uncertainty are Increasing Risk
paragraphs: Currently, nearly every major segment of Ares I and Orion faces  knowledge gaps in the development of required hardware and technology  and many are being affected by uncertainty in requirements. For example,  computer modeling is showing that thrust oscillation within the first stage  of the Ares I could cause excessive vibration throughout the Ares I and  Orion.  Resolving this issue could require redesigns to both the Ares I and  Orion vehicles that could ultimately impact cost, schedule, and  performance. Furthermore, the addition of a fifth segment to the Ares I  first stage has the potential to impact qualification efforts for the first  stage and could result in costly requalification and redesign efforts.  Additionally, the J-2X engine represents a new engine development effort  that, both NASA and Pratt and Whitney Rocketdyne recognize, is likely to  experience failures during development. Addressing these failures is likely  to lead to design changes that could impact the project's cost and  schedule. With regard to the Orion project, there is currently no industry  capability for producing a thermal protection system of the size required  by the Orion. NASA has yet to develop a solution for this gap, and given  the size of the vehicle and the tight development schedule, a feasible  thermal protection system may not be available for initial operational  capability to the space station. The table 1 describes these and other  examples of knowledge gaps in the development of the Ares I and Orion  vehicles.
section_title: Constellation Cost Estimates Are Preliminary Due to Requirements Uncertainty
paragraphs: NASA’s preliminary cost estimates for the Constellation Program are likely  to change when requirements are better defined. NASA will establish a  preliminary estimate of life cycle costs for the Ares I and Orion in support  of each project’s system definition review. A formal baseline of cost,  however, is not expected until the projects’ preliminary design reviews are  completed. NASA is working under a self-imposed deadline to deliver the  new launch vehicles no later than 2015 in order to minimize the gap in  human spaceflight between the Space Shuttle’s retirement in 2010 and the  availability of new transportation vehicles. The Constellation Program’s  budget request maintains a confidence level of 65 percent (i.e., NASA is 65  percent certain that the actual cost of the program will either meet or be  less than the estimate) for program estimates based upon a 2015 initial  operational capability. Internally, however, the Ares I and Orion projects  are working toward an earlier initial operational capability (2013), but at a  reduced budget confidence level—33 percent. However, NASA cannot  reliably estimate the money needed to complete technology development,  design, and production for the Ares I and Orion projects until  requirements are fully understood.
NASA has identified the potential for a life cycle cost increase as a risk for  the Orion program. According to NASA’s risk database, given the historical  cost overruns of past NASA systems and the known level of uncertainty in  the current Orion requirements, there is a possibility that Orion's life cycle  cost estimate may increase over time.  NASA acknowledges that such  increases are often caused by the unknown impacts of decisions made  during development. One factor currently contributing to cost increases is  the addition of new requirements. NASA is working to formulate the best  life cycle cost estimate possible during development, is identifying and  monitoring costs threats, and is implementing management tools all aimed  at addressing this risk.
section_title: Schedule Pressures Add Additional Risks for Ares I and Orion
paragraphs: There are considerable schedule pressures facing both the Ares I and  Orion projects.  These are largely rooted in NASA’s desire to minimize the  gap between the retirement of the space shuttle and availability of the new  vehicles.  Because of this scheduling goal, NASA is planning to conduct  many interdependent development activities concurrently—meaning if  one activity should slip in schedule, it could have cascading effects on  other activities.  Moreover, some aspects of the program are already  experiencing scheduling delays due to the fact that high-level requirements  are still being defined.
subsections: section_title: Ares I
paragraphs: The development schedule for the J-2X is aggressive, allowing less than 7  years from development start to first flight, and highly concurrent. Due to  the tight schedule and long-lead nature of engine development, the J-2X  project was required to start out earlier in its development than the other  elements on the Ares I vehicle. This approach has introduced a high  degree of concurrency between the setting of overall Ares I requirements  and the development of the J-2X design and hardware. Consequently, the  engine development is out of sync with the first stage and upper stage in  the flow-down and decomposition of requirements, an approach our past  work has shown to be fraught with risk. NASA acknowledges that the  engine development is proceeding with an accepted risk that future  requirements changes may affect the engine design and that the engine  may not complete development as scheduled in December 2012. The J-2X  development effort represents a critical path for the Ares I project.  Subsequently, delays in the J-2X schedule for design, development, test,  and evaluation would have a ripple effect of cost and schedule impacts  throughout the entire Ares I project.
The schedule for the first stage also presents a potential issue for the  entire Ares I project. Specifically, the critical design review for the first  stage is out of sync with the Ares I project-level critical design review.  NASA has scheduled two critical design reviews for the first stage. The  first critical design review is scheduled for November 2009, 5 months  before the Ares I project critical design review. At this point, however, the  project will not have fully tested the first stage development motors. The  second critical design review, in December 2010, occurs after additional  testing of developmental motors is conducted. By conducting the Ares I  critical design review before the first stage project critical design review,  the project could prematurely begin full-scale test and integration  activities a full 9 months before the first stage design has demonstrated  maturity. If problems are found in the first stage design during the later  testing, implementing solutions could result in costly rework and redesign  and delay the overall project schedule.
section_title: Orion
paragraphs: Cost and schedule reporting on the Orion project indicates that the Orion  project’s efforts to mature requirements and design and to resolve weight  issues is placing pressure on the Orion schedule. Specifically, activities  aimed at assessing alternate designs to reduce overall vehicle mass,  rework to tooling concepts, and late requirements definition have  contributed to the project falling behind schedule.  Further, the Orion risk  system indicates that schedule delays associated with testing may occur.  The current Orion design has high predicted vibration and acoustic levels.  Historically, components designed and qualified for uncertain vibration  and acoustic environments have resulted in some failures and required  subsequent redesign and retest. Failures during qualification testing of  Orion components may lead to schedule delays associated with  redesigning components.
NASA’s Administrator has publicly stated that if Congress provided the  Agency an additional $2 billion that NASA could accelerate the  Constellation program’s initial operational capability date to 2013.  We  believe that this assessment is highly optimistic.  The development  schedule for the J-2X engine, the critical path for the Ares I development,  is already recognized as aggressive, allowing less than 7 years for  development.  The development of the Space Shuttle Main engine by  comparison took 9 years. Further, NASA anticipates that the J-2X engine is  likely to require 29 rework cycles to correct problems identified during  testing. Given the linear nature of a traditional test-analyze-fix-test cycle,  even large funding increases offer no guarantee of program acceleration,  particularly when the current schedule is already compressed and existing  NASA test facilities are already maximized.
section_title: Test Facilities for Ares I and Orion Insufficient
paragraphs: According to NASA, at this time, existing test facilities are insufficient to  adequately test the Ares I and Orion systems. Existing altitude test  facilities are insufficient to test the J-2X engine in a relevant environment.  To address this issue, NASA is in the process of constructing a new  altitude test facility at Stennis Space Center for the J-2X. Also, current  facilities are inadequate to replicate the Orion vibration and acoustic  environment. Further, Pratt and Whitney Rocketdyne—the J-2 X upper  stage engine contractor—indicated that existing test stands that could  support J-2X testing will be tied up supporting the Space Shuttle program  until 2010. NASA has taken steps to mitigate J-2X risks by increasing the  amount of component-level testing, procuring additional development  hardware and test facilities, and working to make a third test stand  available to the contractor earlier than originally planned. NASA has  compensated for this schedule pressure on the Ares I project by adding  funds for testing and other critical activities. But it is not certain that  added resources will enable NASA to deliver the Ares I when expected.
With respect to Orion’s thermal protection system, facilities available from  the Apollo era for testing large-scale heat shields no longer exist.  Therefore, NASA must rely on two facilities that fall short in providing the  necessary capability and scheduling to test ablative materials needed for  Orion. Additionally, NASA has no scheduled test to demonstrate the  thermal protection system needed for lunar missions. NASA is exploring  other options, including adding a lunar return flight test and building a  new improved test facility. Due to the scheduled first lunar flight, any  issues identified during such testing would need to be addressed in the  time between the flight test and the first flight.
section_title: Oversight Based on Best Practices and Key Indicators Important for Program Success
paragraphs: NASA is poised to invest a significant amount of resources to implement  the Vision over the long term and specifically to develop the Ares I and  Orion projects over the next several years. Accordingly, you asked us to  articulate indicators that Congress could use to assess progress. Our prior  work has shown that investment decisions of this magnitude need to be  based on an established and executable business case and that there are  several key indicators that Congress could be informed of to assess  progress throughout development. These include areas commonly  underestimated in space programs, such as weight growth and software  complexity, as well as indicators used by best practice organizations to  assess readiness to move forward in the development cycle. Space  programs which we have studied in detail in the past have tended to  underestimate cost in some of these areas.
Our previous work on government-funded space systems has shown that  weight growth is often not anticipated even though it is among the highest  drivers of cost growth for space systems. Weight growth can affect the  hardware needed to support a system, and, in the case of launch vehicles,  the power or thrust required for the system. As the weight of a particular  system increases, the power or thrust required for that system will also  increase. This could result in the need to develop additional power or  thrust capability to lift the system, leading to additional costs, or to  stripping down the vehicle to accommodate current power or thrust  capability. For example, NASA went through the process to zero-base the  design for the Orion to address weight concerns. Continual monitoring of  system weight and required power/thrust, as well as margins or reserves  for additional growth, can provide decision makers with an indicator of  whether cost increases can be anticipated.
The complexity of software development on a system, often denoted by  the number of lines of code on a system, can also be used as an indicator  to monitor whether a program will meet cost and schedule goals. In our  work on software development best practices, we have reported that the  Department of Defense has attributed significant cost and schedule  overruns on software-intensive systems to developing and delivering  software. Generally, the greater the number of lines of code, the more  complicated the system development. Changes to the amount of code  needed to be produced can indicate potential cost and schedule problems.  Decision makers can monitor this indicator by continually asking for  information on the estimated amount of code needed on a system and  inquiring about any increases in need and their impact on cost and  schedule.
There are other areas, such as the use of heritage systems and industrial  base capability that are commonly underestimated in space programs as  well. However, weight increases and software growth are more  quantifiable and thus useful for oversight purposes.
Indicators that Can be Used to Assess Knowledge Gap at Key  Junctures  Additionally, since the mid-1990s, GAO has studied the best practices of  leading commercial companies. On the basis of this information, and  taking into account the differences between commercial product  development and major federal acquisitions, we have outlined a best  practices product development model—known as a knowledge-based  approach to system development. This type of approach calls for  investment decisions to be made on the basis of specific, measurable  levels of knowledge at critical junctures before investing more money and  proceeding with development.
Importantly, our work has shown the most leveraged decision point is  matching the customer’s needs with the developer’s resources (time,  dollars, technology, people, etc.) because it sets the stage for the eventual  outcome—desirable or problematic. The match is ultimately achieved in  every development program, but in successful development programs, it  occurs before product development is formally initiated (usually the  preliminary design review).  If the knowledge attained at this and other  critical junctures does not confirm the business case on which the  acquisition was originally justified, the best practice organizations we have  studied do not allow the program to go forward.
We have highlighted the three critical junctures at which developers must  have knowledge to make large investment decisions—the preliminary  design review, the critical design review, and the production review—and  the numerous key indicators that can be used to increase the chances of  successful outcomes.
In assessing the Orion and Ares programs, the Congress and NASA  decision-makers can use these indicators in order to reliably gauge  whether there is a sufficient business case for allowing the programs to  proceed forward.
Preliminary design review: Before product development is started, a  match must be made between the customers’ needs and the available  resources—technical and engineering knowledge, time, and funding. To  provide oversight at this juncture, NASA could provide Congress with  information to verify that the following have indicators been met:    All critical technologies are demonstrated to a high level of technology  maturity, that is demonstrated that they can perform in a realistic or,  more preferably, operational environment. A technology readiness  level 6 or 7 would indicate that this has been achieved.  One approach  to ensure that technology readiness is reliably assessed is to use  independent testing;    Project requirements are defined and informed by the systems    Cost and schedule estimates established for the project are based on  knowledge from the preliminary design using systems engineering  tools;    Additional resources are in place, including needed workforce, and a  decision review is conducted following completion of the preliminary  design review.
A critical enabler for success in this phase of development is performance  and requirements flexibility.  Customers and product developers both  need to be open to reducing expectations, deferring them to future  programs, or to investing more resources up front to eliminate gaps  between resources and expectations. In successful programs we have  studied, requirements were flexible until a decision was made to commit  to product development because both customers and developers wanted  to limit cycle time. This makes it acceptable to reduce, eliminate, or defer  some customer wants so that the product’s requirements could be  matched with the resources available to deliver the product within the  desired cycle time.
Critical design review: A product’s design must demonstrate its ability  to meet performance requirements and be stable about midway through  development. To provide oversight at this juncture, NASA could provide  Congress with information to verify that the following indicators have  been met:    At least 90 percent of engineering drawings are complete;    All subsystem and system design reviews have been completed;    The design meets requirements demonstrated through modeling,  simulation, or prototypes;    Stakeholders’ concurrence that drawings are complete and producible    Failure modes and effects analysis have been completed;    Key system characteristics are identified;    Critical manufacturing processes are identified;    Reliability targets are established and a growth plan based on  demonstrated reliability rates of components and subsystems is  developed; and    A decision review is conducted following the completion of the critical  design review.
Production Review: The developer must show that the product can be  manufactured within cost, schedule, and quality targets and is  demonstrated to be reliable before production begins. To provide  oversight at this juncture, NASA could provide Congress with information  to verify that the following indicators have been met:    Manufacturing processes have been demonstrated;    Production representative prototypes have been built;    Production representative prototypes have been tested and have    Production representative prototypes have been demonstrated in an  operational environment through testing;    Statistical process control data have been collected;    Critical processes have been demonstrated to be capable and that they  are in statistical control;     A decision review is conducted following completion of the production  readiness review.
Over the past 2 years, we have recommended that NASA incorporate a  knowledge-based approach in its policies and take steps to implement this  type of approach in its programs and projects. NASA has incorporated  some knowledge-based concepts into its acquisition policies. For example,  NASA now requires a decision review between each major phase of the  acquisition life cycle and has established general entrance and success  criteria for the decision reviews. In addition, we have reported that this  type of approach is being embraced by the Ares I project.
section_title: Concluding Observations
paragraphs: In conclusion, the President’s Vision for Space Exploration is an ambitious  effort, not just because there will be technical and design challenges to  building systems needed to achieve the Vision’s goals, but because there  are limited resources within which this can be accomplished.  Moreover,  the long-term nature of the Vision means that commitments for funding  and to the goals of the Vision will need to be sustained across presidential  administrations and changes in congressional leadership.  For these  reasons, it is exceedingly important that the right decisions are made early  on and that decision-makers have the right knowledge going forward so  that they can make informed investment decisions.
In looking at the first major investments, the Ares I and Orion projects, it is  important to recognize that they are risky endeavors, largely due to their  complexity, scope, and interdependencies.  It is also important to  recognize that the desire to minimize the gap in human space flight adds  considerable risk, since it could limit NASA's ability to study emerging  problems and pursue alternative ways of addressing them.  For these  reasons, as well as the magnitude of investment at stake, it is imperative  that NASA be realistic and open about the progress it is making and to be  willing to make changes to the architecture and design if technical  problems can not be solved without overly compromising performance.  Additionally, Congress needs to be well-informed about the extent to  which knowledge gaps remain and what tradeoffs or additional resources  are needed to close those gaps and to support changes if they are  determined to be necessary.  The upcoming preliminary design review  milestones represent perhaps the most critical juncture where these  assessments can take place and where hard decisions can be made as to  whether the programs should proceed forward.  It may well be the last  opportunity to make significant adjustments before billions of dollars are  spent and long term commitments become solidified.
Mr. Chairman, this concludes my prepared statement. I would be pleased  to answer any questions that you may have at this time.
section_title: GAO Contacts and Staff Acknowledgements
paragraphs: For further questions about this statement, please contact Cristina T.  Chaplain at (202) 512-4841. Individuals making key contributions to this  statement include James L. Morrison, Meredith A. Kimmitt, Lily Chin, Neil  Feldman, Rachel Girshick, Shelby S. Oakley, and John S. Warren, Jr.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.