NASA: Assessments of Selected Large-Scale Projects
Published: 2010-02-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Aeronautics and Space Administration (NASA) plans to invest billions in the coming years in science and exploration space fl ight initiatives. The scientifi c and technical complexities inherent in NASA's mission create great challenges in managing its projects and controlling costs. In the past, NASA has had diffi culty meeting cost, schedule, and performance objectives for many of its projects. The need to effectively manage projects will gain even more importance as NASA seeks to manage its wide-ranging portfolio in an increasingly constrained fi scal environment. This report provides an independent assessment of selected NASA projects. In conducting this work, GAO compared projects against best practice criteria for system development including attainment of knowledge on technologies and design. GAO also identifi ed other programmatic challenges that were contributing factors in cost and schedule growth of the projects reviewed. The projects assessed are considered major acquisitions by NASA--each with a life-cycle cost of over $250 million. No recommendations are provided in this report; however, GAO has reported extensively and made recommendations on NASA acquisition management in the past. GAO has designated NASA's acquisition management as a high risk area since 1990.
section_title: What GAO Found
paragraphs: GAO assessed 19 NASA projects with a combined life-cycle cost of more than $66 billion. Of those 19 projects, 4 are still in the formulation phase where cost and schedule baselines have yet to be established, and 5 just entered the implementation phase in fi scal year 2009 and therefore do not have any cost and schedule growth. However, 9 of the 10 projects that have been in the implementation phase for several years experienced cost growth ranging from 8 to 68 percent, and launch delays of 8 to 33 months, in the past 3 years. These 10 projects had average development cost growth of almost $121.1 million--or 18.7 percent--and schedule growth of 15 months, and a total increase in development cost of over $1.2 billion, with over half of this total--or $706.6 million--occurring in the last year. In some cases, cost growth was higher than is reported because it occurred before project baselines were established in response to the statutory requirement in 2005 for NASA to report cost and schedule baselines for projects and implementation with estimated life-cycle cost of more than $250 million. Additionally, NASA was recently appropriated over $1 billion through the American Recovery and Reinvestment Act of 2009. Many of the projects GAO reviewed experienced challenges in developing new or retrofi tting older technologies, stabilizing engineering designs, managing the performance of their contractors and development partners, as well as funding and launch planning issues. Reducing the kinds of problems this assessment identifi es in acquisition programs hinges on developing a sound business case for a project. Based, in part, on GAO's previous recommendations, NASA has acted to adopt practices that would ensure programs proceed based on a sound business case and undertaken initiatives aimed at improving program management, cost estimating, and contractor oversight. Continued attention to these efforts and effective, disciplined implementation should help maximize NASA's acquisition investments.

section_title: Letter
paragraphs: I am pleased to present GAO’s second annual assessment of selected large- scale NASA projects.  This report provides a snapshot of how well NASA  plans and executes major acquisitions—a topic that has been on GAO’s  High-Risk List since the list’s inception in 1990.
In this year’s report, we found that NASA frequently exceeded its own  acquisition cost and schedule estimates, even when those estimates  were relatively new.  In fact, 9 out of 10 projects that have been in  implementation for several years signiﬁ cantly exceeded their cost or  schedule baseline estimates—all in the last 3 years.
NASA’s ongoing struggle to meet budget and schedule demands comes at a  time when the agency is on the verge of major changes.  The Space Shuttle  is slated to retire this year after nearly 30 years of service, the International  Space Station draws closer to its scheduled retirement in 2016, and a new  means of human space ﬂ ight is under development, and the very future of   which has been hotly debated and recently reviewed by an independent  commission, and awaits a presidential decision.
Amid all this change, one thing that will remain constant is NASA’s need to  manage programs and projects with a budget that has remained relatively  constant in recent years.  This will require hard choices among competing  priorities within the organization, which must balance its core missions in  science, aeronautics, and human space ﬂ ight and exploration.  In addition,  NASA will be competing for an ever-shrinking share of discretionary  spending against other national priorities, such as the economy, combatting  terrorism, and health care reform.
We believe that this report can provide insights that will help NASA place  programs in a better position to succeed and help the agency maximize  its investments.  Our work has shown that reducing the project challenges  that can lead to cost and schedule growth this report identiﬁ es hinges  on developing a sound business case that includes ﬁ rm requirements,  mature technologies, a knowledge-based acquisition strategy, realistic cost  estimates, and sufﬁ cient funding.  To its credit, NASA has continued to take  steps to improve its acquisition process along these lines.  The revisions aim  to provide key decision-makers with increased knowledge needed to make  informed decisions before a program starts, and to maintain discipline once  it begins.  Implementation of these revisions, however, will require senior  NASA leaders to have the will to terminate projects that do not measure  up, to recognize and reward savings, and to hold appropriate parties  accountable for poor outcomes.
The National Aeronautics and Space Administration’s (NASA) portfolio  of major projects ranges from highly complex and sophisticated space  transportation vehicles, to robotic probes, to satellites equipped with  advanced sensors to study the earth. In many cases, NASA’s projects are  expected to incorporate new and sophisticated technologies that must  operate in harsh, distant environments. These projects have also produced  ground-breaking research and advanced our understanding of the universe.  However, one common theme binds most of the projects—they cost more  and take longer to develop than planned.
We reported last year that 10 out of 13 NASA projects experienced  signiﬁ cant cost and/or schedule growth from baselines established only  2 or 3 years earlier. For example, the Glory project, a science satellite  designed to help understand how the sun and particles in the atmosphere  affect Earth’s climate, saw its development costs increase more than 50  percent—from $169 to $259 million—since 2008. Congress reauthorized   the Glory project in ﬁ scal year 2009 and new cost and schedule baselines  were then established. Similarly, technical issues delayed the Mars Science  Laboratory by 2 years, and the project, which was already over budget, is  now scheduled to cost over $660 million more than estimated in 2007—an  increase of over 68 percent in development costs. In prior years, programs  such as the X-33 and X-34, which were meant to demonstrate technology  for future reusable launch vehicles, were cancelled because of technical  difﬁ culties and cost overruns after NASA spent more than $1 billion on the  programs.
NASA acknowledges the problem and is striving to improve its cost  estimating and program execution. The agency notes that most missions  are one of a kind and complex and that external factors, such as launch  scheduling and spotty performance by development partners, also cause  delays and cost increases. Although space development programs are  complex and difﬁ cult by nature, our work consistently ﬁ nds that inherent  risks are exacerbated by poor acquisition management. Moreover, the  reality of cost and schedule increases can have secondary impacts when  projects that are seemingly on track end up being the bill-payer for troubled  projects. This also makes it hard to manage the portfolio and make  investment decisions.
Congress has expressed concern about NASA’s performance and has  identiﬁ ed the need to standardize the reporting of cost, schedule, and  content for NASA research and development projects. In 2005, Congress  required NASA to report cost and schedule baselines—benchmarks against  which changes can be measured—for all NASA programs and projects with  estimated life-cycle costs of at least $250 million that have been approved  to proceed to the development stage, known as implementation, in which  components begin to take physical form. It also required that NASA  report to Congress when development cost is likely to exceed the baseline  estimate by 15 percent or more, or when a milestone is likely to be delayed  beyond the baseline estimate by 6 months or more.  In response, NASA  began establishing cost and schedule baselines in 2006 and has been using  them as the basis for annual project performance reports to the Congress  provided in its annual budget submission each year. While establishing the  baselines required by the Congress enabled a more consistent reporting  among NASA projects, it also made past cost and schedule growth less  transparent. Consequently, the cost and schedule breaches presented in  this report represent only increases from the baselines established after the  2005 congressional requirement.
Recently, NASA was appropriated over $1 billion through the American  Recovery and Reinvestment Act of 2009 to help spur technological  advances in science. The agency’s Science and Exploration Systems Mission  Directorates were each appropriated $400 million under this supplemental  appropriation. As of October 2009, the projects covered in this assessment  are scheduled to receive $470 million as a part of the total allocation that  NASA intends to use to assist with such items as developing instruments  and spacecraft, maintaining the current workforce, and building test  facilities. Appendix V provides a listing of NASA projects in our review  receiving funding under the American Recovery and Reinvestment Act of  2009 and the intended use of those funds for each project.
The explanatory statement of the House Committee on Appropriations  accompanying the Fiscal Year 2009 Omnibus Appropriations Act directed  GAO to prepare project status reports on selected large-scale NASA  programs, projects, or activities. This report responds to that mandate by  assessing 19 NASA projects, each with an estimated life-cycle cost over  $250 million. The combined estimated life-cycle cost for these 19 projects  exceeds $66 billion. Each assessment is presented in a two-page summary  that analyzes the project’s cost and schedule status and project challenges  we identiﬁ ed with the objective to identify risks that, if mitigated, could put  NASA in a better position to succeed. We also provide general observations  about the performance of NASA’s major projects and the agency’s  management of those projects during development. In doing so, the report  expands on the importance of developing a knowledge-based acquisition  strategy and to provide decision-makers with an independent, knowledge- based assessment of individual systems that identiﬁ es potential risks and  allows the decision-makers to take actions to put projects that are early  in the development cycle in a better position to succeed. NASA provided  updated cost and schedule data as of October 2009 for 14 of the 19 projects.   We reviewed and compared that data to previously established cost and  schedule baselines for each of those 14 projects. We took appropriate steps  to address data reliability.
Our approach included an examination of the current phase of a project’s  development and how each project was advancing. Each project we  reviewed was in either the formulation phase or the implementation  phase of the project life cycle. In the formulation phase, the project  deﬁ nes requirements—what the project is being designed to do—matures  technology, establishes a schedule, estimates costs, and produces a plan for  implementation. In the implementation phase, the project carries out these  plans, performing ﬁ nal design and fabrication as well as testing components  and system assembly, integrating these components and testing how they  work together, and launching the project. This phase also includes the  period from project launch through mission completion. We assessed each  project’s cost and schedule and characterized growth in either as signiﬁ cant  if it exceeded the baselines that trigger reporting to the Congress under the  law.
Based on our previous reviews and discussions with project ofﬁ cials and  drawing on GAO’s established criteria for knowledge-based acquisitions and  on other GAO work on system acquisitions, we identiﬁ ed six challenges that  can contribute to cost and schedule growth in these projects: technology  maturity, design stability, contractor performance, development partner  performance, funding issues, and launch manifest issues. This list of  challenges is not exhaustive, and we believe these challenges will evolve  as we continue this work into the future. To assess technology maturity,  we examined the projects’ reported critical technology readiness levels—a  measure that NASA devised and that is now used at other agencies as  well. We looked at the technology readiness level at the time of the  project’s preliminary design review, which occurs just before it enters the  implementation phase, and compared that against the level of maturity that  best practices call for at that stage to minimize risks. Based in part on our  discussions with ofﬁ cials for the individual projects and data submitted by  the projects, we identiﬁ ed the extent to which project cost and schedule  were negatively impacted by challenges integrating heritage—or pre- existing—technology into their projects. To assess design stability, we  examined the percentage of engineering drawings completed or projected  to be completed by the critical design review—which is usually held about  midway through the project’s development. We asked project ofﬁ cials to  provide this information, and we compared it against GAO’s best practices’  metric of 90 percent of drawings released by the critical design review. We  also discussed the extent to which contractors’ and development partners’  challenges in developing and delivering project hardware affected overall  project cost and schedule. To assess funding issues, we interviewed project  ofﬁ cials and reviewed budget documents to determine if increases to cost  or schedule resulted from interrupted or delayed funding, or if project  ofﬁ cials indicated that the project had poor phasing of the project’s funding  plan. To assess launch manifest issues, we interviewed launch services  ofﬁ cials to determine what projects had to reschedule launch dates based  on an inability to be ready for launch or other factors. The individual project  ofﬁ ces were given an opportunity to provide comments and technical  clariﬁ cations on our assessments prior to their inclusion in the ﬁ nal  product.
We conducted this performance audit from April 2009 to February 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufﬁ cient,  appropriate evidence to provide a reasonable basis for our ﬁ ndings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our ﬁ ndings and conclusions based  on our audit objectives. Appendix II contains detailed information on our  scope and methodology. We do not provide recommendations in this report.
subsections: section_title: A Sound Business Case Underpins Successful Acquisition Outcomes
paragraphs: Many of NASA’s projects are one-time articles, meaning that there is little  opportunity to apply knowledge gained to the production of a second, third,  or future increments of spacecraft. In addition, NASA often partners with  other domestic partners and other space-faring countries, including several  European nations, Japan, and Argentina. These partnerships go a long way  to foster international cooperation in space, but they also subject NASA  projects to added risk such as when partners do not meet their obligations  or run into technical obstacles they cannot easily overcome. While space  development programs are complex and difﬁ cult by nature, and most are  one-time efforts, the nature of its work should not preclude NASA from  achieving what it promises when requesting and receiving funds. We have  reported that NASA would beneﬁ t from a more disciplined approach to its  acquisitions.
The development and execution of a knowledge-based business case for  these projects can provide early recognition of challenges, allow managers  to take corrective action, and place needed and justiﬁ able projects in a  better position to succeed. Our studies of best practice organizations show  the risks inherent in NASA’s work can be mitigated by developing a solid,  executable business case before committing resources to a new product  development. In its simplest form, this is evidence that (1) the customer’s  needs are valid and can best be met with the chosen concept, and (2) the  chosen concept can be developed and produced within existing resources— that is, proven technologies, design knowledge, adequate funding, and  adequate time to deliver the product when needed. A program should not  go forward into product development unless a sound business case can be  made. If the business case measures up, the organization commits to the  development of the product, including making the ﬁ nancial investment.  Our best practice work has shown that developing business cases based  on matching requirements to resources before program start leads to more  predictable program outcomes—that is, programs are more likely to be  successfully completed within cost and schedule estimates and deliver  anticipated system performance.
At the heart of a business case is a knowledge-based approach to product  development that is a best practice among leading commercial ﬁ rms. Those  ﬁ rms have created an environment and adopted practices that put their  program managers in a good position to succeed in meeting expectations.  A knowledge-based approach requires that managers demonstrate high  levels of knowledge as the program proceeds from technology development  to system development and, ﬁ nally, production. In essence, knowledge  supplants risk over time. This building of knowledge can be described over  the course of a program, as follows: When a project begins development, the customer’s needs should match the  developer’s available resources—mature technologies, time, and funding.  An indication of this match is the demonstrated maturity of the technologies  needed to meet customer needs—referred to as critical technologies.  If the project is relying on heritage—or pre-existing— technology, that  technology must be in appropriate form, ﬁ t, and function to address the  customer’s needs within available resources. The project will normally enter  development after completing the preliminary design review, at which time  a business case should be in hand.
Then, about midway through the product’s development, its design  should be stable and demonstrate it is capable of meeting performance  requirements. The critical design review takes place at that point in time  because it generally signiﬁ es when the program is ready to start building  production-representative prototypes. If design stability is not achieved, but  a product development continues, costly re-designs to address changes to  project requirements and unforeseen challenges can occur. By the critical  design review, design should be stable and capable of meeting performance  requirements.
Finally, by the time of the production decision, the product must be  shown to be producible within cost, schedule, and quality targets and  have demonstrated its reliability, and the design must demonstrate that it  performs as needed through realistic system-level testing. Lack of testing  increases the possibility that project managers will not have information  that could help avoid costly system failures in late stages of development or  during system operations.
Our best practices work has identiﬁ ed numerous other actions that can  be taken to increase the likelihood that a program can be successfully  executed once that business case is established. These include ensuring  cost estimates are complete, accurate, and updated regularly and holding  suppliers accountable through such activities as regular supplier audits  and performance evaluations of quality and delivery. Moreover, we have  recommended using metrics and controls throughout the life cycle to gauge  when the requisite level of knowledge has been attained and when to direct  decision makers to consider criteria before advancing a program to the next  level and making additional investments.
The consequence of proceeding with system development without  establishing and adhering to a sound business case is substantial. GAO and  others have reported that NASA has experienced cost and schedule growth  in several of its projects over the past decade, resulting from problems that  include failing to adequately identify requirements and underestimating  complexity and technology maturity. We have found that the need to  meet schedule is one of the main reasons why programs cannot execute  as planned. Short cuts, such as developing technology while design work  and construction are already underway, and delaying or reducing tests,  are taken to meet schedule. Ultimately, when a schedule is set that cannot  accommodate the work that needs to be done, costs go up and capability  is delayed. Delaying the delivery of these capabilities can also have a ripple  effect throughout NASA projects as staff must then stay on a given project  longer than intended, thus increasing the project’s costs, and crippling other  projects that had counted on using newly available staff to move forward.
section_title: NASA Continues Efforts to Improve Its Acquisitions
paragraphs: In 2005, we reported that NASA’s acquisition policies did not conform to  best practices for product development because those policies lacked major  decision reviews at several key points in the project life-cycle that would  allow decision makers to make informed decisions about whether a project  should be authorized to proceed in the development life cycle. Based in part  on our recommendations, NASA issued a revised policy in March 2007 that  institutes several key decision points (KDP) in the development life cycle  for space ﬂ ight programs and projects. At each KDP, a decision authority is  responsible for authorizing the transition to the next life-cycle phase for the  project. In addition, NASA’s acquisition policies also require that  technologies be sufﬁ ciently mature at the preliminary design review before  the project enters implementation, that the design is appropriate to support  proceeding with full-scale fabrication, assembly, integrating and test at the  critical design review, and that the system can be fabricated within cost,  schedule, and performance speciﬁ cations. These changes brought the policy  more in line with best practices for product development. A more detailed  discussion of NASA’s acquisition policy and how it relates to best practices  is provided in appendix III of this report.
Further, in response to GAO’s designation of NASA acquisition management  as a high risk area, NASA developed a corrective action plan to improve  the effectiveness of NASA’s program/project management. The approach  focuses on how best to ensure the mitigation of potential issues in  acquisition decisions and better monitor contractor performance. The  plan identiﬁ es ﬁ ve areas for improvement—program/project management,  cost reporting processes, cost estimating and analysis, standard business  processes, and management of ﬁ nancial management systems—each of  which contains targets and goals to measure improvement. As part of this  initiative, NASA has taken a positive step to improve management oversight  of project cost, schedule, and technical performance with the establishment  of a baseline performance review reporting to NASA’s senior management.  Through monthly reviews, NASA intends to highlight projects that are  predicted to exceed internal NASA cost and/or schedule baselines, which  are set lower than cost and schedule baselines submitted to Congress, so  the agency can take preemptive actions to minimize the projects’ potential  cost overruns or schedule delays. During our data collection efforts, we  reviewed several projects’ monthly and quarterly status reports, which gave  us insight into their status, risks, and issues. While this reporting structure  might enable management to be aware of the issues projects are facing, it  is too early to tell if the monthly reviews are having the intended impact of  enabling NASA management to take preemptive cost saving actions, such as  delaying a design review or canceling a project.
As a part of the continuing effort to improve its acquisition processes, NASA  has begun a new initiative—Joint Cost and Schedule Conﬁ dence Levels  (JCL)—to help programs and projects with management, cost and schedule  estimating, and maintenance of adequate levels of reserves. Under this new  policy, cost, schedule, and risk are combined into a complete picture to help  inform management of the likelihood of a project’s success. Utilizing JCL,  each project will receive a cost estimate with a corresponding conﬁ dence  level—the percentage probability representing the likelihood of success  at the speciﬁ ed funding level. NASA believes the application of this policy  will help reduce the cost and schedule growth in its portfolio and improve  transparency, and increase the probabilities of meeting those expectations.  NASA’s goal is for all projects that have entered the implementation phase  to have a JCL established by spring 2010.
While these efforts are positive steps, it is too early to assess their impact  and they will be limited if project ofﬁ cials are not held accountable for  demonstrating that elements of a knowledge-based business case are  demonstrated at key junctures in development. For projects to have better  outcomes not only must they demonstrate a high level of knowledge at key  junctures, but decision makers must also use this information to determine  whether and how best a project should proceed through the development  life cycle. If done successfully, these measures should enable NASA to  foster the expansion of a business-oriented culture, reduce persistent cost  growth and schedule delays, and maximize investment dollars.
section_title: Our Observations
paragraphs: We assessed 19 large-scale NASA projects in this review. Four of these  projects were in the formulation phase where cost and schedule baselines  have yet to be established, while 15 had entered implementation. Nine of  the 15 projects experienced signiﬁ cant cost and/or schedule growth from  their project baselines, while ﬁ ve of the remaining projects had just  entered implementation and their cost and schedule baselines were  established in ﬁ scal year 2009. NASA provided cost and schedule data for 14  of the 15 projects in the implementation phase of the project life cycle.  Despite being in implementation, NASA did not provide cost or schedule  data for the Magnetospheric Multiscale (MMS) project. NASA will not  formally release its baseline cost and schedule estimates for this project  until the ﬁ scal year 2011 budget submission to Congress, and late in our  review process agency ofﬁ cials notiﬁ ed us that they will not provide project  estimates to GAO until that time. NASA also did not provide formal cost and  schedule information for the projects in formulation, citing that those  estimates were still preliminary. See ﬁ gure 1 for a summary of these  projects.
Based on our analysis, development costs for projects in our review  increased by an average of over 13 percent from their baseline cost  estimates—including one project that increased by over 68 percent— and an average delay of almost 11 months to their launch dates. These  averages were signiﬁ cantly higher when the four projects that just  entered implementation are excluded. Speciﬁ cally, there are 10 projects  of analytical interest because (1) they are in the implementation phase,  and (2) their baselines are old enough to begin to track variances. Most of  these 10 projects have experienced signiﬁ cant cost and/or schedule growth,  often both. These projects had an average development cost growth of  18.7 percent—or almost $121.1 million—and schedule growth of over 15  months, and a total increase in development cost of over $1.2 billion. Over  half of this total increase in development cost—or $706.6 million—occurred  in the last year. These cost growth and schedule delays have all occurred  within the last 3 years, and a number of these projects had experienced  considerable cost growth before baselines were established in response to  the 2005 statutory reporting requirement. See table 1 below for the cost  and schedule growth of the NASA projects in the implementation phase.
Despite having baselines established in ﬁ scal year 2008, two projects have  sought reauthorization from Congress because of development cost growth  in excess of 30 percent. Congress reauthorized the Glory project in ﬁ scal  year 2009, and new cost and schedule baselines were established after the  project experienced a 53 percent cost growth and 6-month launch delay  from original baseline estimates. The Glory project has since breached its  revised schedule baseline by 16 months and exceeded its development cost  baseline by over 14 percent—for a total development cost growth of over 75  percent in just 2 years. Project ofﬁ cials also indicated that recent technical  problems could cause additional cost growth. Similarly, the Mars Science  Laboratory project is currently seeking reauthorization from Congress after  experiencing development cost in excess of 30 percent.
section_title: Project Challenges
paragraphs: All six factors we assessed can lead to project cost and schedule growth:  technology maturity, design stability, contractor performance, development  partner performance, funding issues, and launch manifest issues. These  factors—characterized as project challenges—were evident in the projects  that had reached the implementation phase of the project life cycle, but  many of them began in the formulation phase. We did not speciﬁ cally  correlate individual project challenges with speciﬁ c cost and/or schedule  changes in each project. The degree to which each speciﬁ c challenge  contributed to cost and schedule growth varied across the projects in this  review and we did not assign any speciﬁ c challenge as a primary factor for  cost and/or schedule growth. Table 2 depicts the extent to which each of  the six challenges occurred for each of the 19 projects we reviewed.
Technology Maturity was by far the most prevalent challenge, affecting 15  of the 19 projects. When combined with design instability—another metric  related to technical difﬁ culty—17 projects were affected. A discussion of  each challenge follows.
subsections: section_title: Technology Maturity
paragraphs: Our past work on systems acquisition has shown that beginning an  acquisition program before requirements and available resources are  matched can result in a product that fails to perform as expected, costs  more, or takes longer to develop. We have found that these problems are  largely rooted in the failure to match customer’s needs with the developer’s  resources—technical knowledge, timing, and funding—when starting  product development. In other words, commitments were made to deliver  capability without knowing whether the technologies needed could really  work as intended. Time and costs were consistently underestimated, and  problems that surfaced early cascaded throughout development and  magniﬁ ed the risks facing the program. Our best practices work has shown  that a technology readiness level (TRL) of 6— demonstrating a technology  as a fully integrated prototype in a relevant environment—is the level of  maturity needed to minimize risks for space systems entering product  development. NASA’s acquisition policy states that a TRL of 6 is desirable  prior to integrating a new technology. Technology maturity is a  fundamental element of a sound business case, and the absence is a marker  for subsequent problems, especially in design.
Similarly, our work has shown that the use of heritage technology—proven  components that are being modiﬁ ed to meet new requirements—can also  cause problems when the items are not sufﬁ ciently matured to meet form,  ﬁ t, and function standards by the preliminary design review (PDR). NASA  states in its Systems Engineering Handbook that particular attention must  be given to heritage systems because they are often used in architectures  and environments different from those in which they were designed to  operate. Although NASA distinguishes critical technologies from heritage  technologies, our best practices work has found critical technologies to  be those that are required for the project to successfully meet customer  requirements, regardless of whether or not they are based on existing  or heritage technology. Therefore, whether technologies are labeled as  “critical” or “heritage,” if they are important to the development of the  spacecraft or instrument—enabling it to move forward in the development  process—they should be matured by PDR.
Of the 14 projects for which we received data and that had entered the  implementation phase, four entered this phase without ﬁ rst maturing all  their critical technologies, and 10 encountered challenges in integrating or  modifying heritage technologies. Additionally, two projects in formulation— Ares I and Orion—also encountered challenges with critical or heritage  technologies. These projects did not build in the necessary resources for  technology modiﬁ cation. For instance, the recent cost and schedule growth  in the Mars Science Laboratory (MSL) highlights the problems that can be  realized when a project proceeds past the formulation phase with immature  technologies. MSL reported seven critical technologies were not mature at  the time of its preliminary design review, and over a year later two of these  technologies were still immature at the critical design review; however, the  project moved forward into the implementation phase with established cost  and schedule baselines and the lack of technology maturity contributed  to an unstable design. In part as a result of immature technologies and  an unstable design, MSL delayed its launch date by 25 months, and  development costs have grown by more than $660 million. In November  2008, the GRAIL project also moved beyond its PDR with an immature  heritage technology—the reaction wheel assembly. This technology has  been ﬂ own on other NASA missions, but the project team must modify it for  GRAIL by integrating electronics into the assembly.
NASA acknowledges in its Systems Engineering Handbook that  modiﬁ cation of heritage systems is a frequently overlooked area in  technology development and that there is a tendency on the part of project  management to overestimate the maturity and applicability of heritage  technology. NASA recognizes that as a result of not placing enough  emphasis on the development of heritage technologies, key steps in the  development process are not given appropriate attention, and critical  aspects of systems engineering are overlooked.
section_title: Design Stability
paragraphs: The importance of establishing a stable design at a project’s critical design  review (CDR) is also critical. The CDR provides assurance that the design is  mature and will meet performance requirements. An unstable design can  result in costly re-engineering and re-work efforts, design changes, and  schedule slippage. Quantitative measures employed at CDR, such as  percentage of engineering drawings, can provide evidence that the design is  stable and “freeze” it to minimize changes in the future. Our work has  shown that release of at least 90 percent of engineering drawings at the  CDR provides evidence that the design is stable. Though NASA’s acquisition  policy does not specify how a project should achieve design stability by  CDR, NASA’s Systems Engineering Handbook adheres to this metric of 90  percent drawings released by the CDR.
Eight projects in our assessment have already held their CDR and were  able to provide us with the number of engineering drawings completed and  released. None of these 8 projects met the 90 percent standard for design  stability at CDR; however, NASA believes that some of these projects had  stable designs and pointed to other activities that occurred prior to CDR as  evidence. Nevertheless, the percentage of engineering drawings released  at CDR by these 8 projects averaged less than 40 percent, and more than  three-fourths of these projects had signiﬁ cant cost and/or schedule growth  from their established baselines after their CDR when their design was  supposed to be stable. Although all the cost and schedule growth for these  projects cannot be directly attributed to a lack of design stability, we believe  that this was a contributing factor.
Discussions with project ofﬁ cials showed the metric was used  inconsistently to gauge design stability. For example, Goddard Space Flight  Center requires greater than 80 percent drawings released at CDR, yet  we were told by several project ofﬁ cials that the rule of thumb for NASA  projects is between 70 and 90 percent drawings released at CDR. However,  there was no consensus among the ofﬁ cials. For example, one project  manager from Goddard Space Flight Center told us the project is planning  to have 70 percent of the drawings released at CDR; a project manager  from the Jet Propulsion Laboratory cited that having 85 to 90 percent of  the drawings released is what he prefers at CDR; otherwise, he does not  consider the project design to be complete. Goddard’s Chief Engineer said  that, as a member of a design review board, he will generally question  projects that have less than 95 percent of engineering drawings released,  especially if the project is using heritage technologies. Ofﬁ cials added that  at CDR it is more important to have drawings completed that relate to  critical technologies than those related to integration activities.
In addition to released drawings, NASA often relies on subject matter  experts in the design review process and other methods to assure that  a project has a stable design. Some projects indicated that completing  engineering models, which are preproduction prototypes, and holding  sub-system level CDR’s for instruments and components helped to assess  design stability, at least in part. Ofﬁ cials for these projects indicated that  use of engineering models helps decrease risk of ﬂ ight unit development;  projects that did not use engineering models indicated they might have  caught problems earlier had they used them. For example, at CDR the Mars  Science Laboratory’s engineering models were incomplete and could have  been a cause of concern. Mars Science Laboratory project ofﬁ cials were  aware that avionics were an issue at CDR, but were unaware of future  problems with other project components, such as the actuators. Project  ofﬁ cials told us that if the engineering models for all subsystems had been  completed at CDR, many of the later problems would have been caught and  mitigated earlier in the process, thereby avoiding schedule delays. However,  these project ofﬁ cials added that engineering models are expensive to  employ and not all projects have the available funding required to utilize  them.
section_title: Contractor Performance
paragraphs: NASA relies heavily on the work of its contractors. Ofﬁ cials at ﬁ ve of the  projects we reviewed indicated that the contractors for their projects had  trouble moving their work forward after experiencing technical and design  problems with hardware that disrupted development progress. Since about  85 percent of NASA’s annual budget is spent by its contractors, the  performance of these contractors is instrumental to the success of the  projects.
Shifts in the industrial base and a lack of expertise at the contractors  affected performance. For example, project ofﬁ cials for the SOFIA project  reported that the contractor for the aircraft modiﬁ cation was bought and  sold several times during the development process. Project ofﬁ cials further  reported that the contractor had limited experience with this type of work  and did not fully understand the statement of work. Consequently, the  contractor had difﬁ culty completing this work, which led to signiﬁ cant cost  overruns. While project ofﬁ cials told us that issues with that contractor  have since been resolved, this year another SOFIA contractor that is  responsible for developing hardware and software has performed poorly,  which ofﬁ cials attribute to a recent buyout of the company. In addition,  agency ofﬁ cials said that NASA is a low priority for the contractor, and the  project is ﬁ nding it difﬁ cult to exert pressure to ensure better performance.  Project ofﬁ cials told us that they currently have three people at the  contractor’s site as a permanent presence. They added that if the contract  were to be cancelled due to poor performance, this work would be brought  in-house and would result in a one year delay. In addition, the Glory project  has struggled for several years to develop a key instrument. The Glory  project manager cited management inefﬁ ciencies with the instrument’s  contractor including senior leadership changes, a loss of core competencies  because of a plant closure, and a lack of proper decision authority. The  contractor agreed that the plant closure and the need to re-staff were major  project challenges.
section_title: Development Partner Performance
paragraphs: Six projects in our review encountered challenges with their development  partners. In these cases the development partner could not meet their  commitments to the project within the planned schedules. For example,  NASA collaborated with the European Space Agency (ESA) on the Herschel  space observatory. NASA delivered its two instruments to ESA in a timely  manner, but ESA encountered difﬁ culties developing its instruments, and  the result was a 14-month delay in Herschel’s schedule. Because of this  delay, NASA incurred an estimated $39 million in cost growth because of  the need to fund component developers for a longer period of time than  originally planned. We found that of the projects that are currently in  implementation and have experienced cost and/or schedule growth, those  with international or domestic partners experienced more than one-and-a- half times as much schedule growth on average as those with no partner.  Table 3 below shows the average schedule growth for projects with  partners as compared to those without partners.
section_title: Funding Issues
paragraphs: During the course of our review, we identiﬁ ed six projects in the  implementation phase, as well as three projects still in formulation, that had  experienced issues related to the project’s funding because of issues such  as agency-directed funding cuts early in the project life-cycle and projects  whose budgets do not match the work expected to be accomplished. For  example, NASA management cut $35 million from the Kepler project’s ﬁ scal  year 2005 budget—a cut amounting to one-half of the project’s budget for  the year. Contractor ofﬁ cials told us that this forced the shutdown of  signiﬁ cant work, interrupted the overall ﬂ ow and scheduling for staff and  production, and required a renegotiation of contracts. This funding  instability, according to a NASA project ofﬁ cial, contributed to an overall  20-month delay in the project’s schedule and about $169 million in cost  growth.
The funding instability for Kepler affected more than that one project.  The WISE project had to extend the formulation phase since funding was  unavailable at the time of the conﬁ rmation review in November 2005.
According to NASA and contractor ofﬁ cials, the WISE project experienced  funding cuts when NASA took money from that project to offset increased  costs for the Kepler project. As a result of the extended formulation phase,  the WISE project manager told us that development costs increased and the  launch readiness date slipped 11 months. This is an example of how, when  problems arise, one project can become the bill-payer for another project,  making it difﬁ cult to manage the portfolio and make investment decisions.
We also identiﬁ ed several projects where, according to NASA ofﬁ cials,  the projected budget was inadequate to perform work in certain ﬁ scal  years. For example, the Constellation program’s poorly phased funding  plan has diminished both the Ares I and Orion projects’ ability to deal with  technical challenges. NASA initiated the Constellation program relying on  the accumulation of a large rolling budget reserve in ﬁ scal years 2006 and  2007 to fund program activities in ﬁ scal years 2008 through 2010. Thereafter,  NASA anticipated that the retirement of the space shuttle program in  2010 would free funding for the Constellation program. The program’s  risk management system identiﬁ ed this strategy as high risk, warning that  shortfalls could occur in ﬁ scal years 2009 through 2012. According to the  Constellation program manager, the program’s current funding shortfalls  have reduced the ﬂ exibility to resolve technical challenges. In addition, the  James Webb Space Telescope project had to delay its scheduled launch date  by one year in part because of poor phasing of the project’s funding plan.
section_title: Launch Manifest Issues
paragraphs: We identiﬁ ed four projects in our assessment that are experiencing launch  delays or other launch manifest-related challenges. By their nature, launch  delays can contribute signiﬁ cantly to cost and schedule growth, as months  of delay can translate into millions of dollars in cost increases. For example,  the Solar Dynamics Observatory (SDO) project missed its scheduled launch  date in August 2008 because of test scheduling and spacecraft parts  problems. This delay resulted in the SDO project moving to the end of the  manifest for the Atlas V launch vehicles on the East coast, causing an  18-month launch delay and $50 million cost increase. While the primary  reason for the cost growth is that the SDO project could not meet its  original schedule for launch, the project is incurring additional costs to  maintain project staff longer than originally planned as they await their turn  in the launch queue. According to SDO ofﬁ cials, this has also affected  stafﬁ ng at Goddard Space Flight Center since these personnel were  scheduled to move to other projects. Furthermore, launch delays of one  project can potentially impact the launch manifest for other projects. The  25-month delay of the Mars Science Laboratory project has the potential to  cause disruptions for other projects on the launch manifest in late 2011,  including those outside of NASA, since planetary missions—those missions  that must launch in a certain window because of planetary alignments— receive launch priority to take advantage of optimal launch windows.
Some NASA projects are also experiencing launch manifest-related  challenges. For example, the Gravity Recovery and Interior Laboratory  project is monitoring the availability of trained launch personnel as that  mission is the last to launch on the Delta II vehicle. United Launch Alliance   ofﬁ cials told us that they are taking active steps, such as cross-utilizing the  Delta II personnel with other launch vehicles, to ensure that trained launch  personnel are available for all the remaining Delta II launches. In addition,  the recent failure of the Taurus XL launch vehicle during the launch of the  Orbiting Carbon Observatory has the potential to delay the Glory mission if  the Taurus XL is not cleared for use before Glory has corrected its technical  problems.
section_title: Project Assessments
paragraphs: The 2-page assessments of the projects we reviewed provide a proﬁ le of  each project and describe the challenges we identiﬁ ed. On the ﬁ rst page, the  project proﬁ le presents a general description of the mission objectives for  each of the projects; a picture of the spacecraft or aircraft; a schedule  timeline identifying key dates for the project; a table identifying  programmatic and launch information; and a table showing the baseline  year cost and schedule estimates and the most current available cost and  schedule data; a table showing the challenges relevant to the project; and a  project status narrative. On the second page of the assessment, we provide  an analysis of the project challenges and the extent to which each project  faces cost, schedule, or performance risk because of these challenges. In  addition, NASA project ofﬁ ces were provided an opportunity to review  drafts of the assessments prior to their inclusion in the ﬁ nal product, and  the projects provided both technical corrections and more general  comments. We integrated the technical corrections as appropriate and  characterized the general comments below the detailed project discussion.  See ﬁ gure 2 below for an illustration of the layout of each two-page  assessment.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to NASA for review and comment. In its  written response, NASA agrees with our ﬁ ndings and states that it will strive  to address the challenges that lead to cost and schedule growth in its  projects. NASA agrees that GAO’s cost and schedule growth ﬁ gures reﬂ ect  what the agency has experienced since the baselines were established in  response to the 2005 statutory reporting requirements. Importantly, NASA  has begun to provide more data regarding cost growth prior to these  baselines, and we look forward to working with NASA to increase  transparency into cost and schedule information of large-scale projects  even further in the future.
NASA noted that its projects are high-risk and one-of-a-kind development  efforts that do not lend themselves to all the practices of a “business case”  approach that we outlined since essential attributes of NASA’s project  development differ from those of a commercial or production industry. We  agree, however NASA could still beneﬁ t from a more disciplined approach  to its acquisitions whereby decisions are based upon high levels of  knowledge. Currently, inherent risks are being exacerbated due to projects  moving forward with immature technologies and unstable designs and  difﬁ culties working with contractors and international partners, leading to  cost and schedule increase which make it hard for the agency to manage its  portfolio and make informed investment decisions.
NASA’s comments are reprinted in appendix I. NASA also provided  technical comments, which we addressed throughout the report as  appropriate and where sufﬁ cient evidence was provided to support  signiﬁ cant changes.
We will send copies of the report to NASA’s Administrator and interested  congressional committees. We will also make copies available to others  upon request. In addition, the report will be available at no charge on GAO’s  Web site at http://www.gao.gov.
Should you or your staff have any questions on matters discussed in this  report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact  points for our Ofﬁ ces of Congressional Relations and Public Affairs  may be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix VI.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: Our objectives were to report on the status and challenges faced by NASA  systems with life-cycle costs of $250 million or more and to discuss broader  trends faced by the agency in its management of system acquisitions.
In conducting our work, we evaluated performance and identiﬁ ed  challenges for each of 19 major projects. We summarized our assessments  of each individual project in two components—a project proﬁ le and a  detailed discussion of project challenges. We did not validate the data  provided by the National Aeronautics and Space Administration (NASA).  However, we took appropriate steps to address data reliability. Speciﬁ cally,  we conﬁ rmed the accuracy of NASA-generated data with multiple sources  within NASA and, in some cases, with external sources. Additionally,  we corroborated data provided to us with published documentation. We  determined that the data provided by NASA project ofﬁ ces were sufﬁ ciently  reliable for our engagement purposes.
We developed a standardized data collection instrument (DCI) that was  completed by each project ofﬁ ce. Through the DCI, we gathered basic  information about projects as well as current and projected development  activities for those projects. The cost and schedule data estimates  that NASA provided were the most recent updates as of October 2009;  performance data that NASA provided were the most recent updates as of  November 2009. At the time we collected the data, 4 of the 19 projects were  in the formulation phase and 15 were in the implementation phase. NASA  only provided cost and schedule data for 14 projects in implementation.  Despite being in the implementation phase, NASA did not provide cost or  schedule data for the Magnetospheric Multiscale (MMS) project. To further  understand performance issues, we talked with ofﬁ cials from most project  ofﬁ ces and NASA’s Ofﬁ ce of Program Analysis and Evaluation (PA&E).
The results collected from each project ofﬁ ce, Mission Directorate, and  PA&E were summarized in a 2-page report format providing a project  overview; key cost, contract, and schedule data; and a discussion of the  challenges associated with the deviation from relevant indicators from  best practice standards. The aggregate measures and averages calculated  were analyzed for meaningful relationships, e.g. relationship between cost  growth and schedule slippage and knowledge maturity attained both at  critical milestones and through the various stages of the project life cycle.  We identiﬁ ed cost and/or schedule growth as signiﬁ cant where, in either  case, a project’s cost and/or its schedule exceeded the baselines that trigger  reporting to the Congress.
To supplement our analysis, we relied on GAO’s work over the past years  examining acquisition issues across multiple agencies. These reports  cover such issues as contracting, program management, acquisition policy,  and estimating cost. GAO also has an extensive body of work related to  challenges NASA has faced with speciﬁ c system acquisitions, ﬁ nancial  management, and cost estimating. This work provided the context and basis  for large parts of the general observations we made about the projects we  reviewed. Additionally, the discussions with the individual NASA projects  helped us identify further challenges faced by the projects. Together, the  past work and additional discussions contributed to our development of  a short list of challenges discussed for each project. The challenges we  identiﬁ ed and discussed do not represent an exhaustive or exclusive list.  They are subject to change and evolution as GAO continues this annual  assessment in future years.
Our work was performed primarily at NASA headquarters in Washington,  D.C. In addition, we visited NASA’s Marshall Space Flight Center in  Huntsville, Alabama; Dryden Flight Research Center at Edwards Air Force  Base in California; and Goddard Space Flight Center in Greenbelt, Maryland,  to discuss individual projects. We also met with representatives from  NASA’s Jet Propulsion Lab in Pasadena, California and a provider of NASA  launch services, the United Launch Alliance.
subsections: section_title: Data Limitations
paragraphs: NASA only provided speciﬁ c cost and schedule estimates for 14 of the 19  projects in our review. For one project, the Magnetospheric Multiscale  project, NASA will not formally release its baseline cost and schedule  estimates until the ﬁ scal year 2011 budget submission to Congress, and late  in our review process agency ofﬁ cials notiﬁ ed us that they will not provide  project estimates to GAO until that time. For three of the projects that had  not yet entered implementation, NASA provided internal preliminary  estimated total (life-cycle) cost ranges and associated schedules, from key  decision point B (KDP-B), solely for informational purposes. NASA  formally establishes cost and schedule baselines, committing itself to cost  and schedule targets for a project with a speciﬁ c and aligned set of planned  mission objectives at key decision point C (KDP-C), which follows a non- advocate review (NAR) and preliminary design review (PDR). KDP-C  reﬂ ects the life-cycle point where NASA approves a project to leave the  formulation phase and enter into the implementation phase. NASA  explained that preliminary estimates are generated for internal planning and  ﬁ scal year budgeting purposes at KDP-B, which occurs mid-stream in the  formulation phase, and hence, are not considered a formal commitment by  the agency on cost and schedule for the mission deliverables. NASA ofﬁ cials  contend that because of changes that occur to a project’s scope and  technologies between KDP-B and KDP-C, estimates of project cost and  schedule can change signiﬁ cantly heading toward KDP-C. Finally, NASA did  not provide data for the Global Precipitation Measurement mission because  NASA ofﬁ cials said it did not have a requirement for a KDP-B review,  because it was authorized to be formulated prior to the requirements of  NPR 7120.5D being in place.
section_title: Project Proﬁ le Information on Each Individual Two-Page Assessment
paragraphs: This section of the 2-page assessment outlines the essentials of the project,  its cost and schedule performance, and its status. Project essentials reﬂ ect  pertinent information about each project, including, where applicable, the  major contractors and partners involved in the project. These organizations  have primary responsibility over a major segment of the project or, in some  cases, the entire project.
Project performance is depicted according to cost and schedule changes in  the various stages of the project life cycle. To assess the cost and schedule  changes of each project we obtained data directly from NASA PA&E and  from NASA’s Integrated Budget and Performance documents. For systems  in implementation, we compared the latest available information with  baseline cost and schedule estimates set for each project in the ﬁ scal year  2007, 2008, or 2010 budget request.
All cost information is presented in nominal “then year” dollars for  consistency with budget data. Baseline costs are adjusted to reﬂ ect the  cost accounting structure in NASA’s ﬁ scal year 2009 budget estimates.  For the ﬁ scal year 2009 budget request, NASA changed its accounting  practices from full-cost accounting to reporting only direct costs at the  project level. The schedule assessment is based on acquisition cycle time,  which is deﬁ ned as the number of months between the project start, or  formulation start, and projected or actual launch date. Formulation  start generally refers to the initiation of a project; NASA refers to project  start as key decision point A, or the beginning of the formulation phase.  The preliminary design review typically occurs during the end of the  formulation phase, followed by a conﬁ rmation review, referred to as key  decision point C, which allows the project to move into the implementation  phase. The critical design review is held during the ﬁ nal design period  of implementation and demonstrates that the maturity of the design is  appropriate to support proceeding with full scale fabrication, assembly,  integration, and test. Launch readiness is determined through a launch  readiness review that veriﬁ es that the launch system and spacecraft/ payloads are ready for launch. The implementation phase includes the  operations of the mission and concludes with project disposal.
We assessed the extent to which NASA projects exceeded their cost and  schedule baselines. To do this, we compared the project baseline cost and  schedule estimates with the current cost and schedule data reported by the  project ofﬁ ce in October 2009.
section_title: Project Challenges Discussion on Each Individual Two-Page Assessment
paragraphs: To assess the project challenges for each project, we submitted a data  collection instrument to each project ofﬁ ce. We also held interviews with  representatives from most of the projects to discuss the information on the  data collection instrument. These discussions led to identiﬁ cation of further  challenges faced by NASA projects. These challenges were largely apparent  in the projects that had entered the implementation phase. We then  reviewed pertinent project documentation, such as the project plan,  schedule, risk assessments, and major project reviews.
To assess technology maturity, we asked project ofﬁ cials to assess  the technology readiness levels (TRL) of each of the project’s critical  technologies at various stages of project development. Originally developed  by NASA, TRLs are measured on a scale of one to nine, beginning  with paper studies of a technology’s feasibility and culminating with a  technology fully integrated into a completed product. (See appendix IV  for the deﬁ nitions of technology readiness levels.) In most cases, we did  not validate the project ofﬁ ces’ selection of critical technologies or the  determination of the demonstrated level of maturity. However, we sought to  clarify the technology readiness levels in those cases where the information  provided raised concerns, such as where a critical technology was reported  as immature late in the project development cycle. Additionally, we asked  project ofﬁ cials to explain the environments in which technologies were  tested.
Our best practices work has shown that a technology readiness level of  6— demonstrating a technology as a fully integrated prototype in a relevant  environment—is the level of maturity needed to minimize risks for space  systems entering product development. In our assessment, the technologies  that have reached technology readiness level 6 are referred to as fully  mature because of the difﬁ culty of achieving technology readiness level 7,  which is demonstrating maturity in an operational environment—space.  Projects with critical technologies that did not achieve maturity by the  preliminary design review were assessed as having a technology maturity  project challenge. We did not assess technology maturity for those projects  which had not yet reached the preliminary design review at the time of this  assessment.
To assess the complexity of heritage technology, we asked project ofﬁ cials  to assess the TRL of each of the project’s heritage technologies at various  stages of project development. We also interviewed project ofﬁ cials about  the use of heritage technologies in their projects. We asked them what  heritage technologies were being used, what effort was needed to modify  the form, ﬁ t, and function of the technology for use in the new system,  whether the project encountered any problems in modifying the technology,  and whether the project considered the heritage technology as a risk to the  project. Heritage technologies were not considered critical technologies  by several of the projects we reviewed. Based on our interviews, review  of data from the data collection instruments, and previous GAO work on  space systems, we determined whether complexity of heritage technology  was a challenge for a particular project.
To assess design stability, we asked project ofﬁ cials to provide the  percentage of engineering drawings completed or projected for completion  by the preliminary and critical design reviews and as of our current  assessment. In most cases, we did not verify or validate the percentage  of engineering drawings provided by the project ofﬁ ce. However, we  collected the project ofﬁ ces’ rationale for cases where it appeared that  only a small number of drawings were completed by the time of the design  reviews or where the project ofﬁ ce reported signiﬁ cant growth in the  number of drawings released after CDR. In accordance with GAO’s best  practices, projects were assessed as having achieved design stability if  they had released at least 90 percent of projected drawings by the critical  design review. Projects that had not met this metric were determined to  have a design stability project challenge. Though some projects used other  methods to assess design stability, such as computer and engineering  models and analyses, we did not analyze the use of these other methods and  therefore could not assess the design stability of those projects. We could  not assess design stability for those projects that had not yet reached the  critical design review at the time of this assessment.
To assess whether projects encountered challenges with contractor  performance, we interviewed project ofﬁ cials about their interaction and  experience with contractors. We also relied on interviews we held in 2008  with contractor representatives from Orbital Sciences Corporation, Ball  Aerospace and Technologies Corporation, and Raytheon Space Systems  about their experiences contracting with NASA. We were informed about  contractor performance problems pertaining to their workforce, the  supplier base, and technical and corporate experience. We also discussed  the use of contract fees with NASA and contractor’s representatives.  We  assessed a project as having this challenge if these contractor performance  problems—as conﬁ rmed by NASA and, where possible, the project  contractor—caused the project to experience a cost overrun, schedule  delay, or decrease in mission capability. For projects that did not have a  major contractor, we considered this challenge inapplicable to the project.
To assess whether projects encountered challenges with development  partner performance, we interviewed NASA project ofﬁ cials about  their interaction with international or domestic partners during project  development. Development partner performance was considered a  challenge for the project if project ofﬁ cials indicated that domestic or  foreign partners were experiencing problems with project development  that impacted the cost, schedule, or performance of the project for NASA.  These challenges were speciﬁ c to the partner organization or caused by  a contractor to that partner organization. For projects that did not have  an international or domestic development partner, we considered this  challenge not applicable to the project.
To assess whether projects encountered challenges with funding, we  interviewed ofﬁ cials from NASA’s Program Analysis and Evaluation  Division, NASA project ofﬁ cials, and project contractors about the  stability of funding throughout the project life-cycle. Funding stability  was considered a challenge if ofﬁ cials indicated that project funding had  been interrupted or delayed resulting in an impact to the cost, schedule, or  performance of the project, or if project ofﬁ cials indicated that the project  budgets do not have sufﬁ cient funding in certain years based on the work  expected to be accomplished. We corroborated the funding changes and  reasons with budget documents when available.
To assess whether projects encountered challenges with their launch  manifests, we interviewed NASA Launch Services ofﬁ cials and ofﬁ cials  from one of NASA’s contracted providers for launch services about  project launch scheduling, launch windows, and projects that missed  their opportunities. Launch manifest was considered a challenge if, after  establishing a ﬁ rm launch date, a project had difﬁ culty rescheduling its  launch date because it was not ready, if the project could be affected by  another project slipping its launch, or if there were launch vehicle ﬂ eet  issues. Projects that have not yet entered into the implementation phase  have not yet set a ﬁ rm launch date and were therefore not assessed.
In addition, NASA received an appropriation from the American Recovery  and Reinvestment Act of 2009 (ARRA).  NASA provided a record of projects  involved in our review that received ARRA funds.
The individual project ofﬁ ces were given an opportunity to comment on  and provide technical clariﬁ cations to the 2-page assessments prior to their  inclusion in the ﬁ nal product.
We conducted this performance audit from April 2009 to February 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufﬁ cient,  appropriate evidence to provide a reasonable basis for our ﬁ ndings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our ﬁ ndings and conclusions based  on our audit objectives.
section_title: Appendix III: NASA Life Cycle For Flight Systems Compared to a Knowledge-Based Approach
paragraphs: GAO has previously conducted work on NASA’s acquisition policy for space- ﬂ ight systems, and in particular, on its alignment with a knowledge-based  approach to system acquisitions.  The ﬁ gure below depicts this alignment.
As the ﬁ gure shows, NASA’s policy deﬁ nes a project life cycle in two  phases—the formulation and implementation phases, which are  further divided into incremental pieces: Phase A through Phase F.  Project  formulation consists of Phases A and B, during which time the projects  develop and deﬁ ne the project requirements and cost/schedule basis and  design for implementation, including an acquisition strategy.  During the  end of the formulation phase, leading up to the preliminary design review  (PDR) and non-advocate review (NAR), the project team completes its  preliminary design and technology development.  NASA Interim Directive  NM 7120-81 for NASA Procedural Requirements 7120.5D, NASA Space  Flight Program and Project Management Requirements, specify that the  project complete development of mission-critical or enabling technology,  as needed, with demonstrated evidence of required technology qualiﬁ cation  (i.e., component and/or breadboard validation in the relevant environment)  documented in a technology readiness assessment report.  The project must  also develop, document, and maintain a project management baseline   that includes the integrated master schedule and baseline life-cycle cost  estimate.  Implementing these requirements brings the project closer to  ensuring that resources and needs match, but it is not fully consistent with  knowledge point 1 of the knowledge-based acquisition life-cycle.  Our best  practices show that demonstrating technology maturity at this point in the  system life cycle  should include a system or subsystem model or prototype  demonstration in a relevant environment, not only component validation.   As written, NASA’s policy does not require full technology maturity before a  project enters the implementation phase.
After project conﬁ rmation, the project begins implementation, consisting  of phases C, D, E, and F.  During phases C and D, the project performs  ﬁ nal design and fabrication as well as testing of components and system  assembly, integration, test, and launch.  Phases E and F consist of  operations and sustainment and project closeout.  A second design review,  the critical design review (CDR),  is held during the implementation phase  toward the end of phase C.  The purpose of the CDR is to demonstrate that  the maturity of the design is appropriate to support proceeding with full  scale fabrication, assembly, integration, and test.  Though this review is not  a formal decision review, its requirements for a mature design and ability  to meet mission performance requirements within the identiﬁ ed cost and  schedule constraints are similar to knowledge expected at knowledge point  2 of the knowledge-based acquisition life-cycle.  Furthermore, after CDR,  the project must be approved at KDP D before continuing into the next  phase.
The NASA acquisition life-cycle lacks a major decision review at knowledge  point 3 to demonstrate that production processes are mature.  According to  NASA ofﬁ cials, the agency rarely enters a formal production phase due to  the small quantities of space systems that they build.
section_title: Appendix IV: Technology Readiness Levels
paragraphs: None (paper studies and  analysis)
Invention begins.  Once basic  principles are observed, practical  applications can be invented.  The  application is speculative and there  is no proof or detailed analysis to  support the assumption.  Examples  are still limited to paper studies.
None (paper studies and  analysis)
Active research and development  is initiated. This includes analytical  studies and laboratory studies  to physically validate analytical  predictions of separate elements  of the technology.  Examples  include components that are not yet  integrated or representative.
Analytical studies  and demonstration of  nonscale individual  components (pieces of  subsystem).
Basic technological components  are integrated to establish that the  pieces will work together.  This is  relatively “low ﬁ delity” compared  to the eventual system.  Examples  include integration of “ad hoc”  hardware in a laboratory.
Low ﬁ delity breadboard.  Integration of nonscale  components to show  pieces will work  together.  Not fully  functional or form or  ﬁ t but representative  of technically feasible  approach suitable for  ﬂ ight articles.
Fidelity of breadboard technology  increases signiﬁ cantly.  The basic  technological components are  integrated with reasonably realistic  supporting elements so that the  technology can be tested in a  simulated environment.  Examples  include “high ﬁ delity” laboratory  integration of components.
High ﬁ delity breadboard.   Functionally equivalent  but not necessarily  form and/or ﬁ t (size  weight, materials, etc).  Should be approaching  appropriate scale.  May  include integration of  several components  with reasonably  realistic support  elements/subsystems  to demonstrate  functionality.
Lab demonstrating functionality  but not form and ﬁ t. May include  ﬂ ight demonstrating breadboard  in surrogate aircraft.  Technology  ready for detailed design studies.
Representative model or prototype  system, which is well beyond the  breadboard tested for TRL 5, is  tested in a relevant environment.    Represents a major step up in  a technology’s demonstrated  readiness.  Examples include  testing a prototype in a high ﬁ delity  laboratory environment or in  simulated realistic environment.
Prototype.
Should be very close  to form, ﬁ t and function.  Probably includes the  integration of many  new components and  realistic supporting  elements/subsystems if  needed to demonstrate  full functionality of the  subsystem.
High-ﬁ delity lab demonstration  or limited/restricted ﬂ ight  demonstration for a relevant  environment.  Integration of  technology is well deﬁ ned.
Prototype near or at planned  operational system.  Represents a  major step up from TRL 6, requiring  the demonstration of an actual  system prototype in a realistic  environment, such as in an aircraft,  vehicle or space.  Examples include  testing the prototype in a test bed  aircraft.
Prototype. Should be  form, ﬁ t and function  integrated with other  key supporting  elements/subsystems  to demonstrate  full functionality of  subsystem.
Flight demonstration in  representative realistic environment  such as ﬂ ying test bed or  demonstrator aircraft.
Technology is well substantiated  with test data.
Technology has been proven to  work in its ﬁ nal form and under  expected conditions. In almost all  cases, this TRL represents the  end of true system development.   Examples include developmental  test and evaluation of the system  in its intended weapon system  to determine if it meets design  speciﬁ cations.
Actual application of the technology  in its ﬁ nal form and under  mission conditions, such as those  encountered in operational test  and evaluation.   In almost all  cases, this is the end of the last  “bug ﬁ xing” aspects of true system  development.  Examples include  using the system under operational  mission conditions.
section_title: Appendix V: NASA Projects Receiving Additional Funding
paragraphs: There are 7 NASA projects in our review, including all of those in  formulation, that are receiving money  from the American Recovery and  Reinvestment Act (ARRA) of 2009.  See table 4 below for the NASA  projects in our review receiving this funding and the intended use of those  funds.
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Jim Morrison, Assistant Director;   Jessica M. Berkholtz; Greg Campbell; Richard A. Cederholm; Kristine R.  Hassinger; Jeff R. Jensen; Kenneth E. Patton; Brian A. Tittle; and Letisha T.  Watson made key contributions to this report.