NASA: Projects Need More Disciplined Oversight and Management to Address Key Challenges
Published: 2009-03-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: This testimony discusses the National Aeronautics and Space Administration's (NASA) oversight and management of its major projects. As you know, in 1990, GAO designated NASA's contract management as high risk in view of persistent cost growth and schedule slippage in the majority of its major projects. Since that time, GAO's high-risk work has focused on identifying a number of causal factors, including antiquated financial management systems, poor cost estimating, and undefinitized contracts. Because cost growth and schedule delays persist, this area - now titled acquisition management because of the scope of issues that need to be resolved - remains high risk. To its credit, NASA has recently made a concerted effort to improve its acquisition management. In 2007, NASA developed a comprehensive plan to address systemic weaknesses related to how it manages its acquisitions. The plan specifically seeks to strengthen program/project management, increase accuracy in cost estimating, facilitate monitoring of contractor cost performance, improve agency wide business processes, and improve financial management. While we applaud these efforts our recent work has shown that NASA needs to pay more attention to effective project management. It needs to adopt best practices that focus on closing gaps in knowledge about requirements, technologies, funding, time and other resources before it makes commitments to large-scale programs. For instance, the Mars Science Laboratory, which was already over budget, recently announced a 2-year launch delay. Current estimates suggest that the price of this delay may be $400 million--which drives the current project life-cycle cost estimate to $2.3 billion; up from its initial confirmation estimate of $1.6 billion. Also, in just one year, the development costs of NASA's Glory mission increased by 54 percent, or almost $100 million, because of problems NASA's contractor is having developing a key sensor. Total project costs for another project, Kepler have increased almost another $100 million within 2 fiscal years because of similar issues. Taken together, these and other unanticipated cost increases hamper NASA's ability to fund new projects, continue existing ones, and pave the way to a post-shuttle space exploration environment. Given the constrained fiscal environment and pressure on discretionary spending it is critical that NASA get the most out of its investment dollars for its space systems. The agency is increasingly being asked to expand its portfolio to support important scientific missions including the study of climate change. Therefore, it is exceedingly important that these resources be managed as effectively and efficiently as possible for success. The recent launch failure of the Orbiting Carbon Observatory is an all-too-grim reminder of how much time, hard work, and resources can be for naught when a space project cannot execute its mission.
section_title: What GAO Found
paragraphs: We assessed 18 projects in NASA's current portfolio. Four were in the "formulation" phase, a time when system concepts and technologies are still being explored and 14 were in the "implementation" phase, where system design is completed, scientific instruments are integrated, and a spacecraft is fabricated. When implementation begins, it is expected that project officials know enough about a project's requirements and what resources are necessary to meet those requirements that they can reliably predict the cost and schedule necessary to achieve its goals. Reaching this point requires investment. In some cases, projects that we reviewed spent 2 to 5 years and up to $100 million or more before being able to formally set cost and schedule estimates. Ten of the projects in our assessment for which we received data and that had entered the implementation phase experienced significant cost and/or schedule growth from their project baselines.3 Based on our analysis, development costs for projects in our review increased by an average of almost 13 percent from their baseline cost estimates--all in just two or three years--including one that went up more than 50 percent. It should be noted that a number of these projects had experienced considerably more cost growth before a baseline was established in response to statutory reporting requirements. Our analysis also shows that projects in our review had an average delay of 11 months to their launch dates. We found challenges in five areas that occurred throughout the various projects we reviewed that can contribute to project cost and schedule growth. These are not necessarily unique to NASA projects and many have been identified in other weapon and space systems that we have reviewed and have been prevalent in the agency for decades. (1) Technology maturity. Four of the 13 projects in our assessment for which we received data and that had entered the implementation phase did so without first maturing all critical technologies, that is they did not know that technologies central to the project's success could work as intended before beginning the process of fabricating the spacecraft. (2) Design stability. The majority of the projects in our assessment that held a critical design review did so without first achieving a stable design. If design stability is not achieved, but product development continues, costly re-designs to address changes to project requirements and unforeseen challenges can occur. (3) Complexity of heritage technology. More than half the projects in the implementation phase--eight of them--encountered challenges in integrating or modifying heritage technologies. (4) Contractor performance. Six of the seven projects that cited contractor performance as a challenge also experienced significant cost and/or schedule growth. (5) Development partner performance. Five of the thirteen projects we reviewed encountered challenges with a development partner. In these cases, the development partner could not meet its commitments to the project within planned timeframes. This may have been a result of problems within the specific development partner organization or as a result of problems faced by a contractor to that development partner.

section_title: Letter
paragraphs: Thank you for inviting me to discuss the National Aeronautics and Space  Administration’s (NASA) oversight and management of its major projects.  As you know, in 1990, GAO designated NASA’s contract management as  high risk in view of persistent cost growth and schedule slippage in the  majority of its major projects. Since that time, GAO’s high-risk work has  focused on identifying a number of causal factors, including antiquated  financial management systems, poor cost estimating, and undefinitized  contracts. Because cost growth and schedule delays persist, this area –  now titled acquisition management because of the scope of issues that  need to be resolved – remains high risk.
To its credit, NASA has recently made a concerted effort to improve its  acquisition management. In 2007, NASA developed a comprehensive plan  to address systemic weaknesses related to how it manages its acquisitions.  The plan specifically seeks to strengthen program/project management,  increase accuracy in cost estimating, facilitate monitoring of contractor  cost performance, improve agency wide business processes, and improve  financial management.
While we applaud these efforts our recent work has shown that NASA  needs to pay more attention to effective project management. It needs to  adopt best practices that focus on closing gaps in knowledge about  requirements, technologies, funding, time and other resources before it  makes commitments to large-scale programs. For instance, the Mars  Science Laboratory, which was already over budget, recently announced a  2-year launch delay. Current estimates suggest that the price of this delay  may be $400 million—which drives the current project life-cycle cost  estimate to $2.3 billion; up from its initial confirmation estimate of $1.6  billion. Also, in just one year, the development costs of NASA’s Glory  mission increased by 54 percent, or almost $100 million, because of  problems NASA’s contractor is having developing a key sensor. Total  project costs for another project, Kepler have increased almost another  $100 million within 2 fiscal years because of similar issues. Taken  together, these and other unanticipated cost increases hamper NASA’s  ability to fund new projects, continue existing ones, and pave the way to a  post-shuttle space exploration environment.
Given the constrained fiscal environment and pressure on discretionary  spending it is critical that NASA get the most out of its investment dollars  for its space systems. The agency is increasingly being asked to expand its  portfolio to support important scientific missions including the study of  climate change. Therefore, it is exceedingly important that these resources  be managed as effectively and efficiently as possible for success. The  recent launch failure of the Orbiting Carbon Observatory is an all-too-grim  reminder of how much time, hard work, and resources can be for naught  when a space project cannot execute its mission.
In response to congressional direction, we have prepared a comprehensive  report on the management and oversight of NASA’s major projects. It  contains summaries of 18 projects with a combined life-cycle cost  exceeding $50 billion. It also contains an assessment of issues affecting  projects across-the-board. A copy of this report is now available on GAO’s  Website (www.gao.gov). In conducting this work, we compared projects  against best practice criteria for system development including attainment  of knowledge on technologies and design, as well as various aspects of  program management. We expect to continue this assessment on an  annual basis and to continually refine our examination so that our work  can inform your oversight and NASA’s own efforts to improve in the high  risk area of acquisition management.
In responding to our report, NASA asserted that the unique nature of its  work and external factors beyond its control make it difficult to apply the  same criteria that we apply to other major government acquisitions,  particularly those with large production runs. We disagree. The criteria we  used to assess NASA’s projects represent commonly accepted,  fundamental tenets of disciplined project management, regardless of  complexity or quantity. In fact, the concept of the knowledge-based  approach we use has been adopted in NASA’s own acquisition policy. Key  criteria that we use have been developed by NASA and/or incorporated  into its engineering policy. Moreover, facing long-standing cost and  schedule growth and performance shortfalls, the Department of Defense  (DOD) acknowledges the need for a knowledge based approach in the Air  Force’s “back to basics” policy for space systems. Lastly, we remain open  to discussions with NASA as to whether additional criteria can and should  be applied to its systems to ensure that decisions to move forward in  development are well-informed and ultimately, that taxpayer dollars are  well spent.
Today I will be highlighting the results of this work, the actions NASA is  taking to address the concerns raised in our high risk report and better  position its projects to meet their goals, and what we believe is necessary  to make these actions successful. Because we also have responsibility for  examining military space systems, we will also highlight common  challenges with space acquisitions within NASA and the Department of  Defense (DOD). This testimony is based on previously issued GAO work,  which was conducted in accordance with generally accepted government  auditing standards.
subsections: section_title: Acquisition Management Problems Persist
paragraphs: We assessed 18 projects in NASA’s current portfolio. Four were in the  “formulation” phase, a time when system concepts and technologies are  still being explored and 14 were in the “implementation” phase, where  system design is completed, scientific instruments are integrated, and a  spacecraft is fabricated. When implementation begins, it is expected that  project officials know enough about a project’s requirements and what  resources are necessary to meet those requirements that they can reliably  predict the cost and schedule necessary to achieve its goals. Reaching this  point requires investment. In some cases, projects that we reviewed spent  2 to 5 years and up to $100 million or more before being able to formally  set cost and schedule estimates.
Ten of the projects in our assessment for which we received data and that  had entered the implementation phase experienced significant cost and/or  schedule growth from their project baselines. Based on our analysis,  development costs for projects in our review increased by an average of  almost 13 percent from their baseline cost estimates—all in just two or  three years—including one that went up more than 50 percent. It should  be noted that a number of these projects had experienced considerably  more cost growth before a baseline was established in response to  statutory reporting requirements. Our analysis also shows that projects in  our review had an average delay of 11 months to their launch dates.
We found challenges in five areas that occurred throughout the various  projects we reviewed that can contribute to project cost and schedule  growth. These are not necessarily unique to NASA projects and many have  been identified in other weapon and space systems that we have reviewed  and have been prevalent in the agency for decades.
Technology maturity. Four of the 13 projects in our assessment for which  we received data and that had entered the implementation phase did so  without first maturing all critical technologies, that is they did not know  that technologies central to the project’s success could work as intended  before beginning the process of fabricating the spacecraft. This means that  knowledge needed to make these technologies work remained unknown  well into development. Consequences accrue to projects that are still  working to mature technologies well into system development, when they  should be focusing on maturing system design and preparing for  production. Simply put, projects that start with mature technologies  experience less cost growth than those that start with immature  technologies.
Design stability. The majority of the projects in our assessment that held  a critical design review did so without first achieving a stable design. If  design stability is not achieved, but product development continues, costly  re-designs to address changes to project requirements and unforeseen  challenges can occur. All of the projects in our assessment that had  reached their critical design review and that provided data on engineering  drawings experienced some growth in the total number of design drawings  after their critical design review. Growth ranged from 8 percent to, in the  case of two projects, well over 100 percent. Some of this increase can be  attributed to change in system design after critical design review.
Complexity of heritage technology. More than half the projects in the  implementation phase—eight of them—encountered challenges in  integrating or modifying heritage technologies. Additionally, two projects  in formulation—Ares I and Orion—also encountered this problem. We  found that the projects that relied on heritage technologies  underestimated the effort required to modify them to the necessary form,  fit, or function.
Contractor performance. Six of the seven projects that cited contractor  performance as a challenge also experienced significant cost and/or  schedule growth. For example, through our discussions with the project  offices, we were informed that contractors encountered technical and  design problems with hardware that disrupted development progress.
Development partner performance. Five of the thirteen projects we  reviewed encountered challenges with a development partner. In these  cases, the development partner could not meet its commitments to the  project within planned timeframes. This may have been a result of  problems within the specific development partner organization or as a  result of problems faced by a contractor to that development partner.
subsections: section_title: Common Acquisition Management Challenges Persist between NASA and DOD
paragraphs: The challenges we identified in the NASA assessment are similar to ones  we have identified in other weapon systems, including Defense space  systems. We testified last year that DOD space system cost growth was  attributable to programs starting before they have assurance that  capabilities being pursued can be achieved within available resources and  time constraints. For example, DOD’s National Polar-orbiting Operational  Environmental Satellite System (NPOESS) has increased in cost from  roughly $6 billion to over $11 billion because of challenges with maturing  key technologies. We have also tied acquisition problems in space systems  to inadequate contracting strategies and contract and program  management weaknesses. Further, we issued a report in 2006 that found  DOD space system cost estimates were consistently optimistic. For  example, DOD’s Space-Based Infrared High program was originally  expected to cost about $4 billion and is now expected to cost more than  $10 billion.
We have found these problems are largely rooted in the failure to match  the customer’s needs with the developer’s resources—technical  knowledge, timing, and funding—when starting product development. In  other words, commitments were made to achieving certain capabilities  without knowing whether technologies and/or designs being pursued  could really work as intended. Time and costs were consistently  underestimated. As we have discussed in previous work on space systems  at both DOD and NASA, a knowledge-based approach to acquisitions,  regardless of the uniqueness or complexity of the system is beneficial  because it allows program managers the opportunity to gain enough  knowledge to identify potential challenges earlier in development and  make more realistic assumptions about what they can achieve.
section_title: NASA Is Making a Concerted Effort to Reduce High Risk in Acquisition Management but More Needs to Be Done
paragraphs: NASA has also taken significant steps to improve in the high risk area of  acquisition management. For example, NASA revised its acquisition and  engineering polices to incorporate elements of a knowledge-based  approach that should allow the agency to make informed decisions. The  agency is also instituting a new approach whereby senior leadership is  reviewing acquisition strategies earlier in the process and has developed  broad procurement tenets to guide the agency’s procurement practices.  Further, NASA is working to improve management oversight of project  cost, schedule, and technical performance with the establishment of a  baseline performance review with senior management. In order to  improve it’s contracting and procurement process, NASA has instituted an  agency wide standard contract-writing application intended to ensure all  contracts include the most up-to-date NASA contract clauses and to  improve the efficiency of the contracting process. NASA is also requiring  project managers to quantify the program risks they identify and collect  more consistent data on project cost and technologies. It is taking other  actions to enhance cost estimating methodologies and to ensure that  independent estimates are used.
These changes brought the policy more in line with best practices for  product development. However, as we previously reported, NASA lacks  defined requirements across centers and mission directorates for  consistent metrics that demonstrate knowledge attainment through the  development cycle. In order for a disciplined approach to take hold, we  would expect project officials across the agency to be held accountable  for following the same required policies.
More steps also need to be taken to manage risk factors that NASA  believes are outside of its control. NASA asserts that contractor  deficiencies, launch manifest issues, partner performance, and funding  instability are to blame for the significant cost and schedule growth on  many of its projects that we reviewed. Such unforeseen events, however,  should be addressed in project-level, budgeting and resource planning  through the development of adequate levels of contingency funds. NASA  cannot be expected to predict unforeseen challenges, but being disciplined  while managing resources, conducting active oversight of contractors, and  working closely with partners can put projects in a better position to  mitigate these risks should they occur. Realistically planning for and  retiring technical or engineering risks early in product development allows  the project to target reserves to issues NASA believes are outside of its  control.
In conclusion, managing resources as effectively and efficiently as possible  is more important than ever for NASA. The agency is undertaking a new  multi-billion dollar program to develop the next generation of spacecraft  for human spaceflight and at a time when it is faced with increasing  demands to support important scientific missions, including the study of  climate change, and to increase aeronautics research and development. By  allowing major investment commitments to continue to be made with  unknowns about technology and design readiness, contractor capabilities,  requirements, and/or funding, NASA will merely be exacerbating the  inherent risks it already faces in developing and delivering new space  systems. Programs will likely continue to experience problems that require  more time and money to address than anticipated. Over the long run, the  extra investment required to address these problems may well prevent  NASA from pursuing more critical science and space exploration missions.  By contrast, by continuing to implement its acquisition management  reforms and ensuring programs do not move forward with such  unknowns, NASA can better align customer expectations with resources,  minimize problems that could hurt programs, and maximize it ability to  meet increased demands.
Madam Chairwoman, this concludes my statement. I will be happy to  answer any questions that you have.
section_title: GAO Contacts and Staff Acknowledgments
paragraphs: For additional information, please contact Cristina Chaplain at 202-512- 4841 or chaplainc@gao.gov. Individuals making contributions to this  testimony include Jim Morrison, Assistant Director; Shelby S. Oakley,  Assistant Director; Greg Campbell; Richard A. Cederholm; Brendan S.  Culley; Deanna R. Laufer; Kenneth E. Patton; and Letisha T. Watson.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.