NASA: Preliminary Observations on Major Acquisition Projects and Management Challenges
Released: 2016-02-25T00:00:00

Letter

Background

NASA’s mission is to drive advances in science, technology, aeronautics, and space exploration and contribute to education, innovation, our country’s economic vitality, and the stewardship of the Earth. To accomplish this mission, NASA establishes programs and projects that rely on complex instruments and spacecraft. NASA’s portfolio of major projects ranges from space satellites equipped with advanced sensors to study the Earth to a spacecraft which plans to return a sample from an asteroid to a telescope intended to explore the universe to spacecraft to transport humans and cargo to and beyond low-Earth orbit. Some of NASA’s projects are expected to incorporate new and sophisticated technologies that must operate in harsh, distant environments.

The life cycle for NASA space flight projects consists of two phases— formulation, which takes a project from concept to preliminary design, and implementation, which includes building, launching, and operating the system, among other activities. NASA further divides formulation and implementation into phase A through phase F. Major projects must get approval from senior NASA officials at key decision points before they can enter each new phase. Figure 1 depicts NASA’s life cycle for space flight projects.

Formulation culminates in a review at key decision point C, known as project confirmation, where cost and schedule baselines are established and documented in a decision memorandum. To inform those baselines, each project with a life-cycle cost estimated to be greater than $250 million must also develop a joint cost and schedule confidence level (JCL). The JCL initiative, adopted in January 2009, is a point-in-time estimate that, among other things, includes all cost and schedule elements, incorporates and quantifies known risks, assesses the impacts of cost and schedule to date, and addresses available annual resources. NASA policy requires that projects be baselined and budgeted at the 70 percent confidence level.

Our ongoing work on NASA’s major projects includes assessments of 18 major NASA projects. Figure 2 includes more information on the projects.

NASA Cost and Schedule Performance and Implementation of Best Practices

Our ongoing work indicates that NASA has made progress over the past 5 years in a number of key acquisition management areas, but it faces significant risks in some of its major projects. On the positive side, the cost and schedule performance of NASA’s portfolio of major projects in development has improved and most current projects are adhering to their committed cost and schedule baselines. In addition, NASA has maintained recent improvements in the implementation of key product development best practices, which can help reduce risk in projects. Our preliminary results indicate that although NASA’s overall performance has improved, its portfolio of major projects continues to experience cost and schedule growth and development risks in major projects, such as Orion and the Space Launch System, warrant the committee’s continued attention.

Overall Cost Performance of the Portfolio

Our preliminary results show that the cost and schedule performance of NASA’s portfolio of major projects in development continues to improve. In 2016, overall development cost growth for the portfolio of 12 development projects, excluding the James Webb Space Telescope (JWST), fell to 1.3 percent and launch delays averaged 4 months. Both of those measures are at or near the lowest levels we have reported since we began our annual reviews in 2009 (see fig. 3).

NASA has made positive changes in the past 5 years that have helped contribute to the improved performance of its projects. Among other things, we previously reported that NASA adopted a new policy to help project officials with management, cost and schedule estimating, and maintenance of adequate levels of reserves; established a management review process to enable NASA’s senior management to more effectively monitor a project’s performance, including cost, schedule, and cross- cutting technical and nontechnical issues; and has improved external oversight by increasing transparency into project costs. Congressional action has also helped improve visibility into NASA’s cost and schedule performance. In 2005, Congress required NASA to report cost and schedule baselines for all programs and projects with estimated life-cycle costs of at least $250 million that have been approved to proceed to implementation. Congress also required NASA to report to it when development cost growth or schedule delays exceeded certain thresholds.

Our ongoing work indicates that NASA’s most recent improvements in its overall cost performance have also been driven, in part, by the addition of new, large programs to the portfolio. The cost and schedule performance of any portfolio is affected by its composition. New projects are less likely to have experienced cost and schedule growth than older ones, so they generally help improve portfolio performance. Eight of the 12 major projects in development established baselines within the last 2 years, and cost and schedule performance collectively has improved as projects in the portfolio have become, on average, younger. We will continue to monitor these trends as NASA’s current major projects progress through the project life cycle to see if the improvements in the portfolio’s cost and schedule performance are sustained.

Project Rebaselines

Our ongoing work shows that most current NASA projects have stayed within the cost and schedule estimates in their development baselines, both this year and throughout their life cycles, but the portfolio continues to experience cost and schedule growth. This growth was driven by projects that experienced significant cost growth and exceeded their development cost baselines. When a project exceeds its development cost baseline by 30 percent, it is rebaselined if it is to be continued. NASA has rebaselined a major project each year for 8 out of the last 9 years. Table 1 shows the development cost growth for each of the rebaselined projects.

Our ongoing work also shows that the cost growth associated with rebaselined projects often overwhelms the positive cost performance within the remainder of the portfolio both on an annual and life-cycle basis. In July 2015, NASA approved a new baseline for the Space Network Ground Segment Sustainment (SGSS) project, which increased its estimated development costs from $368 million to $677 million and extended its completion date from June 2017 to September 2019. Cost growth from the SGSS was not offset by better performing projects, such as the Origins-Spectral Interpretation-Resource Identification-Security- Regolith Explorer (OSIRIS-REx) asteroid sampling mission. OSIRIS-REx reported lower than expected development costs for the second consecutive year, even though it is at a stage in the life cycle when projects often realize cost growth. The project attributes its $78.2 million decrease in development cost to several factors, including a mature mission concept and rigorous risk management process.

Our preliminary results indicate that the projects in NASA’s current portfolio with the highest development costs, including Space Launch System and Orion, are entering the stage when most rebaselines occur. Projects appear most likely to rebaseline between their critical design and system integration reviews. All eight major projects that rebaselined during the last 9 years did so after their critical design review and the three projects in the 2016 portfolio that rebaselined did so before holding their systems integration review. Nine projects in the current portfolio are in this stage of development—Exploration Ground Systems; Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2); Ionospheric Connection (ICON); JWST; Orion; SGSS; Space Launch System; Solar Probe Plus (SPP); and Transiting Exoplanet Survey Satellite (TESS). Three projects—ICESat-2, JWST, and SGSS—have already rebaselined. If a rebaseline occurs on any of the other six projects, it could add anywhere from almost $60 million to more than $2 billion to the development cost of the portfolio. We will continue to examine these nine projects as part of our annual assessments until they launch, but they also warrant the committee’s continued oversight attention.

Our ongoing work has also found that the Space Launch System and Orion, the two largest projects in this critical stage of development, face cost, schedule, and technical risks. For example, the Space Launch System program has expended significant amounts of schedule reserve over the past year to address delays with development of the core stage, which is the Space Launch System’s propellant tank and structural backbone. The Orion program continues to face design challenges, including redesigning the heat shield following the determination that the previous design used in the first flight test in December 2014 would not meet requirements for the first uncrewed flight. The standing review boards for each program have raised concerns about the programs’ ability to remain within their cost and schedule baselines. If cost overruns materialize on these programs, they could have a ripple effect on the portfolio and result in the potential postponement or even force the cancellation of projects in earlier stages of development. We have ongoing work on both of these programs and we plan to issue reports on them later this summer.

Implementation of Development Best Practices

Our ongoing work indicates that NASA has maintained recent improvements in the technology maturity and design stability of its projects as measured against best practices. As of 2015, 9 of the 11 major projects in NASA’s 2016 portfolio that have passed the preliminary design review have matured all heritage or critical technologies to a technology readiness level (TRL) 6—a large increase since 2010 (see fig. 4). The 12th project in development, Exploration Ground Systems, did not report any critical or heritage technologies, so it was omitted from this analysis.

Our prior best practices work has shown that reaching a TRL 6—which indicates that a representative prototype of the technology has been demonstrated in a relevant environment that simulates the harsh conditions of space—can minimize risks for space systems entering product development. Projects falling short of this standard before the preliminary design review, a milestone that generally precedes the project’s final design and fabrication phase, may experience subsequent technical problems, which can result in cost growth and schedule delays.

Our ongoing work indicates that NASA has also sustained improvements it has made since 2010 in the design stability of its major projects. The average percentage of engineering drawings released at critical design review for NASA’s 2016 portfolio of major projects was 72 percent, roughly the same percentage as last year. This is a significant improvement since 2010, but is still short of the GAO-identified best practice of 90 percent (see fig. 5). Further, a majority of projects in development maintained mass and power reserves that met or exceeded NASA requirements. NASA projects have also continued to minimize design changes after the critical design review—another measure of design stability.

Our prior work on product development best practices shows that at least 90 percent of engineering drawings should be releasable by the critical design review to lower the risk of subsequent cost and schedule growth. The NASA Systems Engineering Handbook also includes this metric. In 2012, NASA established additional technical leading indicators to assess design maturity. These indicators include (1) the percentage of actual mass margin versus planned mass margin and (2) the percentage of actual power margin versus planned power margin. NASA has updated its project management policy and its systems engineering policy to require projects to track these metrics. Projects that do not achieve design stability by critical design review may experience design changes and manufacturing problems, which can result in cost growth and schedule delays.

Management Challenges

NASA’s portfolio is composed of a few large projects that face a lot of pressures and challenges. Any cost growth within these projects can have grave consequences for smaller projects that are critical to a number of scientific endeavors. In November 2015, the NASA Office of the Inspector General issued its annual report on NASA’s top management and performance challenges. Examples of challenges identified in the report include managing NASA’s science portfolio, space flight operations in low earth orbit, positioning NASA for deep space exploration, and securing NASA’s information technology systems and data. We agree with the challenges identified by the Inspector General and our ongoing and prior work has highlighted additional areas where it will be important for NASA to continue its efforts to reduce acquisition risk, including implementing project management tools, demonstrating sustained cost and schedule performance, and developing plans that will help the agency appropriately direct future investments.

Implementation of Management Tools

As part of our ongoing work, we found that NASA is taking steps to improve its project management tools but has not yet fully implemented best practices.

Earned Value Management. NASA has made progress implementing earned value management (EVM) analysis—a key project management tool—but the agency has not yet fully implemented a formal EVM surveillance plan in accordance with best practices. EVM has been a critical part of the agency’s efforts to understand project development needs and to reduce cost and schedule growth. When implemented well, EVM integrates information on a project’s cost, schedule, and technical efforts for management and decision makers by measuring the value of work accomplished in a given period and comparing it with the planned value of work scheduled for that period and the actual cost of work accomplished. NASA has made progress rolling out EVM at its centers and is supporting these efforts with training, including classroom and online training to projects at its various centers.

In 2012, we recommended that NASA require projects to implement formal EVM surveillance programs. NASA partially concurred, but according to NASA officials, they have not implemented the recommendation due to resource constraints. Proper surveillance of EVM contractor data is a best practice in the NASA Earned Value Management Implementation Handbook and GAO’s Cost Estimating and Assessment Guide. Without implementing proper surveillance, a project may be utilizing unreliable EVM data to inform its cost and schedule decision making. NASA has taken other steps to address the intent of our recommendation, but we continue to find issues with the quality of EVM data. In our December 2015 review of the James Webb Space Telescope, we found project EVM data anomalies and recommended that project officials require the contractors to explain and document all such anomalies in their monthly EVM reports. A continuous surveillance program could have identified these anomalies earlier, allowing the project to pursue corrective action with its contractors. NASA concurred with this recommendation and recently sent us documentation concerning steps it has taken to address it. We are currently reviewing that information to determine if NASA has implemented the recommendation.

Joint Confidence Level. In 2009, in order to ensure that cost and schedule estimates were realistic and projects thoroughly planned for anticipated risks, NASA began requiring that programs and projects with estimated life-cycle costs of $250 million or more develop a JCL prior to key decision point C. However, there is no requirement for NASA projects to update their JCLs and our prior work has found that projects do not regularly update cost risk analyses to take into account newly emerged risks. Our cost estimating best practices recommend that cost estimates should be updated to reflect changes to a program or kept current as it moves through milestones. As new risks emerge on a project, an updated cost risk analysis can provide realistic estimates to decision-makers, including the Congress. This is especially true for NASA’s largest projects as updated estimates may require the Congress to consider a variety of actions.

Schedule Development. Our best practices work stresses the importance of a reliable schedule because not only is it a road map for systematic project execution, but also a means by which to gauge progress, identify and resolve potential problems, and promote accountability. According to NASA officials, a project’s ability to efficiently execute a quality JCL analysis is directly tied to the quality of the underlying data, especially a project schedule. Independent assessors—a group of technical experts within NASA who do not actively work on a specific project or program—noted that when they are reviewing a project’s JCL, one of the most common areas that projects struggle with is developing a reliable schedule. For example, our ongoing work found that the Orion program’s standing review board raised concerns that the program’s schedule is missing activities which could affect the program’s ability to accurately identify what is driving the schedule. Officials in NASA’s Cost Analysis Division told us that various schedule related tools have been developed and already made available to projects and additional tools are in development.

Sustained Cost and Schedule Performance

A key management challenge that NASA faces is whether the improvement in the cost and schedule performance we have seen in the agency’s overall portfolio of major projects can be translated to new, large projects that have been recently baselined and added to the portfolio. These additions include its human spaceflight projects, which includes the Space Launch System, Orion, and Exploration Ground Systems program that is developing systems and infrastructure to support assembly, test, and launch of the Space Launch System and Orion. In our February 2015 High Risk Update, we noted that NASA’s human spaceflight projects are at critical points in implementation and, as I noted earlier, we found that all three projects are entering the stage where most project rebaselines appear most likely to occur—between their critical design and system integrations reviews. This is an area where the agency has not been tested since a similarly large and complex project, the James Webb Space Telescope, underwent a replan in September 2011 that resulted in a 78 percent increase in life-cycle costs—increasing to $8.835 billion— and a schedule delay of 52 months—delaying the planned launch date to October 2018. In addition, NASA will have to demonstrate that it is able to sustain cost and schedule performance in its Commercial Crew Program, which is NASA’s effort to facilitate the private demonstration of safe and reliable transportation services to carry NASA astronauts and cargo to and from the International Space Station. NASA is partnering with commercial providers and its approach includes tailoring its spaceflight project life cycle. Our high-risk report identified key areas where NASA could better anticipate and mitigate risks with respect to these human spaceflight programs, including ensuring that adequate and ongoing assessments of risks are conducted given that the impacts of any potential miscalculations will be felt across the portfolio, ensuring that projects’ JCLs are updated regularly, and ensuring that the long-term project costs are understood.

Long-Term Planning and Stability

Our ongoing and prior work has also found that NASA has established cost and schedule baselines for the Space Launch System, Orion, and Exploration Ground Systems, but the baselines provide little visibility into long-term planning and costs. The baselines for the Space Launch System and Exploration Ground Systems are through the first Exploration Mission (EM-1), during which NASA plans to fly an uncrewed Orion some 70,000 kilometers beyond the moon, and the Orion program’s baseline is through the second Exploration Mission (EM-2), which NASA plans to fly beyond the moon to further test performance with a crewed Orion vehicle. In October 2015, NASA issued its Journey to Mars, which NASA identifies as a document that, among other things, communicates its strategy and plans to get to Mars. However, the document does not provide additional details on future exploration missions, making it difficult to understand NASA’s vision for what type and how many missions it will take to get to Mars. Without this information, decisionmakers do not have visibility into how NASA expects to invest to develop, operate, and sustain a capability over the long term. Having a complete picture of costs can enable both the Congress and the administration to set priorities for both the short and long term. In May 2014, we recommended that NASA establish separate cost and schedule baselines for each additional capability that encompass all life-cycle costs, to include operations and sustainment. NASA partially concurred with our recommendation and stated that it had established separate programs for Space Launch System, Orion, and Exploration Ground Systems. Further, NASA stated that the Space Launch System program had gone further by adopting a block upgrade approach to ensure more realistic long-range investment planning and more effective resource allocations through the budget process. However, NASA stated that it does not intend to carry life-cycle estimates for the Space Launch System program through an end-of- program date because the strategic parameters of such analysis are in the process of being defined. NASA has yet to take action on this recommendation.

The Space Leadership Preservation Act of 2015

The various provisions of the act being discussed today propose changes in NASA’s leadership structure and long-term contracting authorities, among other areas. In a prior testimony, sponsors of the act emphasized that the provisions are aimed at making NASA more professional and less political by giving the agency greater stability. The concept of stability is an important one for NASA since projects require heavy investments— both in terms of time and money—and require cooperation and support from a variety of communities, who sometimes have competing interests, including academic institutions, partnering countries, the science community, and industry, to name a few. We have not studied how the act’s specific provisions, including the types of leadership structures being proposed, could affect stability for NASA’s projects. However, based on our prior work on NASA’s and the Department of Defense’s (DOD) acquisition management efforts, we would like to offer the following observations: If NASA were to implement a board of directors as outlined in the proposed legislation, the board itself must be willing to hold program managers and leadership accountable by canceling programs that do not perform well. If programs with an unsound business case are allowed to continue, their poor performance could have dramatic consequences on the overall portfolio. Insight into program performance, independent assessments, and regular reporting on progress are all necessary tools to enable leadership to hold managers accountable.

DOD has used multiyear contracts under other authorities to acquire weapon systems and believes these tools are helpful in negotiating lower prices. However, longer term commitments to contracts will not necessarily produce better results if they are not accompanied by best practices.

Our past work at DOD has found that it is difficult to precisely determine the impact of multiyear contracting executed under a different authority on actual procurement costs and that savings did not appear to have materialized as expected in budget justifications to Congress in three case studies we looked at, and ultimately more funding was needed to buy the systems. Further, multiyear procurement contracts can provide stability for contractors doing business with the government, but they also can reduce Congress’s and NASA’s flexibility in making changes to programs and budgets unless the government is willing to pay the cancellation fees associated with doing so.

In closing, I would like to emphasize that achieving stability through leadership and contracting changes may offer benefits, but one of the most important factors in achieving stability is a sound business case that balances the necessary resources—technologies, design knowledge, funding, and time—needed to transform a chosen concept into a product. As our ongoing and prior work shows, more effort is still needed to improve NASA’s cost estimating, scheduling practices, and contractor oversight. Robust, long-term plans and realistic estimates are also needed to guide decisions and to secure longer term support. We look forward to continuing to work with NASA and this Committee in instituting these improvements.

Chairman Smith, Ranking Member Johnson, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time.

GAO Contact and Staff Acknowledgments

If you or your staff have any questions about this testimony, please contact Cristina T. Chaplain, Director, Acquisition and Sourcing Management at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this statement include Ronald Schwenn, Assistant Director; Molly Traci, Assistant Director; Laura Greifner; Kurt Gurka; Katherine Lenane; Erin Preston; Roxanna Sun; and Kristin Van Wychen.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.