Space Launch System: Management Tools Should Better Track to Cost and Schedule Commitments to Adequately Monitor Increasing Risk
Published: 2015-07-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: SLS is NASA's first heavy-lift launch vehicle for human space exploration in over 40 years. For development efforts related to the first flight of SLS, NASA established its cost and schedule commitments at $9.7 billion and November 2018, respectively. The program, however, has continued to pursue more aggressive internal goals for cost and schedule.
GAO was asked to assess a broad range of issues related to the SLS program. This report focuses on NASA's cost estimate for the initial phases of SLS and other management tools needed to control costs. Specifically, this report examines the extent to which SLS's (1) cost and schedule estimates for its first test flight are reliable; (2) cost and schedule reserves are available to maintain progress toward this flight test; and (3) EVM data provides meaningful insight into progress. To do this work, GAO examined documents supporting the cost and schedule estimates, contractor EVM data, and other relevant program documentation, and interviewed relevant officials.
section_title: What GAO Found
paragraphs: The cost and schedule estimates for the National Aeronautics and Space Administration's (NASA) Space Launch System (SLS) program substantially complied with five of six relevant best practices, but could not be deemed fully reliable because they only partially met the sixth best practice—credibility. While an independent NASA office reviewed the estimate developed by the program and as a result the program made some adjustments, officials did not commission the development of a separate independent estimate to compare to the program estimate to identify areas of discrepancy or difference. In addition, the program did not cross-check its estimate using an alternative methodology. The purpose of developing a separate independent estimate and cross-checking the estimate is to test the program's estimate for reasonableness and, ultimately, to validate the estimate. The continued accuracy of the estimates is also questionable because officials have no plans to update the original estimates created in 2013. GAO's cost estimating best practices call for estimates to be continually updated through the life of the program to provide decisionmakers with current information to assess status. Moreover, as stressed in prior GAO reports, SLS cost estimates only cover one SLS flight in 2018 whereas best practices call for estimating costs through the expected life of the program.
Limited cost and schedule reserves place the program at increased risk of exceeding its cost and schedule commitments. Although the SLS program is committed to a November 2018 launch readiness date, it has been pursuing an internal goal for launch readiness of December 2017, with the time between December 2017 and November 2018 being designated as schedule reserve. The SLS program expects to use a significant amount of schedule reserve, in part to address some technical challenges, and plans to shift its internal goal from December 2017 to tentatively July 2018. This shift will reduce the amount of available schedule reserve from 11 months to just 4 months. In addition, the program planned for cost reserves of less than 4 percent each year and has already allocated those funds for this year, which leaves no reserve funding available to address unanticipated issues.
Earned value management (EVM) data for SLS remains incomplete and provides limited insight into progress toward the program's external committed cost and schedule baselines because it tracks progress relative to the program's internal goals—which have proven unrealistic. EVM data is intended to provide an accurate assessment of program progress and alert managers of impending schedule delays and cost overruns. GAO analysis of available SLS contractor EVM data indicated that the contractors may incur cost overruns ranging from about $367 million to about $1.4 billion, which is significantly higher than what the contractors were reporting—$89 million. SLS is implementing a program-level EVM system that, once complete, will include all contractor work and work conducted in-house by NASA and may provide more comprehensive information on program progress relative to internal goals. Tracking to internal goals, however, provides limited information relative to progress toward external commitments. At present, the SLS program lacks comprehensive program-level reporting to alert managers of impending delays and cost overruns to external commitments.
section_title: What GAO Recommends
paragraphs: NASA should direct SLS program officials to update the cost and schedule estimates at least annually, and to implement a mechanism that reports progress relative to external committed cost and schedule baselines on a quarterly basis, among other actions. NASA concurred with GAO's recommendations.

section_title: Letter
paragraphs: The National Aeronautics and Space Administration (NASA) is in the  midst of developing its first exploration-class heavy-lift launch vehicle in  over 40 years—the Space Launch System (SLS). It is being developed to  launch astronauts and carry cargo into space, beyond low-Earth orbit.  The SLS program, along with the Orion Multi-Purpose Crew Vehicle  (Orion) and the associated ground systems program, is estimated to cost   This dollar estimate  nearly $23 billion to demonstrate initial capabilities.includes the first planned SLS flight in 2018, the ground systems for that  effort, and the first two Orion flights currently planned for fiscal years 2018  and 2021 or 2022. This amount represents a significant portion of NASA’s  planned development budget for major projects during that period and  also a considerable portion of government-wide launch-related research  and development funding.
GAO, High Risk Series: An Update, GAO-15-290, (Washington, D.C.: Feb. 11, 2015). development schedule in our July 2014 report by delaying the program’s  committed launch readiness date for its first flight from December 2017  until November 2018. To this point, however, the SLS program has  continued to pursue an internal, more aggressive goal for launch  readiness of December 2017. Given the array of cost and schedule  challenges facing the SLS program, you requested that we assess a  broad range of issues related to the cost estimate and risks faced by the  SLS program. We have already reported on NASA’s preliminary cost  estimates for human exploration systems as well as technical and   This report focuses on NASA’s cost  management risks facing SLS.estimate for the first flight of SLS as well as other management tools  needed to control costs. Specifically, for this review, we examined the  extent to which (1) the cost and schedule estimates for SLS through  launch readiness for the first flight test, Exploration Mission 1 (EM-1),  scheduled for November 2018, are reliable; (2) the program has cost and  schedule reserves available to maintain progress to the first flight test;  and (3) the program’s earned value management (EVM) system provides  insight into progress relative to the program’s cost and schedule  baselines for the first test flight.
GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and  Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: Mar. 2, 2009) and  GAO Schedule Assessment Guide: Best Practices for Project Schedules, GAO-12-120G  (Washington, D.C.: May 30, 2012). program risk assessments; and met with senior agency officials, program  management, and program budget specialists to discuss the program’s  budget and reserve postures. To assess the insight that the program’s  EVM system provides relative to the program’s cost and schedule  baselines for the first test flight, we obtained and analyzed contractor cost  and schedule reporting, or EVM, data for the three contractors for which it  was available, determined whether the contractors are forecasting cost  and schedule growth, and calculated our own forecasts of likely cost and  schedule growth. We also collected available program-level EVM data to  determine whether it could be used as an indicator for program progress.
We conducted this performance audit from September 2014 to July 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The National Aeronautics and Space Administration Authorization Act of  2010 directed NASA to, among other things, develop a space launch  system as a follow-on to the Space Shuttle and as a key component in  expanding human presence beyond low-Earth orbit. In 2011, NASA  formally established the SLS program in response to this direction, and  the Congress has provided continued support for the program. For  example, the Congress has appropriated additional funding for SLS in  each of the past 3 fiscal years above the level requested by the program.  The cumulative additional funding totals about $610 million more than  requested for SLS for fiscal years 2013, 2014, and 2015.
NASA plans to develop three SLS launch vehicle capabilities,  complemented by Orion, to transport humans and cargo into space. The  first version of the SLS is a 70-metric ton launch vehicle known as Block  I.the first in 2018 and the second in 2021/22. The vehicle is scheduled to  fly an uncrewed Orion some 70,000 kilometers beyond the moon during   NASA has committed to conduct two test flights of the Block I vehicle— the first test flight, known as EM-1, and to fly a second mission—EM-2— beyond the moon to further test performance with a crewed Orion vehicle.  After 2021, NASA intends to build 105- and 130-metric ton launch  vehicles, known respectively as Block IA/B and Block II, which it expects  to use as the backbone of manned spaceflight for decades. NASA  anticipates using the Block IA/B vehicles for destinations such as near- Earth asteroids and Lagrange points and the Block II vehicles for eventual  Mars missions.infrastructure and systems needed to support processing and launch of  Orion and SLS at Kennedy Space Center. See figure 1.
The JCL is a quantitative probability analysis that requires the project to combine its  cost, schedule, and risks into a complete quantitative picture to help assess whether the  project will be successfully completed within cost and on schedule. NASA introduced the  analysis in 2009, and it is among the agency’s initiatives to reduce acquisition  management risk. The move to probabilistic estimating marks a major departure from  NASA’s prior practice of establishing a point estimate and adding a percentage on top of  that point estimate to provide for contingencies. NASA’s procedural requirements state  that Mission Directorates should plan and budget programs and projects with an estimated  life-cycle cost greater than $250 million based on a 70 percent JCL, or at a different level  as approved by the Decision Authority, and any JCL approved at less than 70 percent  must be justified and documented. NASA Procedural Requirements (NPR) 7120.5E,  NASA Space Flight Program and Project Management Requirements, paras. 2.4.4 and  2.4.4.1 (Aug. 14, 2012) (hereinafter cited as NPR 7120.5E (Aug. 14, 2012)).
2018 as schedule reserve and the $1.3 billion difference between the  $8.4 billion goal and the $9.7 billion baseline as funding for that schedule  reserve. Unlike cost reserves, however, that funding largely corresponds  to those 11 months and cannot be used separately from schedule to  address problems as they arise.
In December 2014, we testified that the ground systems and Orion  programs will likely not be ready to support EM-1 before November 2018  even if SLS is able to meet its earlier internal goal. During this testimony,  NASA witnesses stated that the SLS program would not be able to meet  its internal goal of December 2017 and that the program would likely slip  the internal goal to summer 2018. See table 1 for more specifics on  estimated launch dates.
section_title: The SLS Cost and Schedule Estimates Were Generally Developed in Accordance with Best Practices, but Officials Have No Plans to Update the Cost Estimate with Actual Costs
paragraphs: NASA generally followed best practices in preparing the SLS cost and  schedule baseline estimates for the limited portion of the program life  cycle covered, that is, through launch readiness for the first test flight of  SLS. We found that the SLS program cost and schedule estimates for this  limited portion of development substantially met three of four cost  characteristics—comprehensive, well documented, and accurate—and  both schedule characteristics—comprehensive and well constructed— that GAO considers best practices for preparing a reliable estimate.  However, because the cost estimate only partially met best practice  criteria for credibility, the fourth cost characteristic, the estimates could  not be deemed fully reliable. See figure 2.
While the cost and schedule estimates were prepared largely in  accordance with best practices, they only represent costs for the first  flight of SLS, EM-1, as opposed to the program’s full life cycle. In May  2014, we recommended that NASA establish a separate cost and  schedule baseline for missions beyond EM-1 and report this information  via its annual budget submission. Additionally, we recommended that  NASA establish life-cycle cost and schedule baselines for each upgraded  block of the SLS. NASA partially concurred with our recommendations,  citing that actions it plans to take to track costs and actions already in  place, such as establishing a block upgrade approach for SLS, met the  intent of our recommendations. To this point, however, NASA has not put  forth any estimates or baselines projecting the costs of future blocks of  the SLS.
subsections: section_title: Cost Estimate
paragraphs: Comprehensive: The SLS cost estimate substantially met the criteria for  being comprehensive through launch readiness for EM-1 but did not  include any costs beyond the first flight. To develop the estimate, officials  used a detailed work breakdown structure—the structure used to define in  detail the work necessary to accomplish program objectives—that is  traceable to the cost of each work element and the contract statement of  work and documented ground rules and assumptions. To fully meet the  criteria for being comprehensive, however, the estimate should define in  detail all costs through the expected life of a program. The estimate  satisfied NASA’s cost estimating approach for Human Exploration  programs by including life-cycle costs through launch readiness for EM-1,  but did not include any costs for deployment and operation and  maintenance of SLS beyond the first flight. These costs will likely far  exceed the costs of development through the first flight. For example, in  October 2009, the Review of U.S. Human Spaceflight Plans Committee  reported that the fixed costs of the facilities and infrastructure associated  with the Shuttle program were about $1.5 billion a year. Given that  NASA hopes to operate the SLS for decades, it is reasonable to expect  that the deployment and operation and maintenance costs—which should  be included in a reliable estimate of life-cycle costs—alone for the SLS  will outweigh the agency’s current estimated cost of $9.7 billion. NASA  has stated that cost estimates do not need to cover the program from  “cradle to grave.” Rather, NASA’s position is that the program’s estimate  is only meaningful up to the time that the SLS is delivered for its first  launch, because the agency is taking what it calls a capability approach.  Therefore, NASA has only estimated costs of the program to the point at  which the capability will be achieved. Furthermore, NASA has yet to  determine the number of launches, their missions, or the operating  lifetime for the program, which according to agency officials makes it  difficult to estimate the total costs of the program. Nevertheless, Office of  Management and Budget guidance and GAO’s Cost Assessment Guide  indicate that life-cycle cost estimates should encompass the full life cycle  of a program.
Well Documented: The SLS cost estimate substantially met the criteria  for being well documented; however, some explanations to support the  estimate were missing. A well-documented cost estimate includes  thorough documentation and is traceable to information sources. We  found that the SLS cost estimate documentation discusses, and is  consistent with, the program’s technical baseline and provides evidence  that the cost estimate was reviewed by management. The estimate also  included explanations for how the estimates for the underlying  components were created through an assessment of likely costs for each  part of the system supplemented with an engineering review. In some  instances, however, explanations of how historical data were normalized,  that is, adjusted to support the estimate, were missing. For example, the  cost estimate does not explain how historical costs for the space shuttle  main engine were normalized to support the estimate. The purpose of  data normalization is to make a given data set consistent with and  comparable to other data used in the estimate so that they can be used  for comparison analysis or as a basis for projecting future costs.  Insufficient documentation of how the historical data were adjusted can  hinder understanding and proper use of the estimate.
Accurate: The SLS cost estimate substantially met the criteria for being  accurate, but the continued accuracy of the estimate is in question  because officials have no plans to periodically update the estimate.  Accurate cost estimates are based on assessments of most likely costs,  adjusted properly for inflation, and contain few, if any, minor mistakes. In  addition, a cost estimate should be updated regularly to reflect significant  changes in the program. The SLS cost estimate meets most of these  characteristics as it is based on an assessment of likely costs, is adjusted  properly for inflation, and contains few if any mistakes. Contrary to best  practices, however, NASA does not periodically update the estimate  based on actuals, which limits its use as a management tool for  monitoring progress and planning future work. The program prepared its  cost estimate and JCL in calendar year 2013. GAO’s cost estimating best  practices call for estimates to be continually updated through the life of  the project, ideally every month as actual costs are reported in earned  value management reports. Best practices also call for a risk analysis and  risk simulation exercise—like the JCL analysis—to be conducted  periodically through the life of the program, as risks can materialize or  change throughout the life of a program. Unless properly updated on a  regular basis, the cost estimate cannot provide decisionmakers with  accurate information to assess the current status of the project.
Agency officials have indicated that the SLS program has no plans to  update the cost and schedule estimates underlying the JCL or the JCL  itself, which calls into question the continued accuracy of the estimates.  NASA’s policy for space flight program and project management requires  a program’s committed cost estimate to be updated (rebaselined) if it  exceeds the external baseline committed cost by 30 percent or more, and  if a project is rebaselined, the JCL should be recalculated and approved  as part of the rebaselining process. The NASA Cost Estimating  Handbook, however, indicates that program cost estimates should be  updated when program content changes and as programs move through  their life-cycle phases and conduct milestone reviews, and recognizes  that estimates regularly updated based on actual program performance  give decisionmakers a clearer picture for major decisions. In addition,  through our work assessing large scale programs at the Department of  Defense we have found that some programs update cost estimates  annually and regularly report progress relative to both threshold and  objective targets.
Credible: The SLS cost estimate only partially met the criteria for  credibility because the SLS program did not cross-check the results of the  estimate and did not commission an independent cost estimate. The  purpose of developing a separate independent estimate and cross- checking the estimate is to test the program’s estimate for  reasonableness and, ultimately, to validate the estimate. Consistent with  best practices, the program conducted a risk and uncertainty analysis that  calculated the likely cost and schedule consequences on the program for  each risk identified and conducted a duration sensitivity analysis to  determine how varying the lengths of different tasks affected the program.  Contrary to best practices, however, the SLS program did not cross- check the results of its cost estimate. The main purpose of cross- checking is to determine whether alternative estimating methods produce  similar results. If cross-checking confirms the results of the estimate, then  confidence in the estimate increases, leading to greater credibility. In  addition, project officials did not commission an independent cost  estimate—a separate estimate produced by an organization outside of  the SLS program chain of command—which is considered one of the best  and most reliable estimate validation methods because it provides an  independent view of expected program costs that tests the program  office’s estimate for reasonableness. An estimate that has not been  reconciled with an independent cost estimate has an increased risk of  being underfunded because the independent cost estimate provides an  objective and unbiased assessment of whether the project estimate is  realistic. Because the cost estimate only partially met the criteria for  credibility, the estimate does not fully reflect the characteristics of a  quality estimate and cannot be considered fully reliable.
While the program did not commission an independent estimate, NASA’s  Independent Program Assessment Office (IPAO)—which reviews NASA  programs at key decision points in the life cycle to support approval  decisions by the agency leadership—did review the program’s cost  estimate at the program’s Key Decision Point C (KDP-C) review. KDP-C  is the point in NASA’s project life cycle where baseline cost and schedule  estimates are established and projects begin implementation. During this  review, the IPAO found that the SLS JCL process and cost model were  sound; however, the IPAO also found that the program’s initial SLS cost  estimate appeared optimistic relative to predictions based on historical  data from similar programs. For example, the IPAO reported that the  program was underestimating the likely range of cost growth for four key  elements—software development, core stage qualification, core stage  testing, and procurement of the interim cryogenic propulsion stage.  Senior agency officials indicated that, based in part on the results of the  IPAO assessment, the program increased its estimate and the agency  established higher cost and schedule baseline commitments for the  program.
section_title: Schedule Estimate
paragraphs: Comprehensive: The SLS schedule estimate substantially met best  practice criteria for being comprehensive through launch readiness for  EM-1 but the schedule estimate did not account for work beyond that  point. A comprehensive schedule includes all activities for both the  government and its contractors necessary to accomplish a project’s  objectives as defined in the project’s work breakdown structure. The SLS  schedule reflected all activities in the program cost work breakdown  structure, resources were appropriately allocated to the schedule, and the  schedule realistically reflected how long each activity would take.  Contrary to best practices, however, the schedule estimate, like the cost  estimate, did not fully account for the deployment or operation and  maintenance of the program—specifically work for flights beyond EM-1.
Well Constructed: The SLS schedule estimate substantially met best  practice criteria for being well constructed but not all activities on the  critical path are directly affecting the finish date of the project. A schedule  is well constructed if all its activities are logically sequenced in the most  straightforward manner possible and it has a reliable critical path that  determines which activities drive the project’s earliest completion date.  We found relatively few instances of activities that were not logically  sequenced, and anomalies within the schedule were, in general, justified  by program officials. For example, program officials explained that some  of the anomalies were due to activities representing external deliveries to  vendors. We found, however, that the schedule did not fully meet the  criteria for a well-constructed schedule because not all activities on the  critical path are truly affecting the finish date of the project, which could  mask delays in the schedule.
section_title: Significant Projected Decrease in Schedule Reserves and Limited Cost Reserves Could Put Launch Readiness Commitment at Risk
paragraphs: The SLS program has limited cost and schedule reserves to address  potential issues as it enters its most challenging period. Schedule reserve  is extra time, with the money to pay for it, in the program’s overall  schedule in the event that there are delays or unforeseen problems. For  the SLS program, the 11-month difference between the program’s internal  launch readiness goal and its committed schedule baseline represents  the program’s schedule reserves. Cost reserves are additional funds that  can be used to mitigate problems during the development of a program.  For example, cost reserves can be used to buy additional materials to  replace a component or, if a program needs to preserve schedule, cost  reserves can be used to accelerate work by adding extra shifts to  expedite manufacturing and save time. Because NASA anticipated a  relatively flat budget for SLS, the agency chose to limit cost reserves and  rely on schedule reserve—the 11 months between the internal launch  readiness goal in December 2017 and the committed baseline in  November 2018—as the primary way to mitigate risk. The SLS program,  however, is planning to use 7 of the 11 months of schedule reserve,  which would delay its planned goal for launch readiness for EM-1 from  December 2017 to, tentatively, July 2018. At this point, the agency has  not, however, delayed its baseline commitment date of November 2018.  As a result, the agency would have only 4 months of schedule reserve  remaining between July 2018 and November 2018 to address any further  problems that it may encounter. See figure 3.
Complex development efforts like SLS must plan to address myriad risks  and unforeseen technical challenges. As mentioned above, cost and  schedule reserves are one way to address risks and challenges. NASA’s  Marshall Space Flight Center, which manages the SLS program, has  guidance requiring programs to present their planned cost and schedule  reserves for approval prior to key milestones, but the guidance does not  establish specific requirements for reserve levels. However, other NASA  centers, such as the Goddard Space Flight Center—the NASA center  with responsibility for managing other complex NASA programs such as  the James Webb Space Telescope—has requirements for the level of  both cost and schedule reserves that projects must have in place at KDP- C. At KDP-C, Goddard flight projects are required to have cost reserves  of 25 percent or more and 1 month of schedule reserve for each year of  development from KDP-C to the start of integration and testing, 2 months  per year for integration and test through shipment to the launch site, and  1 week per month from delivery to launch site to actual launch.
As a result of flat funding requests, the SLS program has very low levels  of cost reserves compared to other programs and to cost reserve  guidance of NASA centers. The IPAO noted that the program’s planned  cost reserves at the time of its review—6 percent—were too low and  compared poorly to other development programs which normally had 30  percent cost reserves at a similar stage of development. To execute  within the anticipated flat funding profile, the program extended its  development schedule and limited the amount of cost reserves  available—about $50 million each year, which is about 3.7 percent of the  fiscal year 2016 budget request for the program. Program officials stated  that these cost reserves were completely allocated to technical risks  during the budget planning process, which leaves the program with  schedule reserves as its sole resource to address unanticipated issues  throughout the year. Operating with schedule reserves as the only option  for addressing challenges, however, increases risk to the program’s  launch readiness date because any issue that occurs will impact the  overall schedule.
On a program like SLS such challenges are likely. For example, the  current internal launch date delay is due, at least in part, to problems that  are requiring the program to modify one of the four contracts for its major  elements (the core stage, boosters, main engines, and interim upper  stage). According to program officials, the program is modifying the core  stage contract, in part, because the tooling that will be used to  manufacture the 212-foot-tall core stage was vertically misaligned by the  subcontractor during its installation. According to officials, the  misalignment would have prevented production of the core stage. The  necessary repairs are currently scheduled to be completed in August  2015.
To address this challenge, as mentioned above, the program is  anticipating using 7 of its 11 months of schedule reserve and will have  only 4 months of schedule reserve to address risks with 3.5 years  remaining until the program’s committed baseline launch readiness date.  The program, however, has yet to begin integration and testing where we  have previously found projects can expect to encounter challenges that  will impact schedule. Similarly, as part of its analysis of the SLS cost  and schedule estimates, the IPAO reported that, based on its review of 20  historical NASA projects, the majority of schedule growth occurs after  critical design review, which for SLS is currently scheduled for summer  2015. While the current delay has not impacted the program’s November  2018 baseline commitment, it will increase risk to the committed date  because—as noted above—the project has limited cost reserves and will  now have limited schedule reserves to address any future problems or  delays at the point when problems are most likely to occur.
Using schedule reserve alone, rather than in combination with cost  reserves, does not provide the program with the same level of flexibility to  mitigate risks to maintain planned cost and schedule. GAO’s Cost  Estimating and Assessment Guide states that all development programs  should have cost reserves. Problems always occur, and program  managers need ready access to funding in order to resolve them without  adversely affecting the program. In the case of SLS, if further problems  arise associated with the program’s main cost drivers—such as design  and manufacturing of the core stage or developing the integrated flight  software—the program officials will likely have to use what remains of the  program’s schedule reserve to resolve these problems. As the program  has limited cost reserves, this would put the committed launch readiness  date at risk.
This approach—using only schedule reserve to address challenges—also  leaves the program in the position of potentially having to, for example,  descope planned work or to further delay test events or launch readiness  in order to address technical risks and challenges, should those  challenges exceed available resources. Both options have long-term  risks. For example, if certain development efforts are deferred beyond  EM-1, technical risk to EM-1, EM-2, or both may increase and could put  additional cost and schedule pressure on EM-2—the first crewed flight— because more work would then be required within EM-2’s schedule  before the program could achieve launch readiness. However, if  development work—such as data analysis or a test event—is eliminated  altogether, that loss of potential information may increase technical risk to  EM-1. Similarly, delaying planned work is generally not a viable long-term  solution. NASA has found that deferring planned work to stay within  available budget often leads to increased costs in the long term. For  example, in October 2010, the Independent Comprehensive Review  Panel that reviewed the James Webb Space Telescope program found  that delaying work due to inadequate cost reserves did not control  program costs. Doing so delayed and increased costs—typically two or  threefold—due to inefficiencies visited upon other dependent tasks.
section_title: Earned Value Management System Remains Incomplete and Provides Limited Insight into Progress toward External Committed Cost and Schedule Baselines
paragraphs: NASA is using contractor earned value management (EVM) data as an  additional means to monitor costs for SLS, but the EVM data remain  incomplete and provide limited insight into progress toward the program’s   Program officials  external committed cost and schedule baselines.indicated that the current SLS contractor performance measurement  baselines—which establish the program scope, schedule, and budget  targets to measure progress against—are all based on the program’s  more aggressive internal goal for launch readiness for EM-1 in December  2017 and not its external committed date of November 2018. EVM  systems rely on performance measurement baselines as the basis for  measuring performance and use past performance trends to project an  estimate at completion (EAC). An EAC is a calculation of the cost to  complete authorized work based on a contractor’s EVM performance to  date. We reviewed the EAC provided by three prime contractors and  found that they were below what our calculations predicted. Specifically,  our analysis of the contractors’ EVM data for the three prime contracts— main engines, boosters, and core stage—indicates that the contracts  could incur cost overruns ranging from about $367 million to about $1.4  billion, which is substantially higher than the combined overrun of $89  million that the three contractors were projecting at the time of our review.  These projections do not take into account all of the in-house work  conducted for the SLS program.
The SLS program is in the process of instituting a program-level EVM  system that may improve insight into the program’s progress by providing  aggregated tracking of both in-house and contractor performance on the  SLS program that will present a more comprehensive assessment of  program progress at least to internal cost and schedule goals. The SLS  program-level EVM system, however, is not yet fully implemented.
Specifically, because the core stage contract is being modified, EVM data  will have to be reset to new performance measurement baselines. The  Office of Management and Budget Circular A-11 and Capital  Programming Guide requires EVM for major acquisitions with  developmental effort. According to this guidance, when there is both  government and contractor work, the data from the two EVM systems  must be consolidated at the reporting level for total program management  and visibility. Pulling together EVM data from multiple levels into a  program-level report gives officials a comprehensive outlook of its cost  and schedule and provides the project manager with early warning of  potential cost and schedule overruns. In November 2012, we  recommended that NASA establish a time frame by which all new  spaceflight projects will be required to implement NASA’s newly  developed program-level EVM system. This system should improve  management and oversight of its spaceflight projects by providing a  consolidated tracking mechanism for all of the contract and NASA work  conducted for a program. NASA officials stated that this SLS program- level EVM system would allow the program to better monitor progress  relative to the committed cost and schedule baselines by comparing the  aggregated EAC for both NASA and contractor work to the committed  cost and schedule baselines.
The delay in implementing program-level EVM data is, in part, due to the  fact that the program was still in the process of modifying the core stage  contract. According to program officials, they identified discrepancies  between the agency’s and the contractor’s positions during the core stage  contract integrated baseline review—a review conducted by program  management to ensure a mutual understanding of the contract’s  performance measurement baseline. Specifically, program officials  determined that the core stage contract needs to be modified to  accommodate delays associated with addressing technical issues with  misaligned tooling and to resolve differences between the program and  the contractor regarding the level of funding available to begin work on  the exploration upper stage. Further, NASA officials stated that the  program-level EVM will not fully reflect program progress until a new  internal program launch readiness goal is set and contract performance  measurement baselines are updated to reflect these new goals for all  NASA and contractor activities. The officials also stated that the internal  program launch readiness goals will not be set until after the agency has  a better understanding of the expected fiscal year 2016 appropriations  and has completed the planning for the fiscal year 2017 budget request.  Operating without program-level EVM leaves SLS at risk of encountering  unexpected cost and schedule growth.
Both contractor and program-level EVM data, however, is only reported  relative to the December 2017 date, according to program officials. The  potential impact of cost and schedule growth relative to the program’s  external committed cost and schedule baseline of November 2018 is  neither reported nor readily apparent and renders the EVM data less  useful in support of management decisions and external oversight. Major  NASA programs are required by statute to report annually to certain  congressional committees on changes that occurred over the prior year to  the program’s committed cost and schedule baseline, among other   As this report reflects cost and schedule overruns that have  things.already occurred, it does not serve as a mechanism to regularly and  systematically track progress against committed baselines so that  decisionmakers have visibility into program progress and can take  proactive steps as necessary before cost growth or schedule delays are  realized.
section_title: Conclusions
paragraphs: The SLS program has not updated its cost and schedule estimates since  2013, rendering them less reliable as program management and  oversight tools. Since the estimates were created, the program has  received substantial funding increases in 2014 and 2015 but has also  realized both expected and unexpected risks that are forcing the program  to delay its internal goal for launch readiness. The program’s cost and  schedule estimates, however, do not reflect all of these changes or the 2  years of additional contractor performance data that the program has  accumulated since 2013. Without regular updates, developed in  accordance with best practices—including cross-checking to ensure  credibility and thorough explanations of how the updates were prepared— cost estimates lose their usefulness as predictors of likely outcomes and  as benchmarks for meaningfully tracking progress. Updated cost and  schedule estimates would provide program and agency officials with a  more informed basis for decision making and provide the Congress with  more accurate information to support the appropriation process. In  addition, tracking and reporting progress relative to the external cost and  schedule baseline commitments on only an annual basis and only once  issues have already occurred limits NASA’s ability to continually monitor  progress and forecast potential risks to these cost and schedule  baselines, such that proactive action can be taken by decisionmakers.  Until a comprehensive program-level EVM system, reporting to a realistic  performance measurement baseline, is fully implemented, agency  leadership and external stakeholders will lack an accurate measure of the  program’s true progress. Further, by pursuing internal launch readiness  dates that are unrealistic, the program leaves itself and others in a  knowledge void wherein progress relative to the agency’s commitments is  difficult to ascertain. As the EVM system only tracks progress toward the  program’s internal goals, the program lacks a mechanism to track  progress to its external cost and schedule baseline commitments. Without  such a tracking tool, stakeholders will lack early insight into potential  overruns and delays.
section_title: Recommendations
paragraphs: To ensure that the SLS cost and schedule estimates better conform with  best practices and are useful to support management decisions, GAO  recommends that the NASA Administrator direct SLS officials to update  the SLS cost and schedule estimates, at least annually, to reflect actual  costs and schedule and record any reasons for variances before  preparing their budget requests for the ensuing fiscal year. To the extent  practicable, these updates should also incorporate additional best  practices including thoroughly documenting how data were adjusted for  use in the update and cross-checking results to ensure they are credible.
To provide more comprehensive information on program performance,  the NASA administrator should direct the SLS program to expedite  implementation of the program-level EVM system.
To ensure that decisionmakers are able to track progress toward the  agency’s committed launch readiness date, the NASA administrator  should direct the SLS program to include as part of the program’s  quarterly reports to NASA headquarters a reporting mechanism that  tracks and reports program progress relative to the agency’s external  committed cost and schedule baselines.
section_title: Agency Comments
paragraphs: NASA provided written comments on a draft of this report. These  comments are reprinted in appendix II.
In written comments on a draft of this report, NASA concurred with our  three recommendations. NASA agreed that updating cost and schedule  estimates is a program management best practice and indicated that  moving forward the program would use its budget formulation processes  to update its cost and schedule estimates for the first demonstration of  the initial SLS capability on Exploration Mission 1 (EM-1) following the  enactment of NASA’s final appropriations each fiscal year. We are  encouraged that NASA plans to update its cost estimate for SLS annually.  To satisfy the recommendation, we would expect that any updates would  address the deficiencies we identified in NASA’s original estimate for  SLS, including thoroughly documenting how data were adjusted for the  update and cross-checking the results to ensure credibility. NASA also  recognized that having an EVM system in place is a program  management best practice useful for ensuring effective program control.  NASA indicated that it is in the process of implementing program-level  EVM but that the system would not be fully in place until the stages  contract is modified and an independent baseline review is completed in  the third quarter of fiscal year 2016. We appreciate that NASA plans to  implement the program level EVM system as soon as the stages contract  is rebaselined, but we remain concerned that the stages contract will  remain undefinitized until spring 2016. We reported in July 2014 that the  stages contract had remained undefinitized for an extended period and  that leaving the SLS contracts undefinitized for extended periods placed  the government at increased risk of cost and schedule growth and limits  the program’s ability to monitor contractor progress. Likewise, until  certified EVM data from the stages contract is available, the SLS program  will lack a key tool for tracking and forecasting progress as the program  approaches its committed November 2018 launch readiness date. NASA  also agreed that the program should implement a reporting mechanism  that tracks and reports program progress relative to the agency’s external  committed cost and schedule baselines. NASA indicated that the SLS  program-level EVM system would satisfy this recommendation when it is  fully implemented and included as part of the SLS program’s normal  quarterly reporting to the Exploration Systems Development Division at  NASA Headquarters. Given that EVM data is typically reported relative to  contractual targets, in this case the program’s internal goals, we would  expect that these quarterly reports would also include additional elements  addressing progress relative to the program’s external committed cost  and schedule baselines.
We are sending this report to NASA’s Administrator and to appropriate  congressional committees. In addition, the report will be available at no  charge on GAO’s website at http://www.gao.gov.
Should you or your staff have any questions on matters discussed in this  report, please contact me at (202) 512-4841 or chaplainc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. Other key  contributors to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess the reliability of the National Aeronautics and Space  Administration’s (NASA) Space Launch System (SLS) cost and schedule  estimates, we determined the extent to which the estimates were  consistent with best practices for cost estimating and scheduling as  identified in GAO’s Cost Assessment Guide and Schedule Assessment  Guide. We examined documents supporting the cost and schedule  estimates, such as detailed spreadsheets that contain cost, schedule, and  risk information and the timing and availability of funding and reserves.  We also met with independent reviewers of the SLS Joint Cost and  Schedule Confidence Level (JCL), reviewed their report on the program,  and determined the extent to which the SLS program addressed any  concerns the reviewers raised concerning the JCL estimate. In addition,  we met with program and agency officials to discuss the baseline cost  and schedule estimates, potential program schedule changes, and the  program’s cost and schedule reserve postures, among other issues. We  did not assess the credible or controlled criteria of the schedule estimate  as the estimate was completed to support a JCL. The controlled criterion  deals, in part, with updating the schedule periodically. A JCL, however, is  not designed to be used as an updating tool. The credible criterion  relates, in part, to a risk assessment of the schedule. The summary  schedule estimate, however, was developed specifically for use in  developing the SLS program’s JCL, which requires its own risk  assessment to calculate probability of meeting cost and schedule  baselines, and would make the schedule estimate risk assessment  unnecessary in this instance.
To assess the availability of program cost and schedule reserves for  resolving development challenges, we analyzed budget requests,  appropriations, and program budget projections. We also reviewed prior  NASA reviews that called the program’s reserves into question, briefings  to program and agency management that confirmed the program’s  internal goals, and program risk assessments that outlined program risks  and potential impacts. In addition, we met with senior agency officials,  program management, and program budget specialists to discuss the  program’s budget and reserve postures.
To assess the insight that the program’s earned value management  (EVM) system provides into progress relative to the program’s cost and  schedule baseline commitments we obtained and analyzed contractor  cost and schedule reporting, or EVM, data for the three contractors for  which it was available, determined whether the contractors are  forecasting cost and schedule growth, and calculated our own forecasts  of likely cost and schedule growth. We also collected available program- level EVM data to determine whether it could be used as an indicator for  program progress. We met with program EVM and schedule managers to  discuss any concerns with contractor cost and schedule reporting as well  as the creation of the program-level EVM process.
We conducted this performance audit from September 2014 to July 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the National Aeronautics and Space Administration
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Cristina T. Chaplain (202) 512-4841 or chaplainc@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Shelby S. Oakley (Assistant  Director), Jennifer K. Echard, Laura Greifner, Kristine Hassinger, Sylvia  Schatz, Dina Shorafa, Ryan Stott, Ozzy Trevino, and John S. Warren, Jr.  made key contributions to this report.