Space Launch System: Management Tools Should Better Track to Cost and Schedule Commitments to Adequately Monitor Increasing Risk
Released: 2015-07-16T00:00:00

Letter

Background

The National Aeronautics and Space Administration Authorization Act of 2010 directed NASA to, among other things, develop a space launch system as a follow-on to the Space Shuttle and as a key component in expanding human presence beyond low-Earth orbit. In 2011, NASA formally established the SLS program in response to this direction, and the Congress has provided continued support for the program. For example, the Congress has appropriated additional funding for SLS in each of the past 3 fiscal years above the level requested by the program. The cumulative additional funding totals about $610 million more than requested for SLS for fiscal years 2013, 2014, and 2015.

NASA plans to develop three SLS launch vehicle capabilities, complemented by Orion, to transport humans and cargo into space. The first version of the SLS is a 70-metric ton launch vehicle known as Block I.the first in 2018 and the second in 2021/22. The vehicle is scheduled to fly an uncrewed Orion some 70,000 kilometers beyond the moon during NASA has committed to conduct two test flights of the Block I vehicle— the first test flight, known as EM-1, and to fly a second mission—EM-2— beyond the moon to further test performance with a crewed Orion vehicle. After 2021, NASA intends to build 105- and 130-metric ton launch vehicles, known respectively as Block IA/B and Block II, which it expects to use as the backbone of manned spaceflight for decades. NASA anticipates using the Block IA/B vehicles for destinations such as near- Earth asteroids and Lagrange points and the Block II vehicles for eventual Mars missions.infrastructure and systems needed to support processing and launch of Orion and SLS at Kennedy Space Center. See figure 1.

The JCL is a quantitative probability analysis that requires the project to combine its cost, schedule, and risks into a complete quantitative picture to help assess whether the project will be successfully completed within cost and on schedule. NASA introduced the analysis in 2009, and it is among the agency’s initiatives to reduce acquisition management risk. The move to probabilistic estimating marks a major departure from NASA’s prior practice of establishing a point estimate and adding a percentage on top of that point estimate to provide for contingencies. NASA’s procedural requirements state that Mission Directorates should plan and budget programs and projects with an estimated life-cycle cost greater than $250 million based on a 70 percent JCL, or at a different level as approved by the Decision Authority, and any JCL approved at less than 70 percent must be justified and documented. NASA Procedural Requirements (NPR) 7120.5E, NASA Space Flight Program and Project Management Requirements, paras. 2.4.4 and 2.4.4.1 (Aug. 14, 2012) (hereinafter cited as NPR 7120.5E (Aug. 14, 2012)).

2018 as schedule reserve and the $1.3 billion difference between the $8.4 billion goal and the $9.7 billion baseline as funding for that schedule reserve. Unlike cost reserves, however, that funding largely corresponds to those 11 months and cannot be used separately from schedule to address problems as they arise.

In December 2014, we testified that the ground systems and Orion programs will likely not be ready to support EM-1 before November 2018 even if SLS is able to meet its earlier internal goal. During this testimony, NASA witnesses stated that the SLS program would not be able to meet its internal goal of December 2017 and that the program would likely slip the internal goal to summer 2018. See table 1 for more specifics on estimated launch dates.

The SLS Cost and Schedule Estimates Were Generally Developed in Accordance with Best Practices, but Officials Have No Plans to Update the Cost Estimate with Actual Costs

NASA generally followed best practices in preparing the SLS cost and schedule baseline estimates for the limited portion of the program life cycle covered, that is, through launch readiness for the first test flight of SLS. We found that the SLS program cost and schedule estimates for this limited portion of development substantially met three of four cost characteristics—comprehensive, well documented, and accurate—and both schedule characteristics—comprehensive and well constructed— that GAO considers best practices for preparing a reliable estimate. However, because the cost estimate only partially met best practice criteria for credibility, the fourth cost characteristic, the estimates could not be deemed fully reliable. See figure 2.

While the cost and schedule estimates were prepared largely in accordance with best practices, they only represent costs for the first flight of SLS, EM-1, as opposed to the program’s full life cycle. In May 2014, we recommended that NASA establish a separate cost and schedule baseline for missions beyond EM-1 and report this information via its annual budget submission. Additionally, we recommended that NASA establish life-cycle cost and schedule baselines for each upgraded block of the SLS. NASA partially concurred with our recommendations, citing that actions it plans to take to track costs and actions already in place, such as establishing a block upgrade approach for SLS, met the intent of our recommendations. To this point, however, NASA has not put forth any estimates or baselines projecting the costs of future blocks of the SLS.

Cost Estimate

Comprehensive: The SLS cost estimate substantially met the criteria for being comprehensive through launch readiness for EM-1 but did not include any costs beyond the first flight. To develop the estimate, officials used a detailed work breakdown structure—the structure used to define in detail the work necessary to accomplish program objectives—that is traceable to the cost of each work element and the contract statement of work and documented ground rules and assumptions. To fully meet the criteria for being comprehensive, however, the estimate should define in detail all costs through the expected life of a program. The estimate satisfied NASA’s cost estimating approach for Human Exploration programs by including life-cycle costs through launch readiness for EM-1, but did not include any costs for deployment and operation and maintenance of SLS beyond the first flight. These costs will likely far exceed the costs of development through the first flight. For example, in October 2009, the Review of U.S. Human Spaceflight Plans Committee reported that the fixed costs of the facilities and infrastructure associated with the Shuttle program were about $1.5 billion a year. Given that NASA hopes to operate the SLS for decades, it is reasonable to expect that the deployment and operation and maintenance costs—which should be included in a reliable estimate of life-cycle costs—alone for the SLS will outweigh the agency’s current estimated cost of $9.7 billion. NASA has stated that cost estimates do not need to cover the program from “cradle to grave.” Rather, NASA’s position is that the program’s estimate is only meaningful up to the time that the SLS is delivered for its first launch, because the agency is taking what it calls a capability approach. Therefore, NASA has only estimated costs of the program to the point at which the capability will be achieved. Furthermore, NASA has yet to determine the number of launches, their missions, or the operating lifetime for the program, which according to agency officials makes it difficult to estimate the total costs of the program. Nevertheless, Office of Management and Budget guidance and GAO’s Cost Assessment Guide indicate that life-cycle cost estimates should encompass the full life cycle of a program.

Well Documented: The SLS cost estimate substantially met the criteria for being well documented; however, some explanations to support the estimate were missing. A well-documented cost estimate includes thorough documentation and is traceable to information sources. We found that the SLS cost estimate documentation discusses, and is consistent with, the program’s technical baseline and provides evidence that the cost estimate was reviewed by management. The estimate also included explanations for how the estimates for the underlying components were created through an assessment of likely costs for each part of the system supplemented with an engineering review. In some instances, however, explanations of how historical data were normalized, that is, adjusted to support the estimate, were missing. For example, the cost estimate does not explain how historical costs for the space shuttle main engine were normalized to support the estimate. The purpose of data normalization is to make a given data set consistent with and comparable to other data used in the estimate so that they can be used for comparison analysis or as a basis for projecting future costs. Insufficient documentation of how the historical data were adjusted can hinder understanding and proper use of the estimate.

Accurate: The SLS cost estimate substantially met the criteria for being accurate, but the continued accuracy of the estimate is in question because officials have no plans to periodically update the estimate. Accurate cost estimates are based on assessments of most likely costs, adjusted properly for inflation, and contain few, if any, minor mistakes. In addition, a cost estimate should be updated regularly to reflect significant changes in the program. The SLS cost estimate meets most of these characteristics as it is based on an assessment of likely costs, is adjusted properly for inflation, and contains few if any mistakes. Contrary to best practices, however, NASA does not periodically update the estimate based on actuals, which limits its use as a management tool for monitoring progress and planning future work. The program prepared its cost estimate and JCL in calendar year 2013. GAO’s cost estimating best practices call for estimates to be continually updated through the life of the project, ideally every month as actual costs are reported in earned value management reports. Best practices also call for a risk analysis and risk simulation exercise—like the JCL analysis—to be conducted periodically through the life of the program, as risks can materialize or change throughout the life of a program. Unless properly updated on a regular basis, the cost estimate cannot provide decisionmakers with accurate information to assess the current status of the project.

Agency officials have indicated that the SLS program has no plans to update the cost and schedule estimates underlying the JCL or the JCL itself, which calls into question the continued accuracy of the estimates. NASA’s policy for space flight program and project management requires a program’s committed cost estimate to be updated (rebaselined) if it exceeds the external baseline committed cost by 30 percent or more, and if a project is rebaselined, the JCL should be recalculated and approved as part of the rebaselining process. The NASA Cost Estimating Handbook, however, indicates that program cost estimates should be updated when program content changes and as programs move through their life-cycle phases and conduct milestone reviews, and recognizes that estimates regularly updated based on actual program performance give decisionmakers a clearer picture for major decisions. In addition, through our work assessing large scale programs at the Department of Defense we have found that some programs update cost estimates annually and regularly report progress relative to both threshold and objective targets.

Credible: The SLS cost estimate only partially met the criteria for credibility because the SLS program did not cross-check the results of the estimate and did not commission an independent cost estimate. The purpose of developing a separate independent estimate and cross- checking the estimate is to test the program’s estimate for reasonableness and, ultimately, to validate the estimate. Consistent with best practices, the program conducted a risk and uncertainty analysis that calculated the likely cost and schedule consequences on the program for each risk identified and conducted a duration sensitivity analysis to determine how varying the lengths of different tasks affected the program. Contrary to best practices, however, the SLS program did not cross- check the results of its cost estimate. The main purpose of cross- checking is to determine whether alternative estimating methods produce similar results. If cross-checking confirms the results of the estimate, then confidence in the estimate increases, leading to greater credibility. In addition, project officials did not commission an independent cost estimate—a separate estimate produced by an organization outside of the SLS program chain of command—which is considered one of the best and most reliable estimate validation methods because it provides an independent view of expected program costs that tests the program office’s estimate for reasonableness. An estimate that has not been reconciled with an independent cost estimate has an increased risk of being underfunded because the independent cost estimate provides an objective and unbiased assessment of whether the project estimate is realistic. Because the cost estimate only partially met the criteria for credibility, the estimate does not fully reflect the characteristics of a quality estimate and cannot be considered fully reliable.

While the program did not commission an independent estimate, NASA’s Independent Program Assessment Office (IPAO)—which reviews NASA programs at key decision points in the life cycle to support approval decisions by the agency leadership—did review the program’s cost estimate at the program’s Key Decision Point C (KDP-C) review. KDP-C is the point in NASA’s project life cycle where baseline cost and schedule estimates are established and projects begin implementation. During this review, the IPAO found that the SLS JCL process and cost model were sound; however, the IPAO also found that the program’s initial SLS cost estimate appeared optimistic relative to predictions based on historical data from similar programs. For example, the IPAO reported that the program was underestimating the likely range of cost growth for four key elements—software development, core stage qualification, core stage testing, and procurement of the interim cryogenic propulsion stage. Senior agency officials indicated that, based in part on the results of the IPAO assessment, the program increased its estimate and the agency established higher cost and schedule baseline commitments for the program.

Schedule Estimate

Comprehensive: The SLS schedule estimate substantially met best practice criteria for being comprehensive through launch readiness for EM-1 but the schedule estimate did not account for work beyond that point. A comprehensive schedule includes all activities for both the government and its contractors necessary to accomplish a project’s objectives as defined in the project’s work breakdown structure. The SLS schedule reflected all activities in the program cost work breakdown structure, resources were appropriately allocated to the schedule, and the schedule realistically reflected how long each activity would take. Contrary to best practices, however, the schedule estimate, like the cost estimate, did not fully account for the deployment or operation and maintenance of the program—specifically work for flights beyond EM-1.

Well Constructed: The SLS schedule estimate substantially met best practice criteria for being well constructed but not all activities on the critical path are directly affecting the finish date of the project. A schedule is well constructed if all its activities are logically sequenced in the most straightforward manner possible and it has a reliable critical path that determines which activities drive the project’s earliest completion date. We found relatively few instances of activities that were not logically sequenced, and anomalies within the schedule were, in general, justified by program officials. For example, program officials explained that some of the anomalies were due to activities representing external deliveries to vendors. We found, however, that the schedule did not fully meet the criteria for a well-constructed schedule because not all activities on the critical path are truly affecting the finish date of the project, which could mask delays in the schedule.

Significant Projected Decrease in Schedule Reserves and Limited Cost Reserves Could Put Launch Readiness Commitment at Risk

The SLS program has limited cost and schedule reserves to address potential issues as it enters its most challenging period. Schedule reserve is extra time, with the money to pay for it, in the program’s overall schedule in the event that there are delays or unforeseen problems. For the SLS program, the 11-month difference between the program’s internal launch readiness goal and its committed schedule baseline represents the program’s schedule reserves. Cost reserves are additional funds that can be used to mitigate problems during the development of a program. For example, cost reserves can be used to buy additional materials to replace a component or, if a program needs to preserve schedule, cost reserves can be used to accelerate work by adding extra shifts to expedite manufacturing and save time. Because NASA anticipated a relatively flat budget for SLS, the agency chose to limit cost reserves and rely on schedule reserve—the 11 months between the internal launch readiness goal in December 2017 and the committed baseline in November 2018—as the primary way to mitigate risk. The SLS program, however, is planning to use 7 of the 11 months of schedule reserve, which would delay its planned goal for launch readiness for EM-1 from December 2017 to, tentatively, July 2018. At this point, the agency has not, however, delayed its baseline commitment date of November 2018. As a result, the agency would have only 4 months of schedule reserve remaining between July 2018 and November 2018 to address any further problems that it may encounter. See figure 3.

Complex development efforts like SLS must plan to address myriad risks and unforeseen technical challenges. As mentioned above, cost and schedule reserves are one way to address risks and challenges. NASA’s Marshall Space Flight Center, which manages the SLS program, has guidance requiring programs to present their planned cost and schedule reserves for approval prior to key milestones, but the guidance does not establish specific requirements for reserve levels. However, other NASA centers, such as the Goddard Space Flight Center—the NASA center with responsibility for managing other complex NASA programs such as the James Webb Space Telescope—has requirements for the level of both cost and schedule reserves that projects must have in place at KDP- C. At KDP-C, Goddard flight projects are required to have cost reserves of 25 percent or more and 1 month of schedule reserve for each year of development from KDP-C to the start of integration and testing, 2 months per year for integration and test through shipment to the launch site, and 1 week per month from delivery to launch site to actual launch.

As a result of flat funding requests, the SLS program has very low levels of cost reserves compared to other programs and to cost reserve guidance of NASA centers. The IPAO noted that the program’s planned cost reserves at the time of its review—6 percent—were too low and compared poorly to other development programs which normally had 30 percent cost reserves at a similar stage of development. To execute within the anticipated flat funding profile, the program extended its development schedule and limited the amount of cost reserves available—about $50 million each year, which is about 3.7 percent of the fiscal year 2016 budget request for the program. Program officials stated that these cost reserves were completely allocated to technical risks during the budget planning process, which leaves the program with schedule reserves as its sole resource to address unanticipated issues throughout the year. Operating with schedule reserves as the only option for addressing challenges, however, increases risk to the program’s launch readiness date because any issue that occurs will impact the overall schedule.

On a program like SLS such challenges are likely. For example, the current internal launch date delay is due, at least in part, to problems that are requiring the program to modify one of the four contracts for its major elements (the core stage, boosters, main engines, and interim upper stage). According to program officials, the program is modifying the core stage contract, in part, because the tooling that will be used to manufacture the 212-foot-tall core stage was vertically misaligned by the subcontractor during its installation. According to officials, the misalignment would have prevented production of the core stage. The necessary repairs are currently scheduled to be completed in August 2015.

To address this challenge, as mentioned above, the program is anticipating using 7 of its 11 months of schedule reserve and will have only 4 months of schedule reserve to address risks with 3.5 years remaining until the program’s committed baseline launch readiness date. The program, however, has yet to begin integration and testing where we have previously found projects can expect to encounter challenges that will impact schedule. Similarly, as part of its analysis of the SLS cost and schedule estimates, the IPAO reported that, based on its review of 20 historical NASA projects, the majority of schedule growth occurs after critical design review, which for SLS is currently scheduled for summer 2015. While the current delay has not impacted the program’s November 2018 baseline commitment, it will increase risk to the committed date because—as noted above—the project has limited cost reserves and will now have limited schedule reserves to address any future problems or delays at the point when problems are most likely to occur.

Using schedule reserve alone, rather than in combination with cost reserves, does not provide the program with the same level of flexibility to mitigate risks to maintain planned cost and schedule. GAO’s Cost Estimating and Assessment Guide states that all development programs should have cost reserves. Problems always occur, and program managers need ready access to funding in order to resolve them without adversely affecting the program. In the case of SLS, if further problems arise associated with the program’s main cost drivers—such as design and manufacturing of the core stage or developing the integrated flight software—the program officials will likely have to use what remains of the program’s schedule reserve to resolve these problems. As the program has limited cost reserves, this would put the committed launch readiness date at risk.

This approach—using only schedule reserve to address challenges—also leaves the program in the position of potentially having to, for example, descope planned work or to further delay test events or launch readiness in order to address technical risks and challenges, should those challenges exceed available resources. Both options have long-term risks. For example, if certain development efforts are deferred beyond EM-1, technical risk to EM-1, EM-2, or both may increase and could put additional cost and schedule pressure on EM-2—the first crewed flight— because more work would then be required within EM-2’s schedule before the program could achieve launch readiness. However, if development work—such as data analysis or a test event—is eliminated altogether, that loss of potential information may increase technical risk to EM-1. Similarly, delaying planned work is generally not a viable long-term solution. NASA has found that deferring planned work to stay within available budget often leads to increased costs in the long term. For example, in October 2010, the Independent Comprehensive Review Panel that reviewed the James Webb Space Telescope program found that delaying work due to inadequate cost reserves did not control program costs. Doing so delayed and increased costs—typically two or threefold—due to inefficiencies visited upon other dependent tasks.

Earned Value Management System Remains Incomplete and Provides Limited Insight into Progress toward External Committed Cost and Schedule Baselines

NASA is using contractor earned value management (EVM) data as an additional means to monitor costs for SLS, but the EVM data remain incomplete and provide limited insight into progress toward the program’s Program officials external committed cost and schedule baselines.indicated that the current SLS contractor performance measurement baselines—which establish the program scope, schedule, and budget targets to measure progress against—are all based on the program’s more aggressive internal goal for launch readiness for EM-1 in December 2017 and not its external committed date of November 2018. EVM systems rely on performance measurement baselines as the basis for measuring performance and use past performance trends to project an estimate at completion (EAC). An EAC is a calculation of the cost to complete authorized work based on a contractor’s EVM performance to date. We reviewed the EAC provided by three prime contractors and found that they were below what our calculations predicted. Specifically, our analysis of the contractors’ EVM data for the three prime contracts— main engines, boosters, and core stage—indicates that the contracts could incur cost overruns ranging from about $367 million to about $1.4 billion, which is substantially higher than the combined overrun of $89 million that the three contractors were projecting at the time of our review. These projections do not take into account all of the in-house work conducted for the SLS program.

The SLS program is in the process of instituting a program-level EVM system that may improve insight into the program’s progress by providing aggregated tracking of both in-house and contractor performance on the SLS program that will present a more comprehensive assessment of program progress at least to internal cost and schedule goals. The SLS program-level EVM system, however, is not yet fully implemented.

Specifically, because the core stage contract is being modified, EVM data will have to be reset to new performance measurement baselines. The Office of Management and Budget Circular A-11 and Capital Programming Guide requires EVM for major acquisitions with developmental effort. According to this guidance, when there is both government and contractor work, the data from the two EVM systems must be consolidated at the reporting level for total program management and visibility. Pulling together EVM data from multiple levels into a program-level report gives officials a comprehensive outlook of its cost and schedule and provides the project manager with early warning of potential cost and schedule overruns. In November 2012, we recommended that NASA establish a time frame by which all new spaceflight projects will be required to implement NASA’s newly developed program-level EVM system. This system should improve management and oversight of its spaceflight projects by providing a consolidated tracking mechanism for all of the contract and NASA work conducted for a program. NASA officials stated that this SLS program- level EVM system would allow the program to better monitor progress relative to the committed cost and schedule baselines by comparing the aggregated EAC for both NASA and contractor work to the committed cost and schedule baselines.

The delay in implementing program-level EVM data is, in part, due to the fact that the program was still in the process of modifying the core stage contract. According to program officials, they identified discrepancies between the agency’s and the contractor’s positions during the core stage contract integrated baseline review—a review conducted by program management to ensure a mutual understanding of the contract’s performance measurement baseline. Specifically, program officials determined that the core stage contract needs to be modified to accommodate delays associated with addressing technical issues with misaligned tooling and to resolve differences between the program and the contractor regarding the level of funding available to begin work on the exploration upper stage. Further, NASA officials stated that the program-level EVM will not fully reflect program progress until a new internal program launch readiness goal is set and contract performance measurement baselines are updated to reflect these new goals for all NASA and contractor activities. The officials also stated that the internal program launch readiness goals will not be set until after the agency has a better understanding of the expected fiscal year 2016 appropriations and has completed the planning for the fiscal year 2017 budget request. Operating without program-level EVM leaves SLS at risk of encountering unexpected cost and schedule growth.

Both contractor and program-level EVM data, however, is only reported relative to the December 2017 date, according to program officials. The potential impact of cost and schedule growth relative to the program’s external committed cost and schedule baseline of November 2018 is neither reported nor readily apparent and renders the EVM data less useful in support of management decisions and external oversight. Major NASA programs are required by statute to report annually to certain congressional committees on changes that occurred over the prior year to the program’s committed cost and schedule baseline, among other As this report reflects cost and schedule overruns that have things.already occurred, it does not serve as a mechanism to regularly and systematically track progress against committed baselines so that decisionmakers have visibility into program progress and can take proactive steps as necessary before cost growth or schedule delays are realized.

Conclusions

The SLS program has not updated its cost and schedule estimates since 2013, rendering them less reliable as program management and oversight tools. Since the estimates were created, the program has received substantial funding increases in 2014 and 2015 but has also realized both expected and unexpected risks that are forcing the program to delay its internal goal for launch readiness. The program’s cost and schedule estimates, however, do not reflect all of these changes or the 2 years of additional contractor performance data that the program has accumulated since 2013. Without regular updates, developed in accordance with best practices—including cross-checking to ensure credibility and thorough explanations of how the updates were prepared— cost estimates lose their usefulness as predictors of likely outcomes and as benchmarks for meaningfully tracking progress. Updated cost and schedule estimates would provide program and agency officials with a more informed basis for decision making and provide the Congress with more accurate information to support the appropriation process. In addition, tracking and reporting progress relative to the external cost and schedule baseline commitments on only an annual basis and only once issues have already occurred limits NASA’s ability to continually monitor progress and forecast potential risks to these cost and schedule baselines, such that proactive action can be taken by decisionmakers. Until a comprehensive program-level EVM system, reporting to a realistic performance measurement baseline, is fully implemented, agency leadership and external stakeholders will lack an accurate measure of the program’s true progress. Further, by pursuing internal launch readiness dates that are unrealistic, the program leaves itself and others in a knowledge void wherein progress relative to the agency’s commitments is difficult to ascertain. As the EVM system only tracks progress toward the program’s internal goals, the program lacks a mechanism to track progress to its external cost and schedule baseline commitments. Without such a tracking tool, stakeholders will lack early insight into potential overruns and delays.

Recommendations

To ensure that the SLS cost and schedule estimates better conform with best practices and are useful to support management decisions, GAO recommends that the NASA Administrator direct SLS officials to update the SLS cost and schedule estimates, at least annually, to reflect actual costs and schedule and record any reasons for variances before preparing their budget requests for the ensuing fiscal year. To the extent practicable, these updates should also incorporate additional best practices including thoroughly documenting how data were adjusted for use in the update and cross-checking results to ensure they are credible.

To provide more comprehensive information on program performance, the NASA administrator should direct the SLS program to expedite implementation of the program-level EVM system.

To ensure that decisionmakers are able to track progress toward the agency’s committed launch readiness date, the NASA administrator should direct the SLS program to include as part of the program’s quarterly reports to NASA headquarters a reporting mechanism that tracks and reports program progress relative to the agency’s external committed cost and schedule baselines.

Agency Comments

NASA provided written comments on a draft of this report. These comments are reprinted in appendix II.

In written comments on a draft of this report, NASA concurred with our three recommendations. NASA agreed that updating cost and schedule estimates is a program management best practice and indicated that moving forward the program would use its budget formulation processes to update its cost and schedule estimates for the first demonstration of the initial SLS capability on Exploration Mission 1 (EM-1) following the enactment of NASA’s final appropriations each fiscal year. We are encouraged that NASA plans to update its cost estimate for SLS annually. To satisfy the recommendation, we would expect that any updates would address the deficiencies we identified in NASA’s original estimate for SLS, including thoroughly documenting how data were adjusted for the update and cross-checking the results to ensure credibility. NASA also recognized that having an EVM system in place is a program management best practice useful for ensuring effective program control. NASA indicated that it is in the process of implementing program-level EVM but that the system would not be fully in place until the stages contract is modified and an independent baseline review is completed in the third quarter of fiscal year 2016. We appreciate that NASA plans to implement the program level EVM system as soon as the stages contract is rebaselined, but we remain concerned that the stages contract will remain undefinitized until spring 2016. We reported in July 2014 that the stages contract had remained undefinitized for an extended period and that leaving the SLS contracts undefinitized for extended periods placed the government at increased risk of cost and schedule growth and limits the program’s ability to monitor contractor progress. Likewise, until certified EVM data from the stages contract is available, the SLS program will lack a key tool for tracking and forecasting progress as the program approaches its committed November 2018 launch readiness date. NASA also agreed that the program should implement a reporting mechanism that tracks and reports program progress relative to the agency’s external committed cost and schedule baselines. NASA indicated that the SLS program-level EVM system would satisfy this recommendation when it is fully implemented and included as part of the SLS program’s normal quarterly reporting to the Exploration Systems Development Division at NASA Headquarters. Given that EVM data is typically reported relative to contractual targets, in this case the program’s internal goals, we would expect that these quarterly reports would also include additional elements addressing progress relative to the program’s external committed cost and schedule baselines.

We are sending this report to NASA’s Administrator and to appropriate congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other key contributors to this report are listed in appendix III.

Appendix I: Scope and Methodology

To assess the reliability of the National Aeronautics and Space Administration’s (NASA) Space Launch System (SLS) cost and schedule estimates, we determined the extent to which the estimates were consistent with best practices for cost estimating and scheduling as identified in GAO’s Cost Assessment Guide and Schedule Assessment Guide. We examined documents supporting the cost and schedule estimates, such as detailed spreadsheets that contain cost, schedule, and risk information and the timing and availability of funding and reserves. We also met with independent reviewers of the SLS Joint Cost and Schedule Confidence Level (JCL), reviewed their report on the program, and determined the extent to which the SLS program addressed any concerns the reviewers raised concerning the JCL estimate. In addition, we met with program and agency officials to discuss the baseline cost and schedule estimates, potential program schedule changes, and the program’s cost and schedule reserve postures, among other issues. We did not assess the credible or controlled criteria of the schedule estimate as the estimate was completed to support a JCL. The controlled criterion deals, in part, with updating the schedule periodically. A JCL, however, is not designed to be used as an updating tool. The credible criterion relates, in part, to a risk assessment of the schedule. The summary schedule estimate, however, was developed specifically for use in developing the SLS program’s JCL, which requires its own risk assessment to calculate probability of meeting cost and schedule baselines, and would make the schedule estimate risk assessment unnecessary in this instance.

To assess the availability of program cost and schedule reserves for resolving development challenges, we analyzed budget requests, appropriations, and program budget projections. We also reviewed prior NASA reviews that called the program’s reserves into question, briefings to program and agency management that confirmed the program’s internal goals, and program risk assessments that outlined program risks and potential impacts. In addition, we met with senior agency officials, program management, and program budget specialists to discuss the program’s budget and reserve postures.

To assess the insight that the program’s earned value management (EVM) system provides into progress relative to the program’s cost and schedule baseline commitments we obtained and analyzed contractor cost and schedule reporting, or EVM, data for the three contractors for which it was available, determined whether the contractors are forecasting cost and schedule growth, and calculated our own forecasts of likely cost and schedule growth. We also collected available program- level EVM data to determine whether it could be used as an indicator for program progress. We met with program EVM and schedule managers to discuss any concerns with contractor cost and schedule reporting as well as the creation of the program-level EVM process.

We conducted this performance audit from September 2014 to July 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the National Aeronautics and Space Administration

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Cristina T. Chaplain (202) 512-4841 or chaplainc@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Shelby S. Oakley (Assistant Director), Jennifer K. Echard, Laura Greifner, Kristine Hassinger, Sylvia Schatz, Dina Shorafa, Ryan Stott, Ozzy Trevino, and John S. Warren, Jr. made key contributions to this report.