NASA Human Space Exploration: Opportunity Nears to Reassess Launch Vehicle and Ground Systems Cost and Schedule
Released: 2016-07-27T00:00:00

Letter

Background

The National Aeronautics and Space Administration Authorization Act of 2010 directed NASA to, among another things, develop a Space Launch System as a follow-on to the Space Shuttle and prepare infrastructure at Kennedy Space Center to enable processing and launch of the Space Launch System as a key component in expanding human presence beyond low-Earth orbit. To fulfill this direction, NASA formally established the SLS program in 2011. The agency plans to develop three progressively more capable SLS launch vehicles, complemented by Orion, to transport humans and cargo into space. The first version of the SLS that NASA is developing is a 70-metric ton (mt) launch vehicle known as Block I.

In accordance with direction contained in the NASA Authorization Act of 2010, NASA’s acquisition approach for building the initial variant of the SLS is predicated on the use of legacy systems, designs, and contracts from the Space Shuttle and its intended successor Constellation program, which was terminated in 2010 due to factors that included cost and schedule growth. Figure 1 provides details about the heritage of each SLS hardware element and its source as well as identifying the major portions of the Orion crew vehicle.

NASA plans to use heritage hardware and new designs as follows:

RS-25 engines remaining from the Space Shuttle program to provide power for up to four flights of the SLS, five-segment solid rocket boosters that were developed under the now-canceled Constellation program to provide thrust during the initial minutes of SLS flight, a cryogenic rocket stage used on United Launch Alliance’s Delta IV launch vehicle modified to operate as the Interim Cryogenic Propulsion Stage (ICPS) to provide in-space power for SLS during EM-1, a new core stage, which functions as the SLS’s fuel tank and structural backbone, derived from the Shuttle’s external tank and Ares I upper stage from the Constellation program, a new launch vehicle stage adaptor to attach and integrate the ICPS to the core stage; and a new multi-purpose crew vehicle stage adaptor to attach and integrate the SLS with the Orion vehicle.

NASA has committed to be ready to conduct one test flight, EM-1, of the Block I vehicle no later than November 2018. During EM-1, the Block I vehicle is scheduled to launch an uncrewed Orion to a distant orbit some 70,000 kilometers beyond the moon. All three programs—SLS, Orion, and EGS—must be ready on or before this launch readiness date to support this test flight.

NASA also intends to build 105- and 130-mt launch vehicles, known respectively as Block IB and Block II, which it expects to use as the backbone of manned spaceflight for decades. NASA anticipates using the Block IB vehicles for destinations such as near-Earth asteroids and Lagrange points and the Block II vehicles for eventual Mars missions. When complete, the 130-mt vehicle is expected to have more launch capability than the Saturn V vehicle, which was used for Apollo missions, and be significantly more capable than any recent or current launch vehicle.

To enable processing and launch of the SLS and Orion, NASA established the Ground Systems Development and Operations program in 2012 at Kennedy Space Center. The Ground Systems Development and Operations program consists of the 21st Century Space Launch Complex Initiative and the EGS program. NASA created the 21st Century Space Launch Complex Initiative prior to the establishment of the SLS and Orion programs as a way for Kennedy Space Center to continue to make infrastructure improvements to benefit multiple users in the absence of an ongoing major human exploration program. The EGS program was established to renovate parts of Kennedy Space Center to prepare for SLS and Orion. The program consists of nine major components: the Vehicle Assembly Building, Mobile Launcher, Software, Launch Pad 39B, Crawler-Transporter, Launch Equipment Test Facility, Spacecraft Offline Processing, Launch Vehicle Offline Processing, and Landing and Recovery. See figure 2 for pictures of the Mobile Launcher, Vehicle Assembly Building, Launch Pad 39B, and Crawler-Transporter, and appendix III for a description of the nine EGS components.

As the SLS and Orion programs began development, NASA shifted focus away from the 21st Century Space Launch Complex Initiative to the EGS program. For example, in fiscal year 2011, Congress appropriated NASA $142.8 million for the 21st Century Space Launch Complex Initiative and this declined to $39 million in fiscal year 2013, which was a year after EGS began receiving funding. Further, in the fiscal year 2017 president’s budget request, NASA requested $12 million to support the 21st Century Space Launch Complex Initiative.

Space Systems and Acquisition Best Practices

Space launch vehicle development efforts are high risk from technical and programmatic perspectives. The technical risk is inherent for a variety of reasons, including the environment in which launch vehicles operate, complexity of technologies and designs, and limited room for error in the fabrication and integration process. Managing the development process is complex for reasons that go well beyond technology and design. For instance, at the strategic level, because launch vehicle programs can span many years and be very costly, programs can face difficulties securing and sustaining funding commitments and support. At the program level, if the lines of communication between engineers, managers, and senior leaders are not clear, risks that pose significant threats could go unrecognized and unmitigated. If there are pressures to deliver a capability within a short period of time, programs may be incentivized to overlap development and production activities or delete tests, which could result in late discovery of significant technical problems that require more money and ultimately much more time to address. For these reasons, it is imperative that launch vehicle development efforts adopt disciplined practices and lessons learned from past programs.

Best practices for acquisition programs indicate that establishing baselines that match cost and schedule resources to requirements and rationally balancing cost, schedule, and performance are key steps in establishing a successful acquisition program. Our work has also shown that validating this match before committing resources to development helps to mitigate the risks inherent in complex acquisition programs such as SLS and EGS. We have reported that within NASA’s acquisition life cycle, resources should be matched to requirements at key decision point (KDP)-C, the review that commits the program to formal cost and schedule baselines and marks the transition from the formulation phase into the implementation phase. Best practices for acquisition programs also indicate that about midway through development, the product’s design should be stable and demonstrate that it is capable of meeting performance requirements. The critical design review is the vehicle for making this determination. These programmatic milestones are called out relative to NASA’s acquisition life-cycle in figure 3 below.

NASA approved EM-1 cost and schedule baselines for the SLS program in August 2014 and the EGS program in September 2014, following the completion of each program’s respective KDP-C review. The agency baseline commitment for the SLS program is at the 70 percent confidence level and the agency baseline commitment for the EGS program is at the 80 percent confidence level, which are both in line with NASA’s acquisition policies (see table 1). The confidence level is a probabilistic analysis that provides assurance to stakeholders that programs will meet cost and schedule targets.

In addition to the committed cost and launch readiness dates, both programs are working towards internal goals of earlier launch readiness dates and lower costs. NASA considers the time between the programs’ internal goals and their committed launch readiness dates as funded schedule reserve, which is extra time, with the money to pay for it, in the program’s overall schedule in the event that there are delays or unforeseen problems. In July 2015, we found that the SLS program’s internal goal for launch readiness for EM-1 had slipped from December 2017 to July 2018. This reduced the program’s schedule reserve from eleven months to four months. In May 2016, the SLS program further delayed its internal goal for launch readiness from July 2018 to September 2018, reducing program schedule reserve to two months. EGS’s internal goal for launch readiness for EM-1 is September 2018, meaning the program currently has two months of funded schedule reserve.

SLS Has Resolved Some Technical Issues and Matured Its Design, but Pressure Remains on Reduced Cost and Schedule Reserves

The SLS program has made solid progress in resolving some technical issues and maturing the SLS design, but the program’s management of known risks as well as the program’s upcoming integration and test phase puts pressure on the program’s reduced cost and schedule reserves. This pressure threatens the program’s committed November 2018 launch readiness goal. The SLS program has made progress in resolving some technical issues that we previously reported on. For example, prime contractor officials for the core stage stated that they had implemented all corrective actions necessary to repair a problem with the stage’s tooling. Further, the program met its design goals by demonstrating the program’s design was stable enough to warrant continuation. As the program continues with final design and fabrication, the program faces known risks. Such risks are not unusual for large-scale programs, especially human exploration programs, but the program’s management of these risks may increase pressure on reduced cost and schedule reserves. For example, the SLS program has not positioned itself well to provide accurate assessments of progress with the core stage—including forecasting impending schedule delays, cost overruns, and estimates of anticipated costs at completion—because, at the time of our review, NASA did not have a performance measurement baseline necessary to support full earned value management reporting on the core stage contract. Finally, unforeseen technical challenges are likely to arise once the program reaches its next phase, final integration for SLS and integration of SLS with its related Orion and EGS human spaceflight programs that will likely place further pressure on cost and schedule reserves.

SLS Program Has Made Progress Resolving Some Technical Issues and Maturing Design

The SLS program has made solid technical progress developing its primary elements, but at times, the progress has had associated cost increases or schedule delays. Examples of this development progress— and the unexpected difficulties encountered achieving that progress— include the following:

Core stage. In November 2015, prime contractor officials for the core stage stated that they had implemented all corrective actions necessary to repair a subcontractor’s improper installation of the welding tool used to manufacture the 212-foot-tall stage. These actions were necessary because, as we reported in July 2015, NASA officials told us that they would have prevented production of the core stage. As we reported in March 2016, identifying and implementing the corrective actions was the major contributor to a decrease in the program’s schedule reserves from 11 months to 4 months. In addition to resolving the tooling’s misalignment, the SLS program is making progress with fabricating test articles for core stage component testing, constructing new test stands where those components will be subjected to structural testing, and modifying an existing test stand to support hot-fire testing of the assembled core stage. SLS program officials stated that they have also made progress fabricating the EM-1 flight engine section.

RS-25 engines. In 2015, the program successfully tested RS-25 developmental engines and in March 2016 performed hot-fire testing of a flight engine. According to NASA officials, these tests demonstrated the engine could be operated under the conditions it will encounter when integrated into SLS. The program also began production of the new engine controller, which directs the RS-25 engines during flight. The contractor, however, is forecasting a potential cost overrun of $113 million on the engine contract, largely due to overruns stemming from developing the controller. According to NASA officials, however, the potential overrun has not affected the overall program cost or schedule. The factors that contributed to the overrun include higher than expected parts costs, resolving anomalies discovered in developmental test, and increasing staffing levels at the subcontractor to meet schedule demands. NASA officials indicated that the controller design has been tested in development and the controller’s qualification testing is front-loaded to drive out problems early in the test sequence; however, the new controller will not complete all testing before engine deliveries begin. According to NASA, if that testing uncovers the need for modifications to the controller, engines already delivered may have to be brought back from the flight line so that modifications can be implemented.

Solid Rocket Boosters. The program completed the first qualification test of a fully assembled booster in March 2015. Prior X-ray examination of a booster segment had revealed the presence of unexpected unbonds between the solid rocket propellant, the propellant liner, and the new asbestos-free insulation of the solid rocket boosters that could have potentially caused an explosion. Resolving the unbond issue contributed to a delay of 20 months in full-scale qualification testing and, according to NASA officials, the contractor’s forecast of a potential $129 million cost variance on the contract did not affect the overall program cost or schedule. The program is planning to complete a second qualification test of a fully assembled booster sometime between May and July 2016, which NASA officials anticipate will further confirm resolution of the unbond issue.

ICPS. In October 2015, the SLS program completed work on the test version of the ICPS. Additionally, in December 2015 the SLS program began construction of the ICPS liquid oxygen tank, which will provide liquid oxygen to help power the ICPS.

In addition, the program as a whole met best acquisition practices design goals by releasing approximately 92 percent of design drawings for the program-level Critical Design Review (CDR) in July 2015. Because the CDR is the time in a project’s life cycle when the integrity of a project’s design and its ability to meet mission requirements are assessed, it is important that a project’s design is stable enough to warrant continuation with design and fabrication, which is evidenced by release of 90 percent of design drawings at CDR. A stable design allows projects to “freeze” the design and minimize changes prior to beginning the fabrication of hardware. It also helps to avoid re-engineering and rework efforts due to design changes that can be costly to the project in terms of time and funding.

SLS Program’s Management of Known Risks Could Increase Pressure on Already Reduced Cost and Schedule Reserves

As the program continues with final design and fabrication, the program faces known risks. Such risks are not unusual for large-scale programs, especially human exploration programs which are inherently complex and difficult. The program’s management of these risks, however, may increase pressure on already reduced cost and schedule reserves.

Although the program is making progress resolving some technical challenges with the core stage, the core stage development schedule remains aggressive and any additional delays will threaten the SLS program’s readiness for its internal goal of launch readiness by September 2018. As of May 2016, the core stage development effort had approximately 50 days of schedule margin—or time within the schedule where activities can be delayed before affecting a key milestone, which for the core stage is delivery to Kennedy Space Center to begin integrated operations with the Orion and EGS programs. Figure 4 shows the approximately 50 days of core stage schedule margin as well as the 2 months of SLS program schedule reserve.

In addition, because the core stage is the SLS program’s critical path— the path of longest duration through the sequence of activities that determines the program’s earliest completion date—any delay in its development reduces schedule reserve for the whole program. And with only 2 months of schedule reserve remaining between the program’s internal goal and committed launch readiness date of November 2018, any reduction in program reserves threatens the committed launch readiness date.

As of April 2016, the SLS program was tracking core stage risks, including late component delivery and concerns about application of the thermal protection system that provides heat shielding, which could require the program to use some of the core stage’s margin. Further, the SLS Standing Review Board—an independent NASA team responsible for reviewing SLS at each major program milestone—found in a 2015 report that it was unlikely the core stage would be able to support the SLS program’s committed date for launch readiness. The Board cited several factors for its finding, including a steep learning curve for the handling and alignment of such a large the potential for human access issues to avionics and propulsion plumbing once the stage is assembled, and that the green run test—the culminating test of core stage development where the actual EM-1 core stage flight article will be integrated with the cluster of four RS-25 engines and fired for 500 seconds under simulated flight conditions—carries risks because it is the first time the four RS-25 engines cluster will be fired, the first time the integrated engine and core stage auxiliary power units will be tested in flight-like conditions, and the first time flight and ground software will be used in an integrated flight vehicle. Green run test activities are currently scheduled to begin in October 2017.

Boeing and SLS program officials stated that they are working to establish additional margin within the core stage schedule, but whether the core stage stays on schedule is largely dependent on the success of the green run test. Boeing officials told us that they originally had margin in their schedule for a second green run test if needed, but that it was removed due to the tight schedule. NASA officials acknowledged that this schedule existed; however, they also stated that the contingency test was considered “unauthorized work” for the contractor and the program baseline only calls for one test. Further, NASA officials stated that if the test is not successful, then a re-test may have to occur. Additionally they stated that under current plans, any time required to conduct a re-test would have to come from program schedule margin or reserve. As a result, if the program uncovers unexpected performance issues during green run testing, maintaining the core stage schedule—and thus the program schedule—may prove difficult.

The SLS program has also not positioned itself well to provide accurate assessments of progress with the core stage because it has never had a performance measurement baseline for the core stage that is necessary to support full earned value management reporting. Earned value, or the planned cost of completed work and work in progress, can provide accurate assessments of project progress, produce early warning signs of impending schedule delays and cost overruns, and provide unbiased estimates of anticipated costs at completion. The use of earned value management, which integrates the project scope of work with cost, schedule, and performance elements for optimum project planning and control, is advocated by both GAO’s best practices for cost estimating and NASA’s own guidance. According to a SLS program official, when the program and contractor conducted its integrated baseline review—a joint assessment of the performance measurement baseline by the government and contractor—the program realized the contractor’s plans assumed synergies between the core stage and exploration upper stage efforts that would produce cost savings for the contractor but NASA did not have the funding to begin this work under the same time frames identified by the contractor. A SLS program official told us that NASA asked Boeing to start replanning activities with a proposal that removed the exploration upper stage development from this contract action. In May 2016, NASA and Boeing signed the contract replan—with a cost increase of approximately $1 billion, from about $4.2 billion to about $5.2 billion. However, according to program officials it will probably be summer 2016 before the program receives contractor earned value management data derived from the new performance measurement baseline—some 4.5 years after contract award. Without this information, the program has been in a poor position to understand the extent to which technical challenges with the core stage are having schedule implications or the extent to which they may require reaching into the program’s cost reserves. The latter is of concern because as we found in July 2015, NASA maintains low cost reserves for this program—about $50 million per year—because program officials stated it has been necessary to sustain a flat funding profile for SLS as compared to other programs. Further, at SLS’s KDP-C, NASA approved the program to proceed with cost reserves of less than 2 percent leading to launch readiness, even though requirements for Marshall Space Flight Center—the NASA center with responsibility for the SLS program—indicate that standard cost reserves for launch vehicle programs should be 20 percent at KDP-C.

In addition to cost and schedule pressures stemming from the core stage, development of the flight software—the software that controls the first phase of SLS flight from liftoff through booster separation and up to main engine cut off—may require more time than the SLS program anticipates because the program made a decision to defer its most rigorous testing until software development nears completion. SLS software developers have been testing flight software at the end of each of the first five primary SLS software releases, with the scope of testing in each release isolated to the set of requirements for that respective release. They plan to perform the most rigorous testing of the software when the development reaches the release that will be used for flight qualification testing, beginning in March 2016, which will include testing against the most comprehensive set of requirements at that point. The deferral of the most rigorous testing until the flight qualification release, however, means that the program’s understanding of the defects to this point may not be as complete as it believes. This may, in turn, delay completion of software development while the program takes the time necessary to resolve defects. As we found in a September 2015 report assessing a Veterans Benefits Administration software-based processing system, successful system testing includes appropriately identifying and handling defects that are discovered during testing. In addition, we found that outstanding defects can delay the release of functionality to end users, denying them the benefit of features. The program has allotted one future contingency release at the end of the software effort for defect repairs, but delaying the discovery of defects increases the risk that potential problems will remain undiscovered until the point when few cost or schedule reserves are available to correct deficiencies.

Systems Integration and Testing for EM-1 May Add Even More Pressure to the Already Reduced Cost and Schedule Reserves

Even if the development phase does not consume any additional cost and schedule reserves, the SLS program’s EM-1 integration and test phase may require those resources. Our prior work has shown that this period often reveals unforeseen challenges leading to cost growth and schedule delays. Likewise, although superseded through revision, NASA program management guidance from 2010 states that integration and testing are among the periods of peak spending, when schedule delays are most costly, and that programs should maintain sufficient reserves to address issues encountered during that time, and unknown risks can be managed only by maintaining sufficient reserves. Compounding this already risky time period is that the threat to SLS program reserves is two-fold because SLS EM-1 launch readiness involves in essence two integration efforts. The first integration effort is to assemble SLS as a launch vehicle and the second is a cross-program integration effort, which means integrating SLS, Orion, and EGS to achieve launch readiness in 2018.

Integrated launch readiness for EM-1 is dependent on the success of the individual SLS, Orion, and EGS integration efforts. If delays materialize during individual systems integration and testing, for example, there could be a cascading effect of cross-program problems. Booster component shelf life provides a good illustration of this point. According to program officials, there is a limit on the amount of time the SLS boosters may remain in a stacked configuration that, if exceeded, would necessitate destacking and replacement of limited-life items. Program officials told us that NASA will review these limited-life items prior to stacking the integrated vehicle, but if launch is delayed longer than limited-life time frames allow, NASA would have to disassemble SLS from Orion back in the Vehicle Assembly Building. Such an effort could have broad cost and schedule impacts across the three programs.

NASA’s Human Exploration and Operations Mission Directorate, which oversees development of the SLS, EGS, and Orion programs, plans to conduct a “build-to-synchronization” review in summer 2016 to demonstrate that the integrated launch vehicle, crew vehicle, and ground systems will perform as expected to meet EM-1 objectives. Human Exploration and Operations Mission Directorate officials told us that there is no existing NASA guidance to direct what the build-to-synchronization review should entail, but that they are tailoring requirements, with agency leadership concurrence, from NASA program management guidance for critical design review. According to these officials, the review will serve essentially as an EM-1 integration critical design review for the programs. According to NASA program management requirements, a critical design review for a NASA program would not only evaluate the integrated design, but also evaluate whether it meets mission requirements with appropriate margins and acceptable risk within cost and schedule constraints. As of March 2016, officials leading the planning efforts for the build-to-synchronization review told us that they were currently working on developing the terms of reference—which include review objectives and success criteria—but that they anticipate only limited discussion of cost and schedule because the review will focus first and foremost on the hardware and software design maturity of the three programs. Understanding the technical scope required for EM-1 integrated readiness, however, goes hand-in-hand with knowledge about how much money and time the individual programs will require to achieve that readiness. By foregoing a re-evaluation of cost and schedule reserves at the time it assesses technical scope for EM-1, especially in light of known pressures on the SLS program’s reserves, NASA risks missing an opportunity to reevaluate whether sufficient resources are available to respond to unforeseen challenges during the integration and testing phase.

Beyond EM-1, the SLS program continues to face technical as well as cost and schedule risks. For example, for Exploration Mission 2 (EM-2), the program will be transitioning from the ICPS in-space propulsion element to an exploration upper stage providing both ascent performance and in-space capability. NASA had intended to use the ICPS for EM-2, which is planned to launch a crewed Orion vehicle beyond the moon to further test performance. However, the ICPS is not certified to support crewed flight and NASA estimated it would have to spend at least $150 million on that effort to fly a crewed mission. The Explanatory Statement to the Consolidated Appropriations Act, 2016, while not law, prohibited the use of NASA funds to human-rate the ICPS. In addition, as part of the fiscal year 2016 NASA Exploration appropriation, Congress provided that no less than $85 million of the appropriations should be used for the development of a new exploration upper stage necessary to build the Block IB vehicle for deployment on EM-2. NASA officials told us that the agency intends to have the exploration upper stage complete for EM-2. They also stated that they are currently developing a test plan, which includes examining the risk of performing only ground testing of the exploration upper stage because current plans do not allow for a separate flight test of the stage prior to EM-2.

The EGS Program Is Making Progress Completing Modifications, but Technical Challenges Are Consuming Cost and Schedule Reserves

The EGS program is maturing selected systems, but the program is encountering technical challenges that require both time and money to fix. Further, the program has reduced cost and schedule reserves remaining to address risks if they come to fruition. This pressure threatens the program’s committed November 2018 launch readiness goal. Program management has identified the Vehicle Assembly Building and Mobile Launcher as projects along the critical path and software as a high risk component of the EGS program. All three of these projects have experienced delays and the Vehicle Assembly Building and Mobile Launcher have no schedule margin remaining to overcome any future technical challenges. As a result, any future delays would have to be accommodated by using the overall program’s schedule reserve. The program’s schedule reserve, however, has been reduced over time and now has 2 months of reserve remaining. Further, the program is operating with reduced cost reserves to address any future construction and software risks. These reserves will likely be tested further once the program begins integration with SLS and Orion, as delays in any one program can have a cascading effect.

Selected EGS Systems Are Maturing, but the Program Has Encountered Technical Challenges Leading to the Systems Having Little or No Schedule Margin Remaining

The Vehicle Assembly Building Has No Schedule Margin Remaining to Meet EGS’s Internal Schedule Goal

The Vehicle Assembly Building was built in 1966 as a facility to assemble the Apollo program’s Saturn V moon rocket, and part of the building is being refurbished by the EGS program to accommodate SLS and Orion. Updating the building is a large undertaking as it includes removing about 150 miles of Apollo-era cabling, improving the elevators, upgrading cranes, and incorporating fire safety improvements. EGS officials stated that the age of the building adds even more challenges, such as dealing with outdated building drawings and uncertain field conditions. The most significant of the Vehicle Assembly Building projects is the fabrication and installation of 10 new platforms which will allow access to the integrated SLS and Orion vehicles during final assembly. See figure 5 for a photograph of the Vehicle Assembly Building and an illustration of the building’s platforms.

Complications with the Vehicle Assembly Building’s platform design and installation have required an additional $16 million to resolve—funding which the EGS program used from program reserves and the launch pad project, a project that has development remaining. Additionally, the project has exhausted its schedule margin, and any additional delays would have to be addressed through the use of program level schedule reserves. During testing, NASA observed that the test platform could not roll out properly and the program was forced to modify the design of the platforms midway through construction. Resolution of these design issues involved modifying key mechanical components and installing shims to properly align the platform during rollout. Program officials said that this issue has been addressed and is being implemented on all nine subsequent platforms. NASA is prepared to install additional shimming during platform installation if necessary. NASA’s interim assessment of the design contractor for the platforms highlighted numerous quality issues during design; however, NASA officials ultimately found the design product at an acceptable level of quality with cost and schedule requirements having been met.

Additionally, in December 2015, the first platform was installed in the Vehicle Assembly Building, but was removed shortly thereafter because of an installation issue. According to agency officials, the platform “flexed” slightly when it was lifted via crane for installation due to the weight of the platform in relation to the lifting points. This flexing kept the platform from fitting as designed on the bracket that allows the platforms to be moved to different elevations. The program has designed and fabricated an installation tool to prevent the platform from flexing when it is lifted for installation. EGS officials estimate that, if platform design challenges continue, they could delay the completion of the Vehicle Assembly Building by up to 3 months, which would affect the EGS program’s schedule overall. For example, construction on the building’s platforms is slated to end immediately before the Mobile Launcher is moved into the Vehicle Assembly Building; officials said there is no margin for additional delays on the building if it is to be ready for the Mobile Launcher in time. If additional delays materialize with the Vehicle Assembly Building, the program would need to reduce its overall schedule reserve.

The Mobile Launcher Has No Schedule Margin Remaining and Has Consolidated Future Schedule Activities to Meet Program’s Goals

The Mobile Launcher was originally developed as part of the Constellation program, but was never used because of the program’s cancellation in 2010. After the cancellation, EGS began modifying the Mobile Launcher to support what is now SLS. The EGS program is modifying the Mobile Launcher to support the assembly, testing, prelaunch check-out and servicing of the SLS rocket, as well as to transfer SLS and Orion to the launch pad and provide the platform from which they will launch. According to EGS officials, the Mobile Launcher is the most complex EGS component because it contains more than 900 pieces of ground support equipment needed to support SLS and Orion. Ground support equipment includes subsystems for propellant and gases, electronic control systems, communication systems, and access platforms. Figure 6 is a photograph of the Mobile Launcher.

The EGS program has experienced delays and design challenges with the Mobile Launcher and has no project-level schedule margin remaining in order to meet the program’s internal goals for operations and launch readiness. Any additional delays would have to be addressed through the use of program level schedule reserves. The EGS program has completed all major structural changes to the Mobile Launcher, such as adding reinforcements to the Mobile Launcher’s structure to accommodate SLS height and weight, but the program must still complete the design and installation of the ground support equipment and the nine umbilicals that connect the Mobile Launcher directly to the SLS and Orion. The program has experienced design challenges and late hardware deliveries with two of these umbilicals: the ICPS umbilical, which supplies power, fuel, and cooling between the SLS upper stage and the Mobile Launcher, and the tail service mast umbilical, which provides liquid hydrogen and oxygen to SLS during launch.

Further, there have been ground support equipment and umbilical design changes both during and after the Mobile Launcher’s design phase because of vehicle requirement changes from SLS and Orion. EGS used nearly 22 percent of its schedule margin to accommodate these changes.

Additionally, requirement changes during and after ground support equipment subsystems’ design have led to the Mobile Launcher’s ground support equipment being designed concurrently with its installation. The program has identified a program risk that conducting these activities concurrently could lead to a potential cost increase of up to $10 million and schedule delays of up to 8 months. The Mobile Launcher project plans to begin its project-level verification and validation before installation of the ground support equipment and umbilicals are complete because the project has no schedule margin remaining. Officials acknowledged that conducting the mobile launcher’s verification and validation concurrent with ground support equipment systems and umbilicals installation increases risk because of uncertainties regarding how systems not yet installed may affect the systems already installed. EGS officials indicated that these concurrent installations and verification and validation meets all program test objectives and enables the Mobile Launcher effort to stay on schedule to support the program’s internal launch readiness date.

Interdependencies with SLS and Orion Have Increased Delays and Risk to EGS’s Software Development Efforts

EGS’s software development efforts—Spaceport Command and Control System (SCCS) and Ground Flight Application Software (GFAS)—are behind schedule as compared to program plans. The development efforts face challenges that include the need for requirements-related information from the SLS and Orion programs. EGS is developing these two software systems concurrently—SCCS is to operate and monitor ground equipment needed to launch and communicate with the integrated SLS and Orion vehicles, and GFAS is to interface with flight systems and ground crews. EGS software was immature at the program’s Critical Design Review, and EGS’s Standing Review Board considers the program’s software development effort the highest risk area. The Standing Review Board found in February 2016 that the SCCS and GFAS developments are currently underperforming, are understaffed, and are waiting on requirements definition from the two flight element programs. SCCS and GFAS’s completion are dependent on the SLS and Orion also finishing work on schedule; however, because SCCS and GFAS are among the last EGS activities scheduled to finish prior to integrated operations, delays in their completion could force a delay in the program’s committed November 2018 launch readiness date.

SCCS Architecture: SCCS development is behind its planned software release schedule. Program officials attributed the delays, in part, to requirements maturing late from SLS and Orion. For example, according to EGS officials, there were initially supposed to be two content drops, wherein additional functionality is added, for the last two versions of SCCS. However, as of the program’s critical design review in late 2015, the two drops had evolved into six content drops. Program officials stated that the evolution from two drops to six enables content to be released in an as-needed phased approach to meet stakeholders’ needs and utilize resources in a more efficient manner. In March 2016, the NASA Office of the Inspector General reported that SCCS is more than a year behind schedule and significantly over cost, and that, because of cost and timing pressures, several planned software capabilities have been deferred, including the ability to automatically detect the root cause of specific equipment and system failures. The Office of Inspector General concluded that these issues largely result from unanticipated complexity in the way NASA has approached SCCS’s development. Likewise, program officials told us that developers initially expected ground systems, Orion, and SLS to require a total of 300,000 compact unique identifiers, or information fields. However, these officials said that because EGS is developing software as Orion and SLS are developed, complete information on how many information fields were necessary for each program was unavailable at the beginning of the development effort. SCCS officials have identified a risk that there may be a need for more than 300,000 total information fields, which could degrade the software system’s performance and result in cost and schedule overruns.

As the program’s Standing Review Board concluded, much of EGS’s software development work is heavily dependent on the final requirements of the SLS and Orion programs, both of which are still in development. Program officials indicated that, as all three programs have developed and EGS has received more information about the requirements of SLS, Orion, and ground systems, SCCS’s complexity has increased. To address the added complexity, the EGS program increased its workforce, but the overall schedule is challenged by hiring difficulties in a highly competitive environment. The EGS program is using the same developers to develop content for multiple phased deliveries, and the next content drop has been threatened by the developers’ delayed transition from prior drops.

GFAS Application Software: GFAS development is facing challenges because necessary operational requirements from SLS and Orion are not yet available. GFAS officials told us that they were optimistic in their planning regarding the availability of requirements from SLS and Orion to support software development. For example, EGS officials said that they expected more mature information about operational requirements to come out of the Orion and SLS CDRs than what they received. In September 2015, after EGS officials did not receive early information as they had anticipated, the program conducted a schedule replan and said they planned to hire more staff to reduce the risk to the program. GFAS is currently planning on delivering their last content drop in February 2018, after the program has begun integrated operations with SLS and Orion. Figure 7 illustrates the GFAS content drop schedule with EGS’s schedule milestones.

The EGS program has identified two program risks that development of GFAS could be delayed by a combined maximum of up to 9 months and costs could increase by up to $3.2 million combined because it is dependent in part on SCCS development progress. According to the program’s Standing Review Board, the risk is that the necessary software will not be available when needed to meet EGS critical milestones and could affect the agency’s November 2018 launch readiness commitment date.

EGS Program Overall Has Limited Cost and Schedule Reserves Remaining

Overall, the EGS program is operating with limited cost reserves to address any future construction and software risks. The EGS program is operating in fiscal year 2016 with cost reserves of about $13 million, or about 3 percent of its fiscal year 2016 budget. Program budget documents indicate that the program expects its cost reserve posture to improve to 13 percent and 9 percent in fiscal years 2017 and 2018 and to level out at around 5 percent in subsequent years. Kennedy Space Center, which manages the EGS program, does not have guidance for cost reserves. However, other NASA centers, such as the Goddard Space Flight Center—the NASA center with responsibility for managing other complex NASA programs such as the James Webb Space Telescope—have requirements for the level of both cost and schedule reserves that projects must have in place at KDP-C. At KDP-C, Goddard flight projects are required to have cost reserves of 25 percent or more through operational readiness. At EGS’s KDP-C, however, the program had cost reserves of only 4 percent leading to launch readiness.

According to EGS’s Standing Review Board in 2016, the remaining cost risks to EGS are greater than the program’s current reserve balance. Our analysis of the maximum potential impact of Mobile Launcher and Vehicle Assembly Building cost risks on EGS cost reserves support this assessment. For example, based on the program’s February 2016 risk assessment, the EGS program could see maximum cost increases of $10 million for the Mobile Launcher and $11 million for the Vehicle Assembly Building, which is almost double the program’s fiscal year 2016 reserve. Although these cost increases may not occur in only one fiscal year and could be less than the maximum value, they could still impact the program if the planned reserves are either not available as expected or are not sufficient to cover needs.

The EGS program is also operating with reduced schedule reserves to address future construction and software issues. At the time NASA established EGS’s agency baseline commitment, the program had 5 months of funded schedule reserve between its internal planning date (June 2018) and its committed launch readiness date (November 2018). The program is now internally planning to a launch readiness date of September 2018, which reduces the program’s schedule reserve to 2 months. However, the EGS program must be ready well in advance of this launch readiness date in order to integrate SLS and Orion at the Kennedy Space Center and the program plans to be ready to begin integrated operations with SLS in January 2018. EGS has 3 months of margin before the start of integrated operations, and 1 additional month of margin before its internal goal for launch. See figure 8 for a timeline of EGS’s lifecycle relative to SLS for EM-1.

Moving forward, relying on the critical path to determine available reserves may prove problematic because the program’s scheduling practices are fairly limited. The EGS program identifies its critical path as including the Vehicle Assembly Building and the Mobile Launcher in program quarterly management reports, but we were not able to replicate this critical path. Our analysis of the EGS critical path identified inconsistencies between the critical path identified by the software used to create and maintain the program’s integrated master schedule and the critical path called out in the program’s quarterly management reports. Our best practices for scheduling indicate that a program’s integrated master schedule should identify the program’s critical path rather than critical activities being selectively chosen based on what management has determined to be important. Establishing a valid critical path is necessary for examining the effects of any activities slipping along this path. Based on our limited review, the critical path in the program’s integrated master schedule does not match the critical path in the program’s quarterly management reports. EGS program management acknowledged that the two paths did not match, and indicated that they intentionally do not rely solely on the scheduling software’s generated critical path because it includes non-EGS development activities, such as SLS and Orion flight hardware deliveries. We plan to further research the inconsistencies we identified as part of planned future work on NASA’s human exploration systems.

Integration of EGS with the SLS and Orion programs will be reviewed by the Human Exploration and Operations Mission Directorate at the build- to-synchronization review in summer 2016. As with the SLS program, the same holds true for EGS that integrated flight readiness for EM-1 is dependent on the technical and programmatic stability of all three human spaceflight programs—EGS, SLS and Orion. Further, threats to the margin and schedule reserve for EGS can occur from delays within the program or delays within the Orion and SLS programs. In particular, if the SLS core stage delivery date to Kennedy Space Center slips beyond the March 2018 date depicted in the above figure, NASA will have less time for integrated operations and that will ultimately threaten the launch readiness date of November 2018.

Conclusions

NASA established the SLS and EGS programs to support deep space exploration by humans, but the ability to launch its first exploration mission with these programs by the committed date of November 2018 is threatened. In some cases, the threat comes from technical challenges that are not unusual for large-scale projects, but may take more time and money than the program has reserves to address. In other cases, NASA’s approach to dealing with the known risks is exacerbating the challenges. For example, in some cases the SLS program has not positioned itself well to accurately forecast and proactively manage potential schedule delays and cost overruns, which in turn, may ultimately lead to cost and schedule growth that could stretch the program beyond its committed baseline. An opportunity is nearing in NASA’s upcoming build-to- synchronization process to not only determine whether the integrated launch vehicle, crew vehicle, and ground systems will perform as expected to meet EM-1 objectives, but to also revisit whether cost and schedule reserves are sufficient. Given the mission of the EM-1 test flight, NASA does not have to meet a specific schedule window for its launch date as it often does with planetary missions. As a result, NASA is in the position of being able to make an informed decision about what is a realistic launch readiness date. By not re-evaluating the cost and schedule reserves, both programs may continue to make decisions that result in reduced knowledge to meet a schedule that is not realistic. Until such a re-evaluation occurs, the American public and Congress, who are the beneficiaries of NASA’s technological advances, will not have a clear picture of the time and money needed to support these efforts.

Recommendation for Executive Action

In order to ensure available cost and schedule margins are sufficient to meet the synchronized goals for launch readiness and related activities, we recommend the NASA administrator direct the Human Exploration and Operations Mission Directorate as it finalizes its schedule and plans for EM-1 during the planned build-to-synchronization review to re-evaluate SLS and EGS cost and schedule reserves based on results of the integrated design review in order take advantage of all available time resources and maximize the benefit of available cost reserves, and to verify that the November 2018 launch readiness date remains feasible.

Agency Comments

We provided a draft of this report to NASA for review and comment. Its written comments are reprinted in appendix IV of this report.

NASA concurred with our recommendation to re-evaluate SLS and EGS cost and schedule reserves based on results of the build-to- synchronization review, but stated that further direction from the Administrator to the program is not necessary, as this activity is already underway. We are encouraged that since our discussions with the program regarding the scope of the build-to-synchronization review, and providing NASA with a draft of this report for comment, the agency has incorporated plans to address the processes and capabilities in place to continue managing the enterprise within cost and schedule constraints, including available margins, as part of the build-to-synchronization review and that the SLS management agreement to EM-1 is being updated to align with program and enterprise execution plans. To further satisfy our recommendation’s intent, we anticipate that NASA’s actions could encompass a full examination of the integrated schedule for the programs, to help ensure that an individual program does not anticipate using limited reserves to meet the planned launch readiness date, if the November 2018 date is not feasible for all the programs. NASA stated that the results of its build-to-synchronization review will be reported to the NASA Program Management Council by November 30, 2016.

NASA also provided technical comments on the draft report, which we incorporated as appropriate.

We are sending copies of this report to NASA’s Administrator and to appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report , please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Exploration Ground Systems Components beyond the Space Launch System and Orion

According to Exploration Ground Systems (EGS) officials, the program does not track how EGS investments could benefit users beyond the Space Launch System (SLS) and Orion, but we found that the majority of EGS funds are being used to develop major components that will be exclusively used by SLS and Orion or require some modification to be used by another user. We found that EGS components fall into three categories: components that could be used for users beyond SLS and Orion with no modifications, providing they are not in use by SLS and Orion; components that could be used with some modification; and components that are solely for SLS and Orion.

For example, the Mobile Launcher has nine umbilicals and, according to EGS officials, over 900 pieces of ground support equipment to support SLS and Orion. According to National Aeronautics and Space Administration (NASA) officials, while the steel structure and platform of the Mobile Launcher could be used for another user, that user would have to meet weight limits of the structure and would need to design and install entirely new specialized equipment. Five components—among them, the Vehicle Assembly Building, Crawler-Transporter, and the Multi-Payload Processing Facility (part of Spacecraft Offline Processing)—have received funding from the 21st Century Space Launch Complex Initiative, which focuses on modernizing the infrastructure to support multiple users at Kennedy, in addition to EGS funding. These components can, with some or no modifications, be used by other users. The Crawler- Transporter, for example, has been upgraded by EGS in order to support the combined weight of the Mobile Launcher, SLS, and Orion, but according to EGS officials could be used by any user to transport equipment as long as the equipment was within the Crawler-Transporter’s carrying capacity.

The majority of EGS funds obligated to date are to develop components that require some modifications or will be exclusively used by SLS and Orion. See table 2 for allocation of Ground Systems Development and Operations (GSDO) funding between EGS and 21st Century Space Launch Complex Initiative.

As seen in the above table and its accompanying notes, from fiscal year 2012, when the EGS program started, to fiscal year 2015, the EGS program obligated $1,495.4 million to develop components for SLS and Orion. In the same years, $49.8 million from the 21st Century Space Launch Complex Initiative was used for some EGS components that may benefit users beyond SLS and Orion.

Appendix II: Scope and Methodology

To assess the extent to which the Space Launch System (SLS) program made progress in meeting cost and schedule commitments, we compared current program status with National Aeronautics and Space Administration’s (NASA) cost and schedule baselines for executing Exploration Mission 1 (EM-1) in 2018. We reviewed top SLS program and element-level risks as identified by NASA; analyzed the results of the SLS July 2015 critical design review to determine what software and hardware efforts present the highest risk to program cost and schedule; and reviewed monthly earned value management reports to identify the largest impacts on cost and schedule. In addition, we assessed SLS design production maturity against established knowledge-based, best practice standards. We compared the status of flight software development efforts and progress against NASA’s planned release schedule and reviewed the metrics NASA is using to assess software development status. During the course of our review, we examined other program documents that included program plans, quarterly program status review reports, assessments of SLS preliminary and critical design reviews by the NASA Standing Review Board that reviewed the program’s status at preliminary and critical design review independent from the program; and an assessment of the flight software development by a NASA Independent Verification and Validation team that reviewed software development status independent from the SLS program. We met with the SLS program, element-level and flight software officials at Marshall Space Flight Center in Huntsville, Ala.; representatives from the core stage contractor, Boeing, in Huntsville, Ala.; and officials from the Standing Review Board and the Independent Verification and Validation teams, which are composed of members from various NASA locations.

To assess the extent to which the Exploration Ground Systems (EGS) program has made progress in completing modifications to key components and ground support equipment at Kennedy Space Center, we identified EGS’s major components by reviewing program plans, critical design review documents, quarterly program status review documents, and budget materials. We identified the Vehicle Assembly Building, Mobile Launcher, and software as key construction and development efforts for our review because they are among the top program risks or the most expensive EGS projects. We observed EGS components during a site visit to Kennedy Space Center and discussed modification of the components with NASA officials. To evaluate the progress made in preparing these components and software to support the EM-1 test flight, we reviewed program plans and compared them to program status to assess whether EGS components and software were progressing as expected, critical design review documents to determine design maturity, quarterly program status reviews to identify risks, budget information to assess development costs, and contractor progress reports to identify any issues contractors faced that could impact cost and schedule. We also reviewed NASA’s Standing Review Board assessment from EGS’s preliminary and critical design reviews. We also evaluated the program’s integrated master schedule against the GAO’s best acquisition practices for scheduling in order to assess the validity of the EGS program’s critical path. Additionally, to determine the extent to which major ground system components at Kennedy Space Center directly support the SLS and Orion programs, we reviewed NASA budget and accounting data and interviewed agency officials.

We conducted this performance audit from September 2015 to July 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Major Components of Exploration Ground Systems

Appendix IV: Comments from the National Aeronautics and Space Administration

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Cristina T. Chaplain (202) 512-4841 or chaplainc@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Molly W. Traci (Assistant Director), Michael Armes, Matt Bader, Nabajyoti Barkakati, John Bauckman, Erin Cohen, Juana Collymore, Tana Davis, Juli Digate, Jennifer Echard, Laura Greifner, Jason Lee, Sylvia Schatz, Roxanna Sun, and John S. Warren, Jr. made key contributions to this report.