NASA Commercial Crew Program: Schedule Uncertainty Persists for Start of Operational Missions to the International Space Station
Released: 2019-06-20T00:00:00

Letter

Background

NASA awarded firm-fixed-price contracts in 2014 to Boeing and SpaceX, valued at up to $4.2 billion and $2.6 billion, respectively, for the development of crew transportation systems that meet NASA requirements and for the initial service missions to the ISS.

Figure 1 shows the spacecraft and launch vehicles for Boeing and SpaceX’s crew transportation systems.

These contracts encompass the firm-fixed-price design, development, test, and evaluation work needed to support NASA’s certification of the contractors’ spacecraft, launch vehicle, and ground support systems and begin operational missions to the ISS.

The Commercial Crew Program manages two processes in order to support the contractors’ uncrewed test flight, crewed test flight, and certification milestone. The contractors must submit evidence, which the Commercial Crew Program must review and approve for both processes.

A three-phased safety review process informs the program’s quality assurance activities and is intended to ensure that the contractors have identified all safety-critical hazards and implemented associated controls prior to the first crewed test flight.

In phase one, the contractors identify risks in their designs and develop reports on potential hazards, the controls they put in place to mitigate them, and explanations for how the controls will mitigate the hazards.

In phase two, the program reviews and approves the contractors’ hazard reports and develops strategies to verify and validate that the controls are effective.

In phase three, the contractors will conduct the verification activities and submit the hazard reports to the program for approval.

The verification closure notice process is used to verify that the ISS requirements, applicable to any spacecraft flying to the ISS, and

Commercial Crew Program requirements.

After the contractor has successfully completed its uncrewed and crewed test flights and the above processes, the program determines at the contractor’s certification milestone whether the crew transportation system meets NASA’s requirements for human spaceflight. Following this contract milestone is an agency certification review, which authorizes the use of a contractor’s system to transport NASA crew to and from the ISS. It is at this point that the contractors can begin operational missions. Figure 2 shows the path leading to operational missions.

Contractors Are Making Progress on Vehicles, but Certification Date Remains Unclear

Both contractors have made progress building and testing hardware, including SpaceX’s uncrewed test flight. But continued schedule delays and remaining work for the contractors and the program create continued uncertainty about when either contractor will be certified to begin conducting operational missions to the ISS. The program has made progress reviewing the contractors’ certification paperwork, but contractor delays in submitting evidence for NASA approval may compound a ‘bow wave’ of work, which creates uncertainty about when either contractor will be certified. NASA acknowledged the schedule uncertainty in February 2019, when it announced plans to purchase two additional Soyuz seats from Russia, citing concerns about the difficulties associated with achieving first flights in the final year of development.

Construction and Testing of Contractors’ Hardware Is Progressing

Both contractors are building several spacecraft, some of which are near completion. Each contractor’s spacecraft includes two main modules:

Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module and a service module. The crew module will carry the crew and cargo. It also includes communication systems, docking mechanisms, and return systems for Earth landing. The service module provides propulsion on-orbit and, if needed, in abort scenarios—when a failure prevents continuation of the mission and a return is required for crew survival—as well as radiators for thermal control.

SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we refer to as the crew module, and a trunk, which we refer to as the support module. The crew module will carry the crew and cargo. It also includes avionics, docking mechanisms, and return systems for a water landing. The support module includes solar arrays for on-orbit power and guidance fins for escape abort scenarios.

Different spacecraft will be used for the uncrewed test flight and the crewed test flight, as well as to support other test events. See table 1 for a description of each contractor’s hardware builds, current status, and upcoming events.

Additional details on select hardware testing follow.

In June 2018, Boeing experienced an anomaly while testing its launch abort engines. During a test firing, four of the eight total valves in the four launch abort engines failed to close after a shutdown command was sent. In response to this event, Boeing initiated an investigation to identify the root cause. According to Boeing officials, Boeing plans to replace components on all of its service modules except for the uncrewed test flight service module. This is because the abort system will not be active for the uncrewed test flight. Boeing plans to resume testing its launch abort engines in May 2019. A NASA official told us that addressing this anomaly and identifying its root cause resulted in a 12-month schedule delay to launch abort propulsion system testing.

In March 2019, SpaceX conducted its uncrewed test flight, which demonstrated that the capsule could dock with the ISS and return to Earth. NASA officials described SpaceX’s uncrewed test flight as a success with key systems such as the guidance, navigation, and control and the parachutes performing as expected. A SpaceX official told us that this was a very successful test and represented significant risk reduction from a schedule and technical perspective. Subsequently, the spacecraft used in the uncrewed test flight was destroyed in a testing anomaly. The anomaly occurred during a test that SpaceX was conducting in advance of an in-flight abort test scheduled for this summer. As of May 2019, SpaceX was investigating the anomaly.

Repeated Delays and Remaining Work Create Continued Uncertainty for Certification

Continued schedule delays create uncertainty about when NASA will certify either contractor to begin conducting operational missions to the ISS. We have previously found that the contractors’ schedules regularly changed, and this pattern continues. As of May 2019, both contractors have delayed their certification milestone nine times since establishing dates in their original contracts. In the span of less than a year, since our July 2018 report, Boeing has again delayed its certification milestone four times and by 12 months, while SpaceX has again delayed its certification milestone three times and by 7 months. Both contractors are now planning for certification to occur more than 2 years beyond the original dates in their contracts–Boeing in January 2020 and SpaceX in September 2019, though this date is under review and could further slip (see figure 3).

Over time, both program and contractor officials have told us that they struggle to establish stable schedules. In 2018, the Commercial Crew Program manager told us that she relied on her previous experience to estimate schedule time frames as opposed to relying on the contractors’ schedules, which were overly optimistic. In March 2019, a senior NASA official told us that the agency has struggled to establish schedules with both contractors, often needing to negotiate dates with senior company officials. Further, SpaceX officials explained that they would not know the schedule for the crewed test flight until they conducted the uncrewed test flight. However, even having conducted the uncrewed test flight in March 2019 and before the April 2019 anomaly, SpaceX and NASA were still re- evaluating the schedule for the crewed test flight.

Contractors’ Technical Risks Create Continued Uncertainty for Certification

Both contractors are continuing to mitigate technical risks identified by program officials that need to be addressed in order to reach certification. The program will close a risk when the contractor is able to fully mitigate it. If all mitigation activities are exhausted, but a risk still remains, the program will determine if the risk is acceptable as part of the agency’s rationale for flight. As the contractors address these technical risks and proceed through integration and testing, any issues that arise during testing or the test flights could further delay certification.

Program risks for Boeing include:

Parachute System Certification. Boeing is conducting five parachute system qualification tests to demonstrate that its system meets the Commercial Crew Program’s requirements, which will be validated on two spacecraft flight tests. However, in August 2018, Boeing identified a faulty release mechanism for its drogue parachute—which initially slows down the capsule—during its third parachute qualification test that successfully deployed all parachutes. Identifying and fixing the faulty mechanism delayed its fourth parachute qualification test. According to a NASA official, Boeing is conducting testing to qualify an alternative design, and Boeing must qualify this alternative design before the crewed test flight.

Launch Vehicle Engine Anomaly. Boeing is addressing a safety risk related to a launch vehicle component. Specifically, during a 2018 launch, the launch vehicle engine position during ascent deviated from commands but the launch vehicle provider stated that it achieved all mission objectives. Program officials told us that they have insight into the launch vehicle manufacturer’s ongoing investigation and have participated in a separate independent review team. Boeing will implement a set of corrective actions for the uncrewed test flight, and will continue testing the engines for the crewed test flight.

Spacecraft-Generated Debris. Boeing is addressing a risk that under normal operating procedures the initiators that trigger separation events, such as the separation of the crew and service module prior to re-entry, may generate debris and damage the spacecraft. These components function as expected, but Boeing plans to install hardware to contain debris generated when the initiators fire. Program officials told us that they believe Boeing has identified a solution that will be sufficient for the uncrewed and crewed test flights, but the program is continuing to explore a possible redesign for future operational missions.

Spacecraft Forward Heat Shield. We had previously found that Boeing was addressing a risk that during descent a portion of the spacecraft’s forward heat shield may re-contact the spacecraft after it is jettisoned and damage the parachute system. Since our last report, Boeing tested the performance of the forward heat shield in worst-case scenarios and found there was no damage to the parachute system or the spacecraft. After reviewing test data, the program determined that Boeing had completed the mitigation activities and, as of February 2019, no additional steps were needed.

Program risks for SpaceX include:

Parachute System Certification. Like Boeing, SpaceX is conducting several parachute tests to demonstrate that its system meets the Commercial Crew Program’s requirements. However, SpaceX experienced two anomalies with its parachute system in August 2018. As a result, a SpaceX official told us they enhanced the parachute design to improve robustness. NASA officials told us SpaceX’s enhanced parachutes performed well on its uncrewed test flight. Prior to the crewed test flight, SpaceX must demonstrate the performance of its parachute system. SpaceX plans to continue to test its parachutes, and according to a SpaceX official, will take all steps necessary to ensure that the flight design meets or exceeds minimum performance levels.

Propellant Loading Procedures. SpaceX is continuing to address a safety risk related to its plans to conduct launch vehicle propellant loading procedures after the astronauts are on board the spacecraft. SpaceX officials told us that this loading process has been used in other configurations for multiple SpaceX flights. The Commercial Crew program has approved SpaceX’s proposed loading procedures, including the agreed upon demonstration of the loading procedure five times from the launch site in the final crew configuration before the crewed test flight. The five events include the uncrewed test flight and in-flight abort test. As of March 2019, SpaceX had completed the first two events.

Redesigned Composite Overwrap Pressure Vessel. SpaceX is continuing to address a risk that its launch vehicle’s redesigned composite overwrap pressure vessel, which is intended to contain helium under high pressure, may serve as an ignition source. The program and SpaceX conducted tests on the redesigned vessel and the program determined that all possible ignition sources, with one exception, have a low likelihood of creating ignition. The program continues to assess this ignition source. According to a NASA official, there were no indications of any issues during SpaceX’s uncrewed test flight. SpaceX officials also told us that the redesigned vessel has successfully flown on multiple flights. The program will need to determine whether to accept the risk associated with this technical issue prior to SpaceX’s crewed test flight.

Engine Turbine Cracking. NASA continues to assess a SpaceX risk related to the design of its launch vehicle engines, which has previously resulted in the turbine wheel cracking. To mitigate the turbine cracking risk, SpaceX conducted additional qualification testing and developed an operational strategy that resulted in no cracks. Consequently, the program accepted this risk for SpaceX’s uncrewed test flight but levied a constraint on the crewed test flight. Specifically, SpaceX has agreed to conduct a follow-on test campaign of the engines to demonstrate that it meets NASA’s standards in order to launch its crewed test flight. Program officials said SpaceX plans to build the launch vehicle engines for its crewed test flight concurrently with this follow-on testing series.

Program Office Workload Is a Continued Schedule Risk to Certification

The Commercial Crew Program’s ability to process certification data packages for its two contractors continues to create uncertainty about the timing of certification. Specifically, the program is concurrently reviewing and approving both contractors’ phased safety reviews and verification closure notices. We previously reported that program officials, the contractors, and independent review organizations had concerns about a “bow wave” of work for the program. For example, at that time, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification.

Three-Phased Safety Reviews. The program continues to make progress conducting its phased safety reviews, but it has not yet completed them. In February 2017, we found that the program was behind schedule completing its phased safety reviews and, as of April 2019, it had yet to complete this process. As shown in Table 2, the program is near completion of phase two reviews and phase three reviews are in progress.

Program officials told us that they have started work on many of the phase three safety reviews, but the data only reflect their efforts once they complete a phased safety report in its entirety. Any additional delays to complete this process, however, would delay the crewed test flights and create uncertainty about when NASA will certify the contractors to begin operational flights.

Verification Closure Notices. NASA has made progress verifying that the contractors have met ISS and Commercial Crew Program requirements, but much work remains. When a contractor is ready for NASA to verify that it has met a requirement, such as that the contractor’s system can detect and alert the crew to critical faults that could result in a catastrophic event, the contractor submits data for NASA to review through a verification closure notice. Table 3 shows the agency’s progress approving verification closure notices for each contractor.

Program officials told us that, because the contract solicitation did not require an uncrewed test flight, they had not previously determined the minimum number of Commercial Crew Program requirements that the contractors should meet prior to an uncrewed test flight. Subsequently, both contractors included an uncrewed test flight as part of their schedules. As these test flights approached, NASA determined that it must verify that the contractors met approximately 20 percent of the program’s requirements before the contractors’ uncrewed test flight and the remaining 80 percent before the contractors’ crewed test flights. The program made this determination based on ensuring the contractors met requirements related to the spacecraft safely approaching and docking to the ISS; ensuring the safety of the ISS and its crew; and meeting any mission-specific requirements for cargo.

Both contractors originally planned for the program to verify they had met more than 20 percent of the Commercial Crew Program requirements before the uncrewed test flight but have subsequently changed their plans. For both contractors, the program is allowing the contractors to submit more verification closure notices between the uncrewed and crewed test flight than initially envisioned. Program officials told us that contractors proposed deferring the submission of verification closure notices because they were having difficulties meeting the original targets. Figure 4 includes SpaceX and Boeing’s original and current plans for verification of requirements compared to the Commercial Crew Program’s minimum level of requirements it determined was necessary for the uncrewed test flight. As reflected in the figure, these new plans, which defer submission of work to the crewed test flight, may compound the program’s bow wave of work and create uncertainty about the timing of certification.

Further, the Commercial Crew Program will need to reassess a subset of requirements closed for the uncrewed test flight prior to the crewed test flight. For example, of the 78 requirements Boeing plans to close prior to the uncrewed test flight, the program will re-assess 16; for SpaceX’s 49 requirements, the program will re-assess 32. Program officials told us that some of this work is expected based on known changes to the contractors’ systems between the uncrewed and crewed test flight. For example, officials told us that they approved a verification closure notice for SpaceX’s air conditioning system in order to support the uncrewed test flight, but they know that they will need to re-assess it because SpaceX is making changes before its crewed test flight. While these types of changes and those that are identified through testing are not uncommon, they further add to the program’s workload and create uncertainty about the timing of certification.

Among the requirements that must be closed before the crewed test flight is loss of crew, which is a metric that captures the probability of death or permanent disability to one or more crew members. According to program risk charts, the program’s top safety risk continues to be that neither contractor will meet the contractual requirement of a 1 in 270 probability of incurring loss of crew. We previously found that NASA lacked a consistent approach for how to assess loss of crew and recommended that key parties, including the program manager, collectively determine and document how the agency will determine its risk tolerance level prior to certifying either contractor. NASA partially concurred with that recommendation, stating that, if neither contractor can meet the loss of crew requirement, the program will request a waiver through the human rating certification process to ensure transparency. As of March 2019, NASA officials told us they have not taken steps to address this recommendation. Officials told us that the Commercial Crew Program is currently reviewing Boeing’s loss of crew verification closure notice and SpaceX’s draft verification closure notice in order to verify if the contractors have met the loss of crew requirement.

According to program officials, one of the biggest challenges for the program is balancing its workload to support the two contractors, but officials are making an effort to review each contractor’s data products as they are submitted. For example, program officials told us that they were able to review SpaceX submissions during the summer of 2018, while Boeing’s submissions slowed as it focused on addressing the test anomaly with its launch abort engines. However, based on current schedules, the program must complete its reviews of certification paperwork while supporting uncrewed, crewed, and abort system test flights for both contractors before the end of 2019. Both contractors said they have concerns about NASA’s ability to maintain its pace of processing paperwork in order to support the contractors’ planned test flights and certification dates. The potential bow wave of work continues to create uncertainty about the timing of certification for either contractor, which could result in delays to the first operational mission to the ISS.

NASA Is Taking Steps to Mitigate Delays to Start of Operational Missions

In February 2019, NASA announced plans to buy two more Soyuz seats from Russia, thereby acknowledging that delays to certification of the Commercial Crew Program contractors could continue. These seats would extend U.S. access to the ISS from November 2019 through September 2020. According to a senior NASA official, NASA is not purchasing a new Soyuz spacecraft, which we have previously found requires a 3-year lead time. Instead, two additional seats became available on existing vehicles after changes to the Soyuz manifest. In 2015, NASA paid approximately $82 million per seat through its contract with the Russian Federal Space Agency (Roscomos). Program officials stated they could not publicly disclose the price NASA paid for these two new additional seats, but noted that the cost was 5 percent higher per seat than the previous contract modification to purchase Soyuz seats and is consistent with inflation.

In addition, NASA plans to extend the duration of Boeing’s crewed test flight. In March 2018, NASA modified its contract with Boeing to allow NASA to add a third crew member and extend the length of the crewed test flight. In July 2018, we reported that NASA was considering this option as one way to maintain a U.S. presence on the ISS, but noted it had limited usefulness if Boeing’s crewed test flight slipped past the return date of the last Soyuz flight.

NASA’s actions—purchasing the two additional Soyuz seats and implementing an extended duration crewed test flight for Boeing—do not fully address our July 2018 recommendation to develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified. NASA concurred with this recommendation but, to fully implement it, NASA needs to provide additional support regarding planning efforts to ensure uninterrupted access to the ISS if delays with the Commercial Crew Program contractors continue beyond September 2020. Continued NASA attention on this issue is needed given the uncertainty associated with the final certification dates.

Agency Comments

We provided a draft of this product to NASA for comment. In its response, reproduced in appendix I, NASA generally agreed with our findings and included an update on the progress made by Boeing and SpaceX. NASA also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to NASA Administrator and interested congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Comments from the National Aeronautics and Space Administration

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Cristina T. Chaplain at (202) 512-4841 or chaplainc@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Molly Traci, Assistant Director; Lorraine Ettaro; Laura Greifner; Kurt Gurka; Joy Kim; Christopher Lee; Katherine Pfeiffer; Roxanna T. Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report.