NASA Human Space Exploration: Integration Approach Presents Challenges to Oversight and Independence
Released: 2017-10-19T00:00:00

Letter

Background

Human spaceflight at NASA began in the 1960s with the Mercury and Gemini programs leading up to the Apollo moon landings. After the last lunar landing, Apollo 17, in 1972, NASA shifted its attention to low earth orbit operations with human spaceflight efforts that included the Space Shuttle and International Space Station programs through the remainder of the 20th century. In the early 2000s, NASA once again turned its attention to cislunar and deep space destinations, and in 2005 initiated the Constellation program, a human exploration program that was intended to be the successor to the Space Shuttle. The Constellation program was canceled, however, in 2010 due to factors that included cost and schedule growth and funding gaps.

Following Constellation, the National Aeronautics and Space Administration Authorization Act of 2010 directed NASA to develop a Space Launch System, to continue development of a crew vehicle, and prepare infrastructure at Kennedy Space Center to enable processing and launch of the launch system. To fulfill this direction, NASA formally established the SLS program in 2011. Then, in 2012, the Orion project transitioned from its development under the Constellation program to a new development program aligned with SLS. To transition Orion from Constellation, NASA adapted the requirements from the former Orion plan with those of the newly created SLS and the associated ground systems programs. In addition, NASA and the European Space Agency agreed that it would provide a portion of the service module for Orion. Figure 1 provides details about the heritage of each SLS hardware element and its source as well as identifies the major portions of the Orion crew vehicle.

The EGS program was established to modernize the Kennedy Space Center to prepare for integrating hardware from the three programs as well as processing and launching SLS and Orion and recovery of the Orion crew capsule. EGS is made up of nine major components, including: the Vehicle Assembly Building, Mobile Launcher, Launch Control Center and software, Launch Pad 39B, Crawler-Transporter, Launch Equipment Test Facility, Spacecraft Offline Processing, Launch Vehicle Offline Processing, and Landing and Recovery. See figure 2 for pictures of the Mobile Launcher, Vehicle Assembly Building, Launch Pad 39B, and Crawler-Transporter.

NASA’s Exploration Systems Development (ESD) organization is responsible for directing development of the three individual human spaceflight programs—SLS, Orion, and EGS—into a human space exploration system. The integration of these programs is key because all three systems must work together for a successful launch. The integration activities for ESD’s portfolio occur at two levels in parallel throughout the life of the programs: as individual efforts to integrate the various elements managed within the separate programs and as a joint effort to integrate the three programs into an exploration system.

The three ESD programs support NASA’s long term goal of sending humans to distant destinations, including Mars. NASA’s approach to developing and demonstrating the technologies and capabilities to support their long term plans for a crewed mission to Mars includes three general stages of activities—Earth Reliant, Proving Ground, and Earth Independent.

Earth Reliant: From 2016 to 2024, NASA’s planned exploration is focused on research aboard the International Space Station. On the International Space Station, NASA is testing technologies and advancing human health and performance research that will enable deep space, long duration missions.

Proving Ground: From the mid-2020s to early-2030s, NASA plans to learn to conduct complex operations in a deep space environment that allows crews to return to Earth in a matter of days. Primarily operating in cislunar space—the volume of space around the moon featuring multiple possible stable staging orbits for future deep space missions—NASA will advance and validate capabilities required for humans to live and work at distances much farther away from our home planet, such as on Mars.

Earth Independent: From the early-2030s to the mid-2040s, planned activities will build on what NASA learns on the space station and in deep space to enable human missions to the vicinity of Mars, possibly to low-Mars orbit or one of the Martian moons, and eventually the Martian surface.

The first launch of the integrated ESD systems, EM-1, is a Proving Ground mission. EM-1 is planned as an uncrewed test flight currently planned for no earlier than October 2019 that will fly about 70,000 kilometers beyond the moon. The second launch, Exploration Mission 2 (EM-2), which will utilize an evolved SLS variant with a more capable upper stage, is also a Proving Ground mission planned for no later than April 2023. EM-2 is expected to be a 10- to 14-day crewed flight with up to four astronauts that will orbit the moon and return to Earth to demonstrate the baseline Orion vehicle capability. NASA eventually plans to develop larger and more capable versions of the SLS to support Proving Ground and Earth Independent missions after EM-2.

As noted above, in April 2017 we found that given the combined effects of ongoing technical challenges in conjunction with limited cost and schedule reserves, it was unlikely that the ESD programs would achieve the November 2018 launch readiness date. We recommended that NASA confirm whether the EM-1 launch readiness date of November 2018 was achievable, as soon as practicable but no later than as part of its fiscal year 2018 budget submission process. We also recommended that NASA propose a new, more realistic EM-1 date if warranted. NASA agreed with both recommendations and stated that it was no longer in its best interest to pursue the November 2018 launch readiness date. Further, NASA stated that, in fall 2017, it planned to establish a new launch readiness date. Subsequently, in June 2017, NASA sent notification to Congress that EM-1’s recommended launch date would be no earlier than October 2019.

The life cycle for NASA space flight projects consists of two phases— formulation, which takes a project from concept to preliminary design, and implementation, which includes building, launching, and operating the system, among other activities. NASA further divides formulation and implementation into pre-phase A through phase F. Major projects must get approval from senior NASA officials at key decision points before they can enter each new phase. The three ESD programs are completing design and fabrication efforts prior to beginning Phase D system assembly, integration and test, launch and checkout. Figure 3 depicts NASA’s life cycle for space flight projects.

NASA’s Integration Approach Offers Some Benefits but Complicates Oversight and Impairs Independence

NASA’s approach for integrating and assessing programmatic and technical readiness, executed by ESD, differs from prior NASA human spaceflight programs. This new approach offers some cost and potential efficiency benefits. However, it also brings challenges specific to its structure. In particular, there are oversight challenges because only one of the three programs, Orion, has a cost and schedule estimate for EM-2. NASA is already contractually obligating money on SLS and EGS for EM- 2, but the lack of cost and schedule baselines for these programs will make it difficult to assess progress over time. Additionally, the approach creates an environment of competing interests because it relies on dual- hatted staff to manage technical and safety aspects on behalf of ESD while also serving as independent oversight of those same areas.

Integration Approach Differs from Past Human Spaceflight Programs

NASA is managing the human spaceflight effort differently than it has in the past. Historically, NASA used a central management structure to manage human spaceflight efforts for the Space Shuttle and the Constellation programs. For example, both the Shuttle and Constellation programs were organized under a single program manager and used a contractor to support integration efforts. Additionally, the Constellation program was part of a three-level organization—the Exploration Systems Mission Directorate within NASA headquarters, the Constellation program, and then projects, including the launch vehicle, crew capsule, ground systems, and other lunar-focused projects, managed under the umbrella of Constellation. Figure 4 illustrates the three-level structure used in the Constellation program.

In the Constellation program, the programmatic workforce was distributed within the program and projects. For example, systems engineering and integration organizations—those offices responsible for making separate technical designs, analyses, organizations and hardware come together to deliver a complete functioning system—were embedded within both the Constellation program and within each of the projects.

NASA’s current approach is organized with ESD, rather than a contractor, as the overarching integrator for the three separate human spaceflight programs—SLS, Orion, and EGS. ESD manages both the programmatic and technical cross-program integration, and primarily relies on personnel within each program to implement its integration efforts. Exploration Systems Integration, an office within ESD, leads the integration effort from NASA headquarters. ESD officials stated that this approach is similar to that used by the Apollo program, wherein the program was also managed out of NASA headquarters. Within Exploration Systems Integration, the Cross-Program Systems Integration sub-office is responsible for technical integration and the Programmatic and Strategic Integration sub-office is responsible for integrating the financial, schedule, risk management, and other programmatic activities of the three programs. The three programs themselves perform the hardware and software integration activities. This organizational structure that consists of two levels is shown in figure 5.

ESD is executing a series of six unique integration-focused programmatic and technical reviews at key points within NASA’s acquisition life cycle, as shown in figure 6, to assess whether NASA cost, schedule, and technical commitments are being met for the three-program enterprise.

These reviews cover the life cycle of the integrated programs to EM-1, from formulation to readiness to launch. Some of these reviews are unique to ESD’s role as integration manager, For example, ESD established two checkpoints—Design to Sync in 2015 and Build to Sync in 2016. The purpose of Design to Sync was to assess the ability of the integrated preliminary design to meet system requirements, similar to a preliminary design review and the purpose of Build to Sync was to assess the maturity of the integrated design in readiness for assembly, integration, and test, similar to a critical design review (CDR). At both events, NASA assessed the designs as ready to proceed. Key participants in these integration reviews include ESD program personnel and the Cross-Program Systems Integration and Programmatic and Strategic Integration staff that are responsible for producing and managing the integration activities.

ESD’s Integration Approach Offers Some Cost Avoidance and Potential Efficiency Gains

ESD’s integration approach offers some benefits in terms of cost avoidance relative to NASA’s most recent human spaceflight effort, the Constellation program. NASA estimated it would need $190 million per year for the Constellation program integration budget. By comparison, between fiscal years 2012 and 2017, NASA requested an average of about $84 million per year for the combined integration budgets of the Orion, SLS, EGS, and ESD. This combined average of about $84 million per year represents a significant decrease from the expected integration budget of $190 million per year under the Constellation program. In addition, as figure 7 shows, NASA’s initial estimates for ESD’s required budget for integration are close to the actuals for fiscal years 2012-2017. NASA originally estimated that ESD’s budget for integration would require approximately $30 million per year. ESD’s integration budget was less than $30 million in fiscal years 2012 and 2013 and increased to about $40 million in fiscal year 2017—an average of about $30 million a year.

According to NASA officials, some of the cost avoidance can be attributed to the difference in workforce size. The Constellation program’s systems engineering and integration workforce was about 800 people in 2009, the last full year of the program; whereas ESD’s total systems engineering and integration workforce in 2017 was about 500 people, including staff resident in the individual programs.

ESD officials also stated that, in addition to cost avoidance, their approach provides greater efficiency. For example, ESD officials said that decision making is much more efficient in the two-level ESD organization than Constellation’s three-level organization because the chain of command required to make decisions is shorter and more direct. ESD officials also indicated that the post-Constellation elimination of redundant systems engineering and integration staff at program and project levels contributed to efficiency. Additionally, they stated that program staff are invested in both their respective programs and the integrated system because they work on behalf of the programs and on integration issues for ESD. Finally, they said another contribution to increased efficiency was NASA’s decision to establish SLS, Orion, and EGS as separate programs, which allowed each program to proceed at its own pace.

One caveat to this benefit, however, is that ESD’s leaner organization is likely to face challenges to its efficiency in the integration and test phases of the SLS, Orion, and EGS programs. We analyzed the rate at which ESD has reviewed and approved the different types of launch operations and ground processing configuration management records for integrated SLS, Orion, and EGS operations, and found that the process is proceeding more slowly than ESD anticipated. For example, as figure 8 illustrates, ESD approved 403 fewer configuration management records than originally planned in the period from March 2016 through June 2017. According to an ESD official, the lower-than-planned approval rate resulted from the time necessary to establish and implement a new review process as well as final records being slower to arrive from the programs for review than ESD anticipated. Additionally, the official stated that the records required differing review timelines because they varied in size and scope.

As figure 8 shows, ESD originally expected the number of items that needed review and approval to increase and create a “bow wave” during 2017 and 2018. In spring 2017, ESD re-planned its review and approval process and flattened the bow wave. The final date for review completion is now aligned with the new planned launch readiness date of no earlier than October 2019, which added an extra year to ESD’s timeframe to complete the record reviews. While the bow wave is not as steep as it was under the original plan, ESD will continue to have a large number of records that require approval in order to support the launch readiness date. An ESD official stated that NASA had gained experience managing such a bow wave as it prepared for Orion’s 2014 exploration flight test launch aboard a Delta IV rocket and as part of the Constellation program’s prototype Ares launch in 2009, but acknowledged that ESD will need to be cautious that its leaner staff is not overwhelmed with documentation, which could slow down the review process.

ESD’s Approach Complicates Oversight Because There Is No Mechanism to Assess Affordability beyond First Mission

ESD is responsible for overall affordability for SLS, Orion, and EGS, while each of the programs develops and maintains an individual cost and schedule baseline. The baseline is created at the point when a program receives NASA management approval to proceed into final design and production. In their respective baselines, as shown in table 1, SLS and EGS cost and schedule are baselined to EM-1, and Orion’s are baselined to EM-2. NASA documentation indicates that Orion’s baselines are tied to EM-2 because that is the first point at which it will fulfill its purpose of carrying crew. Should NASA determine it is likely to exceed its cost estimate baseline by 15 percent or miss a milestone by 6 months or more, NASA is required to report those increases and delays—along with their impacts—to Congress. In June 2017, NASA sent notification to Congress that the schedule for EM-1 has slipped beyond the allowed 6- month threshold, but stated that cost is expected to remain within the 15 percent threshold.

NASA has not established EM-2 cost baselines or expected total life- cycle costs for SLS and EGS, including costs related to the larger and more capable versions of SLS needed to implement the agency’s plans to send crewed missions to Mars. GAO’s Cost Estimating and Assessment Guide, a guidebook of cost estimating best practices developed in concert with the public and private sectors, identifies baselines as a critical means for measuring program performance over time and addresses how a baseline backed by a realistic cost estimate increases the probability of a program’s success. In addition, prior GAO work offers insight into the benefits of how baselines enhance a program’s transparency. For example, we found in 2009 that costs for the Missile Defense Agency’s (MDA) ballistic missile defense system had grown by at least $1 billion, and that lack of baselines for each block of capability hampered efforts to measure progress and limited congressional oversight of MDA’s work. MDA responded to our recommendation to establish these baselines and, in 2011, we reported that MDA had a new process for setting detailed baselines, which had resulted in a progress report to Congress more comprehensive than the one it provided in 2009.

To that end, we have made recommendations in the past on the need for NASA to baseline the programs’ costs for capabilities beyond EM-1; however, a significant amount of time has passed without NASA taking steps to fully implement these recommendations. Specifically, in May 2014, we recommended that, to provide Congress with the necessary insight into program affordability, ensure its ability to effectively monitor total program costs and execution, and to facilitate investment decisions, NASA’s Administrator should direct the Human Exploration and Operations Mission Directorate to:

Establish a separate cost and schedule baseline for work required to support the SLS for EM-2 and report this information to the Congress through NASA’s annual budget submission. If NASA decides to fly the SLS configuration used in EM-2 beyond EM-2, establish separate life cycle cost and schedule baseline estimates for those efforts, to include funding for operations and sustainment, and report this information annually to Congress via the agency’s budget submission; and

Establish separate cost and schedule baselines for each additional capability that encompass all life cycle costs, to include operations and sustainment, because NASA intends to use the increased capabilities of the SLS, Orion, and ground support efforts well into the future and has chosen to estimate costs associated with achieving the capabilities.

As part of the latter recommendation, we stated that, when NASA could not fully specify costs due to lack of well-defined missions or flight manifests, the agency instead should forecast a cost estimate range— including life cycle costs—having minimum and maximum boundaries and report these baselines or ranges annually to Congress via the agency’s budget submission.

In its comments on our 2014 report, NASA partially concurred with these two recommendations, noting that much of what it had already done or expected to do would address them. For example, the agency stated that establishing the three programs as separate efforts with individual cost and schedule commitments met GAO’s intent as would its plans to track and report development, operations, and sustainment costs in its budget to Congress as the capabilities evolved. In our response, we stated that while NASA’s prior establishment of three separate programs lends some insight into expected costs and schedule at the broader program level, it does not meet the intent of the two recommendations because cost and schedule identified at that level is unlikely to provide the detail necessary to monitor the progress of each block against a baseline. Further, reporting the costs via the budget process alone will not provide information about potential costs over the long term because budget requests neither offer all the same information as life-cycle cost estimates nor serve the same purpose. Life-cycle cost estimates establish a full accounting of all program costs for planning, procurement, operations and maintenance, and disposal and provide a long-term means to measure progress over a program’s life span.

In 2016, NASA requested closure of these recommendations, citing, among other factors, changes to the programs’ requirements, design, architecture, and concept of operations. However, NASA’s request did not identify any steps taken to meet the intent of these two recommendations, such as establishing cost and schedule baselines for EM-2, baselines for each increment of SLS, Orion, or ground systems capability, or documentation of life cycle cost estimates with minimum and maximum boundaries. Further, a senior level ESD official told us that NASA does not intend to establish a baseline for EM-2 because it is not required to do so. The limited scope that NASA has chosen to use as the basis for formulating the programs’ cost baselines does not provide the transparency necessary to assess long-term affordability. Plainly, progress cannot be assessed without a baseline that serves as a means to compare current costs against expected costs; consequently, it becomes difficult to assess program affordability and for Congress to make informed budgetary decisions.

NASA’s lack of action in regards to our 2014 recommendations means that it is now contractually obligating NASA to spend billions of dollars in potential costs for EM-2 and beyond without a baseline against which to assess progress. For example: in fiscal year 2016, the SLS program awarded two contracts to Aerojet Rocketdyne: a $175 million contract for RL-10 engines to power the exploration upper stage during EM-2 and EM-3 and a $1.2 billion contract to restart the RS-25 production line required for engines for use beyond EM-4, and to produce at least 4 additional RS-25 engines; in 2017, SLS modified the existing Boeing contract upwards by $962 million for work on the exploration upper stage that SLS will use during EM-2 and future flights; and on a smaller scale, in fiscal year 2016 the EGS program obligated $4.8 million to support the exploration upper stage and EM-2.

As illustrated by these contracting activities, the SLS program is obligating more funds for activities beyond EM-1 than Congress directed. Specifically, of approximately $2 billion appropriated for the SLS program, the Consolidated Appropriations Act, 2016 directed that NASA spend not less than $85 million for enhanced upper stage development for EM-2. NASA has chosen to allocate about $360 million of its fiscal year 2016 SLS appropriations towards EM-2, including enhanced upper stage development, additional performance upgrades, and payload adapters, without a baseline to measure progress and ensure transparency. The NASA Inspector General (IG) also recently reported that NASA is spending funds on EM-2 efforts without a baseline in place and expressed concerns about the need for EM-2 cost estimates. Because NASA has not implemented our recommendations, it may now be appropriate for Congress to take action to require EM-2 cost and schedule baselines for SLS and EGS, and separate cost and schedule baselines for additional capabilities developed for Orion, SLS, and EGS for missions beyond EM-2. These baselines would be important tools for Congress to make informed, long-term budgetary decisions with respect to NASA’s future exploration missions, including Mars.

Organizational Structure Impairs Independence of Engineering and Safety Technical Oversight

NASA’s governance model prescribes a management structure that employs checks and balances among key organizations to ensure that decisions have the benefit of different points of view and are not made in isolation. As part of this structure, NASA established the technical authority process as a system of checks and balances to provide independent oversight of programs and projects in support of safety and mission success through the selection of specific individuals with delegated levels of authority. The technical authority process has been used in other parts of the government for acquisitions, including the Department of Defense and Department of Homeland Security. ESD is organizationally connected to three technical authorities within NASA.

The Office of the Chief Engineer technical authority is responsible for ensuring from an independent standpoint that the ESD engineering work meets NASA standards,

The Office of Safety and Mission Assurance technical authority is responsible for ensuring from an independent standpoint that ESD products and processes satisfy NASA’s safety, reliability, and mission assurance policies, and

The Office of Chief Health and Medical technical authority is responsible for ensuring from an independent standpoint that ESD programs meet NASA’s health and medical standards.

These NASA technical authorities have delegated responsibility for their respective technical authority functions directly to ESD staff. According to NASA’s project management requirements, the program or project manager is ultimately responsible for the safe conduct and successful outcome of the program or project in conformance with governing requirements and those responsibilities are not diminished by the implementation of technical authority.

ESD has established an organizational structure in which the technical authorities for engineering and safety and mission assurance (S&MA) are dual hatted to also serve simultaneously in programmatic positions. The chief engineer technical authority also serves as the Director of ESD’s Cross Program System Integration Office and the S&MA technical authority also serves as the ESD Safety and Mission Assurance Manager. In their programmatic roles for ESD, the individuals manage resources, including budget and schedule, to address engineering and safety issues. In their technical authority roles, these same individuals are to provide independent oversight of programs and projects in support of safety and mission success. Having the same individual simultaneously fill both a technical authority role and a program role creates an environment of competing interests where the technical authority may be subject to impairments in their ability to impartially and objectively assess the programs while at the same time acting on behalf of ESD in programmatic capacities. This duality makes them more subject to program pressures of cost and schedule in their technical authority roles. Figure 9 describes some of the conflicting roles and responsibilities of these officials in their two different positions.

The concurrency of duties leaves the positions open to conflicting goals of safety, cost, and schedule and increases the potential for the technical authorities to become subject to cost and schedule pressures. For example: the dual-hatted engineering and S&MA technical authorities serve on decision-making boards both in technical authority and programmatic capacities, making them responsible for providing input on technical and safety decisions while also keeping an eye on the bottom line for ESD’s cost and schedule; and the technical authorities are positioned such that they have been the reviewers of the ESD programmatic areas they manage—in essence, “grading their own homework.” For example, at ESD’s Build to Sync review in 2016, the engineering and S&MA technical authorities evaluated the areas that they manage in their respective capacities as ESD Director of Cross Program System Integration and ESD Safety and Mission Assurance Manager. This process relied on their abilities as individuals to completely separate the two hats—using one hand to put on the ESD hat and manage technical and safety issues within programmatic cost and schedule constraints and using the other hand to take off that hat and assess the same issues with an independent eye.

NASA officials identified several reasons why the dual-hat structure works for their purposes. Agency officials stated that one critical factor to successful dual-hatting is having the “right” people in those dual-hat positions—that is, personnel with the appropriate technical knowledge to do the work and the ability to act both on behalf of ESD and independent of it. Officials also indicated that technical authorities retain independence because their technical authority reporting paths and performance reviews are all within their technical authority chain of command rather than under the purview of the ESD chain of command.

Additionally, agency officials said that dual-hat roles are a commonplace practice at NASA and cited other factors in support of the approach, including that: it would not be an efficient use of resources to have an independent technical authority with no program responsibilities because that person would be unlikely to have sufficient program knowledge to provide useful insight and could slow the program’s progress; a technical authority that does not consider cost and schedule is not helpful to the program because it is unrealistic to disregard those aspects of program management; a strong dissenting opinion process is in place and allows for issues to be raised through various levels to the Administrator level within NASA; and

ESD receives additional independent oversight through three NASA internal organizations—the independent review teams that provide independent assessments of a program’s technical and programmatic status and health at key points in its life cycle; the NASA Engineering and Safety Center that conducts independent safety and mission success-related testing, analysis, and assessments of NASA’s high- risk projects; and the Aerospace Safety Advisory Panel (ASAP) that independently oversees NASA’s safety performance.

These factors that NASA officials cite in support of the dual-hat approach minimize the importance of having independent oversight and place ESD at risk of fostering an environment in which there is no longer a balance between preserving safety with the demands of maintaining cost and schedule. The Columbia Accident Investigation Board (CAIB) report—the result of an in-depth assessment of the technical and organizational causes of the Columbia accident—concluded that NASA’s organization for the Shuttle program combined, among other things, all authority and responsibility for schedule, cost, safety, and technical requirements and that this was not an effective check and balance. The CAIB report recommended that NASA establish a technical authority to serve independently of the Space Shuttle program so that employees would not feel hampered to bring forward safety concerns or disagreements with programmatic decisions. The Board’s findings that led to this recommendation included a broken safety culture in which it was difficult for minority and dissenting opinions to percolate up through the hierarchy; dual Center and programmatic roles vested in one person that had confused lines of authority, responsibility, and accountability and made the oversight process susceptible to conflicts of interest; and oversight personnel in positions within the program, increasing the risk that these staffs’ perspectives would be hindered by too much familiarity with the programs they were overseeing.

ESD officials stated that they had carefully and thoughtfully implemented the intent of the CAIB; they said they had not disregarded its finding and recommendations but instead established a technical authority in such a way that it best fit the context of ESD’s efforts. These officials did acknowledge, though, that the dual hat approach does not align with the CAIB report’s recommendation to separate programmatic and technical authority or with NASA’s governance framework. Further, over the course of our review, we spoke with various high-ranking officials outside and within NASA who expressed some reservations about ESD’s dual hat approach. For example:

The former Chairman of the CAIB stated that, even though the ESD programs are still in development, he believes the technical authority should be institutionally protected against the pressures of cost and schedule and added that NASA should never be lulled into dispensing of engineering and safety independence because human spaceflight is an extremely risky enterprise.

Both NASA’s Chief Engineer and Chief of S&MA acknowledged there is inherent conflict in the concurrent roles of the dual hats, while also expressing great confidence in the ESD staff now in the dual roles.

NASA’s Chief of S&MA indicated that the dual-hat S&MA structure is working well within ESD, but he believes these dual-hatted roles may not necessarily meet the intent of the CAIB’s recommendation because the Board envisioned an independent safety organization completely outside the programs.

NASA’s Chief Engineer stated that he believes technical authority should become a separate responsibility and position as ESD moves forward with integration of the three programs and into their operation as a system.

As these individuals made clear, ensuring the ESD engineering and S&MA technical authorities remain independent of cost and schedule conflicts is key to human spaceflight success and safety. Along these lines, the ASAP previously conveyed concerns about NASA’s implementation of technical authority that continue to be valid today. In particular, the ASAP stated in a 2013 report that NASA’s technical authority was working at that time in large measure due to the well- qualified, strong personnel that had been assigned to the process. The panel noted, however, that should there be a conflict or weakening of the placement of strong individuals in the technical authority position, this could introduce greater risk into a program. Although a current ASAP official stated she had no concerns with ESD’s present approach to technical authority, the panel’s prior caution remains applicable, and the risk that the ASAP identified earlier could be realized if not mitigated by eliminating the potential for competing interests within the ESD engineering and S&MA positions.

NASA is currently concluding an assessment of the implementation of the technical authority role to determine how well that function is working across the agency. According to the official responsible for leading the study, the assessment includes examining the evolution of the technical authority role over the years and whether NASA is spending the right amount of funds for those positions. NASA expects to have recommendations in 2017 on how to improve the technical authority function, but does not expect to address the dual hat construct. A principle of federal internal controls is that an agency should design control activities to achieve objectives and respond to risks, which includes segregation of key duties and responsibilities to reduce the risk of error, misuse, or fraud. By overlapping technical authority and programmatic responsibilities, NASA will continue to run the risk of creating an environment of competing interests for the ESD engineering and S&MA technical authorities.

ESD Risk Posture Has Improved, but Key Risk Areas Remain for the Integration Effort

Despite the development and integration challenges associated with a new human spaceflight capability, ESD has improved its overall cross- program risk posture over the past 2 years. Nonetheless, it still faces key integration risk areas within software development and verification and validation (V&V). Both are critical to readiness for EM-1 because software acts as the “brain” that ties SLS, Orion, and EGS together in a functioning body, while V&V ensures the integrated body works as expected. The success of these efforts forms the foundation for a launch, no matter the date of EM-1.

ESD’s Cross-Program Risk Posture Has Improved

We have previously reported on individual SLS, Orion, and EGS program risks that were contributing to potential delays within each program. For example, in July 2016, we found that delays with the European Service Module—which will provide services to the Orion crew module in the form of propulsion, consumables storage, and heat rejection and power—could potentially affect the Orion program’s schedule. Subsequently, in April 2017, we found that those delays had worsened and were contributing to the program likely not making a November 2018 launch readiness date. All three programs continue to manage such individual program risks, which is to be expected of programs of this size and complexity. The programs may choose to retain these risks in their own risk databases or elevate them to ESD to track mitigation steps. A program would elevate a risk to ESD when decisions are needed by ESD management, such as a need for additional resources or requirement changes. Risks with the greatest potential for negative impacts are categorized as top ESD risks. In addition to these individual programs risks that are elevated to ESD, ESD is also responsible for overseeing cross-program risks that affect multiple programs. An example of a cross-program risk is the potential for delayed delivery of data from SLS and Orion to affect the EGS software development schedule.

ESD has made progress reducing risks over the last 2 years, from the point of the Design to Sync preliminary design review equivalent for the integrated programs to the Build to Sync critical design review equivalent. As figure 10 illustrates, ESD has reduced its combined total of ESD and cross program risks from 39 to 25 over this period, and reduced the number of high risks from about 49 percent of the total to about 36 percent of the total.

The ESD risk system is dynamic, with risks coming into and dropping out of the system over time as development proceeds and risk mitigation is completed. A total of 29 of the 39 risks within the ESD risk portfolio were removed from the register and 15 risks were added to the register between November 2014, prior to Design to Sync, and March 2017, after Build to Sync. Examples of risks removed over this time period include risks associated with late delivery of Orion and SLS ground support equipment hardware to EGS and establishing a management process to identify risks stemming from the programs being at differing points in development.

Nine risks remained active in the system over the 2-year period we analyzed, and NASA experienced delays in the length of time it anticipated it would take to complete mitigation of the majority of these nine risks. Three of these nine risks that have remained active in the risk system since before Design to Sync are still classified as high risk; the remaining six are classified as medium risk. Mitigation is an action taken to eliminate or reduce the potential severity of a risk, either by reducing the probability of it occurring, by reducing the level of impact if it does occur, or both. ESD officials indicated a number of reasons why risks could take longer to mitigate. For instance, risks with long-term mitigation strategies may go for extended periods of time without score changes. In addition, ESD may conduct additional risk assessments and determine that certain risks need to be reprioritized over time and that resources should be focused towards higher risks. In addition, some risk mitigation steps are tied to hardware delivery and launch dates, and as those delay, the risk mitigation steps will as well. As illustrated in table 2, we found that six of these nine risks were related to software and V&V and represented some of the primary causes in terms of estimated completion delays. On average, the estimated completion dates for these six risks were delayed about 16 months. In addition, the two V&V risks that have remained active since before Design to Sync were still considered top ESD risks as of March 2017 when we completed this analysis.

Software Development Is a Key Risk Area Facing the Integration Effort

Software development is one of the top cross-program technical issues facing ESD as the programs approach EM-1. Software is a key enabling technology required to tie the human spaceflight systems together. Specifically, for ESD to achieve EM-1 launch readiness, software developed within each of the programs has to be able to link and communicate with software developed in other programs in order to enable a successful launch. Furthermore, software development continues after hardware development and is often used to help resolve hardware deficiencies discovered during systems integration and test.

ESD has defined six critical paths—the path of longest duration through the sequence of activities that determines the program’s earliest completion date—for its programs to reach EM-1, and three are related to software development. These three software critical paths support interaction and communication between the systems the individual programs are developing—SLS to EGS software, Orion to EGS software, and the Integrated Test Laboratory (ITL) facility that supports Orion software and avionics testing as well as some SLS and EGS testing. The other critical paths are development of the Orion crew service module, SLS core stage, and the EGS Mobile Launcher. Because of software’s importance to EM-1 launch readiness, ESD is putting a new method in place to measure how well these software efforts are progressing along their respective critical paths. To that end, it is currently developing a set of “Key Progress Indicators” milestones that will include baseline and forecast dates. Officials indicated that these metrics will allow ESD to better track progress of the critical path software efforts toward EM-1 during the remainder of the system integration and test phase. ESD officials have indicated, however, that identifying and establishing appropriate indicators is taking longer than expected and proving more difficult than anticipated.

One of the software testing critical paths, the ITL, has already experienced delays that slipped completion of planned software testing from September 2018 until March 2019, a delay of 6 months. Officials told us that this delay was primarily due to a series of late avionics and software deliveries by the European Space Agency for Orion’s European Service Module. The delay in the Orion testing in turn affects SLS and EGS software testing and integration because those activities are informed by the completion of the Orion software testing. Furthermore, some EGS and SLS software testing scheduled to be conducted within the ITL has been re-planned as a result of the Orion delays.

The Orion program indicates that it has taken action to mitigate ITL issues as they arise. For example, the European Service Module avionics and software delivery delay opened a 125-day gap between completion of crew module testing and service module testing. Orion officials indicated that the program had planned to proceed directly into testing of the integrated crew module and service module software and systems, but the integrated testing cannot be conducted until the service module testing is complete.

As illustrated by figure 11, to mitigate the impact of the delay, Orion officials indicated that the program filled this gap by rescheduling other activities at the ITL such as software integration testing and dry runs for the three programs. These adjustments narrowed the ITL schedule gap from 125 days to 24 days. The officials stated that they will continue to adjust the schedule to eliminate gaps.

The other two software critical paths—SLS to EGS and Orion to EGS software development—are also experiencing software development issues. In July 2016, for example, we found that delays in SLS and Orion requirements development, as well as the programs’ decisions to defer software content to later in development, were delaying EGS’s efforts to develop ground command and control software and increasing cost and schedule.

Furthermore, ESD reports show that delays and content deferral in the Orion and SLS software development continue to affect EGS software development and could delay launch readiness. For example, the EGS data throughput risk that both ESD and EGS are tracking is that the ground control system software is currently not designed to process the amount of telemetry it will receive and provide commands to SLS and ground equipment as required during launch operations. EGS officials stated that, if not addressed, the risk is that if there is a SLS or Orion failure, the ground control system software may not display the necessary data to launch operations technicians. EGS officials told us that the reason for the mismatch between the data throughput being sent to the ground control software and how much is it designed to process is that no program was constrained in identifying its data throughput. These officials stated that retrospectively, they should have established an interface control document to manage the process. The officials also stated that the program is taking steps to mitigate this risk, including defining or constraining the data parameters and buying more hardware to increase the amount of data throughput that can be managed, but will not know if the risk is fully mitigated until additional data are received and analyzed during upcoming tests. For example, EGS officials stated that the green run test will provide additional data to help determine if the steps they are taking address this throughput risk. If the program determines the risk is not fully mitigated and additional software redesign is required, it could lead to schedule delays.

ESD officials overseeing software development acknowledged that software development for the integrated systems is a difficult task and said they expect to continue to encounter and resolve software development issues during cross-program integration and testing. As we have found in past reviews of NASA and Department of Defense systems, software development is a key risk area during system integration and testing. For example, we found in April 2017 that software delivery delays and development problems with the U.S. Air Force’s F-35 program experienced during system integration and testing were likely to extend that program’s development by 12 months and increase its costs by more than $1.7 billion.

Verification and Validation Will Remain Key Risk Area to Monitor as NASA Establishes and Works towards New Launch Readiness Date

Verification and validation (V&V) is acknowledged by ESD as a top cross- program integration risk that NASA must monitor as it establishes and works toward a new EM-1 launch readiness date. V&V is a culminating development activity prior to launch for determining whether integrated hardware and software will perform as expected. V&V consists of two equally important aspects: verification is the process for determining whether or not a product fulfills the requirements or specifications established for it at the start of the development phase; and validation is the assessment of a planned or delivered system ability to meet the sponsor’s operational need in the most realistic environment achievable during the course of development or at the end of development.

Like software development and testing, V&V is typically complex and made even more so by the need to verify and validate how SLS, Orion, and EGS work together as an integrated system.

ESD’s V&V plans for the integrated system have been slow to mature. In March 2016, leading up to ESD’s Build to Sync review, ESD performed an audit of V&V-related documentation for the program CDRs and ESD Build to Sync. The audit found that 54 of 257 auditable areas (21 percent) were not mature enough to meet NASA engineering policy guidance for that point in development. According to ESD documentation, there were several causes of this immaturity, including incomplete documentation and inconsistent requirements across the three programs. NASA officials told us that our review prompted ESD to conduct a follow-up and track the status of these areas. As of June 2017, 53 of the 54 auditable areas were closed, which means these areas are at or have exceeded CDR level of maturity—6 months after Build to Sync was completed. NASA officials indicated that the remaining one auditable area, which is related to the test plan for the integrated communication network, was closed in August 2017.

Nevertheless, other potential V&V issues still remain. According to ESD officials, distributing responsibility for V&V across the three programs has created an increased potential for gaps in testing. If gaps are discovered during testing, or if integrated systems do not perform as planned, money and time for modifications to hardware and/or software may be necessary, as well as time for retesting. This could result in delayed launch readiness. As a result, mature V&V plans are needed to ensure there are no gaps in planned testing. ESD officials indicated that a NASA Engineering and Safety Center review of their V&V plans, requested by ESD’s Chief Engineer to address concerns about V&V planning, would help define the path forward for maturing V&V plans. V&V issues add to cost and schedule risk for the program because they may take more time and money to resolve than ESD anticipates. In some cases, they may have a safety impact as well. For example, if the structural models are not sufficiently verified, it increases flight safety risks. Each of the programs bases its individual analyses on the models of the other programs. As a result, any deficiencies discovered in one can have cascading effects through the other systems and programs. We will continue to monitor ESD’s progress mitigating risks as NASA approaches EM-1.

Conclusions

NASA is at the beginning of the path leading to human exploration of Mars. The first phase along that path, the integration of SLS, Orion, and EGS, is likely to set the stage for the success or failure of the rest of the endeavor. Establishing a cost and schedule baseline for NASA’s second mission is an important initial step in understanding and gaining support for the costs of SLS, Orion, and EGS, not just for that one mission but for the Mars plan overall. NASA’s ongoing refusal to establish this baseline is short-sighted, because EM-2 is part of a larger conversation about the affordability of a crewed mission to Mars. While later stages of the Mars mission are well in the future, getting to that point in time will require a funding commitment from the Congress and other stakeholders. Much of their willingness to make that commitment is likely to be based on the ability to assess the extent to which NASA has met prior goals within predicted cost and schedule targets.

Furthermore, as ESD moves SLS, Orion, and EGS from development to integrated operations, its efforts will reach the point when human lives will be placed at risk. Space is a severe and unforgiving environment; the Columbia accident showed the disastrous consequences of mistakes. As the Columbia Accident Investigation Board report made clear, a program’s management approach is an integral part of ensuring that human spaceflight is as safe and successful as possible. The report also characterized independence as key to achieving that safety and success. ESD’s approach, however, tethers independent oversight to program management by vesting key individuals to wear both hats at the same time. As a result, NASA is relying heavily on the personality and capability of those individuals to maintain independence rather than on an institutional process, which diminishes lessons learned from the Columbia accident.

Matter for Congressional Consideration

We are making the following matter for congressional consideration.

Congress should consider requiring the NASA Administrator to direct the Exploration Systems Development organization within the Human Exploration and Operations Mission Directorate to establish separate cost and schedule baselines for work required to support SLS and EGS for Exploration Mission 2 and establish separate cost and schedule baselines for each additional capability that encompass all life cycle costs, to include operations and sustainment. (Matter for Consideration 1)

Recommendation for Executive Action

We are making the following recommendation to the Exploration Systems Development organization.

Exploration Systems Development should no longer dual-hat individuals with both programmatic and technical authority responsibilities. Specifically, the technical authority structure within Exploration Systems Development should be restructured to ensure that technical authorities for the Offices of the Chief Engineer and Safety and Mission Assurance are not fettered with programmatic responsibilities that create an environment of competing interests that may impair their independence. (Recommendation 1)

Agency Comments and Our Evaluation

NASA provided written comments on a draft of this report. These comments are reprinted in appendix II. NASA also provided technical comments, which were incorporated as appropriate.

In responding to a draft of our report, NASA partially concurred with our recommendation that the Exploration Systems Development (ESD) organization should no longer dual-hat individuals with both programmatic and technical authority responsibilities. Specifically, we recommended that the technical authority structure within ESD should be restructured to ensure that technical authorities for the Offices of Chief Engineer and Safety and Mission Assurance are not fettered with programmatic responsibilities that create an environment of competing interests that may impair their independence. In response to this recommendation, NASA stated that it created the technical authority governance structure after the Columbia Accident Investigation Board report and that the dual- hat technical authority structure has been understood and successfully implemented within ESD. NASA recognized, however, that as the program moves from the design and development phase into the integration and test phase, it anticipates that the ESD environment will encounter more technical issues that will, by necessity, need to be quickly evaluated and resolved. NASA asserted that within this changed environment it would be beneficial for the Engineering Technical Authority role to be performed by the Human Exploration and Operations Chief Engineer (who reports to the Office of the Chief Engineer). NASA stated that over the next year or so, it would solicit detailed input from these organizations and determine how to best support the program while managing the transition to integration and test and anticipated closing this recommendation by September 30, 2018.

We agree that NASA should solicit detailed input from key organizations within the agency as it transitions away from the dual hat technical authority structure to help ensure successful implementation of a new structure. In order to implement this recommendation, however, NASA needs to assign the technical authority role to a person who does not have programmatic responsibilities to ensure they are independent of responsibilities related to cost and schedule performance. To fulfill this, this person may need to reside outside of the Human Exploration and Operations Mission Directorate and NASA should solicit input from the Office of the Chief Engineer when making this decision to ensure that there are no competing interests for the technical authority. Moreover, in its response, NASA does not address the dual-hat technical authority role for Safety and Mission Assurance. We continue to believe that similar changes for this role would be appropriate as well.

Further, in response to this recommendation, NASA makes two statements that require additional context. First, NASA stated that GAO’s recommendation was focused on overall Agency technical authority management. While this review involved meeting with the heads of the Office of Chief Engineer and the Office of Safety and Mission Assurance, the scope of this review and the associated recommendation are limited to ESD. Second, NASA stated “As you found, we agree that having the right personnel in senior leadership positions is essential for a Technical Authority to be successful regardless of how the Technical Authority is implemented.” To clarify, this perspective is attributed to NASA officials in our report and does not represent GAO’s position.

We are sending copies of this report to NASA’s Administrator and to appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report assesses (1) the benefits and challenges of the National Aeronautics and Space Administration’s (NASA) approach for integrating and assessing the programmatic and technical readiness of Orion, SLS, and EGS; and (2) the extent to which the Exploration Systems Development (ESD) organization is managing cross-program risks that could affect launch readiness.

To assess the benefits and challenges of NASA’s approach for integrating and assessing the programmatic and technical readiness of its current human spaceflight programs relative to other selected programs, we reviewed and analyzed NASA policies governing program and technical integration, including cost, schedule, and risk. We obtained and analyzed ESD implementation plans to assess the role of ESD in cross program integration of the three programs. We reviewed the 2003 Columbia Accident Investigation Board’s Report’s findings and recommendations related to culture and organizational management of human spaceflight programs as well as the Constellation program’s lessons learned report. We reviewed detailed briefings and documentation from Cross-Program Systems Integration and Programmatic and Strategic Integration teams explaining ESD’s approach to programmatic and technical integration, including implementation of systems engineering and integration. We interviewed NASA officials to discuss the benefits and challenges of NASA’s integration approach and their roles and responsibilities in managing and overseeing the integration process. We met with the technical authorities and other representatives from the NASA Office of the Chief Engineer, Office of Safety and Mission Assurance, Crew Health and Safety, addressed cost and budgeting issues with the Chief Financial Officer and discussed and documented their roles in executing and overseeing the ESD programs. We also interviewed outside subject matter experts to gain their insight of ESD’s implementation of NASA’s program management policies on the independent technical authority structure. Additionally, we compared historical budget data from the now- cancelled Constellation program to ESD budget data and quantified systems engineering and integration budget savings through preliminary design review, the point at which the Constellation program was cancelled. In addition, we assessed the scope of NASA’s funding estimates for the second exploration mission and beyond against best practices criteria outlined in GAO’s cost estimating guidebook. We assessed the reliability of the budget data obtained using GAO reliability standards as appropriate. We compared the benefits and challenges of NASA’s integration approach to that of other complex, large-scale government programs, including NASA’s Constellation and the Department of Defense’s Missile Defense Agency programs.

To determine the extent to which ESD is managing cross-program risks that could affect launch readiness, we obtained and reviewed NASA and ESD risk management policies; detailed monthly and quarterly briefings; and documentation from Cross-Program Systems Integration and Programmatic and Strategic Integration teams explaining ESD’s approach to identifying, tracking, and mitigating cross-program risks. We reviewed Cross-Program Systems Integration systems engineering and systems integration areas as well as Programmatic and Strategic Integration risks, cost, and schedule to determine what efforts presented the highest risk to cross program cost and schedule. We conducted an analysis of ESD’s risk dataset and the programs’ detailed risk reports, which list program risks and their potential schedule impacts, including mitigation efforts to date. We examined risk report data from Design to Sync to Build to Sync and focused our analyses to identify risks with current mitigation plans to determine if risk mitigation plans are proceeding on schedule. We did not analyze risks that were categorized under “Accept,” “Candidate,” “Research,” “Unknown,” or “Watch” because these risks were not assigned an active mitigation plan by ESD. To assess the reliability of the data, we reviewed related documentation and interviewed knowledgeable agency officials. We determined the data was sufficiently reliable for identifying risks and schedule delays associated with those risks. We examined ESD integrated testing facility schedules to determine the extent to which they can accommodate deviation in ESD’s planned integrated test schedule. We also interviewed program and contractor officials on technical risks, potential impacts, and risk mitigation efforts underway and planned.

We conducted this performance audit from August 2016 to October 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the National Aeronautics and Space Administration

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, Molly Traci (Assistant Director), LaTonya Miller, John S. Warren Jr., Tana Davis, Laura Greifner, Roxanna T. Sun, Samuel Woo, Marie P. Ahearn, and Lorraine Ettaro made key contributions to this report.