NASA Human Space Exploration: Integration Approach Presents Challenges to Oversight and Independence
Released: 2017-10-19T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: NASA is undertaking a trio of closely related programs to continue human space exploration beyond low-Earth orbit. All three programs (SLS, Orion, and EGS) are working toward a launch readiness date of no earlier than October 2019 for the first test flight. Each program is a complex technical and programmatic endeavor. Because all three programs must work together for launch, NASA must integrate the hardware and software from the separate programs into a working system capable of meeting its goals for deep space exploration.

The House Committee on Appropriations report accompanying H.R. 2578 included a provision for GAO to assess the progress of NASA's human space exploration programs. This report assesses (1) the benefits and challenges of NASA's approach for integrating these three programs and (2) the extent to which cross-program risks could affect launch readiness. GAO examined NASA policies, the results of design reviews, risk data, and other program documentation and interviewed NASA and other officials.

section_title: What GAO Found
paragraphs: The approach that the National Aeronautics and Space Administration (NASA) is using to integrate its three human spaceflight programs into one system ready for launch offers some benefits, but it also introduces oversight challenges. To manage and integrate the three programs—the Space Launch System (SLS) vehicle; the Orion crew capsule; and supporting ground systems (EGS)—NASA's Exploration Systems Development (ESD) organization is using a more streamlined approach than has been used with other programs, and officials GAO spoke with believe that this approach provides cost savings and greater efficiency. However, GAO found two key challenges to the approach:

The approach makes it difficult to assess progress against cost and schedule baselines. SLS and EGS are baselined only to the first test flight. In May 2014, GAO recommended that NASA baseline the programs' cost and schedule beyond the first test flight. NASA has not implemented these recommendations nor does it plan to; hence, it is contractually obligating billions of dollars for capabilities for the second flight and beyond without establishing baselines necessary to measure program performance.

The approach has dual-hatted positions, with individuals in two programmatic engineering and safety roles also performing oversight of those areas. As the image below shows, this presents an environment of competing interests.

These dual roles subject the technical authorities to cost and schedule pressures that potentially impair their independence. The Columbia Accident Investigation Board found in 2003 that this type of tenuous balance between programmatic and technical pressures was a contributing factor to that Space Shuttle accident.

NASA has lowered its overall cross-program risk posture over the past 2 years, but risk areas—related to software development and verification and validation, which are critical to ensuring the integrated body works as expected—remain. For example, delays and content deferral in Orion and SLS software development continue to affect ground systems software development and could delay launch readiness. GAO will continue to monitor these risks.

section_title: What GAO Recommends
paragraphs: Congress should consider directing NASA to establish baselines for SLS and EGS's missions beyond the first test flight. NASA's ESD organization should no longer dual-hat officials with programmatic and technical authority responsibilities. NASA partially concurred with our recommendation and plans to address it in the next year. But NASA did not address the need for the technical authority to be independent from programmatic responsibilities for cost and schedule. GAO continues to believe that this component of the recommendation is critical.

section_title: Letter
paragraphs: The National Aeronautics and Space Administration (NASA) is nearing  the point when billions of dollars invested should begin to pay off with the  first launch of systems needed to support deep space exploration by  humans. This deep space exploration requires the capability to transport  crew and large masses of cargo beyond low Earth orbit to distant  destinations including the moon and eventually Mars. The Exploration  Systems Development (ESD) organization within NASA’s Human  Exploration and Operations Mission Directorate is responsible for  managing and integrating the three programs developing the specific  capabilities needed.

The Space Launch System (SLS) program is developing a vehicle to  launch a crew capsule and cargo beyond low-Earth orbit.

The Orion Multi-Purpose Crew Vehicle (Orion) program is developing  a crew capsule to transport humans beyond low-Earth orbit.

The Exploration Ground Systems (EGS) program is developing  systems and infrastructure to support assembly, test, and launch of  the SLS and Orion crew capsule, and recovery of the Orion crew  capsule.

This portfolio of three programs is estimated to cost almost $24 billion—to  include two Orion flights and one each for SLS and EGS—and constitute  more than half of NASA’s planned development budget. All three  programs are necessary for the first integrated test flight, Exploration  Mission 1 (EM-1), and are working to a launch readiness date of no  earlier than October 2019.

NASA intends for ESD’s portfolio of programs—SLS, Orion, and EGS—to  provide an important capability for human exploration missions. Each of  these programs represents a large, complex technical and programmatic  endeavor. In addition, since all three programs must work together for  launch, NASA faces the additional challenge of integrating the hardware  and software from the separate programs into a working system capable  of effectively meeting its goals for deep space exploration. Our prior work  has shown that the integration and test phase often reveals unforeseen  challenges leading to cost growth and schedule delays.

GAO has designated NASA’s management of acquisitions as a high-risk  area for more than two decades. In February 2017, we found that the  agency has continued to make progress in reducing risk on major projects  after previously struggling with poor cost estimation, weak oversight, and  risk underestimation. We also found that the Orion, SLS, and EGS  programs are generally better positioned for success than past crewed  vehicle efforts that were canceled after facing acquisitions problems and  funding-related issues. Nevertheless, as we have reported, management  weaknesses—including overly ambitious schedules, unreliable cost  estimating, limited reserves, and operating for extended periods of time  without definitized contracts—have increased the likelihood that the  programs will incur schedule delays and cost overruns, particularly when  coupled with the technical risks that are inherent in any human spaceflight  development. In April 2017, we found that it was unlikely that the ESD  programs would achieve the planned November 2018 launch readiness  date and recommended that NASA reassess the date. NASA agreed with  this recommendation and stated that it would establish a new launch  readiness date in fall 2017. Subsequently, in June 2017, NASA sent  notification to Congress that EM-1’s recommended launch date would be  no earlier than October 2019.

The House Committee on Appropriations included a provision in its 2015  report for GAO to review the acquisition progress of NASA’s human  exploration programs, including Orion, SLS, and EGS. This report is the  latest in a series of reports addressing the mandate. This report assesses  (1) the benefits and challenges of NASA’s approach for integrating and  assessing the programmatic and technical readiness of Orion, SLS, and  EGS; and (2) the extent to which ESD is managing cross-program risks  that could affect launch readiness.

To assess the benefits and challenges of NASA’s approach for  integration, we obtained and analyzed NASA program policies governing  program and technical integration, including cost, schedule, and risk. We  obtained and analyzed ESD implementation plans to assess the role of  ESD in cross program integration of SLS, Orion, and EGS and reviewed  briefings explaining ESD’s approach to programmatic and technical  integration, including implementation of systems engineering and  integration. In addition, we assessed the scope of NASA’s funding  estimates for the second exploration mission and beyond against best  practices criteria outlined in GAO’s cost estimating guidebook. We  reviewed the 2003 Columbia Accident Investigation Board Report’s  findings and recommendations related to culture and organizational  management of human spaceflight programs as well as the Constellation  program’s lessons learned report. We met with the technical authorities  and other representatives from the NASA Office of the Chief Engineer,  Office of Safety and Mission Assurance, Crew Health and Safety, and  addressed cost and budgeting issues with the Chief Financial Officer, and  discussed and documented their roles in executing and overseeing the  ESD programs. We also interviewed outside subject matter experts to  gain their insight of ESD’s implementation of NASA’s program  management policies on the independent technical authority structure.

To assess the extent to which ESD is managing cross-program risks that  could affect launch readiness, we obtained and reviewed NASA and ESD  risk management policies, detailed monthly and quarterly briefings and  documentation from Cross-Program Systems Integration and  Programmatic and Strategic Integration teams explaining ESD’s approach  to identifying, tracking, and mitigating cross-program risks. We conducted  an analysis of ESD’s risk dataset and the programs’ detailed risk reports  which list program risks and their potential schedule impacts, including  mitigation efforts to date. We examined risk report data from Design to  Synchronization (Design to Sync) to Build to Synchronization (Build to  Sync) and focused our analyses to identify risks with current mitigation  plans to determine if risk mitigation plans are proceeding on schedule.  We supplemented this analysis with interviews of responsible ESD  officials. For more information on our scope and methodology, see  appendix I.

We conducted this performance audit from August 2016 to October 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Human spaceflight at NASA began in the 1960s with the Mercury and  Gemini programs leading up to the Apollo moon landings. After the last  lunar landing, Apollo 17, in 1972, NASA shifted its attention to low earth  orbit operations with human spaceflight efforts that included the Space  Shuttle and International Space Station programs through the remainder  of the 20th century. In the early 2000s, NASA once again turned its  attention to cislunar and deep space destinations, and in 2005 initiated  the Constellation program, a human exploration program that was  intended to be the successor to the Space Shuttle. The Constellation  program was canceled, however, in 2010 due to factors that included cost  and schedule growth and funding gaps.

Following Constellation, the National Aeronautics and Space  Administration Authorization Act of 2010 directed NASA to develop a  Space Launch System, to continue development of a crew vehicle, and  prepare infrastructure at Kennedy Space Center to enable processing  and launch of the launch system. To fulfill this direction, NASA formally  established the SLS program in 2011. Then, in 2012, the Orion project  transitioned from its development under the Constellation program to a  new development program aligned with SLS. To transition Orion from  Constellation, NASA adapted the requirements from the former Orion  plan with those of the newly created SLS and the associated ground  systems programs. In addition, NASA and the European Space Agency  agreed that it would provide a portion of the service module for Orion.  Figure 1 provides details about the heritage of each SLS hardware  element and its source as well as identifies the major portions of the  Orion crew vehicle.

The EGS program was established to modernize the Kennedy Space  Center to prepare for integrating hardware from the three programs as  well as processing and launching SLS and Orion and recovery of the  Orion crew capsule. EGS is made up of nine major components,  including: the Vehicle Assembly Building, Mobile Launcher, Launch  Control Center and software, Launch Pad 39B, Crawler-Transporter,  Launch Equipment Test Facility, Spacecraft Offline Processing, Launch  Vehicle Offline Processing, and Landing and Recovery. See figure 2 for  pictures of the Mobile Launcher, Vehicle Assembly Building, Launch Pad  39B, and Crawler-Transporter.

NASA’s Exploration Systems Development (ESD) organization is  responsible for directing development of the three individual human  spaceflight programs—SLS, Orion, and EGS—into a human space  exploration system. The integration of these programs is key because all  three systems must work together for a successful launch. The integration  activities for ESD’s portfolio occur at two levels in parallel throughout the  life of the programs: as individual efforts to integrate the various elements  managed within the separate programs and as a joint effort to integrate  the three programs into an exploration system.

The three ESD programs support NASA’s long term goal of sending  humans to distant destinations, including Mars. NASA’s approach to  developing and demonstrating the technologies and capabilities to  support their long term plans for a crewed mission to Mars includes three  general stages of activities—Earth Reliant, Proving Ground, and Earth  Independent.

Earth Reliant: From 2016 to 2024, NASA’s planned exploration is  focused on research aboard the International Space Station. On the  International Space Station, NASA is testing technologies and  advancing human health and performance research that will enable  deep space, long duration missions.

Proving Ground: From the mid-2020s to early-2030s, NASA plans to  learn to conduct complex operations in a deep space environment  that allows crews to return to Earth in a matter of days. Primarily  operating in cislunar space—the volume of space around the moon  featuring multiple possible stable staging orbits for future deep space  missions—NASA will advance and validate capabilities required for  humans to live and work at distances much farther away from our  home planet, such as on Mars.

Earth Independent: From the early-2030s to the mid-2040s, planned  activities will build on what NASA learns on the space station and in  deep space to enable human missions to the vicinity of Mars, possibly  to low-Mars orbit or one of the Martian moons, and eventually the  Martian surface.

The first launch of the integrated ESD systems, EM-1, is a Proving  Ground mission. EM-1 is planned as an uncrewed test flight currently  planned for no earlier than October 2019 that will fly about 70,000  kilometers beyond the moon. The second launch, Exploration Mission 2  (EM-2), which will utilize an evolved SLS variant with a more capable  upper stage, is also a Proving Ground mission planned for no later than  April 2023. EM-2 is expected to be a 10- to 14-day crewed flight with up  to four astronauts that will orbit the moon and return to Earth to  demonstrate the baseline Orion vehicle capability. NASA eventually plans  to develop larger and more capable versions of the SLS to support  Proving Ground and Earth Independent missions after EM-2.

As noted above, in April 2017 we found that given the combined effects of  ongoing technical challenges in conjunction with limited cost and  schedule reserves, it was unlikely that the ESD programs would achieve  the November 2018 launch readiness date. We recommended that NASA  confirm whether the EM-1 launch readiness date of November 2018 was  achievable, as soon as practicable but no later than as part of its fiscal  year 2018 budget submission process. We also recommended that NASA  propose a new, more realistic EM-1 date if warranted. NASA agreed with  both recommendations and stated that it was no longer in its best interest  to pursue the November 2018 launch readiness date. Further, NASA  stated that, in fall 2017, it planned to establish a new launch readiness  date. Subsequently, in June 2017, NASA sent notification to Congress  that EM-1’s recommended launch date would be no earlier than October  2019.

The life cycle for NASA space flight projects consists of two phases— formulation, which takes a project from concept to preliminary design, and  implementation, which includes building, launching, and operating the  system, among other activities. NASA further divides formulation and  implementation into pre-phase A through phase F. Major projects must  get approval from senior NASA officials at key decision points before they  can enter each new phase. The three ESD programs are completing  design and fabrication efforts prior to beginning Phase D system  assembly, integration and test, launch and checkout. Figure 3 depicts  NASA’s life cycle for space flight projects.

section_title: NASA’s Integration Approach Offers Some Benefits but Complicates Oversight and Impairs Independence
paragraphs: NASA’s approach for integrating and assessing programmatic and  technical readiness, executed by ESD, differs from prior NASA human  spaceflight programs. This new approach offers some cost and potential  efficiency benefits. However, it also brings challenges specific to its  structure. In particular, there are oversight challenges because only one  of the three programs, Orion, has a cost and schedule estimate for EM-2.  NASA is already contractually obligating money on SLS and EGS for EM- 2, but the lack of cost and schedule baselines for these programs will  make it difficult to assess progress over time. Additionally, the approach  creates an environment of competing interests because it relies on dual- hatted staff to manage technical and safety aspects on behalf of ESD  while also serving as independent oversight of those same areas.
subsections: section_title: Integration Approach Differs from Past Human Spaceflight Programs
paragraphs: NASA is managing the human spaceflight effort differently than it has in  the past. Historically, NASA used a central management structure to  manage human spaceflight efforts for the Space Shuttle and the  Constellation programs. For example, both the Shuttle and Constellation  programs were organized under a single program manager and used a  contractor to support integration efforts. Additionally, the Constellation  program was part of a three-level organization—the Exploration Systems  Mission Directorate within NASA headquarters, the Constellation  program, and then projects, including the launch vehicle, crew capsule,  ground systems, and other lunar-focused projects, managed under the  umbrella of Constellation. Figure 4 illustrates the three-level structure  used in the Constellation program.

In the Constellation program, the programmatic workforce was distributed  within the program and projects. For example, systems engineering and  integration organizations—those offices responsible for making separate  technical designs, analyses, organizations and hardware come together  to deliver a complete functioning system—were embedded within both the  Constellation program and within each of the projects.

NASA’s current approach is organized with ESD, rather than a contractor,  as the overarching integrator for the three separate human spaceflight  programs—SLS, Orion, and EGS. ESD manages both the programmatic  and technical cross-program integration, and primarily relies on personnel  within each program to implement its integration efforts. Exploration  Systems Integration, an office within ESD, leads the integration effort  from NASA headquarters. ESD officials stated that this approach is  similar to that used by the Apollo program, wherein the program was also  managed out of NASA headquarters. Within Exploration Systems  Integration, the Cross-Program Systems Integration sub-office is  responsible for technical integration and the Programmatic and Strategic  Integration sub-office is responsible for integrating the financial, schedule,  risk management, and other programmatic activities of the three  programs. The three programs themselves perform the hardware and  software integration activities. This organizational structure that consists  of two levels is shown in figure 5.

ESD is executing a series of six unique integration-focused programmatic  and technical reviews at key points within NASA’s acquisition life cycle,  as shown in figure 6, to assess whether NASA cost, schedule, and  technical commitments are being met for the three-program enterprise.

These reviews cover the life cycle of the integrated programs to EM-1,  from formulation to readiness to launch. Some of these reviews are  unique to ESD’s role as integration manager, For example, ESD  established two checkpoints—Design to Sync in 2015 and Build to Sync  in 2016. The purpose of Design to Sync was to assess the ability of the  integrated preliminary design to meet system requirements, similar to a  preliminary design review and the purpose of Build to Sync was to assess  the maturity of the integrated design in readiness for assembly,  integration, and test, similar to a critical design review (CDR). At both  events, NASA assessed the designs as ready to proceed. Key  participants in these integration reviews include ESD program personnel  and the Cross-Program Systems Integration and Programmatic and  Strategic Integration staff that are responsible for producing and  managing the integration activities.

section_title: ESD’s Integration Approach Offers Some Cost Avoidance and Potential Efficiency Gains
paragraphs: ESD’s integration approach offers some benefits in terms of cost  avoidance relative to NASA’s most recent human spaceflight effort, the  Constellation program. NASA estimated it would need $190 million per  year for the Constellation program integration budget. By comparison,  between fiscal years 2012 and 2017, NASA requested an average of  about $84 million per year for the combined integration budgets of the  Orion, SLS, EGS, and ESD. This combined average of about $84 million  per year represents a significant decrease from the expected integration  budget of $190 million per year under the Constellation program. In  addition, as figure 7 shows, NASA’s initial estimates for ESD’s required  budget for integration are close to the actuals for fiscal years 2012-2017.  NASA originally estimated that ESD’s budget for integration would require  approximately $30 million per year. ESD’s integration budget was less  than $30 million in fiscal years 2012 and 2013 and increased to about $40  million in fiscal year 2017—an average of about $30 million a year.

According to NASA officials, some of the cost avoidance can be attributed  to the difference in workforce size. The Constellation program’s systems  engineering and integration workforce was about 800 people in 2009, the  last full year of the program; whereas ESD’s total systems engineering  and integration workforce in 2017 was about 500 people, including staff  resident in the individual programs.

ESD officials also stated that, in addition to cost avoidance, their  approach provides greater efficiency. For example, ESD officials said that  decision making is much more efficient in the two-level ESD organization  than Constellation’s three-level organization because the chain of  command required to make decisions is shorter and more direct. ESD  officials also indicated that the post-Constellation elimination of redundant  systems engineering and integration staff at program and project levels  contributed to efficiency. Additionally, they stated that program staff are  invested in both their respective programs and the integrated system  because they work on behalf of the programs and on integration issues  for ESD. Finally, they said another contribution to increased efficiency  was NASA’s decision to establish SLS, Orion, and EGS as separate  programs, which allowed each program to proceed at its own pace.

One caveat to this benefit, however, is that ESD’s leaner organization is  likely to face challenges to its efficiency in the integration and test phases  of the SLS, Orion, and EGS programs. We analyzed the rate at which  ESD has reviewed and approved the different types of launch operations  and ground processing configuration management records for integrated  SLS, Orion, and EGS operations, and found that the process is  proceeding more slowly than ESD anticipated. For example, as figure 8  illustrates, ESD approved 403 fewer configuration management records  than originally planned in the period from March 2016 through June 2017.  According to an ESD official, the lower-than-planned approval rate  resulted from the time necessary to establish and implement a new  review process as well as final records being slower to arrive from the  programs for review than ESD anticipated. Additionally, the official stated  that the records required differing review timelines because they varied in  size and scope.

As figure 8 shows, ESD originally expected the number of items that  needed review and approval to increase and create a “bow wave” during  2017 and 2018. In spring 2017, ESD re-planned its review and approval  process and flattened the bow wave. The final date for review completion  is now aligned with the new planned launch readiness date of no earlier  than October 2019, which added an extra year to ESD’s timeframe to  complete the record reviews. While the bow wave is not as steep as it  was under the original plan, ESD will continue to have a large number of  records that require approval in order to support the launch readiness  date. An ESD official stated that NASA had gained experience managing  such a bow wave as it prepared for Orion’s 2014 exploration flight test  launch aboard a Delta IV rocket and as part of the Constellation  program’s prototype Ares launch in 2009, but acknowledged that ESD will  need to be cautious that its leaner staff is not overwhelmed with  documentation, which could slow down the review process.

section_title: ESD’s Approach Complicates Oversight Because There Is No Mechanism to Assess Affordability beyond First Mission
paragraphs: ESD is responsible for overall affordability for SLS, Orion, and EGS, while  each of the programs develops and maintains an individual cost and  schedule baseline. The baseline is created at the point when a program  receives NASA management approval to proceed into final design and  production. In their respective baselines, as shown in table 1, SLS and  EGS cost and schedule are baselined to EM-1, and Orion’s are baselined  to EM-2. NASA documentation indicates that Orion’s baselines are tied to  EM-2 because that is the first point at which it will fulfill its purpose of  carrying crew. Should NASA determine it is likely to exceed its cost  estimate baseline by 15 percent or miss a milestone by 6 months or  more, NASA is required to report those increases and delays—along with  their impacts—to Congress. In June 2017, NASA sent notification to  Congress that the schedule for EM-1 has slipped beyond the allowed 6- month threshold, but stated that cost is expected to remain within the 15  percent threshold.

NASA has not established EM-2 cost baselines or expected total life- cycle costs for SLS and EGS, including costs related to the larger and  more capable versions of SLS needed to implement the agency’s plans to  send crewed missions to Mars. GAO’s Cost Estimating and Assessment  Guide, a guidebook of cost estimating best practices developed in concert  with the public and private sectors, identifies baselines as a critical means  for measuring program performance over time and addresses how a  baseline backed by a realistic cost estimate increases the probability of a  program’s success. In addition, prior GAO work offers insight into the  benefits of how baselines enhance a program’s transparency. For  example, we found in 2009 that costs for the Missile Defense Agency’s  (MDA) ballistic missile defense system had grown by at least $1 billion,  and that lack of baselines for each block of capability hampered efforts to  measure progress and limited congressional oversight of MDA’s work.  MDA responded to our recommendation to establish these baselines and,  in 2011, we reported that MDA had a new process for setting detailed  baselines, which had resulted in a progress report to Congress more  comprehensive than the one it provided in 2009.

To that end, we have made recommendations in the past on the need for  NASA to baseline the programs’ costs for capabilities beyond EM-1;  however, a significant amount of time has passed without NASA taking  steps to fully implement these recommendations. Specifically, in May  2014, we recommended that, to provide Congress with the necessary  insight into program affordability, ensure its ability to effectively monitor  total program costs and execution, and to facilitate investment decisions,  NASA’s Administrator should direct the Human Exploration and  Operations Mission Directorate to:

Establish a separate cost and schedule baseline for work required to  support the SLS for EM-2 and report this information to the Congress  through NASA’s annual budget submission. If NASA decides to fly the  SLS configuration used in EM-2 beyond EM-2, establish separate life  cycle cost and schedule baseline estimates for those efforts, to  include funding for operations and sustainment, and report this  information annually to Congress via the agency’s budget submission;  and

Establish separate cost and schedule baselines for each additional  capability that encompass all life cycle costs, to include operations  and sustainment, because NASA intends to use the increased  capabilities of the SLS, Orion, and ground support efforts well into the  future and has chosen to estimate costs associated with achieving the  capabilities.

As part of the latter recommendation, we stated that, when NASA could  not fully specify costs due to lack of well-defined missions or flight  manifests, the agency instead should forecast a cost estimate range— including life cycle costs—having minimum and maximum boundaries and  report these baselines or ranges annually to Congress via the agency’s  budget submission.

In its comments on our 2014 report, NASA partially concurred with these  two recommendations, noting that much of what it had already done or  expected to do would address them. For example, the agency stated that  establishing the three programs as separate efforts with individual cost  and schedule commitments met GAO’s intent as would its plans to track  and report development, operations, and sustainment costs in its budget  to Congress as the capabilities evolved. In our response, we stated that  while NASA’s prior establishment of three separate programs lends some  insight into expected costs and schedule at the broader program level, it  does not meet the intent of the two recommendations because cost and  schedule identified at that level is unlikely to provide the detail necessary  to monitor the progress of each block against a baseline. Further,  reporting the costs via the budget process alone will not provide  information about potential costs over the long term because budget  requests neither offer all the same information as life-cycle cost estimates  nor serve the same purpose. Life-cycle cost estimates establish a full  accounting of all program costs for planning, procurement, operations and  maintenance, and disposal and provide a long-term means to measure  progress over a program’s life span.

In 2016, NASA requested closure of these recommendations, citing,  among other factors, changes to the programs’ requirements, design,  architecture, and concept of operations. However, NASA’s request did not  identify any steps taken to meet the intent of these two recommendations,  such as establishing cost and schedule baselines for EM-2, baselines for  each increment of SLS, Orion, or ground systems capability, or  documentation of life cycle cost estimates with minimum and maximum  boundaries. Further, a senior level ESD official told us that NASA does  not intend to establish a baseline for EM-2 because it is not required to do  so. The limited scope that NASA has chosen to use as the basis for  formulating the programs’ cost baselines does not provide the  transparency necessary to assess long-term affordability. Plainly,  progress cannot be assessed without a baseline that serves as a means  to compare current costs against expected costs; consequently, it  becomes difficult to assess program affordability and for Congress to  make informed budgetary decisions.

NASA’s lack of action in regards to our 2014 recommendations means  that it is now contractually obligating NASA to spend billions of dollars in  potential costs for EM-2 and beyond without a baseline against which to  assess progress. For example:  in fiscal year 2016, the SLS program awarded two contracts to Aerojet  Rocketdyne: a $175 million contract for RL-10 engines to power the  exploration upper stage during EM-2 and EM-3 and a $1.2 billion  contract to restart the RS-25 production line required for engines for  use beyond EM-4, and to produce at least 4 additional RS-25  engines;  in 2017, SLS modified the existing Boeing contract upwards by $962  million for work on the exploration upper stage that SLS will use  during EM-2 and future flights; and on a smaller scale, in fiscal year 2016 the EGS program obligated  $4.8 million to support the exploration upper stage and EM-2.

As illustrated by these contracting activities, the SLS program is  obligating more funds for activities beyond EM-1 than Congress directed.  Specifically, of approximately $2 billion appropriated for the SLS program,  the Consolidated Appropriations Act, 2016 directed that NASA spend not  less than $85 million for enhanced upper stage development for EM-2.  NASA has chosen to allocate about $360 million of its fiscal year 2016  SLS appropriations towards EM-2, including enhanced upper stage  development, additional performance upgrades, and payload adapters,  without a baseline to measure progress and ensure transparency. The  NASA Inspector General (IG) also recently reported that NASA is  spending funds on EM-2 efforts without a baseline in place and  expressed concerns about the need for EM-2 cost estimates. Because  NASA has not implemented our recommendations, it may now be  appropriate for Congress to take action to require EM-2 cost and  schedule baselines for SLS and EGS, and separate cost and schedule  baselines for additional capabilities developed for Orion, SLS, and EGS  for missions beyond EM-2. These baselines would be important tools for  Congress to make informed, long-term budgetary decisions with respect  to NASA’s future exploration missions, including Mars.

section_title: Organizational Structure Impairs Independence of Engineering and Safety Technical Oversight
paragraphs: NASA’s governance model prescribes a management structure that  employs checks and balances among key organizations to ensure that  decisions have the benefit of different points of view and are not made in  isolation. As part of this structure, NASA established the technical  authority process as a system of checks and balances to provide  independent oversight of programs and projects in support of safety and  mission success through the selection of specific individuals with  delegated levels of authority. The technical authority process has been  used in other parts of the government for acquisitions, including the  Department of Defense and Department of Homeland Security. ESD is  organizationally connected to three technical authorities within NASA.

The Office of the Chief Engineer technical authority is responsible for  ensuring from an independent standpoint that the ESD engineering  work meets NASA standards,

The Office of Safety and Mission Assurance technical authority is  responsible for ensuring from an independent standpoint that ESD  products and processes satisfy NASA’s safety, reliability, and mission  assurance policies, and

The Office of Chief Health and Medical technical authority is  responsible for ensuring from an independent standpoint that ESD  programs meet NASA’s health and medical standards.

These NASA technical authorities have delegated responsibility for their  respective technical authority functions directly to ESD staff. According  to NASA’s project management requirements, the program or project  manager is ultimately responsible for the safe conduct and successful  outcome of the program or project in conformance with governing  requirements and those responsibilities are not diminished by the  implementation of technical authority.

ESD has established an organizational structure in which the technical  authorities for engineering and safety and mission assurance (S&MA) are  dual hatted to also serve simultaneously in programmatic positions. The  chief engineer technical authority also serves as the Director of ESD’s  Cross Program System Integration Office and the S&MA technical  authority also serves as the ESD Safety and Mission Assurance  Manager. In their programmatic roles for ESD, the individuals manage  resources, including budget and schedule, to address engineering and  safety issues. In their technical authority roles, these same individuals are  to provide independent oversight of programs and projects in support of  safety and mission success. Having the same individual simultaneously  fill both a technical authority role and a program role creates an  environment of competing interests where the technical authority may be  subject to impairments in their ability to impartially and objectively assess  the programs while at the same time acting on behalf of ESD in  programmatic capacities. This duality makes them more subject to  program pressures of cost and schedule in their technical authority roles.  Figure 9 describes some of the conflicting roles and responsibilities of  these officials in their two different positions.

The concurrency of duties leaves the positions open to conflicting goals of  safety, cost, and schedule and increases the potential for the technical  authorities to become subject to cost and schedule pressures. For  example:  the dual-hatted engineering and S&MA technical authorities serve on  decision-making boards both in technical authority and programmatic  capacities, making them responsible for providing input on technical  and safety decisions while also keeping an eye on the bottom line for  ESD’s cost and schedule; and  the technical authorities are positioned such that they have been the  reviewers of the ESD programmatic areas they manage—in essence,  “grading their own homework.” For example, at ESD’s Build to Sync  review in 2016, the engineering and S&MA technical authorities  evaluated the areas that they manage in their respective capacities as  ESD Director of Cross Program System Integration and ESD Safety  and Mission Assurance Manager. This process relied on their abilities  as individuals to completely separate the two hats—using one hand to  put on the ESD hat and manage technical and safety issues within  programmatic cost and schedule constraints and using the other hand  to take off that hat and assess the same issues with an independent  eye.

NASA officials identified several reasons why the dual-hat structure works  for their purposes. Agency officials stated that one critical factor to  successful dual-hatting is having the “right” people in those dual-hat  positions—that is, personnel with the appropriate technical knowledge to  do the work and the ability to act both on behalf of ESD and independent  of it. Officials also indicated that technical authorities retain independence  because their technical authority reporting paths and performance  reviews are all within their technical authority chain of command rather  than under the purview of the ESD chain of command.

Additionally, agency officials said that dual-hat roles are a commonplace  practice at NASA and cited other factors in support of the approach,  including that:  it would not be an efficient use of resources to have an independent  technical authority with no program responsibilities because that  person would be unlikely to have sufficient program knowledge to  provide useful insight and could slow the program’s progress; a technical authority that does not consider cost and schedule is not  helpful to the program because it is unrealistic to disregard those  aspects of program management; a strong dissenting opinion process is in place and allows for issues to  be raised through various levels to the Administrator level within  NASA; and

ESD receives additional independent oversight through three NASA  internal organizations—the independent review teams that provide  independent assessments of a program’s technical and programmatic  status and health at key points in its life cycle; the NASA Engineering  and Safety Center that conducts independent safety and mission  success-related testing, analysis, and assessments of NASA’s high- risk projects; and the Aerospace Safety Advisory Panel (ASAP) that  independently oversees NASA’s safety performance.

These factors that NASA officials cite in support of the dual-hat approach  minimize the importance of having independent oversight and place ESD  at risk of fostering an environment in which there is no longer a balance  between preserving safety with the demands of maintaining cost and  schedule. The Columbia Accident Investigation Board (CAIB) report—the  result of an in-depth assessment of the technical and organizational  causes of the Columbia accident—concluded that NASA’s organization  for the Shuttle program combined, among other things, all authority and  responsibility for schedule, cost, safety, and technical requirements and  that this was not an effective check and balance. The CAIB report  recommended that NASA establish a technical authority to serve  independently of the Space Shuttle program so that employees would not  feel hampered to bring forward safety concerns or disagreements with  programmatic decisions. The Board’s findings that led to this  recommendation included a broken safety culture in which it was difficult  for minority and dissenting opinions to percolate up through the hierarchy;  dual Center and programmatic roles vested in one person that had  confused lines of authority, responsibility, and accountability and made  the oversight process susceptible to conflicts of interest; and oversight  personnel in positions within the program, increasing the risk that these  staffs’ perspectives would be hindered by too much familiarity with the  programs they were overseeing.

ESD officials stated that they had carefully and thoughtfully implemented  the intent of the CAIB; they said they had not disregarded its finding and  recommendations but instead established a technical authority in such a  way that it best fit the context of ESD’s efforts. These officials did  acknowledge, though, that the dual hat approach does not align with the  CAIB report’s recommendation to separate programmatic and technical  authority or with NASA’s governance framework. Further, over the course  of our review, we spoke with various high-ranking officials outside and  within NASA who expressed some reservations about ESD’s dual hat  approach. For example:

The former Chairman of the CAIB stated that, even though the ESD  programs are still in development, he believes the technical authority  should be institutionally protected against the pressures of cost and  schedule and added that NASA should never be lulled into dispensing  of engineering and safety independence because human spaceflight  is an extremely risky enterprise.

Both NASA’s Chief Engineer and Chief of S&MA acknowledged there  is inherent conflict in the concurrent roles of the dual hats, while also  expressing great confidence in the ESD staff now in the dual roles.

NASA’s Chief of S&MA indicated that the dual-hat S&MA structure is  working well within ESD, but he believes these dual-hatted roles may  not necessarily meet the intent of the CAIB’s recommendation  because the Board envisioned an independent safety organization  completely outside the programs.

NASA’s Chief Engineer stated that he believes technical authority  should become a separate responsibility and position as ESD moves  forward with integration of the three programs and into their operation  as a system.

As these individuals made clear, ensuring the ESD engineering and  S&MA technical authorities remain independent of cost and schedule  conflicts is key to human spaceflight success and safety. Along these  lines, the ASAP previously conveyed concerns about NASA’s  implementation of technical authority that continue to be valid today. In  particular, the ASAP stated in a 2013 report that NASA’s technical  authority was working at that time in large measure due to the well- qualified, strong personnel that had been assigned to the process. The  panel noted, however, that should there be a conflict or weakening of the  placement of strong individuals in the technical authority position, this  could introduce greater risk into a program. Although a current ASAP  official stated she had no concerns with ESD’s present approach to  technical authority, the panel’s prior caution remains applicable, and the  risk that the ASAP identified earlier could be realized if not mitigated by  eliminating the potential for competing interests within the ESD  engineering and S&MA positions.

NASA is currently concluding an assessment of the implementation of the  technical authority role to determine how well that function is working  across the agency. According to the official responsible for leading the  study, the assessment includes examining the evolution of the technical  authority role over the years and whether NASA is spending the right  amount of funds for those positions. NASA expects to have  recommendations in 2017 on how to improve the technical authority  function, but does not expect to address the dual hat construct. A  principle of federal internal controls is that an agency should design  control activities to achieve objectives and respond to risks, which  includes segregation of key duties and responsibilities to reduce the risk  of error, misuse, or fraud. By overlapping technical authority and  programmatic responsibilities, NASA will continue to run the risk of  creating an environment of competing interests for the ESD engineering  and S&MA technical authorities.

section_title: ESD Risk Posture Has Improved, but Key Risk Areas Remain for the Integration Effort
paragraphs: Despite the development and integration challenges associated with a  new human spaceflight capability, ESD has improved its overall cross- program risk posture over the past 2 years. Nonetheless, it still faces key  integration risk areas within software development and verification and  validation (V&V). Both are critical to readiness for EM-1 because software  acts as the “brain” that ties SLS, Orion, and EGS together in a functioning  body, while V&V ensures the integrated body works as expected. The  success of these efforts forms the foundation for a launch, no matter the  date of EM-1.
subsections: section_title: ESD’s Cross-Program Risk Posture Has Improved
paragraphs: We have previously reported on individual SLS, Orion, and EGS program  risks that were contributing to potential delays within each program. For  example, in July 2016, we found that delays with the European Service  Module—which will provide services to the Orion crew module in the form  of propulsion, consumables storage, and heat rejection and power—could  potentially affect the Orion program’s schedule. Subsequently, in April  2017, we found that those delays had worsened and were contributing to  the program likely not making a November 2018 launch readiness date.  All three programs continue to manage such individual program risks,  which is to be expected of programs of this size and complexity. The  programs may choose to retain these risks in their own risk databases or  elevate them to ESD to track mitigation steps. A program would elevate a  risk to ESD when decisions are needed by ESD management, such as a  need for additional resources or requirement changes. Risks with the  greatest potential for negative impacts are categorized as top ESD risks.  In addition to these individual programs risks that are elevated to ESD,  ESD is also responsible for overseeing cross-program risks that affect  multiple programs. An example of a cross-program risk is the potential for  delayed delivery of data from SLS and Orion to affect the EGS software  development schedule.

ESD has made progress reducing risks over the last 2 years, from the  point of the Design to Sync preliminary design review equivalent for the  integrated programs to the Build to Sync critical design review equivalent.  As figure 10 illustrates, ESD has reduced its combined total of ESD and  cross program risks from 39 to 25 over this period, and reduced the  number of high risks from about 49 percent of the total to about 36  percent of the total.

The ESD risk system is dynamic, with risks coming into and dropping out  of the system over time as development proceeds and risk mitigation is  completed. A total of 29 of the 39 risks within the ESD risk portfolio were  removed from the register and 15 risks were added to the register  between November 2014, prior to Design to Sync, and March 2017, after  Build to Sync. Examples of risks removed over this time period include  risks associated with late delivery of Orion and SLS ground support  equipment hardware to EGS and establishing a management process to  identify risks stemming from the programs being at differing points in  development.

Nine risks remained active in the system over the 2-year period we  analyzed, and NASA experienced delays in the length of time it  anticipated it would take to complete mitigation of the majority of these  nine risks. Three of these nine risks that have remained active in the risk  system since before Design to Sync are still classified as high risk; the  remaining six are classified as medium risk. Mitigation is an action taken  to eliminate or reduce the potential severity of a risk, either by reducing  the probability of it occurring, by reducing the level of impact if it does  occur, or both. ESD officials indicated a number of reasons why risks  could take longer to mitigate. For instance, risks with long-term mitigation  strategies may go for extended periods of time without score changes. In  addition, ESD may conduct additional risk assessments and determine  that certain risks need to be reprioritized over time and that resources  should be focused towards higher risks. In addition, some risk mitigation  steps are tied to hardware delivery and launch dates, and as those delay,  the risk mitigation steps will as well. As illustrated in table 2, we found that  six of these nine risks were related to software and V&V and represented  some of the primary causes in terms of estimated completion delays. On  average, the estimated completion dates for these six risks were delayed  about 16 months. In addition, the two V&V risks that have remained  active since before Design to Sync were still considered top ESD risks as  of March 2017 when we completed this analysis.

section_title: Software Development Is a Key Risk Area Facing the Integration Effort
paragraphs: Software development is one of the top cross-program technical issues  facing ESD as the programs approach EM-1. Software is a key enabling  technology required to tie the human spaceflight systems together.  Specifically, for ESD to achieve EM-1 launch readiness, software  developed within each of the programs has to be able to link and  communicate with software developed in other programs in order to  enable a successful launch. Furthermore, software development  continues after hardware development and is often used to help resolve  hardware deficiencies discovered during systems integration and test.

ESD has defined six critical paths—the path of longest duration through  the sequence of activities that determines the program’s earliest  completion date—for its programs to reach EM-1, and three are related to  software development. These three software critical paths support  interaction and communication between the systems the individual  programs are developing—SLS to EGS software, Orion to EGS software,  and the Integrated Test Laboratory (ITL) facility that supports Orion  software and avionics testing as well as some SLS and EGS testing. The  other critical paths are development of the Orion crew service module,  SLS core stage, and the EGS Mobile Launcher. Because of software’s  importance to EM-1 launch readiness, ESD is putting a new method in  place to measure how well these software efforts are progressing along  their respective critical paths. To that end, it is currently developing a set  of “Key Progress Indicators” milestones that will include baseline and  forecast dates. Officials indicated that these metrics will allow ESD to  better track progress of the critical path software efforts toward EM-1  during the remainder of the system integration and test phase. ESD  officials have indicated, however, that identifying and establishing  appropriate indicators is taking longer than expected and proving more  difficult than anticipated.

One of the software testing critical paths, the ITL, has already  experienced delays that slipped completion of planned software testing  from September 2018 until March 2019, a delay of 6 months. Officials told  us that this delay was primarily due to a series of late avionics and  software deliveries by the European Space Agency for Orion’s European  Service Module. The delay in the Orion testing in turn affects SLS and  EGS software testing and integration because those activities are  informed by the completion of the Orion software testing. Furthermore,  some EGS and SLS software testing scheduled to be conducted within  the ITL has been re-planned as a result of the Orion delays.

The Orion program indicates that it has taken action to mitigate ITL issues  as they arise. For example, the European Service Module avionics and  software delivery delay opened a 125-day gap between completion of  crew module testing and service module testing. Orion officials indicated  that the program had planned to proceed directly into testing of the  integrated crew module and service module software and systems, but  the integrated testing cannot be conducted until the service module  testing is complete.

As illustrated by figure 11, to mitigate the impact of the delay, Orion  officials indicated that the program filled this gap by rescheduling other  activities at the ITL such as software integration testing and dry runs for  the three programs. These adjustments narrowed the ITL schedule gap  from 125 days to 24 days. The officials stated that they will continue to  adjust the schedule to eliminate gaps.

The other two software critical paths—SLS to EGS and Orion to EGS  software development—are also experiencing software development  issues. In July 2016, for example, we found that delays in SLS and Orion  requirements development, as well as the programs’ decisions to defer  software content to later in development, were delaying EGS’s efforts to  develop ground command and control software and increasing cost and  schedule.

Furthermore, ESD reports show that delays and content deferral in the  Orion and SLS software development continue to affect EGS software  development and could delay launch readiness. For example, the EGS  data throughput risk that both ESD and EGS are tracking is that the  ground control system software is currently not designed to process the  amount of telemetry it will receive and provide commands to SLS and  ground equipment as required during launch operations. EGS officials  stated that, if not addressed, the risk is that if there is a SLS or Orion  failure, the ground control system software may not display the necessary  data to launch operations technicians. EGS officials told us that the  reason for the mismatch between the data throughput being sent to the  ground control software and how much is it designed to process is that no  program was constrained in identifying its data throughput. These officials  stated that retrospectively, they should have established an interface  control document to manage the process. The officials also stated that  the program is taking steps to mitigate this risk, including defining or  constraining the data parameters and buying more hardware to increase  the amount of data throughput that can be managed, but will not know if  the risk is fully mitigated until additional data are received and analyzed  during upcoming tests. For example, EGS officials stated that the green  run test will provide additional data to help determine if the steps they are  taking address this throughput risk. If the program determines the risk is  not fully mitigated and additional software redesign is required, it could  lead to schedule delays.

ESD officials overseeing software development acknowledged that  software development for the integrated systems is a difficult task and  said they expect to continue to encounter and resolve software  development issues during cross-program integration and testing. As we  have found in past reviews of NASA and Department of Defense  systems, software development is a key risk area during system  integration and testing. For example, we found in April 2017 that software  delivery delays and development problems with the U.S. Air Force’s F-35  program experienced during system integration and testing were likely to  extend that program’s development by 12 months and increase its costs  by more than $1.7 billion.

section_title: Verification and Validation Will Remain Key Risk Area to Monitor as NASA Establishes and Works towards New Launch Readiness Date
paragraphs: Verification and validation (V&V) is acknowledged by ESD as a top cross- program integration risk that NASA must monitor as it establishes and  works toward a new EM-1 launch readiness date. V&V is a culminating  development activity prior to launch for determining whether integrated  hardware and software will perform as expected. V&V consists of two  equally important aspects:  verification is the process for determining whether or not a product  fulfills the requirements or specifications established for it at the start  of the development phase; and  validation is the assessment of a planned or delivered system ability  to meet the sponsor’s operational need in the most realistic  environment achievable during the course of development or at the  end of development.

Like software development and testing, V&V is typically complex and  made even more so by the need to verify and validate how SLS, Orion,  and EGS work together as an integrated system.

ESD’s V&V plans for the integrated system have been slow to mature. In  March 2016, leading up to ESD’s Build to Sync review, ESD performed  an audit of V&V-related documentation for the program CDRs and ESD  Build to Sync. The audit found that 54 of 257 auditable areas (21 percent)  were not mature enough to meet NASA engineering policy guidance for  that point in development. According to ESD documentation, there were  several causes of this immaturity, including incomplete documentation  and inconsistent requirements across the three programs. NASA officials  told us that our review prompted ESD to conduct a follow-up and track the  status of these areas. As of June 2017, 53 of the 54 auditable areas were  closed, which means these areas are at or have exceeded CDR level of  maturity—6 months after Build to Sync was completed. NASA officials  indicated that the remaining one auditable area, which is related to the  test plan for the integrated communication network, was closed in August  2017.

Nevertheless, other potential V&V issues still remain. According to ESD  officials, distributing responsibility for V&V across the three programs has  created an increased potential for gaps in testing. If gaps are discovered  during testing, or if integrated systems do not perform as planned, money  and time for modifications to hardware and/or software may be  necessary, as well as time for retesting. This could result in delayed  launch readiness. As a result, mature V&V plans are needed to ensure  there are no gaps in planned testing. ESD officials indicated that a NASA  Engineering and Safety Center review of their V&V plans, requested by  ESD’s Chief Engineer to address concerns about V&V planning, would  help define the path forward for maturing V&V plans. V&V issues add to  cost and schedule risk for the program because they may take more time  and money to resolve than ESD anticipates. In some cases, they may  have a safety impact as well. For example, if the structural models are not  sufficiently verified, it increases flight safety risks. Each of the programs  bases its individual analyses on the models of the other programs. As a  result, any deficiencies discovered in one can have cascading effects  through the other systems and programs. We will continue to monitor  ESD’s progress mitigating risks as NASA approaches EM-1.

section_title: Conclusions
paragraphs: NASA is at the beginning of the path leading to human exploration of  Mars. The first phase along that path, the integration of SLS, Orion, and  EGS, is likely to set the stage for the success or failure of the rest of the  endeavor. Establishing a cost and schedule baseline for NASA’s second  mission is an important initial step in understanding and gaining support  for the costs of SLS, Orion, and EGS, not just for that one mission but for  the Mars plan overall. NASA’s ongoing refusal to establish this baseline is  short-sighted, because EM-2 is part of a larger conversation about the  affordability of a crewed mission to Mars. While later stages of the Mars  mission are well in the future, getting to that point in time will require a  funding commitment from the Congress and other stakeholders. Much of  their willingness to make that commitment is likely to be based on the  ability to assess the extent to which NASA has met prior goals within  predicted cost and schedule targets.

Furthermore, as ESD moves SLS, Orion, and EGS from development to  integrated operations, its efforts will reach the point when human lives will  be placed at risk. Space is a severe and unforgiving environment; the  Columbia accident showed the disastrous consequences of mistakes. As  the Columbia Accident Investigation Board report made clear, a  program’s management approach is an integral part of ensuring that  human spaceflight is as safe and successful as possible. The report also  characterized independence as key to achieving that safety and success.  ESD’s approach, however, tethers independent oversight to program  management by vesting key individuals to wear both hats at the same  time. As a result, NASA is relying heavily on the personality and capability  of those individuals to maintain independence rather than on an  institutional process, which diminishes lessons learned from the Columbia  accident.

section_title: Matter for Congressional Consideration
paragraphs: We are making the following matter for congressional consideration.

Congress should consider requiring the NASA Administrator to direct the  Exploration Systems Development organization within the Human  Exploration and Operations Mission Directorate to establish separate cost  and schedule baselines for work required to support SLS and EGS for  Exploration Mission 2 and establish separate cost and schedule baselines  for each additional capability that encompass all life cycle costs, to  include operations and sustainment. (Matter for Consideration 1)

section_title: Recommendation for Executive Action
paragraphs: We are making the following recommendation to the Exploration Systems  Development organization.

Exploration Systems Development should no longer dual-hat individuals  with both programmatic and technical authority responsibilities.  Specifically, the technical authority structure within Exploration Systems  Development should be restructured to ensure that technical authorities  for the Offices of the Chief Engineer and Safety and Mission Assurance  are not fettered with programmatic responsibilities that create an  environment of competing interests that may impair their independence.  (Recommendation 1)

section_title: Agency Comments and Our Evaluation
paragraphs: NASA provided written comments on a draft of this report. These  comments are reprinted in appendix II. NASA also provided technical  comments, which were incorporated as appropriate.

In responding to a draft of our report, NASA partially concurred with our  recommendation that the Exploration Systems Development (ESD)  organization should no longer dual-hat individuals with both programmatic  and technical authority responsibilities. Specifically, we recommended  that the technical authority structure within ESD should be restructured to  ensure that technical authorities for the Offices of Chief Engineer and  Safety and Mission Assurance are not fettered with programmatic  responsibilities that create an environment of competing interests that  may impair their independence. In response to this recommendation,  NASA stated that it created the technical authority governance structure  after the Columbia Accident Investigation Board report and that the dual- hat technical authority structure has been understood and successfully  implemented within ESD. NASA recognized, however, that as the  program moves from the design and development phase into the  integration and test phase, it anticipates that the ESD environment will  encounter more technical issues that will, by necessity, need to be quickly  evaluated and resolved. NASA asserted that within this changed  environment it would be beneficial for the Engineering Technical Authority  role to be performed by the Human Exploration and Operations Chief  Engineer (who reports to the Office of the Chief Engineer). NASA stated  that over the next year or so, it would solicit detailed input from these  organizations and determine how to best support the program while  managing the transition to integration and test and anticipated closing this  recommendation by September 30, 2018.

We agree that NASA should solicit detailed input from key organizations  within the agency as it transitions away from the dual hat technical  authority structure to help ensure successful implementation of a new  structure. In order to implement this recommendation, however, NASA  needs to assign the technical authority role to a person who does not  have programmatic responsibilities to ensure they are independent of  responsibilities related to cost and schedule performance. To fulfill this,  this person may need to reside outside of the Human Exploration and  Operations Mission Directorate and NASA should solicit input from the  Office of the Chief Engineer when making this decision to ensure that  there are no competing interests for the technical authority. Moreover, in  its response, NASA does not address the dual-hat technical authority role  for Safety and Mission Assurance. We continue to believe that similar  changes for this role would be appropriate as well.

Further, in response to this recommendation, NASA makes two  statements that require additional context.  First, NASA stated that GAO’s  recommendation was focused on overall Agency technical authority  management. While this review involved meeting with the heads of the  Office of Chief Engineer and the Office of Safety and Mission Assurance,  the scope of this review and the associated recommendation are limited  to ESD. Second, NASA stated “As you found, we agree that having the  right personnel in senior leadership positions is essential for a Technical  Authority to be successful regardless of how the Technical Authority is  implemented.” To clarify, this perspective is attributed to NASA officials in  our report and does not represent GAO’s position.

We are sending copies of this report to NASA’s Administrator and to  appropriate congressional committees. In addition, the report is available  at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report assesses (1) the benefits and challenges of the National  Aeronautics and Space Administration’s (NASA) approach for integrating  and assessing the programmatic and technical readiness of Orion, SLS,  and EGS; and (2) the extent to which the Exploration Systems  Development (ESD) organization is managing cross-program risks that  could affect launch readiness.

To assess the benefits and challenges of NASA’s approach for integrating  and assessing the programmatic and technical readiness of its current  human spaceflight programs relative to other selected programs, we  reviewed and analyzed NASA policies governing program and technical  integration, including cost, schedule, and risk. We obtained and analyzed  ESD implementation plans to assess the role of ESD in cross program  integration of the three programs. We reviewed the 2003 Columbia  Accident Investigation Board’s Report’s findings and recommendations  related to culture and organizational management of human spaceflight  programs as well as the Constellation program’s lessons learned report.  We reviewed detailed briefings and documentation from Cross-Program  Systems Integration and Programmatic and Strategic Integration teams  explaining ESD’s approach to programmatic and technical integration,  including implementation of systems engineering and integration. We  interviewed NASA officials to discuss the benefits and challenges of  NASA’s integration approach and their roles and responsibilities in  managing and overseeing the integration process. We met with the  technical authorities and other representatives from the NASA Office of  the Chief Engineer, Office of Safety and Mission Assurance, Crew Health  and Safety, addressed cost and budgeting issues with the Chief Financial  Officer and discussed and documented their roles in executing and  overseeing the ESD programs. We also interviewed outside subject  matter experts to gain their insight of ESD’s implementation of NASA’s  program management policies on the independent technical authority  structure. Additionally, we compared historical budget data from the now- cancelled Constellation program to ESD budget data and quantified  systems engineering and integration budget savings through preliminary  design review, the point at which the Constellation program was  cancelled. In addition, we assessed the scope of NASA’s funding  estimates for the second exploration mission and beyond against best  practices criteria outlined in GAO’s cost estimating guidebook. We  assessed the reliability of the budget data obtained using GAO reliability  standards as appropriate. We compared the benefits and challenges of  NASA’s integration approach to that of other complex, large-scale  government programs, including NASA’s Constellation and the  Department of Defense’s Missile Defense Agency programs.

To determine the extent to which ESD is managing cross-program risks  that could affect launch readiness, we obtained and reviewed NASA and  ESD risk management policies; detailed monthly and quarterly briefings;  and documentation from Cross-Program Systems Integration and  Programmatic and Strategic Integration teams explaining ESD’s approach  to identifying, tracking, and mitigating cross-program risks. We reviewed  Cross-Program Systems Integration systems engineering and systems  integration areas as well as Programmatic and Strategic Integration risks,  cost, and schedule to determine what efforts presented the highest risk to  cross program cost and schedule. We conducted an analysis of ESD’s  risk dataset and the programs’ detailed risk reports, which list program  risks and their potential schedule impacts, including mitigation efforts to  date. We examined risk report data from Design to Sync to Build to Sync  and focused our analyses to identify risks with current mitigation plans to  determine if risk mitigation plans are proceeding on schedule. We did not  analyze risks that were categorized under “Accept,” “Candidate,”  “Research,” “Unknown,” or “Watch” because these risks were not  assigned an active mitigation plan by ESD. To assess the reliability of the  data, we reviewed related documentation and interviewed knowledgeable  agency officials. We determined the data was sufficiently reliable for  identifying risks and schedule delays associated with those risks. We  examined ESD integrated testing facility schedules to determine the  extent to which they can accommodate deviation in ESD’s planned  integrated test schedule. We also interviewed program and contractor  officials on technical risks, potential impacts, and risk mitigation efforts  underway and planned.

We conducted this performance audit from August 2016 to October 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Comments from the National Aeronautics and Space Administration

section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, Molly Traci (Assistant Director),  LaTonya Miller, John S. Warren Jr., Tana Davis, Laura Greifner, Roxanna  T. Sun, Samuel Woo, Marie P. Ahearn, and Lorraine Ettaro made key  contributions to this report.

Fast facts: NASA is developing 3 programs to put astronauts into space—the Orion crew vehicle, Space Launch System, and Exploration Ground Systems. All 3 programs must work together.

We found challenges in NASA's approach to integrating these programs. For example, the technical authorities for engineering and safety also have program roles that include managing resources. When technical authorities must also deal with cost and schedule pressures, it can potentially impair their independence. A review board concluded this type of tension contributed to the 2003 Columbia shuttle accident.

We made recommendations to strengthen oversight and independence.

Question-summary annotations: questions: question: What is NASA attempting to accomplish?
answer: The approach that the National Aeronautics and Space Administration (NASA) is using to integrate its three human spaceflight programs into one system ready for launch offers some benefits, but it also introduces oversight challenges.

question: What are the three spaceflight programs?
answer: To manage and integrate the three programs—the Space Launch System (SLS) vehicle; the Orion crew capsule; and supporting ground systems (EGS)—NASA's Exploration Systems Development (ESD) organization is using a more streamlined approach than has been used with other programs, and officials GAO spoke with believe that this approach provides cost savings and greater efficiency.

question: How will this new approach benefit NASA?
answer: To manage and integrate the three programs—the Space Launch System (SLS) vehicle; the Orion crew capsule; and supporting ground systems (EGS)—NASA's Exploration Systems Development (ESD) organization is using a more streamlined approach than has been used with other programs, and officials GAO spoke with believe that this approach provides cost savings and greater efficiency.

questions: question: What is one challenge with this new approach?
answer: The approach makes it difficult to assess progress against cost and schedule baselines.
child_questions: question: What is an example of this challenge?
answer: SLS and EGS are baselined only to the first test flight.

question: What did GAO recommend to solve this issue?
answer: In May 2014, GAO recommended that NASA baseline the programs' cost and schedule beyond the first test flight.

question: What are the consequences since NASA has not implemented the recommendations?
answer: NASA has not implemented these recommendations nor does it plan to; hence, it is contractually obligating billions of dollars for capabilities for the second flight and beyond without establishing baselines necessary to measure program performance.

questions: question: How has NASA's risk posture changed?
answer: NASA has lowered its overall cross-program risk posture over the past 2 years, but risk areas—related to software development and verification and validation, which are critical to ensuring the integrated body works as expected—remain.
child_questions: question: Which risk areas still remain?
answer: NASA has lowered its overall cross-program risk posture over the past 2 years, but risk areas—related to software development and verification and validation, which are critical to ensuring the integrated body works as expected—remain.

question: What could still delay program readiness?
answer: For example, delays and content deferral in Orion and SLS software development continue to affect ground systems software development and could delay launch readiness.

question: What is GAO's duty in this situation?
answer: GAO will continue to monitor these risks.

questions: question: When are all three programs hoping to be ready by?
answer: All three programs (SLS, Orion, and EGS) are working toward a launch readiness date of no earlier than October 2019 for the first test flight.

question: What must NASA do for program readiness for all three programs?
answer: Because all three programs must work together for launch, NASA must integrate the hardware and software from the separate programs into a working system capable of meeting its goals for deep space exploration.
child_questions: question: What type of programs is NASA undertaking for this project?
answer: NASA is undertaking a trio of closely related programs to continue human space exploration beyond low-Earth orbit.

questions: question: Which report included a provision regarding GAO and NASA?
answer: The House Committee on Appropriations report accompanying H.R. 2578 included a provision for GAO to assess the progress of NASA's human space exploration programs.
child_questions: question: What are the two points this report assesses?
answer: This report assesses (1) the benefits and challenges of NASA's approach for integrating these three programs and (2) the extent to which cross-program risks could affect launch readiness.

question: What actions did GAO take in their assessment?
answer: GAO examined NASA policies, the results of design reviews, risk data, and other program documentation and interviewed NASA and other officials.

questions: question: What should Congress consider doing with regards to NASA?
answer: Congress should consider directing NASA to establish baselines for SLS and EGS's missions beyond the first test flight.
child_questions: question: What is it recommended that NASA’s ESD organization no longer do?
answer: NASA's ESD organization should no longer dual-hat officials with programmatic and technical authority responsibilities.

question: Did NASA agree to this suggestion?
answer: NASA partially concurred with our recommendation and plans to address it in the next year.
child_questions: question: What did NASA fail to address?
answer: But NASA did not address the need for the technical authority to be independent from programmatic responsibilities for cost and schedule.
child_questions: question: What is GAO’s view of this component of the recommendation?
answer: GAO continues to believe that this component of the recommendation is critical.