United States General Accounting Office

GAO Report to the Honorable

James A. Hayes, House of

Representatives

March 1996 SPACE SHUTTLE

Need to Sustain

Launch Risk

Assessment Process

Improvements

G O A

years

1921 - 1996

GAO/NSIAD-96-73

GAO United States

General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-260150 

March 26, 1996

The Honorable James A. Hayes

House of Representatives

Dear Mr. Hayes:

As requested, we reviewed the National Aeronautics and Space Administration’s (NASA)

management of risk associated with flying the space shuttle. This report addresses NASA’s

efforts to create an environment conducive to the free flow of needed risk information and

adopt quantitative methods for assessing risk.

We are sending copies of the report to interested congressional committees, the Administrator

of NASA, and the Director of the Office of Management and Budget. We will also provide copies

to others on request.

Please contact me at (202) 512-8412 if you or your staff have any questions concerning this

report. Other major contributors are listed in appendix V.

Sincerely yours,

David R. Warren, Director

Defense Management Issues
Executive Summary

Purpose The 1986 space shuttle Challenger accident brought into sharp focus the

risks involved in human space flight. The Presidential Commission that

investigated the accident found that it was caused by a poor design of the

joints holding the solid rocket motors together, but the Commission also

cited inadequacies in the National Aeronautics and Space Administration’s

(NASA) processes for identifying, assessing, and managing risk as

contributing factors.

The former Chairman, Subcommittee on Investigations and Oversight,

House Committee on Science, Space, and Technology, asked GAO to review

NASA’s management of risk associated with flying the shuttle. Specifically,

GAO reviewed the actions NASA has taken to improve the free flow of

information in the launch decision process and the progress NASA has

made in adopting quantitative methods for assessing risk.

Background Space systems are inherently risky because of the technology involved and

the complexity of the activity. For example, thousands of people perform

about 1.2 million separate procedures to process a shuttle for flight. While

the risks cannot be completely eliminated, they must be identified and

managed to the extent possible.

Although the Presidential Commission determined that a faulty solid

rocket motor joint design caused the accident, it identified other

contributing factors. The Commission concluded that there were serious

flaws in the decision-making process leading up to launch. It reported

management isolation and communication failures as contributing causes

to the accident. The Commission cited the propensity of some NASA

managers to attempt to resolve potentially serious problems internally

rather than tell higher management levels.

Following the Commission’s investigation, the National Research Council

reviewed NASA’s approach to conducting shuttle risk assessments. The

Council found that NASA was placing too much reliance on qualitative risk

assessments and recommended greater use of quantitative methods, such

as probabilistic risk assessments.1

GAO assessed the current communications environment, in part, by

observing the launch decision process and by interviewing shuttle

managers, representatives of NASA’s safety organization, and managers and

1

Probabilistic risk assessment is a method of systematically examining complex technical systems to

measure both the likelihood that an accident will occur (probability) and the level of damage or loss

that will result (consequences).

Page 2 GAO/NSIAD-96-73 Space Shuttle
Executive Summary

working-level engineers at three shuttle contractors. GAO initially

interviewed officials in groups. The group interviews enabled participants

to exchange their perspectives on communications within the shuttle

program, provided GAO with an understanding of these complex areas, and

produced concrete illustrations. GAO followed up the interviews of NASA

officials with a structured survey to more precisely measure views of

communications issues that emerged from the group discussions.

Results in Brief NASA has been successful in creating an environment conducive to the free

flow of needed risk information. NASA managers and safety representatives

responsible for shuttle operations reported that they believe conditions

governing the flow of information and decision processes are appropriate.

However, some viewed the management information systems as needing

improvement. GAO’s own analysis of communication flows in NASA’s

assessment of a recently identified shuttle hardware problem illustrated

significant improvements in communication of risk information.

Some discussion group participants expressed concern about impending

budget reductions and the transition of shuttle operations to a prime

contractor. The challenge for NASA will be to maintain the principles of

effective communications it has in place now as it continues to reduce

shuttle funding and transfers management of shuttle operations to a single

contractor.

NASA still relies primarily on qualitative risk assessments, but has made

limited progress in adopting quantitative methods, such as probabilistic

risk assessments, for assessing significant shuttle risks. However, there is

not a consensus among shuttle managers and safety representatives on

increasing the use of these methods. NASA lacks an overall strategy on

when to use such methods to supplement engineering judgments. Officials

cited limited resources and a lack of personnel with expertise in these

methods as barriers to their implementation. Officials told GAO that

another reason quantitative methods are not used more routinely is that

needed data is not always available in a readily usable form.

Page 3 GAO/NSIAD-96-73 Space Shuttle
Executive Summary

Principal Findings

NASA Has Improved the

Flow of Information in the

Shuttle Program

GAO asked NASA and contractor managers, safety representatives, and

contractor engineers to assess conditions related to the flow and quality of

information to management. Based on GAO’s interviews and survey

responses, the program’s organizational culture encourages people to

discuss safety concerns and to elevate concerns to higher management if

they believe the issues were not adequately addressed at lower levels.

A variety of communication forums help ensure that NASA and contractor

managers and the safety community are continually apprised of safety

problems and issues that arise during shuttle processing. These forums

include the certification of flight readiness, daily telephone conferences,

and weekly meetings.

NASA managers at the three field centers with primary responsibility for the

shuttle program and at headquarters reported having taken steps to create

an organizational environment that encourages personnel at all levels to

voice their views on safety issues to management. For example, managers

encourage debate at readiness reviews and other meetings and invite

individuals to meet with them at other times about safety concerns.

Although the current program culture encourages open discussion of

safety issues, there was not complete agreement on the kind or level of

detail of information to be discussed at the flight readiness review. NASA

managers widely endorsed 7 of the 15 types of safety issues GAO asked

about as needing to be discussed in detail; however, opinions were divided

in other areas such as whether or not information about hazards and

waivers should always be briefed.

GAO compared how NASA addressed a problem in a solid rocket motor joint

that occurred in 1995 with its handling of the joint problem that caused the

1986 accident. Based on this comparison and observations, NASA was much

more open in dealing with the more recent problem. For example, shuttle

program managers were kept informed and involved in resolving the

problem and NASA held weekly press meetings to discuss its progress.

Funding Reductions,

Downsizing, and

Restructuring

Although NASA has made substantial improvement in the flow of

communications, some managers expressed concern about the impact of

funding reductions particularly with respect to staffing and organizational

Page 4 GAO/NSIAD-96-73 Space Shuttle
Executive Summary

restructuring. NASA must further reduce shuttle operation costs to meet

expected declining budgets. Because of this, NASA plans significant

changes in the way the shuttle program is managed.

Future funding reductions, downsizing, and program restructuring will be

challenging because the program also must maintain the capability to meet

the demanding international space station launch schedule. NASA must

reduce shuttle budgets by an additional $2.5 billion in fiscal years 1996

through 2000, while implementing a very compressed launch schedule.

To help meet the cost reduction and schedule challenges, NASA is planning

to turn shuttle operations over to a single prime contractor. The agency

will reduce its involvement in day-to-day operations but will retain

responsibility for launch decisions. Although not enough information is

available about the plan to assess all of its implications, some of the

shuttle program managers and safety representatives GAO interviewed

expressed concern about continued funding reductions and the transition

from the current way of doing business to the future management of the

program.

GAO’s work shows that NASA follows certain management principles in its

communications processes. These principles include priorities that place

safety above cost or schedule; an environment that encourages timely,

open debate; a culture that encourages people to elevate their safety

concerns; NASA and contractor working relationships that ensure agency

managers obtain continual knowledge of problems and issues; and an

organizational relationship that enables managers to carry out their

responsibility to certify readiness for flight. Survey respondents generally

agreed that these principles should be followed in the future.

NASA Still Relies Primarily

on Qualitative Risk

Assessments

NASA still relies primarily on its engineers’ judgment to assess and prioritize

significant shuttle program risks. It has made some use of quantitative

methods, such as probabilistic risk assessments, but has no overall

strategy on when these methods should be used in shuttle

decision-making. Past quantitative risk assessments have included

proof-of-concept studies, assessments of specific shuttle systems, and

assessments of accident probabilities for launches involving radioactive

material. NASA had a contractor develop a probabilistic risk assessment

model for use in the shuttle program but has not developed a plan for

incorporating this tool into its shuttle program management.

Page 5 GAO/NSIAD-96-73 Space Shuttle
Executive Summary

The contractor that developed the risk assessment model cited potentially

beneficial uses, such as establishing cost objectives for redesigning the

highest risk components. However, some shuttle managers told us that

NASA lacks an overall strategy and specific employee skills to efficiently

and effectively utilize methods such as probabilistic risk assessments.

Some officials stated there is a lack of trust in probabilistic risk

assessments because people do not understand the methodology.

Therefore, acceptance of this risk assessment method as a supplement to

existing qualitative methods is not NASA-wide, and there is much

skepticism about the cost and benefits of using probabilistic risk

assessments.

Data Not Always Available

in Readily Usable Form

NASA has developed two automated database systems to provide shuttle

data for use in decision-making—the Program Compliance Assurance and

Status System and the Problem Reporting and Corrective Action System.

However, some officials told GAO that information from these databases is

not always timely or reliable, and the systems are cumbersome to use.

The Program Compliance Assurance and Status System is based on older

technology, trend and other data are not centralized in the system, and

software needed to convert contractor data to NASA database format has

not been developed. A January 1995 internal study found important

information missing from thousands of entries. Officials also said that the

Problem Reporting and Corrective Action system records are often not

reliable and lack uniformity in categorizing problems.

Recommendations GAO recommends that the Administrator of NASA

• identify guiding principles of good risk management, such as those

described in this report, and ensure that terms and conditions of the

planned shuttle operations contract reflect these principles;

• take steps to ensure that flight readiness review participants understand

and agree on the minimum issues that should always be discussed at the

review and the level of detail that should be provided;

• establish a strategy, including specific milestones, for deciding whether

and how quantitative methods might be used as a supplemental tool to

assess shuttle risk; and

• assess the shuttle program’s centralized database, as well as other

databases, to insure that data required to conduct risk assessments and

inform decisionmakers, is accessible, timely, accurate, and complete.

Page 6 GAO/NSIAD-96-73 Space Shuttle
Executive Summary

Agency Comments In commenting on a draft of this report, NASA concurred with GAO’s four

recommendations and stated that the agency is already taking action to

implement them. GAO made additional changes to the report, where

appropriate, based on NASA’s technical comments. NASA’s comments are in

appendix IV.

Page 7 GAO/NSIAD-96-73 Space Shuttle
Contents

Executive Summary 2

Chapter 1 

Introduction

10

Risk of Space Flight 10

NASA’s Risk Management Program 11

Certifying the Shuttle for Flight 13

Objectives, Scope, and Methodology 16

Chapter 2 

NASA Has Improved

the Shuttle

Communications

Environment and

Needs to Sustain

Improvement in the

New Management

Environment

18

NASA Has Made Changes to Strengthen Shuttle Risk Management 18

NASA Has Improved the Communications Environment 19

Some Managers Concerned About Future Changes 27

Principles Guiding the Communications Process 32

Conclusions 33

Recommendations 34

Agency Comments 34

Chapter 3 

NASA Has Not

Developed an Overall

Strategy for Using

Quantitative Risk

Assessment Methods

in the Shuttle Program

36

National Research Council Recommended Quantitative Approach 36

NASA’s Response to the Council’s Recommendation 37

NASA’s Databases Do Not Provide Sufficient Information for

Quantitative Analysis

41

Conclusions 42

Recommendations 43

Agency Comments 43

Appendixes Appendix I: Methodology for Group Interviews and Survey

Assessing the Flow and Quality of Shuttle Safety-Related

Information

44

Appendix II: Survey Questions and Responses 47

Appendix III: Group Interview Questions 57

Appendix IV: Comments From the National Aeronautics and

Space Administration

61

Appendix V: Major Contributors to This Report 64

Page 8 GAO/NSIAD-96-73 Space Shuttle
Contents

Tables Table 1.1: Potential Effects of Failures of Shuttle Hardware

Components

12

Table 2.1: Differences in NASA’s Approach to the Challenger

Motor Joint Problem and the Recent Motor Joint Problem

26

Figures Figure 1.1: Flight Preparation Process 15

Figure 2.1: Characteristics of the Current Shuttle

Communications Environment

22

Figure 2.2: Safety and Mission Assurance Parallel Assessments 24

Figure 2.3: Respondents Views on Extent to Which Program

Priorities Place Greater Importance on Safety Than on Meeting

Schedule

29

Figure 2.4: Respondents Views on Extent to Which Program

Priorities Place Greater Importance on Safety Than on Cost

Reduction

30

Abbreviations

GAO General Accounting Office

NASA National Aeronautics and Space Administration

Page 9 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

The first space shuttle launch occurred on April 12, 1981. During the 25th

launch on January 28, 1986, the shuttle Challenger was destroyed shortly

after liftoff from Kennedy Space Center. Shuttle flights were suspended

while the accident was investigated by the Presidential Commission. The

shuttle returned to flight on September 29, 1988. Since that time, it has

flown successfully about 50 times.

The Presidential Commission determined that the 1986 accident was

caused by a faulty seal in one of the solid rocket motor joints. The

Commission also found other contributing causes to the accident, such as

management isolation, communications failures, and lack of a properly

staffed, supported, and robust safety organization.

According to the Commission’s June 6, 1986, report, the decision to launch

the Challenger was based on incomplete and sometimes misleading

information, a conflict between engineering data and management

judgments, and a the National Aeronautics and Space Administration

(NASA) management structure that permitted internal flight safety

problems to bypass key shuttle managers.1

 Officials who made the launch

decision were unaware of a recent history of problems with the defective

solid rocket motor joint and of the motor contractor’s initial

recommendation against launching. According to the Commission, if the

decisionmakers had known all of the facts, it is highly unlikely that they

would have decided to launch.

Risk of Space Flight Space flight can never be made risk free because it involves complex

hardware and software systems, harsh operating environments, and the

possibility of human error. A 1995 study by a NASA contractor, for example,

placed the median estimate of a catastrophic shuttle failure at 1 in 145

launches.

According to the advisory committee on the Future of the U. S. Space

Program, “there can be no acceptable objective among those who would

challenge the vastness of space other than perfection.”2

 Unfortunately, as

the Committee’s report points out, the objective of perfection is not readily

met, especially since space missions are fundamentally difficult and

demand undertakings that depend upon some of the world’s most

advanced technology and there are many opportunities for error.

1

Report of the Presidential Commission on the Space Shuttle Challenger Accident, June 6, 1986.

2

Report of the Advisory Committee on the Future of the U.S. Space Program, December 1990.

Page 10 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

The shuttle is an extremely complex system. The program employs

thousands of people and launching a shuttle requires that 1.2 million

separate procedures be accomplished correctly. Also, NASA has identified

over 5,000 critical system components whose failure, either singularly or in

combination, could cause loss of the vehicle or crew. Because these risks

cannot be completely eliminated, they must be identified and properly

managed.

NASA’s Risk

Management Program

NASA’s risk management policy requires that program and project

management communicate to NASA management and all program/project

personnel the significance of assessed risks and the decisions made with

respect to them. At NASA, risk management includes identifying the

primary risk drivers and estimating the likelihood of occurrence,

identifying the ensuing consequences, and determining the cost and

schedule impact.

NASA policy regarding safety is to

• avoid loss of life, injury of personnel, damage, and property loss;

• instill safety awareness in all NASA employees and contractors;

• assure that an organized and systematic approach is utilized to identify

safety hazards and that safety is fully considered from conception to

completion of all agency activities; and

• review and evaluate contractors’ and NASA’s plans, systems, and activities

related to establishing and meeting safety requirements to ensure that

desired objectives are effectively achieved.

Failure modes and effects analyses3

 are conducted for all flight hardware

elements and ground support equipment. This analysis starts with the

identification of all potential failure modes and evaluation of “worst case”

effects. NASA places potential effects of failures into the general categories

shown in table 1.1.

3

The failure mode and effects analysis is a systematic evaluation of each component of the shuttle

system to identify hardware items that are critical to the performance and safety of the vehicle and

mission. The evaluation includes identifying all system components, determining the potential modes

of failure for each component, and recommending corrective action. A critical items list is developed

as a result of the failure modes and effects analysis. The list includes all system components that could

cause loss of life, vehicle, or mission.

Page 11 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

Table 1.1: Potential Effects of Failures

of Shuttle Hardware Components Criticality Potential effect of failure

1 Single failure that could result in loss of life or vehicle.

1R Redundant item(s), all of which failed, could cause loss of life or

vehicle.

2 Single failure that could result in loss of mission.

2R Redundant item(s), all of which failed, could cause loss of

mission.

3 All others.

Hazard analyses4 are conducted to identify potential safety hazards and

means for minimizing the hazards. NASA’s actions to minimize hazards

follow the sequence of (1) system designs that minimize potential hazards,

(2) use of safety devices if the design does not eliminate a potential safety

hazard, (3) use of warning devices to alert the flight or ground crew to

potential hazards, and (4) use of special procedures.

Risk Assessment

Approaches

Approaches for assessing risk can be either quantitative or qualitative,

depending on whether statistical probabilities are assigned to a risk

element. All risk assessment approaches require experts to make

subjective judgments about the risk elements as well as the likelihood of

their occurrence.

Quantitative approaches, such as probabilistic risk assessments, can be

used to assess both the likelihood that an accident will occur (probability)

and the level of damage or loss that will result (consequences).

Quantitative assessment methods mathematically quantify risk on the

basis of engineering judgment, calculated probabilities of component

reliability, analysis of potential human failures, and whether they occur

singly or in combination. A probabilistic risk assessment, for example,

addresses three basic questions: (1) What could go wrong? (2) How likely

is it that this will happen? and (3) What are the consequences?

Qualitative assessments, on the other hand, assess risk through descriptive

information, identifying the nature and components of risk or an ordinal

scale, such as high, medium, and low. Qualitative ratings are usually based

on the judgments of experts after they consider such things as test and

operational experience, analytical results, trends, and other reported data.

4

Hazard analysis is to determine potential sources of danger that could develop while operating and

maintaining the system hardware and software. Hazard analysis also identifies the presence of other

potential risks caused by the environment, crew-machine interfaces, and mission activities.

Page 12 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

Certifying the Shuttle

for Flight

NASA follows a formal review process in certifying the shuttle for flight.

The certification of flight readiness process is a step-by-step activity

designed to certify the readiness of all components of the vehicle assembly

and all aspects of mission support.

The flight preparation process begins with project milestone reviews

including (1) element acceptance, (2) payload readiness, (3) software

readiness, and (4) project preflight readiness reviews. These reviews are

chaired by NASA project managers and the contractors formally certify the

flight readiness of the hardware and software.

The next step in the process is the program milestone reviews. These

reviews are held to assess the readiness for mating the external tank and

solid rocket booster, orbiter and external tank, and ferrying the orbiter

atop the shuttle carrier aircraft when required. These reviews are chaired

by the manager of launch integration and each shuttle element manager

certifies that it has satisfactorily completed the manufacture, assembly,

test, and checkout of the elements, including the contractor’s certification

that design and performance are up to standard.

The final step in the flight preparation process is the flight readiness

review. This review is held about 2 weeks prior to launch and is chaired by

the Associate Administrator for Space Flight.5,6 All shuttle elements,

safety and mission assurance, center directors, and senior representatives

from the major contractors participate in this review. At the end of the

flight readiness review, all organizations must certify that the mission is

ready for launch. The Associate Administrator for Safety and Mission

Assurance7

 is also an active participant.

The safety and mission assurance organization holds parallel reviews to

assess safety issues related to the planned launch. The safety and mission

assurance organization participates in all phases of the flight preparation

process.

5

The Associate Administrator for space Flight is responsible for providing leadership and programmatic

direction to accomplish the NASA human space flight program, including space shuttle, space station,

spacelab, cooperative U.S./Russian human space flight programs, and other related space flight

activities.

6

As a result of organizational changes announced subsequent to the completion of our review, the

Director of the Johnson Space Center will chair future flight readiness reviews.

7

The Associate Administrator for Safety and Mission Assurance is responsible for providing leadership,

policy direction, functional management, and coordination for the safety, reliability, maintainability,

and quality assurance for all NASA programs.

Page 13 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

Two days before a scheduled launch, a mission management team holds a

review to assess flight readiness. Its agenda includes close out of any open

work, close out of any flight readiness review action items, discussion of

new or continuing anomalies,8

 and an updated briefing on anticipated

weather conditions at the launch site and at abort landing sites in different

parts of the world. The mission management team meets every day after

the launch –2 day review up to the conclusion of the mission. Figure 1.1

illustrates NASA’s flight preparation process.

8

Anomalies are unexpected events; hardware or software damage; a departure from established

procedures or performance; or a deviation of system, subsystem, and/or hardware or software

performance outside certified design/performance specification limits.

Page 14 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

Figure 1.1: Flight Preparation Process

Element


acceptance


review

Element


acceptance


review

Element


acceptance


review

Element


acceptance


review

Payload


readiness


review

External tank/solid


rocket booster


mate review

 Orbiter roll-out/external


tank mate


review

Software


readiness


review

Flight readiness


review

Orbiter ferry review


(as required)

Program/project


pre-flight readiness


reviews

Element


acceptance


review

Element


acceptance


review Element


acceptance


review

Space shuttle


main engine

Orbiter

External tank

Reusable solid


rocket motor


and solid


rocket booster

Payloads

Horizontal payloads

Vertical payloads

NASA’s safety organization provides an independent channel for assessing

shuttle flight safety. Each center’s safety organization participates in the

element acceptance reviews as well as the flight readiness review and the

mission management team. Participation in these reviews provides the

opportunity for NASA’s safety organization to express any residual concerns

about the safety of an upcoming mission. The organization also holds

Page 15 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

independent prelaunch assessment reviews. In addition, the Associate

Administrator for Safety and Mission Assurance attends the flight

readiness review and has a direct communications link to the NASA

Administrator.

Other program briefings and reviews are also a part of the certification of

flight readiness process. For example, the program manager holds an early

morning telephone conference with the shuttle centers and headquarters

each day to discuss the status of progress and problems. Likewise, about

midday the working level shuttle managers hold a telephone conference to

provide updated information. Safety and mission assurance personnel

attend all of the shuttle program and project meetings and contribute their

independent views.

Objectives, Scope,

and Methodology

The former Chairman, Subcommittee on Investigations and Oversight,

House Committee on Science, Space, and Technology, asked us to review

NASA’s management of risk associated with flying the space shuttle.

Specifically, we reviewed the actions NASA has taken to improve the free

flow of information in the launch decision process and the progress NASA

has made in adopting quantitative methods for assessing risk.

To assess the communications environment, we reviewed policies,

procedures, and practices related to management of the shuttle program

used by the agency in making launch decisions; we observed various

shuttle processing reviews, including a shuttle launch; and discussed

various aspects of the program with those responsible for its management.

We also conducted discussions of these topics with groups of shuttle and

safety managers at NASA Headquarters, and the Johnson, Marshall, and

Kennedy field centers. Together these individuals represented almost all of

the top NASA officials responsible for shuttle launch decisions and

management of most shuttle manufacturing and processing work. To

understand the flow of risk information within shuttle contractor

organizations and between NASA and its shuttle contractors, we also held

discussions with groups of program and safety managers and

working-level engineers at three of NASA’s prime shuttle contractors. We

chose the three contractors because the work is among the more complex

and highest risk in the program.

Group discussions are very useful for exploring the various facets of

communications issues and processes. However, they did not enable us to

Page 16 GAO/NSIAD-96-73 Space Shuttle
Chapter 1 

Introduction

determine how many participants held a particular view or the intensity of

their views. Therefore, to more precisely measure the themes that

emerged from the group discussions, we sent a structured questionnaire to

the NASA interview participants and some safety representatives who did

not participate in the group interviews. Appendixes I through III contain a

more detailed discussion of our group interview and survey methodology.

To evaluate NASA’s use of quantitative risk assessment methodologies, we

reviewed policies, procedures, and practices related to NASA’s shuttle risk

management program and held discussions with senior shuttle managers

and NASA’s safety and mission assurance organization. We also discussed

the use of quantitative risk assessment methodologies with other federal

agencies that are responsible for managing complex systems to establish a

benchmark for the use of such methods within the federal government.

This work included the Nuclear Regulatory Commission and the Federal

Aviation Administration. We also obtained information on the

Environmental Protection Agency’s use of quantitative risk assessment in

the management of superfund cleanup sites. In addition, we consulted

outside experts to obtain their views on the usefulness of quantitative risk

assessments to NASA.

We conducted our review primarily at NASA Headquarters, Washington,

D.C.; Marshall Space Flight Center, Alabama; Johnson Space Center,

Texas; Kennedy Space Center, Florida; Thiokol Corporation, Ogden, Utah;

and Rocketdyne Division of Rockwell International, Canoga Park,

California.

We conducted our review between June 1994 and December 1995 in

accordance with generally accepted government auditing standards.

Page 17 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Good communications is one of the keys to effective risk management.

Without adequate information about risks, launch decisions may be flawed

as they were in the case of the Challenger accident. Interviews with key

shuttle program officials, survey data, and our observations indicate that

NASA has been successful in creating communication channels and an

organizational culture that encourages people to discuss safety concerns

and to bring those concerns to higher management if necessary.

NASA has announced plans to make fundamental changes in the way it

manages the shuttle program—turning day-to-day management over to a

single prime contractor and reducing direct NASA involvement. Some

managers expressed concern about the potential impact of this change,

particularly with respect to staffing and organizational restructuring.

NASA’s challenge will be to ensure adherence to the communications

principles that are essential to promoting shuttle safety.

NASA Has Made

Changes to

Strengthen Shuttle

Risk Management

According to the Presidential Commission, prior to the Challenger

accident, project managers for the various elements of the shuttle program

felt more accountable to their center management than to the shuttle

program organization. As a result, vital program information frequently

bypassed the program manager, who was located at the Johnson Space

Center. The Commission recommended that NASA give the program

manager authority over all program funding and work. In response, NASA

centralized program management in a shuttle program director at

headquarters with overall responsibility for shuttle operations and

budgets. Also, the program manager at the Johnson Space Center was

made a headquarters employee in order to minimize center-to-center

communications problems. Effective January 31, 1996, however, shuttle

program management responsibility was transferred from the

headquarters director to the Johnson Space Center director. Because NASA

has not yet prepared a detailed plan for implementing this change, we

could not fully evaluate its implications. However, according to NASA

officials in the Office of Human Space Flight, the Johnson Center director

will have full authority over the shuttle resources and work at all

participating centers and will report directly to the NASA administrator.

NASA has also given astronauts a role in certifying the shuttle for launch

and encouraged them to move into shuttle management positions, as

recommended by the Presidential Commission.

Page 18 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

NASA also established the Headquarters Office of Safety and Mission

Assurance1

 under the direction of an associate administrator reporting

directly to the NASA administrator. The agency strengthened the safety

organizations at its shuttle field centers so that each director of safety and

mission assurance reports to a center director rather than the engineering

organization. NASA also increased the number of people assigned to the

safety organization. In addition, NASA established a safety reporting system

to provide an avenue for NASA and contractor personnel to confidentially

report problems to safety and program management officials that could

result in loss of life or mission capability, injury, or property damage.

NASA Has Improved

the Communications

Environment

Participants in our discussion groups—both within NASA and in the

contractor organizations—described a communication environment that is

more open than the one that existed at the time of the accident.

Respondents in our follow-up survey portrayed the culture as encouraging

contractors and employees to discuss and, if necessary, elevate safety

concerns. Discussion groups also identified multiple channels, both formal

and informal, for communicating flight safety information. In some cases,

these communication channels represent independent, parallel paths for

assessing risk. Our own observations and analysis of NASA’s approach to

dealing with a recent problem illustrated the openness with which agency

officials address safety issues.

Current Culture

Encourages Open

Discussion of Safety Issues

In group discussions with key NASA and contractor shuttle managers and

contractor working-level engineers, we asked them to assess conditions

related to the flow of safety information to top management. All of the

groups reported that the shuttle program’s organizational culture

encourages people to discuss safety concerns and bring concerns to

higher management if they believe the issues were not adequately

addressed at lower levels. As one manager noted, because of the

complexity of the shuttle program, open communication, group

discussions, and the sharing of information are essential to flight and work

place safety.

NASA managers at the three field centers with primary responsibility for

managing shuttle elements and at NASA headquarters reported having taken

steps to create an organizational environment that encourages personnel

at all levels to voice their views on safety to management. One manager

1

When established, this organization was the Office of Safety, Reliability, Maintainability, and Quality

Assurance.

Page 19 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

noted that people are not afraid to surface their mistakes to management

when they discover mistakes have occurred. Another manager said, “If . . .

I got the idea that I had a manager in the system who wasn’t allowing their

people to feel comfortable in bringing [up] things, probably that’s the time

I think I would change that person’s job because . . . our people need to

feel that they can come without attribution and talk about what they need

to talk about.”

Managers in each group we interviewed cited various techniques they use

to create an organizational environment that encourages personnel at all

levels to voice their professional viewpoints on safety issues to

management, even if dissenting. For example, managers invite people to

express their concerns by

• trying to keep every line of communication open and telling people that

bringing up a problem does not reflect poor performance;

• holding extensive dialogue over shuttle safety issues, beginning early in

the problem identification stage, so that everyone fully understands the

issues;

• encouraging people to come in or call their managers if they want to talk

about a safety concern, no matter how small the issue; and

• not only encouraging, but expecting, open expression of professional

differences at all levels.

The contractor managers also described a working relationship with NASA

that they believe encourages open communication and the elevation of

safety concerns. They described the flow of information between NASA and

shuttle contractors as continual, open, and comprehensive. From their

perspective, daily contact between contractor and NASA working-level

personnel contributes to the exchange of information. Contractor support

to and participation in flight readiness reviews and other shuttle

processing meetings, and their reporting of safety information directly into

NASA’s centralized information systems are among the other mechanisms

that achieve that exchange.

One manager noted that the Challenger accident prompted a change in his

contractor’s management approach. Before the accident, company

meetings were closed to the NASA site representatives. Since the accident,

NASA representatives attend all technical meetings. Managers from two

other contractors said that they would not hesitate to go to the highest

levels of NASA management to ensure that safety issues received

appropriate attention.

Page 20 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Contractor working-level engineers portrayed their organizations as

supportive of engineers elevating shuttle safety issues and concerns to

management. For example, at one contractor facility, program teams are

structured so that minority opinions about the handling of safety problems

can be elevated to a higher level board. At another contractor facility, the

work environment was described as one that encourages debate,

discussion, and never keeping a safety concern quiet. At the third

contractor plant, the formal reporting process ensures that NASA and

contractor managers are continually apprised of issues, review how issues

are resolved, and can request more work if they do not agree with the

resolution of a safety issue.

The managers and safety representatives who responded to our survey

also gave very favorable ratings to NASA’s current communications culture.

For example, 90 percent of those responding to the survey said that to a

great or very great extent NASA’s organizational culture encourages civil

service employees to discuss safety concerns with management.

As shown in figure 2.1, more than 80 percent of the respondents to our

survey rated the following current shuttle communications and

information flow conditions very favorably.

Page 21 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Figure 2.1: Characteristics of the Current Shuttle Communications Environment

0 20 40 60 80 100

Percentage of total respondents

Culture encourages employees 

to elevate concerns if issues

not adequately addressed

Culture encourages contractors 

to raise safety issues

Sufficient independent 

assessment and/or review 

functions

Culture encourages employees 

to discuss safety concerns

Note: The chart presents the percentage of 39 respondents rating each characteristic as present

to a great or very great extent.

As part of our review, we attended numerous certification of flight

readiness and prelaunch assessment reviews for shuttle mission STS-64,

including the flight readiness review and launch. We observed open and

candid discussions, debate of issues, and a structure that required the

recording and follow-up of unresolved issues. At most reviews,

presentations appeared thorough and participants asked many probing

questions to ensure they had an adequate understanding of the issues

being briefed. If participants did not believe they adequately

Page 22 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

understood an issue or additional work was required to resolve an issue, it

was listed as an open item to be resolved prior to launch.

Numerous

Communications Paths Are

Available

Managers, safety personnel, and working-level engineers described shuttle

program and contractor procedures and structures that provide multiple

avenues for continual communication with contractors, across centers,

and with headquarters to discuss safety issues. These avenues include the

certification of flight readiness process, daily telephone conferences, and

weekly meetings. In response to our survey, almost all NASA program

managers and safety representatives believe the opportunities to discuss

and communicate shuttle issues and concerns meet, or even exceed, the

needs of the program in terms of the number of forums held and the types

and levels of expertise represented.

The certification of flight readiness process requires the involvement of all

centers and projects on issues that could affect safety or mission success.

In preparation for a launch, NASA relies on a number of reviews to ensure

that the shuttle is safe for flight. These reviews are designed to ensure

compliance with requirements, that prior problems/failures have been

corrected, planned work has been completed, and operational support is

in place for the mission.

Managers also reported other, sometimes less formal, channels for

communicating safety information. For example, the shuttle program

manager holds an early morning telephone conference daily, enabling NASA

managers at headquarters and the centers to discuss problems and draw

upon the experience of others. The manager of launch integration also

conduct a daily “noon board” telephone conference to discuss shuttle

issues, status, and required changes related to vehicle processing at the

Kennedy Space Center. Project representatives from the various shuttle

centers participate if the issue involves their shuttle element. Also, NASA’s

shuttle program manager chairs a weekly Program Requirements Control

Board meeting that is the controlling authority for all changes to the

shuttle program baseline. Safety and mission assurance engineers

participate in all of these meetings. Further, NASA safety and project

representatives at contractor plants help ensure a continual flow of

information on contractor issues. In addition, the NASA Safety Reporting

System (an anonymous reporting system) provides another opportunity

for people to report safety concerns.

In addition to taking part in all of the program and project reviews for the

certification of flight readiness, NASA’s Office of Safety and Mission

Page 23 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Assurance conducts prelaunch assessment reviews of all major shuttle

elements. The office’s System Safety Review Panel also conducts several

reviews, including a review of in-flight anomalies from previous missions.

These safety office reviews are conducted independently of the project

offices responsible for the various shuttle elements. Results of the safety

office reviews are presented at the flight readiness review. The safety

organization continues to monitor shuttle missions up to and during

launch. Figure 2.2 illustrates the parallel assessments by safety and

mission assurance and the shuttle program and project offices.

Figure 2.2: Safety and Mission Assurance Parallel Assessments

Safety and Mission


Assurance Prelaunch


Assessment


Reviews

System Safety


Review Panel


Review of 


Previous 


Mission In- 


Flight 


Anomalies

Safety and


Mission


Assurance


Prelaunch


Assessment


Review with


Headquarters

Safety and 


Mission


Assurance


Prelaunch


Assessment


Review Update


with 


Headquarters

Safety and 


Mission


Assurance


Prelaunch


Assessment


Review Update


with 


Headquarters

System Safety


Review Panel


Hazard Report


Review

System Safety


Review Panel


Integrated


Cargo


Hazard 


Analysis

System Safety


Review Panel


for the


Launch Minus


2-Day Review

Mate


Reviews


Flight


Readiness


Review

Launch 


Minus 2-Day 


Review


with Mission


Management


Team

Safety and Mission 


Assurance 


Independent 


Readiness 


Assessments

Flight Preparation


Process Contractor
 Launch


Element


Acceptance


Reviews

Page 24 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

We asked contractor working-level engineers what avenues are open to

them to communicate their views in the event that they disagree with a

safety decision made at higher levels of management, either within their

organization or within NASA. A variety of communication routes were cited:

a company ombudsman, the firm’s safety manager, NASA counterparts, or

higher levels of management within the contractor’s organization and the

NASA Safety Reporting System.

Not Complete Agreement

on Type and Amount of

Information Needed

While there was a high level of agreement that the current culture

encourages and enables contractors and employees to discuss safety

issues and concerns, there was not universal agreement about the kinds of

risk information needed for final launch decisions. We asked NASA

managers and safety representatives to designate the types of safety issues

that should always be briefed in detail to corporate-level management at

the final flight readiness review. Seven of the 15 types of issues we asked

about were widely endorsed as needing the board’s review; however,

opinions were divided in other areas. For example, the views of the board

members tended to differ from those of the other managers and safety

representatives regarding whether hazards and new waivers should always

be briefed in detail. Opinions were also divided about the level of detail

that should be provided when there are changes that affect procedures or

processes involving the flight crew, operations, software, or shuttle

hardware.

We also observed differences in the amount of detail provided during two

flight readiness reviews. At the first review, we observed that the review

board’s chairman required less detail about issues and concerns than at

the second review. The second review meeting we observed was chaired

by a different official. This official requested a greater level of detail about

issues being discussed. Thus, the change in personnel caused some initial

confusion about the type and amount of information needed to make

corporate-level launch decisions.

NASA Demonstrated

Openness in Dealing With

Recent Motor Joint Issue

To provide a better understanding of the cultural and communication path

changes within NASA, we compared NASA’s approach to handling the motor

joint issue at the time of Challenger with a recent issue concerning

another joint in the solid rocket motor. On two successive flights in 1995,

hot gas penetrated beyond the joint’s sealer compound and made very

small singe marks on the joint’s primary o-ring. NASA was more cautious in

its approach to handling the latest motor joint problem. For example, NASA

Page 25 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

immediately halted shuttle launches and publicly aired the problem. NASA

held weekly press meetings to discuss the problem and progress in

correcting it. Shuttle and contractor managers at all organizational levels

were heavily involved in the issue and the safety organization provided an

independent assessment of the problem. NASA did not resume shuttle

launches until it was confident that the problem was understood and

corrected. Table 2.1 describes our observations.

Table 2.1: Differences in NASA’s

Approach to the Challenger Motor

Joint Problem and the Recent Motor

Joint Problem

Challenger problem Recent problem

Nature of the problem

Design flaw. NASA did not make timely

attempt to develop and verify a new seal

after the initial design was shown to be

defective.

Process problem. Process enhancements

were initiated as soon as gas paths were

detected. Improvements are continuing.

Hot gas penetrated past primary o-ring to

secondary o-ring on STS 51-C (Jan. 24,

1985) prior to Challenger.

Hot gas made very small singe marks on

primary o-ring on two successive flights.

Information flow

All program managers were not informed of

problem prior to the launch of Challenger.

All program managers were heavily

involved in the issue.

NASA solid rocket motor manager waived

constraint to launch for six consecutive

launches prior to Challenger. He was

required to notify higher levels but did not.

All levels were aware of and understood

the problem. NASA stopped shuttle

launches until the anomaly was resolved

and repairs made.

Differing views at lower levels of

management were not raised to the

appropriate levels.

Relevant information was raised to

appropriate levels of contractor and NASA

management. NASA and the hardware

contractor reached a consensus on cause

and corrective actions.

Top decisionmakers were not aware of all

the facts; so, flight was allowed to proceed.

Top decisionmakers were fully informed,

understood the facts, and stopped shuttle

flights.

NASA culture

Culture not conducive to airing problems. Organizations and management

encourage people to elevate problems

and concerns.

Inadequate public airing of problems. Held weekly press meetings to discuss

problems and progress.

Safety organization oversight

Safety and Mission Assurance reporting

channels varied among centers.

Safety and mission assurance

organizations are independent of

engineering and project management

throughout NASA.

No formal process to facilitate confidential

reporting of safety concerns.

NASA Safety Reporting System allows

confidential reporting of safety concerns.

(continued)

Page 26 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Challenger problem Recent problem

Little or no trend analysis was performed on

motor joint and hot gas blow-by problems.

Engineering organizations compile and,

along with safety and mission assurance,

evaluate trend data.

Failure mode and effects analysis/critical

items list and hazard analyses were minimal.

Indepth failure mode and effects

analysis/critical items list and hazard

analyses were used during the

investigation and repair planning.

Postflight inspections included case and

case to nozzle joints and seals, but only

limited nozzle inspection. Nozzle joints were

not inspected. Delayed reporting of

inspections results.

Indepth case and nozzle postflight

inspection performed, including all joints.

Formal and timely reporting of all identified

discrepancies.

Some Managers

Concerned About

Future Changes

Some discussion group participants told us they are concerned about the

impacts of continued cost reductions and planned program changes. Over

the next 5 years, plans call for NASA to make significant additional

reductions in shuttle costs while maintaining the capability to meet the

demanding schedule for international space station assembly and support.

Although final decisions have not been made, NASA has initiated a number

of actions to further reduce shuttle operation costs, including turning

shuttle operations over to a single prime contractor. Some participants in

our discussion groups expressed concern about the effect of continued

cost reductions and the transition to contractor management of the

program.

In July 1995, we reported on the schedule pressures created by the

International Space Station assembly requirements.2

 Based on our own

analysis and internal NASA studies, we concluded that the shuttle’s ability

to meet station launch requirements appeared questionable.3

 To meet the

station’s “assembly complete” milestone, shuttle officials had designed a

very compressed launch schedule. During certain periods of the station

assembly, clusters of shuttle flights are scheduled to be launched within

very short time frames. For example, the schedule calls for five launches

within a 6-month period in fiscal year 2000 and seven launches during a

9-month period in fiscal year 2002. Because the schedule is so compressed

at times, there is very little margin for error. There is little flexibility in the

2

Space Shuttle: Declining Budget and Tight Schedule Could Jeopardize Space Station Support

(GAO/NSIAD-95-171, July 28, 1995).

3

In commenting on a draft of the aforementioned report on June 23, 1995, NASA stated that “Although

our space station assembly schedule is demanding and funding is tight, we are currently on schedule

and within budget. We are committed to achieving the space shuttle enhancements and launches

required to assemble a productive station on time for ourselves and our international partners.”

Page 27 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

schedule to meet major contingencies, such as late delivery of station

hardware, or technical problems with the orbiter.

We reported in June 19954

 that NASA had reduced shuttle operations

funding requirements by a cumulative amount of $2.9 billion between

fiscal years 1992 and 1995 when the fiscal year 1992 budget request is

compared to the fiscal year 1995 request. In our survey, we asked NASA

managers and safety representatives what actions had been taken to

accommodate the funding reductions and whether these actions, in their

opinion, had enhanced, degraded, or had little or no effect on the

accuracy, completeness, and timeliness of shuttle safety-related

information. Generally, their assessment was that the actions either had

little or no effect on quality or somewhat degraded quality. For example,

of nine respondents who reported funding reductions accomplished by

delaying safety improvements, six said the delay somewhat degraded the

quality of safety-related information. However, some respondents reported

actions taken to cut costs actually enhanced the quality of information.

Just over 75 percent of NASA managers and safety representatives we

surveyed believed that NASA emphasized safety over shuttle schedule to a

great or very great extent. Figure 2.3 illustrates NASA managers and safety

representative responses to our survey question on the extent to which

program priorities place greater importance on safety than on meeting

schedule.

4

Space Shuttle: NASA Must Reduce Costs Further to Operate Within Future Projected Funds

(GAO/NSIAD-95-118, June 15, 1995).

Page 28 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Figure 2.3: Respondents Views on

Extent to Which Program Priorities

Place Greater Importance on Safety

Than on Meeting Schedule

49% • Very great extent

28%•

Great extent

•10%

Moderate extent

•13%

Some extent

Note: The chart presents the responses of 39 NASA shuttle managers and safety officials. No

respondents designated “little or no extent.”

Just over 60 percent of NASA managers and safety representatives we

surveyed believe that to a great or very great extent NASA emphasizes

safety over reducing cost. Figure 2.4 illustrates responses to our survey

question on the extent to which program priorities place greater

importance on safety than on cost reduction.

Page 29 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Figure 2.4: Respondents Views on

Extent to Which Program Priorities

Place Greater Importance on Safety

Than on Cost Reduction

31% • Very great extent

31% • Great extent

23%•

Moderate extent

•13%

Some extent

•

3%

Little or no extent

Note: Percents do not add to 100 due to rounding. The chart presents the responses of 39 NASA

shuttle managers and safety officials.

Contractor managers and working-level engineers also told us that past

funding reductions had not affected the quality of safety-related

information they develop. According to the contractor managers,

reductions in the shuttle flight rate and various contractor productivity

enhancements have enabled them to accommodate past personnel cuts

without, they believe, sacrificing the quality of shuttle information they

develop.

Some working-level engineers in the group interviews cited a variety of

concerns about the effects of funding reductions. For example, the

engineers said (1) investigations of lower priority issues take longer to

complete because there is not enough time to devote to them, (2) keeping

people with the required skill level is a concern, and (3) there is a lack of

storage in automated databases to archive safety information. In addition,

some engineers told us that the funding reductions have adversely

Page 30 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

impacted employee morale because people are being asked to accomplish

more with fewer resources and some employees fear losing their job.

Some engineers said, however, that although morale was lower, they did

not believe it adversely affected flight safety.

In November 1995, the Associate Administrator for Space Flight testified

that NASA plans an additional $2.5 billion cumulative reduction from total

shuttle funding requirements in fiscal years 1996 through 2000 against the

fiscal year 1996 budget request.5

 According to the Associate Administrator,

the program will achieve the budget reductions through restructuring and

other workforce and content reductions.

Both NASA and contractor managers in our discussion groups expressed

concerns about how they would cope with additional funding cuts. For

example, the project managers for two contractors said that workforce

reductions can impact their timeliness in responding to situations that

arise. One contractor manager noted that while the company measures

various indexes such as “first time quality” and overtime, it is difficult to

specify the point at which additional program changes to accommodate

funding cuts might reduce quality. Another contractor manager noted that

at some point, funding reductions could translate into not having enough

people, so that maintaining the required quality will mean continual

schedule delays—a signal to the contractor that their program cannot be

reduced further.

Although firm estimates are not available, NASA expects to achieve

significant cost savings by turning shuttle operations over to a prime

contractor. The contractor would be responsible for shuttle processing

and launch, but NASA will retain the responsibility for making the final

launch decision. The single prime contractor would combine many of the

tasks now performed under 28 separate shuttle program contracts. Savings

are expected to accrue because shuttle operations would be more efficient

and require fewer civil service employees. Current plans are to award the

contract by fiscal year 1997.

During our discussion groups, some NASA managers expressed concern

about the transition of shuttle operations to a single prime contractor.

They feel that over the years NASA has assembled an expert shuttle

operations team and there are many unknowns about making a transition

to a new way of doing business. For example, the safety and mission

5

Statement of Dr. J. Wayne Littles, Associate Administrator for Space Flight, National Aeronautics and

Space Administration, before the Subcommittee on Space and Aeronautics, Committee on Science,

House of Representatives, Nov. 9, 1995.

Page 31 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

assurance organization maintains independent oversight of shuttle

operations. NASA’s projections are that the quality assurance oversight role

will be reduced under the single prime contractor concept of operations.

Although managers expressed concern about transitioning to a single

operations contractor, in response to our survey, 76 percent of the

managers and safety representatives said that quality assurance

inspections and reviews should be decreased. According to NASA, there will

continue to be independent oversight and the agency has plans to assure

that the oversight/insight will be properly focused with the reduced level

of resources expected.

NASA will retain decision authority and direct oversight over work that is

considered out-of-family (those events/activities that may contain a level

of risk beyond the known and accepted level). In addition, NASA will retain

the developmental effort for new hardware. This work will transition to

the single prime contractor, but only after all the unknowns are

understood by NASA. Further, NASA will return to an oversight mode when

there is an indication that there is an increase in the understood level of

risk for any reason. The single prime contractor will be required to

propose a process for performing risk assessment and to demonstrate that

they are able to institute and properly manage the process. This includes

the process for keeping NASA informed of issues that have the potential for

increasing risk.

Principles Guiding the

Communications

Process

Through our discussion groups, individual interviews, and observations,

we identified several management principles related to communication

and information flow that appear to guide shuttle communications. We

also identified additional management principles that we believe are

essential to promoting shuttle safety in the future. In our survey, we listed

these principles and asked NASA managers and safety representatives to

identify those guiding principles that they believe are essential to

promoting shuttle program safety as NASA deals with budget constraints,

associated downsizing, and restructuring in the near term, and with

continuation of shuttle flights in the long term. A large percentage of

managers and safety representatives we surveyed agreed that the

following principles are essential to promoting shuttle safety.

• The organizational environment and structures for both contractor and

NASA personnel encourage timely, open discussion and debate to ensure

managers have the benefit of all relevant knowledge of shuttle program

issues.

Page 32 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

• Managers (civil service and contractors) stress safety over schedule and

cost and those managers foster these values among employees.

• The organizational environment encourages people (civil service and

contractor) to elevate concerns to higher management if they believe the

issues were not adequately addressed at lower levels.

• The working arrangement between NASA and contractors ensures agency

managers obtain continual knowledge of problems and issues so that

appropriate decisions can be made.

• Organizational mechanisms enable NASA corporate-level managers to carry

out their decision-making responsibilities for certifying readiness for

flight.

• NASA uses the most appropriate analytic and quantitative methods available

to assess shuttle risks and conduct sufficient assessments and reviews to

carry out the agency’s oversight of shuttle work processes.

• Management information systems, including databases, are accessible,

accurate, complete, and timely for shuttle program oversight and

decision-making.

• The NASA environment is a self-evaluative one that monitors its

effectiveness in communication and information flow and seeks ways to

improve it.

In addition to the principles previously listed, some NASA managers

provided additional principles that they believe are essential to promoting

shuttle safety as NASA deals with budget constraints, downsizing, and

restructuring.

• Management of changes in the program receives adequate attention and

time to ensure that (1) program priorities are adhered to, (2) government

and contractor responsibilities for the reporting and resolution of

safety-related issues are clearly defined, and (3) changes to the shuttle

program are appropriately evaluated before implementation.

• Appropriate training is conducted to ensure that personnel can effectively

and efficiently carry out their work when changes in program operations,

processes, and staffing occur.

• Morale and the working environment of employees are considered key

elements in assuring a safe and quality program.

• Prime contractor management methods ensure quality of subcontractor

work.

Conclusions NASA has created an organizational culture that encourages shuttle

program and contractor employees at all levels to bring safety concerns to

Page 33 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

the attention of NASA’s top management. NASA has also established policies

and procedures to ensure the free flow of needed safety-related

information. However, in response to our survey, some shuttle program

personnel expressed concern about whether NASA might be emphasizing

cost reductions over flight safety as planned budget reductions and

operational changes occur. Also, in response to our survey, several types

of issues were endorsed as always needing the flight readiness review

board’s attention. However, opinions were divided in other areas,

suggesting that managers and safety representatives may not be clear on

each other’s expectations about the issues that should always be briefed.

If, as is likely, the planned shuttle operations contractor assumes more of

the burden of providing information to the flight readiness review, it will

be important to clearly specify the type and level of detail of information

to be provided.

NASA has adopted certain management principles that help guide the

shuttle launch decision process. These include such steps as stressing

safety over schedule and cost and developing an organizational culture

that encourages both contractor and NASA personnel to elevate concerns to

higher management if they believe the issues were not adequately

addressed at lower levels.

Recommendations We recommend that the Administrator of NASA identify guiding principles

of good risk management, such as those contained in this chapter, and

ensure that terms and conditions of the planned shuttle operations

contract reflect these principles.

We also recommend that the Administrator take steps to ensure that flight

readiness review participants understand and agree on the minimum

issues that should always be discussed at the review and the level of detail

that should be provided.

Agency Comments In commenting on a draft of this report, NASA agreed with our first

recommendation and stated that the agency is taking steps to implement

it. According to NASA, the shuttle flight operations contract request for

proposal and statement of work have been carefully reviewed and these

documents reflect the principles of good risk management described in

this report. NASA said that it will ensure that the contract terms and

conditions are compatible with these principles.

Page 34 GAO/NSIAD-96-73 Space Shuttle
Chapter 2 

NASA Has Improved the Shuttle

Communications Environment and Needs to

Sustain Improvement in the New

Management Environment

Regarding the second recommendation, NASA said that it is appropriate and

the agency has recently completed an activity to update and clarify the

roles and responsibilities of each program element and organization

relative to the flight readiness review. The new procedure is to be fully

implemented in support of shuttle flight STS-78 in June 1996.

We made additional changes to the report, where appropriate, based on

NASA’s technical comments.

Page 35 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

The National Research Council recommended in 1988 that NASA apply

quantitative risk assessments to the shuttle program. However, NASA still

relies primarily on qualitative methods to assess and prioritize significant

shuttle risk. This approach relies heavily on the judgment of shuttle

engineers to identify significant risk items that could cause loss of a

shuttle or crew. Although NASA awarded a contract for development of a

quantitative method model known as a probabilistic risk assessment1

 for

the shuttle program, NASA has not fully assessed the potential benefits of

using the tool in routine shuttle decision-making. The agency also has not

developed an overall strategy for assuring use of this method where it is

appropriate. In addition, databases are not always timely, complete,

accessible, or reliable enough to be used in these type analyses.

National Research

Council

Recommended

Quantitative

Approach

The National Research Council investigation of NASA’s risk assessment

approach following the Challenger accident found that quantitative

assessment methods had not been used to directly support NASA

decision-making related to the space shuttle. The Council recommended

that probabilistic risk assessment approaches be applied to the shuttle at

the earliest possible date. They also recommended that databases be

expanded to support probabilistic risk assessments, trend analysis, and

other quantitative analysis and that NASA develop a statistical sciences

capability to perform necessary risk assessments.

Quantitative Methods Used

in Other High-Risk Areas

Quantitative methods, such as probabilistic risk assessments, have been

used in the decision-making process by other federal agencies involved in

high-risk ventures. For example, the Nuclear Regulatory Commission uses

probabilistic assessments in its regulation and oversight of nuclear power

plants. These techniques are used to assess the safety of operating reactor

events and as an integral part of the design certification review process for

advanced reactor designs. Commission officials stated they have found

probabilistic risk assessments to be an effective tool for making

plant-by-plant examinations to determine areas needing more emphasis,

such as how long it takes a utility to respond to problems. Commission

officials told us that, in their experience, probabilistic risk assessments

can help identify and focus their attention on risk areas that require the

most resources.

1

Probabilistic risk assessment is a systematic methodology for evaluating the probability that an event

will occur and predicting the consequences should the event occur.

Page 36 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

The Environmental Protection Agency uses quantitative risk assessments

to determine the health risks posed by superfund hazardous waste sites.

The agency reviews contaminated sites for investigation and cleanup. One

element of the investigation is a baseline risk assessment—an evaluation

of current or potential threat to human health. The evaluation establishes

probabilities that are used to decide whether a site requires cleanup. For

example, if the risk of humans developing cancer from site chemicals is

greater than 1 in 10,000, Environmental Protection Agency policy requires

that the site be cleaned.

NASA pointed out that it is important to make a clear distinction between

quantitative risk assessments in general and the specific probabilistic risk

assessment method when determining the value of applying these methods

to space hardware issues. NASA said it recognized that probabilistic risk

assessments had proven valuable at the Nuclear Regulatory Commission

and the Environmental Protection Agency. However, this method did not

have comparable utility at NASA. Reactor design and certification risk

assessments are based on failure rates compiled from hundreds of plants

and facilities while the shuttle has significantly less hard data available to

quantify risk. In addition, NASA said the public health risk posed by nuclear

power plant accidents or toxic waste sites argues for a multimillion dollar

investment in risk assessment that can span years of analysis. In contrast,

according to NASA most shuttle risk issues must be resolved in a shorter

time frame.

NASA’s Response to

the Council’s

Recommendation

In response to the Council’s interim report, NASA began taking tentative

steps toward the use of probabilistic analysis by initiating contractor trial

probabilistic risk assessments of some shuttle elements. In parallel with

this, NASA began developing a procedure to prioritize the shuttle’s highest

risk elements. This proposed technique would lend itself to the

incorporation of quantitative measures of risk and probabilities of

occurrence as these measures were developed. NASA planned to assess the

benefits and applicability of this method to the shuttle risk management

process based on the results of the contractor studies. A former Associate

Administrator for Safety and Mission Assurance indicated that he would

personally evaluate the probabilistic risk assessment technique and

develop a strategy for introducing it throughout NASA. However, the

strategy has not yet been developed.

Regarding its databases, NASA responded by developing a centralized

database designed to improve the quality of information by providing an

Page 37 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

integrated view of the status of shuttle problems in near real time. The

Council recommended that development of this system be given a high

priority. NASA developed a database to provide information, but, as

discussed in other sections of this chapter, that database has limitations.

NASA officials told us that while some progress has been made, the use of

probabilistic methods have not reached a mature state at NASA. NASA has

made limited use of probabilistic risk assessments of the shuttle, including

proof-of-concept studies, assessment of some specific shuttle systems, and

required assessments of accident probabilities for launches involving

radioactive material. A 1994 survey of probabilistic methods used in

structural design, which included some shuttle projects, found that there

is no agreed-upon approach across centers for preferred methods,

practices, or software and that the various quantitative tools have not been

fully examined, evaluated, and accepted by NASA centers.

In early 1994, the NASA Administrator and the Office of Space Flight

concluded that a probabilistic risk assessment of shuttle risk was needed

to guide safety improvement decision-making. According to a safety

official, NASA contracted with Science Applications International

Corporation in January 1994 to conduct a probabilistic risk assessment of

the space shuttle. This was the first assessment to include a complete

shuttle mission. The contractor was required to develop and apply a risk

model of the shuttle during flight and to quantify in-flight safety risk. The

analysis was to identify, quantify, and prioritize risk contributors for the

shuttle. According to the model’s author, secondary objectives were to

provide a vehicle for introducing and transferring probabilistic risk

assessment technology to NASA, and to demonstrate the value of the

technology. The model was completed in April 1995.

According to the contractor who developed the probabilistic risk

assessment model, the model could be a useful tool in NASA’s management

of shuttle risks. For example, the model might be used to establish

realistic cost objectives for redesigning the high risk components, helping

to assure that limited resources are focused toward solving those

problems that will have the most impact on safety. The National Research

Council also noted that a detailed quantitative risk assessment provides

decision makers with a better basis for managing risk.

An internal shuttle program survey of managers, safety experts, and senior

engineers revealed mixed reactions to the model. Although generally

positive, respondents cited some concerns. For example, some

Page 38 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

respondents commented that more use of actual failure data would have

benefited the analysis and that some assumptions used were debatable.

Some found fault with the excess use of expert opinion and the lack of

thoroughness in delineating certain assumptions. Following the survey,

the Deputy Associate Administrator for Space Flight informed shuttle and

safety managers that they should feel free to use the report and model as a

“limited tool in the risk management tool box.”

According to some NASA safety officials, the model has not been routinely

used by NASA personnel as a risk assessment tool because officials are still

evaluating the utility of the model and barriers exist to its use by NASA

employees. For example, there is no instruction manual for using the

model and it requires use of contractor owned software. According to

safety officials, NASA does not have current copies of the required software

and older inadequate versions are limited. According to these officials,

only one NASA employee has been able to use the model on a NASA

computer using the older software. In addition, no firm decisions have

been made regarding maintenance and update of the model to reflect

shuttle changes, such as the super light weight external tank. Safety

officials stated they are continuing to assess the model to determine its

utility within NASA.

2

NASA project and safety officials compile a list of significant shuttle risk

issues for each project to target resources and manage risk reduction

efforts. Only risks that can be reduced by incorporating hardware or

procedure modifications are included in the assessment. According to

NASA’s April 1995 shuttle safety risk ranking methodology guidance, the

source of risk information currently used in the rankings is qualitative and

the process ranks catastrophic events by judgmentally derived

prioritization matrices. The guidelines state that many comparisons of

catastrophic events could be made but are sometimes subjective,

emotional, and rely on different techniques. A complete probabilistic risk

assessment would be the most desirable analysis, according to the

guidelines, but probabilistic analyses are labor-intensive efforts that

require many system experts, a complete understanding of the

methodology, and proper management of the effort.

2

After our field work was completed, NASA informed us that the Office of Safety and Mission

Assurance and the Office of Space Flight were working together to procure the necessary software to

operate the shuttle probabilistic risk assessment model. The software is being procured to (1) maintain

the currency of the model by the addition of new flight and test data as they become available;

(2) modify the model, as appropriate, to reflect the most current shuttle design configuration; and

(3) permit possible “reverse engineering” of the model to enable the use of its major components, both

separately and together, using less expensive and more commonly used software applications.

Page 39 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

Limited Use of

Quantitative Risk

Assessments

NASA has made limited progress in adopting the National Research

Council’s recommendations that the agency assess risk with quantitative

methods, such as probabilistic risk assessments. NASA uses a variety of

methods to assess shuttle risk issues, and efforts are underway to increase

the use of quantitative methods. Qualitative methods are still widely used

when risk issues are thought to be well understood. NASA has made limited

use of the classical probabilistic risk assessment method of analysis. Cost,

lack of specific expertise, and lack of data are the reasons cited for limited

use.

According to shuttle and safety managers, lack of a strategy for

incorporating the methods into decision-making processes has impeded

NASA’s progress in adopting the National Research Council’s

recommendations on risk assessments. Also, insufficient expertise exists

at NASA to conduct specific quantitative analyses, such as probabilistic risk

assessments.

NASA project and safety officials told us that progress in implementing

quantitative risk assessment methods has been impeded because NASA

does not have a working strategy for formalizing these methods for the

shuttle program. Such a strategy would include clear and measurable

goals, resource requirements, assessments of current utilization and skills

within NASA, and training needs, including the need to learn by doing

selected projects. Without this focus, projects and safety organizations are

skeptical about the cost and benefits of using the probabilistic risk

assessment model.

Project and safety officials at several centers expressed concerns about

the applicability of probabilistic risk assessments to the shuttle program.

While officials stated they recognize probabilistic risk assessments could

be used as an effective additional tool to assess risk, they see a need for

more training on the methodology and the need to learn by doing selected

projects. Several stated they do not have the resources needed for this

type analysis but are stretched just to operate their programs. Several

officials stated they believe there is a lack of trust in the probabilistic risk

assessment method because people do not understand it. Many officials

expressed concern about the complexity of the shuttle probabilistic risk

assessment model, the lack of good data, and the dependence upon the

contractor to make needed changes to the model. Several officials

commented that NASA needs a “champion” at headquarters to provide a

focused effort to emphasize use of these tools when appropriate.

Page 40 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

NASA headquarters safety and mission quality officials stated they have not

developed a master plan for formalizing quantitative techniques within

NASA or made the progress they would like in this area. However, steps are

being taken to address several of the concerns expressed by project and

safety officials at the centers. For example, training courses in risk

management and assessment are being planned that will be offered to

safety and other NASA personnel. Reference manuals on sources for data

and techniques on risk assessments are under contract. According to NASA

safety officials, the first effort to develop these type documents began in

1989 but was unsuccessful and the documents were not published.

However, NASA has established a coordination committee to develop a

standard, comprehensive approach to introduce structural design methods

that can be used in the shuttle program.

NASA is also trying to give this issue visibility as the agency plans to move

to a single prime contractor and to assure that the statement of work

contains provisions that the contractor use quantitative risk assessment

techniques where appropriate.

According to the National Research Council, decisionmakers within NASA

must be supported by people skilled in the statistical sciences to aid in the

transformation of complex data into useful information. The Council

recommended that NASA develop a staff of experts in these areas to

provide improved analytical support for risk management. NASA officials at

several centers and at NASA Headquarters told us they lack sufficient

personnel with these skills, and in one case, a center lost needed

contractor skills that caused the delay or termination of a planned

analytical project. A 1994 NASA survey of probabilistic methods used in

structural design work found that a wide variance of knowledge exists at

the centers and that a majority of working-level engineers are not familiar

with and do not use probabilistic methods.

NASA’s Databases Do

Not Provide Sufficient

Information for

Quantitative Analysis

Another factor that has hindered development of quantitative methods of

risk assessment is that NASA’s databases do not always provide timely,

accessible, accurate, and complete information. A large percentage of

managers and safety representatives we surveyed believe that NASA should

provide management information systems, including databases that are

accessible, accurate, complete, and timely for shuttle program oversight

and decision-making. However, more than half assessed NASA’s current

management information systems as needing improvement.

Page 41 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

NASA has developed automated database systems to provide shuttle data

used in decision-making. One system, called the Program Compliance

Assurance and Status System, is a central database designed to integrate

existing data, such as in-flight anomalies, from various sources in the

program. Another system, the Problem Reporting and Corrective Action

system, provides data to the central system and is designed to document

and track problems in the program.

According to NASA officials, the Program Compliance Assurance and Status

System is neither timely nor fully utilized. The system is cumbersome to

use because it is based on older technology, some trend and other data is

not centralized in the system, and software needed to convert contractor

data to NASA database format has not been developed. Program officials

told us they maintain trends on some aspects of the shuttle program, but

have found the centralized system to be difficult to use and not compatible

with other existing databases. The officials stated that the required

conversion programs have never been developed to input some contractor

data into the system. In some cases, safety officials must obtain data

directly from contractors to conduct quantitative risk assessments.

Because the system is hard to use in real-time and the data is not always

current, some officials stated they are using a different software program

with faster computers to access and correlate data more rapidly.

A January 1995 internal report on shuttle problem reporting system data

integrity at two centers found missing criticality codes on thousands of

entries. Blank entries could, therefore, be interpreted as either not

applicable or inadvertently omitted. A NASA Headquarters official was not

aware of any corrective action on this matter. Officials told us that the

Problem Reporting and Corrective Action System records are often not

reliable, lack data needed for quantitative risk assessments, and lack

uniformity in categorizing problems. The system also contains entries that

may not meet the definition of a “real problem.” NASA safety officials

acknowledged that the system needs improvement but stated no firm

decision has been made regarding the extent of improvements pending the

transition to a single prime contractor.

Conclusions NASA has made limited progress in adopting the National Research

Council’s recommendation that the agency assess risk with quantitative

methods, such as probabilistic risk assessments. NASA officials, for the

most part, rely on qualitative methods for assessing risk in the shuttle

program when they believe risk issues are well understood. Although

Page 42 GAO/NSIAD-96-73 Space Shuttle
Chapter 3 

NASA Has Not Developed an Overall

Strategy for Using Quantitative Risk

Assessment Methods in the Shuttle Program

some progress has been made, NASA lacks an overall strategy with focused

management emphasis to incorporate methods, such as probabilistic risk

assessments into the shuttle program, when appropriate. Resource

constraints and specific expertise are cited as barriers to increased use of

these methods. In addition, NASA’s databases need improvement and are

not fully utilized by decisionmakers nor are they adequate to support the

use of quantitative risk assessment methodologies.

Recommendations We recommend that the Administrator of NASA establish a strategy, to

include specific milestones, for deciding whether and how quantitative

methods, such as probabilistic risk assessments, might be used as a

supplemental tool to assess shuttle risk.

We also recommend that the Administrator assess the shuttle program’s

centralized database, as well as other databases, to insure that data

required to conduct risk assessments and inform decisionmakers is

accessible, timely, accurate, and complete.

Agency Comments NASA agreed with the need to establish a strategy, with milestones, for

incorporation of quantitative risk assessment methods into the shuttle’s

risk management program. According to NASA, the agency will establish a

team to develop the strategy.

NASA also agreed that the shuttle program’s centralized databases need to

be assessed. In this regard, NASA will form a team of engineers to

thoroughly examine the Program Compliance Assurance and Status

system. The team will be tasked to determine the adequacy of what

presently exists and make recommendations for improvements as

necessary. The assessment team will report to the shuttle program

manager. In addition, the Problem Reporting and Corrective Action

System is being examined at each center by a reengineering team. This

team is searching out deficiencies and will recommend needed

improvements that must be implemented by the shuttle flight operations

contractor.

We made additional changes to the report, where appropriate, based on

NASA’s technical comments.

Page 43 GAO/NSIAD-96-73 Space Shuttle
Appendix I 

Methodology for Group Interviews and

Survey Assessing the Flow and Quality of

Shuttle Safety-Related Information

Overview This appendix describes the methodology we used to study the flow and

quality of safety-related information in the shuttle program. Appendix II

shows the questions and results of our survey of shuttle program officials

and appendix III provides the questions used in our group interviews.

We conducted group interviews with National Aeronautics and Space

Administration (NASA) managers located at the three NASA field centers

with primary responsibility for managing shuttle program elements and at

the program’s headquarters. The group interviews enabled participants to

exchange their perspectives on communication within the shuttle

program, provided us with an understanding of these complex areas, and

produced concrete illustrations.

While the interviews provided insights that may only arise in a group

setting, we also sent a brief survey to these managers and to safety

officials responsible for shuttle hardware components to obtain more

precise measures of the themes that emerged in the group discussions. We

included personnel in the shuttle program’s safety and mission assurance

and engineering organizations in our interviews and survey as well as

project and program managers in order to obtain a full range of

perspectives on communication. The objectivity and accuracy of our

interpretation of the transcribed group discussions were verified through

several approaches.

Another component of our design was to interview personnel with three

shuttle program contractors. We selected the contractors responsible for

the solid rocket motor and the shuttle main engine because these systems

are complex, high-risk elements. We also selected the contractor

responsible for processing the shuttle for launch because it is a

labor-intensive effort. We interviewed the program managers and senior

safety officials for shuttle program contractors in order to understand the

flow of information between NASA and contractors from the contractor’s

perspective. We held separate group interviews with working-level

engineers with each of the three contractors in order to understand the

flow of information within the contractor’s organization. The viewpoints

expressed in the contractor interviews cannot be generalized to other

shuttle program contractors.

NASA Group

Interviews

We included shuttle project and program managers (or their designated

alternates) located at three centers and at headquarters in the NASA group

interviews. We held a group interview composed of 6 to 14 managers at

Page 44 GAO/NSIAD-96-73 Space Shuttle
Appendix I 

Methodology for Group Interviews and

Survey Assessing the Flow and Quality of

Shuttle Safety-Related Information

each of the following locations: Johnson Space Center, Kennedy Space

Center, Marshall Space Flight Center, and the shuttle program’s

headquarters organization. In total, 40 individuals in the following

positions or alternates designated by NASA participated: shuttle program

and project managers and senior managers in the safety and mission

assurance and the program’s engineering organizations. The group

interviews were conducted between April and June 1995.

The main questions in our NASA group interviews focused on information

conveyed at the Level I flight readiness review, the extent to which various

conditions ensure that serious issues come to the attention of

management, and funding reductions and restructuring as they relate to

morale and the transmittal of high quality information for safety

assessment.

We moderated the interviews that were audio-recorded. A co-moderator

took notes in the event that audio-recordings were not complete or clear.

The transcription of each interview was systematically analyzed by the

moderator to extract the discussion themes and illustrations of these

themes.

We ensured the adequacy of our analysis and interpretation of the group

interviews through several steps. A summary of each group discussion

with NASA personnel was developed by our staff and independently audited

by another staff member who traced each statement in the summary back

to the portion of the transcribed text which supported the statement. The

summaries were then reviewed by a NASA official with in-depth knowledge

of the shuttle program who had not participated in the group interviews.

The reviewer was asked to assess the summaries for their technical

correctness and objectivity.

As a final verification step, each participant in the group interviews

received the summary of the interview he or she participated in, along

with a copy of the transcription of the discussion. The participants were

asked to verify that the summary accurately reflected his or her input and

the communication themes that emerged. The suggested clarifications

from the NASA reviewer and the NASA group interview participants were

incorporated in the summaries.

Contractor Interviews We held interviews with the project manager and senior safety official for

the solid rocket motor, shuttle main engine, and shuttle processing

Page 45 GAO/NSIAD-96-73 Space Shuttle
Appendix I 

Methodology for Group Interviews and

Survey Assessing the Flow and Quality of

Shuttle Safety-Related Information

contractors during May and July 1995. The main questions in these

interviews focused on NASA reporting requirements, information conveyed

at Level III flight readiness reviews and other reviews, the flow of

shuttle-related information within the contractor’s organization and with

NASA, and funding reductions and restructuring as they relate to morale

and the transmittal of high-quality information for safety assessment.

We also held a group interview composed of 9 to 12 working-level

engineers at the three shuttle program contractors in our study. The

interviews were conducted in May and July 1995. We attempted to include

engineers from each of the contractor’s major work areas, including

contractor safety organizations. The main questions in these group

interviews focused on the types of safety issues and concerns the

engineers brief to their management, the flow of information within their

organizations, and funding reductions and restructuring as they relate to

morale and the transmittal of high quality information for safety

assessment.

The contractor interviews were audio-recorded. The transcription of each

interview was systematically analyzed by our staff to extract the

discussion themes and illustrations of these themes. A summary of each

interview was developed by a staff member and independently audited by

another staff member. As a final verification step, each interviewee

received a copy of the summary and the transcribed interview and was

asked to verify that the summary accurately reflected the discussion and

the major themes that emerged. The suggested clarifications from the

contractor group interview participants were incorporated into the

summaries.

Survey Methodology We pretested a questionnaire on communications with managers at the

three space centers included in the group interview portion of our study

and at NASA Headquarters. Headquarters officials also performed a

technical review of the survey questions. The questionnaire was

distributed in August 1995, to each participant in the NASA group

interviews, except for two managers who had retired from the program at

the time of the survey. We also sent questionnaires to the safety officials

responsible for shuttle hardware components. Survey recipients were told

that the survey results would be reported in summary form in our report

and that any discussion of individual answers would omit information

identifying the respondent. Of the 44 surveys we distributed, we received

39 responses, representing a response rate of 89 percent.

Page 46 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

This appendix provides the exact text of the survey questions. Transitional

phrases used to guide the respondents from one topic to another are not

included. The appendix provides the combined number of managers and

safety representatives endorsing the response options accompanying each

survey question as well as the number of respondents eligible to answer an

item who did not answer.

Page 47 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 1: During 1994 and 1995 a variety of forums provided

opportunities for personnel to discuss and

communicate shuttle safety issues and concerns

(e.g., FRRs, morning tag-ups, technical panels and

working group meetings, change review boards, and

ad-hoc meetings). Overall, to what extent, if at

all, did these forums meet the safety needs of the

Shuttle Program with regard to the number of forums

held and the types of expertise and levels of

authority present at the meetings?

Number of Respondents in Each Rating Categorya

COMMUNICATION

FORUMS

Greatly deficient

for needs

Somewhat

deficient for

needs

Exactly met

needs or about

right

Somewhat

exceeded

needs

Greatly

exceeded

needs

1. Number of forums 22 15 1

2. Types of expertise

present at each forum

30 9

3. Levels of

authority/management

present at each forum

1 30 7 1

a

Item 1 was answered by 38 managers and safety representatives. Items 2 and 3 were each answered by 39

managers and safety representatives.

Page 48 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 2: One approach to quality assurance stresses reliance

on the use of inspectors or reviewers to check work

performed by others. Another approach stresses

reliance on individual "hands-on" workers to meet

performance standards when completing their tasks,

relying less on subsequent inspections and reviews

to assure quality. Given the need to maintain

current shuttle safety standards, do you believe

that the present level of reliance on each of these

two approaches should be increased, decreased, or

remain the same?a

Number of Respondents in Each Rating Categorya

QUALITY ASSURANCE

APPROACHES

Should be

greatly

decreased

Should be

somewhat

decreased

Should remain

the same or

about the

same

Should be

somewhat

increased

Should be

greatly

increased

1. Current reliance on

inspections and review

24 8 1

2. Current reliance on

individual "hands-on"

workers meeting

performance standards,

relying less on inspections

and reviews

4 1 6 24 6

a

Respondents were asked to consider work performed by both shuttle contractors and NASA or other civil

service personnel in the organizational unit(s) they managed, or if not managers, for the unit worked in. Items 1

and 2 were each answered by 37 managers and safety representatives. Two respondents did not answer.

Page 49 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 3: To what extent, if at all, would the increased or

decreased reliance you suggest improve (1) the

efficiency of shuttle quality assurance processes

and (2) the accuracy, reliability, and timeliness of

shuttle safety-related information?

Number of Respondents in Each Rating Category

QUALITY ASSURANCE

APPROACHES

To little or no

extent

To some

extent

To a

moderate

extent

To a great

extent

To a very

great extent

1. Improve efficiency of

Shuttle quality assurance

processes

8 11 10 1

2. Improve the accuracy,

reliability, and timeliness

of shuttle safety-related

information

10 7 8 4

a

Thirty-four managers and safety representatives were eligible to answer this question because they recommended

an increase or decrease in one or both of the quality assurance approaches listed in question 2. Item 1 was

answered by 30 of these managers and safety representatives and item 2 by 29 of them.

Page 50 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 4: Regardless of what was done in the past, which

types of issues do you believe should always be

briefed in detail at the COFR 6 FRR when the issues

apply to the launch or mission being reviewed?

Number of respondentsa

TYPE OF ISSUEB Brief issue

in detail

Do not

brief issue

in detail

1. CRIT 1 or 1R hardware or software performance outside certified

design/performance specification limits 35 3

2. Other hardware or software (below CRIT 1 or 1R) performance outside

certified design/performance specification limits 13 25

3. Hardware or software performance outside historical experience; out-offamily performance not seen before 34 4

4. Unexplained anomalies 32 6

5. Other unresolved problems or open issues 26 12

6. Potential generic problems from another shuttle or another launch

vehicle 33 5

7. Issues or events leading to an increase in risk posture 36 2

8. New or changed procedures or processes for crew, operations, software,

or hardware 15 22

9. Any changes in flight margins to include performance, loads, or any

envelope expansion beyond previous flight experience 28 9

10. Constraints to launch 35 3

11. Explained problems (full corrective actions not possible and risks must

be accepted)

22 16

12. Hazards (where risks had previously been accepted) if new problems or

events affect-might affect them 27 10

13. New waivers 25 13

14. Items related to the NASA Safety Reporting System (NSRS) 26 12

15. Any issue that a participant at the FRR or other Shuttle Program

individual believes constitutes a reason for not flying or an issue that an

individual wants the FRR Board to provide an opinion/perspective on

37 1

a

This question was answered by 38 managers and representatives except for items 8, 9, and 12 which were each

answered by 37 managers and representatives.

b

In almost all the issue areas listed above, a few managers wanted detailed briefings at the FRR restricted to

issues that were critical to safety of flight, for example, issues involving items at the criticality 1 or 1R level.

Page 51 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 5: Currently, to what extent, if at all, does the

Shuttle Program have the following communication and

information flow conditions?a

Number of Respondents in Each Categorya

SHUTTLE PROGRAM CONDITIONS To little or

no extent

To some

extent

To a

moderate

extent

To a great

extent

To a very

great extent

1. An organizational culture that encourages

civil service employees to discuss safety

concerns with management

2 2 20 15

2. An organizational culture that encourages

civil service employees to elevate concerns

to higher management if they believe the

issues were not adequately addressed at

lower levels

3 2 18 16

3. A culture that encourages contractors to

raise safety issues with their government

counterparts

3 4 20 12

4. Sufficient parallel assessment and/or

review functions, such as SR&QA, with

independent reporting chains to top

management

1 4 21 13

5. Program priorities which place greater

importance on safety than on meeting

schedule

5 4 11 19

6. Program priorities which place greater

importance on safety than on cost

reductions

1 5 9 12 12

7. Management information systems which

contain accessible, accurate, complete, and

up-to-date information for shuttle risk

assessment and decision-making

11 11 13 4

a

The question was answered by each of the 39 managers or safety representatives who responded to our survey.

Page 52 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 6: During the last 3 years, has your shuttle work had

any funding reductions? (Answer only for the

organizational unit(s) you manage, or if not a

manager, the unit you work in)

Thirty-six respondents said that their shuttle work had had

funding reductions during the last 3 years while 3 had not.

Page 53 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Page 54 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Question 8: As NASA deals with budget constraints,

associated downsizing, and restructuring in the

near term, and with continuation of Shuttle

flight in the long term, what guiding

principles related to communication and

information flow do you believe are essential

to promote Shuttle Program safety? (Check all

that apply)

Number of Respondentsa

THE GUIDING PRINCIPLE THAT: Essential Not essential

1. Managers (civil service and contractor) have values that stress safety over

schedule and cost and that managers foster these values among their

people

39

2. The organizational environment and structures for both contractor and

NASA personnel encourage timely, open discussion and debate to ensure

managers have the benefit of all relevant knowledge of shuttle program

issues

39

3. The organizational environment encourages people (civil service and

contractors) to elevate concerns to higher management if they believe the

issues were not adequately addressed at lower levels

39

4. The working arrangement between NASA and contractors ensures agency

managers obtain continual knowledge of problems and issues so that

appropriate shuttle decisions can be made

36 3

5. Organizational mechanisms enable NASA corporate-level managers to

carry out their decision-making responsibilities for certifying shuttle

readiness for flight

37 2

6. NASA use the most appropriate analytic and quantitative methods

available to assess shuttle risks and conduct sufficient assessments and

reviews to carry out the agency’s oversight of shuttle work processes

37 2

7. Management information systems, including databases, are accessible,

accurate, complete, and timely for shuttle program oversight and decisionmaking

31 8

8. The NASA environment is a self-evaluative one that monitors its

effectiveness in communication and information flow and seeks ways to

improve it

34 5

a

This question was answered by each of the 39 managers and safety representatives who responded to our survey.

Page 55 GAO/NSIAD-96-73 Space Shuttle
Appendix II 

Survey Questions and Responses

Several managers and safety representatives identified additional guiding

principles necessary for safe operation as they undergo organizational and

funding changes:

The guiding principles that:

• Management of changes in the program receives adequate attention and

time to ensure that (1) program priorities are adhered to, (2) government

and contractor responsibilities for the reporting and resolution of

safety-related issues are clearly defined, and (3) changes to the Shuttle

Program are appropriately evaluated before implementation.

• Appropriate training is conducted to ensure that personnel can effectively

and efficiently carry out their work when changes in program operations,

processes, and staffing occur.

• Morale and the working environment of employees are considered key

elements in assuring a safe and quality program.

• Prime contractor management methods ensure the quality of

subcontractor work.

Page 56 GAO/NSIAD-96-73 Space Shuttle
Appendix III 

Group Interview Questions

This appendix presents the questions used in our group interviews. We

used different sets of main questions for the interviews we held with NASA

personnel, program managers and safety officials with Shuttle Program

contractors, and contractor engineers at the working level. Different sets

of probes, not listed below, were prepared for each set of main questions.

NASA Group Interview

Questions

1. In order for the Board to provide a corporate decision on launch, what

types of safety issues do you think always need to be briefed in detail at

the FRR—regardless of what was done in the past?

2. Unintentional filtering can occur if information is lost or important

details are missing as messages are transmitted within an organization.

Several conditions can minimize the opportunity for such unintentional

filtering to occur in the Shuttle Program. For example,

• An organizational culture that encourages people to discuss safety

concerns with management and to elevate concerns to higher management

if they believe the issues were not adequately addressed at lower levels.

• A culture that encourages contractors to raise safety issues with their

government counterparts.

• A parallel assessment and/or review function, such as SR&QA, with an

independent reporting chain to the Associate Administrator for Safety and

Mission Quality.

• Program priorities which do not tempt managers to override safety

considerations in order to meet schedule or cut costs.

• Databases that permit timely retrieval of complete and accurate

information relevant to shuttle risk assessment and decision-making.

The question is: To what extent, if at all, does the Shuttle Program have

these or other conditions that ensure that management is informed of

serious safety issues? Please discuss your position on each of these

conditions.

3. (Headquarters only) What impact, if any, have funding reductions during

the last 3 years had on the morale of civil service employees at

Headquarters and at each of your Centers?

4. (Centers only) During the last 3 years, what impact, if any, have funding

reductions or related restructuring or downsizing had on the morale of

civil service employees that you work with?

Page 57 GAO/NSIAD-96-73 Space Shuttle
Appendix III 

Group Interview Questions

5. (Headquarters only) What impact, if any, do you think the funding

reductions have had on the quality of safety-related information you

develop or receive?

6. (Centers only) Are there any functions, processes, or tasks that have

been changed or eliminated during the last 3 years because of these

factors? (If yes) What effect, if any, have these changes or cuts had on the

quality of safety-related information you develop?

7. (Centers only) While changes or cuts may not have a noticeable effect

on the quality of safety-related information, they could increase the risk of

degraded quality beyond what is acceptable. What techniques were used to

ensure that these changes or cuts did not increase the risk beyond what is

acceptable?

Interviews With Contractor

Program Managers and Safety

Officials

1. What kinds of problems, hazards, or other safety issues or concerns

does NASA require you to report to them? How does NASA require that you

document these safety issues or concerns? Are there any types of issues or

concerns that are not documented?

2. Over about the last 2 years, what types of safety issues did your program

office discuss in detail at your reviews that prepare for the Level III FRRs?

3. If there is disagreement among contractor personnel about whether a

safety issue should be brought forward to NASA, how is it handled?

4. What do you see as your role and responsibilities at the Level III FRR or

in support of this review? How about the COFR 6 review?

5. Over about the last 2 years, what types of safety issues or concerns did

your office brief NASA on in detail at the Level III FRR? How about at COFR

6 reviews?

6. Finally, as a contractor, what types of safety issues do you think your

office should always brief in detail at NASA reviews, regardless of what was

done in the past?

7. If you had a safety concern that in your view did not get surfaced at the

appropriate NASA review level or if you disagreed with a safety decision

made by NASA, what avenues are open to you to communicate your

concerns?

Page 58 GAO/NSIAD-96-73 Space Shuttle
Appendix III 

Group Interview Questions

8. See item 2, NASA Group Interview Questions. The identical question was

asked of contractor officials except for the framing of the last part of the

question, as shown below.

The question is: To what extent, if at all, does the Shuttle Program and

your organization have these or other conditions that ensure that

management is informed of serious safety issues? Please discuss your

position on each of the conditions. Please consider conditions first in NASA

and then within your own organization.

9. Has your shuttle work had any funding reductions or related

restructuring or downsizing during the last 3 years? (If yes) What impact, if

any, have these factors had on the morale of contractor employees you

work with?

10. Were any functions, processes, or tasks related to your work on the

shuttle changed or eliminated during the last 3 years because of reductions

or associated downsizing or restructuring? (If yes) What effect, if any, did

these have on the quality of shuttle information you develop?

Group Interviews With

Working-Level Engineers

1. Over about the last 2 years, did you brief your management in detail on

any safety issues or concerns, and if so, what types of safety issues or

concerns were these?

2. If there is a disagreement among contractor personnel about whether a

safety issue should be brought forward to your higher levels of

management, how is it handled?

3. If you had a safety concern that did not get surfaced at the appropriate

level, or if you disagreed with a safety decision made at higher levels,

either within your contractor’s plant or NASA, what avenues are open to

you to communicate your views?

4. Given the organizational culture within your organization, how

acceptable or unacceptable do you believe it is to voice safety concerns?

5. Has your shuttle work had any funding reductions or related

restructuring or downsizing during the last three years? (If yes) During the

last three years, what impact, if any, have these factors had on the morale

of contractor employees you work with?

Page 59 GAO/NSIAD-96-73 Space Shuttle
Appendix III 

Group Interview Questions

6. Are there any functions, processes, or tasks related to your work on the

shuttle that have been changed or eliminated during the last 3 years

because of funding reductions or associated downsizing or restructuring?

(If yes) What effect, if any, have these changes or cuts had on the quality

of shuttle information you develop?

Page 60 GAO/NSIAD-96-73 Space Shuttle
Appendix IV 

Comments From the National Aeronautics

and Space Administration

Note:We are not

publishing enclosures 2

and 3 to NASA’s

comments. These

enclosures included

suggestions for technical

changes, which we

incorporated where

appropriate.

Page 61 GAO/NSIAD-96-73 Space Shuttle
Appendix IV 

Comments From the National Aeronautics

and Space Administration

Page 62 GAO/NSIAD-96-73 Space Shuttle
Appendix IV 

Comments From the National Aeronautics

and Space Administration

Page 63 GAO/NSIAD-96-73 Space Shuttle
Appendix V 

Major Contributors to This Report

National Security and

International Affairs

Division, Washington,

D.C.

Lee Edwards

James Beard

Fred Felder

Richard Irving

Julia Kennon

Marilyn Mauch

Terry Wyatt

(709066) Page 64 GAO/NSIAD-96-73 Space Shuttle
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100