Space Shuttle: Need to Sustain Launch Risk Assessment Process Improvements
Released: 1996-03-26T00:00:00

Letter

Introduction

The first space shuttle launch occurred on April 12, 1981. During the 25th launch on January 28, 1986, the shuttle Challenger was destroyed shortly after liftoff from Kennedy Space Center. Shuttle flights were suspended while the accident was investigated by the Presidential Commission. The shuttle returned to flight on September 29, 1988. Since that time, it has flown successfully about 50 times.

The Presidential Commission determined that the 1986 accident was caused by a faulty seal in one of the solid rocket motor joints. The Commission also found other contributing causes to the accident, such as management isolation, communications failures, and lack of a properly staffed, supported, and robust safety organization.

According to the Commission’s June 6, 1986, report, the decision to launch the Challenger was based on incomplete and sometimes misleading information, a conflict between engineering data and management judgments, and a the National Aeronautics and Space Administration (NASA) management structure that permitted internal flight safety problems to bypass key shuttle managers. Officials who made the launch decision were unaware of a recent history of problems with the defective solid rocket motor joint and of the motor contractor’s initial recommendation against launching. According to the Commission, if the decisionmakers had known all of the facts, it is highly unlikely that they would have decided to launch.

Risk of Space Flight

Space flight can never be made risk free because it involves complex hardware and software systems, harsh operating environments, and the possibility of human error. A 1995 study by a NASA contractor, for example, placed the median estimate of a catastrophic shuttle failure at 1 in 145 launches.

According to the advisory committee on the Future of the U. S. Space Program, “there can be no acceptable objective among those who would challenge the vastness of space other than perfection.” Unfortunately, as the Committee’s report points out, the objective of perfection is not readily met, especially since space missions are fundamentally difficult and demand undertakings that depend upon some of the world’s most advanced technology and there are many opportunities for error.

The shuttle is an extremely complex system. The program employs thousands of people and launching a shuttle requires that 1.2 million separate procedures be accomplished correctly. Also, NASA has identified over 5,000 critical system components whose failure, either singularly or in combination, could cause loss of the vehicle or crew. Because these risks cannot be completely eliminated, they must be identified and properly managed.

NASA’s Risk Management Program

NASA’s risk management policy requires that program and project management communicate to NASA management and all program/project personnel the significance of assessed risks and the decisions made with respect to them. At NASA, risk management includes identifying the primary risk drivers and estimating the likelihood of occurrence, identifying the ensuing consequences, and determining the cost and schedule impact.

NASA policy regarding safety is to avoid loss of life, injury of personnel, damage, and property loss; instill safety awareness in all NASA employees and contractors; assure that an organized and systematic approach is utilized to identify safety hazards and that safety is fully considered from conception to completion of all agency activities; and review and evaluate contractors’ and NASA’s plans, systems, and activities related to establishing and meeting safety requirements to ensure that desired objectives are effectively achieved.

Failure modes and effects analyses are conducted for all flight hardware elements and ground support equipment. This analysis starts with the identification of all potential failure modes and evaluation of “worst case” effects. NASA places potential effects of failures into the general categories shown in table 1.1.

Hazard analyses are conducted to identify potential safety hazards and means for minimizing the hazards. NASA’s actions to minimize hazards follow the sequence of (1) system designs that minimize potential hazards, (2) use of safety devices if the design does not eliminate a potential safety hazard, (3) use of warning devices to alert the flight or ground crew to potential hazards, and (4) use of special procedures.

Risk Assessment Approaches

Approaches for assessing risk can be either quantitative or qualitative, depending on whether statistical probabilities are assigned to a risk element. All risk assessment approaches require experts to make subjective judgments about the risk elements as well as the likelihood of their occurrence.

Quantitative approaches, such as probabilistic risk assessments, can be used to assess both the likelihood that an accident will occur (probability) and the level of damage or loss that will result (consequences). Quantitative assessment methods mathematically quantify risk on the basis of engineering judgment, calculated probabilities of component reliability, analysis of potential human failures, and whether they occur singly or in combination. A probabilistic risk assessment, for example, addresses three basic questions: (1) What could go wrong? (2) How likely is it that this will happen? and (3) What are the consequences?

Qualitative assessments, on the other hand, assess risk through descriptive information, identifying the nature and components of risk or an ordinal scale, such as high, medium, and low. Qualitative ratings are usually based on the judgments of experts after they consider such things as test and operational experience, analytical results, trends, and other reported data.

Certifying the Shuttle for Flight

NASA follows a formal review process in certifying the shuttle for flight. The certification of flight readiness process is a step-by-step activity designed to certify the readiness of all components of the vehicle assembly and all aspects of mission support.

The flight preparation process begins with project milestone reviews including (1) element acceptance, (2) payload readiness, (3) software readiness, and (4) project preflight readiness reviews. These reviews are chaired by NASA project managers and the contractors formally certify the flight readiness of the hardware and software.

The next step in the process is the program milestone reviews. These reviews are held to assess the readiness for mating the external tank and solid rocket booster, orbiter and external tank, and ferrying the orbiter atop the shuttle carrier aircraft when required. These reviews are chaired by the manager of launch integration and each shuttle element manager certifies that it has satisfactorily completed the manufacture, assembly, test, and checkout of the elements, including the contractor’s certification that design and performance are up to standard.

The final step in the flight preparation process is the flight readiness review. This review is held about 2 weeks prior to launch and is chaired by the Associate Administrator for Space Flight. All shuttle elements, safety and mission assurance, center directors, and senior representatives from the major contractors participate in this review. At the end of the flight readiness review, all organizations must certify that the mission is ready for launch. The Associate Administrator for Safety and Mission Assurance is also an active participant.

The safety and mission assurance organization holds parallel reviews to assess safety issues related to the planned launch. The safety and mission assurance organization participates in all phases of the flight preparation process.

Two days before a scheduled launch, a mission management team holds a review to assess flight readiness. Its agenda includes close out of any open work, close out of any flight readiness review action items, discussion of new or continuing anomalies, and an updated briefing on anticipated weather conditions at the launch site and at abort landing sites in different parts of the world. The mission management team meets every day after the launch –2 day review up to the conclusion of the mission. Figure 1.1 illustrates NASA’s flight preparation process.

NASA’s safety organization provides an independent channel for assessing shuttle flight safety. Each center’s safety organization participates in the element acceptance reviews as well as the flight readiness review and the mission management team. Participation in these reviews provides the opportunity for NASA’s safety organization to express any residual concerns about the safety of an upcoming mission. The organization also holds independent prelaunch assessment reviews. In addition, the Associate Administrator for Safety and Mission Assurance attends the flight readiness review and has a direct communications link to the NASA Administrator.

Other program briefings and reviews are also a part of the certification of flight readiness process. For example, the program manager holds an early morning telephone conference with the shuttle centers and headquarters each day to discuss the status of progress and problems. Likewise, about midday the working level shuttle managers hold a telephone conference to provide updated information. Safety and mission assurance personnel attend all of the shuttle program and project meetings and contribute their independent views.

Objectives, Scope, and Methodology

The former Chairman, Subcommittee on Investigations and Oversight, House Committee on Science, Space, and Technology, asked us to review NASA’s management of risk associated with flying the space shuttle. Specifically, we reviewed the actions NASA has taken to improve the free flow of information in the launch decision process and the progress NASA has made in adopting quantitative methods for assessing risk.

To assess the communications environment, we reviewed policies, procedures, and practices related to management of the shuttle program used by the agency in making launch decisions; we observed various shuttle processing reviews, including a shuttle launch; and discussed various aspects of the program with those responsible for its management.

We also conducted discussions of these topics with groups of shuttle and safety managers at NASA Headquarters, and the Johnson, Marshall, and Kennedy field centers. Together these individuals represented almost all of the top NASA officials responsible for shuttle launch decisions and management of most shuttle manufacturing and processing work. To understand the flow of risk information within shuttle contractor organizations and between NASA and its shuttle contractors, we also held discussions with groups of program and safety managers and working-level engineers at three of NASA’s prime shuttle contractors. We chose the three contractors because the work is among the more complex and highest risk in the program.

Group discussions are very useful for exploring the various facets of communications issues and processes. However, they did not enable us to determine how many participants held a particular view or the intensity of their views. Therefore, to more precisely measure the themes that emerged from the group discussions, we sent a structured questionnaire to the NASA interview participants and some safety representatives who did not participate in the group interviews. Appendixes I through III contain a more detailed discussion of our group interview and survey methodology.

To evaluate NASA’s use of quantitative risk assessment methodologies, we reviewed policies, procedures, and practices related to NASA’s shuttle risk management program and held discussions with senior shuttle managers and NASA’s safety and mission assurance organization. We also discussed the use of quantitative risk assessment methodologies with other federal agencies that are responsible for managing complex systems to establish a benchmark for the use of such methods within the federal government. This work included the Nuclear Regulatory Commission and the Federal Aviation Administration. We also obtained information on the Environmental Protection Agency’s use of quantitative risk assessment in the management of superfund cleanup sites. In addition, we consulted outside experts to obtain their views on the usefulness of quantitative risk assessments to NASA.

We conducted our review primarily at NASA Headquarters, Washington, D.C.; Marshall Space Flight Center, Alabama; Johnson Space Center, Texas; Kennedy Space Center, Florida; Thiokol Corporation, Ogden, Utah; and Rocketdyne Division of Rockwell International, Canoga Park, California.

We conducted our review between June 1994 and December 1995 in accordance with generally accepted government auditing standards.

NASA Has Improved the Shuttle Communications Environment and Needs to Sustain Improvement in the New Management Environment

Good communications is one of the keys to effective risk management. Without adequate information about risks, launch decisions may be flawed as they were in the case of the Challenger accident. Interviews with key shuttle program officials, survey data, and our observations indicate that NASA has been successful in creating communication channels and an organizational culture that encourages people to discuss safety concerns and to bring those concerns to higher management if necessary.

NASA has announced plans to make fundamental changes in the way it manages the shuttle program—turning day-to-day management over to a single prime contractor and reducing direct NASA involvement. Some managers expressed concern about the potential impact of this change, particularly with respect to staffing and organizational restructuring. NASA’s challenge will be to ensure adherence to the communications principles that are essential to promoting shuttle safety.

NASA Has Made Changes to Strengthen Shuttle Risk Management

According to the Presidential Commission, prior to the Challenger accident, project managers for the various elements of the shuttle program felt more accountable to their center management than to the shuttle program organization. As a result, vital program information frequently bypassed the program manager, who was located at the Johnson Space Center. The Commission recommended that NASA give the program manager authority over all program funding and work. In response, NASA centralized program management in a shuttle program director at headquarters with overall responsibility for shuttle operations and budgets. Also, the program manager at the Johnson Space Center was made a headquarters employee in order to minimize center-to-center communications problems. Effective January 31, 1996, however, shuttle program management responsibility was transferred from the headquarters director to the Johnson Space Center director. Because NASA has not yet prepared a detailed plan for implementing this change, we could not fully evaluate its implications. However, according to NASA officials in the Office of Human Space Flight, the Johnson Center director will have full authority over the shuttle resources and work at all participating centers and will report directly to the NASA administrator. NASA has also given astronauts a role in certifying the shuttle for launch and encouraged them to move into shuttle management positions, as recommended by the Presidential Commission.

NASA also established the Headquarters Office of Safety and Mission Assurance under the direction of an associate administrator reporting directly to the NASA administrator. The agency strengthened the safety organizations at its shuttle field centers so that each director of safety and mission assurance reports to a center director rather than the engineering organization. NASA also increased the number of people assigned to the safety organization. In addition, NASA established a safety reporting system to provide an avenue for NASA and contractor personnel to confidentially report problems to safety and program management officials that could result in loss of life or mission capability, injury, or property damage.

NASA Has Improved the Communications Environment

Participants in our discussion groups—both within NASA and in the contractor organizations—described a communication environment that is more open than the one that existed at the time of the accident. Respondents in our follow-up survey portrayed the culture as encouraging contractors and employees to discuss and, if necessary, elevate safety concerns. Discussion groups also identified multiple channels, both formal and informal, for communicating flight safety information. In some cases, these communication channels represent independent, parallel paths for assessing risk. Our own observations and analysis of NASA’s approach to dealing with a recent problem illustrated the openness with which agency officials address safety issues.

Current Culture Encourages Open Discussion of Safety Issues

In group discussions with key NASA and contractor shuttle managers and contractor working-level engineers, we asked them to assess conditions related to the flow of safety information to top management. All of the groups reported that the shuttle program’s organizational culture encourages people to discuss safety concerns and bring concerns to higher management if they believe the issues were not adequately addressed at lower levels. As one manager noted, because of the complexity of the shuttle program, open communication, group discussions, and the sharing of information are essential to flight and work place safety.

NASA managers at the three field centers with primary responsibility for managing shuttle elements and at NASA headquarters reported having taken steps to create an organizational environment that encourages personnel at all levels to voice their views on safety to management. One manager noted that people are not afraid to surface their mistakes to management when they discover mistakes have occurred. Another manager said, “If . . . I got the idea that I had a manager in the system who wasn’t allowing their people to feel comfortable in bringing things, probably that’s the time I think I would change that person’s job because . . . our people need to feel that they can come without attribution and talk about what they need to talk about.”

Managers in each group we interviewed cited various techniques they use to create an organizational environment that encourages personnel at all levels to voice their professional viewpoints on safety issues to management, even if dissenting. For example, managers invite people to express their concerns by trying to keep every line of communication open and telling people that bringing up a problem does not reflect poor performance; holding extensive dialogue over shuttle safety issues, beginning early in the problem identification stage, so that everyone fully understands the issues; encouraging people to come in or call their managers if they want to talk about a safety concern, no matter how small the issue; and not only encouraging, but expecting, open expression of professional differences at all levels.

The contractor managers also described a working relationship with NASA that they believe encourages open communication and the elevation of safety concerns. They described the flow of information between NASA and shuttle contractors as continual, open, and comprehensive. From their perspective, daily contact between contractor and NASA working-level personnel contributes to the exchange of information. Contractor support to and participation in flight readiness reviews and other shuttle processing meetings, and their reporting of safety information directly into NASA’s centralized information systems are among the other mechanisms that achieve that exchange.

One manager noted that the Challenger accident prompted a change in his contractor’s management approach. Before the accident, company meetings were closed to the NASA site representatives. Since the accident, NASA representatives attend all technical meetings. Managers from two other contractors said that they would not hesitate to go to the highest levels of NASA management to ensure that safety issues received appropriate attention.

Contractor working-level engineers portrayed their organizations as supportive of engineers elevating shuttle safety issues and concerns to management. For example, at one contractor facility, program teams are structured so that minority opinions about the handling of safety problems can be elevated to a higher level board. At another contractor facility, the work environment was described as one that encourages debate, discussion, and never keeping a safety concern quiet. At the third contractor plant, the formal reporting process ensures that NASA and contractor managers are continually apprised of issues, review how issues are resolved, and can request more work if they do not agree with the resolution of a safety issue.

The managers and safety representatives who responded to our survey also gave very favorable ratings to NASA’s current communications culture. For example, 90 percent of those responding to the survey said that to a great or very great extent NASA’s organizational culture encourages civil service employees to discuss safety concerns with management.

As shown in figure 2.1, more than 80 percent of the respondents to our survey rated the following current shuttle communications and information flow conditions very favorably.

As part of our review, we attended numerous certification of flight readiness and prelaunch assessment reviews for shuttle mission STS-64, including the flight readiness review and launch. We observed open and candid discussions, debate of issues, and a structure that required the recording and follow-up of unresolved issues. At most reviews, presentations appeared thorough and participants asked many probing questions to ensure they had an adequate understanding of the issues being briefed. If participants did not believe they adequately understood an issue or additional work was required to resolve an issue, it was listed as an open item to be resolved prior to launch.

Numerous Communications Paths Are Available

Managers, safety personnel, and working-level engineers described shuttle program and contractor procedures and structures that provide multiple avenues for continual communication with contractors, across centers, and with headquarters to discuss safety issues. These avenues include the certification of flight readiness process, daily telephone conferences, and weekly meetings. In response to our survey, almost all NASA program managers and safety representatives believe the opportunities to discuss and communicate shuttle issues and concerns meet, or even exceed, the needs of the program in terms of the number of forums held and the types and levels of expertise represented.

The certification of flight readiness process requires the involvement of all centers and projects on issues that could affect safety or mission success. In preparation for a launch, NASA relies on a number of reviews to ensure that the shuttle is safe for flight. These reviews are designed to ensure compliance with requirements, that prior problems/failures have been corrected, planned work has been completed, and operational support is in place for the mission.

Managers also reported other, sometimes less formal, channels for communicating safety information. For example, the shuttle program manager holds an early morning telephone conference daily, enabling NASA managers at headquarters and the centers to discuss problems and draw upon the experience of others. The manager of launch integration also conduct a daily “noon board” telephone conference to discuss shuttle issues, status, and required changes related to vehicle processing at the Kennedy Space Center. Project representatives from the various shuttle centers participate if the issue involves their shuttle element. Also, NASA’s shuttle program manager chairs a weekly Program Requirements Control Board meeting that is the controlling authority for all changes to the shuttle program baseline. Safety and mission assurance engineers participate in all of these meetings. Further, NASA safety and project representatives at contractor plants help ensure a continual flow of information on contractor issues. In addition, the NASA Safety Reporting System (an anonymous reporting system) provides another opportunity for people to report safety concerns.

In addition to taking part in all of the program and project reviews for the certification of flight readiness, NASA’s Office of Safety and Mission Assurance conducts prelaunch assessment reviews of all major shuttle elements. The office’s System Safety Review Panel also conducts several reviews, including a review of in-flight anomalies from previous missions. These safety office reviews are conducted independently of the project offices responsible for the various shuttle elements. Results of the safety office reviews are presented at the flight readiness review. The safety organization continues to monitor shuttle missions up to and during launch. Figure 2.2 illustrates the parallel assessments by safety and mission assurance and the shuttle program and project offices.

We asked contractor working-level engineers what avenues are open to them to communicate their views in the event that they disagree with a safety decision made at higher levels of management, either within their organization or within NASA. A variety of communication routes were cited: a company ombudsman, the firm’s safety manager, NASA counterparts, or higher levels of management within the contractor’s organization and the NASA Safety Reporting System.

Not Complete Agreement on Type and Amount of Information Needed

While there was a high level of agreement that the current culture encourages and enables contractors and employees to discuss safety issues and concerns, there was not universal agreement about the kinds of risk information needed for final launch decisions. We asked NASA managers and safety representatives to designate the types of safety issues that should always be briefed in detail to corporate-level management at the final flight readiness review. Seven of the 15 types of issues we asked about were widely endorsed as needing the board’s review; however, opinions were divided in other areas. For example, the views of the board members tended to differ from those of the other managers and safety representatives regarding whether hazards and new waivers should always be briefed in detail. Opinions were also divided about the level of detail that should be provided when there are changes that affect procedures or processes involving the flight crew, operations, software, or shuttle hardware.

We also observed differences in the amount of detail provided during two flight readiness reviews. At the first review, we observed that the review board’s chairman required less detail about issues and concerns than at the second review. The second review meeting we observed was chaired by a different official. This official requested a greater level of detail about issues being discussed. Thus, the change in personnel caused some initial confusion about the type and amount of information needed to make corporate-level launch decisions.

NASA Demonstrated Openness in Dealing With Recent Motor Joint Issue

To provide a better understanding of the cultural and communication path changes within NASA, we compared NASA’s approach to handling the motor joint issue at the time of Challenger with a recent issue concerning another joint in the solid rocket motor. On two successive flights in 1995, hot gas penetrated beyond the joint’s sealer compound and made very small singe marks on the joint’s primary o-ring. NASA was more cautious in its approach to handling the latest motor joint problem. For example, NASA immediately halted shuttle launches and publicly aired the problem. NASA held weekly press meetings to discuss the problem and progress in correcting it. Shuttle and contractor managers at all organizational levels were heavily involved in the issue and the safety organization provided an independent assessment of the problem. NASA did not resume shuttle launches until it was confident that the problem was understood and corrected. Table 2.1 describes our observations.

Some Managers Concerned About Future Changes

Some discussion group participants told us they are concerned about the impacts of continued cost reductions and planned program changes. Over the next 5 years, plans call for NASA to make significant additional reductions in shuttle costs while maintaining the capability to meet the demanding schedule for international space station assembly and support. Although final decisions have not been made, NASA has initiated a number of actions to further reduce shuttle operation costs, including turning shuttle operations over to a single prime contractor. Some participants in our discussion groups expressed concern about the effect of continued cost reductions and the transition to contractor management of the program.

In July 1995, we reported on the schedule pressures created by the International Space Station assembly requirements. Based on our own analysis and internal NASA studies, we concluded that the shuttle’s ability to meet station launch requirements appeared questionable. To meet the station’s “assembly complete” milestone, shuttle officials had designed a very compressed launch schedule. During certain periods of the station assembly, clusters of shuttle flights are scheduled to be launched within very short time frames. For example, the schedule calls for five launches within a 6-month period in fiscal year 2000 and seven launches during a 9-month period in fiscal year 2002. Because the schedule is so compressed at times, there is very little margin for error. There is little flexibility in the schedule to meet major contingencies, such as late delivery of station hardware, or technical problems with the orbiter.

We reported in June 1995 that NASA had reduced shuttle operations funding requirements by a cumulative amount of $2.9 billion between fiscal years 1992 and 1995 when the fiscal year 1992 budget request is compared to the fiscal year 1995 request. In our survey, we asked NASA managers and safety representatives what actions had been taken to accommodate the funding reductions and whether these actions, in their opinion, had enhanced, degraded, or had little or no effect on the accuracy, completeness, and timeliness of shuttle safety-related information. Generally, their assessment was that the actions either had little or no effect on quality or somewhat degraded quality. For example, of nine respondents who reported funding reductions accomplished by delaying safety improvements, six said the delay somewhat degraded the quality of safety-related information. However, some respondents reported actions taken to cut costs actually enhanced the quality of information.

Just over 75 percent of NASA managers and safety representatives we surveyed believed that NASA emphasized safety over shuttle schedule to a great or very great extent. Figure 2.3 illustrates NASA managers and safety representative responses to our survey question on the extent to which program priorities place greater importance on safety than on meeting schedule.

Just over 60 percent of NASA managers and safety representatives we surveyed believe that to a great or very great extent NASA emphasizes safety over reducing cost. Figure 2.4 illustrates responses to our survey question on the extent to which program priorities place greater importance on safety than on cost reduction.

Contractor managers and working-level engineers also told us that past funding reductions had not affected the quality of safety-related information they develop. According to the contractor managers, reductions in the shuttle flight rate and various contractor productivity enhancements have enabled them to accommodate past personnel cuts without, they believe, sacrificing the quality of shuttle information they develop.

Some working-level engineers in the group interviews cited a variety of concerns about the effects of funding reductions. For example, the engineers said (1) investigations of lower priority issues take longer to complete because there is not enough time to devote to them, (2) keeping people with the required skill level is a concern, and (3) there is a lack of storage in automated databases to archive safety information. In addition, some engineers told us that the funding reductions have adversely impacted employee morale because people are being asked to accomplish more with fewer resources and some employees fear losing their job. Some engineers said, however, that although morale was lower, they did not believe it adversely affected flight safety.

In November 1995, the Associate Administrator for Space Flight testified that NASA plans an additional $2.5 billion cumulative reduction from total shuttle funding requirements in fiscal years 1996 through 2000 against the fiscal year 1996 budget request. According to the Associate Administrator, the program will achieve the budget reductions through restructuring and other workforce and content reductions.

Both NASA and contractor managers in our discussion groups expressed concerns about how they would cope with additional funding cuts. For example, the project managers for two contractors said that workforce reductions can impact their timeliness in responding to situations that arise. One contractor manager noted that while the company measures various indexes such as “first time quality” and overtime, it is difficult to specify the point at which additional program changes to accommodate funding cuts might reduce quality. Another contractor manager noted that at some point, funding reductions could translate into not having enough people, so that maintaining the required quality will mean continual schedule delays—a signal to the contractor that their program cannot be reduced further.

Although firm estimates are not available, NASA expects to achieve significant cost savings by turning shuttle operations over to a prime contractor. The contractor would be responsible for shuttle processing and launch, but NASA will retain the responsibility for making the final launch decision. The single prime contractor would combine many of the tasks now performed under 28 separate shuttle program contracts. Savings are expected to accrue because shuttle operations would be more efficient and require fewer civil service employees. Current plans are to award the contract by fiscal year 1997.

During our discussion groups, some NASA managers expressed concern about the transition of shuttle operations to a single prime contractor. They feel that over the years NASA has assembled an expert shuttle operations team and there are many unknowns about making a transition to a new way of doing business. For example, the safety and mission assurance organization maintains independent oversight of shuttle operations. NASA’s projections are that the quality assurance oversight role will be reduced under the single prime contractor concept of operations. Although managers expressed concern about transitioning to a single operations contractor, in response to our survey, 76 percent of the managers and safety representatives said that quality assurance inspections and reviews should be decreased. According to NASA, there will continue to be independent oversight and the agency has plans to assure that the oversight/insight will be properly focused with the reduced level of resources expected.

NASA will retain decision authority and direct oversight over work that is considered out-of-family (those events/activities that may contain a level of risk beyond the known and accepted level). In addition, NASA will retain the developmental effort for new hardware. This work will transition to the single prime contractor, but only after all the unknowns are understood by NASA. Further, NASA will return to an oversight mode when there is an indication that there is an increase in the understood level of risk for any reason. The single prime contractor will be required to propose a process for performing risk assessment and to demonstrate that they are able to institute and properly manage the process. This includes the process for keeping NASA informed of issues that have the potential for increasing risk.

Principles Guiding the Communications Process

Through our discussion groups, individual interviews, and observations, we identified several management principles related to communication and information flow that appear to guide shuttle communications. We also identified additional management principles that we believe are essential to promoting shuttle safety in the future. In our survey, we listed these principles and asked NASA managers and safety representatives to identify those guiding principles that they believe are essential to promoting shuttle program safety as NASA deals with budget constraints, associated downsizing, and restructuring in the near term, and with continuation of shuttle flights in the long term. A large percentage of managers and safety representatives we surveyed agreed that the following principles are essential to promoting shuttle safety.

The organizational environment and structures for both contractor and NASA personnel encourage timely, open discussion and debate to ensure managers have the benefit of all relevant knowledge of shuttle program issues.

Managers (civil service and contractors) stress safety over schedule and cost and those managers foster these values among employees.

The organizational environment encourages people (civil service and contractor) to elevate concerns to higher management if they believe the issues were not adequately addressed at lower levels.

The working arrangement between NASA and contractors ensures agency managers obtain continual knowledge of problems and issues so that appropriate decisions can be made.

Organizational mechanisms enable NASA corporate-level managers to carry out their decision-making responsibilities for certifying readiness for flight.

NASA uses the most appropriate analytic and quantitative methods available to assess shuttle risks and conduct sufficient assessments and reviews to carry out the agency’s oversight of shuttle work processes.

Management information systems, including databases, are accessible, accurate, complete, and timely for shuttle program oversight and decision-making.

The NASA environment is a self-evaluative one that monitors its effectiveness in communication and information flow and seeks ways to improve it.

In addition to the principles previously listed, some NASA managers provided additional principles that they believe are essential to promoting shuttle safety as NASA deals with budget constraints, downsizing, and restructuring.

Management of changes in the program receives adequate attention and time to ensure that (1) program priorities are adhered to, (2) government and contractor responsibilities for the reporting and resolution of safety-related issues are clearly defined, and (3) changes to the shuttle program are appropriately evaluated before implementation.

Appropriate training is conducted to ensure that personnel can effectively and efficiently carry out their work when changes in program operations, processes, and staffing occur.

Morale and the working environment of employees are considered key elements in assuring a safe and quality program.

Prime contractor management methods ensure quality of subcontractor work.

Conclusions

NASA has created an organizational culture that encourages shuttle program and contractor employees at all levels to bring safety concerns to the attention of NASA’s top management. NASA has also established policies and procedures to ensure the free flow of needed safety-related information. However, in response to our survey, some shuttle program personnel expressed concern about whether NASA might be emphasizing cost reductions over flight safety as planned budget reductions and operational changes occur. Also, in response to our survey, several types of issues were endorsed as always needing the flight readiness review board’s attention. However, opinions were divided in other areas, suggesting that managers and safety representatives may not be clear on each other’s expectations about the issues that should always be briefed. If, as is likely, the planned shuttle operations contractor assumes more of the burden of providing information to the flight readiness review, it will be important to clearly specify the type and level of detail of information to be provided.

NASA has adopted certain management principles that help guide the shuttle launch decision process. These include such steps as stressing safety over schedule and cost and developing an organizational culture that encourages both contractor and NASA personnel to elevate concerns to higher management if they believe the issues were not adequately addressed at lower levels.

Recommendations

We recommend that the Administrator of NASA identify guiding principles of good risk management, such as those contained in this chapter, and ensure that terms and conditions of the planned shuttle operations contract reflect these principles.

We also recommend that the Administrator take steps to ensure that flight readiness review participants understand and agree on the minimum issues that should always be discussed at the review and the level of detail that should be provided.

Agency Comments

In commenting on a draft of this report, NASA agreed with our first recommendation and stated that the agency is taking steps to implement it. According to NASA, the shuttle flight operations contract request for proposal and statement of work have been carefully reviewed and these documents reflect the principles of good risk management described in this report. NASA said that it will ensure that the contract terms and conditions are compatible with these principles.

Regarding the second recommendation, NASA said that it is appropriate and the agency has recently completed an activity to update and clarify the roles and responsibilities of each program element and organization relative to the flight readiness review. The new procedure is to be fully implemented in support of shuttle flight STS-78 in June 1996.

We made additional changes to the report, where appropriate, based on NASA’s technical comments.

NASA Has Not Developed an Overall Strategy for Using Quantitative Risk Assessment Methods in the Shuttle Program

The National Research Council recommended in 1988 that NASA apply quantitative risk assessments to the shuttle program. However, NASA still relies primarily on qualitative methods to assess and prioritize significant shuttle risk. This approach relies heavily on the judgment of shuttle engineers to identify significant risk items that could cause loss of a shuttle or crew. Although NASA awarded a contract for development of a quantitative method model known as a probabilistic risk assessment for the shuttle program, NASA has not fully assessed the potential benefits of using the tool in routine shuttle decision-making. The agency also has not developed an overall strategy for assuring use of this method where it is appropriate. In addition, databases are not always timely, complete, accessible, or reliable enough to be used in these type analyses.

National Research Council Recommended Quantitative Approach

The National Research Council investigation of NASA’s risk assessment approach following the Challenger accident found that quantitative assessment methods had not been used to directly support NASA decision-making related to the space shuttle. The Council recommended that probabilistic risk assessment approaches be applied to the shuttle at the earliest possible date. They also recommended that databases be expanded to support probabilistic risk assessments, trend analysis, and other quantitative analysis and that NASA develop a statistical sciences capability to perform necessary risk assessments.

Quantitative Methods Used in Other High-Risk Areas

Quantitative methods, such as probabilistic risk assessments, have been used in the decision-making process by other federal agencies involved in high-risk ventures. For example, the Nuclear Regulatory Commission uses probabilistic assessments in its regulation and oversight of nuclear power plants. These techniques are used to assess the safety of operating reactor events and as an integral part of the design certification review process for advanced reactor designs. Commission officials stated they have found probabilistic risk assessments to be an effective tool for making plant-by-plant examinations to determine areas needing more emphasis, such as how long it takes a utility to respond to problems. Commission officials told us that, in their experience, probabilistic risk assessments can help identify and focus their attention on risk areas that require the most resources.

The Environmental Protection Agency uses quantitative risk assessments to determine the health risks posed by superfund hazardous waste sites. The agency reviews contaminated sites for investigation and cleanup. One element of the investigation is a baseline risk assessment—an evaluation of current or potential threat to human health. The evaluation establishes probabilities that are used to decide whether a site requires cleanup. For example, if the risk of humans developing cancer from site chemicals is greater than 1 in 10,000, Environmental Protection Agency policy requires that the site be cleaned.

NASA pointed out that it is important to make a clear distinction between quantitative risk assessments in general and the specific probabilistic risk assessment method when determining the value of applying these methods to space hardware issues. NASA said it recognized that probabilistic risk assessments had proven valuable at the Nuclear Regulatory Commission and the Environmental Protection Agency. However, this method did not have comparable utility at NASA. Reactor design and certification risk assessments are based on failure rates compiled from hundreds of plants and facilities while the shuttle has significantly less hard data available to quantify risk. In addition, NASA said the public health risk posed by nuclear power plant accidents or toxic waste sites argues for a multimillion dollar investment in risk assessment that can span years of analysis. In contrast, according to NASA most shuttle risk issues must be resolved in a shorter time frame.

NASA’s Response to the Council’s Recommendation

In response to the Council’s interim report, NASA began taking tentative steps toward the use of probabilistic analysis by initiating contractor trial probabilistic risk assessments of some shuttle elements. In parallel with this, NASA began developing a procedure to prioritize the shuttle’s highest risk elements. This proposed technique would lend itself to the incorporation of quantitative measures of risk and probabilities of occurrence as these measures were developed. NASA planned to assess the benefits and applicability of this method to the shuttle risk management process based on the results of the contractor studies. A former Associate Administrator for Safety and Mission Assurance indicated that he would personally evaluate the probabilistic risk assessment technique and develop a strategy for introducing it throughout NASA. However, the strategy has not yet been developed.

Regarding its databases, NASA responded by developing a centralized database designed to improve the quality of information by providing an integrated view of the status of shuttle problems in near real time. The Council recommended that development of this system be given a high priority. NASA developed a database to provide information, but, as discussed in other sections of this chapter, that database has limitations.

NASA officials told us that while some progress has been made, the use of probabilistic methods have not reached a mature state at NASA. NASA has made limited use of probabilistic risk assessments of the shuttle, including proof-of-concept studies, assessment of some specific shuttle systems, and required assessments of accident probabilities for launches involving radioactive material. A 1994 survey of probabilistic methods used in structural design, which included some shuttle projects, found that there is no agreed-upon approach across centers for preferred methods, practices, or software and that the various quantitative tools have not been fully examined, evaluated, and accepted by NASA centers.

In early 1994, the NASA Administrator and the Office of Space Flight concluded that a probabilistic risk assessment of shuttle risk was needed to guide safety improvement decision-making. According to a safety official, NASA contracted with Science Applications International Corporation in January 1994 to conduct a probabilistic risk assessment of the space shuttle. This was the first assessment to include a complete shuttle mission. The contractor was required to develop and apply a risk model of the shuttle during flight and to quantify in-flight safety risk. The analysis was to identify, quantify, and prioritize risk contributors for the shuttle. According to the model’s author, secondary objectives were to provide a vehicle for introducing and transferring probabilistic risk assessment technology to NASA, and to demonstrate the value of the technology. The model was completed in April 1995.

According to the contractor who developed the probabilistic risk assessment model, the model could be a useful tool in NASA’s management of shuttle risks. For example, the model might be used to establish realistic cost objectives for redesigning the high risk components, helping to assure that limited resources are focused toward solving those problems that will have the most impact on safety. The National Research Council also noted that a detailed quantitative risk assessment provides decision makers with a better basis for managing risk.

An internal shuttle program survey of managers, safety experts, and senior engineers revealed mixed reactions to the model. Although generally positive, respondents cited some concerns. For example, some respondents commented that more use of actual failure data would have benefited the analysis and that some assumptions used were debatable. Some found fault with the excess use of expert opinion and the lack of thoroughness in delineating certain assumptions. Following the survey, the Deputy Associate Administrator for Space Flight informed shuttle and safety managers that they should feel free to use the report and model as a “limited tool in the risk management tool box.”

According to some NASA safety officials, the model has not been routinely used by NASA personnel as a risk assessment tool because officials are still evaluating the utility of the model and barriers exist to its use by NASA employees. For example, there is no instruction manual for using the model and it requires use of contractor owned software. According to safety officials, NASA does not have current copies of the required software and older inadequate versions are limited. According to these officials, only one NASA employee has been able to use the model on a NASA computer using the older software. In addition, no firm decisions have been made regarding maintenance and update of the model to reflect shuttle changes, such as the super light weight external tank. Safety officials stated they are continuing to assess the model to determine its utility within NASA.

NASA project and safety officials compile a list of significant shuttle risk issues for each project to target resources and manage risk reduction efforts. Only risks that can be reduced by incorporating hardware or procedure modifications are included in the assessment. According to NASA’s April 1995 shuttle safety risk ranking methodology guidance, the source of risk information currently used in the rankings is qualitative and the process ranks catastrophic events by judgmentally derived prioritization matrices. The guidelines state that many comparisons of catastrophic events could be made but are sometimes subjective, emotional, and rely on different techniques. A complete probabilistic risk assessment would be the most desirable analysis, according to the guidelines, but probabilistic analyses are labor-intensive efforts that require many system experts, a complete understanding of the methodology, and proper management of the effort.

Limited Use of Quantitative Risk Assessments

NASA has made limited progress in adopting the National Research Council’s recommendations that the agency assess risk with quantitative methods, such as probabilistic risk assessments. NASA uses a variety of methods to assess shuttle risk issues, and efforts are underway to increase the use of quantitative methods. Qualitative methods are still widely used when risk issues are thought to be well understood. NASA has made limited use of the classical probabilistic risk assessment method of analysis. Cost, lack of specific expertise, and lack of data are the reasons cited for limited use.

According to shuttle and safety managers, lack of a strategy for incorporating the methods into decision-making processes has impeded NASA’s progress in adopting the National Research Council’s recommendations on risk assessments. Also, insufficient expertise exists at NASA to conduct specific quantitative analyses, such as probabilistic risk assessments.

NASA project and safety officials told us that progress in implementing quantitative risk assessment methods has been impeded because NASA does not have a working strategy for formalizing these methods for the shuttle program. Such a strategy would include clear and measurable goals, resource requirements, assessments of current utilization and skills within NASA, and training needs, including the need to learn by doing selected projects. Without this focus, projects and safety organizations are skeptical about the cost and benefits of using the probabilistic risk assessment model.

Project and safety officials at several centers expressed concerns about the applicability of probabilistic risk assessments to the shuttle program. While officials stated they recognize probabilistic risk assessments could be used as an effective additional tool to assess risk, they see a need for more training on the methodology and the need to learn by doing selected projects. Several stated they do not have the resources needed for this type analysis but are stretched just to operate their programs. Several officials stated they believe there is a lack of trust in the probabilistic risk assessment method because people do not understand it. Many officials expressed concern about the complexity of the shuttle probabilistic risk assessment model, the lack of good data, and the dependence upon the contractor to make needed changes to the model. Several officials commented that NASA needs a “champion” at headquarters to provide a focused effort to emphasize use of these tools when appropriate.

NASA headquarters safety and mission quality officials stated they have not developed a master plan for formalizing quantitative techniques within NASA or made the progress they would like in this area. However, steps are being taken to address several of the concerns expressed by project and safety officials at the centers. For example, training courses in risk management and assessment are being planned that will be offered to safety and other NASA personnel. Reference manuals on sources for data and techniques on risk assessments are under contract. According to NASA safety officials, the first effort to develop these type documents began in 1989 but was unsuccessful and the documents were not published. However, NASA has established a coordination committee to develop a standard, comprehensive approach to introduce structural design methods that can be used in the shuttle program.

NASA is also trying to give this issue visibility as the agency plans to move to a single prime contractor and to assure that the statement of work contains provisions that the contractor use quantitative risk assessment techniques where appropriate.

According to the National Research Council, decisionmakers within NASA must be supported by people skilled in the statistical sciences to aid in the transformation of complex data into useful information. The Council recommended that NASA develop a staff of experts in these areas to provide improved analytical support for risk management. NASA officials at several centers and at NASA Headquarters told us they lack sufficient personnel with these skills, and in one case, a center lost needed contractor skills that caused the delay or termination of a planned analytical project. A 1994 NASA survey of probabilistic methods used in structural design work found that a wide variance of knowledge exists at the centers and that a majority of working-level engineers are not familiar with and do not use probabilistic methods.

NASA’s Databases Do Not Provide Sufficient Information for Quantitative Analysis

Another factor that has hindered development of quantitative methods of risk assessment is that NASA’s databases do not always provide timely, accessible, accurate, and complete information. A large percentage of managers and safety representatives we surveyed believe that NASA should provide management information systems, including databases that are accessible, accurate, complete, and timely for shuttle program oversight and decision-making. However, more than half assessed NASA’s current management information systems as needing improvement.

NASA has developed automated database systems to provide shuttle data used in decision-making. One system, called the Program Compliance Assurance and Status System, is a central database designed to integrate existing data, such as in-flight anomalies, from various sources in the program. Another system, the Problem Reporting and Corrective Action system, provides data to the central system and is designed to document and track problems in the program.

According to NASA officials, the Program Compliance Assurance and Status System is neither timely nor fully utilized. The system is cumbersome to use because it is based on older technology, some trend and other data is not centralized in the system, and software needed to convert contractor data to NASA database format has not been developed. Program officials told us they maintain trends on some aspects of the shuttle program, but have found the centralized system to be difficult to use and not compatible with other existing databases. The officials stated that the required conversion programs have never been developed to input some contractor data into the system. In some cases, safety officials must obtain data directly from contractors to conduct quantitative risk assessments. Because the system is hard to use in real-time and the data is not always current, some officials stated they are using a different software program with faster computers to access and correlate data more rapidly.

A January 1995 internal report on shuttle problem reporting system data integrity at two centers found missing criticality codes on thousands of entries. Blank entries could, therefore, be interpreted as either not applicable or inadvertently omitted. A NASA Headquarters official was not aware of any corrective action on this matter. Officials told us that the Problem Reporting and Corrective Action System records are often not reliable, lack data needed for quantitative risk assessments, and lack uniformity in categorizing problems. The system also contains entries that may not meet the definition of a “real problem.” NASA safety officials acknowledged that the system needs improvement but stated no firm decision has been made regarding the extent of improvements pending the transition to a single prime contractor.

Conclusions

NASA has made limited progress in adopting the National Research Council’s recommendation that the agency assess risk with quantitative methods, such as probabilistic risk assessments. NASA officials, for the most part, rely on qualitative methods for assessing risk in the shuttle program when they believe risk issues are well understood. Although some progress has been made, NASA lacks an overall strategy with focused management emphasis to incorporate methods, such as probabilistic risk assessments into the shuttle program, when appropriate. Resource constraints and specific expertise are cited as barriers to increased use of these methods. In addition, NASA’s databases need improvement and are not fully utilized by decisionmakers nor are they adequate to support the use of quantitative risk assessment methodologies.

Recommendations

We recommend that the Administrator of NASA establish a strategy, to include specific milestones, for deciding whether and how quantitative methods, such as probabilistic risk assessments, might be used as a supplemental tool to assess shuttle risk.

We also recommend that the Administrator assess the shuttle program’s centralized database, as well as other databases, to insure that data required to conduct risk assessments and inform decisionmakers is accessible, timely, accurate, and complete.

Agency Comments

NASA agreed with the need to establish a strategy, with milestones, for incorporation of quantitative risk assessment methods into the shuttle’s risk management program. According to NASA, the agency will establish a team to develop the strategy.

NASA also agreed that the shuttle program’s centralized databases need to be assessed. In this regard, NASA will form a team of engineers to thoroughly examine the Program Compliance Assurance and Status system. The team will be tasked to determine the adequacy of what presently exists and make recommendations for improvements as necessary. The assessment team will report to the shuttle program manager. In addition, the Problem Reporting and Corrective Action System is being examined at each center by a reengineering team. This team is searching out deficiencies and will recommend needed improvements that must be implemented by the shuttle flight operations contractor.

We made additional changes to the report, where appropriate, based on NASA’s technical comments.