Aviation Safety: Safer Skies Initiative Has Taken Initial Steps to Reduce Accident Rates by 2007
Published: 2000-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Federal Aviation Administration's (FAA) Safer Skies Initiative, focusing on: (1) to what extent addressing the safety problems to be addressed by the initiative will help reduce the fatal accident rate; (2) what progress the initiative has made in identifying and implementing interventions to address each of these safety problems; (3) what progress has been made in assessing the effectiveness of those interventions; and (4) how FAA is coordinating the Safer Skies initiative with other safety activities conducted throughout the agency, in partnership with the aviation industry, and by other federal agencies.
section_title: What GAO Found
paragraphs: GAO noted that: (1) the Safer Skies initiative addresses the safety problems that have contributed to fatal accidents in the past, and in conjunction with other safety problems, it can be expected to reduce the fatal accident rate and thus enhance the safety of the nation's air passengers; (2) in commercial aviation, the initiative addresses safety problems that accounted for over three-quarters of the fatal accidents in those operations in 1988-1997; (3) in general aviation, the Safer Skies initiative plans to address safety problems that appear to be the most common causes of fatal accidents; (4) the initiative has adopted a less aggressive goal in general aviation of reducing the number of fatal accidents to 350 in 2007, which represents about a 20-percent reduction; (5) the initiative addressed four safety problems in cabin safety; (6) to date, safety improvement efforts by FAA and the initiative have focused on reducing the causes of past accidents and incidents, which may not be entirely predictive of future ones; (7) as of April 1, 2000, Safer Skies teams had started work on 13 of the 16 safety problems and had begun implementing interventions for 5 of these--2 in commercial aviation and 3 in cabin safety; (8) since most of the interventions developed under the Safer Skies initiative are in early implementation stages, little progress has been made in evaluating their effectiveness; (9) of the five Safer Skies teams that have begun implementing interventions, only one has developed a performance measure to evaluate whether the interventions it has selected are helping to reduce the safety problems that cause fatal accidents and are worth what they cost; (10) FAA has coordinated extensively with aviation experts from industry, other federal government agencies, and its own staff, but GAO's review identified three coordination problems that could undermine the implementation and evaluation of Safer Skies' interventions; (11) although FAA officials have repeatedly committed to funding interventions agreed upon by all parties, skepticism still exists among some participants as to whether this commitment can or will be honored; (12) furthermore, if funding is limited, it remains unclear what process will be used to reprioritize available resources to ensure funding for interventions that emerge later but have greater potential for reducing the fatal accident rate; and (13) the Safer Skies initiative, FAA, and the Department of Transportation (DOT) have not agreed on how they will measure progress in achieving the accident reduction goal for commercial aviation.

section_title: Letter
paragraphs: As requested, we are reporting on the actions taken by the Federal Aviation Administration (FAA), through the Safer Skies initiative, towards the goal of reducing the nation’s fatal aviation accident rates by 2007. Our report contains recommendations designed to ensure that the implementation of interventions approved by FAA and the Safer Skies steering committees is tracked and that the interventions are evaluated for their effectiveness in meeting the goal.
As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 7 days after the date of this letter. At that time, we will send copies to o the appropriate congressional committees; the Honorable Rodney E. Slater, Secretary of Transportation; the Honorable Jane F. Garvey, Administrator, FAA; and the Honorable Jacob Lew, Director, Office of Management and Budget. We will also make copies available to others upon request.
If you have any questions about this report please contact me or Robert White at (202) 512-2834. Other key contributors to this report are listed in appendix I.
subsections: section_title: Background
paragraphs: The Federal Aviation Administration (FAA) has forecast continued growth for commercial and general aviation over the next decade. Growth over the past few decades brought innovations to improve flight safety that contributed to a dramatic lowering of the accident rate by the mid-1970s. Further reductions in the accident rate have, however, remained elusive. Unless the current accident rate can be reduced, the number of fatal accidents is likely to increase as aviation operations continue to grow. During the 1990s, FAA, the aviation industry, and the Congress all acknowledged and studied this potential danger. They set ambitious targets for reducing the accident rate, made over a thousand recommendations for improving aviation safety, and implemented a number of safety initiatives. In spite of these efforts, the accident rate, which is already low, has remained fairly steady. The FAA Administrator, White House and congressional task forces, and aviation industry groups have concluded that FAA and the aviation industry must coordinate their efforts to prioritize safety recommendations and focus resources on those with the most potential to decrease the accident rate. In 1998, the FAA Administrator announced the Safer Skies initiative, a joint government- industry effort to identify and address the greatest threats to aviation safety in order to reduce the fatal accident rate by 80 percent by the year 2007.
subsections: section_title: FAA Expects Continued Growth in Aviation
paragraphs: Over the past several decades, aviation has grown substantially in the United States, and FAA expects this growth to continue into the next century. Commercial aviation has grown consistently since 1982, while growth in general aviation has been less consistent. One key measure of aviation activity shows that the number of flight hours for commercial aircraft more than doubled from 8 million hours in 1982 to nearly 18 million hours in 1999. In contrast, general aviation activity dropped fairly steadily from the early 1980s until 1995. While general aviation has grown since 1995, it has not yet returned to 1990 levels. The number of general aviation flight hours decreased by nearly 9 percent from 32.6 million hours in 1982 to 29.9 million hours in 1999. (See fig. 1.)
FAA has forecast continued growth for commercial aviation as well as for general aviation into the next century. The number of planes will increase, and these aircraft will fly more miles, spend more hours in the air, and carry more people. For example, FAA estimates that commercial aviation flight hours will grow to 24 million hours in 2007—an increase of 37 percent from 1999. In commercial aviation, FAA projects that the use of large air carriers will grow at an annual rate of 4 percent, while the use of commuter air carriers will grow at 3 percent per year. Although growth has been more erratic in general aviation than in commercial aviation, FAA projects an annual growth rate of 2.2 percent for general aviation into the next century. FAA estimates that general aviation flight hours will increase to about 36 million hours in 2007, a growth of nearly 19 percent over 1999.
section_title: Fatal Accident Rates Have Decreased for U.S. Aviation
paragraphs: Even with the growth in aviation, fatal accidents remain relatively rare, especially in commercial aviation. Fatal accident rates for U.S. aviation are low and have decreased over the past decades for both commercial and general aviation. The fatal accident rate can be calculated as the number of accidents with one or more fatalities divided by a measure of aviation activity, such as the number of aircraft miles flown, aircraft hours flown, or departures.
subsections: section_title: More Fatal Accidents Occur in General Aviation, but Commercial Aviation Accidents Can Be Catastrophic
paragraphs: In the 10-year period preceding the initiative, 4,471 fatal aviation accidents occurred in the United States, resulting in a total of 9,802 deaths. Table 2 shows the distribution of accidents and deaths for commercial aviation, which includes large and commuter air carriers, and general aviation, which includes on-demand air taxis. General aviation accounted for the largest number of fatal accidents and deaths in 1988-97. The initiative addresses both commercial and general aviation, but increased attention is focused on further improving the safety of commercial aviation because large and commuter air carriers are the primary forms of air transportation for most Americans. While fatal commercial aviation accidents are rare, large airplane accidents can cause more deaths in an instant than most events, other than wars or natural disasters. They consequently raise concerns with both the public and the media, and commercial aviation is held to a higher standard of safety than other forms of transportation. With commercial aviation expected to grow steadily into the next century, aviation accidents will occur with a frequency that will be unacceptable to the public unless steps are taken to decrease the fatal accident rate. While such accidents remain rare, FAA recognizes that the public demands a high standard of safety and expects continued improvement.
section_title: How Fatal Accident Rates Are Calculated
paragraphs: FAA tracks the number of passenger fatalities for various types of aviation operations and calculates accident rates. Basically, the rates are calculated by dividing the number of accidents with one or more fatalities by one of the various measures of aviation activity. For example, the fatal accident rate for commercial aviation for 1988-97 is 0.058 per 100,000 flight hours, which was calculated by dividing the number of fatal accidents (85)by the number of flight hours (151 million). This translates into about one fatal accident for every 2 million hours flown. The three activity measures generally used to calculate fatal accident rates are the number of individual flights (referred to as departures), aircraft miles flown, and aircraft hours flown. Each activity measure reflects different exposures to the risks associated with flying. For example, most commercial aviation accidents occur during takeoff or landing, rather than during the cruise phase, which constitutes the largest part of the total mileage and hours flown. For this reason, we believe that departures are usually the best measure of exposure to risk.
For large and commuter air carriers, all three fatal accident rates are tracked. But for general aviation, the only measure of exposure is the number of flight hours estimated from survey data. Thus, fatal accident rates for commercial aviation (large and commuter air carriers) are generally expressed in terms of the number of fatal accidents per 100,000 departures, while fatal accident rates for general aviation are expressed as the number of fatal accidents per 100,000 flight hours as estimated by FAA’s annual survey.General aviation flight hours are not sufficiently reliable for use in calculating a fatal accident rate for general aviation because they are estimated from a voluntary survey, according to FAA.
section_title: Fatal Accident Rates Have Decreased
paragraphs: Over the past few decades, the annual rate of fatal aviation accidents has decreased significantly for both commercial and general aviation. While the accident rates are low, they have shown little improvement recently. For large commercial air carriers, the U.S. accident rate was 26 fatal accidents per million departures in 1959. Following the advent of large jet aircraft in the 1960s, the rate fell to one or fewer fatal accidents per million departures and has remained fairly steady for three decades. The fatal accident rate for commuter aircraft has also fallen over the last several decades. The accident rate for commuter air carriers fell from about 2 fatal accidents per million departures in 1982 to 3 per 10 million departures in 1996. While there were no fatal commuter accidents in 1998, the five fatal accidents in 1999 resulted in a fatal accident rate of nine per million departures. This increase in the fatal accident rate reflects a 1997 narrowing in the definition of commuter air carrier to include only small aircraft with nine or fewer seats.Similarly, the accident rate for general aviation aircraft has dropped since 1960. The fatal accident rate of six per 100,000 flight hours in 1960 fell to less than two by the early 1980s. The fatal accident rate for general aviation continued to decrease fairly steadily through the 1980s, increased slightly in the early 1990s, and has dropped steadily since 1995. In 1999, the fatal accident rate for general aviation was 1.2 fatal accidents per 100,000 flight hours. (See fig. 2.)
The reductions in the fatal accident rates resulted from a combination of technological advances that improved safety. In commercial aviation, these advances included the replacement of large, piston-engine aircraft with jet aircraft with far more reliable engines, the development of navigational equipment to warn pilots of impending crashes, better ground navigation aids, improved aircraft instrumentation, and increased air traffic radar coverage. Some of these improvements have also benefited smaller commuter and general aviation aircraft. As commuter air carriers switched from small aircraft to sophisticated turboprop aircraft, the accident rate among the larger commuter aircraft became comparable to that of large air carriers.
section_title: If Greater Numbers of Fatalities Are to Be Avoided, the Fatal Accident Rate Must Be Reduced
paragraphs: If the current fatal accident rate holds steady and aviation activity grows as FAA has projected, the increased air traffic will result in greater numbers of crashes and fatalities. We estimate that the average of six fatal commercial aviation accidents per year in 1994-96 will likely rise to nine per year by 2007. Similarly, the fatal accidents for general aviation will probably mount from an average of 380 in 1996-98 to 484 in 2007.Table 3 shows our projections of the number of fatal accidents in 2007 calculated from FAA’s growth estimates and the current fatal accident rate for each type of aviation operation.
The prospect of more accidents and deaths is unacceptable to the public, FAA, and the aviation industry. Avoiding that outcome means reducing the fatal accident rate significantly. The final report of the National Civil Aviation Review Commission concluded in 1997 that the “anticipated growth in aviation between now and the first quarter of the next century will almost certainly lead to an occurrence of aviation accidents with a frequency that will be wholly unacceptable to the public.” The Commission called for a joint industry-government effort to reduce the accident rate substantially.
section_title: FAA and the Aviation Industry Made Previous Efforts to Reduce the Fatal Accident Rate
paragraphs: During the 1990s, FAA and aviation industry groups had separate and joint efforts under way to use available data to identify and address the major causes of accidents. A series of fatal crashes and concern that the number of accidents and fatalities will increase as air traffic increases prompted these efforts to reduce the accident rate. Many of the reports that resulted from these efforts set specific goals and included recommendations for decreasing aviation accidents. Although FAA and the aviation industry acted on some of these recommendations, the fatal accident rate has remained fairly stable but low.
The effectiveness of previous efforts to reduce the fatal accident rate is believed to have been undercut by their limited scope and a lack of coordination between government and industry groups. Many of the studies issued during the 1990s were under the leadership of either FAA or a particular segment of the aviation industry. For example, FAA, on its own, studied controlled flight into terrain (CFIT)accidents and runway incursions. Separately, the Flight Safety Foundation brought together participants from many segments of the aviation industry to study CFIT and approach and landing but initially had only limited FAA involvement. The Aerospace Industries Association initiated an extensive study on the causes of safety-related problems in aircraft engines, including uncontained engine failure.(For a list of key aviation studies and our related reports, see app. II.) According to FAA and industry officials we interviewed, efforts to address specific safety issues were generally unsuccessful when one group failed to coordinate its work with that of other groups that had important roles in aviation safety.
Many of these reports issued during the 1990s set specific goals for reducing the overall fatal accident rate or for addressing specific aviation safety problems that result most often in fatalities. They also included numerous specific recommendations to FAA and the aviation industry to help meet these goals. Among the key reports were the following:
In 1993, the Flight Safety Foundation led an international task force on CFIT, the leading cause of fatal commercial aviation accidents worldwide. The task force provided specific recommendations and training aids aimed at reducing CFIT accidents. The task force set a goal of reducing these accidents 50 percent worldwide by 1998 and other goals targeting improvements in the regions of the world with the highest CFIT rates.
In January 1995, over 1,000 government, industry, and union officials attended an FAA-sponsored safety conference. The officials agreed that they shared responsibility for pursuing a goal of zero accidents. Their report identified 173 high-priority safety initiatives in the areas of crew training, air traffic control and weather, safety data collection and use, applications of emerging technologies, aircraft maintenance procedures and inspections, and development of flight operating procedures.
Following the May 1996 ValuJet crash, an FAA task force recommended in September 1996 that FAA target agency resources to safety risks, improve the certification and oversight of new air carriers, and address concerns about inspector guidance and resources.
In February 1997, the White House Commission on Aviation Safety and Security recommended that the government and the aviation industry establish a national goal to reduce the aviation fatal accident rate by a factor of five (meaning 80 percent) within 10 years. To achieve that goal, the Commission made specific recommendations for reengineering FAA’s regulatory and certification programs.The Commission did not explicitly state whether the national goal should apply to all types of aviation operations.
In December 1997, the National Civil Aviation Review Commission recommended that the government and the aviation industry work together to achieve the White House Commission’s goal of an 80-percent reduction in the accident rate over the next 10 years and recommended specific safety improvements for achieving that goal.While the Commission did not explicitly state whether the 80-percent goal should apply to all types of aviation operations, the Commission specifically discussed the accident rates for large jets, commuter air carriers, general aviation operations, and air taxis.
Both the White House Commission on Aviation Safety and Security and the National Civil Aviation Review Commission called for FAA and the aviation industry to work together on aviation safety issues.
section_title: The Safer Skies Initiative Continued Ongoing Efforts to Use Data Analysis to Address Safety Problems
paragraphs: On April 14, 1998, the Vice President, the Secretary of Transportation, and the FAA Administrator announced the Safer Skies initiative, a new aviation safety program committed to reducing the fatal accident rate by 80 percent by 2007. Under the initiative, experts from FAA, the aviation industry, and other government agencies with responsibility for aviation are to jointly analyze U.S. and global data to identify the most serious threats to aviation and to find the root causes of accidents. They will then determine the best actions to break the chain of events that lead to accidents and direct resources first to those actions. These actions are also referred to as interventions.
subsections: section_title: FAA Invited Members of Ongoing Industry and Government Safety Groups to Join the Safer Skies Initiative
paragraphs: When FAA announced the Safer Skies initiative, the agency invited participants from a number of ongoing industry and government safety groups to join in creating a unified safety agenda. In establishing the agenda for the initiative, the commercial and general aviation steering committees joined with and expanded the preexisting efforts. To develop the unified agenda, key government and industry aviation officials are to conduct data analyses to identify the causes of fatal accidents and determine what interventions are needed to prevent them.
Several of the preexisting safety groups were already using data-driven approaches to study aviation safety issues. Representatives of air carriers, aircraft and engine manufacturers, and related associations had established a commercial aviation group in January 1997 to analyze fatal commercial aviation accidents and to recommend ways to prevent them. Before joining the initiative, this group had outlined a process for obtaining accident data from U.S. and international sources and for reaching consensus on the safety problems to be addressed. Another industry group analyzing data on uncontained engine failure had developed a process for analyzing safety data, using case studies to identify root causes, and evaluating the feasibility of proposed interventions. A third group that represented a cross-section of various general aviation constituencies, such as pilots and small aircraft manufacturers, was addressing the causes of fatal general aviation accidents.A joint government-industry group sponsored by FAA was continuing work on issues pertaining to the safety of passengers and crew members in the aircraft cabin that had been started separately by FAA, industry associations, and unions representing flight attendants.FAA invited members from all four of these groups to participate in the initiative.
section_title: Steering Committees Selected 16 Safety Problems for the Safer Skies Initiative to Address
paragraphs: Safer Skies formed steering committees of safety experts from government and industry to lead the work in each of its three agenda areas: commercial aviation, general aviation, and cabin safety. Each steering committee has co-chairs and participants from both FAA and industry. The commercial and general aviation steering committees used available data to select the safety problems to be addressed in their respective agenda areas. In contrast, the cabin safety steering committee continued the work on safety problems that had already been under way as a joint FAA-industry effort that preceded Safer Skies. The three Safer Skies steering committees ultimately chose to address 16 safety problems: 6 in commercial aviation, 6 in general aviation, and 4 in cabin safety. The commercial aviation and general aviation steering committees selected several of the same safety problems, including weather and loss of control over the aircraft. Because safety problems can affect large and small aircraft differently, the commercial and general aviation steering committees planned to have separate teams study each safety problem with one exception. A joint team will study runway incursions because commercial and general aviation aircraft often share the same runways and accidents have occurred involving both types of aircraft. Table 4 lists and briefly explains each safety problem.
section_title: Objectives, Scope, and Methodology
paragraphs: At the request of the Chairman and Ranking Democratic Member of the Subcommittee on Aviation, House Committee on Transportation and Infrastructure, we reviewed the design and implementation of the Safer Skies initiative. Specifically, they asked us to determine (1) to what extent addressing the safety problems selected by the Safer Skies initiative will help reduce the fatal accident rate; (2) what progress the initiative has made in identifying and implementing interventions to address each of these safety problems; (3) what progress the Safer Skies initiative has made in assessing the effectiveness of those interventions; and (4) how FAA is coordinating the Safer Skies initiative with other safety activities conducted throughout the agency, in partnership with the aviation industry, and by other federal agencies.
Because Safer Skies is a 10-year project that hopes to reach its goals in 2007, we analyzed domestic flight operations and accident data for the decade that preceded the 1998 announcement of Safer Skies and for the decade to come. We examined data on fatal accidents and their causes for all types of aviation operations in the United States from 1988 through 1997. We also examined projected data for aviation operations and accidents through 2007.
To determine whether addressing the safety problems chosen by Safer Skies will help reduce the fatal accident rate, we interviewed FAA officials responsible for overseeing Safer Skies, officials at the Department of Defense and the National Aeronautics and Space Administration involved in aviation safety, and the chairs and many members of the steering committees for commercial aviation, general aviation, and cabin safety. We reviewed documents related to each of these steering committees as well as data used by these groups in choosing the problems on which Safer Skies would focus. We also discussed the problems Safer Skies selected as priorities with staff at the National Transportation Safety Board, the Flight Safety Foundation, and other aviation safety groups.
To determine what progress has been made in identifying, developing, and implementing intervention strategies for the Safer Skies initiative, we interviewed the FAA and industry chairs of the teams formed to address each problem under study. We obtained and reviewed team reports completed for each safety problem to understand the analysis process, modifications made to it by successive work groups, and actions planned to improve aviation safety in each problem area.
To determine what progress has been made to date in assessing the effectiveness of its actions to improve aviation safety, we reviewed implementation plans to determine whether schedules were being met and whether ways had been chosen to measure the success of such actions. We also reviewed available team reports and relevant data to determine whether sufficient data were available to measure Safer Skies’ progress in improving aviation safety.
To determine how FAA coordinated the Safer Skies initiative with safety activities conducted throughout FAA and in partnership with the aviation industry, we reviewed information on related industry and government safety activities. Specifically, we sought information on activities under the auspices of FAA, the Department of Defense, the National Aeronautics and Space Administration, the National Transportation Safety Board, selected engine and aircraft manufacturers, several major air carriers, the Air Transport Association, and the Aircraft Owners and Pilots Association. During our interviews with members of the Safer Skies steering committees and teams, we discussed efforts to coordinate their work with other government and industry safety activities. We also reviewed the reports from each Safer Skies team for safety problems where coordination would be appropriate. We discussed the budgetary implications of the Safer Skies initiatives and the criteria for prioritizing resources with FAA officials and steering committee members.
We conducted our work from August 1999 through June 2000 in accordance with generally accepted government auditing standards.
section_title: The Safer Skies Initiative Should Help Improve Aviation Safety
paragraphs: Addressing the 16 safety problems chosen by the Safer Skies initiative should help reduce the nation’s fatal accident rate. In commercial aviation, eliminating the six safety problems to be addressed by the initiative would approach the 80-percent goal. Other FAA initiatives are addressing additional safety problems in commercial aviation, which should complement efforts under the Safer Skies initiative. In general aviation, the initiative will address six problems that appear to be among the most common causes of fatal accidents for this type of operation, according to available accident data. While the initiative has adopted the 80-percent goal in commercial aviation, which transports most passengers who fly in the United States, the initiative adopted a less aggressive goal for general aviation, which accounted for the vast majority of the fatal aviation accidents. The goal in general aviation is to reduce the number of fatal accidents to 350 in 2007, which represents about a 20-percent reduction. Finally, the initiative addressed four problems in cabin safety. Improving cabin safety will have little impact on lowering the fatal accident rate because cabin safety accounted for only two U.S. fatalities in commercial aviation in 1988-97. No quantitative goal was set for safety improvements in cabin safety. To date, safety improvement efforts by FAA and the Safer Skies initiative have focused on past accidents and incidents, which may not be entirely predictive of future ones. Studying growth and technological changes in the aviation industry can help anticipate and prevent the safety problems and accidents that are likely to arise from such changes. Preliminary international efforts have been initiated to address future hazards, and coordinating these efforts with Safer Skies work could enhance the initiative’s efforts to reduce the fatal accident rate.
subsections: section_title: The Safer Skies Initiative Addresses Major Safety Problems in Commercial Aviation
paragraphs: The Safer Skies initiative plans to address six safety problems that accounted for 79 percent of the fatal accidents in commercial aviation in 1988-97. If past accident causes continue, completely eliminating these six safety problems might approach the 80-percent goal. FAA also has safety initiatives under way to address several of the safety problems in commercial aviation not addressed by the initiative. These include sabotage, fuel tank explosions, and structural problems. In combination with the Safer Skies initiative, FAA’s safety initiatives have potential for reducing the fatal accident rate in commercial aviation. For commercial aviation, the Safer Skies initiative has established a goal of reducing the fatal accident rate by 80 percent in 2007 in accordance with the goal envisioned by the White House and congressional commissions on aviation safety.
subsections: section_title: The Safer Skies Initiative Identified Six Major Safety Problems
paragraphs: The Safer Skies initiative will address six safety problems that accounted for 79 percent of the fatal commercial aviation accidents in 1988-97. Three of these safety problems were major ones both worldwide and in the United States: pilots’ losing control of their aircraft, pilots’ flying otherwise controllable aircraft into the ground or water (CFIT), and accidents during approach and landing. These three safety problems accounted for 58 of the 85 fatal accidents in U.S. commercial aviation during this period. The commercial aviation teams are examining 34 of these accidents, which involved larger aircraft. The commercial aviation steering committee referred the remaining 24 fatal accidents to the general aviation steering committee for review because they involved small commuter aircraft with nine or fewer seats that operated scheduled commercial service. This was done because (1) the aircraft involved are more similar to general aviation aircraft than to larger commercial aircraft, (2) the types of operating environments and safety problems that caused the accidents more closely resemble those of general aviation than those of commercial aviation, and (3) the interventions to address safety problems in general aviation are more likely to correct these safety problems than interventions designed for large commercial aircraft. We reviewed the National Transportation Safety Board’s (NTSB) reports for the 24 small commuter accidents and found that most of the accidents happened in Alaska when pilots flew into mountains after deteriorating weather reduced visibility. On the basis of our review, we concur with the commercial aviation steering committee’s assessment that these accidents more closely resemble general aviation accidents and would likely benefit from the interventions that emerge to address these safety problems in general aviation aircraft.
The potential for improving safety in these smaller commuter aircraft exists with a number of the interventions proposed by the general aviation teams working on weather and CFIT. It is unclear whether the initiative or FAA has mechanisms in place to ensure that small commuter operators will benefit from the interventions developed. For example, many of the interventions involve providing additional training to pilots on weather conditions and assessing the risk factors associated with each flight. Because the initiative plans to deliver much of this training jointly with the Aircraft Owners and Pilots Association, it is essential that notification about this training also be provided to small commuter operators and pilots who could benefit from this training but may not be members. Although members of several other organizations participate in the general aviation steering committee and study teams,neither the initiative nor FAA has made specific provisions to ensure that such interventions are also directed at small commuter aircraft operators and pilots, as well as at general aviation pilots. Because small commuter accidents accounted for 28 percent of the 85 fatal accidents in commercial aviation in 1988-97, reducing the fatal accident rate by 80 percent depends on addressing these safety problems in small commuter aircraft, as well as in large commercial aircraft.
To further reduce the fatal accident rate for commercial aviation, the initiative will address three additional safety problems that have resulted in fewer fatal accidents in the United States from 1988 through 1997. The steering committee chose runway incursions, uncontained engine failure, and weather, each of which resulted in from two to four fatal accidents. These three safety problems accounted for an additional 9 accidents, or 11 percent of the 85 fatal accidents. The committee selected these problems because they caused past fatal accidents or serious incidents that could have cost many lives. These areas were also included because each occurred with greater frequency in the United States than worldwide and because FAA or the aviation industry had already begun work on these safety problems. (See fig. 3.)
Our analysis of aviation data and review of safety reports confirmed that the initiative is addressing three major safety problems that caused fatal accidents in commercial aviation, as well three other safety problems that have the potential to cause accidents with large numbers of fatalities. Reducing or eliminating safety problems resulting from CFIT, loss of control, approach and landing, runway incursions, weather, and uncontained engine failure should help lower the fatal accident rate. Safer Skies participants, FAA officials, and industry aviation experts whom we interviewed also believe that the initiative is addressing the most important aviation safety problems. Most of these aviation experts indicated strong support for addressing such major safety concerns as CFIT, approach and landing, and loss of control. Furthermore, because of the increasingly global nature of commercial aviation, addressing these safety problems means that many of the interventions recommended by the initiative might have applicability worldwide, as well as in the United States. Many aviation experts we interviewed also supported the inclusion of safety problems with fewer fatalities but with a high potential for fatalities, such as runway incursions and uncontained engine failure. They agreed that reducing or eliminating these safety problems should help reduce the fatal accident rate.
section_title: Addressing Additional Safety Problems Could Further Reduce the Fatal Accident Rate
paragraphs: In addition to successfully addressing the major safety problems discussed above, addressing additional safety problems could further reduce the fatal accident rate in commercial aviation. FAA has a number of aviation safety initiatives under way that potentially can contribute to improvements in the safety of smaller commuter aircraft sometimes used in commercial aviation. For example, FAA’s Capstone Project focuses on improving general aviation safety in Alaska by providing additional navigational aids but also has potential application for addressing the safety problems of small commuter aircraft used elsewhere. FAA also has ongoing initiatives to address the causes of 4 of the 18 commercial aviation accidents not being addressed by Safer Skies teams. These include programs overseen by the agency’s Office of Civil Aviation Security to reduce the threat of sabotage, hijacking, and the transportation of hazardous cargo. Other FAA initiatives are under way to address the structural problems of aging aircraft and fuel tank explosions. FAA has, for example, published a notice proposing requirements for design reviews and mandatory maintenance actions for fuel tank systems on large transport aircraft.
Of the remaining fatal accidents in commercial aviation that the initiative is not addressing, 12 were on-ground fatalities, and 2 resulted from other causes.The on-ground accidents each involved the death of a single worker or unauthorized individual at the airport. Most of these accidents occurred near the boarding gate or ramp. For example, several employees were fatally injured when struck by an aircraft’s propeller or nose gear during the course of their work. Of the on-ground fatalities, two resulted from individuals gaining unauthorized access to airport areas that should have been secured, nine involved various airline or airport employees who sustained injuries in the workplace, and one involved a passenger who fell out of an aircraft catering door and onto the ground. Because on-ground accidents accounted for 14 percent of the 85 fatal accidents in commercial aviation in 1988-97, reducing the fatal accident rate by 80 percent by 2007 will be difficult if these safety problems are not addressed.
FAA has initiatives to address some of the safety problems that caused on- ground fatalities, but it is unclear how systematically these problems are being addressed. Specifically, FAA’s Office of Civil Aviation Security oversees airline and airport programs to limit access to secure areas to authorized individuals. The status of FAA’s efforts to address workplace safety issues that resulted in on-ground fatalities is less clear. FAA is responsible for regulating the safety and health aspects of the work environment of aircraft crew members when the aircraft is in operation. However, FAA has not promulgated specific regulations that address all employee safety and health issues associated with working conditions on aircraft. FAA held a public meeting in December 1999 to gather information on issues associated with working conditions on and around aircraft and to determine whether additional regulations should be proposed. However, FAA does not currently have a group addressing workplace safety issues and could not identify any regulations, guidance, or other initiatives that have been developed to address the types of workplace safety problems that caused most of the on-ground fatalities.
section_title: Improving Commercial Aviation Safety Involves Considering More Than Reducing the Fatal Accident Rate
paragraphs: Looking at the number of fatalities associated with various safety problems, as well as their contribution to the fatal accident rate, provides additional perspective on Safer Skies’ commercial aviation agenda. Reductions in the fatal accident rate are closely linked to reductions in the number of fatal accidents. Following this logic, the greatest reductions in the fatal accident rate can be achieved by eliminating the safety problems that caused the greatest number of accidents with one or more fatalities. However, strict adherence to the goal of reducing the fatal accident rate could result in focusing attention and resources on the causes of accidents that resulted in single fatalities, rather than on those causes that result in multiple fatalities, as well as multiple accidents. In choosing which safety problems to address, the commercial aviation steering committee selected safety problems that will help reduce fatalities, as well as the fatal accident rate.
The fatal accident rate in commercial aviation can most quickly be reduced by addressing the three safety problems that form the core of the Safer Skies agenda in commercial aviation: CFIT, loss of control, and approach and landing. These three safety problems accounted for 34 fatal accidents involving larger aircraft that commercial aviation teams are handling and 24 additional small commuter accidents that general aviation teams are handling. (See table 5.) If the initiative is successful in developing and implementing interventions to eliminate these three safety problems for both large aircraft and small commuter aircraft, it would make progress toward preventing the kinds of safety problems that caused 68 percent of the fatal accidents in 1988-97. If the initiative could successfully eliminate all six safety problems on its agenda for commercial aviation, it would approach the goal of an 80-percent reduction in the fatal accident rate. However, other safety problems actually resulted in more fatal accidents and thus could reduce the fatal accident rate more quickly if eliminated.
The initiative could approach the 80-percent goal more quickly by eliminating on-ground accidents, which caused more fatal accidents in commercial aviation than all other safety problems except loss of control and CFIT. On-ground accidents caused 12 fatal accidents in commercial aviation—14 percent of the total. While the safety problems that caused on- ground accidents merit addressing, the safety problems that Safer Skies’ commercial aviation team has chosen to address resulted in multiple fatal accidents and many more fatalities. For this reason, the initiative will probably have more impact on improving the safety of air transportation for the majority of the nation’s passengers than addressing other safety problems, such as on-ground accidents, whose elimination could reduce the fatal accident rate more but would save fewer lives.
While focusing on reducing the fatal accident rate by addressing the safety problems that caused the most commercial aviation accidents, the approach taken by the initiative also resulted in choices that recognized where the greatest number of fatalities have occurred or could occur. The three major problems addressed by the initiative’s commercial aviation teams (CFIT, loss of control, and approach and landing) accounted for 57 percent of the 1,756 fatalities in 1988-97. This rises to 66 percent when all six safety problems on the commercial aviation agenda are considered. The additional small commuter accidents that are to be addressed by general aviation teams account for another 5 percent of the total fatalities. The only other safety problems that resulted in hundreds of fatalities were sabotage and fuel tank explosions. The initiative did not focus on these problems for two reasons. First, only one fatal accident resulted from each of these safety problems in 1988-97. Second, FAA already has initiatives under way to address both sabotage and fuel tank explosions. The initiative did include two safety problems on its commercial aviation agenda that each accounted for only about 1 percent of the fatalities in U. S. commercial aviation during this period. However, the initiative recognized the potential of runway incursions, which accounted for 25 U.S. fatalities, to result in hundreds of fatalities. While weather resulted in few commercial aviation accidents and 16 fatalities, the commercial aviation steering committee members felt that the problems of turbulence and icing merited attention. In contrast, the 12 on-ground accidents each resulted in a single fatality that together accounted for fewer than 1 percent of the nation’s commercial aviation fatalities. Eliminating the safety problems that caused on-ground fatalities could reduce the fatal accident rate more quickly than eliminating either CFIT or approach and landing accidents that involved large commercial aircraft. The commercial aviation steering committee has selected safety problems that will help reduce fatalities, as well as the fatal accident rate. (See fig. 4.)
section_title: The Safer Skies Initiative Has Adopted the 80-Percent Goal for Commercial Aviation
paragraphs: The Safer Skies initiative and FAA have adopted the goal of reducing the fatal accident rate for commercial aviation by 80 percent by 2007. Specifically, the goal is to reduce the fatal accident rate for commercial aviation from a 1994-96 baseline of 0.037 fatal accidents per 100,000 flight hours to 0.007 fatal accidents per 100,000 flight hours in 2007. The meaning of this goal can be more readily understood by considering the current number of fatal commercial aviation accidents and the number of accidents projected for 2007 if further safety improvements are not undertaken. In 1994-96, the United States averaged six fatal commercial aviation accidents each year. Given the projected growth of commercial aviation, we estimate that this number could increase to nine in 2007 if safety is not improved. If the initiative achieves the goal of an 80-percent reduction in the fatal accident rate for commercial aviation, we estimate that the number of fatal accidents expected in 2007 would drop to two. (See table 6.)
section_title: Accident Data and Other Resources Were Used to Identify Safety Problems That Caused Many Fatal Accidents in General Aviation
paragraphs: The general aviation steering committee used available accident data, safety reports, and professional expertise in aviation to identify safety problems that caused many of the fatal accidents in general aviation. The six safety problems chosen were controlled flight into terrain, loss of control, aeronautical decision-making, runway incursions, weather, and survivability. Steering committee members told us that they selected these safety problems after reviewing the available data on general aviation accidents and past industry and government-sponsored safety reports on general aviation. They said that the NTSB accident reports were challenging to analyze because many lacked the detail needed to determine the root causes of accidents. They noted, for example, that most general aviation aircraft are not equipped with such key equipment as flight data recorders that would help identify the safety problems that caused the accidents. To meet these additional challenges, FAA developed a training course tailored to the needs of those responsible for analyzing general aviation accidents. Both FAA and industry members attended this course before starting the analysis phase. The general aviation analysis reports on CFIT and weather also included recommendations to address problems with the quality of the data on general aviation accidents. In response to these recommendations, the general aviation steering committee chartered a team in April 2000 to develop strategies to (1) provide increased detail about factors that have contributed to or caused general aviation accidents and incidents and (2) improve the quality and timeliness of estimates of general aviation activity. Members of the steering committee told us that, in cases where the safety problems that caused the fatal accidents were unclear, they used their experience as either pilots or experts in general aviation to determine the possible causal factors involved in the accidents.
Members of the steering committee also examined past industry and government reports on the causes of general aviation accidents. One key report was the Nall Report, a report on general aviation accident trends and factors published annually by the Aircraft Owners and Pilots Association’s Air Safety Foundation. According to the 1998 Nall Report, the major causes of fatal general aviation accidents were weather, loss of control or other errors during flights in which the pilot was maneuvering the plane, and accidents on approach to the airport. Another key report was FAA’s study of the causes of general aviation CFIT accidents.FAA’s study concluded that CFIT accidents accounted for 17 percent of the general aviation fatalities and 32 percent of general aviation accidents in weather conditions requiring pilots to have instrument ratings to fly.
Steering committee members also told us that several reports indicated growing problems with runway incursions involving general aviation aircraft. For example, a study by DOT’s Office of Inspector General showed that general aviation pilots caused the majority of runway incursions attributable to pilot error in 1990-96.Members of the steering committee told us that they also decided to address survivability in an effort to decrease the number of fatalities among those who survive the impact of a crash but ultimately die from their injuries.
Although the data available on general aviation accidents are less detailed than those available on commercial aviation accidents, the general aviation problems the initiative plans to address represent reasonable choices. Most of the safety problems chosen have been identified in past safety reports and NTSB accident reports as major causes of fatal accidents in general aviation. These include weather, loss of control, CFIT, and runway incursions. Aeronautical decision-making has also been cited repeatedly as a factor in such safety problems as weather, when pilots exercise judgment about whether to depart or turn back when faced with potential danger. In addition, aeronautical decision-making includes those decisions made relating to aircraft maintenance. Most of the Safer Skies participants, FAA officials, and aviation experts we interviewed concurred that the six general aviation safety problems to be addressed by the initiative are reasonable ones that will help to reduce the fatal accident rate.
subsections: section_title: The Safer Skies Initiative Adopted a Goal of Reducing Fatal Accidents in General Aviation to 350 in 2007
paragraphs: Although both the White House and congressional commissions on aviation safety called for an 80-percent reduction in the nation’s fatal accident rate, FAA and the Safer Skies initiative applied this goal only to commercial aviation and adopted a less aggressive accident reduction goal for general aviation. The goal is to reduce the number of fatal general aviation accidents to 350 in 2007. This represents a 20-percent reduction in the number of fatal accidents that would likely result from projected growth in general aviation. Because general aviation accounted for 98 percent of U.S. fatal accidents in 1988-97, the goal of an 80-percent reduction in the nation’s fatal accident rate set forth by the two major aviation commissions is unreachable if these fatal accidents are not greatly reduced. The congressionally mandated commission on aviation safety discussed the fatal accident rates for all kinds of aviation operations, including general aviation. Because this commission did not explicitly apply the 80-percent goal to general aviation, it remains unclear whether it intended the goal to apply to general aviation as well as commercial aviation. The goal adopted—350 fatal accidents—contrasts sharply with the 97 fatal accidents that would likely result if the 80-percent goal were achieved. (See table 6.)
The steering committee did not adopt the 80-percent goal for general aviation because of strong objections from the general aviation community. Representatives of the general aviation community argued that, given the varied experience levels of its pilots, reducing fatal accidents by 80 percent would be impossible without grounding the fleet. One general aviation representative said that there was a prevailing concern in the general aviation community that any agreement on a solid goal would lead to more regulation and less growth. In addition, these representatives objected to establishing a goal that involved a fatal accident rate. The fatal accident rate for general aviation is calculated by dividing the number of fatal accidents by the number of flight hours. Data on general aviation flight hours are estimated using an annual survey of general aviation operators conducted by FAA. Response to the survey is voluntary. Because the flight hours are estimated on the basis of this survey, representatives of the general aviation community questioned the reliability of these data and expressed concern about using flight hours to calculate past and future fatal accident rates. As a result, the Safer Skies steering committee for general aviation agreed not to use the survey data on flight hours to calculate a fatal accident rate until the data are more reliable. Instead, the accident reduction goal for general aviation was expressed in terms of the number of fatal accidents, rather than the fatal accident rate.
To set its goal of reducing fatal accidents to 350, the general aviation steering committee reviewed available data on fatal accidents. The steering committee found the number had declined fairly steadily since 1990 in response to past initiatives to improve safety. The data used by the steering committee showed that, in 1996-98, an average of 379 fatal general aviation accidents occurred each year.The steering committee used this average and the 1.6-percent annual growth expected in general aviation to project that 437 accidents would occur in 2007 if additional safety initiatives were not undertaken. They agreed that a reduction to 350 fatal accidents would be achievable. This represents a 20-percent reduction in the number of fatal accidents that they estimated would occur without additional safety initiatives (437). According to projections by the steering committee and the general aviation community, a reduction of this magnitude would prevent 363 accidents from 2000 through 2007.
The goal of reducing the number of fatal accidents to 350 in 2007 is probably achievable, but this goal is not likely to push the general aviation community toward more safety improvement as aggressively as it could.
We believe that this goal is achievable for two reasons. First, although the level of general aviation activity has increased, the number of fatal accidents decreased to 354 in 1999, a decrease of 17 percent since 1994. Both FAA and industry officials attributed this decrease in part to ongoing safety initiatives. The goal of 379 accidents established for each of the next 3 years represents a 7-percent growth in the current number of fatal accidents. Second, the goal of 350 accidents set for 2007 is only 4 fewer fatal accidents than occurred last year. Hence, the long-term goal is achievable if the general aviation community is able to hold its number of fatal accidents steady as its air traffic grows by an expected 2.2 percent per year in the coming decade.
We recognize that an 80-percent reduction in fatal accidents is probably not achievable in general aviation at this time because of the diversity in pilots’ experience levels, aircraft types, and operating environments. However, we believe that a more aggressive goal would encourage greater efforts by general aviation operators, manufacturers, associations, and FAA to make safety improvements in general aviation operations that could save lives.
section_title: Improving Cabin Safety Is Important but Will Have Little Impact on Lowering the Fatal Accident Rate
paragraphs: Improving cabin safety is unlikely to have much impact on reducing the overall fatal accident rate. In contrast to the safety problems addressed by the commercial and general aviation steering committees, the safety problems addressed by the cabin safety steering committee have not resulted in numerous fatalities, and few data are available on any injuries that result from these problems. The Safer Skies initiative identified only two fatalities in U.S. commercial aviation in 1988-97 related to cabin safety problems.While passengers and crew have been injured in the cabin environment, few data exist on these incidents because air carriers are not required to report such incidents unless they involve a serious injury or fatality. The study of cabin safety problems thus relies more on information shared by flight attendants and air carriers than on analysis of the limited data available. Because cabin safety resulted in few fatalities and affords few data for analysis, it is arguable whether it was appropriate to include cabin safety issues in an initiative directed at reducing the fatal accident rate through a data-driven analysis of safety problems.
Although not appropriate for Safer Skies’ focus on the safety problems that caused fatal aviation accidents, cabin safety issues are an appropriate topic for FAA to address jointly with the aviation industry. NTSB and flight crews have raised concerns about the potential for injuries and fatalities in the cabin. FAA and industry were jointly studying cabin safety problems before the initiative was announced. The safety problems under study included those involving child restraint systems, passenger seatbelt use, passenger interference with crew, and carry-on baggage. Concerns about these safety problems are not new. For example, NTSB has long advocated FAA’s requiring the use of child restraints for passengers under the age of 2. NTSB was concerned enough about the use of child restraints to launch a campaign aimed at making parents aware of the benefit of putting children in approved child restraint systems and to declare 1999 as the “year of child transportation safety.” Similarly, representatives of air carrier crews have expressed concern that the incidents of passengers interfering with crew members are increasing.
section_title: Additional Work on Future Hazards Could Help Anticipate and Prevent Fatal Accidents
paragraphs: In December 1997, the congressionally mandated commission on aviation safety recommended that FAA and the aviation industry jointly develop a strategic plan to improve aviation safety and that the process “begin with analysis of both previous and potential failures to meet safety expectations.” These failures include accidents, incidents, insight from flight operational data, and aviation system changes. The analysis of the causes of past accidents provides insights into safety problems that exist within the current aviation system, while the analysis of aviation system changes can help anticipate future hazards that may arise from such changes as growth and technological advances (e.g., vertical takeoff and landing by aircraft). The approaches to the analysis of past safety problems and future hazards are distinct. A data-driven approach is particularly useful for analyzing the safety problems that caused past fatal accidents. Data on nonfatal accidents and incidents can also be used to identify and address safety problems that did not result in fatalities but could have. The data-driven approach is based on the assumption that identifying a problem is possible where historical data are available. While this approach can be used to address the safety problems in the current operating environment, other types of analyses may be more useful for anticipating and preventing the safety problems that could result in new types of fatal accidents. For example, the anticipated growth in air traffic will lead to more congestion around airports, increasing the possibility of runway incursions and midair collisions near airports. Anticipating how changes in the aviation industry may increase existing safety problems or bring about new ones can better position both FAA and the aviation industry to prevent accidents.
While FAA, Safer Skies, and industry groups have made progress in the analysis of the causes of past accidents and incidents, efforts to analyze and anticipate future hazards are more preliminary. The Joint Safety Strategy Initiative in Europehas formed a work group to develop a method for examining future hazards. A number of FAA staff participate in this work group, which should facilitate the cooperative exchange of ideas and information on this topic. As of April 2000, the Safer Skies initiative had not established a process for analyzing future hazards. A systematic analysis of the changes occurring in the aviation industry could enhance Safer Skies’ ongoing efforts to reduce the fatal accident rate. Several of the aviation experts interviewed suggested that the initiative could benefit from going beyond the analysis of data on past accidents to consider safety problems that may arise from rapid changes in the aviation operating environment. Participants on Safer Skies’ commercial aviation steering committee also indicated that while data-driven approaches are helpful, it is also important to consider future hazards. FAA’s Director of the Aircraft Certification Servicesaid that the initiative’s first priority was to understand and eliminate the safety problems that caused past accidents but that the commercial aviation steering committee also plans to address future hazards and recently added this topic to its agenda for consideration. Because work on future hazards could help anticipate and prevent fatal accidents, this topic is important for the Safer Skies steering committees to address, especially as it applies to commercial aviation. Coordinating this effort with the work initiated by European and FAA staff on future hazards should help avoid duplication of effort and foster awareness of and solutions to these potential problems internationally.
section_title: Conclusions
paragraphs: The premise of both the White House and congressional commissions on aviation safety was that data on past and possible future causes of accidents could be used to focus resources on substantially reducing the fatal accident rate. While the Safer Skies initiative has made significant strides, it has not yet carried out this mandate as fully as it could. The six safety problems that the initiative is addressing accounted for almost 80 percent of the fatal accidents in commercial aviation in 1988-97. Our review showed that the initiative and FAA have work under way to address these and other safety problems in commercial aviation. However, the initiative has not challenged all sectors of the aviation community to push aggressively for safety improvements. Although the initiative has adopted the challenging goal of reducing the fatal accident rate for commercial aviation by 80 percent by 2007, general aviation is not being asked to set a similarly challenging goal. While an 80-percent reduction in fatal accidents is probably not achievable in general aviation at this time, the goal adopted by the initiative does not push the general aviation community toward implementing the kinds of interventions that could substantially lower the fatal accident rate. A more rigorous goal would encourage greater efforts by general aviation operators, manufacturers, associations, and FAA to make needed safety improvements. In addition, many of the interventions developed to improve general aviation safety could also benefit small commuter operators and pilots, but this benefit will not be realized without a systematic way of ensuring that training and other interventions are also directed at small commercial aviation operations. Finally, the Safer Skies initiative and most aviation safety studies to date have focused on the causes of past accidents. While analyses of accident data are useful for determining the causes of past accidents, reducing fatal accidents during a period of rapid growth in aviation will probably require the analysis of the changing aviation environment to anticipate future safety problems. Preliminary international efforts have been initiated to consider future hazards, and integrating these efforts with Safer Skies’ work would enhance the initiative’s efforts to reduce the fatal accident rate.
section_title: Recommendations
paragraphs: To further reduce the nation’s fatal accident rate and save lives in the type of aviation operation that causes the most fatal accidents and fatalities, we recommend that the Secretary of Transportation direct the FAA Administrator to work with the general aviation community to set a more challenging goal for reducing the number of fatal general aviation accidents by 2007, set interim goals to assess progress toward this new goal, and ensure that training and other interventions that emerge from general aviation teams are communicated to small commuter operators and pilots who may benefit from them.
section_title: Agency Comments
paragraphs: DOT and FAA officials concurred with our recommendations aimed at setting a more challenging interim goal and long-term goals for general aviation and said that they planned to do so in the future. However, the officials noted that existing general aviation accident data are too inaccurate to be used as the basis for setting an accident reduction goal. The general aviation steering committee has established a work group to recommend ways to improve the quality of general aviation data. The officials stated that FAA and the general aviation community would review the accident reduction goal when the quality of the data improves.
DOT and FAA officials disagreed with our recommendation aimed at ensuring that training and other interventions emerging from general aviation teams are communicated to small commuter operators because they believe that mechanisms already exist to do this. The officials explained that a number of associations representing smaller commuter aircraft participate on the general aviation steering committee and on its analysis and implementation teams. These organizations provide conduits for transmitting interventions developed by the general aviation teams to small commuter operators. We agree that these organizations may facilitate the transfer of safety interventions developed by the general aviation teams to small commuter air carriers. However, it will be difficult to achieve the mandated 80-percent reduction in commercial aviation fatalities without systematic improvements in the safety record of small commuter air carriers, which accounted for 28 percent of fatal commercial aviation accidents. We believe that Safer Skies would benefit from a systematic plan for ensuring that interventions developed by general aviation teams are communicated to and implemented by small commuter operators. For this reason, we did not modify or delete our recommendation.
DOT and FAA officials disagreed with our recommendation calling for an analysis of future safety problems arising from the rapid growth and changes in aviation. The officials noted that efforts involving FAA, Safer Skies, and the European aviation industry are already under way to address future hazards in aviation. On the basis of the information presented by DOT and FAA officials, we withdrew this recommendation.
section_title: The Safer Skies Initiative Has Made Progress in Selecting and Implementing Interventions
paragraphs: Joint FAA and industry teams have started work on 13 of the 16 problems being addressed by the initiative. A two-part process has been developed for use by these teams to first analyze accident and incident data and then to use that analysis to identify, select, and implement safety interventions to help prevent accidents in the future. That process is reasonable and has allowed FAA and industry groups to reach consensus on how to address safety problems identified under the initiative. This process was not used to address cabin safety problems because the cabin safety steering committee had already begun its work before the process was developed. The Safer Skies teams have made progress primarily in those areas that had been studied extensively in the past for which widely supported recommendations already existed. The interventions recommended for five problems are now being implemented: uncontained engine failure and CFIT in commercial aviation and passenger seatbelt use, child restraint systems, and carry-on baggage in cabin safety. The process being used will require more extensive analysis in the future as teams begin to address safety problems that have not been studied previously. Finally, the success of the interventions that the Safer Skies teams have chosen to address these long-standing safety problems depends in part on effective implementation. Our past work has shown that FAA does not consistently follow through on implementing key safety recommendations. Furthermore, FAA and the aviation industry began implementing some of the Safer Skies safety interventions before having a process in place to track their progress. The initiative has developed a process for tracking the implementation of interventions to improve safety in commercial aviation. However, the implementation of Safer Skies’ interventions is not assured because the tracking system for commercial aviation is not sufficiently detailed to assess progress in implementing interventions. Furthermore, the cabin safety steering committee implemented its interventions without having a tracking process in place, and the general aviation steering committee is working toward the final approval of interventions to address two safety problems without having a tracking process. Without a complete tracking process, FAA and the industry cannot ensure that the initiative will improve aviation safety in each of these areas.
subsections: section_title: The Safer Skies Methodology Is Based on Previous Efforts to Identify Safety Problems
paragraphs: For the Safer Skies initiative, FAA and the aviation industry jointly developed a two-part process to analyze accident data and then to choose from among the possible interventions. This process grew out of a previous FAA effort that used a data-driven approach to identify threats to aviation safety and develop interventions to address those threats. During the first part of this process, an analysis team reviews accident data to determine what went wrong, why it went wrong, and what interventions might be the most effective in preventing similar accidents in the future. The second part of the process involves another team that assesses the feasibility of each potential intervention, prioritizes the interventions on the basis of their effectiveness and feasibility, and submits plans for implementing projects to the steering committee for approval. However, as we discuss later in this chapter, the steering committee addressing cabin safety problems did not use this process.
subsections: section_title: The Initiative Uses a Two- Part Process to Analyze Data and Identify Interventions
paragraphs: The initiative uses a two-part process to analyze data and identify interventions to address safety problems in commercial aviation and general aviation. This process is modeled on an analysis of the most significant threats to aviation safety conducted in 1997 by staff from FAA’s Aircraft Certification Service. The two-part process was developed for use by the teams addressing safety problems in commercial aviation but has also been used by the general aviation teams with some modifications. Under the process, the steering committee forms an analysis team for each aviation safety problem. The team, which includes members from FAA and the aviation industry, reviews accident data, determines accident causes, and identifies possible interventions to prevent future accidents. For selected accidents, the team develops a detailed sequence of events that includes the actions by pilots and air traffic controllers as well as any system or equipment failure. The team determines what went wrong and why and then considers various interventions that could have prevented the accident. In its final report, the analysis team ranks all of the identified interventions by their effectiveness in preventing similar accidents and presents them to the steering committee for further action.
Once the analysis team completes its work on a safety problem, the steering committee forms a second team to assess the feasibility of implementing the interventions suggested by the analysis team. The implementation team assesses feasibility in six areas: the cost of the intervention; the time needed to implement it; whether it requires regulatory changes; technical feasibility; the practicality of the project within the operating environment or the nationwide aviation system; and political feasibility. The implementation team prioritizes the interventions by both effectiveness and feasibility and then presents the resulting prioritized list to the steering committee. Once the steering committee initially approves an intervention, the implementation team develops a detailed project plan for implementation that is sent to the steering committee for final approval. Once detailed plans are approved, the interventions are then implemented by the responsible organizations.
The general aviation teams have made some modifications to the analysis process initially developed for use by the commercial aviation teams. Although the first commercial aviation analysis team considered feasibility as well as effectiveness, the two-part process ultimately approved for commercial aviation teams considers only effectiveness at the analysis stage. Any consideration of such matters as cost and the need for developing new regulations is left to the implementation team. In contrast, the general aviation analysis teams consider both effectiveness and feasibility. Our review of the general aviation analysis team’s reports for CFIT and weather confirmed that such feasibility criteria as cost and the need for new regulations have been considered far earlier in the assessment of general aviation interventions than in the process now used by commercial aviation teams. While other feasibility factors are also considered, cost, the avoidance of interventions that would require new regulations, and acceptability to the general aviation community have weighed heavily in the choice of interventions to address general aviation safety problems. In emphasizing cost and acceptability to the aviation community, the general aviation teams have selected training and other interventions that will be more affordable to general aviation pilots.
section_title: The Cabin Safety Team Used a Different Approach
paragraphs: While the initiative is using a systematic, defined approach to consider ways to address safety problems in commercial and general aviation, a different approach was used to address cabin safety problems. Several months before the announcement of the Safer Skies initiative, FAA established the Partners in Cabin Safety (PICS) team to provide information to the public about four cabin safety problems: passenger interference with flight crews, the safety benefits of greater use of seat belts by passengers, the safety benefits of child safety restraints, and potential safety issues arising from the stowage of carry-on baggage. According to PICS team members, FAA identified these problems before assigning them as tasks to the team in January 1998. Team members discussed such additional issues as in-flight medical emergencies and cabin air quality but settled on the four that were eventually included. Unlike the commercial and general aviation teams, the PICS team limited the possible interventions to ones that did not require that FAA create new regulations, a process that was viewed by some participants as too slow and unlikely to result in consensus among various industry and government participants. Consequently, the PICS team focused on interventions that involved educating passengers.
The four cabin safety problems addressed differ in several important ways from those safety problems addressed by the Safer Skies teams in commercial and general aviation. First, the cabin safety problems resulted in only two fatalities in U.S. commercial aviation from 1988 through 1997, both involving passengers not using their seat belts when the aircraft encountered turbulence. In contrast, during the same period of time, there were more than 9,800 fatalities in all commercial and general aviation accidents. Second, air carriers are not required to maintain or submit data on cabin safety incidents unless they involve fatalities or serious injuries. Since only limited historical data on cabin safety accidents and injuries were available for analysis, the PICS team did not conduct a causal analysis as has been done by the analysis and implementation teams for both commercial and general aviation.
The PICS team disbanded in January 1999 after it completed the development of passenger education materials. As part of its passenger education efforts, the team distributed brochures on child restraint systems from a previous campaign by FAA. In addition, the Luggage and Leather Goods Manufacturers of America, along with FAA, developed a brochure addressing carry-on baggage concerns, which the team members were asked to distribute to airlines, luggage stores, and airports. It was also put on FAA’s World Wide Web site for further distribution by interested parties. Steering committee members and FAA officials also told us that the team worked with air carriers to develop additional cabin announcements for the stowage of carry-on baggage and the importance of seat belt usage. Finally, the PICS team developed a passenger safety checklist for publication on FAA’s Web site, which addressed passenger interference with flight crews, seat belt usage, child restraint systems, and carry-on baggage. This checklist, however, is not currently available on FAA’s Web site. According to an official at FAA’s Flight Standards Service, the passenger safety checklist project is on hold until the agency appoints a new national resource specialist for cabin safety who will review the document before it is made available to the public.
section_title: The Safer Skies Initiative Has Made the Most Progress With Problems Studied Previously
paragraphs: Since the FAA Administrator announced the Safer Skies initiative in April 1998, work has started on 9 of the 12 safety problems to be addressed in the commercial and general aviation safety areas. Teams have made the most progress in selecting interventions for safety problems when they could build on previous studies for which widely supported recommendations exist. The commercial aviation steering committee plans to have work started on all of the identified problems before the end of fiscal year 2000, but the general aviation steering committee has not yet determined when work on three of its six problems will begin. Table 7 shows the status of the work on each of the 12 safety problems to be addressed in commercial and general aviation as of April 1, 2000.
The general aviation CFIT and weather reports were presented to us as final reports.
However, in responding to our draft report, FAA told us that these reports had not received final approval.
The steering committees charged with developing interventions for each of the safety problems first formed analysis teams to work on problems for which major studies had already been done or were under way. The ongoing and completed studies conducted by FAA and the industry provided information the analysis teams could use to identify the causes of accidents and potential interventions. For example, the Flight Safety Foundation had completed an extensive study on CFIT, examining over 250 accidents and incidents worldwide. The foundation had also developed training materials for pilots and made other recommendations to prevent CFIT accidents. In another instance, the team analyzing weather-related accidents involving general aviation aircraft identified 11 safety studies that had preceded its efforts, all of which recommended interventions similar to the ones the team ultimately identified. FAA participants on the Safer Skies commercial aviation steering committee told us that beginning with previously studied safety problems helped team members make progress in developing the team’s two-part process for analyzing data and identifying interventions and become comfortable with the analysis and selection process before moving onto more complex issues that may involve original research and analysis. However, this approach meant that work on another area that is important for reducing the fatal accident rate in commercial aviation did not start work until September 1999—17 months after the initiative’s announcement. FAA identified loss of control as the single largest cause of fatal commercial aviation accidents involving U.S. operators.
To date, the implementation of interventions has concentrated mostly in areas for which analysis and implementation were well under way or complete when the initiative began. The first of the interventions to be implemented addressed uncontained engine failure: FAA issued a series of airworthiness directives requiring enhanced inspections of high-speed rotating parts in certain jet engines.The directives require industry maintenance personnel to perform additional, more detailed inspections to check for cracks and other signs of irregularities whenever an engine is disassembled for overhaul or maintenance. According to staff at FAA’s Engine and Propeller Directorate, these directives affect more than 90 percent of the jet engines that U.S. airlines currently use. FAA and the industry are also taking steps to implement an intervention endorsed by the commercial aviation team examining CFIT. The team has recommended that enhanced navigational equipment be installed on new and existing aircraft to warn pilots of impending crashes. Air carriers began installing the enhanced navigational equipment to prevent CFIT accidents in their aircraft before FAA issued the final rule requiring that the equipment be installed in the commercial fleet. Specifically, air carriers began installing the enhanced navigational equipment to prevent CFIT accidents in their aircraft before FAA issued its final rule in March 2000 and before the commercial aviation team working on CFIT issued its final report in June 2000. This equipment is now being included on some new aircraft, and airlines had equipped about 4,000 aircraft already in service with the new technology by December 31, 1999.
The timetable for analysis and implementation teams addressing the problems included under the initiative has changed since the initiative was announced in April 1998. According to the chairs of the Safer Skies steering committees, some of these schedule changes occurred because the analysis process took longer than anticipated. In other cases, changes to the analysis approach required rescheduling Safer Skies’ efforts. For example, FAA officials explained that the final report date for the commercial aviation CFIT implementation team was rescheduled after the steering committee decided that combining the CFIT and approach and landing teams for the implementation analysis made sense because of overlap in the interventions they had identified. Several high-priority interventions to address CFIT accidents in commercial aviation were, however, forwarded to the steering committee for final approval and implementation without waiting for the implementation team’s final report. An FAA co-chair of the general aviation steering committee told us that they changed the start dates for several of the general aviation teams because general aviation accidents are more numerous than commercial aviation accidents and analyzing them proved more time-consuming than anticipated. This FAA official also said that some general aviation groups participating in the initiative do not have enough people or resources to serve on multiple teams simultaneously. While we believe that these decisions were justified, they also effectively mean that the interventions to resolve some key safety problems will not be identified or implemented until later than originally anticipated.
section_title: Early Experience Indicates That Future Problems Will Require More Analysis
paragraphs: Additional analysis will be needed to identify interventions to address current and future safety problems for which few or no previous studies exist. Safer Skies teams relied initially on a limited number of case study analyses to identify the causes of accidents and incidents, as well as the interventions that could prevent them in the future. The teams compared the results of these case studies with the causes and interventions identified by previous studies to determine whether they are consistent. For example, the team working on CFIT in commercial aviation completed detailed event sequences for 10 accidents and found that the causes identified and the interventions it recommended were similar to those of prior studies. Safer Skies teams working on approach and landing accidents in commercial aviation and weather-related accidents in general aviation also compared the results of their analyses with those of prior studies.
Along with other changes as the Safer Skies initiative has evolved, this approach has been modified as teams addressed additional safety problems. For example, the runway incursion analysis team expanded its case studies to include incidents because there were so few fatal accidents involving runway incursions. Similarly, the analysis team now working on loss of control in commercial aviation has selected a larger number of case studies because this safety problem has not been the subject of extensive prior analysis.
section_title: Effective Implementation Is Critical Next Step in Making Progress Toward the Goals Set for Reducing Fatal Accidents
paragraphs: The Safer Skies initiative has identified the major safety problems to be addressed, has made progress in identifying their root causes, and has developed interventions to address some of them. Reducing fatal accidents depends in part on the effective implementation of these interventions. As discussed in chapter one, however, many of these safety problems are long- standing ones that have persisted in spite of previous studies and recommendations. In addition, FAA has not consistently followed through on implementing safety recommendations in the past. The Safer Skies initiative does not yet have in place a process to track the implementation of these interventions that is sufficiently detailed and covers interventions chosen to improve safety in commercial aviation, general aviation, and cabin safety.
subsections: section_title: The Success of Safer Skies Interventions Depends on Effective Implementation
paragraphs: Reducing the fatal accident rate in commercial aviation and the number of general aviation accidents will depend in part on effective implementation of the interventions chosen by the Safer Skies teams. Many of the safety problems that the initiative addresses are long-standing ones that have been studied extensively in the past. Actually resolving these problems has proven difficult in the past and remains very challenging. Similar interventions have been recommended, but the desired reductions in fatal accident rates have not been achieved. For example, extensive prior studies of CFIT and approach and landing accidents in commercial aviation recommended many of the same interventions that are now being implemented by the Safer Skies commercial aviation steering committee. Furthermore, reaching the 80-percent goal in commercial aviation will depend heavily on the successful implementation of interventions to address the safety problems that caused the most fatal accidents: loss of control, CFIT, and approach and landing. To reach the goal in commercial aviation, interventions must be effectively implemented for both small commuter aircraft and large commercial air carriers. Even after safety interventions have been identified, implementing them has proven challenging.
As DOT’s Inspector General and we have reported previously, FAA does not consistently follow through on implementing safety recommendations. Our review showed that FAA usually agreed with the recommendations on aviation safety made by GAO, NTSB, and DOT’s Inspector General. FAA had implemented 64 percent of the 256 recommendations that we reviewed; however, FAA had not completed actions to implement the remaining 36 percent of the recommendations.We found that FAA sometimes did not establish time frames for implementing the recommendations or did not meet established times for implementing them. Similarly, DOT’s Inspector General found that of the 23 near-term actions FAA planned for addressing runway incursions in its 1998 Action Plan, 15 had not been completed on time.We found that even safety recommendations that received specialized attention, intensive follow-up, and heightened awareness among industry, the Congress, and the public have not been fully implemented. For example, NTSB considered runway incursions so serious that it repeatedly placed this safety problem on its lists of critical safety recommendations in the early 1990s. Although FAA concurred with NTSB’s recommendations, our review found that several of the corrective actions needed had not been implemented, including actions to improve (1) visibility at airports; (2) runway lighting, signage, and surface markings; and (3) radar and related equipment to alert air traffic controllers to impending runway incursions. FAA developed several plans in the 1990s to decrease runway incursions. In spite of these programs, the actual number of runway incursions has increased. DOT’s Inspector General noted in 1999 that the number of runway incursions had increased from 292 in 1997 to 325 in 1998, in part because FAA had not set aside the funds needed to support the initiatives and projects in the runway incursion action plan.As a result, FAA has made limited progress in implementing its plan, and milestones have been missed and extended. DOT’s latest performance report for fiscal year 1999 shows continuing problems in this area. The actual number of runway incursions (321) was 19 percent higher than the goal of 270 established in DOT’s performance plan.
Industry participants in the Safer Skies initiative have voiced concern that some interventions may not be implemented promptly or at all. Some of the same Safer Skies participants questioned whether enough resources would be available to complete the implementation of the selected interventions. Without assurance of adequate resources, it is likely that the choice of interventions by Safer Skies teams will be constrained by cost considerations and the implementation of recommended interventions will be incomplete. Effective implementation will also depend on having a process for tracking the implementation of interventions to be carried out by all Safer Skies participants, including FAA; other government agencies; manufacturers; airlines; and other industry participants.
section_title: The Steering Committees Have Not Yet Developed Effective Processes for Tracking the Implementation of Interventions
paragraphs: FAA and the aviation industry began implementing some of the Safer Skies safety interventions before developing a systematic way of tracking the progress being made. This occurred in part because the steering committees incorporated some safety initiatives already under way and endorsed the resulting interventions before they developed a systematic tracking process. In addition, Safer Skies teams have recommended that a few high-priority safety initiatives be started before final implementation reports are issued. While moving forward on important safety initiatives makes sense, ensuring their successful implementation depends on effective tracking. Interventions have been implemented in both commercial aviation and cabin safety with no tracking process in place. The general aviation steering committee is moving toward approval of interventions for CFIT and weather but has not yet developed a tracking process. Several of the Safer Skies participants we interviewed voiced some concerns about whether all the interventions being identified would eventually be implemented, given FAA’s past problems in implementing recommended safety improvements.
section_title: Tracking Has Been Limited and Not Systematic
paragraphs: In its December 1997 report, the congressionally mandated commission on aviation safety recommended that FAA’s and the industry’s strategic plan include milestones for accomplishing specific tasks. The commission noted that the plan should be detailed enough that milestones for accomplishing specific tasks can be readily recognized by agency management and the industry, as well as the public. In addition, the commission directed FAA to report periodically on where initiatives stand, why any delays are occurring, and whether and why changes are being made to the plan. These recommendations are in accordance with sound internal controls for program management.
The Safer Skies initiative, which was announced in April 1998, implemented a number of interventions without first developing a process for tracking their progress. In some cases, these were interventions that were developed by teams whose work was incorporated into the Safer Skies effort. In commercial aviation, for example, FAA, relying on the work of the uncontained engine failure team, published airworthiness directives beginning in April 1999 to require more extensive inspections of aircraft engines. The commercial aviation team working on CFIT also implemented several interventions in or before September 1999. These included interventions to verify the operational status of radar equipment to provide minimum safe altitude warnings to pilots and to develop a template for standard operating procedures to be used by airlines in training their pilots in techniques to avoid CFIT accidents.
In September 1999, the commercial aviation steering committee recognized the need for the systematic tracking of interventions and directed a work group to develop a proposal. At the commercial aviation steering committee’s meeting in January 2000, the work group presented its proposal for a Joint Implementation Measurement Team. The team designed the tracking process to provide a high-level report on whether each intervention is being implemented as planned. Specifically, this team’s responsibilities will include tracking whether the implementation of approved interventions complies with the implementation plans and their milestones; helping to predict the potential effectiveness of the proposed interventions; and identifying ways of measuring whether the intervention is achieving the desired risk reduction. The team will also provide a brief explanation of what is causing noncompliance with the plan and whether a solution has been found to resolve the problem. As conceived, the tracking report is to be a high-level progress report that does not intrude on the internal planning of the organizations responsible for carrying out the interventions. The tracking report thus does not provide detailed information on interim and long-term milestones or identify individuals responsible for implementing the plan and preparing progress reports for the tracking committee. Without more detailed information than is currently provided in the proposed tracking report, it may be difficult for the steering committee to assess progress in implementing interventions. For example, the tracking team’s January report notes that FAA has completed a plan for implementing two programs critical to gaining access to safety dataand that other industry and government groups have plans in development. However, the tracking report provides no information about the milestones established by FAA’s plan for establishing these key programs, both of which have experienced delays in the past. After we identified concerns about the tracking system, the commercial aviation steering committee agreed that improvements are needed, and it is working on revisions. A draft version provided for our review in June 2000 still lacked key information about major commitments, deliverables, and milestones.
Tracking implementation is even more critical for the more complex initiatives whose success depends on coordinated efforts by both FAA and the aviation industry. For example, successful implementation of the highest-priority intervention to prevent CFIT accidents in commercial aircraft—the installation of enhanced aircraft navigational equipment to warn pilots of impending crashes—requires coordination among many parties:
FAA must certify that the equipment works, issue technical standards for manufacturers, and issue a final regulation to require that the equipment be installed on new and existing aircraft.
Aircraft manufacturers need to make the equipment standard on new aircraft and retrofit it in older aircraft.
Air carriers need to incorporate the appropriate procedures for maintaining and using this equipment into maintenance and flight manuals and to train pilots in its use.
FAA needs to update its guidance to its inspectors so that they can ensure that air carriers properly carry out their responsibilities for training, maintenance, and use of the equipment.
Without a tracking system that provides more detailed information on the implementation of complex interventions, the commercial aviation steering committee will not have the information needed to ensure that they are fully implemented in accordance with planned milestones.
The implementation of interventions to improve cabin safety has also not been adequately tracked. The cabin safety steering committee, which completed the development of passenger education materials before it disbanded in January 1999, carried out most of its interventions with no Safer Skies tracking process in place. However, we found that educational materials related to passenger interference with crew had not been distributed or made available on FAA’s Web page as of April 2000. Furthermore, according to a member of the cabin safety steering committee, the distribution of other cabin safety brochures was, in some instances, never completed. The absence of a systematic process for tracking Safer Skies interventions may have contributed to inaccuracies in reporting on the status of cabin safety interventions. Specifically, the DOT FY 2001 Performance Plan and FY 1999 Performance Report states that all initiatives relating to cabin safety were completed as planned. However, planned actions to include material on passenger interference with crew had not been completed as of April 2000.
Finally, although the general aviation steering committee is reviewing draft implementation team reports that recommend interventions to address CFIT and weather, it has no process in place to track the implementation of interventions once they are approved. According to the FAA co-chairs of the general aviation steering committee, this group has committed to track the interventions selected but has not yet developed a process for doing that and plans to discuss this issue at a future meeting. Without coordinated, detailed implementation plans that assign responsibilities, FAA and the Safer Skies steering committees will not be able to ensure that all parties complete their portion of the plan and that implementation occurs on time. In addition, as part of the Safer Skies process, FAA and the general aviation community identified efforts that could be accomplished in the short term or were already under way to address the safety areas to be addressed by the initiative. FAA and the industry implemented a number of these short-term initiatives, such as the development and distribution of various safety videos and training aids. However, Safer Skies did not track the implementation of these interventions or evaluate their effectiveness.
section_title: Conclusions
paragraphs: The progress made by the initiative to date has resulted in the implementation of interventions for five safety problems—two in commercial aviation and three in cabin safety. However, a coordinated, centralized method of tracking will be necessary to ensure full implementation of these and future interventions. In the past, FAA has developed plans to make safety improvements but has not consistently implemented them successfully. An effective tracking system would provide for identifying the individuals or entities responsible for implementation, setting milestones, establishing resource estimates, and preparing progress reports. Without a systematic tracking mechanism, there is no assurance that any of the selected interventions will be fully implemented. While the commercial aviation steering committee has developed a system to track the implementation of the interventions it approves, this system is not sufficiently detailed to ensure their implementation. The general aviation steering committee, which is nearing final approval on interventions to address safety problems related to weather and controlled flight into terrain, is only now developing a tracking system modeled after the one used by the commercial aviation steering committee. Finally, nothing comparable has been developed to track interventions recommended by the cabin safety teams.
section_title: Recommendations
paragraphs: To ensure that interventions are implemented and that effective and feasible interventions are identified in the future for issues that the initiative has yet to address, we recommend that the Secretary of Transportation direct the FAA Administrator to advise the Safer Skies steering committees to take the following actions:
Develop a systematic way of tracking the implementation of interventions approved by all Safer Skies steering committees. This tracking system should include the identification of responsibility for implementation, the establishment of short- and long-term milestones and resource estimates, and the preparation of progress reports. The progress reports should provide information on the detailed steps to be taken by all government and industry participants to ensure the successful implementation of each intervention. Progress reports should highlight and explain any delays in meeting the milestones. This system should be shared with the relevant Safer Skies steering committees and FAA’s focal point for the initiative as well as with the team that recommended the intervention.
section_title: Agency Comments
paragraphs: DOT and FAA officials concurred with our recommendation on the need to track the implementation of interventions to achieve results, but they disagreed with the level of detail we advised. The officials stated that the commercial aviation steering committee’s draft revised tracking system provides better information for tracking the major commitments and deliverables. The expectation is that more detailed implementation plans will be maintained within each implementing organization. The officials do not believe that it is realistic for steering committees to review the details of every organization’s action plan. They also noted that the general aviation steering committee is developing a tracking system similar to that used to track commercial aviation interventions. We agree that the Safer Skies initiative has taken steps to improve its tracking system for commercial aviation and to work toward the development of a similar system for general aviation. However, the revised tracking system provided for our review in June 2000 did not clearly identify and include time frames for major commitments and deliverables for each of the interventions approved by the commercial aviation steering committee. We agree that individual FAA and industry organizations responsible for implementing Safer Skies interventions would logically have far more detailed systems for tracking implementation than the steering committees. However, without a reliable tracking system in place that contains basic information on major deliverables, responsibilities, and time frames, FAA and Safer Skies will not be in a position to ensure that recommended interventions are implemented to improve aviation safety.
DOT and FAA officials disagreed with our recommendation that FAA and the Safer Skies steering committees should analyze a sample of safety problems that were not studied previously. The officials presented information that showed some Safer Skies’ work groups were using or would be using a sample of previously unexamined safety problems in their work. For this reason, we withdrew the recommendation.
section_title: The Safer Skies Initiative Has Not Yet Developed Performance Measures to Evaluate the Effectiveness of Most Interventions
paragraphs: Of the five Safer Skies teams that have begun implementing interventions, only one has developed a performance measure to evaluate whether the interventions it has selected are helping to reduce the safety problems that cause fatal accidents and are worth what they cost. Such evaluations depend on performance measures that serve as the yardsticks for measuring progress toward program goals. The initiative’s ultimate goal is saving lives by reducing fatal accidents. Federal law requires that federal departments evaluate the effectiveness of the program activities for which they request funding. FAA will evaluate progress toward its broad goals for aviation safety using performance measures based on reducing the fatal accident rate for commercial aviation and the number of fatal accidents in general aviation. However, additional performance measures will be needed for evaluating the effectiveness of the interventions selected by the teams working on each of the safety problems. Most teams are still analyzing data on safety problems and selecting safety interventions and thus have not yet determined how to evaluate the effectiveness of interventions selected. Although teams working on 5 of the 16 safety problems have recommended interventions that are being implemented, only one of these teams developed an adequate performance measure before its interventions were implemented.
subsections: section_title: Federal Law Requires the Development of Performance Measures as Part of the Budget Process
paragraphs: To ensure that programs achieve their objectives and that funds are expended wisely, federal law requires that each department develop performance measures as part of its budget request. Performance measures are the yardsticks used to evaluate the effectiveness of the activities undertaken as part of federal programs. The initiative plans to develop performance measures to evaluate the effectiveness of the interventions it recommends to save lives by addressing the safety problems that cause fatal accidents. However, developing good performance measures can be difficult. While it is useful to establish a baseline of information about past fatal accidents, they occur too rarely to serve as performance measures to evaluate the effectiveness of interventions. Years may elapse between specific types of fatal accidents, such as uncontained engine failure, making it difficult to see trends or evaluate the effectiveness of interventions. Instead, the initiative must develop performance measures based on events that occur more frequently and that can be linked closely to interventions.
A congressional mandate exists for the measurement and evaluation of all federal programs. Performance measurement is a central premise of the Government Performance and Results Act of 1993 (Results Act). This act requires annual performance plans to cover each program activity set out in a federal agency’s budget. Among other requirements, performance plans are to (1) establish performance indicators to be used in measuring or assessing the outcomes of each program activity, (2) determine how to compare actual results with the performance goals, and (3) describe the means to verify and validate information used to report on performance. In accordance with this law, DOT develops annual plans that include performance measures for specific programs and activities. Agencies under DOT, such as FAA, develop more detailed plans and performance measures for each program activity.
Because of its impact on FAA’s programmatic and budgeting activities, the Safer Skies initiative falls under the Results Act’s requirement to evaluate program performance. Moreover, it was developed in response to the National Civil Aviation Review Commission’s report, which specifically directed FAA and the aviation industry to establish performance measures and milestones to assess the initiative’s progress in meeting safety goals, to review priorities periodically, and to monitor progress. The Safer Skies initiative incorporates the idea of establishing performance measures to evaluate progress toward safety goals. As a result, the Safer Skies teams that recommend interventions are tasked with developing the performance measures for those interventions approved by the steering committees.
For a performance measure to be useful, a baseline must be established against which to measure the effect of the intervention. Good evaluation criteria include (1) definitions of baseline information on the extent of the safety problem over a particular period prior to the implementation of the intervention and (2) timeframes for evaluating changes using the performance measure. Goals and time frames must also be established to determine what the program is expected to achieve and by when. For the initiative, appropriate baseline information includes both the total number of fatal accidents and the number of fatal accidents caused by each safety problem within each type of aviation operation (i.e., commercial aviation and general aviation).
Good performance measures have several key features: the event to be measured (e.g., a runway incursion) or desired outcome (a reduction in the number of runway incursions) is measurable; data on the event are or could be collected; and the event occurs with sufficient frequency between evaluations for progress to be measurable. The performance measures under development to evaluate Safer Skies’ initiatives can be assessed against these criteria.
section_title: Determining the Effectiveness of Safer Skies’ Initiatives Will Require the Development of Additional Performance Measures
paragraphs: Determining the effectiveness of Safer Skies interventions will require the development of performance measures other than the overall goals set for commercial and general aviation. Fatal aviation accidents occur so infrequently that their usefulness is limited as a measure of the success of Safer Skies’ interventions. This is especially true for commercial aviation, which had a total of 85 fatal accidents in the United States from 1988 through 1997. The fact that a particular type of accident has not occurred for several years does not mean that the underlying safety problem has been successfully addressed. Furthermore, for several reasons it may be difficult or impossible to match a specific implementation plan to a numerical reduction in fatal accidents overall or attributable to a specific safety problem. For example, in general aviation the lack of detail in accident reporting makes it difficult to determine specific accident causes; the lack of pilot profiles makes it difficult to evaluate the effectiveness of pilot training strategies; and it is hard to predict how many aircraft owners will install new safety equipment in the future. Thus, to determine to what extent an intervention is reducing fatal accidents attributed to a specific safety problem, teams will need to develop additional performance measures. The commercial aviation steering committee recognized early on this need to develop interim measures to evaluate the unique effect of individual interventions.
Even if a team identifies suitable performance measures for a specific safety problem, it may be difficult to determine whether a particular intervention, cluster of interventions, or other outside factors influenced changes in the performance being measured. This is especially true for situations in which teams choose numerous interventions to address a safety problem. While the uncontained engine failure team developed a single primary intervention, the team working on CFIT in commercial aviation has already initiated several interventions and is contemplating dozens more. Similarly, the general aviation team working on weather recommended 17 interventions. Without some way to independently evaluate the effectiveness of individual interventions or clusters of interventions, the initiative will have little way of knowing whether particular interventions save lives and are thus worth the time or money being expended on them.
In developing performance measures, one option involves using the precursors to accidents as proxies for the likelihood of fatal accidents. Precursors are events that, although they typically precede a particular type of fatal accident, often occur without culminating in a crash. For example, approach and landing accidents are almost always preceded by unstable approaches to the airport, but many unstable approaches may culminate in a hard or late landing that does not result in injuries or a crash. Performance measures based on precursors have been developed to evaluate initiatives for one of the safety problems the initiative is addressing, uncontained engine failure. The success of this approach depends on identifying appropriate accident precursors that can serve as proxies for the specific safety problem the team is addressing. Precursors are most useful when they follow the criteria for good performance measures: they are measurable, relevant data on them are available, and they occur with sufficient frequency.
section_title: Most Safer Skies’ Interventions Are Being Implemented Without Determining How to Evaluate Their Effectiveness
paragraphs: Of the 16 Safer Skies teams, 8 have recommended safety interventions for implementation; interventions from 5 of these teams have been or are being implemented; but only one has developed a performance measure that can show whether the intervention is effective at saving lives. Most Safer Skies teams are still analyzing data on safety problems and selecting interventions and have not yet determined how to evaluate the effectiveness of interventions selected. Of the five teams whose recommendations are being implemented, three have developed some performance measures. Only the uncontained engine failure team has developed two quantifiable performance measures that are based on accident precursors. In contrast, the general aviation teams working on CFIT and weather developed some general performance measures for reducing accidents resulting from these safety problems but did not quantify these measures. No performance measures were developed to evaluate the educational interventions implemented to address the four cabin safety problems. Finally, the team working on CFIT accidents in commercial aviation has implemented one intervention in advance of the team’s final report. While this team has not yet developed a performance measure for this intervention, it is considering using an accident precursor. Performance measures based on accident precursors have potential for use in evaluating the effectiveness of additional interventions being considered to address CFIT and other safety problems. FAA does not presently collect data on some accident precursors that could be used to evaluate the effectiveness of Safer Skies interventions and faces significant barriers to collecting such data.
subsections: section_title: The Uncontained Engine Failure Team Has Chosen Two Accident Precursors as Performance Measures
paragraphs: The Safer Skies team working on uncontained engine failure chose two accident precursors as performance measures for evaluating the effectiveness of the intervention it recommended: more extensive engine inspections. Because uncontained engine failure caused just two fatal accidents in the United States in 1988-97, fatal accidents are too infrequent to serve as a performance measure. But well-established trend data show that the safety problem occurs much more frequently, resulting not in fatal accidents but in incidents with severe or serious consequences on an average of about 1.5 times a year.The team chose the rate of these incidents as the primary performance measure for its recommended intervention. The team also chose another accident precursor as a second performance measure: the number of cracks detected in engine disks when engines are overhauled.Data analysis identified cracked disks as the primary cause of uncontained engine failure. According to staff at FAA’s Engine and Propeller Directorate, each crack detected during inspections probably avoids an uncontained engine failure that could have had severe or serious consequences.
Both accident precursors chosen—the rate of uncontained engine failure with severe or serious consequences and the detection of cracks in engine disks—have some of the attributes of a good performance measure. Both can be counted, and reporting mechanisms are in place for collecting the key data needed for both measures. Hence, it will be possible to evaluate whether the more extensive engine inspections lead to the detection of more cracks and fewer instances of uncontained engine failure with severe or serious consequences. However, good performance measures track events that occur often enough between evaluations to show whether progress is being made. Uncontained engine failure with severe or serious consequences occurs from one to three times a year, according to data from 1992-98, while cracks in engine disks are likely to be discovered about once in 25,000 inspections, according to staff at FAA’s Engine and Propeller Directorate. Hence, 2 to 5 years may elapse before the effectiveness of the more extensive engine inspections can be judged. Nonetheless, tracking both measures should provide sufficient data for reasonable interim and final performance measures, and the enhanced inspections provide an opportunity to avert potentially catastrophic accidents.
The uncontained engine failure team established much of the information needed to use its performance measures to evaluate the effectiveness of enhanced engine inspections. During our review, we worked with FAA staff on the team to develop additional information to provide a more complete context for how that intervention relates to the overall Safer Skies effort and to the fatal accident rate in commercial aviation. We then developed a template for this information that can serve as a model for other Safer Skies implementation teams. (See table 8.) The template displays the data critical for understanding the extent of the safety problem and the baseline for measuring progress in addressing it, including the frequency of the problem’s occurrence in 1988-97 and projections of its occurrence with and without the recommended intervention by 2007, the target year for Safer Skies to achieve an 80-percent reduction in the overall fatal accident rate. The template reflects the team’s goal of reducing the rate and projected number of uncontained engine failures with severe or serious consequences by 50 percent by 2007.
section_title: General Aviation Teams Did Not Develop Quantified, Specific Performance Measures
paragraphs: The general aviation implementation teams for CFIT and weather have completed their draft reports but did not develop quantified, specific performance measures to evaluate the effectiveness of the interventions they recommended. The general aviation CFIT team recommended 5 interventions subdivided into 22 distinct subinterventions. None of the 22 subinterventions included specific, quantified performance measures. For example, the CFIT team recommended developing criteria for standardizing the marking of wires, towers, and support structures to help decrease the number of CFIT accidents that occur when pilots of low-flying aircraft, such as helicopters and small planes, fly into these obstacles. As one measure of effectiveness, the team chose a decrease in the number of CFIT accidents involving wires or towers. However, the team did not provide any baseline information about the number of past CFIT accidents that involved wires or towers or the types of aircraft involved. To the extent that such baseline information is available, it provides a yardstick against which to measure progress in reducing these accidents. Furthermore, the team did not provide any specific interim or long-term accident reduction goals for the number of accidents or the percentage of the fleet affected. Without such information, it will be impossible to determine whether or by how much CFIT accidents involving wires or towers have decreased. Other performance measures for general aviation CFIT initiatives share this lack of quantification and specificity. Without baseline information on the occurrence of the problem prior to the implementation of the intervention and specific quantified goals, it will be impossible to evaluate the effectiveness of the interventions implemented.
The general aviation team working on weather experienced similar problems in setting performance measures for its interventions. The team’s final report recommended 17 interventions subdivided into 49 distinct subinterventions. Of the 49 subinterventions, only 1 included a quantified, specific performance measure. The rest had either no performance measures or performance measures that were not quantified or specific. Some of the interventions for which no performance measures were established involve research that is still ongoing to develop the technology suggested in the intervention. For example, NASA has the lead in developing equipment to sense turbulence and warn flight crews so that they can avoid or reduce the dangers associated with turbulence. Because research on this technology is preliminary, the performance measures are described broadly as reducing fatalities and injuries. It is likely too early to establish performance measures for these interventions. The performance measures included for many other subinterventions were too broad to allow actual evaluation of their effectiveness. The performance measure for most of these was a “decrease in the number of weather-related accidents.” These performance measures are neither quantified nor linked in any specific way to the interventions, which makes it impossible to determine what portion of the reduction, if any, is attributable to individual interventions or clusters of interventions. Of the performance measures developed, several measure progress in implementing training interventions, rather than the effectiveness of the training in reducing safety threats. For example, one intervention involves training Flight Service Station specialists and supervisors on in-depth weather analysis and interpretation to improve the weather briefings given to general aviation pilots. The associated performance measure involves training all of these FAA staff by 2002, rather than measuring the effectiveness of that training. In other cases, the team did not include a performance measure when one could have been developed. For example, one intervention involves conducting a refresher clinic for flight instructors to update them about current weather information and provide appropriate training materials for them to use with general aviation pilots. No performance goal was specified for this intervention. To measure how well this intervention has been implemented, it is possible to determine the number of flight instructors, to establish a goal for how many attend this training each year, and to have them provide information on how many pilots they subsequently train using the information. To determine whether the intervention is effective, the pilots who receive the training could later be surveyed to determine whether they had used the weather information provided or their safety records could be compared with the records of pilots who did not have the training. The link between accident reduction and such training is more tenuous than the link between crack detection and the prevention of uncontained engine failure, but it is possible to gain at least some information about the effectiveness of the training. Without such feedback, it is difficult to determine whether the training is effective and should be continued.
Without more specific baseline information on these performance measures prior to the implementation of the interventions and interim and long-term goals for progress, the initiative will not be able to evaluate the impact of these interventions. In responding to our draft report, FAA noted that the implementation teams for CFIT and weather relied on the expertise of team members, following analysis of the root causes of accidents, to determine the probable effectiveness of the interventions. Safer Skies analysis and implementation reports described problems with the quantity, quality, and type of data currently available about general aviation. These problems include shortcomings in the data for the types and numbers of operations and in the level of detail of the actual accident investigations. FAA concluded that the problems with general aviation data make it difficult to measure the effectiveness of individual intervention strategies by the traditional approach of how they affect accident rates. While we acknowledge the need to improve general aviation data, we also believe that such data can provide some indication of the relative frequency and importance of the causes of fatal accidents. Such information is also important for making decisions about which interventions to fund and expedite, considering their potential effectiveness and the number of fatal accidents that their use might prevent. While it may not be possible to develop quantitative performance measures for all interventions proposed by the implementation teams, good performance measures depend on having measurable events, a way to collect data on those events, and an event that occurs with sufficient frequency between evaluations for progress to be measurable. The performance measures for both general aviation weather and CFIT could be improved where possible by identifying and quantifying baseline information, ensuring that a means exists for collecting data on the performance measure, and setting interim and long-term goals against which to measure progress in implementing the intervention.
section_title: The Safer Skies Initiative Did Not Develop a Strategy for Evaluating Cabin Safety Interventions
paragraphs: The Safer Skies cabin safety steering committee completed work on four safety problems and implemented most interventions without developing any strategy for evaluating the interventions. Although the steering committee completed its work in January 1999, it did not develop performance measures for the interventions it selected. While the initiative’s broad goal is reducing the fatal accident rate, the broad goal for cabin safety is educating the flying public about four areas: passenger interference with flight crews, passenger use of seat belts, child restraint systems, and carry-on baggage. The steering committee distributed brochures about carry-on baggage and the importance of child restraint systems and worked with air carriers to develop additional cabin announcements to remind passengers to use their seat belts. The team did not, however, set up any evaluation to show whether the public’s knowledge about these issues improved as a result of these interventions and whether that improved knowledge results in fewer fatalities.
While useful performance measures could be defined in each of the four cabin safety areas, the steering committee did not develop a strategy for evaluating the impact of its educational initiatives. For example, the steering committee did not plan or track the distribution of the flyers it issued about carry-on baggage or child restraint systems, and it developed no performance measures for evaluating the effectiveness of these initiatives to educate the public. Furthermore, FAA does not have a mechanism for consistently collecting data about any of these areas. Airlines are required to report information related to cabin safety only if something happens in the cabin that results in serious injuries or death. As a consequence, the agency does not have baseline data for measuring improvements that may result from its initiatives. Thus, the Safer Skies initiative has no way of measuring the effectiveness of its educational efforts in the cabin safety area.
section_title: Precursors of Accidents Have Potential for Use as Performance Measures in Other Safer Skies Areas
paragraphs: Precursors to accidents have the potential for use as performance measures for evaluating interventions to address at least three other Safer Skies safety problems: CFIT, runway incursions, and approach and landing. Precursors are needed because fatal aviation accidents caused by all three safety problems occur rarely. The precursors for each safety problem have at least some of the attributes of good performance measures.
subsections: section_title: Navigational Alerts Could Serve as a Performance Measure for One CFIT Intervention
paragraphs: The Safer Skies team working on CFIT accidents in commercial aviation is considering using an accident precursor to evaluate the effectiveness of one of its interventions: the installation of enhanced navigational equipment on aircraft that sounds alerts to warn pilots of impending crashes. The equipment tracks data on the frequency of the alerts and the situations in which they occur. Although these data are not currently collected by FAA, they could be used to develop a performance measure based on the alerts sounded as precursors to CFIT accidents. The performance measure of alerts sounded could indicate the number of dangerous situations avoided.
Alerts sounded by this navigational equipment have several features of a good performance measure. First, the alerts can be measured. Second, the equipment itself tracks such warnings. Finally, the alerts are sounded with sufficient frequency to be useful as a performance measure. According to the manufacturer, enhanced navigational equipment was installed in over 4,000 aircraft from March 1996 through December 1999. In 14 instances, the alerts enabled pilots to recover from impending crashes.
section_title: Runway Incursion Incidents Could Serve as a Performance Measure
paragraphs: Runway incursion incidents that do not result in accidents provide another useful performance measure and are being used as such by FAA. From 1988 through 1997, 2,345 runway incursions resulted in five fatal accidents and 59 fatalities in the United States.However, runway incursions have the potential to cause much greater numbers of fatalities; the collision of two large aircraft on the ground in the Canary Islands in 1977 resulted from a runway incursion and took more than 580 lives. Because runway incursion incidents are increasing in the United States and have the potential to lead to fatal accidents, FAA’s Performance Plan for FY 2000 has used these incidents to establish a performance measure for a series of safety recommendations designed to reduce accidents caused by runway incursions. The Safer Skies team addressing runway incursions has not yet identified interventions, but FAA’s ongoing work offers some useful performance measures for measuring progress in addressing this safety problem.
Runway incursion incidents have all three features of a good performance measure. First, the incidents can be counted. Second, the data can be collected because FAA already has a mechanism for reporting runway incursions.Moreover, FAA has collected data on them for years, and therefore has historical data that can be used to establish baselines against which the effectiveness of interventions intended to reduce runway incursions can be measured. For example, one intervention now in use by FAA involves deploying action teams to airports that have experienced high numbers of runway incursion incidents to determine the causes and develop action plans to resolve them. Data on runway incursion incidents can be used to determine whether the use of action teams reduces such incidents at the airports in question. Finally, runway incursion incidents occur with sufficient frequency to make it possible to measure progress between evaluations. Several hundred runway incursion incidents have been reported each year this decade.
section_title: Unstable Approaches Could Serve as a Performance Measure for Approach and Landing
paragraphs: The Safer Skies team working on approach and landing accidents in commercial aviation is considering using an accident precursor to evaluate the effectiveness of training and other related interventions. The team determined that unstable aircraft approaches to airports were clearly precursors to many approach and landing accidents.Several problems can contribute to unstable approaches, including excess speed on approach, aircraft flaps not in position, and an approach that is too steep or too shallow. Data on each of these key aspects are recorded on an aircraft’s flight data recorder. Thus, the team has an opportunity to develop a performance measure based on reducing the number of unstable approaches.
Unstable approaches have some features of good performance measures. First, they are measurable. Second, data on them can be obtained from flight data recorders. However, there are barriers to obtaining these data that must be overcome before unstable approaches can be used as a performance measure for approach and landing interventions. Finally, unstable approaches occur frequently enough to measure progress resulting from interventions.
section_title: Potential Barriers Exist to the Use of Some Accident Precursors as Performance Measures
paragraphs: Barriers exist to using some accident precursors as performance measures. For example, the use of unstable approaches as a performance measure depends on access to information from aircraft flight data recorders. While some airlines use data from flight recorders to analyze the causes of safety problems on routine flights, there are barriers to sharing this information with FAA or with other airlines. Logistical barriers include (1) the limited information tracked by older flight data recorders still in use and (2) differences in the ways that air carriers have programmed flight data recorders to track key information. Because of these differences, the kinds of data items needed to track unstable approaches are not being captured with enough consistency for this measure to be a good indication of performance throughout commercial aviation.
Other potential barriers also prevent the use of unstable approaches as a performance measure. Among these barriers are the ongoing debate about how data from flight recorders are to be shared, who should have access to these data, and whether legal enforcement cases can be initiated on the basis of these data. Numerous major aviation safety reports in this decade have advocated a program that would gather and analyze information from flight data recorders about routine flights. FAA has for years promised to establish such a program.However, the inability of FAA, the aviation industry, and other federal agencies to reach consensus on key aspects of this program has delayed its finalization. While shared data can move safety forward, concerns about potential litigation, criminal indictments, and the violation of an air carrier employee’s privacy have served as barriers to the establishment of the program. Such concerns have also delayed the finalization of other programs to enhance the sharing of aviation safety data. For example, safety reports have for years recommended the establishment of Aviation Safety Action Programs to encourage voluntary self-reporting of safety violations by pilots; FAA issued an advisory circular providing guidance for these programs on March 17, 2000.
section_title: Conclusions
paragraphs: Most Safer Skies teams have not finished analyzing the causes of the safety problems they are working on and have not yet selected interventions to prevent the problems. Thus, these teams have not developed methods to evaluate the effectiveness of their interventions. But when interventions have been selected, most have been implemented without first determining how to evaluate their effectiveness. Neither FAA nor the aviation industry will have the information that will be critical in determining whether the interventions have made progress in resolving the safety problems until appropriate performance measures are developed. Evaluating the impact of safety interventions depends on having good baseline data on the extent of the problem prior to the implementation of the intervention, explicit short- and long-term goals against which to measure progress, and performance measures that are clearly linked to the safety problem being addressed. In addition, as Safer Skies teams select interventions to address the safety problems that caused fatal aviation accidents, it would be useful to identify clearly any existing barriers to the development of performance measures. These barriers include differences in aircraft equipment and the absence of needed data. Once such problems are clearly identified, FAA and the aviation industry can work jointly to resolve them.
section_title: Recommendations
paragraphs: To improve the ability to determine the effectiveness of Safer Skies interventions, we recommend that the Secretary of Transportation direct the FAA Administrator to work with the Safer Skies steering committees to direct the teams to identify the extent of fatal accidents resulting from the safety problems they are working on. If possible, data should be developed to establish a consistent baseline against which to measure the progress that results from the Safer Skies initiative. If an analysis team has already completed its report, the implementation team working on the same safety problem should develop these baseline data. More specifically, to better measure progress toward overall safety goals, we recommend that the FAA Administrator work with the Safer Skies steering committees to revise the implementation guidance to (1) develop an overall performance measure or measures to determine progress toward eliminating the safety problem the team is addressing; (2) consider using accident precursors as performance measures for the safety problem in question; and (3) identify any barriers that may impede the implementation of performance measures.
section_title: Agency Comments
paragraphs: DOT and FAA officials agree in principle with the need for baseline data on the extent of each safety problem and performance measures to determine progress toward overall safety goals. They concur with the potential of accident precursors as possible performance measures and with the importance of identifying any barriers that may impede the implementation of performance measures.
section_title: Coordination Has Been Extensive but Needs Improvement for the Safer Skies Initiative to Succeed
paragraphs: FAA coordinated extensively with numerous representatives from the aviation industry, other federal agencies involved in aviation safety, and its own staff on the identification of safety problems and the selection of interventions. However, efforts to prioritize, fund, and evaluate Safer Skies initiatives could be better coordinated with industry and within FAA and the Department of Transportation (DOT). Joint government-industry efforts to improve safety are not new, but participants noted that the initiative was more inclusive than prior joint efforts. This inclusive approach should help FAA gain consensus on which interventions will best address aviation safety problems. However, our review identified three coordination problems that could undermine the implementation and evaluation of Safer Skies interventions. First, although FAA officials have repeatedly committed to funding interventions agreed upon by all parties working on the initiative, skepticism still exists among some participants as to whether this commitment can or will be honored. This is particularly true in general aviation. It also remains unclear what process will be used, if funding is limited, to reprioritize available resources to ensure funding for interventions that emerge later but have greater potential for reducing the fatal accident rate. Finally, Safer Skies steering committees, FAA, and DOT have not coordinated how they will measure progress in achieving the accident reduction goal for commercial aviation.
subsections: section_title: The Safer Skies Initiative Involves an Unprecedented Level of Coordination Between Industry and Government
paragraphs: FAA included aviation experts from a wide range of government and industry organizations on the Safer Skies steering committees and the teams working on the 16 safety problems. Many participants represent groups that are directly responsible for the nation’s aviation safety, such as the air carriers and the manufacturers of aircraft and engines. Other participants come from trade associations that represent various aviation groups or from federal agencies that share responsibility for aviation safety, including the Department of Defense and the National Aeronautics and Space Administration. In addition, while giving priority initially to reducing the U.S. accident rate, the initiative recognized the increasingly global nature of aviation. In an effort to address both domestic and worldwide aviation safety problems, the commercial aviation steering committee included representatives from two international aviation authorities, the Joint Aviation Authorities and the International Civil Aviation Organization.
Joint efforts between industry and government officials to study aviation safety problems are not new. In prior years, government and industry convened various joint teams to review aviation safety issues and make recommendations; however, according to Safer Skies participants, those earlier teams did not always include representatives from major organizations who were responsible for aviation safety. As a result, FAA was not always successful in obtaining consensus on the safety interventions that those teams recommended. Safer Skies participants noted that the level of participation and cooperation for this initiative is unprecedented among the major groups responsible for aviation safety and should enhance FAA’s chances of implementing the safety interventions made by the various teams.
Moreover, the initiative coordinated ongoing aviation safety work that was being conducted independently by FAA, industry, and other federal agencies. For example, aircraft manufacturers had initiated an exhaustive study on ways to prevent uncontained engine failure. FAA eventually joined the aircraft manufacturers in this study, and it subsequently became part of the Safer Skies agenda. In addition, the industry and FAA had been conducting independent studies on runway incursions and CFIT. Under the initiative, representatives from the aircraft manufacturers, airline industry, and government are members of the teams studying 16 safety problems, and together they will decide on the strategies to address them.
section_title: The Funding, Prioritization, and Evaluation of Safer Skies Interventions Could Be Better Coordinated
paragraphs: While coordination between government and industry organizations participating in the initiative has been extensive, we identified three areas in which coordination could be improved. First, although FAA has committed to funding interventions approved by the Safer Skies steering committees, uncertainty remains about the agency’s ability to fund these safety interventions. The steering committees for commercial aviation and general aviation have both sought commitment to the implementation and funding of interventions before giving final approval to move forward. However, FAA’s commitment has come at different points in the approval process for interventions recommended by these steering committees, and FAA’s commitment to the general aviation interventions was still uncertain even after some industry and FAA officials believed the steering committee had given its final approval. As a consequence, general aviation participants were more skeptical about whether FAA would implement or fund their safety interventions. Second, it remains unclear what process will be used to reprioritize available resources if funding is limited. Finally, Safer Skies steering committees, FAA, and DOT have not coordinated how they will measure Safer Skies’ progress in achieving the goal of reducing the fatal accident rate in commercial aviation by 80 percent by 2007.
subsections: section_title: Skepticism Persists About FAA’s Ability to Fund Safety Interventions
paragraphs: Skepticism persists about whether FAA can or will be able to honor its commitments to fund the interventions approved by the Safer Skies steering committees to reduce the fatal accident rate. This is especially true in the general aviation community. This skepticism results partly because the process for approving and funding Safer Skies interventions has worked differently for general aviation than it has for commercial aviation thus far. This has contributed to differing perceptions about the likelihood of the funding and implementation of interventions. These perceptions have resulted in part from the different processes used by the two steering committees to seek approval and funding from participating organizations, from the way interventions have moved forward within these two Safer Skies committees, and from FAA’s handling of the interventions recommended by them.
subsections: section_title: The Process for Final Approval of Interventions Has Worked Differently in the Two Steering Committees
paragraphs: The final approval of recommended safety interventions has worked differently in the commercial aviation and general aviation steering committees. The commercial aviation steering committee has documented its process for approving interventions, which involves members’ gaining the approval of their respective organizations for both implementation and funding. This approval comes in two stages. First, steering committee members brief their respective organizations on the general concept of each intervention under consideration and seek preliminary approval of each intervention. Changes and modifications may be suggested by the organizations. For organizations that will be involved in the implementation of an intervention, the preliminary approval also involves a tentative commitment to fund the cost of implementing any interventions for which they are responsible. Once members grant preliminary approval, the steering committee asks the team to draw up detailed implementation plans for each intervention. These implementation plans are then submitted to the steering committee for the next level of approval. Members subsequently seek final approval of these plans from the organizations they represent, including firm resource and funding commitments if appropriate. When participating organizations concur with the detailed implementation plans, the steering committee grants final approval. To date, most of the commercial aviation teams have forwarded a few interventions at a time for final approval by the steering committee, rather than complete lists of interventions to address multiple aspects of complex safety problems, such as CFIT. Thus, when the commercial aviation steering committee has given its final approval for an intervention, members interviewed told us they assumed that the intervention had a high priority and that implementation would take place because the organizations responsible for implementation had already committed both the staff and funding needed.
In contrast, the general aviation steering committee had not documented its process for approving interventions at the time of our review, although it recently developed draft procedures, according to FAA’s response to our draft report.Furthermore, commitment to provide resources for them is still pending, although some members of both FAA and industry who serve on the steering committee understood that final approval had been given to the interventions chosen to address CFIT and weather. Once these two implementation teams submitted their draft reports to the general aviation steering committee, the steering committee asked members to have their organizations review and comment on each intervention. This process resulted in preliminary approval or disapproval of the concept of each intervention, in some cases after the intervention was modified. Organizations responsible for the implementation of interventions also were expected to give a tentative commitment to fund the cost of their implementation. The steering committee then asked the teams to develop detailed implementation plans for each intervention and to submit those for its final approval. These two teams recommended and developed plans for a total of 17 interventions, many of which involve subinterventions and will require substantial resources either in the form of staff or funding from FAA. Because of the number and potential cost of interventions contained in the two general aviation reports, FAA requested that the general aviation steering committee prioritize the interventions. The general aviation steering committee prioritized the interventions in the letter that transmitted the final CFIT and weather reports to the FAA Administrator in March 2000. Unlike the commercial aviation teams, which have presented one intervention at a time to the steering committee, the general aviation teams have presented their complete series of interventions for each safety problem.
As the general aviation CFIT and weather reports moved toward final approval, however, confusion arose. Some industry and FAA participants believed that these reports had received final approval. This perception is supported by a March 22, 2000, letter from the industry and FAA co-chairs of the general aviation steering committee transmitting to the FAA Administrator the final CFIT and weather implementation reports with their detailed implementation plans. The letter and accompanying reports identified high-priority interventions for immediate implementation. These participants were concerned because FAA was still undecided which interventions would actually be implemented and funded. In contrast, FAA’s informal written comments in response to our draft report state that final approval has not been given to either implementation report and depends on the completion of detailed implementation plans by the FAA offices responsible for carrying out the implementation. According to the Director of Aircraft Certification, confusion arose because some members of the steering committee had “misperceptions” about what levels of approval had been agreed to.
section_title: FAA’s Internal Review and Funding Process for Safer Skies Interventions Has Led to Some Uncertainty About Whether Some Interventions Will Be Funded
paragraphs: FAA’s internal review and funding process for Safer Skies interventions has led to uncertainty about whether some interventions will be funded, in part because interventions forwarded by the commercial aviation and general aviation steering committees have been handled somewhat differently thus far. Like the other organizations participating in the initiative, FAA must commit its own resources to the interventions that it is responsible for implementing. In October 1999, FAA formed an executive council to help coordinate the implementation of the agency’s safety agenda, including how to provide funding and staff resources for Safer Skies interventions. The executive council includes the heads of each of FAA’s major program offices, its general counsel, and a regional administrator. The executive council has not yet documented its process for approving and funding interventions, however, and it remains unclear at what point FAA is committing resources to implement Safer Skies interventions. This uncertainty has led to different perceptions on the part of some FAA and industry participants about the likelihood that interventions will be implemented and funded.
FAA staff working on the initiative described differences in the way the executive council has handled interventions proposed by the two steering committees. These differences have resulted in a clear indication of funding for commercial aviation interventions before that steering committee’s final approval is given, while the general aviation steering committee’s final approval was given on a series of weather and CFIT interventions that have yet to be approved and funded by FAA. When proposed Safer Skies interventions are under serious consideration by the steering committees, they are also presented to the executive council for discussion of their possible impact on workload and budget, according to FAA staff who serve as co-chairs of the two steering committees. The executive council provides feedback to the steering committees before interventions are approved. FAA staff serving on Safer Skies committees presented conflicting views, however, of when FAA commits to funding interventions. Several of the FAA staff interviewed said that FAA’s commitment of staffing and funding to commercial aviation interventions occurs before that steering committee gives its final approval to interventions.
However, the Director of FAA’s Aircraft Certification Service, who serves as co-chair of the commercial aviation steering committee, described the executive council’s role as having more room for interpretation of the intervention and a subsequent determination of whether funding is available. She said that, once the intervention is approved, the executive council again discusses it, determines whether to accept it as stated or to modify it, assigns it to an FAA office for implementation, and determines how it fits in with the office’s existing priorities. The program office then reviews the intervention, can suggest modifications that will achieve the same goal, and determines whether the intervention can be accomplished with existing resources or requires a request for additional funding. She said that the executive council could also request that the steering committee modify or prioritize interventions. For example, she said that FAA agreed to implement the commercial aviation CFIT team’s recommendation to develop precisionlike airport approaches,concluded that the agency’s resources would not permit the completion of approaches for all airports in the time frame envisioned by the intervention, and is now working with the steering committee to identify which airports present the greatest risks and should be completed first. Similarly, she said that the council asked that the general aviation steering committee approve a different way to accomplish one intervention without hiring additional staff and prioritize its list of CFIT and weather interventions according to which ones will have the most impact on improving safety and reducing fatalities. Because the executive council’s role is new and its procedures remain undocumented, confusion persists about when FAA commits its resources to implementing the safety interventions approved by the steering committees. For example, although FAA’s executive safety council had agreed in principle to the highest priority interventions to address general aviation safety problems caused by weather and CFIT, FAA’s response to our draft report indicated that final approval and funding depend on the completion of detailed implementation plans. As a consequence, several Safer Skies participants from FAA and industry, especially those working on general aviation issues, expressed some concern about whether the recommended interventions would be funded or implemented.
These concerns stem partly from FAA’s past record for implementing safety recommendations. FAA’s budget does not specifically identify and commit resources to implementing Safer Skies interventions. For example, FAA has no funds set aside in its budgets for fiscal years 2000 or 2001 for general aviation interventions. However, FAA’s Deputy Associate Administrator for Regulation and Certification said that the agency’s approach to budgeting is to retain flexibility by not identifying specific budget amounts for such efforts as the Safer Skies initiative. While we do not advocate including specific Safer Skies line items in FAA’s budget, the uncertainty about funding and implementation also exists because FAA has either not fully funded or not implemented some safety recommendations in the past. Several industry participants in the initiative specifically mentioned concerns about FAA’s lack of follow through on safety recommendations to decrease the number of runway incursions. Although FAA has received many recommendations for reducing runway incursions, continuing problems in this area have been partially attributable to insufficient funding of the safety plans FAA developed, according to DOT’s Inspector General.Additionally, after initially planning to fund the agency’s new inspection system,FAA has still not provided funding to hire analysts to review inspection data on the nation’s 10 major airlines for possible safety concerns. While FAA has implemented many safety recommendations over the years, concerns still persist about the agency’s ability to fund new safety initiatives. Greater assurance about the implementation of Safer Skies interventions could be provided in two ways. First, as mentioned in chapter 3, stronger mechanisms for tracking the implementation of interventions from all three steering committees need to be established. Second, clarifying FAA’s process for committing resources for implementing interventions would provide greater assurance of their implementation. Both of these steps would improve coordination between FAA and other Safer Skies participants.
Thus far, the interventions approved by steering committees have not required a major commitment of time and resources by either FAA or industry groups. But future interventions may require substantial resources not included in FAA’s current budget, and choices may have to be made about which interventions to fund. Furthermore, FAA addresses and funds many issues beyond those on the Safer Skies agenda, including security issues and improvements to the air traffic control and airport infrastructure. FAA’s executive council provides a forum for agency managers to discuss and prioritize program and resource needs. However, without clear priorities and a unified aviation safety agenda that also takes such issues into account, FAA will continue to address aviation piecemeal, rather than as an integrated system. While the Safer Skies initiative represents a major step in the direction of coordinating the nation’s aviation safety agenda, a more far-reaching effort has not yet been undertaken to coordinate the nation’s complete aviation agenda.
section_title: The Initiative Does Not Have A Process for Prioritizing Interventions to Ensure the Implementation of Those With the Greatest Potential to Reduce the Fatal Accident Rate
paragraphs: The initiative has not developed a process for prioritizing interventions to ensure the implementation of those with the greatest potential to reduce the fatal accident rate if funding is limited. The initiative has involved prioritization at several points thus far. First, the teams addressing safety problems in commercial aviation and general aviation have prioritized the interventions they considered. For example, the general aviation weather team considered numerous possible safety interventions and eventually developed a list of 17 that it presented in order of priority. The steering committees have also prioritized interventions. For example, the commercial aviation steering committee has moved quickly on several interventions that the CFIT implementation team considered as having a high priority and potential for effectiveness. At the request of the executive council, the general aviation team created a unified list to prioritize its CFIT and weather interventions. Given the constraints of FAA’s budget, such prioritization is critical to ensuring that funds are expended on the interventions that will be most effective in reducing the fatal accident rate.
The ability to reprioritize resources for Safer Skies interventions and other aviation work may also become critical. The Safer Skies team has just begun work on loss of control—the safety problem that caused the greatest number of fatal accidents in commercial aviation in 1988-97. Interventions to address loss of control are thus likely to be critical for reducing the fatal accident rate. If funding is limited, this may mean reprioritizing funding from existing programs and Safer Skies interventions that have already been approved to those with more potential to reduce the fatal accident rate and save lives. The initiative’s success will depend in part on its ability to identify those interventions with the most potential impact and to prioritize their implementation and funding. Safer Skies steering committees and FAA’s executive council have not yet established any process for reprioritizing interventions if funding is limited.
section_title: Safer Skies Steering Committees, FAA, and DOT Have Different Ways of Measuring Progress in Reducing Commercial Aviation’s Fatal Accident Rate
paragraphs: A lack of coordination among Safer Skies steering committees, FAA, and DOT has resulted in their having different ways of measuring whether the goal of reducing the fatal accident rate for commercial aviation by 80 percent is achievable by 2007. DOT is responsible for setting safety goals for all modes of transportation under its authority, including aviation. Generally, FAA and other agencies under DOT have established specific goals and use measurements that evaluate their progress in meeting those goals that are in line with those set by DOT. But currently, DOT and FAA measure progress toward the goal of an 80-percent reduction in the fatal accident rate for commercial aviation in different ways. DOT’s Performance Plan for fiscal year 2001 establishes goals for reducing the fatal accident rate in commercial aviation that rely on the Safer Skies initiatives. To determine the progress made in reducing the rate, DOT’s plan uses aircraft flight hours as the activity measure. In contrast, the commercial aviation steering committee and FAA use aircraft departures as the measure of aviation activity. Because DOT, FAA, and Safer Skies all share a common goal of reducing the fatal accident rate, consistency would be desirable in the aviation activity measure they use to calculate the progress being made toward that goal. Since most commercial aviation accidents occur during takeoff and landing, we believe that using departures would better measure the effectiveness of the Safer Skies interventions for commercial aviation.
section_title: Conclusions
paragraphs: Additional steps need to be taken to ensure that those safety interventions most critical to reducing the nation’s fatal accident rate are given top priority and funding. If FAA’s process for prioritizing and funding Safer Skies interventions is not clarified, there is no assurance that the agency will be able to implement these interventions. If funding is limited, a process may well be needed for reprioritizing available staffing and funding to ensure that the interventions with the greatest potential for reducing the nation’s fatal accident rate and saving lives are implemented first. Even if Safer Skies steering committees and FAA agree on the priorities for the nation’s safety agenda, these priorities will continue to compete for resources with other aviation needs until FAA develops a unified aviation agenda. Finally, FAA, the Safer Skies commercial aviation steering committee, and DOT are not using the same aviation activity measure to calculate the progress of Safer Skies interventions in reducing the fatal accident rate for commercial aviation. Consequently, they may reach different conclusions about the effectiveness of the Safer Skies interventions in achieving the goal of reducing the fatal commercial aviation accident rate by 80 percent by 2007.
section_title: Recommendations
paragraphs: To ensure the implementation of the Safer Skies safety interventions, we recommend that the Secretary of Transportation direct the FAA Administrator to clarify the executive council’s process for committing to the funding and implementation of interventions and coordinate with the Safer Skies steering committees about the meaning and timing of this commitment. To ensure that the interventions with the greatest potential for reducing the fatal accident rate and improving aviation safety receive needed resources, we recommend that the Secretary of Transportation direct the FAA Administrator to ensure that the executive council has a process in place for reprioritizing interventions if funding is limited.
To ensure that the extent of progress toward reducing the fatal accident rate for commercial aviation is measured consistently, we recommend that the Secretary of Transportation ensure that DOT, FAA, and the Safer Skies commercial aviation steering committee all use departures as the activity measure for calculating the rate.
section_title: Agency Comments
paragraphs: DOT and FAA officials concurred with our recommendations to clarify the executive council’s process for committing to the funding and implementation of interventions and to use departures as the activity measure for calculating the fatal accident rate in commercial aviation. They disagreed with our recommendation that FAA’s executive council should develop a process for reprioritizing interventions if funding is limited. The officials said that such reprioritization falls under the agency’s normal processes for reprogramming funding. However, the role of the executive council is to help coordinate the implementation of the agency’s safety agenda—including how to provide funding and staff resources for Safer Skies interventions. We believe that it would be useful for the executive council to establish some basic criteria and processes for evaluating and comparing the potential impact of existing and emerging safety interventions. For this reason, we did not modify or withdraw our recommendation.