Aviation Safety: Enhanced Oversight and Improved Availability of Risk- Based Data Could Further Improve Safety
Released: 2011-10-13T00:00:00

Letter

Background

The terminal area is the area around an airport extending from the airfield or surface to about 10,000 feet vertically and out to about 40 miles in any direction. The terminal area includes airport surface areas such as runways, taxiways, and ramps, as well as the airspace covered by air traffic control towers—typically within 5 miles of a towered airport—and by terminal radar approach control (TRACON) facilities, which typically handle air traffic to within about 40 miles of an airport (see fig. 1).

Terminal area safety incidents can occur on the surface at airports or in the airspace around them. Surface incidents may threaten the safety of aircraft, passengers, and airport workers, among others. Terminal area safety incidents that happen on runways and taxiways include incursions and excursions. Runway incursions typically involve the incorrect presence of an aircraft, vehicle, or person on a runway, and runway excursions generally occur when an aircraft veers off or overruns a runway (see fig. 2). Ramp incidents can involve aircraft or airport vehicles, such as baggage carts or ground handling vehicles, as well as airline and airport employees and others.

Airborne safety incidents in the terminal area often involve a loss of the minimum required distance between aircraft—as airplanes fly too close to each other—or as individual aircraft fly too close to terrain or obstructions.

These incidents are called “losses of separation,” because there is a violation of FAA separation standards that ensure established distances are maintained between aircraft or other obstacles while under the con of air traffic controllers. Generally, air traffic controllers must maintain either vertical or horizontal separation between aircraft (see fig. 3), and losses of separation occur when both of these measures are violated, based on phase of flight and size of the aircraft.

Safety in the terminal area is a shared responsibility among FAA, airlines, pilots, and airports. FAA air traffic controllers oversee activity on runways and taxiways, and airlines and airports provide primary safety oversight in ramp areas. Several FAA offices have a role in terminal area safety including:  The Office of Runway Safety (Runway Safety) within the Air Traffic Organization (ATO) Safety Office was established in 1999 and le and coordinates the agency’s runway safety efforts. Its primary mission is to improve runway safety by decreasing the number a severity of runway incursions. Runway Safety is responsible for nd developing a national runway safety plan and performance measures for runway safety and evaluating the effectiveness of runway safety activities. The office currently has an acting director. Other FAA offices, including the Office of Aviation Safety, the Office of Airports, other components of ATO, and regional offices sup port Runway Safety’s work to identify hazards and analyze risk.  ATO manages air tr affic control and develops and maintains runway safety technology.  The Office of Aviation Safety and Flight Standards Service (Flight Standards) within it conduct safety inspections of airlines, audit air traffic safety issues, and administer a program to obtain information about safety incidents involving pilots.  The Office of Airports oversees airport-related safety, including airpo infrastructure. This includes issuing airport operating certificates to commercial service airports, establishing airport design and safety standards, and inspecting certificated airports. The Office of Airports also provides Airport Improvement Program (AIP) grants to airports to help support safety improvements.

Airlines and airports typically oversee the safety of operations in ra areas. Ramp areas are typically small, congested areas in which departing and arriving aircraft are serviced by ramp workers, who include baggage, catering, and fueling personnel. These areas can be dange for ground workers and passengers. As noted in our 2007 report on runway and ramp safety, FAA’s oversight of ramp areas is generally provided indirectly through its certification of airlines and airports. Bot NTSB and OSHA investigate accidents in the ramp area. Thus, NTSB investigates ramp accidents—and other accidents involving aircraft—that occur from the time any person boards an aircraft with the intention t o fly until the time the last person disembarks the aircraft, if the accident h results in serious or fatal injury or substantial aircraft damage. OSHA can conduct an inspection in response to a fatality, injuries, or a complaint, unless it is preempted by an exercise of statutory authority by FAA.

FAA collects and analyzes information about various safety incidents in the terminal area in order to track incidents, identify their causes, and assign severity levels. Currently data are collected at towered airports for runway incursions, some other surface incidents, and for airborne incidents. By contrast, no complete data are collected for incidents in ramp areas. FAA categorizes incidents according to the actions or inactions of air traffic controllers, pilots, or others, such as pedestrians or vehicle operators. Table 1 provides hypothetical examples of each type of incident. Depending on the type of incident identified—air traffic control surface event, operational error or deviation, pilot deviation, or pedestrian/vehicle deviation—different offices within FAA are responsible for investigating individual incidents.

FAA is in the process of implementing a data-driven, risk-based approach to safety oversight that FAA expects will help it continuously improve safety by identifying hazards, assessing and mitigating risk, and measuring performance. For decades, the aviation industry and FAA have used data reactively to identify the causes of aviation accidents and incidents and take actions to prevent their recurrence. Using a safety management system approach, the agency plans to use aviation safety data to identify conditions that could lead to incidents, allowing it to address risks proactively. FAA’s current approach for analyzing information about safety in the terminal area includes separate approaches for surface and airborne incidents.  Surface incidents. For runway incursions, Runway Safety collects information from the Administrator’s Daily Bulletin and the Air Traffic Quality Assurance database (ATQA)—a mandatory reporting system with incident information recorded by FAA air traffic controller supervisors, support specialists, and managers—and other sources such as incident investigations. Runway Safety determines how an event will be categorized (e.g., air traffic control, pilot, or vehicle/pedestrian deviation, etc.), and the runway incursion severity classification team, which consists of representatives from the Office of Airports, Flight Standards, and ATO Terminal Services, determines the severity of the incursion.  Airborne incidents. For terminal area incidents that occur in the air, the primary source of information is ATQA. FAA recently adopted a new process for analyzing these incident data and has taken steps to increase the amount and quality of information collected. FAA officials stated that, prior to this change, data were limited to the information collected in ATQA from FAA managers and supervisors, with limited input from individual controllers through controller statements gathered during incident investigations. We will discuss the new system in more detail in the following section.

FAA Has Taken Actions to Reduce Risk in the Terminal Area

Procedural and Technological Changes to Improve Runway Safety

FAA has taken several steps since 2007 to further improve surface safety at airports, focusing most notably on efforts to reduce the number and severity of runway incursions—the agency’s key performance measures for this area. (See fig. 4.) As part of its 2007 Call to Action Plan, the agency implemented new safety approaches and developed milestones for the implementation of various mid- and long-term initiatives, such as conducting safety reviews of 20 airports where incursions were of greatest concern, upgrading airport markings at airports, and reviewing cockpit and air traffic procedures. Additionally, FAA’s 2009–2011 National Runway Safety Plan establishes priorities for each FAA office involved in reducing incursion risks and identifies performance targets for reducing the risk of runway incursions, including an overall goal to reduce total runway incursions by 10 percent from 1,009 in fiscal year 2008 to 908 incursions by the end of fiscal year 2013. In 2010, FAA issued an order that further strengthened the role of Runway Safety as the agency’s focus for addressing incursions and improving runway safety. FAA has also proposed new rules related to airport safety management systems that address ramp areas.

Additionally, FAA established local and regional runway safety action teams that assess runway safety issues at particular airports, formulate runway safety action plans to address these concerns, and execute their runway safety programs. FAA also established the Runway Safety Council (Council) with aviation industry stakeholders to develop a systemic approach to improving runway safety. The Council’s Root Cause Analysis Team—comprised of representatives from FAA and airlines—investigates severe runway incursions to determine root causes in order to identify systemic risks. The Root Cause Analysis Team presents recommendations to the Council, which in turn, assigns accepted recommendations to FAA or the aviation industry, based on which is best able to address root causes and prevent further incursions. The Council is responsible for tracking recommendations and ensuring that they get implemented.

FAA’s layered approach to addressing runway safety includes a range of actions, such as encouraging airport improvements, including improving runway safety areas; changes to airport layout and runway markings, signage, and lighting; providing training for pilots and air traffic controllers; mitigating wildlife hazards; and researching, testing, and deploying new technology. According to its 2009–2011 National Runway Safety Pla annual safety reports, FAA’s efforts to decrease the risk of surface incidents include: Improving runway safety areas. In order to reduce fatalities and injuries from runway excursions, the Office of Airports has provide between $200 and $300 million annually since 2000 through AIP grants to improve runway safety areas, which are unobstructed areas surrounding runways.  Outreach to general aviation pilots. Regional runway safety action team meetings, briefings, and clinics for general aviation pilots and flight instructors discuss the importance of runway safety and how to avoid incursions. FAA also provided training, printed materials, and electronic media such as DVDs explaining runway safety.  New terminology. FAA adopted international air traffic co terminology for taxi clearance instructions to help avoid miscommunication between pilots on the taxiway and runway a traffic controllers. These included new mandatory detailed taxi instructions, including directional turns, fo taxiing to and from ramps and runways. r all aircraft and vehicles  Upgraded markings. Markings—such as enhanced centerlines drawn on taxiways and runways—wer commercial airports in 2010. e installed at all 549 FAA-certificated  Hot spot identification. Hot spots—locations on runways or taxiways with a history of collisions or incursions or the heightened potentia such incidents—have been identified on airport diagrams to alert pilots of complex locations on runways and taxiways.  Airport layout. Some airports have relocated taxiways, allowing pilots to avoid crossing active runways during the taxi phase. These “end around taxiways” facilitate ground movement and minimize conflict with aircraft operating on runways. The Office of Airports has alsoreleased guidance on the design of taxiways and aprons to help prevent runway incursions.  Training. FAA has developed video programs, training modules, and best practices for pilots, controllers, and airport personnel aimed at heightening awareness of situations that could lead to incursions. F now also requires that runway incursion prevention be included in ning for controllers, pilots, and all certificated airport refresher trai employees.  Research and development of best practices and other useful information. Runway Safety’s Web site has resources, best prac and statistics on runway safety. Moreover, Runway Safety has produced DVDs and Pilot’s Guide brochures, as well as runway incursion safety alerts for airport operators.  Wildlife Hazard Mitigation. In addition to an active research program for developing practical techniques for mitigating bird strikes, F encouraged all certificated airports to conduct wildlife hazard assessments and is pursuing rulemaking to make it mandatory for certificated airports to do so. FAA currently provides AIP funds to hire qualified wildlife biologists to develop assessments and mitigation plans, as needed.

A number of available technological systems are intended to help re the number and severity of runway incursions, and FAA has made progress installing several of these systems since 2007. For example order to prevent collisions, FAA completed installation of the Airport Surface Detection Equipment, Model X (ASDE-X) system at 35 major airports, which provides air traffic controllers a visual representation of traffic on runways and taxiways (see fig. 5). Other systems that will provide safety information directly to pilots are being installed or tested. For example, runway status lights, an automatic series of lights that giv pilots a visible warning when runways are not clear to enter, cross, or depart on, are planned to be installed at 23 airports by August 2016.

See appendix II for more information on safety.

To date, runway excursions have not received the same level of attention from FAA as incursions. However, excursions can be as dangerous as incursions; according to research by the Flight Safety Foundation, excursions have resulted in more fatalities than incursions globally. FAA is now planning efforts to track and assess excursions as well. According to FAA officials, in response to recommendations that we and others hav made, Runway Safety will begin overseeing runway excursions on October 1, 2011. Specific responsibilities include collecting and analyzing data to develop steps to reduce the risk of such incidents. According to FAA officials, the office plans to develop an official definition of an excursion, develop a data collection instrument and performance metrics e that would enable it to collect and evaluate excursion data, and develop training and steps to help mitigate excursions. According to a timeline from Runway Safety, it will be several years before this program is totally implemented and FAA has detailed information about excursions.

FAA recently issued two proposed rules for airports under the agency’s authority to issue airport operating certificates. The first proposed rule, issued in October 2010, would require airports to establish safety management systems for ramps areas, as well as other parts of the airfield, including runways and taxiways. As previously noted, FAA historically has not primarily overseen safety in ramp areas, which are typically controlled by airlines or—to a lesser extent—airports using their own practices. FAA’s proposal would require airports to establish safety management systems for the entire airfield environment in order to ensure that individuals are trained on the surface of the terminal area; safety implications of working on the  hazards are identified proactively, and analysis systems are in place;  data analysis, tracking, and reporting syste ms are available for trend analysis and to gain lessons learned; and there is timely communication of safety issues to all stakeholders.

A second proposed rule, issued in February 2011, would establish minimum training standards for all personnel who access ramp areas.

Required training would occur at least yearly and include familiarization with airport markings, signs, and lighting, as well as procedures for operating in the nonmovement (ramp) area. The public comment period for these proposed rules closed during July 2011. FAA has not indicated when the rules would be finalized.

We reported in 2007 that FAA lacked ground handling standards for ramp areas. In the absence of agency standards, other organizations have developed tools to improve ramp safety. For example, the Flight Safety Foundation has collected best practices and developed a template of standard operating procedures to assist ramp supervisors in developing their own procedures. The guidelines are wide ranging and include the reporting of safety information, ramp safety rules, the positioning of equipment and safety cones, refueling, and caring for passengers, among other areas. In addition to the Flight Safety Foundation guidelines, the International Air Transport Association, an international airline association, has developed a safety audit program for ground handling companies aimed at improving safety and cutting airline costs by drastically reducing ground accidents and injuries. The program is available to all ground service providers, who, after successfully completing the audit, are placed on a registry. As of August 2011, Seattle- Tacoma International Airport is the only domestic airport participating in the program.

Data Collection and Risk- Based Analysis of Airborne Aviation Safety Information

Controllers are required to report any occurrence that may be an operational deviation, operational error, proximity event, or air traffic incident if the reported issue is known only to the employee and occurs while the employee is directly providing air traffic services to aircraft or vehicles or first level watch supervision. concerns have been addressed through informal discussions between ATSAP officials and FAA facilities.

In other efforts to obtain more safety data, FAA has taken other steps to make incident reporting less punitive. For example, in July 2009, FAA changed its incident reporting policy such that individually identifying information, such as air traffic controller names and performance records, is no longer associated with specific incidents in ATQA, the central FAA database used by air traffic control managers or supervisors to report incidents. In addition, in July 2010, FAA also stopped issuing incident “not to exceed” targets to individual air traffic control facilities (e.g., towers, TRACONS, or en route facilities). According to officials, these targets created an incentive for underreporting of less serious incidents by supervisors at the facility level, and the targets were discontinued in order to encourage increased reporting at the agency.

FAA is also implementing new technologies, specifically, the Traffic Analysis and Review Program (TARP), an error detection system that can be used to automatically capture losses of separation that occur while aircraft are under the control of air traffic control towers and TRACONs. Historically, FAA relied on air traffic controllers and their supervisors to manually report on operational errors, something we have noted in the past may negatively impact data quality and completeness. TARP automatically captures data on all airborne losses of separation, which, according to officials, will increase the volume of data FAA gathers on air traffic safety incidents and enable FAA to obtain a more complete picture of potential safety hazards. According to the fiscal year 2010 FAA Performance and Accountability Report, FAA has deployed TARP at 150 air traffic control tower and TRACON facilities. According to FAA officials, TARP is currently being used as an audit tool for approximately 2 hours per month at some facilities—in lieu of full-time use at all facilities—but further implementation of the system has been delayed as the agency evaluates the impact of the system on controllers and determines how the system will be used and how to handle the additional workload that will be created as more incidents are captured and require investigation. Following the completion of these steps, FAA will take 210 days to fully deploy TARP. Currently, incidents identified through TARP are being included in official incident counts.

FAA is also shifting to a new, risk-based process for assessing a select category of airborne losses of separation. FAA began using the Risk Analysis Process (RAP)—which is adapted from a similar process used by the European Organization for the Safety of Air Navigation (Eurocontrol)—in fiscal year 2010. While the new process is being established, RAP will be used in tandem with the existing system. RAP is currently limited to Losses of Standard Separation (LoSS). This subset of airborne incidents includes those in which the separation maintained is less than 66 percent of the minimum separation standard for the planes involved. Under RAP, FAA determines both the severity and the repeatability of selected LoSS events (that is, how likely a certain LoSS event will occur again at any airport under similar circumstances based on a number of factors). These factors include proximity of planes to one another at the time of the event, rate of closure between planes, controller and pilot recovery, and whether or not Traffic Collision Avoidance System (TCAS) technology is triggered by the incident. Prior to the development of RAP, FAA categorized losses of separation based on proximity alone: the greater the loss of separation between two planes, the greater the severity of the incident in question. Operational errors were then rated on an A–C scale, with those that retained more than 90 percent of required minimum separation categorized as proximity events. (Fig. 7 compares the threshold for review of incidents in each system.) Officials stated that RAP is more robust than the previous system because it is able to take numerous factors into account when determining event severity, as well as overall risk to air traffic safety. In addition, the RAP will assess risk for LoSS events that were not assigned a severity category under the old system. As a result, they said the agency will be better equipped to identify systemic issues in air traffic safety and to issue related corrective action requests. Based on analysis of systemic issues identified across incidents, RAP released its first five corrective action requests on July 19, 2011, which were developed to mitigate specific hazards that contribute to what RAP has identified as the highest risk events.

We, the Department of Transportation Inspector General (IG), and NTSB have raised concerns about terminal area safety. For example, we recommended, in 2007, that FAA take several steps to enhance runway and ramp safety, such as updating its national runway safety plan, collecting data on runway excursions, and working with OSHA and industry to collect and analyze better information on ramp accidents. In 2007, FAA put in place a Director for Runway Safety and issued a Call to Action aimed at reducing the risk of incursions following several high- profile incidents (see table 2 for select recommendations to FAA). The IG also made recommendations to FAA about runway safety issues and recommended that FAA take several steps to reduce the risk of airborne incidents and improve oversight of this area. For example, the IG recommended that FAA clearly document the severity ratings used by FAA for runway incursions, revise the national plan for runway safety, and realign the Office of Runway Safety. With regard to airborne incidents, the IG recommended establishing a process to rate the severity of pilot deviations and corresponding performance goals, developing milestones for implementing TARP, and assuring that Flight Standards works with ATO Safety Services to determine whether losses of separation are pilot or controller errors, among other recommendations. Further, NTSB continues to include runway safety, safety management systems, and pilot and air traffic controller professionalism issues on its list of most wanted safety improvements.

Reported Surface and Airborne Incidents Have Increased, and Several Key Factors Likely Contribute to Trends

Rate of Reported Runway Incursions Has Increased since 2004, but Serious Incidents Have Significantly Declined

In fiscal years 2009 and 2010, the agency met its interim goals toward reducing the total number of runway incursions at towered airports, but the rate of incursions per million operations continued to increase (see fig. 8). As noted in our 2007 report, both the number and rate of incursions reached a peak in fiscal year 2001, prompting FAA to focus on runway safety. The number and rate of incursions at towered airports decreased dramatically for a few years thereafter, though the impact of FAA’s efforts on these outcomes is uncertain. Beginning in 2004, however, both the number and rate of incursions began increasing again. For example, in fiscal year 2004, there were 733 incursions at a rate of 11.4 incursions per 1 million tower operations, compared with fiscal year 2010, when there were 966 incursions at a rate of 17.8 incursions per 1 million such operations. Although the rate of incursions at towered airports continues to increase, the number of incursions at these airports peaked in fiscal year 2008.

The most serious runway incursions at towered airports—where collisions are narrowly avoided—decreased by a large amount from fiscal year 2001 to 2010, and FAA met or exceeded its goals for reducing the rate of these incidents. FAA classifies the severity of runway incursions into four categories—A through D—and its performance targets call for the reduction of the most severe incursions (category A and B) to a rate of no more than 0.45 per million air traffic control tower operations by fiscal year 2010 and for the rate to remain at or below that level through fiscal year 2013. The number of the most severe incidents at towered airports also dropped from fiscal year 2001 to 2010. Thus, category A and B incursions at these airports decreased from 53 to just 6 during that time, with category A incursions decreasing from 20 to 4, and category B incursions decreasing from 33 to 2 (see fig. 9).

In fiscal year 2010, the majority of incursions at towered airports were classified as pilot deviations (65 percent), followed by vehicle/pedestrian deviations (19 percent), and operational errors and deviations by air traffic controllers (16 percent) (see fig. 10). Further, for every year since 2001, pilot deviations comprised the majority of runway incursions at these airports, and the proportion involving these errors steadily increased from about 55 percent of all incursions in fiscal year 2001 to 65 percent in fiscal year 2010.

We previously reported that most runway incursions at towered airports involved general aviation aircraft and that trend continues. General aviation aircraft make up nearly a third of total operations at towered airports but have consistently accounted for about 60 percent of incidents each year since 2001. More specifically, the rate of incursions per million tower operations involving at least one general aviation aircraft is higher than the rate of incursions not involving general aviation aircraft, and the rate has increased every year since fiscal year 2004 (see fig. 11). Further, general aviation aircraft were involved in over 70 percent of the most serious—category A and B—incursions from fiscal year 2001 through the second quarter of fiscal year 2011. According to FAA officials, general aviation pilots may be more susceptible to incursions and other incidents because of their varying degrees of experience and frequency of flying. Furthermore, general aviation pilots do not generally undergo the same training as commercial airline pilots do.

With regard to runway excursions, our review of NTSB data found that general aviation aircraft are also involved in most runway excursions. Although FAA does not yet formally collect information on runway excursions, NTSB provided us with accident investigation reports on 493 accidents that involved runway overruns or excursions since 2008. Seven of these accidents were fatal, resulting in 14 fatalities. Our review of these reports found that 97 percent of the accidents involving excursions referred to the involvement of at least one general aviation aircraft.

In our 2007 report, we found that efforts to address the occurrence of safety incidents in ramp areas were hindered by the lack of data on the nature, extent, and cost of ramp incidents and accidents and by the absence of industrywide ground handling standards. As discussed above, FAA collects no comprehensive data on incidents in the ramp area, and NTSB does not routinely collect data on ramp incidents that do not result in injury or aircraft damage. Likewise, as mentioned above, OSHA, the primary source of ramp fatality data, does not collect data on incidents that do not result in at least three serious injuries or fatalities. In the ramp area, OSHA data on worker fatalities show the number of deaths in the ramp area to have varied between 3 and 11 from 2000 to 2010. The rate remained constant—between 4 and 6 deaths per year—from 2008 to 2010.

Rate and Number of Reported Airborne Operational Errors Increased since 2007, Including the Most Serious Incidents

The rate of reported airborne operational errors in the terminal area increased considerably in recent years. From the second quarter of fiscal year 2008 to the second quarter of fiscal year 2011, the rate and number of reported airborne operational errors increased significantly. During this time period, the rate of reported airborne operational errors in the terminal area nearly doubled, increasing 97 percent, and the number of reported airborne operational errors increased from 220 to 378. The rate of incidents began a notable climb in the fourth quarter of fiscal year 2009, peaked in the second quarter of fiscal year 2010, and remained at rates higher than the historical average through the second quarter of 2011 (see fig. 12). FAA officials attributed at least some portion of the spike in reported incidents during the second quarter of fiscal year 2010 to approximately 150 events that occurred as the result of the misinterpretation of an arrival waiver at one TRACON facility.

While the rate of airborne operational errors has increased over time in both the TRACON and tower environments, the rate of errors in the TRACON environment has increased more. Between the second quarters of fiscal years 2008 and 2011, the rate of operational errors in the TRACON environment increased from 8.5 to 22.6 operational errors per million air traffic control operations—a 166 percent increase (see fig. 13). In comparison, operational errors increased by 53 percent in the tower environment.

Overall, the rate of the most severe airborne operational errors more than doubled between the second quarter of fiscal year 2008 and the second quarter of 2011. While the least severe (category C) incidents are more numerous than the most severe, the most severe (category A) incidents increased from 5 in the second quarter of 2008 to 14 in the second quarter of 2011. In comparison, category C operational errors increased by 135 percent, and category B operational errors decreased by 5 percent.

These incident rates do not meet FAA goals under both the prior severity system and using new risk assessment measures. In fiscal year 2010, FAA reported 3.32 category A and B operational errors per million air traffic control operations, significantly exceeding its targeted rate for fiscal year 2010 of 2.05 per million operations. In fiscal year 2011, FAA replaced its operational error measure with a new measure—the System Risk Event Rate (SRER)—a 12-month rolling rate of the most serious LoSS events per thousand such events. The rate of high-risk events also increased using this measure. According to data provided by FAA, the number of the most serious LoSS events—called high-risk events in the new risk assessment process—spiked from 9 events in December 2010 to 16 events in January 2011 but has since decreased. However, the overall SRER increased significantly between December 2010 and February 2011 (from 21.9 to 29.9 high-risk LoSS events per 1000 LoSS events) and remains significantly elevated above FAA’s target of 20 serious LoSS events for every thousand such events. The SRER for the 12-month period ending in April 2011 was 28.97. According to FAA officials, the agency’s target of 20 LoSS events per thousand LoSS incidents represents the system performance baseline gathered using human reporting and may therefore be an unrealistic target as the agency moves to gathering data electronically. FAA plans to continue to collect data on and categorize events using both the old and new systems. Once FAA has completed a 2-year baseline period, it has committed to conduct an independent review of both metrics to determine whether any improvements are needed.

Several Key Factors Likely Contribute to Trends in Runway Incursions and Airborne Operational Errors

Several factors have likely contributed to recent trends in runway incursions and airborne operational errors. The agency has noted that recent increases in runway incursions and airborne operational errors are primarily attributable to changes in FAA’s reporting practices, which encourage increased reporting of incidents. We found evidence to suggest that changes to reporting policies and processes have likely contributed to the increased number of incidents reported—both into ATQA, the official database for incidents—and into ATSAP, the nonpunitive reporting system for air traffic controllers. In addition, the implementation of new technologies and procedures in the terminal area also likely contributed to an increase in the number of reported airborne incidents and runway incursions. FAA has carried out changes aimed at increasing reporting, and each of these factors may have contributed to an increase in the number of reported incidents. That said, it is possible that the increase in safety incidents in the terminal area may also reflect some real increase in the occurrence of safety incidents. As a comparison, we looked at the rate of en route operational errors, which are captured automatically by airplane tracking technology and would therefore not be expected to substantially increase by a change in reporting practices or procedures at the agency. We found that the average rate of en route operational errors in fiscal year 2010 was 38 percent higher than the year before, and that the overall rate increased 13 percent from the second quarter of fiscal year 2008 to the same quarter in 2011. According to FAA officials, some of the increase in reported en route errors may be attributable to increased confidence in the nonpunitive nature of the system—reflected by a decrease in the number of requests for reclassification of incidents from en route facilities.

Changes to reporting processes and policies at FAA may explain in part the recent upward trend in reported runway incursions and airborne operational errors. Since operational errors and other losses of separation in both the tower and TRACON environments are currently reported manually by FAA supervisors and quality assurance staff into ATQA, changes in reported error rates may be partially attributable to changes made to encourage more comprehensive reporting of incidents. Most notably, as previously discussed, FAA changed its incident reporting policy in July 2009 such that individually identifying information, such as air traffic controller names and performance records, are no longer associated with specific events in the ATQA database. According to officials, this change may encourage controllers to share more information about incidents with supervisors and quality assurance officers. In addition, in fiscal year 2010, FAA stopped issuing incident “not to exceed” targets to individual facilities. According to officials, these targets may have led supervisors in the past to underreport less serious incidents in order to meet these targets. These policy changes may have increased reporting to an extent that these effects are apparent in incident rates. (See fig. 15 for recent FAA changes to reporting practices overlaid on report operational errors.)

Implementation of a nonpunitive, confidential, system of reporting for air traffic controllers may have also contributed to the real increase in the occurrence of operational errors, according to FAA officials. While the implementation of ATSAP may affect reporting rates—either by increasing reporting or by lowering the number of reports to supervisors given that the system satisfies reporting requirements—officials told us that it could also inadvertently lead to an actual increase in the occurrence of operational errors or deviations. According to these officials, the reduced personal accountability ATSAP provides may make some air traffic controllers less risk averse in certain situations. In addition, officials also noted that ATSAP may present a barrier to managerial efforts to directly manage controller performance. For example, if a report is filed into ATSAP, a supervisor may have limited options to assign training or take other corrective actions in response to an incident, even if he or she is aware that an error was made by an individual air traffic controller, presuming the incident did not involve alcohol or drug use or other such violations.

The implementation of ATSAP may have resulted in increased reporting of incidents, although reporting into this system does not directly affect official trends in operational errors. According to FAA officials, the confidential, nonpunitive nature of ATSAP has contributed to a positive change in the reporting at FAA. As a result, errors that previously may have gone unreported by air traffic controllers are now being reported to ATSAP. However, data entered into ATSAP are not directly available to FAA and do not feed into ATQA (see fig. 16). In addition, it is possible that some incidents that would have previously been reported to FAA are now being reported only to ATSAP, thus decreasing the number of incidents reported to FAA. According to ATSAP data, approximately 35 percent of all incidents reported to ATSAP in 2010 were “known” to FAA—meaning that the incident was reported into ATQA by a supervisor or manager, as well as into ATSAP by an air traffic controller—while the other 65 percent were not official reported to FAA.

Implementation of new technologies in the terminal area may also impact recent trends in surface incidents and airborne losses of separation. For example, since FAA’s ongoing implementation of TARP will allow FAA to automatically capture losses of separation in the tower and TRACON environments, it will also likely increase the number of reported losses of separation. According to officials, during its limited testing at facilities, TARP has already captured errors that were not being reported by air traffic controllers. For surface incidents, the ASDE-X system alerts controllers when aircraft or vehicles are at risk of colliding on runways, resulting in the identification of incidents that controllers might otherwise not be aware of. Designed as a surface surveillance system, ASDE-X helps to prevent collisions by raising alarms when aircraft appear to be at risk of colliding. As these alerts draw attention to near misses or potential collisions, they also serve to notify personnel of possible incursions and thus may have contributed to an increase in reported events, even as they may have prevented accidents. We analyzed the number of reported incursions at airports with ASDE-X and found that, at many of these airports, the number of reported incursions actually increased after their ASDE-X systems became operational. (For more information about our analysis of how the number of runway incursions changed after the installation of ASDE-X, see app. III.) Officials with the Sensis Corporation, the developer of ASDE-X, acknowledged to us that this may be a side effect of the deployment of the system.

Enhanced Oversight and Additional Information about Incidents Could Help Improve Safety in the Terminal Area

FAA has taken steps to improve safety in the terminal area since 2007 and has both reduced the number of serious incursions and undertaken successful efforts to increase reporting of incidents, but we identified two areas in which FAA could further improve management of data and technology in order to take a more proactive, systemic approach to improving terminal area safety. These areas include: (1) enhanced oversight of terminal area safety, including the management of runway excursions and ramp areas, and (2) assurance that data for risk assessments are complete, meaningful, and available to decision makers.

Federal Oversight of Terminal Areas Could Be Enhanced

Runway Safety Oversight Remains Limited

Stakeholders we spoke with generally lauded Runway Safety’s efforts on incursions, but FAA could do more to expand to other aspects of runway safety—notably runway excursions—as well as playing a more active oversight role in ramp areas. As we noted earlier, FAA is rolling out a new program to gather and analyze data on excursions, which should allow the agency to better understand why excursions happen and develop programs to mitigate risk. FAA is exercising some additional authority over ramps by proposing rules to address airport safety management systems and training for personnel accessing ramp areas, but these efforts are limited and involve requiring airports to develop and implement their own safety guidelines. In 2007, we reported that the lack of standards for ramp operations hindered safety, and an upcoming report by the Airport Cooperative Research Program (ACRP) continues to find that no comprehensive standards exist with regard to ramp area markings, ground operations, or safety training. The two proposed rules by FAA on airport safety management systems and training establish some standards for the ramp area, but proposed federal oversight would still be limited. The proposed rule implementing safety management systems for airports would require airports to develop plans to identify and address hazards in the ramp area and on the airfield, in addition to ensuring that all employees with access to runways, taxiways, and ramps receive training on operational safety and on the airport’s safety management system. Other aspects of ground handling, such as surface marking and ground operations, would continue to largely be overseen by airlines and the ground handling companies that are contracted by them.

The placement of the Runway Safety within the ATO Safety Office may limit its ability to serve as an effective focal point for runway and terminal area safety, given that aspects of runway and terminal area safety fall under the purview of several parts of FAA, including ATO, the Office of Airports, and the Office of Aviation Safety. In 2010, the IG recommended that the placement of Runway Safety within ATO be reconsidered, because the office may be limited in its ability to carry out cross-agency risk management efforts. Subsequently, the IG determined that Runway Safety had demonstrated effectiveness within ATO, but pointed to a need to periodically review organizational structures and processes to ensure that it continues to be effective. Runway Safety oversees data, assessments, and performance measures across a number of safety areas—air traffic control, pilot actions and training, outreach to general aviation, airport infrastructure, and technologies, among others—which are under the purview of different offices within FAA. As a result, Runway Safety has the potential to serve as the focal point for risk management in the terminal area.

FAA Data for Risk Assessment May Not Be Complete, Meaningful, or Available to Decision Makers

Impact of Changes in Reporting Policies and Processes on Measures of Incidents and Risk Is Unclear

Multiple changes to reporting policies and processes in recent years make it difficult to know the extent to which the recent increases in some terminal area incidents are due to more accurate reporting or an increase in the occurrence of safety incidents or both. For example, FAA officials have specifically attributed the increase in airborne operational errors to changes in reporting practices following the implementation of ATSAP, but, as previously mentioned, the relationship between the implementation of ATSAP and an increase in errors is uncertain. Likewise, other changes to performance measures and internal reporting policies, such as removal of individually identifying information from ATQA, further obscure the source of recent trends. Without accurate and consistent measures of safety outcomes, FAA cannot assess the risks posed to aircraft or passengers over time or the impacts of its efforts to improve safety.

As we noted in a 2010 report, FAA has embarked on a data-driven, risk- based safety oversight approach. As part of this effort, FAA has established a new, risk-based measure to track losses of separation, but measures for runway incursions are not risk based, reflecting instead a simple count of incidents. Thus, FAA currently rates the severity of incursions based on proximity and the response time to avoid a collision and does not differentiate between types of aircraft—or the number of lives put at risk—as part of its severity calculation. While any loss of life is catastrophic, the impact of an accident involving a commercial aircraft carrying hundreds of passengers can have different implications than an accident involving smaller aircraft. According to industry stakeholders, the use of proximity as the main criterion for severity of incursions is overly simplistic. As a result, FAA may be unable to use incursion data to identify the most serious systemic safety issues. Similarly, the application of risk assessment to measures of runway safety could allow FAA to focus individually on the risk posed by incursions by large commercial aircraft, as well as the risk posed by an ever-increasing incursion rate among general aviation operations. Additional attention to the root causes of incidents involving general aviation could potentially identify strategies addressing this ongoing challenge, which may include the installation of low-cost ground surveillance systems. While FAA officials did not detail immediate plans to alter the measure for incursions, officials did state that the agency plans to introduce surface incidents into RAP at the beginning of fiscal year 2012. The joint FAA-aviation industry Runway Safety Council is a first step toward the effort to reduce incursion risks. By identifying causes and making recommendations that could help determine what changes would be needed to make measures more risk- based, the Council’s Root Cause Analysis Team can help reduce incursion risks. Further, according to FAA officials, the new measure being developed for excursions will be risk-based; however, this measure will not be fully in use until 2014.

By contrast, FAA has taken steps to improve its ability to assess the risk of airborne operational errors and to collect more information. However, under the new risk assessment system, far fewer incidents are subject to analysis than were included in previous, nonrisk-based iterations, and the measure may therefore not account for all potential risk. For instance, RAP does not yet have procedures to assess losses of separation with terrain and with airspace boundaries. Currently, LoSS events in which at least 66 percent of minimum separation is maintained between aircraft are not assessed through FAA’s recently launched RAP. According to FAA officials, the 66 percent threshold for inclusion in RAP was adopted in recognition of the resources required for the enhanced risk-analysis process. This initial threshold is not based on specific risk-based criteria. Furthermore, losses of separation eligible for inclusion in RAP are currently limited to those that occur between two or more radar-tracked aircraft. As a result, many incidents—such as those that occur between aircraft and terrain or aircraft and protected airspace—are currently excluded from FAA’s process for assessing systemic risk. According to FAA officials, this exclusion is in part because there is no system in place through which the current RAP proximity inclusion threshold could be applied to these types of incidents, although FAA officials stated that an effort is under way to expand RAP to include these areas. In addition, FAA’s new measure for assessing air traffic risk levels—the SRER—does not include many losses of separation that were tracked under the old measure, although it does expand the assessment process to include some errors caused by pilots. Further, while the technology has been developed to collect data automatically for potential operational errors involving losses of separation, FAA has delayed the full implementation of TARP in air traffic control towers and TRACONS. According to FAA officials, the implementation of TARP may create workload challenges for quality assurance staff, as the technology is likely to capture hundreds of potential losses of separation that were not previously being reported through existing channels. In 2009, the IG recommended that FAA establish firm timelines for the full implementation of TARP.

Data collected through ATSAP, the nonpunitive reporting system, have limitations. There is the potential for serious events to be reported only to ATSAP and to therefore not be included in the official ATQA database or in RAP. Such events are referred to as “unknown” events. In 2010, 65 percent of incidents reported to ATSAP were unknown to FAA. (See fig. 17.) FAA officials acknowledged that there are a large proportion of unknown incidents but stated that these incidents are likely to be minor. In addition, some information about incidents reported to ATSAP is available for analysis by FAA and the aviation industry via the Aviation Safety Information Analysis and Sharing (ASIAS) program, as de-identified ATSAP data are shared with ASIAS. Further, they noted, ATSAP reports may be procedural, rather than reports of incidents or operational errors. Available data from the ATSAP program office indicates, however, that some of the “unknown” reports in the system were potentially serious events. For example, between the program’s launch in July 2008 and June 2011, 74 out of 253 ATSAP reports that were classified as potentially hazardous did not appear in ATQA, accounting for 29 percent of these reports. In June 2011—the most recent month for which data are available—approximately one third of all ATSAP reports classified as potentially major events, and 42 percent of those classified as hazardous, did not appear in ATQA. According to officials, ATSAP allows controllers to report incidents that may have otherwise gone unreported, and the program facilitates early detection and improved awareness of operational deficiencies and adverse trends. These unknown events, FAA officials point out, would not likely have been reported into ATQA before the implementation of ATSAP.

Information sharing challenges may impact the ability of FAA to analyze safety data and understand safety trends. Multiple FAA programs and data systems assign contributing factors to incidents, but factors are not coordinated across programs. For instance, both ATSAP and RAP have developed sets of factors that are identified as contributing to incidents during the incident investigation process. However, despite the fact that these two programs look at some of the same type of incidents (airborne losses of separation), program officials have not coordinated their development of the categories used to describe incidents. As a result, officials we interviewed stated that it is difficult to compare data across systems. For example, both ATSAP and RAP issued internal reports identifying top factors contributing to reported incidents, but there is no apparent overlap between the two lists. In addition, while the ATQA database contains more than 50 contributing factors for operational errors, FAA and the ATSAP program office do not use these data to identify systemic safety issues (see table 3). According to FAA officials, FAA is currently developing a common set of contributing factors for ATSAP and RAP, as well as a translation capability that will allow for the inclusion of historical data on contributing factors in future analyses. The IG raised concerns about the quality of ATQA data on contributing factors in a 2009 report, noting that FAA does not consistently include fatigue issues in contributing factor data it collects on operational errors. FAA has added contributing factors related to fatigue to ATSAP and is exploring ways to gather objective shift, schedule, and related resource management data to support enhanced fatigue analysis.

Regional and local access to and awareness of data related to both individual incidents and incident trends may be limited. According to FAA officials we interviewed at the regional level, it is difficult for supervisors at the regional and facility levels to obtain information on incident trends specific to their area of supervision in part because key databases, such as ATQA, do not have the capability to allow regional supervisors to run region- or facility-specific data queries. In addition, while multiple data resources may be available, officials stated that information on incidents is scattered, and no central source exists where employees can identify available data resources. While FAA has made advances in the quantity and comprehensiveness of the data it collects on incidents in the terminal area, officials stated that the agency faces difficulty in developing sophisticated databases with which to perform queries and modeling of the data. According to FAA officials, the full implementation of CEDAR will address many of the deficiencies identified by regional and local offices.

The nature and scope of ramp accidents are still unknown, just as they were when we reported in 2007, and we were told by officials with Airports Council International that it can be difficult to for airports to get data on incidents in the ramp area—areas typically overseen by airlines. This will pose a challenge as airports move to implement safety management systems and seek to identify and mitigate hazards. As one aviation expert explained, even if data are available locally—which they may not be—the number of incidents at individual airports can be too few to allow for the identification of root causes or the proactive identification of risk.

Conclusions

The Office of Runway Safety focuses on improving safety by reducing the number and severity of runway incursions. However, risk management in terminal areas involves more than just incursions—notably runway excursions and incidents in ramp areas. Runway Safety plans to start tracking runway excursions in October 2011, but it will take several years to develop processes for identifying and tracking incidents, identifying and mitigating risks, and measuring outcomes. Likewise, FAA does not track incidents in ramp areas, although we previously recommended that FAA work with the aviation industry and OSHA to develop a mechanism to collect and analyze data on ramp accidents. Airports implementing plans for safety management systems under FAA’s proposed rule will need data that are useful, complete, and meaningful in order to accurately assess risk and plan for safety, but FAA cannot yet provide meaningful data for the assessment and management of risks posed by runway excursions or ramp areas. Without information on incidents in these areas, FAA and its safety partners are hampered in their ability to identify risk, develop mitigation strategies, and track outcomes.

FAA addresses runway incursions as a specific type of incident and does not distinguish between commercial and general aviation in its performance measures. However, risks posed by runway safety incidents to passengers and aircraft in the national airspace system are different for commercial aircraft and general aviation. FAA performance measures for runway incursions—including the number, rate, and severity—do not reflect differences between commercial and general aviation and are not risk-based. The agency has installed risk-reduction technologies at larger commercial service airports, for example, but in the absence of risk-based performance measures, it lacks the ability to prioritize projects or measure effectiveness. With regard to general aviation, this traffic currently accounts for about a third of total tower operations, but 60 percent of runway incursions involve these aircraft. While Runway Safety has acknowledged that general aviation has caused more runway incursions, without performance measures that reflect risk, FAA may not be able identify appropriate mitigation strategies to address the large—and growing—proportion of runway incidents—including both incursions and excursions—involving general aviation aircraft. Strategies to decrease the risk posed by safety incidents involving general aviation could include additional outreach to these pilots, increased remediation following pilot errors, or the installation of technologies such as low-cost ground surveillance at airports serving general aviation traffic.

Safety in the terminal area is a shared responsibility among FAA, airlines, pilots, and airports, and there are a number of FAA offices that either collect or analyze terminal area incident data, but useful access to complete and meaningful data is limited. The agency currently does not have comprehensive risk-based data, sophisticated databases to perform queries and model data, methods of reporting that capture all incidents, or a level of coordination that would facilitate the comparison of incidents across systems. Technologies aimed at improving reporting have not been fully implemented. As a result, aviation officials managing risk using safety management systems, including local and regional decision makers, have limited—if any—access to FAA incident data. For example, FAA’s official database for air traffic safety does not allow local or regional FAA safety officials to run region- or facility-specific data queries. Further, under the new risk assessment process used for losses of separation, fewer incidents are assessed and accounted for in performance measures—such as losses of separation between aircraft and terrain or aircraft and protected airspace—which may distort risk assessment processes. Finally, according to FAA officials, one reason the agency has not fully implemented TARP is that implementation of TARP may create workload challenges for FAA quality assurance staff, as the technology is likely to capture hundreds of potential losses of separation that were not previously being reported through existing channels. FAA offices and others using a safety management system approach to manage risk should have access to complete and meaningful data to allow for hazard identification and risk management. The ability of FAA and airport officials—and the local Runway Safety Action Teams that they serve on— to identify safety risks, develop mitigation strategies, and measure outcomes is hindered by limited access to complete and meaningful data.

Recommendations for Executive Action

To enhance oversight of terminal area safety to include the range of incidents that pose risks to aircraft and passengers, we recommend that the Secretary of Transportation direct the FAA Administrator to take the following three actions:  develop and implement plans to track and assess runway excursions and extend oversight to ramp safety;  develop separate risk-based assessment processes, measures, and performance goals for runway safety incidents (including both incursions and excursions) involving commercial aircraft and general aviation and expand the existing risk-based process for assessing airborne losses of separation to include incidents beyond those that occur between two or more radar-tracked aircraft; and  develop plans to ensure that information about terminal area safety incidents, causes, and risk assessment is meaningful, complete, and available to appropriate decision makers.

Agency Comments

We provided the Departments of Transportation and Labor, NTSB, and the National Aeronautics and Space Administration (NASA) with a draft of this report for review and comment. The Department of Transportation agreed to consider our recommendations and provided clarifying information about efforts made to improve runway safety, which we incorporated. The Department of Labor, NTSB, and NASA provided technical corrections, which we also incorporated.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 7 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretaries of Transportation and Labor, NTSB, the Administrator of NASA, and interested parties. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me on (202) 512-2834 or at dillinghamg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Gerald L. Dillingham, P Director, Physical Infrastructure Issues h.D.

Appendix I: Objectives, Scope, and Methodology

Our objective was to review aviation safety and update our 2007 report on runway and ramp safety. To do so, we addressed the following questions: (1) What actions has the Federal Aviation Administration (FAA) taken to improve safety in the terminal area since 2007? (2) What are the trends in terminal area safety and the factors contributing to these trends? and (3) What additional actions could FAA take to improve terminal area safety?

To identify actions FAA has taken since 2007 to improve safety in the terminal area and to identify additional actions FAA could take to improve safety, we reviewed our prior reports, as well as documents and reports from FAA, the Department of Transportation Inspector General (IG), the National Transportation Safety Board (NTSB), the International Civil Aviation Organization (ICAO), and others; FAA orders, advisory circulars, and regulations; and applicable laws. We also determined the roles and responsibilities of FAA, NTSB, the Occupational Safety and Health Administration (OSHA), the National Aeronautics and Space Administration (NASA), airports, and airlines involving runway, terminal airborne, and ramp safety. In addition to interviewing officials from FAA, IG, and NTSB, we interviewed aviation experts affiliated with the Air Line Pilots Association, Airports Council International, Air Transportation Association, the Flight Safety Foundation, and the National Air Traffic Controllers Association about terminal area safety practices and technologies. We also interviewed researchers from the Air Cooperative Research Program (ACRP) of the Transportation Research Board and experts affiliated with various aviation technology companies. To obtain information about air traffic control operations, observe the application of key technologies, and interview facility managers, we visited four FAA facilities that were near our GAO offices: the Potomac Consolidated Terminal Radar Approach Control facility, the Washington Air Route Traffic Control Center, and the air traffic control towers at Ronald Reagan Washington National Airport and the Seattle-Tacoma International Airport. We also interviewed airport officials with the Port of Seattle at Seattle- Tacoma International Airport. To obtain information about the Air Traffic Safety Action Program (ATSAP), we interviewed officials with the ATSAP program office and attended an Event Review Committee meeting in Renton, Washington. We also reviewed FAA’s progress in addressing recommendations that we, IG, and NTSB have made in previous years and reviewed the processes that FAA uses to collect and assess runway and air traffic safety data.

To identify and describe recent trends in terminal area safety and the factors contributing to these trends, we obtained and analyzed data from FAA, NTSB, and OSHA on safety incidents in the terminal area. We analyzed FAA runway incursion data collected from fiscal year 2001 through the second quarter of fiscal year 2011, as well as FAA data on airborne operational errors from the Air Traffic Quality Assurance database (ATQA) from the third quarter of fiscal year 2007 through the second quarter of fiscal year 2011. We limited our analysis to airborne operational errors in order to avoid double counting of surface operationa errors that are included in our counts of runway incursions. We used Operations Network data from FAA to determine rates of incursions and airborne operational errors per million operations. Rates of incursions were calculated per million tower operations, and rates of airborne operational errors were calculated per million operations perfo traffic control towers, terminal radar approach control (TRACON) facilitie s, and en route facilities on a quarterly basis. We also reviewed NTSB data involving runway incursions and excursions from 2008 through June 2011 and summarized OSHA data on fatalities in the ramp area from 2001 through 2010. We used statistical models to assess the association between safety incidents and the concentration of general aviatio operations and the implementation of the Airport Surface Detection Equipment, Model X (ASDE-X) surface surveillance system from fiscal year 2001 through April 2011. These models estimated how the number of incursions changed after airports installed ASDE-X or increased the proportion of operations involving general aviation. The models accoun for other factors that may contribute to incursions, such as long-term weather patterns, runway layouts, as well as controller and pilot experience. See appendix III for more information about the methods and results of these analyses. To assess the reliability of FAA data, we (1) reviewed internal FAA documents about its collection, entry, and maintenance of the data and (2) interviewed FAA officials who were knowledgeable about the content and limitations of these data. Both NTSB and OSHA provided information about the reliability of their excursion and fatality data, respectively. We determined that these data t were sufficiently reliable for the descriptive and comparative analyses used in this report.

We conducted this performance audit from February 2011 to October 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the aud it to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Technologies to Improve Runway Safety

Researching, testing, and deploying new technology is a major part of FAA’s risk-reduction strategy. A number of available technological systems are intended to help reduce the number and severity of runway incursions. For example, to give air traffic controllers better visibility of activity on the airfield and help prevent collisions, FAA has installed the ASDE-X system at 35 major airports, while the Airport Surface Detection Equipment, Model 3 (ASDE-3) radar and the Airport Movement Area Safety System (AMASS) provide surface surveillance at 9 additional airports. Runway status lights, which will be installed at 23 airports, are a fully automatic series of lights that give pilots a visible warning when runways are not clear to enter, cross, or depart on. To mitigate the risks posed by runway excursions, FAA conducted research that led to the development of the Engineered Materials Arresting System (EMAS), a bed of crushable concrete designed to stop aircraft from overrunning runway areas. As of July 2011, EMAS has been installed at 52 runways at 36 airports, and there are plans to install 11 EMAS systems at 7 others. According to FAA officials, EMAS has successfully arrested seven overrunning aircraft with no fatalities or serious injuries and little damage to the aircraft to date. (See table 4 for a brief description of technologies designed to improve runway safety.)

Appendix III: Technical Appendix

This appendix summarizes our statistical analysis of the between the ASDE-X airfield surface surveillance system and runway ine focused on ASDE-X, among many other runway safety cursions. W programs, because of its potential for persuasive impact evaluation. We describe how the process FAA used to install ASDE-X created a “quasi- experiment,” which allows u as to compare how incursions changed at sirports that received the technology relative to airports that kept the cursions. W rograms, because of its potential for persuasive impact evaluation. We escribe how the process FAA used to install ASDE-X created a “quasi- xperiment,” which allows u s to compare how incursions changed at irports that received the technology relative to airports that kept the tatus quo.

Evalu ASDE-X on Runway Incursions

A a s lo c m p th key goal of ASDE-X is to make air traffic controllers more aware of ctivities on taxiways and runways in order to avoid collisions. The ystem consists of airfield radar and sensors that collect data on the cation of aircraft and vehicles. Computers transform these data in ontinuously updated maps of the airfield, which are displayed on color onitors in air traffic control towers. The system warns controllers of otential collisions—which may draw attention to possible incursions— rough visual and audible alarms.

AA used a selective and staggered process to install ASDE-X at 34 irports from 2003 through 2011 (of 35 airports slated to receive this ystem). The variation among airports receiving the technology, as well F a s as the times when they received i evaluation. This type of analysis compares the change in incursions over time at airports that installed ASDE- nX with the change at airports that did t, allows for a quasi-experimental valuation. This type of analysis compares the change in incursions over me at airports that installed ASDE- X with the change at airports that did ot receive the technology, also known as a “difference-in-difference.” alysis of site-specific safety and efficiency benefits as compared to site-specific costs. airports provide a plausible comparison group for analysis, but we use variety of other comparison groups to ensure that our findings are robu st.

Many factors that may contribute to incursions are controlled in our analysis here. We control for variation among airports in runway and taxiway layouts, markings, and lighting, in addition to long-term variati in weather, air traffic, and pilot and controller skills. The staggered installation of ASDE-X makes bias due to short-term weather conditions or pilot and controller experience unlikely, because these factors would need to be correlated with 34 installation times throughout the country. In addition, the staggered installation lets us control for factors that affect all airports equally, such as changes in training and procedures made throughout the country at the same time.

Data Sources

The time period of our analysis spans fiscal years 2001 through April 2011. We assembled data on the number of incursions that occurred pe month at each FAA-towered airport in this period, along with data on air traffic control tower operations. The latter data included the number of monthly tower operations at each airport, as well as the mixture of commercial and general aviation operations. The operations data ide the population of interest, including the many smaller airports with no incursions that do not appear in the incursion data.

FAA provided the installation dates and locations for the ASDE-X, runway status lights, FAROS, and low-cost ground surveillance systems. We used these data to identify whether each technology was installed for each airport and month between fiscal year 2001 and April 2011.

Statistical Modeling

We used a statistical model to estimate the association between ASDE-X and the number of incursions for airport i and month t. The model took the form of E(Yit | a,p,tit,xit) = a exp(δp + atit + xitβ), where Yit randomly varies according to the Poisson distribution, a is vector of airport fixed effects, p is a vector of year-month fixed effects, tit indicates whether ASDE-X was operational at airport i in month t, xit are other time-varying covariates, and δ, α, and β are vectors of parameters. We estimated the change in incursions after the installation of ASDE-X using one contemporaneous, before-and-after parameter, α, becaus ASDE-X likely has an immediate effect on incursions once it has been installed that does not change over time. ,4 The covariates xit included the number of air traffic control tower operations (to measure exposure and variation in the nature of activity across airports), indicators for having 25 to 60 percent and greater than 60 percent of operations involving general aviation (excluding 0 to 25 percent), and indicators for having the runway status lights, FAROS, or low-cost ground surveillance systems installed at airport i and time t.

T airport. The 485 airports in the panel ensure that these estimates will be accurate approximations, even if incursions are not Poisson-distributed. As Wooldridge 2003 (674- 675) notes, conditional ML estimators consistently estimate the parameters of a fixed effects model, even with arbitrary forms of over- and under-dispersion, heteroskedasticity, and serial correlation. As a result, we can safely use the Poisson conditional likelihood to estimate the parameters while using cluster-robust standard errors.

The models used several groups to compare the change in incursions before and after ASDE-X was installed, in order to assess the sensitivity of our results to plausible alternatives. The groups included 1. all FAA-towered airports; 2. airports that were included in the FAA benefit-cost analysis above b did not receive ASDE-X; 3. the top 100 airports in tower operations from fiscal year 2001 through 4. airports that had a similar ground surveillance system, ASDE- 3/AMASS, installed prior to the first installation of ASDE-X (baseline 5. airports that did not have ASDE-3/AMASS installed at baseline; and 6. airports that did not have ASDE-3/AMASS installed at baseline and that were among the top 100 airports in tower operations from fiscal year 2001 through April 2011.

Runway Incursions Rise after the Installation of ASDE-X, but Better Reporting May Explain the Change

Figure 18 plots the average monthly incursion rate for airports that did and did not receive ASDE-X, rescaled to a ratio of the over-time mean to better express the trends. The smooth lines summarize the average incursion rate for each group and month using nonparametric locally weighted regression models. The vertical lines show the ASDE-X installation times for each airport.

Prior to the first installation of ASDE-X, the incursion rate changed in roughly the same ways for the ASDE-X and comparison airports. As FAA began to install the system in late 2003, and the incursion rate began to increase for the ASDE-X airports, but it decreased and then increased at a slower rate for the comparison airports. Substituting the other comparison groups in these plots produces similar patterns. Consequently, the raw data suggest that reported incursions increased at airports that received ASDE-X, as compared to the change at airports t did not receive the system.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Heather MacLeod, Assistant Director; Russ Burnett; Martha Chow; Dave Hooper; Delwen Jones; Molly Laster; Brooke Leary; Josh Ormond; and Jeff Tessin made key contributions to this report.