GAO-03-1175R Operational Errors Data 

United States General Accounting Office 

Washington, DC 20548 

September 23, 2003 

The Honorable James L. Oberstar 

Ranking Democratic Member 

Committee on Transportation and Infrastructure 

U.S. House of Representatives 

Subject: Aviation Safety: Information on FAA’s Data on Operational Errors at Air 

Traffic Control Towers 

A fundamental principle of aviation safety is the need to maintain adequate separation 

between aircraft and to ensure that aircraft maintain a safe distance from terrain, 

obstructions, and airspace that is not designated for routine air travel. Air traffic controllers 

employ separation rules and procedures that define safe separation in the air and on the 

ground.1

 An operational error occurs when the separation rules and procedures are not 

followed due to equipment or human error. Data maintained by the Federal Aviation 

Administration (FAA) indicate that a very small number of operational errors occur in any 

given year—on average about three operational errors per day occurred in fiscal year 2002. 

However, some of these occurrences can pose safety risks by directing aircraft onto 

converging courses and, potentially, midair collisions. 

You asked us to provide information on FAA’s data on operational errors and whether this 

data can be used to identify types of air traffic control facilities with greater safety risks. 

Specifically, you asked us to (1) determine what is known about the reliability and validity2

 of 

the data that FAA maintains on operational errors and (2) identify whether comparisons of 

operational errors among air traffic control facilities can be used to determine the facilities’ 

relative safety record. 

 

1

 The Federal Aviation Administration (FAA) has established a separation standard in the en route 

environment of 5 nautical miles horizontally and either 1,000 or 2,000 feet vertically depending on 

altitude. In the terminal environment, horizontal separation is generally between 3 and 5 nautical miles 

depending on the type of aircraft. 

2

 Data reliability refers to the accuracy and completeness of data. We define data as reliable when they 

are (1) complete and (2) accurate. Reliability does not mean that data are error free, but that the data 

is sufficient for the intended purposes. Validity refers to whether the data actually represent what one 

thinks is being measured. See U.S. General Accounting Office, Assessing the Reliability of ComputerProcessed Data, GAO-02-15G (Washington, D.C.: Sept. 2002). 
2 GAO-03-1175R Operational Errors Data

To answer these objectives, we reviewed past GAO studies3 and reports by the Department of 

Transportation (DOT) and DOT’s Inspector General (IG) that pertain to FAA’s data on 

operational errors and applied standard methodological practices for data reliability, validity, 

and analysis.4

 

Data Has Reliability and Validity Limitations 

We identified several potential limitations with FAA’s data on operational errors based on our 

review of issued GAO and DOT reports and application of best methodological practices. 

First, it is very difficult to determine the completeness of the data. FAA collects data on 

operational errors from two sources—self-reporting by air traffic controllers and automatic 

reports of errors detected on the en route portion of a flight. The possibility exists for 

underreporting by air traffic controllers, since some errors are self-reported and some air 

traffic controllers may not self-report every incident. Second, due to the way the data are 

recorded, the severity of many errors cannot be determined or is misleading. Prior to 2001, 

minor errors, such as establishing a 4.5-mile rather than a 5-mile separation, were counted in 

the same way as more serious errors, according to DOT.5

 In 2001, DOT began to address this 

issue by establishing a rating system to identify the severity of, or collision hazard posed by, 

operational errors. The system uses a 100-point scale to rate and categorize operational 

errors as high, moderate, or low severity. However, in 2003, DOT’s IG reported continuing 

concerns with FAA’s data on operational errors.6

 The IG noted that the new rating system 

provides misleading information and that FAA needs to modify the system to more accurately 

identify the most serious operational errors. The DOT IG found that in one instance FAA 

rated an operational error as moderate that was less than 12 seconds from becoming a midair 

collision. The IG believed that this operational error should have been rated as high severity. 

The IG also reported that FAA cannot be sure that air traffic controllers report all operational 

errors. 

Comparison of Operational Errors Alone Does Not Provide Valid Conclusions About 

Safety of Air Traffic Control Facilities 

Comparisons of operational errors among types of air traffic control facilities, such as FAAstaffed facilities versus contractor-staffed facilities, cannot be used alone to provide valid 

conclusions about safety due to three factors that we identified based on standard 

methodological practices and our understanding of FAA’s data. First, such problems as the 

completeness and specificity of data on operational errors are likely to affect the validity of 

comparisons among air traffic control facilities because operational errors may not be 

comparably reported at the types of facilities being compared. For example, as we 

mentioned above, FAA cannot be sure that all operational errors at either FAA-staffed or 

 

3

See, for example, U.S. General Accounting Office, Air Traffic Control: FAA Enhanced the Controllerin-Charge Program, but More Comprehensive Evaluation Is Needed, GAO-02-55 (Washington, D.C.: 

Oct. 31, 2001). 

4

See GAO-02-15G; U.S. General Accounting Office, Government Auditing Standards, GAO-03-673G

(Washington, D.C.: June 2003); and GAO Policy and Procedures Manual, Factors Affecting a Design’s 

Credibility.

5

 U.S. Department of Transportation, Performance Report Fiscal Year 2000, Performance Plan, Fiscal 

Year 2002 (Washington, D.C.: April 2001). 6

 U.S. Department of Transportation, Office of Inspector General, Top Management Challenges, 

Department of Transportation, PT-2003-012 (Washington, D.C.: Jan. 21, 2003) and Safety, Cost, and 

Operational Metrics of the Federal Aviation Administration’s Visual Flight Rule Towers, AV-2003-

057 (Washington, D.C.: Sept. 4, 2003). 
3 GAO-03-1175R Operational Errors Data 

contractor-staffed towers were reported. When such a situation exists, it is difficult, if not 

impossible, to determine whether the comparative results are valid or are an artifact of 

under-reporting at one or both types of air traffic control facilities. Second, in order to make 

valid comparisons a number of factors that might affect the rate of operational errors would 

need to be accounted for in an analysis. For example, air traffic density, other operating 

conditions such as the number of flights, age and experience of air traffic controllers, and 

weather conditions at the time the error occurred all might influence operational errors. 

These factors would have to be accounted for in any analysis comparing operational errors 

among different types of facilities in order to determine if the errors are associated with 

something other than the type of air traffic control facility. Finally, as previously mentioned, 

a very small number of operational errors occur in any given year (6.7 operational errors per 

million operations, on average, across all FAA towers in fiscal year 2002), which may make it 

difficult to detect any real differences in the error rates among facilities. 

Because of these factors, the determination of real differences in the rate of operational 

errors between different types of air traffic control facilities is difficult, and comparisons of 

operational error rates alone are not sufficient to draw conclusions about the relative safety 

records of air traffic control facilities. At a minimum, the additional factors mentioned above 

would need to be considered and analyzed with a technique that models the occurrence of 

rare events and looks at these events over time. This approach, however, is not without risk 

and would depend upon the existence of proper and reliable data on operational error rates, 

operating conditions at the towers at the time the error occurred, and other factors that may 

be associated with operational errors. Such an approach would allow for a more meaningful 

comparison of facilities’ operational errors through ascertaining and accounting for the 

multiple factors that may be associated with such errors. 

As arranged with your office, unless you publicly announce its contents earlier, we plan no 

further distribution of this report until seven days after the date of this report. At that time, 

we will send copies of this report to interested congressional committees. The report will 

also be available on GAO’s home page at http://www.gao.gov. If you have any questions 

about this report, please contact me at (202) 512-2834 or by e-mail at dillinghamg@gao.gov. 

Key contributors to this assignment are Isidro Gomez, Brandon Haller, Teresa Spisak, and 

Alwynne Wilbur. 

Sincerely yours, 

Gerald L. Dillingham 

Director, Civil Aviation Issues 

(540075) 
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability of the

federal government for the American people. GAO examines the use of public 

funds; evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good 

government is reflected in its core values of accountability, integrity, and 

reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web 

site daily. The list contains links to the full-text document files. To have GAO email this list to you every afternoon, go to www.gao.gov and select “Subscribe to 

e-mail alerts” under the “Order GAO Products” heading. 

The first copy of each printed report is free. Additional copies are $2 each. A 

check or money order should be made out to the Superintendent of Documents. 

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to

a single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 

441 G Street NW, Room LM

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

Obtaining Copies of 

GAO Reports and

Testimony 

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Public Affairs