FAA’S PROCESS FOR REPORTING AND 

INVESTIGATING OPERATIONAL ERRORS 

Federal Aviation Administration 

Report Number: AV-2009-045 

Date Issued: March 24, 2009 
U.S. Department of 

 Memorandum

Transportation 

Office of the Secretary 

of Transportation 

Office of Inspector General 

Subject: ACTION: FAA’s Process for Reporting and 

Investigating Operational Errors 

Federal Aviation Administration 

Report Number AV-2009-045 

Date: March 24, 2009 

From: David A. Dobbs 

Deputy Inspector General 

Reply to 

Attn. of: J-2 

To: Acting Federal Aviation Administrator 

This report presents the results of our review of the Federal Aviation Administration’s 

(FAA) process for reporting and investigating operational errors. Operational errors 

(when a controller fails to maintain separation between two aircraft) can be extremely 

serious incidents that can lead to a catastrophic accident. Ensuring that all events 

involving a loss of separation are accurately reported, investigated, and addressed is 

critical to the safe operation of the National Airspace System. 

In October 2007, Congressman James Oberstar, Chairman of the House Committee 

on Transportation and Infrastructure, and Congressman Jerry Costello, Chairman of 

the House Subcommittee on Aviation, requested that our office audit FAA’s process 

for identifying and reporting operational errors. This request was prompted by our 

then-ongoing investigation at the Dallas Fort-Worth (DFW) Terminal Radar 

Approach Control (TRACON) facility, which we conducted after whistleblowers 

alleged that facility management was intentionally misclassifying operational errors.1 

Our investigation at DFW had found that Air Traffic managers at the TRACON were, 

in fact, intentionally misclassifying operational errors as either pilot deviations or 

“non-events.”2

 We identified 62 operational errors and deviations that were either 

incorrectly reported as pilot deviations (39) or misclassified as “non-events” (23). 

Further, FAA’s Service Area and Headquarters safety oversight processes and 

controls failed to uncover this practice despite FAA’s prior assurances to our office in 

2005 that it would not allow operational errors to go unreported. 

 

1

 OIG Report Number CC-2007-083, Alleged Cover-Up of Operational Errors at DFW TRACON,” April 18, 2008. 

OIG reports are available on our website: www.oig.dot.gov. 2

 Non-events are those incidents that facility personnel reviewed but determined there was no loss of separation.

i
Specifically, Chairmen Oberstar and Costello requested that we determine whether 

the operational error reporting problems found at the DFW TRACON were occurring 

at other Air Traffic facilities. Accordingly, our audit objectives were to (1) determine 

whether FAA has adequate policies and procedures to ensure accuracy and 

consistency in operational error reporting and (2) review the roles and responsibilities 

of the Air Traffic Organization and FAA’s Aviation Safety line of business in 

reporting and investigating operational errors. 

We conducted this audit between November 2007 and December 2008 in compliance 

with generally accepted Government Auditing Standards as prescribed by the 

Comptroller General of the United States. Exhibit A details our review scope and 

methodology, and exhibit B lists the FAA offices and facilities we visited. 

BACKGROUND 

To ensure that safe distances are maintained between aircraft while under the control 

of air traffic controllers, FAA has minimum separation standards that must be 

maintained based on the aircraft’s phase of flight and size. During the en route phase 

of flight,3

 aircraft must be separated 5 miles laterally or 1,000 to 2,000 feet vertically 

(depending on altitude). During the departure and arrival phase of flight,4

 aircraft 

must be separated by 3 miles or more horizontally (depending on aircraft size) or 

1,000 feet vertically. 

To maintain these minimum separation standards, controllers are responsible for 

providing instructions to pilots. If a loss of separation occurs between two aircraft, in 

most cases it is classified as either an operational error (if the controller’s actions 

caused the loss), a pilot deviation (if the pilot’s actions caused the loss), or both. 

When a suspected loss of separation occurs, the Air Traffic facility where the incident 

occurred is responsible for initially investigating and classifying the incident and 

preparing a preliminary report. The Air Traffic facility then forwards the operational 

error report to the appropriate Air Traffic Organization (ATO) Service Area (Regional 

office) and Service Unit (Headquarters’ Terminal or En Route office) for review, after 

which it is forwarded to Headquarters ATO Safety Office for final review. Pilot 

deviations are forwarded to the responsible Aviation Safety Flight Standards office, 

which will conduct a full investigation and prepare a final report (see figure 1 below). 

 

3

 The en route phase includes aircraft flying at higher altitudes, generally above 17,000 feet under the control of en route 

facilities. 4

 This phase includes aircraft within the immediate area of an airport, generally within 0 to 40 miles of the airport under the 

control of tower and TRACON (i.e., terminal) facilities. 

ii
Figure 1. FAA’s Process for Reporting and Investigating Incidents Involving a 

Loss of Separation 

RESULTS IN BRIEF 

Controller 

Error 

Facility prepares preliminary 

investigation report and 

forwards report to 

appropriate ATO Service 

Area and Unit for review. 

Pilot 

Error 

Preliminary report 

forwarded to Flight 

Standards inspector who 

completes investigation and 

final report. 

After review by Service Area 

and Unit, report is forwarded 

to Headquarters ATO Safety 

office for review. 

Loss of separation identified 

by Air Traffic facility. 

Initial determination is made 

if the incident was controller 

or pilot error. 

Air Traffic facility completes 

investigation and final report

and forwards report to 

appropriate ATO Service 

Area and Unit for review.

Facility copies 

Service Area and 

Unit and HQ Safety 

on preliminary 

report. 

ATO Safety 

Office conducts 

final review. 

Events at DFW TRACON Were Not Systemic, but Significant 

Weaknesses Exist in FAA’s Processes for Reporting and Investigating 

Incidents Involving a Loss of Separation 

We found that the problems identified at the DFW TRACON were not systemic. To 

determine this, we randomly selected 13 Air Traffic facilities for review and 

statistically selected and reviewed 166 pilot deviations that had a loss of separation. 

We also judgmentally reviewed 206 other “non-events” at the 10 facilities we visited. 

We found only 3 misclassified incidents (out of the total 372 examined) that should 

have been reported as operational errors—significantly less than the 62 reported at 

DFW. Based on this sample, we statistically project that there were between 1 and 

7 pilot deviations systemwide (excluding DFW) in fiscal year (FY) 2007 that should 

have been classified as operational errors.5

However, we did identify control and oversight weaknesses in FAA’s process for 

reporting and investigating losses of separation caused by pilots and controllers that 

could allow similar problems to occur in the future. Contributing factors to these 

weaknesses were inadequate FAA guidance on how to investigate these events and 

insufficient staffing in the ATO Safety Office. We identified four areas where FAA 

should focus its actions to ensure all losses of separation are accurately reported and 

investigated. 

 

5

 Based on a 90-percent confidence level and a universe of 484 pilot deviations involving a loss of separation. 

iii
Additional actions and follow-up are needed from FAA’s Flight Standards 

Service to improve the procedures for reporting and investigating pilot 

deviations. We found that Flight Standards Service did not consistently investigate 

pilot deviations because of control weaknesses in FAA’s guidance. For example, we 

found inspectors did not always verify the accuracy of Air Traffic’s preliminary report 

and even failed to investigate incidents altogether when they were referred to the 

airlines’ Aviation Safety Action Program (ASAP).6

 In other incidents, it was unclear 

if inspectors performed a thorough investigation because they did not document their 

results. 

During our review, FAA issued new guidance to address these control weaknesses. 

For example, inspectors are now required to verify the accuracy of all data in 

preliminary reports, including those referred to ASAP, and must fully describe the 

pilot deviation, including causal factors. FAA trained its managers on the new 

requirements in August 2008 and completed its training for field inspectors in October 

2008. While those actions represent progress, FAA will need to follow up once the 

requirements are implemented to ensure inspectors are complying with the new 

procedures. 

FAA needs to consistently evaluate losses of separation caused by both 

controllers and pilots. Unlike its process for operational errors, FAA does not rate 

the proximity of (i.e., how close two aircraft came to one another) or have a goal for 

reducing the risk of pilot deviations that cause a loss of separation. This is despite the 

fact that pilot deviations can pose the same risk for a catastrophic accident as 

operational errors. FAA rates controller operational errors by proximity—from 

category A to C—and maintains goals for reducing category A and B incidents. 

(Category A is the most serious, i.e., the aircraft came very close to one another. 

Category C is the least serious, i.e., most of the required separation was maintained.) 

For example, on July 6, 2007, while under control of the Atlanta TRACON, two 

regional jets narrowly avoided a collision when a pilot flew in the wrong direction. 

The aircraft were at the same altitude and came within about 3,300 feet of each other 

horizontally. Had this been a controller operational error, it would have been 

classified as a serious event. However, because it was a pilot deviation, the severity 

was not measured and no analysis was conducted to determine if similar deviations 

were occurring so that corrective actions could be taken systemwide. 

We applied FAA’s proximity rating system for operational errors (using distance and 

altitude parameters for categories A through C) to estimate the severity of pilot 

deviations that occurred while under control of en route and TRACON facilities 

during FY 2007. 

 

6

 ASAP is a program in which air carrier employees can report potential safety issues without fear of enforcement action 

from FAA. Incident data obtained by the air carrier through the ASAP process is maintained by the air carrier and 

protected from disclosure. 

iv
Of the 478 pilot deviations, we estimate that 33 percent would have been rated as 

serious category A or B incidents. 

If FAA were to categorize the severity of losses of separation caused by pilots in a 

similar manner, it could focus on the most serious incidents, identify potentially 

systemic issues, and take corrective actions on those that require attention at the 

national level. For example, within our sample of 166 pilot deviations, we found: 

• 53 incidents occurred when the pilots acknowledged and read back the correct 

altitude assigned but then deviated to a different altitude; 

• 27 incidents occurred because the pilots either had equipment malfunctions or did 

not accurately program their aircraft navigational equipment; and 

• 19 incidents occurred because the pilots did not follow established departure or 

arrival procedures, including those established by newly implemented area 

navigation (RNAV) procedures. 

To effectively reduce the risk of mid-air collisions, FAA should establish a process to 

rate the severity of pilot deviations that cause a loss of separation and establish a 

corresponding goal to reduce the most severe incidents. 

FAA needs to implement the Traffic Analysis and Review Program (TARP) as a 

full-time separation conformance tool to ensure all losses of separation are 

accurately reported. Unlike en route centers, terminal facilities do not have a 

system to automatically report losses of separation. Instead, managers at terminal 

facilities rely on controllers to self-report those incidents. FAA is aware of this 

problem and is developing TARP—an automated tool that will allow FAA to identify 

when operational errors (or other losses of separation) occur at terminal facilities. 

FAA plans to implement TARP as an audit tool only at all TRACONs by the end of 

2009. According to FAA managers, additional technical and infrastructure changes 

will be needed before TARP can be used reliably on a full-time basis. However, FAA 

has not established milestones and deadlines for when the system will be fully 

operational. 

While using TARP as an audit tool should help FAA to determine if operational errors 

have been misclassified, it does not provide full assurance that losses of separation at 

TRACON facilities are being reported and investigated. For this to occur, FAA must 

implement the system as a full-time separation conformance tool. To effectively 

deploy TARP, FAA must establish milestones for implementation and operational use 

and hold managers accountable for meeting them. 

Additional FAA actions are needed to improve the ATO’s oversight of the 

reporting and investigation process. We found that the ATO’s Safety Office did 

v
not have adequate oversight processes and controls for reporting and investigating 

loss of separation events. For example, neither the ATO Headquarters nor Service 

Center Safety offices discovered or challenged the incidents that were misclassified 

by DFW TRACON (62 incidents). These weaknesses occurred, in part, because the 

ATO Safety Services did not have enough staff in its Investigation and Evaluations 

office. Over the last 10 years, the number of employees in this office has decreased 

from 79 to 24 (as of March 2008). FAA is aware of this concern and initiated a 

complete review of ATO Safety Services to determine what additional resources are 

needed to meet its oversight responsibilities. 

ATO Safety Services is also making procedural and organizational changes to 

improve its oversight of reporting and classification of losses of separation. For 

example, it plans to establish an independent quality assurance office in the three 

ATO Service Areas. This new office—not the facility manager—will determine 

whether a loss of separation was a pilot or controller error. These changes are 

planned for completion by the summer of 2009 and should improve how FAA 

oversees reports and investigations for all losses of separation. Once the changes are 

implemented, FAA should initiate an internal audit by an independent organization, 

such as its Aviation Safety Oversight Office (AOV), to ensure these actions have been 

implemented and are being complied with. 

SUMMARY OF RECOMMENDATIONS 

Our recommendations include establishing (1) a follow-up mechanism to ensure 

Flight Standards inspectors comply with new guidance for investigating pilot 

deviations, (2) a process to rate the severity of pilot deviations and a corresponding 

goal to reduce the most severe incidents, (3) milestones for fully implementing TARP, 

and (4) an internal audit of the planned changes to the ATO’s safety oversight. We 

are making a total of 7 recommendations, which are listed on page 12 of this report. 

SUMMARY OF AGENCY COMMENTS AND OFFICE OF INSPECTOR 

GENERAL RESPONSE 

We provided FAA with our draft report on December 30, 2008, and received its 

response on March 12, 2009. FAA generally agreed with our recommendations and 

provided acceptable corrective actions with two exceptions. 

First, FAA did not agree with our recommendation to establish a process to rate the 

severity of pilot deviations and a corresponding goal to reduce the most severe 

incidents. In its response, FAA stated that it “addresses the seriousness of the 

violation during the investigation and adjusts punishment according to the degree of 

the violation.” However, our recommendation was not aimed at rating the severity of 

pilot deviations to assess punitive actions against pilots on a case-by-case basis. 

vi
vii

Rather, we intended for FAA to assess the severity of incidents to advance its riskbased approach to safety oversight. This would allow FAA to gather macro-level 

data, which it could use to identify possible trends, potentially systemic issues, and 

corrective actions needed at the national level. 

FAA already uses a similar severity rating approach to identify systemic issues for 

virtually all other aviation incidents including operational errors, near midair 

collisions, and runway incursions. In fact, FAA currently rates the severity of pilot 

deviations that occur on the ground (i.e., runway incursions) but not pilot deviations 

that occur in the air. Given that aviation safety is FAA’s primary mission, we are 

concerned that the Agency would bypass an opportunity to advance its risk-based 

approach at the national level. Accordingly, we believe FAA needs to reconsider its 

position. 

Second, FAA did not agree with our recommendation to assign a regional Flight 

Standards liaison (which could be a collateral position) to assist the ATO Safety 

Services staff in determining whether losses of separation are pilot or controller 

errors. In its response, FAA stated that “ATO-S will continue to work with [aviation 

safety inspectors] to resolve reported violations of air traffic rules and procedures, 

eliminating the need to assign a regional Flight Standards Liaison.” 

However, the intent of our recommendation was to allow Flight Standards to have 

more timely input into losses of separation that were or should have been classified as 

a pilot deviation rather than waiting until after the entire ATO investigation process is 

completed. FAA’s response seems to indicate that it intends to simply maintain the 

status quo. We therefore request that FAA reconsider its position on how it plans to 

ensure that Flight Standards has input into the accurate classification of losses of 

separation at the “front end” of the process. 

FAA’s comments, our response, and further Agency actions required are fully 

discussed on pages 13 and 14. FAA’s entire response is included at the appendix to 

this report. We appreciate the courtesies and cooperation of FAA representatives 

during this audit. If you have any questions concerning this report, please contact me 

at (202) 366-1427 or Daniel Raville, Program Director, at (202) 366-1405. 

# 

cc: Acting FAA Deputy Administrator 

 Anthony Williams, ABU-100 
TABLE OF CONTENTS 

FINDINGS .............................................................................................. 1

Events at DFW TRACON Were Not Systemic .................................. 1

FAA Flight Standards Service Needs To Take Additional 

Actions To Ensure All Losses of Separation Are 

Accurately Reported and Investigated To Reduce 

Potential Safety Risks ....................................................................... 4

FAA Needs To Ensure Losses of Separation Caused by 

Controllers and Pilots Are Evaluated Consistently............................ 6

FAA Needs To Implement TARP as a Full-Time 

Separation Conformance Tool To Ensure That All 

Losses of Separation Are Accurately Reported ................................ 7

FAA Needs To Take Additional Actions To Improve the 

ATO’s Oversight Role of Reported Losses of Separation................. 9

RECOMMENDATIONS........................................................................ 12

AGENCY COMMENTS AND OFFICE OF INSPECTOR 

GENERAL RESPONSE....................................................................... 13

EXHIBIT A. SCOPE AND METHODOLOGY ..................................... 15

EXHIBIT B. ORGANIZATIONS VISITED OR 

CONTACTED....................................................................................... 17

EXHIBIT C. PRIOR AUDIT REPORTS............................................... 18

EXHIBIT D. MAJOR CONTRIBUTORS TO THIS 

REPORT .............................................................................................. 19

APPENDIX. AGENCY COMMENTS................................................... 20
1

FINDINGS 

We found that the events that transpired at DFW TRACON were not systemic. 

However, we did identify control and oversight weaknesses in FAA’s process for 

reporting and investigating losses of separation caused by pilots and controllers 

that could contribute to similar problems in the future. These weaknesses were 

due in part to inadequate FAA guidance for investigating these events and 

insufficient staffing in the ATO Safety Office. Further, FAA’s current processes 

do not ensure that all losses of separation are accurately reported across terminal 

and en route facilities or consistently evaluated for severity. 

FAA has initiated actions to correct some of these weaknesses, but additional 

actions and follow-up are still needed to ensure that all losses of separation are 

accurately reported and investigated. Specifically, FAA needs to improve its 

process for reporting and investigating losses of separation by (1) improving 

inspectors’ procedures for investigating pilot deviations, (2) evaluating losses of 

separation caused by pilots and controllers consistently, and (3) implementing 

TARP as a full-time separation conformance tool. 

FAA also needs to enhance the ATO’s oversight role in ensuring that losses of 

separation are reported accurately by (1) implementing planned organizational 

changes in a timely manner (2) involving Flight Standards early in the event 

determination process, and (3) improving its facility oversight during Air Traffic 

facility audits. 

Events at DFW TRACON Were Not Systemic 

Our sample results identified only 3 incidents (out of the total 372 events 

sampled—166 pilot deviations and 206 other or “non-events”) that should have 

been reported as operational errors or proximity events in FY 2007.7

 Therefore, 

we conclude that the problems identified at DFW were not occurring on a systemwide basis. 

To evaluate whether the reporting problems that occurred at DFW TRACON were 

occurring at other facilities, we reviewed pilot deviations, quality assurance 

reviews (QARs), and Operational Error Detection Program (OEDP) alerts and 

related supporting documentation (e.g., radar and voice data). Our analysis is fully 

discussed below. 

 

7

 Proximity events are minor losses of separation between two aircraft where 90 percent or greater of the required 

separation is maintained in either the horizontal or vertical plane. 

Findings 
2

• Pilot Deviations: Our sample results identified only one pilot deviation (at 

Chicago TRACON) that should have been reported as an operational error. 

We reviewed a statistical sample of 166 pilot deviations (with a loss of 

separation) that occurred during FY 2007 at 13 randomly selected Air Traffic 

facilities. We selected these from a total universe of 484 pilot deviations with 

a loss of separation that occurred at 63 TRACON and en route facilities.8

 

Based on the results of our sample, we can statistically project (with a 

90-percent confidence level) that the percentage of misclassified pilot 

deviations is between 0.2 and 1.4 percent. In other words, we can project that 

between 1 and 7 loss of separation events in FY 2007 were misclassified as 

pilot deviations when they should have been classified as operational errors. 

This is significantly less than the 39 operational errors and deviations that were 

misclassified as pilot deviations at DFW TRACON alone. 

• QAR/OEDP Alerts: We also judgmentally sampled 206 QARs and OEDP 

alerts that occurred in FY 2008 at the 10 facilities we visited. Our sample 

results identified only 2 incidents (1 percent) that should have been reported as 

proximity events (at Chicago TRACON and Denver Center). Again, this is 

significantly less than the 23 operational errors and deviations that were 

erroneously reported as “non-events” at DFW TRACON. 

- QARs are reviews conducted of other Air Traffic incidents that do not 

involve an operational error (e.g., pilot complaints, emergencies, etc.). 

These incidents are recorded on facility daily logs and reviewed by facility 

management or other staff to evaluate controller performance. At DFW 

TRACON, we found that the facility improperly recorded 23 operational 

errors as QAR “non-events.” 

- OEDP alerts are generated by a system at en route facilities that 

automatically detects and alerts management when a loss of separation 

occurs. Each OEDP alert is investigated to determine if it is an operational 

error, pilot deviation, or non-event.9

 At en route centers, we judgmentally 

sampled OEDP alerts that the facility investigated and classified as either a 

pilot deviation or a “non-event.” We were unable to do similar tests at 

TRACON facilities because they do not have a corresponding automatic 

system to identify when a loss of separation occurs. 

 

8

 All TRACONs and en route centers that reported a pilot deviation with a loss of separation. We excluded DFW 

TRACON from our universe. 9

 OEDP alerts may be non-events for several different reasons. For example, an OEDP alert may be for aircraft flying 

under Visual Flight Rules (VFR). Under VFR, FAA prescribed separation standards do not apply and pilots assume 

responsibility for maintaining safe separation from other aircraft. The OEDP may also alert when military formation 

flights are operating; again, FAA separation standards do not apply between the military aircraft in such situations. 

Findings 
3

As shown in the table below, we identified 3 discrepancies at 2 of the 13 facilities 

randomly selected: Chicago TRACON and Denver Center. 

Table. OIG Review of Pilot Deviation and QAR/OEDP Alert 

Sample Results by Facility 

Facility PDs 

Reviewed 

PDs that 

should be 

OEs 

QAR/OEDP 

Alerts 

Reviewed 

QAR/OEDP 

Alerts that 

should be 

OEs 

Atlanta 

TRACON 

11 0 13 0 

Chicago 

TRACON 

10 1 13 1 (PE) 

No. Cal. 

TRACON 

9 0 13 0 

Salt Lake 

TRACON 

5 0 10 0 

So. Cal. 

TRACON 

31 0 14 0 

Atlanta Center 30 0 19 0 

Cleveland Center 11 0 30 0 

Denver Center 11 0 30 1 (PE) 

LA Center 24 0 40 0 

Miami Center 13 0 24 0 

Minneapolis 

TRACON 

1 0 N/A* N/A 

Portland 

TRACON 

2 0 N/A* N/A 

Seattle Center 8 0 N/A* N/A 

TOTAL 166 1 206 2 

Key: PD – Pilot Deviation OE – Operational Error PE – Proximity Event 

* N/A – We did not visit these three facilities and therefore did not review their QAR/OEDP 

alerts.

Findings 
4

FAA Flight Standards Service Needs To Take Additional Actions To 

Ensure All Losses of Separation Are Accurately Reported and 

Investigated To Reduce Potential Safety Risks 

While problems that occurred at DFW TRACON were not systemic, we did 

identify control weaknesses in FAA’s overall reporting and investigating process 

for loss of separation events. Specifically, we found that pilot deviations were not 

always investigated thoroughly due to inadequate FAA guidance. During our 

review, Flight Standards issued guidance to address many of the weaknesses in the 

investigation process. However, additional actions and follow-up are needed to 

ensure that pilot deviations are properly reported, classified, investigated, and 

addressed. 

Pilot Deviations Were Not Always Investigated Thoroughly 

During this review and our prior investigation at DFW TRACON, we found 

evidence that Flight Standards inspectors responsible for investigating pilot 

deviations did not perform a thorough investigation of the incidents. For example: 

• Inspectors did not review pilot deviations that were referred to the airline’s 

Aviation Safety Action Program (ASAP). Instead, we found that some 

inspectors relied solely on the facility’s classification of pilot deviations that 

were referred to ASAP and did not review or validate the accuracy of the 

report. At least 19 of the 166 pilot deviations we reviewed were submitted 

through ASAP. 

• In addition, 20 pilot deviation reports10 from our sample of 166 pilot deviations 

did not appear to be fully investigated by the inspector. For instance, either the 

inspector’s final report did not provide any details about cause of the incident 

or the narrative section of the inspector’s report was entirely blank. Therefore, 

it was unclear if inspector performed a thorough investigation of the incident. 

We also identified incidents where the inspector erroneously concluded that 

the Air Traffic controller caused the loss of separation when, in fact, we 

determined it was due to pilot error. 

Flight Standards Has Issued Guidance To Address Many of the Control 

Weaknesses in Its Pilot Deviation Investigation Process 

In February, March, and July of 2008, Flight Standards issued guidance that 

addressed many of the control weaknesses identified during the DFW TRACON 

investigation and this review. For example, Flight Standards guidance now 

requires inspectors to: 

 

10 Note: Not all final reports were available for review. 

Findings 
5

• verify the accuracy of data in Air Traffic’s preliminary report, including those 

referred to ASAP. 

• include a full description of the pilot deviation, causal factors, and 

recommendations for systemic correction action. For ASAP pilot deviations, 

inspectors are to request this information from the ASAP Event Review 

Committee. 

• have a review of the final report conducted by the office management and the 

regional office. 

• provide full justification when their investigation determines the pilot deviation 

should be reclassified. 

Also, in August 2008, FAA trained its Flight Standards managers on the new 

requirements and completed its training of field inspectors on their responsibilities 

for investigating operational errors and pilot deviations in October 2008. 

While these actions represent progress, FAA Flight Standards will need to follow 

up once the requirements are implemented to ensure inspectors are complying 

with the new procedures. 

FAA Does Not Retain Flight Radar Data, Which Could Be Used To 

Validate Inspector Investigations of Losses of Separation 

Flight Standards Service inspectors have 90 days to complete their investigation of 

an event (e.g., to determine whether the pilot actions constitute a violation of 

Federal Aviation Regulations). However, en route and terminal facilities maintain 

flight radar data for only 15 and 45 days, respectively. Therefore, if an inspector’s 

investigation determines that an event was caused by controller error, key radar 

data from the flight may be unavailable to validate the conclusion. FAA’s current 

guidelines only specifically require that the voice data be retained for 21⁄2 years for 

pilot deviations. Maintaining the radar replay data for pilot deviations with a loss 

of separation for a longer period would greatly increase the value and accuracy of 

third-party reviews of loss of separation events. 

During our review, we were unable to determine if the controller contributed to the 

loss of separation for four pilot deviations because radar replay data was not 

available. For instance, at one en route facility, we questioned whether the 

controller may have contributed to the loss of separation by not instructing the 

pilot to turn the aircraft sooner or in a different direction. However, we were 

unable to verify this because the radar data were not available to evaluate the 

controller’s options at the time of the incident. 

Findings 
6

FAA Needs To Ensure Losses of Separation Caused by Controllers 

and Pilots Are Evaluated Consistently 

Unlike its process for operational errors, FAA does not rate the proximity of (i.e., 

how close aircraft came to one another) or have a goal for reducing the risk of 

pilot deviations that cause a loss of separation. This is despite the fact that pilot 

deviations can pose the same risk for a catastrophic accident as operational errors. 

Loss of separation events can vary from minor incidents (where there was no 

threat of a collision) to severe incidents (where a collision was barely avoided). 

Categorizing the severity of events allows FAA to focus resources on identifying 

the root cause and taking immediate corrective actions for those incidents where 

there is the greatest potential for another, possibly catastrophic, incident. FAA 

rates controller operational errors based on proximity—from A (most serious) to C 

(least serious) and maintains goals for reducing category A and B incidents. 

We noted recent pilot deviations that would have been rated as severe events 

under FAA’s criteria for operational errors. However, because these events were 

pilot deviations, the severity of the event was not measured and no analysis was 

conducted to determine if similar deviations were occurring so that corrective 

actions could be taken systemwide. 

For example, on August 13, 2007, while under control of the Southern California 

TRACON, a pilot took his plane above his assigned altitude and conflicted with 

another aircraft even though he had read back the clearance correctly to the 

controller. Using FAA’s operational error severity classification methodology, the 

event would have been classified as a serious, category A incident as the 2 aircraft 

narrowly missed each other by 100 feet vertically and about 1 half mile 

horizontally. 

Using FAA’s proximity rating system for operational errors, we estimated the 

ratings for pilot deviations that occurred while under control of en route and 

TRACON facilities during FY 2007. Of the 478 pilot deviations, we estimate that 

33 percent would be rated as serious category A or B incidents. 

Figure 2 below is an example of radar display of a pilot deviation (while under the 

control of the Seattle en route center) that, if categorized in the same method as 

operational errors, would have been rated as a category B event. The event 

occurred when a military pilot (EXP094) took his plane above his assigned 

altitude and conflicted with a regional air carrier (QXE397). The aircraft came 

within 300 feet vertically and 2.96 miles horizontally of each other. The Flight 

Standards inspector closed the report with no action, in part, because he did not 

think it was significant enough. 

Findings 
7

Figure 2. Radar Display of a Serious Pilot Deviation 

By evaluating losses of separation caused by pilots in the same manner as those 

caused by controllers, FAA would have a clearer understanding of the causes of 

these incidents and could use this information to improve both the accurate 

classification of the incidents and the safety of the National Airspace System. To 

illustrate, we reviewed pilot deviations in our sample and noted several trends that 

may need national attention. For example we found that: 

• 53 of the incidents occurred when the pilots acknowledged and read back the 

correct altitude assigned but deviated to a different altitude; 

• 27 occurred because the pilots either had equipment malfunctions or did not 

accurately program their aircraft navigational equipment (e.g., forgot to enter 

changes in the Flight Management System, entered the wrong altitude, or set 

the altimeter incorrectly, etc.); and 

• 19 occurred because the pilots did not follow established departure or arrival 

procedures, including those established by newly implemented area navigation 

(RNAV) procedures. 

FAA should establish a process to rate the severity of pilot deviations that cause a 

loss of separation and establish a corresponding goal to reduce the most severe 

incidents. 

FAA Needs To Implement TARP as a Full-Time Separation 

Conformance Tool To Ensure That All Losses of Separation Are 

Accurately Reported 

FAA will not have adequate assurance that losses of separation at TRACON 

facilities are reported accurately until it fully implements TARP as a full-time 

separation conformance tool. TARP is an automated tool that will identify when 

losses of separation occur at terminal facilities. FAA began developing TARP in 

Findings 
8

2005 to improve operational error reporting in response to our and the National 

Transportation Safety Board’s longstanding recommendations. FAA plans to 

implement TARP in three phases. 

• During Phase 1, TARP will be used only to conduct terminal facilities’ 

required 2-hour monthly radar audits. FAA plans to complete this phase by the 

end of calendar year 2009. 

• In Phase 2, facilities will continue to use TARP as an audit tool but for 

increasing periods of time beyond the current 2-hour requirement. FAA will 

not finalize its Phase 2 implementation strategy until April 30, 2009. 

• In Phase 3, facilities will use TARP continuously (24/7) in the operating 

quarters as a separation conformance tool. However, this will require 

additional technical improvements and infrastructure to operate reliably in this 

capacity. At the time of our review, FAA had not established milestones for 

developing an implementation strategy for this phase. 

It will be important that FAA moves expeditiously to implement TARP as a fulltime separation conformance tool. To do this, FAA will need to address potential 

barriers. For example, managers expressed concerns that TARP may have a 

significant impact on workload. That is, if TARP is constantly alerting for very 

minor errors11 that otherwise cannot be detected on the controller’s radar scope, it 

could consume a substantial amount of resources to investigate each alert due to 

their sheer volume. FAA officials indicated that they plan to mitigate such 

workload issues by programming TARP to automatically fill out the proximity 

event form and not require a full investigation. 

While FAA is working to address this potential barrier, history shows that 

additional barriers or challenges are likely to occur during the implementation of 

new systems. One way to proactively mitigate them is to establish a method to 

share best practices among facilities. We interviewed personnel from seven 

facilities that had received TARP (six certified facilities and one where TARP was 

in the testing stage). We found that most thought TARP was a very user-friendly 

system and agreed that a process for sharing best practices in using the system 

would be helpful. In addition, as FAA develops its implementation strategy for 

Phase 2 implementation, it needs to work aggressively in developing milestones 

for implementing Phase 3 of TARP. 

 

11 TARP can identify losses to the hundredth of a mile. 

Findings 
9

FAA Needs To Take Additional Actions To Improve the ATO’s 

Oversight Role of Reported Losses of Separation 

FAA has taken several actions to improve its oversight of reporting and 

investigating losses of separation caused by pilot and controllers, but additional 

actions are still needed to ensure these incidents are reported and classified 

accurately. Prior to our DFW TRACON investigation, there was little or no 

oversight provided over the accurate classification of losses of separation. 

ATO Oversight Processes Were Not Adequate To Identify Potential 

Reporting Problems 

We found that the ATO Safety Office’s oversight process of reporting and 

investigating losses of separation was inadequate. For example, neither the ATO 

Headquarters nor the Service Center Safety offices discovered or challenged the 

incidents that were misclassified by DFW TRACON. In addition, ATO Safety 

Office personnel did not routinely review pilot deviations to ensure they were 

accurately reported. Also, if the Flight Standards inspector disagreed with Air 

Traffic’s initial classification of a pilot deviation, ATO Safety personnel did not 

follow up to verify whether a controller’s actions may have contributed to the 

incident. 

The weaknesses in the ATO’s oversight occurred, in part, because ATO-Safety 

personnel lacked stable leadership and staffing resources to provide adequate 

oversight. The ATO did not have a permanent Vice President (VP) for Safety 

Services for more than 18 months12 and has not had a permanent manager for its 

Evaluations and Investigations office for 2 years. 

Over the last 10 years, the number of employees in this office decreased from 79 

to 24 (as of March 2008). In addition, ATO officials told us that hiring qualified 

and experienced employees is a challenge because Air Traffic facilities do not 

want to give up qualified Air Traffic employees. Further, those officials told us 

that there were financial disincentives for employees to transfer from Air Traffic 

facilities to Headquarters, as they could experience substantial pay cuts—these 

cuts range from $16,000 to $41,000 depending on the pay band. 

FAA Is Taking Actions To Revamp the ATO’s Safety Oversight 

In response to the problems at DFW TRACON, FAA has either taken or plans to 

take several actions to improve its oversight. Specifically: 

 

12 A permanent VP for Safety Services was hired in April 2008. 

Findings 
10

• In 2007 (after problems at DFW TRACON began to surface), FAA 

Headquarters13 and Service Center personnel began to closely scrutinize all 

pilot deviations with a loss of separation to determine if Air Traffic actions 

contributed to the incident. 

• ATO Headquarters Safety personnel improved its oversight of the terminal 

facilities’ 2-hour monthly radar audits. For example, facilities can now be 

selected more than once per year and are not notified in advance that their data 

will be audited by Headquarters. These changes have improved the facility’s 

reporting of operational errors. For example, 6 months prior to implementing 

these changes, facilities reported no operational errors as a result of their 

monthly audits. Three months after these new procedures were implemented, 

facilities identified 10 operational errors and 3 proximity events as a result of 

their monthly audits. 

• FAA’s Evaluation and Investigation office now has 34 permanent staff plus 

9 additional personnel on detail. 

• FAA plans to establish a process whereby ATO Headquarters Safety personnel 

will review pilot deviations after Flight Standards investigations are completed. 

• Finally, in response to our DFW TRACON report, the ATO’s Chief Operating 

Office directed the VP for Safety Services to conduct a complete, “top-tobottom” review of ATO Safety Services to determine what additional resources 

are required to meet its responsibilities. 

As part of this ongoing review, FAA plans to establish an independent quality 

assurance function in the ATO Service Areas that will report directly to the ATO’s 

VP for Safety Services. This function will continually oversee event reporting, 

make event determinations (i.e., whether a loss of separation is pilot or controller 

error), and ensure audit data integrity of facility reports. This change will transfer 

the responsibility for event determination from the facility manager to this new 

office. FAA expects to complete all of these organizational changes by the 

summer of 2009. 

These actions should help alleviate ATO Safety Office staffing issues and improve 

oversight at the national and Service Area levels. Nevertheless, additional actions 

are still needed in three areas. 

• First, FAA needs to closely monitor the planned actions to ensure they are 

completed in a timely manner. These actions include hiring a permanent 

 

13 Reviews are being conducted by ATO staff from Safety’s Evaluations and Investigations Office and En Route and 

Oceanic’s Quality Assurance and Safety Office and by Aviation Safety’s Air Traffic Safety Oversight group. 

Findings 
11

Director for the Evaluations and Investigations office and formalizing 

procedures for the ATO Safety Office’s review of pilot deviations with a loss 

of separation. FAA should initiate an internal audit conducted by its 

independent Aviation Safety Oversight Office (AOV) once the planned 

changes have been implemented to determine that they are being complied 

with. 

• Second, once the ATO transfers the event determination responsibility from 

the facility to its independent quality assurance staff located in the Service 

Areas, Flight Standards should provide a liaison from its regional offices (colocated with the ATO Safety Areas) to assist ATO Safety staff in the event 

determinations when losses of separation occur. By doing so, Flight Standards 

will have more timely input into losses of separations that were or should have 

been classified as a pilot deviation rather than waiting until after the entire 

ATO investigation process is completed. 

• Finally, FAA needs to ensure that it thoroughly evaluates the accuracy of 

reporting during its ATO Safety facility audit process. ATO Safety 

Evaluations staff is required to conduct audits of all Air Traffic facilities once 

every 3 years. The audits provide an independent method of assessing the 

facility’s compliance with FAA directives and procedures, including its 

processes for investigating and reporting losses of separation. 

In the case of DFW TRACON, the facility audit (which was completed just 

4 months prior to our investigation at the facility) did not effectively identify 

operational error reporting weaknesses. Specifically, in March 2007, the ATO’s 

facility evaluations group conducted a facility audit of DFW TRACON but did not 

identify operational error reporting problems. Further, the evaluators concluded 

that the TRACON’s QAR process met requirements and noted that its review of 

QARs revealed a “commendable” process. However, our investigation at DFW 

(started only a few months later in July 2007) revealed that 23 (37 percent) of the 

62 misreported operational errors and deviations were originally recorded as nonevent QARs. Therefore, the thoroughness of facility audits in evaluating the 

accuracy of operational error reporting is highly questionable. 

We also found that the procedures for documenting and investigating QARs varied 

at the facilities we visited. At one facility, we identified significant weaknesses in 

the QAR procedures. For example, at one location visited, after initiating a QAR 

to investigate a possible incident, managers either did not complete the 

investigation or failed to document the results of the investigation as required. 

Such weaknesses could indicate that the facility has a lax investigation process, 

which could lead to inaccurate reporting. Therefore, it is important that during its 

facility audit process, that ATO Safety personnel thoroughly review the facilities’ 

QAR process. 

Findings 
Recommendations 

12

RECOMMENDATIONS 

We recommend that FAA: 

1. Establish a follow-up mechanism to ensure that Flight Standards Service 

inspectors are complying with new guidance for investigating pilot deviations 

issued in February, March, and July of 2008. 

2. Require that radar replay data for pilot deviations with a loss of separation be 

retained for 21⁄2 years (similar to retention requirements for voice data). 

3. Establish a process to rate the severity of pilot deviations that cause a loss of 

separation and establish a corresponding goal to reduce the most severe 

incidents. 

4. Develop milestones for implementing TARP as a full-time separation 

conformance tool. 

5. Initiate an internal AOV audit of the planned changes to the ATO’s safety 

oversight process to ensure compliance with the new procedures. 

6. Assign a regional Flight Standards liaison to assist the ATO Safety Services 

staff in determining whether losses of separation are pilot or controller errors. 

7. Modify the ATO facility audit process by including requirements to 

specifically review facility QARs to determine if incidents reported as “nonevents” were accurately classified. 
13

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 

RESPONSE 

We provided FAA with our draft report on December 30, 2008, and received its 

response on March 12, 2009. In its response, FAA agreed with recommendations 

1, 2, 4, 5, and 7 and provided acceptable corrective actions. FAA did not agree 

with recommendations 3 and 6. FAA’s response is summarized below and 

included in its entirety in the appendix to this report. 

Recommendation 1: FAA concurred and stated that it will develop a follow-up 

process to ensure inspector compliance with new guidance for investigating pilot 

deviations by December 31, 2009. FAA’s response, planned actions, and target 

date meet the intent of our recommendation. 

Recommendation 2: FAA partially concurred and stated that it suggests that the 

recommendation be modified to state that radar and voice data retention 

requirements be the same for both operational errors and pilot deviations with a 

loss of separation, and that retained data be required to allow subsequent audit and 

confirmation of loss categorization. FAA further stated that “Air Traffic 

Oversight Services (ATO-S), the ATO System Operations Service (ATO-R), and 

the Flight Standards Service (AFS) will coordinate their organizations’ respective 

directives to ensure common and compatible data retention requirements are 

published by September 30, 2009.” FAA’s alternative actions and target date 

address the intent of our recommendation. 

Recommendation 3: FAA non-concurred and stated that it “addresses the 

seriousness of the violation during the investigation and adjusts punishment 

according to the degree of the violation.” However, our recommendation was not

aimed at rating the severity of pilot deviations to assess punitive actions against 

pilots on a case-by-case basis. Rather, we intended for FAA to assess the severity 

of incidents to advance its risk-based approach to safety oversight. This would 

allow FAA to gather macro-level data, which it could use to identify possible 

trends, potentially systemic issues, and corrective actions needed at the national 

level. 

FAA already uses a similar severity rating approach to identify systemic issues for 

virtually all other aviation incidents including operational errors, near midair 

collisions, and runway incursions. In fact, FAA currently rates the severity of 

pilot deviations that occur on the ground (i.e., runway incursions) but not pilot 

deviations that occur in the air. Given that aviation safety is FAA’s primary 

mission, we are concerned that the Agency would bypass an opportunity to 

advance its risk-based safety oversight by gathering data on the most severe pilot 

deviations to identify systemic issues and taking actions to address the root causes 

Agency Comments and Office of Inspector General Response 
Agency Comments and Office of Inspector General Response 

14

at the national level. We believe FAA needs to reconsider its position and 

therefore consider this recommendation unresolved. 

Recommendation 4: FAA concurred and stated that it has established in its 

FY 2009 Business Plan several milestones and target dates for implementing 

TARP. FAA’s response, planned actions, and target dates meet the intent of our 

recommendation. 

Recommendation 5: FAA concurred and stated that “AOV currently has an 

established audit mechanism that will evaluate any planned changes in ATO’s 

Safety Oversight process when they are developed.” FAA’s response addresses 

the intent of our recommendation, but we request that FAA provide us with target 

dates for evaluating the planned changes to the ATO’s safety oversight process. 

Recommendation 6: FAA non-concurred and stated that “ATO-S will continue 

to work with [aviation safety inspectors] to resolve reported violations of air traffic 

rules and procedures, eliminating the need to assign a regional Flight Standards 

Liaison.” However, the intent of our recommendation was to allow Flight 

Standards to have more timely input into losses of separations that were or should 

have been classified as a pilot deviation rather than waiting until after the entire 

ATO investigation process is completed. FAA’s response seems to indicate that it 

intends to simply maintain the status quo. We therefore request that FAA 

reconsider its position on how it plans to ensure that Flight Standards has input 

into the accurate classification of losses of separation at the “front end” of the 

process. Accordingly, we consider this recommendation unresolved. 

Recommendation 7: FAA concurred and stated that ATO-S is revising both its 

requirements for conducting Quality Assurance Reviews (QARs) and its audit 

processes in general. FAA expects to complete this effort by September 30, 2009. 

FAA’s response, planned actions, and target date meet the intent of our 

recommendation. 

ACTIONS REQUIRED 

We consider FAA’s planned actions and target dates for recommendations 1, 2, 4, 

and 7 to be responsive. These recommendations are considered resolved pending 

completion of these actions. While FAA’s actions to address recommendation 5 

are also responsive, we request that FAA provide target dates for completion. In 

accordance with DOT Order 8000.1C, we request that FAA reconsider its position 

regarding recommendations 3 and 6. Please provide your written response 

regarding recommendations 3, 5, and 6 within 30 days of this report. 
15

EXHIBIT A. SCOPE AND METHODOLOGY 

We conducted this performance audit in accordance with generally accepted 

government auditing standards. Those standards require that we plan and perform 

the audit to obtain sufficient, appropriate evidence to provide a reasonable basis 

for our findings and conclusions based on our audit objective. We believe that the 

evidence obtained provides a reasonable basis for our findings and conclusions 

based on our audit objective. The audit was conducted between November 2007 

and December 2008. The following scope and methodology were used in 

conducting this review. 

To determine whether the reporting problems that occurred at DFW TRACON 

were occurring at other facilities, we verified the accuracy of how pilot deviation, 

QARs and OEDP alerts were classified. Specifically, we statistically reviewed 

166 pilot deviations with a loss of separation that occurred during FY 2007 at 

13 Air Traffic facilities. We also judgmentally selected 206 QAR and OEDP 

alerts that occurred at 10 facilities in FY 2008, which the facilities determined not 

to be operational errors. Our methodology in reviewing these events was 

consistent with the methodology used during our review of DFW TRACON. 

To determine if pilot deviations, QARs and OEDP alerts were classified 

accurately, we: 

• reviewed radar and voice data. 

• reviewed preliminary and final pilot deviation reports and related 

documentation. 

• reviewed QAR reports and related documentation. 

• reviewed OEDP alert logs and related documentation. 

• interviewed ATC and flight standards personnel as needed. 

• reviewed operational error documentation if the pilot deviation was also an 

operational error. 

To evaluate FAA’s policies and procedures and identify the roles and 

responsibilities for reporting and investigating these incidents we did the 

following: 

• Interviewed representatives from the following Air Traffic Organization 

(ATO) offices: 

-ATO Safety Services, Evaluations and Investigations and Vice President for 

Safety 

Exhibit A. Scope and Methodology 
16

-ATO Terminal Service, Quality Assurance 

-En Route and Oceanic Services, Quality and Safety Assurance 

-Two Service Area, Service Centers (see exhibit B) 

-Ten Air Traffic Control facilities (see exhibit B), including Air Traffic 

Managers, Operations Managers, Quality Assurance and Safety Managers, 

and quality assurance staff. 

• Interviewed representatives from the following Aviation Safety offices: 

-Air Traffic Safety Oversight Services 

-Flight Standards Service-Quality Assurance 

-Five Flight Standards District Offices (see exhibit B) 

-One Regional Flight Standards Division Office 

• Reviewed the following FAA guidance: 

-FAA Order 7210.56C, Air Traffic Quality Assurance Order and related 

Notices and changes. 

-FAA Order 8900.1, Flight Standards Information Management System 

(FSIMS) and related Notices. 

-FAA Order 8020.11B, Aircraft Accident and Incident, Notification, 

Investigation, and Reporting. 

-FAA Order 8020.16, Air Traffic Organization Aircraft Accident and 

Incident, Notification, Investigation, and Reporting. 

-FAA Order 7110.65R, Air Traffic Control. 

-Flight Standards Quality Management System, QPM # AFS 001-020, Pilot 

Deviation Process. 

• Reviewed the implementation status of TARP. 

To determine the accuracy and completeness of the pilot deviation database used 

to select our statistical sample, we judgmentally sampled pilot deviation reports 

not included in our sample at facilities visited to ensure they did not involve a loss 

a separation. We also compared these reports to the pilot deviations listed on the 

national pilot deviation database. 

Exhibit A. Scope and Methodology 
17

EXHIBIT B. ORGANIZATIONS VISITED OR CONTACTED 

FAA Headquarters, Washington, DC 

• ATO Safety Services 

• ATO En Route and Oceanic Services 

• ATO Terminal Services 

• Aviation Safety - Flight Standards Service 

• Aviation Safety - Air Traffic Safety Oversight Service 

Service Areas/Regional Offices 

• Eastern Service Center, Safety Assurance Group 

• Western Service Center, Safety Assurance Group 

• Northwest Mountain Region Flight Standards 

Air Traffic Control Facilities 

• Atlanta En Route Center 

• Atlanta TRACON 

• Chicago TRACON 

• Cleveland En Route Center 

• Denver En Route Center 

• Los Angeles En Route Center 

• Miami En Route Center 

• Northern California TRACON 

• Salt Lake TRACON 

• Southern California TRACON 

Flight Standards District Offices (FSDO) 

• Atlanta FSDO 

• Denver FSDO 

• Los Angeles FSDO 

• Riverside FSDO 

• Sacramento FSDO 

Exhibit B. Organizations Visited or Contacted 
18

EXHIBIT C. PRIOR AUDIT REPORTS 

Since 2000, we have issued three audit reports on operational errors. 

• OIG Report Number AV-2001-11, “Actions To Reduce Operational Errors and 

Deviations Have Not Been Effective,” December 15, 2000. 

• OIG Report Number AV-2003-040, “Operational Errors and Runway 

Incursions,” April 3, 2003. 

• OIG Report Number AV-2004-085, “Controls Over the Reporting of 

Operational Errors,” September 20, 2004. 

In our 2000 and 2004 reports, we specifically addressed the need for FAA to 

improve controls over the reporting operational errors as described below. 

In December 2000, we reported that FAA was at risk of underreporting 

operational errors at terminal facilities because, at these facilities, FAA relied on 

controller self-reporting. Further, adequate documentation was not always 

available to confirm if an operational error occurred because radar and voice tapes 

were retained for only 15 days. We recommended that FAA implement NTSB’s 

recommendation to extend the retention period for voice and radar tapes from 

15 days to 45 days. 

At the time, FAA indicated that it could not extend the retention period for radar 

and voice data because it would create a storage problem. Therefore, it agreed to 

retain voice tapes and radar data for 45 days only for known or suspected 

incidents. Since that time FAA has updated its terminal radar and voice data with 

digital technology and now retains these data for 45 days. 

In September 2004, we again reported that operational errors were at risk of being 

underreported because of FAA’s reliance on self-reporting at terminal facilities. 

At the time of our report, FAA indicated that because of the complexity of the 

airspace in the terminal environment, it could not develop an automated system to 

identify when operational errors occur. 

In response to our 2004 recommendations, FAA established procedures in 2005 

that requires terminal facilities to conduct monthly audits of radar data to identify 

potential unreported operational errors. Additionally, FAA Headquarters officials 

were also required to randomly review facility audit results to ensure compliance. 

Since that time, FAA began to develop a system (TARP) to automatically identify 

operational errors at terminal facilities. FAA is now implementing this system. 

Exhibit C. Prior Audit Reports 
Exhibit D. Major Contributors to This Report 

19

EXHIBIT D. MAJOR CONTRIBUTORS TO THIS REPORT 

 Name Title 

 Daniel Raville Program Director 

 Liz Hanson Project Manager

 Mark Gonzales Senior Analyst 

 Tasha Thomas Analyst

 Kevin Montgomery Analyst 

 Andrea Nossaman Writer-Editor 
20

APPENDIX. AGENCY COMMENTS 

Federal Aviation 

Administration

Memorandum 

Date: March 12, 2009 

To: Lou Dixon, Assistant Inspector General for Aviation and Special Program Audits 

From: Ramesh K. Punwani, Assistant Administrator for Financial Services/CFO 

Prepared by: Anthony Williams, x79000 

Subject: OIG Draft Report: FAA’s Process for Reporting and Investigating Operational 

Errors Federal Aviation Administration 

Thank you for the opportunity to review and comment on the findings and recommendations of 

the subject draft report dated December 30, 2008. The Federal Aviation Administration (FAA) 

concurs with Recommendations 1, 4, 5, and 7; nonconcurs with Recommendations 3, and 6; and 

partially concurs with Recommendation 2. 

The following is the Agency’s response to each of your Recommendations. 

OIG Recommendation 1: Establish a follow-up mechanism to ensure that Flight Standards 

Service inspectors are complying with new guidance for investigating pilot deviations issued in 

February, March, and July 2008. 

FAA Response: Concur. Flight Standards agrees and will develop a follow up process to ensure 

inspector compliance by December 31, 2009. 

OIG Recommendation 2: Require that radar replay data for pilot deviations with a loss of 

separation be retained for 2 1⁄2 years (similar to retention requirements for voice data). 

FAA Response: Partially concur with comment. ATO Directives currently require 2 1⁄2 year 

retention of supporting radar data for pilot deviations similar to those required for voice and radar 

data in operational error investigations. However, current radar data retention requirements do 

not specifically require playback capability. The FAA suggests that this Recommendation be 

modified to state that “radar and voice data retention requirements be the same for both 

operational errors and pilot deviations with a loss of separation, and that retained data be required 

to allow subsequent audit and confirmation of loss categorization”. 

Appendix. Agency Comments 
21

Air Traffic Oversight Service (ATO-S), the ATO System Operations Service (ATO-R), and the 

Flight Standards Service (AFS) will coordinate their organizations respective directives to ensure 

common and compatible data retention requirements are published by September 30, 2009. 

OIG Recommendation 3: Establish a process to rate the severity of pilot deviations that cause a 

loss of separation and establish a corresponding goal to reduce the most severe incidents. 

FAA Response: Nonconcur. Any violation of a Federal Air Regulation contained with the Code 

of Federal Regulation is treated as a serious violation. FAA addresses the seriousness of the 

violation during the investigation and adjusts punishment according to the degree of the 

violation. Each violation is addressed in its entirety. We will continue to focus our efforts on 

pilot education to reduce all forms of violations and continue to look for additional ways to 

measure the events. 

OIG Recommendation 4: Develop milestones for implementing Traffic Analysis and Review 

Program (TARP) as a full-time separation conformance tool. 

FAA Response: Concur. FAA has established in its Fiscal Year 2009 Business Plan the 

following milestones for implementing TARP. Complete TARP audit tool implementation at 

first 50 percent of applicable terminal sites by April 30, 2009. Finalize TARP Phase II 

implementation strategy plan by April 30, 2009. Complete TARP audit tool implementation at 

80 percent of all applicable terminal sites by September 30, 2009. Continue the development and 

deployment of TARP with a targeted completion of Terminal Implementation by December 30, 

2009 and NAS-wide implementation by 

September 30, 2011. 

OIG Recommendation 5: Initiate an internal AOV audit of the planned changes to the ATO’s 

Safety Oversight process to ensure compliance with the new procedures. 

FAA Response: Concur. AOV currently has an established audit mechanism that will evaluate 

any planned changes in ATO's Safety Oversight process when they are developed. 

OIG Recommendation 6: Assign a regional Flight Standards Liaison to assist the ATO Safety 

Services staff in determining whether losses of separation are pilot or controller errors. 

FAA Response: Nonconcur. Aviation Safety Inspectors (ASIs) do not have the background or 

training in the rules and regulations required of Air Traffic Controllers. However, ASIs should 

be responsible for the investigation into any reported violation of a Federal Air Regulation. 

Therefore, ATO-S will continue to work with ASIs to resolve reported violations of air traffic 

rules and procedures, eliminating the need to assign a regional Flight Standards Liaison. 

OIG Recommendation 7: Modify the ATO facility audit process by including requirements to 

specifically review facility QARs to determine if incidents reported as “non-events” were 

accurately classified. 

FAA Response: Concur. ATO-S is currently in the process of revising both its requirements for 

conducting Quality Assurance Reviews (QARs) and its audit processes in general. In both cases, 

it is the goal of ATO-S to improve the ATO’s accuracy and completeness in reporting losses of 

separation and other indicators of risk. ATO-S will develop revised QAR, or similar risk 

Appendix. Agency Comments 
Appendix. Agency Comments 

22

indicator report(s), definition(s) and processes by September 30, 2009. ATO-S will establish 

specific audit processes and requirements to review facilities’ accuracy and effectiveness in 

investigating suspected losses of separation. 
The following pages contain textual versions of the graphs and charts found in this 

document. These pages were not in the original document but have been added here to 

accommodate assistive technology. 
FAA’s Process for Reporting and Investigating Operational Errors 

Section 508 Compliant Presentation 

Figure 1. FAA’s Process for Reporting and Investigating Incidents Involving a 

Loss of Separation 

• Loss of separation identified by Air Traffic facility. Initial determination is made 

if the incident was controller or pilot error. 

• If the loss of separation is determined to be a pilot error, the Air Traffic facility 

forwards the preliminary report to the Flight Standards inspector who completes 

the investigation and final report. The Air Traffic facility also copies the 

appropriate Air Traffic Organization (ATO) Service Area and Unit on the 

preliminary report that it sends to Flight Standards. 

• If the loss of separation is determined to be a controller error, the Air Traffic 

Facility prepares preliminary investigation report and forwards the report to 

appropriate ATO Service Area and Unit for review. The Air Traffic facility 

completes the investigation and final report and forwards the final report to the 

appropriate ATO Service Area and Unit for review. The ATO Service Area and 

Unit review both the preliminary and final reports sent from the Air Traffic 

Facility. After review, the ATO Service Area and Unit forward the report to 

Headquarters ATO Safety office for review. The ATO Safety Office conducts the 

final review. 

Table. Office of Inspector General Review of Pilot Deviation and Quality 

Assurance Reviews (QARs) and Operational Error Detection Program (OEDP) 

Alerts - Sample Results by Facility 

Atlanta 

TRACON 

Pilot deviations 

reviewed: 11 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 13 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Chicago 

TRACON 

Pilot deviations 

reviewed: 10 

Pilot deviations that 

should be classified as 

operational errors: 1 

QAR and OEDP 

alerts reviewed: 13 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 

1 (Proximity 

Event) 
Northern 

California 

TRACON 

Pilot deviations 

reviewed: 9 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 13 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Salt Lake 

TRACON 

Pilot deviations 

reviewed: 5 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 10 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Southern 

California 

TRACON 

Pilot deviations 

reviewed: 31 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 14 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Atlanta Center Pilot deviations 

reviewed: 30 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 19 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Cleveland 

Center 

Pilot deviations 

reviewed: 11 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 30 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Denver 

Center 

Pilot deviations 

reviewed: 11 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 30 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 1 

(Proximity Event) 

Los Angeles 

Center 

Pilot deviations 

reviewed: 24 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 40 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Miami Center Pilot deviations 

reviewed: 13 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: 24 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 0 

Minneapolis 

TRACON 

Pilot deviations 

reviewed: 1 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: not 

applicable (We did 

not visit this facility 

and therefore did 

not review its QAR 

or OEDP alerts.) 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 

not applicable 
Portland 

TRACON 

Pilot deviations 

reviewed: 2 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: not 

applicable (We did 

not visit this facility 

and therefore did 

not review its QAR 

or OEDP alerts.) 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 

not applicable 

Seattle Center Pilot deviations 

reviewed: 8 

Pilot deviations that 

should be classified as 

operational errors: 0 

QAR and OEDP 

alerts reviewed: not 

applicable (We did 

not visit this facility 

and therefore did 

not review its QAR 

or OEDP alerts.) 

QAR and OEDP 

alerts that should be 

classified as 

operational errors: 

not applicable 

Total pilot deviations reviewed: 166. Total pilot deviations that should be classified as 

operational errors: 1 

Total QAR and OEDP alerts reviewed: 206. Total QAR and OEDP alerts that should 

be classified as operational errors: 2 

Figure 2. Radar Display of a Serious Pilot Deviation 

Figure 2 shows a radar display of a pilot deviation (while under the control of the 

Seattle en route center) that, if categorized in the same method as operational errors, 

would have been rated as a category B event. The event occurred when a military 

pilot (labeled on the screen as EXP094) took his plane above his assigned altitude and 

conflicted with a regional air carrier (labeled on the screen as QXE397). The aircraft 

came within 300 feet vertically and 2.96 miles horizontally of each other.