FAA NEEDS TO IMPROVE RISK ASSESSMENT 

PROCESSES FOR ITS AIR TRANSPORTATION 

OVERSIGHT SYSTEM

Federal Aviation Administration

 Report Number: AV-2011-026

 Date Issued: December 16, 2010
U.S. Department of

Memorandum

Transportation

Office of the Secretary

of Transportation

Office of Inspector General

Subject: ACTION: FAA Needs To Improve Risk 

Assessment Processes for Its Air Transportation 

Oversight System 

Federal Aviation Administration

Report Number AV-2011-026

Date: December 16, 2010

From: Jeffrey B. Guzzetti

Assistant Inspector General 

 for Aviation and Special Program Audits

Reply to 

Attn. of: JA-10

To: Federal Aviation Administrator

The Federal Aviation Administration (FAA) uses its Air Transportation Oversight 

System (ATOS) to conduct surveillance of nearly 100 airlines. These airlines 

transport more than 90 percent of U.S. airline passenger and cargo traffic. We 

have consistently reported that ATOS is conceptually sound because it is datadriven and intended to target inspector resources to the highest risk areas. 

However, in 2002 and 2005, we reported that FAA needed to strengthen national 

oversight of ATOS to hold field managers more accountable for consistently 

implementing effective oversight practices.

1

After hearings on the safety issues at Southwest, the Senate Committee on 

Science, Commerce, and Transportation and the House Committee on 

Transportation and Infrastructure requested that we review FAA’s oversight of the 

U.S. airline industry. As a result of the weaknesses identified in our 2008 

testimony,

 In 2008, safety lapses at Southwest 

Airlines exposed serious weaknesses in FAA’s risk-based oversight process at that 

airline. 

2

 1 OIG Report Number AV-2002-088, “Air Transportation Oversight System,” April 8, 2002.

the Committees asked us to determine if similar oversight weaknesses 

existed elsewhere in the system. Accordingly, our audit objectives were to 

determine (1) whether FAA has completed timely ATOS inspections of air 

carriers’ policies and procedures for their most critical maintenance systems;

OIG Report Number AV-2005-062, “FAA Safety Oversight of an Air Carrier Industry in Transition,” June 3, 2005.

 OIG reports are available on our website: www.oig.dot.gov. 2 OIG Testimony Number CC-2008-046, “Actions Needed To Strengthen FAA’s Safety Oversight and Use of 

Partnership Programs,” April 3, 2008. 
2

(2) how effective ATOS performance inspections have been in testing and 

validating that these critical maintenance systems are working properly; and 

(3) how well FAA implemented ATOS for the remaining Part 1213

RESULTS IN BRIEF 

air carriers and 

what, if any, oversight challenges FAA inspection offices face. We conducted the

audit from May 2008 through July 2010 in accordance with government auditing 

standards prescribed by the Comptroller General of the United States. 

FAA has worked to continuously improve its risk-based air carrier oversight 

system since introducing it in 1998. However, we identified three areas where 

FAA could improve its management and oversight of ATOS: 

• FAA did not perform timely ATOS inspections of policies and procedures (i.e., 

design assessments) for air carriers’ most critical maintenance systems. At all 

eight major air carriers we reviewed,

4 inspectors did not complete these 

inspections5

• FAA inspectors did not effectively assess whether critical maintenance systems 

were performing as intended (i.e., performance assessments) due to three 

process weaknesses. First, like the design assessments, FAA did not complete 

these inspections in a timely manner. For example, at the 8 major air carrier 

inspection offices we reviewed, between fiscal year (FY) 2005 and FY 2009, 

inspectors did not complete 340 inspections on time. Second, ATOS design 

flaws allow lower risk maintenance programs to be inspected before higher

risk programs. For example, at one FAA inspection office, the risk level for 

the air carrier’s maintenance manual system had nearly doubled within 1 year;

yet, it received lower inspection priority than the carrier’s cargo handling 

program, which showed minimal risk. Third, inspectors were not using all 

within FAA’s required 5-year interval—some had not been 

completed for nearly 8 years. For example, in three inspection offices, more 

than 5 years had lapsed between inspections of the Airworthiness Directive 

(AD) Management program, even though inspectors noted in ATOS that it had 

increased risk. Further, overdue inspections continued even after FAA reduced 

the number of maintenance programs that inspectors were required to review. 

Inspectors cited confusion with interpreting FAA guidance as one reason for 

not completing timely inspections. We also found that some inspections were 

overdue because FAA does not monitor inspections that have been scheduled 

but not yet assigned to an inspector.

 3 This refers to large, commercial operators regulated under 14 CFR Part 121, Operating Requirements: Domestic, 

Flag, and Supplemental Operations. These carriers operate larger aircraft with primarily scheduled flights. 4 Alaska Airlines, American Airlines, Continental Airlines, Delta Air Lines, Northwest Airlines, Southwest Airlines, 

United Airlines, and US Airways. For this review, we visited each of these FAA air carrier offices except Southwest 

Airlines; instead, we conducted a review of inspection data to supplement our report. 5 The terms “assessment” and “inspection” are used interchangeably in this report. 
3

available safety data to assess risk. For example, inspectors’ risk assessments

did not include analyses of voluntary disclosure data (i.e., maintenance errors 

that air carriers self-report) or changes that occurred in the airline industry. 

These data are readily available and, if analyzed regularly, could be valuable 

risk indicators to monitor air carrier maintenance programs. 

• Finally, FAA completed transitioning the remaining Part 121 air carrier 

inspection offices to ATOS at the end of 2007. However, at the time of our 

review, effective implementation of ATOS was hindered due to inspectors'

frustrations with adapting ATOS principles to smaller carrier operations, citing 

problems with broad or redundant inspection checklist questions, air carrier 

staffing limitations, and insufficient data to support the ATOS “data-driven” 

approach. Our analysis shows that a contributing problem may be long gaps—

3 years or more—between when inspectors received initial system safety 

training and when they began actually using ATOS. While these inspection 

offices have not reached the end of their first 5-year inspection interval, these 

issues warrant attention to avoid overdue design assessments in the future.

Weaknesses in FAA’s current ATOS approach hinder its ability to effectively 

target inspector resources to the areas of greatest need. We are making 

recommendations to FAA to improve its data, training, and risk assessment 

processes for ATOS. 

BACKGROUND 

In October 1998, FAA launched its data-driven, risk-based Air Transportation 

Oversight System at the 10 Part 121 air carriers6

• Design assessments are the most important ATOS function because safety is 

the outcome of a properly designed system. Inspectors evaluate air carriers’

that transported the most 

passengers. Today, FAA inspectors use ATOS to conduct surveillance of air 

carrier maintenance and operations at 94 Part 121 U.S. air carriers. Under the 

ATOS concept, FAA inspectors apply system safety principles and use data 

analysis to focus their inspections on areas that pose the greatest risk and identify 

potential problems before accidents occur. ATOS also permits inspectors to shift 

the focus of their inspections in response to changing conditions within air 

carriers’ operations. ATOS has three primary functions: assessments of air carrier 

system design, assessments of air carrier system performance, and risk 

management. All three of these functions require decisions by FAA inspectors 

and managers; therefore, ATOS is considered a decision support system.

 6 The initial group of ATOS air carriers included: Alaska Airlines, America West Airlines, American Airlines, 

Continental Airlines, Delta Air Lines, Northwest Airlines, Southwest Airlines, Trans World Airlines, United Airlines 

and US Airways.
4

policies and procedures (typically through reviews of air carrier manuals) to 

determine if their operating systems comply with safety regulations and 

standards. Design assessments are required every 5 years. 

• Performance assessments confirm that air carriers’ operating systems produce 

intended results. Inspectors determine whether air carriers are following their 

FAA-approved procedures and that those procedures and operating systems are 

working as intended. Performance evaluations are conducted at prescribed

intervals depending on the criticality level assigned to each air carrier program. 

Programs considered to be high-criticality7

• Risk management is the ATOS function to identify and control unsafe 

situations. Inspectors examine air carrier processes dealing with hazards and 

associated risks that are subject to regulatory control (e.g., enforcement actions 

and rulemaking). FAA uses these analyses as a basis to target resources 

towards the most at-risk programs. Risk is determined by analyzing 

consequences relative to likelihood and severity and the combined effects are 

assessed to determine priorities when multiple risks are identified. 

are inspected every 6 months, 

medium criticality every 12 months, and low-criticality programs every 

36 months. 

FAA HAS NOT COMPLETED TIMELY ATOS INSPECTIONS OF AIR 

CARRIERS’ POLICIES AND PROCEDURES FOR KEY 

MAINTENANCE SYSTEMS 

Overdue ATOS inspections have been a longstanding challenge for FAA in 

implementing its risk-based oversight system at major U.S. air carriers. Over an 

8-year period, inspectors at all 8 major air carrier inspection offices in our review 

did not complete 207 key inspections of carriers’ maintenance policies and 

procedures on time. This is despite changes FAA has made to ATOS over the last 

10 years that actually decreased the number of maintenance programs inspectors 

were required to review and increased the intervals between inspections. 

Confusion over when design assessments should be completed was one reason 

cited by multiple inspectors for missing inspection intervals. Additionally, FAA 

Headquarters is not using available inspection status data to hold these offices 

accountable for overdue ATOS inspections. In our view, tracking all overdue 

inspections and using available data would assist FAA in achieving more timely 

and targeted ATOS inspections. 

 7 FAA defines high-criticality maintenance programs as those programs where the likelihood of a system failure could 

lead to an unsafe condition. Conversely, medium- and low-criticality maintenance programs are those less likely to 

produce an unsafe condition if they fail.
5

FAA Inspection Offices for Major Air Carriers Missed ATOS Design 

Assessments 

From FY 2002 to FY 2009, all eight FAA major air carrier inspection offices we 

reviewed did not complete systemic ATOS reviews of air carrier maintenance 

policies and procedures (i.e., design assessments) on time. As shown in figure 1, 

four of the eight FAA inspection offices completed less than 50 percent of their 

inspection workload at the required interval; the US Airways inspection office 

completed slightly less than one-quarter of its workload on time.8

Figure 1. Percentage of Design Assessments Not Completed Within

Prescribed Intervals

 As a result, any 

risks in the air carrier systems would remain “unknown” until inspections are 

completed.

41

36

30 29

26 24

12

9

54 53 55

48

54 53 52 53

0

10

20

30

40

50

60

No. of Maintenance Programs*

FAA Inspection Offices

No. Not Completed No. of Programs

76% 68% 55% 60% 48% 45% 23% 17%

 Source: OIG analysis of FAA data, FY 2002 – FY 2009

*Due to the unique operations of each air carrier, not all air carriers have the same type or number of maintenance 

programs; therefore, not all will apply. 

As shown in table 1 below, the most commonly missed inspections of 

maintenance programs included reviews of air carrier manuals and training 

requirements for Required Inspection Items (RII).9

 8 FAA office managers stated they had delayed completing these inspections because the air carrier was merging with 

America West Airlines, and they believed it would be more efficient to wait and review the maintenance programs 

once the merger was completed. 9 RII are maintenance tasks that, if not properly performed or if done with improper parts or materials, could result in a 

failure, malfunction, or defect that would endanger the continued safe flight and landing of aircraft.
6

Table 1. Common Overdue Inspections at Eight FAA Offices 

Maintenance Program Description

No. of Offices with 

Overdue Inspections for 

this Maintenance Program

Required Inspection Item Training Requirements 7

Availability (Manuals) 6

Manual Currency 6

Supplemental Operations Manual Requirements 6

Chief Inspector * 6

Director of Maintenance * 6

Parts/Material Control/Suspected Unapproved Parts 6

Engineering/Major Repairs and Alterations 4

Airworthiness Directive Management 3

 Source: OIG analysis of FAA data, FY 2002 – FY 2009

*These inspections are used to validate the qualifications of the individuals holding these positions. 

Two of the maintenance programs shown in table 1—Airworthiness Directive 

Management and Engineering/Major Repairs and Alterations—are considered 

high-criticality maintenance programs. Inspectors for five inspection offices 

identified increased risk levels (i.e., meaning vulnerabilities in these programs 

could negatively impact safety) in these programs during the time when they 

should have been inspected; however, at least 5 to 6 years had lapsed between 

inspections. 

Principal inspectors stated that they missed inspection intervals due to confusion 

over FAA’s guidance on when they should complete ATOS design assessments. 

According to these inspectors, the guidance only “suggested” a 5-year inspection 

cycle for this type of assessment. While this may have been the case when FAA 

issued guidance in 2001, it reissued the guidance in July 2007, in part, to clarify 

inspection requirements. The revised guidance explicitly stated that these 

assessments must be completed every 5 years. Our review showed that even after 

FAA clarified inspection interval requirements, inspections were still not 

completed on time. 

FAA principal inspectors for Alaska Airlines informed us that because their office 

was transitioning to a newer version of ATOS in 2008, FAA Headquarters had 

permitted them to “reset the clock” on inspections nearing the 5-year time 

requirement. At that time, 10 design assessments were nearing their due dates and 

would be considered late under existing guidelines, if not completed. However, 

inspectors were later told to “hurry up and finish” any outstanding assessments. 

These inspectors stated that they believed FAA reversed its position due to the 

scrutiny that overdue ATOS inspections had received during congressional 

hearings in April 2008. These hearings likely served as a trigger to inspectors to 

complete overdue inspections. For example, after the hearings, the 8 major air 

carriers we reviewed completed 188 of 215 inspections (87 percent) that were 

overdue during FY 2002 through FY 2009. 
7

Smaller Air Carrier Inspection Offices Also Face Overdue ATOS 

Design Assessments 

In December 2007, FAA completed transitioning the remaining Part 121 

inspection offices to ATOS oversight. Because FAA guidance requires design 

assessments to be completed every 5 years, the 12 smaller inspection offices in our 

review still have time to complete these assessments. However, at the time of our 

review, only 2 of the 12 FAA inspection offices had completed 50 percent or more 

of their inspections. 

Inspectors for small air carrier offices face challenges in completing their assigned 

ATOS design assessments. This is due, in part, to the fewer number of inspectors 

assigned to oversee these carriers and the distance from their offices to air carrier 

headquarters and main maintenance bases. Inspectors assigned to small air carrier 

offices need more time to complete inspections than their larger air carrier 

counterparts because, while they usually have fewer aircraft to inspect and fewer 

inspectors on staff, they have nearly the same number of maintenance programs as 

large carriers. Table 2 shows the relationship of numbers of inspectors to the 

number of maintenance programs and size of air carrier fleet.

Table 2. Relationship of the Number of FAA Inspectors to Maintenance 

Programs and Aircraft Fleet Size

No. of Inspectors No. of Maint. Programs10 No. of Aircraft

Small Air Carriers 3 to 9 42 to 51 1 to 83

Major Air Carriers 15 to 32 48 to 55 117 to 750

Source: OIG analysis of FAA data

Also, we identified four inspection offices during our review that were either not 

located near the carriers’ headquarters, main maintenance base, or even in the 

same country where the carrier conducts most of its operations. These inspectors 

expressed concerns about their inability to have timely access to observe aircraft 

maintenance operations. For example, FAA inspectors for Executive Airlines 

must travel from their office in Miami, Florida, to San Juan, Puerto Rico—where 

the aircraft are based—to perform inspections of the air carrier’s headquarters and 

maintenance operations. Conversely, inspectors for major air carriers are often 

located near air carrier headquarters and major hubs around the country, allowing 

them easy access to review air carrier maintenance records and observe aircraft 

maintenance operations. 

 10 By March 2010, FAA completed action to eliminate or consolidate 27 air carrier maintenance inspection programs; 

therefore, the number of authorized programs is proportionately less now for each air carrier than during our review.
8

FAA Headquarters Does Not Track Overdue and Unassigned ATOS 

Inspections

FAA recently implemented a quarterly report to monitor field office inspections, 

but the report does not track overdue inspections that remain unassigned to an 

inspector or provide trend analyses of offices with habitually late inspections. 

Since July 2008, the Flight Standards Certification and Surveillance Division 

Manager has been sending these quarterly inspection status reports—commonly 

referred to as the Quarterly ADI11 Completion Report—to regional managers. 

FAA initiated this process in response to our June 2008 report12

Inspections are automatically scheduled in ATOS based on intervals established 

within the system, and office managers assign inspectors to complete them. We 

examined FAA’s quarterly reports from June 2008 through June 2009 and 

identified 237 scheduled inspections

in which we 

recommended that FAA implement a process to monitor field office inspections 

and alert local, regional, and Headquarters management of overdue inspections. 

Although it appears that FAA Headquarters has developed this report to monitor 

field inspections, the report does not include a cumulative data roll-up that could 

be useful in identifying problem offices where inspections are habitually late. As 

a result, Headquarters officials do not readily know if uncompleted inspections—

whether assigned or unassigned—in that quarter would be scheduled and 

completed in the following quarters. 

13 that were left unassigned14 and 

uncompleted. However, FAA did not use the Quarterly ADI Completion Report 

to track any of these to ensure they would be rescheduled and completed. For 

example, our review of ATOS data disclosed 11 inspections that were at least 

90 days past due but not rescheduled. In other instances, inspection offices did 

reschedule unassigned inspections; however, as illustrated in table 3 below, 

projected completion dates for these inspections will be up to 4 years past the 

original due date. Inspectors had previously identified four of these programs as 

having increased levels of risk, meaning vulnerabilities in these programs could 

negatively impact safety. 

 11 ADI: Action, Determination, and Implementation is a process in ATOS designed to permit a principal inspector or 

management official to collect and analyze inspection data in order to make decisions to mitigate risks found during 

inspections of air carriers’ operating programs. 12 OIG Report Number AV-2008-057, “Review of FAA’s Oversight of Airlines and Use of Regulatory Partnership 

Programs,” June 30, 2008. 13 These are Element Performance Inspections designed to assess whether critical maintenance systems are performing 

as intended. 14 Unassigned inspections are scheduled in the ATOS automated system, but managers have not committed inspector 

resources to complete them.
9

Table 3. Unassigned and Overdue Inspections 

With Projected Completion Dates

Inspection Program Original Due 

Date

Revised Due Date Time Past 

Due

Engineering/Major 

Repairs and Alterations*

Dec. 31, 2008 Dec. 31, 2012 4 years

Line Stations Sept. 30, 2008 Sept. 30, 2012 4 years

Other Personnel with Operational 

Control*

Sept. 30, 2008 Dec. 31, 2011 3 years, 

3 months

Training of Flight Attendants* Dec. 31, 2008 June 30, 2010 18 months

Major Repairs and Alterations 

Records*

Sept. 30, 2008 Mar. 31, 2010 18 months

Deicing Program Dec. 31, 2008 Mar. 31, 2010 15 months

Source: FAA ATOS repository, June 30, 2008 – June 30, 2009

*Inspection programs that were identified with elevated risk.

Unassigned inspections pose a significant problem for FAA because managers 

cited a lack of inspector resources needed to complete them. Our analysis of 

ATOS data showed that a lack of inspector staffing was cited in 165 of the 

237 (70 percent) unassigned inspections over a 1-year period. 

FAA Headquarters officials stated that not all scheduled ATOS inspections will be 

completed at the required interval and that, in their opinion, it is not practical or 

desirable to complete all inspections just for the sake of completing inspections. 

Therefore, Headquarters officials do not hold local inspection offices accountable 

for completing unassigned inspections because that would impede the time 

inspectors need to perform quality inspections for areas that pose greater risk. 

While we agree that higher-risk air carrier programs should be inspected ahead of 

lower-risk programs, inspectors still did not complete some of the unassigned 

inspections where they had found increased risk levels. All air carrier programs 

should be inspected at regular intervals to mitigate risk before system failure 

occurs. 

FAA Inspections Were Late Despite Changes to ATOS That 

Decreased Inspectors’ Workload

FAA has not met its ATOS inspection intervals despite cutting the number of 

maintenance programs requiring inspection nearly in half—from 59 to 31—over 

the last 10 years. According to FAA, these changes were intended, in part, to 

reduce redundancy in the system and more efficiently target inspector resources. 

In FY 1999, ATOS required local inspection offices to complete a minimum of 

151 maintenance inspections annually; now, they only have to complete 

37 inspections (as shown in figure 2 below).15

 15 These numbers may be less based on the number of applicable maintenance programs at an air carrier.

 Over time, this resulted in a 

75-percent reduction in workload for FAA inspectors. It also resulted in air 
10

carriers receiving fewer inspections during a time when the air carrier industry was 

facing record financial losses and making unprecedented changes to their 

operations (e.g., retiring and storing aircraft, making personnel cuts, and closing 

maintenance facilities) to regain profitability. An air carrier program, such as 

Airworthiness Directive Management, which had received a minimum of one 

quarterly inspection when ATOS was first implemented, now receives only two 

inspections annually, primarily as a result of FAA’s revisions to ATOS, not 

because of reduced risks. 

Figure 2. Reduction in the Minimum Number of Required Element 

Performance Inspections — FY 1999 - FY 2010

Source: FAA ATOS data

* Number rounded.

While fewer maintenance programs—and therefore fewer inspections—will 

reduce the likelihood of missed inspections, FAA’s changes to ATOS may not 

necessarily result in more effective risk assessment processes.

Specifically, FAA did not base its decision to reduce, eliminate, or consolidate 

maintenance inspections on sound analytical support: 

• In 2003, FAA convened a workgroup to review the risk assessment process in 

ATOS. The workgroup reviewed the criticality level (i.e., the likelihood that a 

system failure could lead to an unsafe condition) assigned to each air carrier 

maintenance program and recommended that 15 of the 25 high-criticality 

programs be downgraded to medium- and low-criticality levels. However, the 
11

workgroup did not consider whether past inspection reports revealed any 

problems that would reverse their decision to make these changes. 

• In March 2010, FAA again revised ATOS. In this revision, FAA eliminated or 

consolidated 27 maintenance programs into the remaining 31 programs, in part, 

to remove redundancy within ATOS. Similar to 2003, FAA did not analyze 

historical ATOS data to support its decision to make these changes. Instead, 

FAA relied on concerns inspectors voiced during their visits to local inspection 

offices (e.g., ATOS was difficult to use). 

FAA views its latest decision to reduce the number of maintenance programs as a 

means of producing a more efficient oversight system by eliminating overlapping 

inspection areas. While this action has merit, it also seemed to focus on

maintenance programs that had habitually late inspections, not just those with 

overlapping areas. For example, four16

ATOS INSPECTIONS WERE NOT EFFECTIVE AT TESTING AND 

VALIDATING THAT CRITICAL MAINTENANCE SYSTEMS WERE 

WORKING PROPERLY 

of the most commonly missed inspections 

(as previously shown in table 1) were eliminated or combined with other 

inspections based on FAA’s revision to ATOS. As a result, these maintenance 

programs no longer exist as separate programs. 

FAA inspectors did not complete ATOS performance assessments on time at all 

8 major and 12 smaller air carrier inspection offices we reviewed despite having 

fewer maintenance programs to inspect. Performance assessments are intended to 

confirm that air carrier systems operate as required by Federal Aviation 

Regulations. Yet, over a 5-year period, inspectors did not complete 

576 inspections at the required intervals. Additionally, ATOS was envisioned to 

be a risk-based oversight system, but we found the risk assessment process—the 

basis for prioritizing inspections for timely completion—does not give priority to 

maintenance programs where FAA inspectors found increased risk. Also, 

inspectors we interviewed were not analyzing voluntary disclosure data (i.e., 

maintenance errors that air carriers self-report) or industry events that could 

impact a carrier’s performance and stability. Voluntary disclosure data and 

changes in the airline industry are important indicators of whether air carriers are 

properly maintaining their aircraft during times of economic downturn. 

 16 Required Inspection Item (RII) Training Requirements, Availability (Manuals), Chief Inspector, and Supplemental 

Operations Manual Requirements.
12

FAA Inspectors for Major Air Carriers Missed Timelines for 

Conducting ATOS Performance Assessments 

FAA inspectors are required to conduct ATOS performance assessments of air 

carriers’ policies and procedures at specified time intervals. Performance 

assessments have different inspection intervals than design assessments. For 

example, depending on the criticality (i.e., the likelihood that a system failure 

could lead to an unsafe condition) of a maintenance program, performance 

assessments are conducted at 6, 12, or 36-month intervals. 

Between FY 2005 and FY 2009, inspectors at 8 major air carrier inspection offices 

did not complete 340 performance assessments within FAA’s required time 

intervals to test whether air carriers’ policies and procedures were operating 

effectively. Of the 340 missed inspections, 180 were high-criticality maintenance 

program inspections; the remaining 160 were medium-criticality inspections. As 

shown in figure 3, the Southwest Airlines inspection office missed the most (57) 

inspections at the required time intervals while the Delta Air Lines inspection 

office missed the least (26). 

Figure 3. Number of Performance Assessments Not Completed at 

Required Intervals for Major Carriers — FY 2005 - FY 2009

32

25 23

20

23

26

20

11

25

26

25 28

21

13

7 15

0

10

20

30

40

50

60

No. of Inspections Not Completed

Air Carrier Inspection Offices

High Criticality Programs Medium Criticality Programs 

Source: OIG analysis of FAA data, FY 2005 – FY 2009

Examples of high-criticality inspections missed most often at major air carriers 

included the following: 
13

• 30 Continuing Analysis and Surveillance System (CASS) inspections

• 28 Aircraft Reliability Program inspections

• 21 AD Management inspections 

All air carrier maintenance programs are important, but these three programs are 

especially key because they are designed to: (1) monitor and analyze the 

performance of air carrier inspection and maintenance programs, (2) monitor 

failure rates of aircraft components, and (3) alert air carriers to potentially unsafe 

manufacturing defects in aircraft parts and components. Yet, our work has 

continually found that FAA has problems completing inspections of these 

programs on time. For example, between 2005 and 2007, FAA inspectors for 

American Airlines did not inspect the carrier’s CASS and Aircraft Reliability 

programs.17 Also, between 2006 and 2007, inspectors for Southwest Airlines did 

not inspect the AD Management program for almost 18 months.18

Small Air Carrier Inspection Offices Also Missed Timelines for 

Conducting ATOS Performance Assessments 

 On average, the 

number of days in which inspections were completed beyond their original due 

dates was 54 days for high-criticality and 96 days for medium-criticality programs. 

Twelve small air carrier FAA inspection offices we visited also did not meet 

inspection baseline intervals for 236 inspections.19

 17 OIG Report Number AV-2010-042, “FAA’s Oversight of American Airlines’ Maintenance Programs,” 

February 16, 2010. 

 These offices are required to 

complete performance assessments at the same time intervals as the major air 

carrier inspection offices. Of the 236 overdue inspections, 116 were highcriticality maintenance program inspections and the remaining 120 were mediumcriticality inspections. As shown in figure 4 below, 7 of the 12 inspection offices 

did not complete 20 or more inspections within the required intervals. 

18 OIG Report Number AV-2008-057, “Review of FAA’s Safety Oversight of Airlines and Use of Regulatory 

Partnership Programs,” June 30, 2008. 19 Because these offices did not transition to ATOS at the same time, our analysis was limited to reviewing available 

data in the ATOS data repository from the time of each office’s transition through the end of FY 2009.
14

Figure 4. Number of Performance Assessments Not Completed at Required 

Intervals for Small Carriers — FY 2007 - FY 2009

15

8

13 13

10

12

9

10

9 9

5

3

11 18

12

11

13 10

11

9 10

9

4

2

0

5

10

15

20

25

30

No. of Inspections Not Completed

Air Carrier Inspection Offices

High Criticality Programs Medium Criticality Programs 

 Source: OIG analysis of FAA data, FY 2007 – FY 2009

* Transitioned to ATOS in FY 2008

Examples of high-criticality inspections missed most often at small carriers 

included the following:

• 20 Engineering/Major Repairs and Alterations Program inspections

• 15 Maintenance Control Program inspections

• 15 CASS inspections 

• 12 Aircraft Airworthiness inspections 

Because these are high-criticality programs, FAA guidance requires that they be 

inspected every 6 months. Two local inspection offices that transitioned to ATOS 

at the end of FY 2007 did not complete timely inspections of the Aircraft 

Airworthiness and Engineering/Major Repairs and Alterations programs. When 

these two programs were finally inspected—12 and 18 months later, 

respectively—inspectors identified deficiencies. For example, at one air carrier, 

inspectors discovered that the carrier had not completed an audit of the 

maintenance records department to determine whether existing processes were 

effectively identifying problems and providing corrective actions. Until these 

overdue inspections were completed, the risks in these programs could only be 

described as “unknown” because inspections were not performed at the required 

intervals to monitor any changes. 
15

ATOS Design Limitations Hinder Timely and Targeted Inspections 

The ATOS system design prompts inspectors to place priority on programs that 

are not necessarily high risk.

20

ATOS is designed to assign high-criticality programs three times the baseline risk 

value of low-criticality programs. As a result, these programs will normally 

receive first inspection priority. For example, if an air carrier’s Airworthiness 

Directive program is operating normally, it will be assigned a baseline risk value 

of 84 while an air carrier’s maintenance manual program, with no risk, will 

receive a baseline risk value of 28. As a low-criticality program, an air carrier’s 

General Maintenance Manual system, by itself, will not result in an unsafe 

condition on an aircraft. However, as the foundation for an effective aircraft 

maintenance program, without accurate manuals, maintenance errors can occur. 

We agree that high-criticality programs warrant vigilant FAA oversight, but they 

may not always present the highest risk to safe air carrier operations if inspectors 

have not identified any hazards in the programs. 

This is because ATOS disproportionately weights

maintenance programs designated as high-criticality over lower-criticality 

maintenance programs even though inspectors have identified deficiencies in the

lower-criticality programs. As a result, lower-criticality maintenance programs 

with increased risk may not receive priority for inspection at required intervals.

Table 4 below shows how considering increased risk along with program 

criticality would impact inspections at one air carrier. As shown in the table, all 

but 1 of this carrier’s top 10 maintenance programs listed in the column “Increased 

Risk Score” are high-criticality programs; therefore, they will receive priority for 

inspection even though 3 of them show no increased risk above their baseline 

score (84). However, the maintenance manual program nearly doubled in risk, but 

ranks only 35th in priority.

 20 Maintenance programs can be designated as high-criticality (likely to produce unsafe results if the program fails), but 

may not be considered “high risk” if FAA inspectors have not found problems with the program.
16

Table 4. Example of an Air Carrier’s Maintenance Programs Prioritized 

According to Highest Risk Score

Maintenance Program

Criticality 

Designation

Baseline

Risk Score

Increased

Risk Score *

Priority 

Ranking

Aircraft Airworthiness H 84 141 1

AD Management H 84 123 2

CASS H 84 117 3

Line Stations H 84 102 4

Maintenance Control H 84 96 5

Outsource Organization H 84 93 6

Weight & Balance M 56 86 7

RII** H 84 84 8

Engineering / Major Repairs & 

Alterations** H 84 84 9

Cargo Handling Equipment, & 

Appliances** H 84 84 10

General Maintenance Manual L 28 53 35

Source: FAA ATOS data repository

*Increased risk scores are derived from inspectors’ analyses of air carrier data and inspection reports, which indicate that 

the individual maintenance program is experiencing problems.

**These programs have no increased risk. 

More emphasis on prioritizing programs with increased risk, regardless of the 

criticality designation, would allow FAA to better target inspector resources. 

Table 5 shows an example where calculating the percentage of increased risk 

should ensure that low-criticality maintenance programs with higher risks will 

receive the same consideration for inspector resources as high-criticality programs. 

Table 5. Example of One Air Carrier’s Maintenance Programs Prioritized 

According to Percentage of Increased Risk 

Maintenance 

Program

Criticality 

Designation

Baseline 

Risk 

Score

Increased 

Risk 

Score

Initial 

Priority 

Ranking

% of 

Increased 

Risk to 

Baseline 

New 

Priority 

Ranking

General 

Maintenance 

Manual L 28 53 35 89% 1

Aircraft 

Airworthiness H 84 141 1 68% 2

Weight & Balance M 56 86 7 54% 3

AD Management H 84 123 2 46% 4

CASS H 84 117 3 39% 5

Maintenance 

Program M 56 70 11 25% 6

Line Stations H 84 102 4 21% 7

Maintenance 

Facility/Main 

Maintenance Base M 56 66 12 18% 8

MIS Reports L 28 33 36 18% 9

Service Difficulty 

Reports L 28 33 38 18% 10

Source: FAA ATOS data repository
17

An internal FAA report issued in November 200821

FAA Inspectors Do Not Analyze All Available Data Sources or 

Relevant Industry Events To Ensure Accurate Risk Assessments 

stated that surveillance 

planning is mainly targeted to high-criticality programs and not enough 

inspections are directed toward low-criticality programs. FAA’s report concluded 

that the ATOS system would miss low-criticality programs with high probability 

of risk and FAA should review how inspection offices target their surveillance 

planning to determine if ATOS should be modified to increase the frequency of 

inspections for low-criticality maintenance programs. We agree with this 

assessment, and FAA officials stated that the Agency is working to address this 

issue.

Inspectors’ risk assessment plans do not consider all available safety data; rather,

they place more emphasis on the results of their surveillance activities and 

enforcement actions and do not fully analyze or trend data from the Safety 

Performance Analysis System (SPAS) database (e.g., hotline complaints and 

Service Difficulty Reports) or the Voluntary Disclosure Reporting Program 

(VDRP).22

None of the FAA inspection offices we visited performed ongoing trend analyses 

of voluntary disclosure reports to identify potential repeat violations. Disclosure 

reports we reviewed indicated inspectors were more concerned about whether the 

number of disclosures had increased rather than trending the data to look for 

repetitive submissions. Trending air carrier self-disclosure data is important to 

ensure that corrective actions are working and prevent abuse of the system. Such 

trend analyses would also benefit the risk assessment process because repetitive 

self-disclosures indicate that an air carrier is unable to correct identified 

maintenance deficiencies and would trigger inspectors to adjust the risk level of 

that maintenance program so that it receives an adequate level of inspection. 

Disclosure data, which are not typically obtained through normal 

surveillance means, could serve as a valuable source of information to assist 

inspectors in monitoring the safety compliance and trends of their assigned 

carriers. For example, repetitive disclosures of mechanic errors at an air carrier 

should indicate to FAA that problems still exist in the carrier’s maintenance 

training program.

FAA recently took steps to ensure inspectors considered VDRP data as part of 

their normal surveillance planning by directing inspectors to review national trend 

analyses of VDRP data and identify potential safety risks. This is a good first step 

to ensure that self-disclosure data will receive heightened consideration. 

 21 FAA internal report dated November 2008. Availability determined by 5 USC §552. 22 The VDRP provides air carriers with the opportunity to notify (self-disclose) FAA of known safety violations 

without fear of legal enforcement action.
18

However, FAA will need to closely monitor whether inspectors consistently apply 

the results of their reviews when developing their risk assessment plans. 

In addition, inspectors do not fully consider events occurring in the air carrier 

industry that could impact maintenance programs within their assigned carrier. 

For example, while Delta Air Lines was merging with Northwest Airlines in 2008, 

FAA inspectors for both offices did not document potential concerns in ATOS to 

acknowledge these events. According to FAA guidance, identification of risks in 

areas such as “Merger or Takeover,” “Reduction in Workforce,” changes in 

“Financial Conditions,” or “Labor-Management Relations” is likely to impact the 

carriers’ ability to effectively manage their workforce. However, FAA inspectors’

risk assessments of Delta and Northwest showed little to no increase for potential 

risk in these areas during the merger. 

We also found that when inspectors did document risks, their risk assessment 

plans frequently lacked substantive supporting statements to justify why inspectors 

raised or lowered risk levels for each of the maintenance programs. ATOS 

guidance recommends that inspectors annotate reasons for adjusting risk levels; 

however, we examined risk assessments for eight major air carrier inspection 

offices and found inconsistencies. For example, inspectors at the Continental 

Airlines inspection office raised the level of risk for the Aircraft Airworthiness 

maintenance program based on vague justifications (e.g., “substantial number of 

human factors performance errors” and “large number of self disclosures”) that 

did not identify the specific safety concerns that existed at the air carrier at the 

time of the risk assessment. Additionally, inspectors at the Southwest Airlines 

inspection office initially identified increased levels of risk for 10 ATOS 

maintenance programs but did not document reasons to support their actions. 

However, more than 5 months later, inspectors realized that they overstated the 

risk; yet, these programs received higher priority to be inspected first. 

FAA stated that it is aware of the inconsistencies in how its field inspection offices 

assess risk and of the need to reevaluate whether all ATOS risk indicators should 

have equal weighting for each maintenance program. While this seems to be a 

practical approach to ensure that ATOS is truly a risk-based system, it could 

reduce the flexibility of ATOS to adapt to ever-changing conditions, especially 

since all air carriers do not operate in the same environment. 

FAA INSPECTORS FOR SMALLER PART 121 AIR CARRIERS DID 

NOT EFFECTIVELY TRANSITION TO ATOS

FAA completed transitioning all of its Part 121 inspection offices to ATOS at the 

end of 2007; however, some inspectors for smaller air carriers expressed concerns 

with adapting ATOS to smaller carrier operations. Our work shows one reason for 
19

this difficulty may be gaps between when inspectors received initial system safety 

training and their actual use of ATOS. Additionally, our discussions with 

supervisory inspectors indicate that they do not fully support or accept the ATOS 

concept due, in part, to the inspection checklist design. Our prior reviews and 

ongoing work, as well as an FAA internal review, have identified lack of 

management support for these initiatives as an obstacle to successful ATOS 

implementation.

Inspectors for Smaller Air Carriers Cited Frustration and Confusion 

with ATOS

Managers and inspectors at 12 FAA inspection offices for smaller air carriers that 

recently transitioned to ATOS cited concerns with the system’s design such as 

inspection checklist questions, air carrier staffing limitations, confusion over how 

to record inspection findings, and insufficient data to effectively support the 

ATOS “data-driven” approach. 

• Inspector Frustration With ATOS Inspection Checklist Questions and 

Procedures: At the time of our review, managers and inspectors we spoke 

with felt that ATOS questions were hard to understand and redundant. These 

inspectors also stated concerns that a system safety approach to air carrier 

oversight that required more time spent in front of a computer compromised 

the time needed to conduct actual hands-on inspections. FAA inspectors have 

voiced this complaint to us since we first began reviewing ATOS 

implementation in 2000. In 2009, FAA implemented changes to ATOS data 

collection tools, which simplified and removed redundant inspection questions. 

FAA also commissioned a Time Allocation Study,23

• Small Air Carrier Staffing Limitations: Inspectors stated that ATOS was 

originally designed for oversight of larger air carriers that have access to fully 

staffed engineering and quality assurance departments. These carriers are 

better equipped to handle needed changes and respond to FAA inspection 

findings in a more timely fashion. Conversely, smaller air carriers typically 

have less staff—in some cases, one or two persons to handle an entire 

engineering program—to address required maintenance program changes. 

Therefore, smaller air carriers require more time to correct deficiencies 

identified during FAA inspections. As a result, air carriers could be operating 

unsafe aircraft or air carrier employees could be following deficient manual 

procedures while management is trying to make corrections to their programs. 

which showed that 

inspectors actually spent less time using computer automation (less than onethird of an inspector’s workweek was devoted to computer usage) than what 

was previously reported by inspectors.

 23 This report was prepared for FAA by Booz Allen Hamilton in August 2009; availability determined by 5 USC §552.
20

• Inspector Confusion Over How To Record Inspection Findings: Several 

inspectors expressed concerns that the process for recording inspection 

findings in ATOS is confusing and difficult to adapt to smaller carrier 

operations. For example, an air carrier providing seaplane passenger service 

between islands in the Caribbean is more prone to experiencing airframe 

corrosion due to long exposure to salt water than aircraft operating in other 

environments. This is unique to this air carrier, and inspectors stated there is 

no place in ATOS to accurately record these findings because the ATOS 

checklist questions are too broad. In another example, an air carrier providing 

service to its sports franchise customers frequently flies into airports where the 

carrier has no established company maintenance, contract facilities, or gate 

personnel. As a result, the air carrier’s mechanics routinely fly to the 

destination and provide on-call maintenance services, as needed. FAA 

inspectors expressed frustration with their ability to schedule and perform 

comprehensive inspections of this carrier due to the “on-demand” nature of the 

carrier’s operations. Also, because these carriers operate in non-traditional 

environments, inspectors found it difficult to associate their inspection findings 

to structured ATOS checklists. 

• Data Limitations: Inspectors did not know if ATOS was actually working for 

the carriers they oversee. Inspectors overseeing air carriers that operate only 

one or two aircraft are not seeing the benefits of a risk-based oversight system 

because there are not enough data to fuel the “data-driven” nature of ATOS. 

The uniqueness of small air carrier operations raises questions about whether 

inspectors can adjust ATOS to the size and complexity of air carrier operations. 

Addressing these questions will be particularly important if FAA plans to expand 

ATOS as its oversight system. According to inspectors, FAA Headquarters was 

aware of the ATOS issues with these air carrier operations, and some of them were 

told that a scaled-down version of ATOS—or “ATOS Lite” as the inspectors 

referred to it—was in the works to address such issues. However, FAA contends 

that ATOS was originally designed to be scalable to any size air carrier operation. 

Frustration With Training and Lack of Management Support Hinder 

Acceptance of ATOS at Smaller Inspection Offices

Inspectors expressed frustration with the gap between the time in which they 

received system safety training and when they actually began using ATOS to 

oversee their assigned air carrier. We surveyed 63 airworthiness inspectors24

 24 Airworthiness includes both maintenance and avionics specialties.

at 

the 12 small inspection offices we reviewed and found that they received initial 

system safety training, but in some cases, this training occurred as much as 6 years 

before they actually began performing inspections. For those inspectors who had 
21

a gap of 3 or more years, nearly 70 percent reported being unable to recall and 

apply system safety concepts to answer ATOS inspection questions. 

Understanding and applying system safety principles is key to ensure that air 

carriers’ maintenance programs work effectively and that ATOS contains accurate 

data. 

In 2008, FAA commissioned an internal report to gauge the effectiveness of the 

transition from existing oversight systems to a newer ATOS version 1.2.25

• Inspectors have a partial understanding of system safety theory and principles.

 The 

report’s findings reiterate our concerns. For example, specific issues regarding 

FAA’s transition disclosed in the report included the following:

• Inspectors do not understand ATOS inspection checklist questions. 

• Inspectors who transitioned to ATOS were generally more negative about the 

newer version of ATOS (1.2).

• Inspectors perceived that the new version does not adequately account for 

either air carrier size or scope of operations when planning for surveillance 

activities. 

This report also highlighted inconsistent ATOS data reviews across local 

inspection offices and regions. More importantly, the report found that training 

for the new ATOS version was poorly received, which negatively affected the 

inspectors’ acceptance of it. This complaint was exacerbated by the fact that 

FAA’s study found that the lack of management buy-in and supervisory support 

for new initiatives were impeding ATOS acceptance and success. We also 

reported this issue with management support in our 2002 audit of the initial ATOS 

implementation. 

The internal report also recommended the following actions to FAA:

• Enhance the workforce knowledge of system safety principles and their 

understanding of the ATOS inspection checklist questions.

• Evaluate the design of the new ATOS version as it relates to scalability across 

all Part 121 air carriers. 

• Develop resources for use by field office managers and supervisors that 

support their understanding of effective ways to demonstrate ATOS buy-in.

• Develop supplemental training for inspectors using the new ATOS version.

• Conduct additional research to further understand ATOS version

1.2 implementation at other offices. 

 25 This report was prepared for FAA by Booz Allen Hamilton in July 2008; availability determined by 5 USC §552.
22

FAA has taken initial steps to address some of these recommendations. For 

example, as of April 2010, FAA has developed and implemented ATOS 

interactive training courses for managers and inspectors to emphasize the 

relationship between system safety principles and ATOS. More work remains to 

fully address the recommendations. 

CONCLUSION

The United States has traditionally operated the safest air carrier system in the 

world, and risk-based oversight of air carriers is critical to ensure that the 

American public continues to have access to safe and reliable air transportation. 

FAA has faced many challenges in implementing ATOS since its inception, 

including the expansion of ATOS to all Part 121 air carriers. While FAA is 

making progress in addressing these challenges, a number of critical actions, such 

as ensuring accountability for overdue inspections and accurate and consistent risk 

assessment, are needed to ensure ATOS operates as originally intended. Without 

these improvements, FAA lessens its ability to effectively oversee the National 

Airspace System. 

RECOMMENDATIONS 

We recommend that FAA:

1. Redesign the Quarterly ADI Completion report to include cumulative roll-up 

data from previous quarters and conduct trend analyses that could be used to

hold regional and local inspection offices accountable for scheduling 

uncompleted inspections. 

2. Develop procedures to document justification for significant changes to 

ATOS (i.e., planned changes to alter the number of data collection tools or 

prescribed inspection time intervals).

3. Redesign the current risk assessment process within ATOS so that it 

appropriately prioritizes maintenance programs with the greatest percentage 

of increased risk (regardless of criticality level) for inspector resources. 

4. Provide training to inspectors to help them more accurately interpret data 

from all available sources (i.e., self-disclosure data, ATOS Data Packages, 

events occurring in the aviation industry) and apply the results more 

consistently when planning risk assessments. 

5. Evaluate ATOS to determine if it is designed to accurately record inspection 

findings unique to smaller air carrier operations.
23

6. Evaluate whether ATOS is scalable across all Part 121 air carriers.

7. Expedite enhancement of ATOS training methods (as identified in its 2008 

internal report) to assist inspectors in understanding how to use ATOS data 

collection tools and increase their proficiency in using ATOS. 

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 

RESPONSE 

We discussed the results of our review with FAA’s Deputy Director of Flight 

Standards Field Operations on July 13, 2010, and provided FAA with our draft 

report on September 23, 2010. We received the Agency’s formal response on 

October 20, 2010, which is included in its entirety as the appendix to this report. 

FAA concurred with four of our seven recommendations and partially concurred 

with three. FAA agreed to implement actions we recommended to evaluate 

whether ATOS is scalable to all carriers; therefore, we are closing 

recommendation 6. FAA generally agreed with the remaining six 

recommendations, but did not propose actions that we consider sufficient to 

address our concerns. Therefore, we consider them as open and unresolved and 

request that FAA provide further information, as discussed below. 

Recommendation 1: FAA stated that it had modified ATOS with a ghost icon to 

indicate that an inspection was not completed. However, this modification still 

does not provide Headquarters with a process that could be used to hold 

local inspection offices accountable for scheduling uncompleted inspections. 

Therefore, we request that FAA clarify its response or provide an alternative 

course of action that would meet the intent of this recommendation. 

Recommendation 2: FAA stated that it has developed a Quality Management 

System (QMS) process to support its decision to revise ATOS Decision Collection 

Tools (DCT). However, during our audit, FAA officials never discussed 

developing a QMS process or how it would be used to support decisions to revise 

ATOS. Therefore, we request that FAA clarify how the QMS process will be used 

to document future changes to ATOS.

Recommendation 3: FAA stated that an ATOS process already exists that permits

principal inspectors to manually adjust inspection priorities and that our 

recommendation to redesign the current risk assessment process is unwarranted. 

However, this process is neither automated nor objective. Further, it does not use 

the percentage of increased risk to determine when a maintenance program should 

be inspected. Therefore, we request that FAA reconsider its response or provide 

an alternative course of action to satisfy the intent of this recommendation. 
24

Recommendation 4 and 7: FAA stated that it revised two ATOS training courses 

designed to provide newly assigned Part 121 inspectors and managers with indepth coverage of risk assessment processes. We found, however, that existing 

inspectors and managers were not using all available data sources when analyzing 

risk. Therefore, we request that FAA clarify its planned action to provide training 

to existing inspectors and managers.

Recommendation 5: FAA stated that it completed revisions to its ATOS DCTs in 

July 2010 after our audit work was completed. Because these revisions to ATOS 

are still relatively new, we request that FAA provide us with additional details on 

how it will determine whether these changes have been effective in improving the 

accuracy of recording inspection findings unique to smaller air carrier operations. 

ACTIONS REQUIRED 

We consider all recommendations—except for recommendation 6—as unresolved 

and open, pending additional information from FAA. In accordance with 

Department of Transportation 8000.1C, we request that FAA clarify its response

on actions it intends to take to fully address all recommendations along with 

estimated target completion dates. We would appreciate receiving this response 

within 30 calendar days. 

We appreciate the courtesies and cooperation of FAA representatives during this 

audit. If you have any questions concerning this report, please contact me at 

(202) 366-0500 or Tina B. Nysted, Program Director, at (404) 562-3770. 

#

cc: FAA Associate Administrator for Aviation Safety

 Director of Flight Standards

 Anthony Williams, AAE-001

Martin Gertel, M-100
25

Exhibit A. Objectives, Scope, and Methodology

EXHIBIT A. OBJECTIVES, SCOPE, AND METHODOLOGY 

Our audit objectives were to determine (1) whether FAA has completed timely 

ATOS inspections of air carriers’ policies and procedures for their most critical 

maintenance systems; (2) how effective ATOS performance inspections have been 

in testing and validating that these critical maintenance systems are working 

properly; and (3) how well FAA implemented ATOS for the remaining Part 121 

air carriers and what, if any, oversight challenges FAA inspection offices face. 

We conducted this audit between May 2008 and July 2010 in accordance with 

generally accepted government auditing standards. Those standards require that 

we plan and perform the audit to obtain sufficient, appropriate evidence to provide 

a reasonable basis for our findings and conclusions based on our audit objectives. 

We believe that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives.

We selected 20 FAA inspection offices for our review. Eight offices were selected 

because they have been under ATOS since FY 1999 and would provide the 

greatest amount of data for analysis. Also, these carriers and their affiliates are, in 

part, responsible for transporting more than 90 percent of passengers and air cargo 

in the United States. The remaining 12 inspection offices were chosen because 

they were new to using ATOS in the last 3 years, and some of these carriers 

participate in multiple code-sharing agreements with major air carriers. At the 

time of our review, there were 104 FAA Part 121 air carrier inspection offices that 

transitioned to ATOS. Exhibit B lists the entities we contacted or visited during 

our review.

We evaluated ATOS oversight and implementation at the national, regional, and 

local levels. To accomplish this, we interviewed officials from FAA Flight 

Standards Service to obtain information regarding ATOS implementation to 

determine the effectiveness of FAA’s national oversight of the ATOS program. 

We also contacted Division Managers for all eight FAA Regional Flight Standards 

Offices (Alaska, Central, Eastern, Great Lakes, Northwest Mountain, Southern, 

Southwest, and Western-Pacific) to obtain their input as to the usefulness of FAA 

Headquarters’ Quarterly ADI Completion Reports. 

We interviewed FAA Managers, Principal Airworthiness Inspectors, Aviation 

Safety Inspectors, and Operations Research Analysts to determine what processes, 

procedures, guidance, and data they use to conduct ATOS inspections and analyze 

data. Our review also included analyses of ATOS design and performance 

assessments obtained from FAA’s Safety Performance Analysis System and the 

ATOS data repository for FY 2002 through FY 2009 to determine the timeliness 

of completing inspections. For our review, we evaluated whether inspectors 

completed design assessments within the 5-year interval and performance 
26

Exhibit A. Objectives, Scope, and Methodology

assessments within the 6, 12, or 36-month interval requirements, as prescribed by 

FAA. Assessments that were not completed within the required intervals were 

considered overdue. 

For this analysis, we determined the date when an inspection of a maintenance 

program was completed and then, using FAA’s guidance for prescribed time 

intervals, we measured the time difference to the next time this same maintenance 

program was inspected. FAA revised the ATOS program in 2004, which included 

increased time intervals; therefore, we chose to begin our analysis of performance 

assessment data starting with FY 2005. Finally, we surveyed all 63 airworthiness 

inspectors from the 12 small inspection offices to obtain their personal 

impressions of system safety and ATOS.
27

Exhibit B. Activities Visited or Contacted

EXHIBIT B. ACTIVITIES VISITED OR CONTACTED 

FEDERAL AVIATION ADMINISTRATION

Headquarters:

Aviation Safety (AVS) Washington, DC

Flight Standards Service (AFS) Washington, DC

Regions*:

Alaska Region Anchorage, AK

Central Region Kansas City, MO

Eastern Region Jamaica, NY

Great Lakes Region Des Plaines, IL

Northwest Mountain Region Renton, WA

Southern Region College Park, GA

Southwest Region Fort Worth, TX

Western-Pacific Region Los Angeles, CA

Local Inspection Offices for: 

Aerodynamics Belleville, MI

Alaska Airlines SeaTac, WA

American Airlines Fort Worth, TX 

Chautauqua Airlines Plainfield, IN

Continental Airlines Houston, TX

Delta Air Lines College Park, GA

Executive Airlines Doral, FL

Falcon Air Doral, FL

Gulfstream Airlines Doral, FL

Lynx Aviation Doral, FL

Miami Air International Doral, FL

Northwest Airlines Bloomington, MN

Republic Airlines Plainfield, IN

Seaborne Virgin Island Doral, FL

Shuttle America Plainfield, IN

Southwest Airlines** Dallas, TX

*Contacted only to obtain clarifying data

**Did not visit; performed data analyses only
28

Exhibit B. Activities Visited or Contacted

Spirit Airlines Belleville, MI

United Airlines Daly City, CA

US Airways Coraopolis, PA

USA Jet Airlines Belleville, MI

Other FAA Offices:

Seattle Aircraft Certification Office Renton, WA

AIR CARRIERS*

Republic Airlines Indianapolis, IN

Shuttle America Indianapolis, IN

*Contacted only to obtain clarifying data
29

Exhibit C. Major Contributors to This Report

EXHIBIT C. MAJOR CONTRIBUTORS TO THIS REPORT 

Tina Nysted Program Director

Name Title

Kevin George Project Manager

Anne Longtin Senior Analyst-in-Charge

Galen Steele Senior Auditor

Jeannette McDonald Senior Analyst

Sara Gragg Analyst

Stefanie McCans Senior Analyst

Travis Wiley Senior Analyst

Andrea Nossaman Writer-Editor
30

Appendix. Agency Comments

APPENDIX. AGENCY COMMENTS 

Federal Aviation 

Administration 

Memorandum

Date: October 20, 2010

To: Jeffrey B. Guzzeti, Assistant Inspector General for Aviation and Special 

Program Audits

From: Clay Foushee, Director, Office Audit and Evaluation, AAE-1

Prepared by: Anthony R. Williams, x79000

Subject: OIG Report: FAA Needs to Improve Risk Assessment Processes for Its 

Air Transportation Oversight System (ATOS)

The Air Transportation Oversight System (ATOS) is an ambitious attempt to utilize the 

best available data sources to detect and predict safety trends, and use those trends as a 

tool to prioritize risk areas that could benefit from increased surveillance. The FAA 

recognizes that ATOS still has limitations, and as such, the system is subject to 

continuous refinement to improve its capabilities. As a result of this continuous 

improvement process, FAA has already addressed many of the issues identified in the 

OIG draft report, which includes cases that date as far back as 1999. Since accurately 

predicting risk factors is an imprecise science, ATOS is just one of a number of methods 

the FAA uses to inform decisions about the allocation of aviation safety inspector 

resources. Despite its limitations, ATOS provides useful information to help better focus 

FAA’s safety inspection activities. As a result, the most critical inspections are now 

being completed in a timely manner, and FAA is better able to identify and defer, low 

risk inspections to better focus its inspection activities on the most critical safety 

priorities.

The draft report’s acknowledgement that ATOS is conceptually sound is reassuring as 

FAA continues to evolve its oversight systems to take full advantage of system safety 

frameworks including the integration of other safety management systems. For example, 

FAA is developing the Safety Assurance System (SAS), with system implementation 

planned for 2013. SAS will employ enhanced analytic processes focused on hazard 

identification, risk mitigation, and validation of risk control effectiveness as a result of 

lessons learned during ATOS development and implementation. We anticipate that SAS 

will further improve FAA’s analytic capabilities, enabling even more precise focus on the 

most important safety priorities.
31

Appendix. Agency Comments

Recommendations and Responses

OIG Recommendation 1:

FAA Response: Partially concur. FAA has already taken action to ensure that 

uncompleted inspections (design and performance assessments) are rescheduled when 

justified by their relative risk. The agency modified the ATOS software in April 2010 to 

employ a “ghost” icon that represents an assessment that did not have resources assigned 

to it and, therefore, was not completed. The icon automatically appears in the calendar 

quarter subsequent to the quarter in which the assessment was not completed. 

Consequently, uncompleted assessments from a previous quarter will compete for 

resources with assessments scheduled for the current quarter, and principal inspectors 

have the opportunity to make priority adjustments based on the fact that an assessment 

has been deferred. 

 Redesign the Quarterly ADI [Assessment Determination and 

Implementation] Completion report to include cumulative roll-up data from previous 

quarters and conduct trend analyses that could be used to hold regional and local 

inspection offices accountable for scheduling uncompleted inspections.

Each calendar quarter, principal inspectors rank all of the ATOS assessments due in that 

quarter according to their relative safety risk. This process provides an additional check 

to ensure that managers assign inspectors to assessments with the highest risk and work 

down the priority list until expending all resources. ATOS-generated inspections may be 

deferred for other acceptable reasons. For example, if an air carrier is required to make 

changes to its flight crewmember training program due to observed deficiencies, there is 

no point in assessing the performance of the training program until the changes are 

complete. Another example would be a principal inspector deferring an assessment so 

something that has not been assessed for an extended period can receive attention, even 

though it might have a lower ATOS risk value. 

After years of experience with ATOS and because of inspector resource limitations, FAA 

recognizes the importance of maintaining high standards for inspection quality, even if 

that means it will not complete every ATOS-scheduled inspection. Principal inspectors 

and their front-line managers are responsible for making decisions on the best use of 

resources, using the best available data and knowledge. As described in FAA’s 

September 2009 Report to Congress,1 FAA depends on front line managers to make 

resource allocation decisions to focus their limited resources as effectively as possible. It 

is unwise to override local decisions on resource allocation based on national trend 

analyses. Further, unexpected incidents and findings from surveillance activities will 

always require priorities to shift and be redefined, and a strict adherence to ATOSdefined schedules would not be in the best interests of aviation safety.

The Quarterly ADI Completion report, referenced in OIG’s recommendation, is used to 

update a dashboard used by the Associate Administrator for Aviation Safety to track the 

status of a variety of programs. The report is also sent to all Flight Standards (AFS) 

 1 Please see FAA’s September 21, 2009, Report to Congress: Semi-Annual Overdue Safety Attribute and Element 

Performance Inspections.
32

Appendix. Agency Comments

regional division managers. It is effective for its intended purpose, which is to hold 

certificate-holding offices accountable for completing “assigned” inspections (those with 

resources assigned to them) in accordance with the principles of a risk-based oversight 

system, i.e., using available resources to accomplish the highest priority work. 

Certificate management teams (CMT) are expected to complete assigned inspections in 

accordance with ATOS policy. After we began tracking trends in accomplishing 

assigned assessments in the 3rd quarter of 2008, 100 percent of them have been completed 

on time. This was not the case before we began tracking them. The Quarterly ADI 

Completion report also tracks “unassigned” assessments (those that did not have 

resources assigned to them) at the regional level. 

The utility of this OIG report finding could be enhanced if it offered greater emphasis on 

assigned ATOS inspections rather than simply focusing upon all uncompleted ATOSscheduled inspections, as these inspections reflect higher priorities. Unassigned 

assessments equate to the “uncompleted inspections” referenced in the OIG 

recommendation. ATOS policy establishes how often inspectors conduct design and 

performance assessments (baseline frequency). Inspectors must conduct design 

assessments every 5 years. FAA considers a design assessment completed in the past 5 

years as up-to-date. We intend to accomplish all design assessments as close as possible 

to the baseline, 5-year interval. However, current ATOS policy permits design 

assessments to be completed beyond their 5-year due dates if Principal Inspectors 

document a risk-based rationale for doing so in ATOS automation.

Similar criteria apply to performance assessments. Inspectors must conduct performance 

assessments every 6 months, 1 year, or 3 years, depending on the criticality2

The intent of this recommendation is met. We request that it be closed.

of a failure 

in the program element assessed. An assessment can be scheduled to be completed 

before its baseline due date. When this occurs and the assessment is not completed, it 

will be an “uncompleted inspection” by the criteria used in the OIG recommendation, but 

it will not be overdue in relation to its baseline frequency. The agency prepares an ad hoc 

report to track overdue assessments. In the future, we will produce this report each 

calendar quarter and provide it to AFS regional division managers.

OIG Recommendation 2: Develop procedures to document justification for significant 

changes to ATOS (i.e., planned changes to alter the number of data collection tools or 

prescribed inspection time intervals).

FAA Response:

 2 The criticality of an air carrier operating system (e.g., deicing, fueling, flight crewmember training) is categorized as 

high, medium, or low based on the likelihood that a failure in the system will result in an unsafe condition.

 Concur. FAA has documented procedures for making significant 

changes to ATOS as part of the agency’s ISO 9001-registered quality management 

system (QMS), which was not considered as part of the OIG audit. These QMS 

processes and work instructions are used when we make changes such as policy 

revisions, automation changes, data collection tool revisions, and certification process 

changes. 
33

Appendix. Agency Comments

The most recent changes to the data collection tools (DCT) were made by AFS subject 

matter experts, who followed a process approved and overseen by management. These 

changes to the DCTs eliminated redundant questions and combined several elements so 

that ATOS procedures and tools would be more efficient and easier to use by small 

CMTs. If a future safety-mandated, significant change becomes necessary, we will 

follow our QMS processes and document the justification for the change.

The intent of this recommendation is met. We request that it be closed.

OIG Recommendation 3:

FAA Response: Partially concur. FAA agrees that the ATOS methodology for 

prioritizing risk could benefit from further improvement and is being continually refined 

as a result of our experience with the system, and with ever-changing risk profiles within 

the industry. Earlier versions of ATOS used an algorithm that calculated percentage of 

increased risk to trigger an increased level of criticality for an inspection element. 

However, this does not necessarily equate to significantly increased safety criticality 

because the element may be a low priority item in which failure would have limited 

consequences. The criticality of an ATOS element should be determined by the 

likelihood that a failure in the system represented by the element could lead to a 

significantly unsafe condition or failure. In other words, critical items are those in which 

failure would more likely lead to an incident or accident. FAA has developed and 

implemented such methodology.

 Redesign the current risk assessment process within ATOS 

so that it appropriately prioritizes maintenance programs with the greatest percentage of 

increased risk (regardless of criticality level) for inspector resources. 

FAA also recently convened a panel of subject matter experts to study ways to improve 

the Air Carrier Assessment Tool (ACAT), including several alternatives that would allow 

for lower-criticality elements to compete more successfully for resources. Given the 

planned future enhanced system assessment tools (SAS), the cost of re-engineering the 

ATOS software to accommodate these changes is not warranted; however, this will be 

addressed in SAS. 

As noted above, a process has been implemented that allows principal inspectors to adjust 

inspection priorities in the current ATOS system. This process allows principal 

inspectors to adjust ATOS assessment priorities if there are overriding factors such as 

significant changes in risk indicators or an extended time since the last assessment. FAA 

Order 8900.1, Flight Standards Information Management System, volume 10, chapter 2, 

includes examples of why a principal inspector may adjust the priority of an assessment. 

Examples include:

• Results of a bottom-line assessment for that element.

• Status of the element in relation to the 6-month, 1-year, 3-year performance 

assessment review cycle, or the 5-year design assessment cycle.

• Any increase or decrease in the risk score over time.
34

Appendix. Agency Comments

• Assignment of assessment completion dates based on the highest risk.

• New elements that are added as a result of major changes to air carrier operations.

• Work that was unassigned in the previous planning cycle that may have a higher 

priority level for the current planning cycle.

We believe these procedures enable principal inspectors to adjust assessment priorities as 

intended by OIG’s recommendation. Consequently, the intent of this recommendation is 

met, and we request that it be closed. 

OIG Recommendation 4: Provide training to inspectors to help them more accurately 

interpret data from all available sources (i.e., self-disclosure data, ATOS Data Packages, 

events occurring in the aviation industry) and apply the results more consistently when 

planning risk assessments. 

FAA Response: Concur. FAA has deployed two new courses for ATOS aviation safety 

inspectors (ASI), principal inspectors, and managers. They are FAA course 21000051, 

ATOS 1.2 Interactive Training for Aviation Safety Inspectors; and FAA course 

21000052, ATOS 1.2 Interactive Training for Principal Inspectors and Managers. Both 

courses underwent a major update this summer (2010). The revised course curriculum 

for course 21000052 for principal inspectors and managers provides in-depth coverage of 

the risk assessment process using ACAT risk indicators and demonstrates how they can 

be affected by reports available in the Safety Performance Analysis System, including the 

ATOS data packages referenced in the OIG recommendation. The course also 

emphasizes using operational research analysts to assist in the interpretation of data and 

adjustment of risk scores. This course is required for all newly assigned principal 

inspectors and managers. By December 31, 2010, we will complete a review of this 

revised course to determine if additional enhancements would be useful.

OIG Recommendation 5: Evaluate ATOS to determine if it is designed to accurately 

record inspection findings unique to smaller air carrier operations.

FAA Response: Partially concur. The examples of CMTs having difficulty using ATOS 

DCTs seem to be unique—in one case a seaplane operator and in the other a service 

provider to sports franchise customers. Redesigning ATOS software to make changes to 

the data recording process would be very expensive and is not justified for a few 

relatively unique operations. 

FAA’s recent revision to the DCTs, completed in July 2010, was largely in response to 

inspector input gathered during extensive listening sessions with small, medium, and 

large CMTs. The modifications combined elements, eliminated redundancies, clarified 

questions, and applied plain language to make it easier to answer DCT questions and 

record results. Inspectors now have the option of entering comments for “yes” answers 

to DCT questions as well as to “no” answers. These revisions, completed in large part 

after the timeframe of OIG’s audit, address most of the concerns of smaller CMTs. 

These actions are responsive to the intent of this recommendation, and FAA requests that 

it be closed. 
35

Appendix. Agency Comments

OIG Recommendation 6: Evaluate whether ATOS is scalable across all Part 121 air 

carriers.

FAA Response: Concur. FAA has completed the recommended evaluation based on our 

responses to recommendations 1 and 5, and is confident that ATOS is as scalable as is 

feasible given ATOS' remaining life span. Scalability is a major design requirement for 

the AFS SAS. Significant attention and resources are being devoted to ensuring the AFS 

SAS can be effectively used by small and large CMTs. Extensive documentation of the 

AFS SAS design requirements is available for the OIG's review. Consequently, we 

believe we have met the intent of this recommendation, and we will enhance the 

scalability of our oversight system when SAS is deployed in 2013. We request that this 

recommendation be closed.

OIG Recommendation 7: Expedite enhancement of ATOS training methods (as 

identified in its 2008 internal report) to assist inspectors in understanding how to use 

ATOS data collection tools and increase their proficiency in using ATOS. 

FAA Response: Concur. Two new courses were developed for ATOS aviation safety 

inspectors (ASI), principal inspectors, and managers. They are FAA course 21000051, 

ATOS 1.2 Interactive Training for Aviation Safety Inspectors; and FAA course 

21000052, ATOS 1.2 Interactive Training for Principal Inspectors and Managers. These 

courses incorporate recommendations from the 2008 internal report. Both courses 

underwent a major update this summer (2010). 

The walk-through and operational try-out of revised course 21000051 were completed on 

August 27 and September 18, 2010, respectively. The course prototype is scheduled for 

October 8 through 14. Course 21000051 for ASIs includes training on understanding and 

use of DCTs as OIG recommends. All newly hired ASIs are required to complete this 

course. The course focuses on ATOS functions that are essential for job performance, 

including data collection and reporting. The course curriculum is available for the OIG’s 

review. 

The prototype of revised course 21000052 was completed on July 29, 2010. Two classes 

of the revised course for principal inspectors and managers have been taught since then. 

All of the recommendations of the 2008 internal report are closed either by incorporating 

them into ATOS training and processes or referring them to the Systems Approach to 

Safety Oversight Program Office for inclusion in the AFS SAS. A copy of the 2008 

report and its disposition is available for the OIG’s review. Consequently, we request 

that this recommendation be closed.