Office of Inspector General

Audit Report

FAA LACKS A RELIABLE MODEL FOR 

DETERMINING THE NUMBER OF FLIGHT 

STANDARDS SAFETY INSPECTORS IT NEEDS 

Federal Aviation Administration

Report Number: AV-2013-099

Date Issued: June 20, 2013
U.S. Department of

Memorandum

Transportation

Office of the Secretary

of Transportation

Office of Inspector General

Subject: ACTION: FAA Lacks a Reliable Model for 

Determining the Number of Flight Standards 

Safety Inspectors It Needs

Federal Aviation Administration

Report No. AV-2013-099

Date: June 20, 2013

From: Jeffrey B. Guzzetti

Assistant Inspector General 

 for Aviation Audits 

Reply to 

Attn. of: JA-10

To: Federal Aviation Administrator

FAA employs approximately 4,000 aviation safety inspectors and 40 analysts1

who play a key role in helping to maintain the United States’ remarkable air 

carrier safety record. However, the February 2009 Colgan Air crash highlighted 

potential weaknesses in FAA’s oversight of Part 1212 air carriers, including 

concerns about whether FAA has enough inspectors. In October 2009, FAA 

introduced a new staffing model, known as the Aviation Safety Staffing Tool and 

Reporting System, to address concerns raised in a 2006 congressionally mandated 

National Research Council3 (NRC) study. The NRC concluded that FAA had an 

ineffective method for identifying how many safety inspectors it needs and where 

they are most needed. 

In the Airline Safety and FAA Extension Act of 2010,4 Congress directed our 

office to evaluate how FAA assigns inspectors to Part 121 air carriers, including 

assessing the number and experience levels of inspectors and analysts, and how 

inspectors use surveillance methods to supplement their regular inspections.

Accordingly, our audit objectives were to (1) determine the status of FAA’s 

implementation of its new staffing model, (2) evaluate the process FAA uses to 

 1 Analysts support safety inspectors by analyzing air carrier and inspection data to identify risk areas and trends. 2 14 CFR Part 121, Operating Requirements: Domestic, Flag, and Supplemental Operations. Mainline air carriers are 

major airlines that generally operate aircraft seating 100 or more passengers. Regional air carriers are airlines that 

generally operate aircraft seating 99 or fewer passengers.

3 The National Research Council is the working arm of the National Academies of the U.S and is organized to associate 

the science and technology community with the purposes of increasing knowledge and advising the Federal 

government.

4 Public Law 111-216 (enacted 2010).
2

assess the number and level of experience of inspectors and analysts assigned to 

each Part 121 carrier, and (3) evaluate FAA’s use of other surveillance processes 

to supplement the inspections performed by assigned oversight offices.

We conducted this review in accordance with generally accepted Government 

auditing standards. To conduct our work, we visited or contacted 28 FAA field 

offices responsible for oversight of Part 121 air carriers in 8 FAA regions, the 

FAA’s staffing model program office, budget and finance officials, and the 

Research and Innovative Technology Administration’s Volpe Center (Volpe).

5 We 

also reviewed documents obtained from both FAA and Volpe. Exhibit A details 

our scope and methodology. Exhibit B lists the organizations we visited or 

contacted.

RESULTS IN BRIEF

FAA established its inspector staffing model over 3 years ago; however, the model 

is faulty—containing incomplete, inaccurate, and outdated data—and cannot be 

relied on to determine the number and placement of inspectors needed. For 

example, key data such as the number of hours worked per task by inspectors are 

still based on outdated surveys rather than actual data. In addition, Volpe has faced 

challenges in identifying which variables and formulas would best predict staffing 

needs. Although FAA has conducted outreach sessions in field offices to 

emphasize the importance of data quality, the Agency has not implemented a 

comprehensive, aggressive strategy for obtaining accurate data or milestones for 

effectively implementing the model. FAA’s problems in using its model are due in 

part to the Agency’s failure to fully address all aspects of the September 2006 

NRC recommendations, such as defining performance measures of the model’s 

success, clearly identifying what work is not performed as a result of staffing 

shortfalls, and obtaining inspector workforce buy-in. Despite these shortcomings, 

FAA officials credit the model with providing better overall information on 

current staffing (i.e., how many inspectors the Agency currently employs). 

However, the model’s data deficiencies preclude FAA from reliably determining

how many inspectors it needs, where they are most needed, and whether the 

Agency’s approach is cost effective. 

Without a reliable inspector staffing model, FAA’s process for assessing the 

number of inspectors and analysts assigned to each Part 121 air carrier does not 

differ significantly from prior methods that the NRC deemed ineffective. 

Currently, the Director of Flight Standards Service, with advice from a committee 

of FAA managers known as the Human Capital Council (HCC), establishes

 5 The Volpe Center assists Federal, State, and local governments; industry; and academia in a number of areas,

including human factors research; system design, implementation, and assessment; strategic investment and resource 

allocation; environmental preservation; and organizational effectiveness.
3

staffing levels for FAA regions. Regional Managers then place inspectors in 

individual offices using processes that vary by region, which can lead to subjective 

and inconsistent staffing decisions. In addition, FAA has not established a formal 

method for determining the number of analysts needed to support FAA inspectors 

assigned to a particular air carrier. As result, FAA cannot be assured that it is 

consistently targeting inspector and analyst resources where they are most needed. 

FAA supplements its regular inspections through its reinstated geographic 

surveillance program.

6 While the geographic surveillance program is a helpful 

oversight tool, we identified concerns that may undermine its success. For 

example, inspectors expressed reluctance to participate in certain aspects of the 

new geographic surveillance program because typically, inspectors in other offices

are not trained on the specific operations of their assigned air carrier. Additionally, 

inspectors noted the cumbersome process of requesting assistance from other 

offices through the program. Similar issues impacted FAA’s previous geographic 

inspection program and ultimately led to its discontinuation—a risk that FAA 

could face again.

We are making recommendations to enhance the effectiveness of FAA’s staffing 

model and address training issues related to the geographic surveillance program.

BACKGROUND

FAA’s approximately 4,000 inspectors and analysts are located in 105 Certificate 

Management Offices (CMO) and Flight Standards District Offices (FSDO) across 

the country.

7 They are charged with oversight of all facets of aviation safety from 

general aviation to air carrier operations. Inspectors overseeing Part 121 air 

carriers are located in 44 CMOs and FSDOs that can be responsible for either one 

air carrier or several smaller air carriers. As shown in table 1, inspectors’ oversight 

covers a vast network of operators and functions.

 6 The Geographic Surveillance Program is the process through which FAA inspectors can request inspectors in other 

offices to assist with surveillance of their assigned air carrier. 7 Certificate Management Office inspectors are assigned the responsibility of monitoring the operations and 

maintenance activities of major air carriers. Flight Standards District Office inspectors are responsible for monitoring 

the safety of various types of aviation operators located in their assigned geographical area or district.
4

Table 1. FAA Flight Standards Inspectors’ Workload

Area of Oversight Number Requiring 

Inspection

Part 121 Air Carriers 89

Active Pilots 747,959

Repair Stations 4,825

Check Airmen* 7,747

Flight Instructors 97,398

FAA Designee Representatives** 3,689

Aircraft 210,463

FAA Licensed Mechanics and Repairmen 378,561

Source: FAA Aviation Safety Workforce Plan as of March 2012.

*Pilots employed by air carriers to evaluate pilot proficiency.

**Individuals authorized by FAA to perform certain safety functions on its behalf.

The 2003 Vision 100-Century of Aviation Reauthorization Act8 directed the NRC

to conduct a study of the assumptions and methods used by FAA to estimate 

staffing standards for FAA inspectors to ensure proper oversight of the aviation 

industry. As required by the Act, the study was to include suggested modifications 

or alternative approaches to FAA’s staffing model and to approximate the cost and 

time of developing a modified or new staffing model. The 2006 NRC report9

recommended that FAA completely overhaul its staffing model, known as the 

Automated Staffing Allocation Model (ASAM). Specifically, the NRC found that 

ASAM did not:

• predict the consequences of staffing shortfalls;

• account for some important factors affecting inspector workload, such as 

managing designees;

10 or

• include validated data.

The NRC further recommended that a new staffing model not only possess the 

attributes listed above, but also have the ability to project the number of inspectors

needed and where they should be located to sustain system performance. In 

response, FAA contracted with Volpe to design a new staffing model and 

established a goal to implement and begin using it by October 2009. 

 8 Public Law 108-176 (enacted December 12, 2003). 9 Staffing Standards for Aviation Safety Inspectors published September 2006.

10 Individuals authorized by FAA to perform certain safety functions on its behalf.
5

FAA HAS NOT FULLY IMPLEMENTED ITS NEW STAFFING 

MODEL DUE TO CONCERNS ABOUT ITS RELIABILITY

Although FAA introduced its inspector staffing model11 in October 2009, FAA 

has not fully relied on the model’s results because of continuing concerns with its 

incomplete and inaccurate data. FAA has not successfully integrated key data that 

would enhance the model’s accuracy. In addition, Volpe faced challenges in 

identifying which variables and formulas would best predict inspector staffing 

needs. Further, the model does not fully address all aspects of the September 2006 

NRC recommendations in key areas, such as defining performance measures of 

the model’s effectiveness and clearly identifying what inspections will not be

performed as a result of staffing shortfalls. 

FAA’s Inspector Staffing Model Does Not Effectively Project Staffing 

Needs or Support Budget Requests

FAA’s inspector staffing model does not yet produce accurate projections for 

Flight Standards Service12 staffing needs. Since 2009, FAA and Volpe have tried 

to determine which data are most effective for predicting proper staffing levels and 

how heavily they should be weighted in the model. However, the model is still 

generating results that fluctuate significantly at the national, regional, and office 

level. For example, as of January 2013, FAA has reported the results of the 

staffing model six times since it was first established over 3 years ago, with each 

iteration showing differing nationwide employee shortages. As shown in figure 1, 

the model has projected employee shortages ranging from 389 to as high as 935.

13

 11 FAA’s staffing model is called the Office of Aviation Safety Staffing Tool and Reporting System (ASTARS). 12 The model also projects staffing needs for other FAA organizations such as aircraft engineers. However, our review 

focused on the Flight Standards Service.

13 FAA determines a shortage or surplus based on the difference in the number of employees on-board and the number 

of employees projected as necessary by the model at the time the model reports staffing needs. These differences do not 

correlate directly to inspector attrition.
6

Figure 1. Model-Projected Employee Shortages

Source: OIG analysis of FAA data.

While FAA Headquarters officials stated the model is useful in accounting for 

current staffing by providing them with data on the number of inspectors on board, 

they are not yet confident in the model’s ability to accurately project staffing 

needs. Therefore, FAA officials have not fully relied on the model’s projections

when justifying resource needs to Congress during the annual budget process. For 

example, when FAA submitted its proposed budget for fiscal year 2012, FAA 

requested 112 additional positions even though the staffing model projected a 

shortage of 463. For fiscal year 2013, FAA did not request any additional 

positions, even though the staffing model projected a shortage of 389 positions.

Incomplete and Inaccurate Data Impede FAA’s Staffing Model 

Effectiveness

FAA’s difficulties with developing an accurate staffing model are due in part to 

problems with incomplete and inaccurate data. FAA has attempted to identify data 

that will enable the model to better predict staffing needs but faces challenges in 

capturing accurate data in a format the model can use. Efforts to identify data 

problems have been mostly ad hoc, without a comprehensive, documented 

analysis of underlying reasons for differences between onboard and projected 

staffing, or a timeline for corrective actions. As a result of these underlying data 

concerns, wide fluctuations regarding the number of inspectors at the individual 

office level are continuing to occur, as shown in table 2.

463

389

646

914 935

430

0

100

200

300

400

500

600

700

800

900

1000

Apr-10 Jan-11 Jul-11 Jan-12 Aug-12 Jan-13

Number of Employees
7

Table 2. Fluctuations in Model Projections for Staffing 

Shortages and Overages, April 2010–January 2013

Office April 

2010

January 

2011

July 

2011

January 

2012

August 

2012

January 

2013

Chicago FSDO -6 -4 -7 -30 -34 -4

Delta CMO -8 +40 +44 -63 -53 +52

American CMO -11 -18 -9 -84 -86 +22

Sacramento FSDO -4 -7 -11 -13 -13 -9

Note: Numbers in red indicate a projected staffing shortage, while numbers in black indicate an overage.

Source: OIG analysis of FAA data.

FAA officials stated that the large fluctuations in model results shown in the table 

could be due to FAA electing not to backfill vacancies as they occurred (such as at 

the Sacramento FSDO). Not backfilling vacancies reduces the number of staff 

onboard, resulting in a change in the model’s projected staffing need. However, 

based on our analysis of staffing data at this office, this only accounts for a portion 

of the employee variance for this FSDO between April 2010 and January 2012.

Moreover, FAA’s cited cause does not explain all the fluctuations for other 

offices. Rather, based on our analysis of FAA’s data, we found that these 

fluctuations are attributable to a range of underlying data issues. For example:

• Inaccurate and outdated line station data—Based on our analysis of FAA 

data, fluctuations in the model results for the Delta Certificate Management 

Office (CMO) were due to inaccurate data on the number of line stations. For 

example, FAA officials attributed the January 2011 model results—which

indicated the Delta CMO was overstaffed by 40 positions—to significant 

underreporting on the number of line stations Delta served. According to FAA

officials, misstating the number of line stations dramatically skews the number 

of positions an oversight office is projected to need. Although a critical 

attribute in determining staffing needs, the number of line stations can be 

difficult to keep current because destinations change frequently, particularly 

for small charter operators. Inspectors at 8 of 2814 oversight offices we 

interviewed stated they did not remove line stations from a database used by

the model when the carrier they oversee suspended service to that location.

• Large fluctuations in model results following database updates—FAA’s 

staffing model program manager attributed the large fluctuations in model 

results for the Delta and American CMOs between July 2011 and January 2012 

to the model’s inability to handle large variations in data. These variations 

 14 We selected a random sample of 30 out of 44 FAA field offices. However, during the course of our audit, oversight 

responsibilities for two air carriers were transferred to other offices in our sample resulting in the need to visit only 28 

offices.
8

occurred as field office inspectors began updating FAA databases at the 

program office’s request. For example, the reported number of Delta 

maintenance locations with training records in FAA’s database increased by 56 

(431 percent) as a result of database updates. Also, FAA inspectors updated the

reported number of pilots-in-command for American Airlines to reflect a 

decrease of 1,455 (30 percent). While FAA indicated that changes in the model 

results could be due to normal variations in the aviation landscape as carriers 

change their operations, the program manager stated that in this case inspectors 

had not updated the pilots-in-command figure in several years. FAA modified

the model in December 2012 to account for these large variations in data.

• Other database accuracy issues—Several databases15 containing inspection, 

personnel, and air carrier statistical data feed into the model (see figure 2). 

However, even though FAA officials overseeing development of the model 

have reminded inspectors and office managers to ensure the accuracy of these 

databases, some data are still outdated, difficult to obtain, or inaccurate. For 

example, some data—such as the amount of time it takes inspectors to perform 

their work—are still based on outdated survey data obtained in 2008, instead of 

actual data. Additionally, some data, such as information on current on-board 

staffing, is obtained through a time consuming manual process that must be 

repeated each time the model is run. 

Figure 2. Databases That Feed Into the Staffing Model 

Model Results

eVID

ATOS

FPPS PTRS

Web 

OPPS

Simulations Model Report/Projections

Source: OIG analysis of FAA data.

 15 Key databases currently feeding the model are the electronic Vital Information Database (eVID), which contains 

statistics on air carriers such as the number of pilots and aircraft the carrier operates; the Program Tracking and 

Reporting System (PTRS), which tracks certain inspection data input by FAA inspectors of non-Part 121 air carriers; 

WebOps, which provides a current count of the number of Part 121 air carriers; Air Transportation Oversight System 

(ATOS), which includes inspection data input by FAA inspectors of Part 121 air carriers; and Federal Personnel Payroll 

System (FPPS), which provides data on current staffing.
9

• A lack of key data on inspector labor hours and operations—In June 2011, 

FAA had planned to improve data quality by having its Labor Distribution 

Reporting (LDR)

16 system feed directly into the staffing model, which 

provides detailed information regarding inspector labor hours by task. FAA 

officials postponed those plans until they could collect 2 years of LDR data, 

and now expect to begin using LDR data in fiscal year 2014. However, Volpe 

representatives expressed concerns as to whether the LDR system contained 

the level of detail to significantly improve the model’s results. They also stated 

that a lack of quality data on how long it takes inspectors to perform their work 

is a significant shortcoming because such information is fundamental to the 

model’s precision. FAA has attempted to address this shortcoming, but so far 

has not been successful. Further, the model does not yet include key data on all 

of the work inspectors perform, such as designee management.

17 FAA officials 

told us that these functions are slated to be added to the model in the future, but 

the timing of that decision is unknown.

According to Volpe representatives, an alternative and better source of data to 

support the model is the operations specifications for each Part 121 air carrier. 

These specifications provide the authorizations, limitations, and procedures an 

air carrier is subject to and include information on the number and type of 

aircraft operated, airports served, maintenance bases, and other data associated 

with each carrier. However, this information is maintained in a format that 

makes it unusable to the model. FAA officials stated that they have been 

working for 2 years to obtain access to the underlying data in a usable form 

and finally established an MOU in September 2012 to gain access. However, 

while Volpe officials are cautiously optimistic that this information would give 

them the data they need, it is too soon to determine whether it will produce a 

reliable model.

Despite these data shortcomings, FAA officials stated that the current staffing 

model represents progress beyond previous models. Further, they credited the new 

model with providing better overall information on current staffing, while calling 

attention to problems with FAA databases that they had not previously discovered.

However, until these underlying data problems are resolved, the true staffing 

needs of FAA offices will remain unknown. Of particular concern is the fact that 

because FAA is not relying on the model’s results, a genuine problem with 

understaffing may go unresolved. For example, the staffing model indicated that 

one oversight office was understaffed by 34 inspectors. During our review, we 

found that managers at this office generally agreed with the assessment that they 

 16 Labor Distribution Reporting: A reporting system by which inspectors code their total time by work tasks/type of 

work. 

17 For example, until recently, a procedure to tell the model to count the inspectors who oversee large helicopter 

emergency medical service operators was inadvertently missing from the model.
10

needed more staffing. Safety inspectors at this office also agreed that they were 

not able to conduct the number of inspections necessary to make comprehensive 

risk assessments of their assigned carriers. Yet, because of the staffing model’s 

overall unreliability, FAA has not reviewed the model fluctuations in detail and 

determined whether there are instances of understaffing that may be accurate and 

require an immediate staffing action to ensure safety.

Design Challenges Further Hinder the Development of a Reliable 

Model

In addition to data inaccuracies, Volpe representatives expressed concerns with 

developing the model in areas related to its key variables and design. They stated 

that they are unable to effectively predict the frequency of surveillance activities at 

the individual office level because each air carrier is unique in the amount and 

type of oversight required. Further, they were concerned that some model 

variables are too simplistic. For example, the model takes into account the total 

number of aircraft flown by a given airline but not the number of different aircraft 

fleets that make up the total.

FAA and Volpe will face additional challenges because of the need to adjust the 

model to accommodate a new oversight system that FAA plans to implement in 

fiscal year 2015. FAA is developing the Safety Assurance System (SAS) to 

replace its current safety oversight system, the Air Transportation Oversight 

System (ATOS).

18 SAS will include new inspection data collection tools that FAA 

and Volpe will need to incorporate into the model. However, the change in model 

data inputs could impact FAA’s ability to reliably determine inspector staffing 

needs. Recognizing these concerns, FAA has formed a work group to study the 

impact that SAS will have on its staffing model.

FAA’s Staffing Model Does Not Fully Address the Recommendations 

of the National Research Council Report

Developing a comprehensive staffing model under tight timeframes is a difficult 

challenge. Yet, over 3 years after its introduction, FAA’s model still does not fully 

comply with the intent of key NRC recommendations that are critical to enhancing 

the model’s success, such as analyzing costs and defining performance goals. 

Despite these prolonged issues, FAA has not taken steps to reassess its approach. 

The NRC recommendations not yet implemented include the following:

• Conduct Detailed Cost Analyses. FAA has not yet performed a 

comprehensive analysis to determine the total cost to operate, upgrade, and 

maintain the new staffing model. The NRC report strongly recommended FAA 

conduct a cost-benefit analysis, during the design phase, to determine what the 

 18 ATOS is FAA’s mechanism to conduct safety inspections and provide regulatory oversight of Part 121 air carriers.
11

Agency was willing to invest to implement and maintain the model versus 

what it hoped to achieve with the model. While NRC estimated developing a 

model would cost between $900,000 and $1.2 million, according to FAA, the 

Agency has spent at least $9.9 million. Further, FAA’s estimates of the cost to 

maintain the model range between $1.8 million and $2.2 million per year.

• Define Performance Measures. FAA has not yet established system and 

individual performance measures to evaluate the effectiveness of the model for 

items such as a comprehensive list of duties an inspector performs, including 

their priority. While the NRC report recommended performance measures be 

developed at the outset of the process, the staffing model program manager 

said FAA just recently began developing them. 

• Identify the Consequences of Staffing Shortfalls. The model cannot quantify 

the consequences of understaffing, which was also a major weakness in FAA’s 

previous staffing model. Without this metric, FAA managers cannot measure 

the impact of one staffing choice versus another so that they can better 

prioritize staffing decisions. FAA officials stated that a new simulation module 

implemented in July 2012 identifies activities that are not being accomplished. 

However, the new module only identifies the number of hours that are not 

covered with current staffing, not specific work activities. Volpe 

representatives said identifying the consequences of staffing shortfalls will 

require FAA to rank all inspector duties in order of importance, something that

the Agency has not yet done. 

• Obtain Management Buy-In at Field Offices. While FAA officials 

responsible for the model at Headquarters have conducted field outreach 

sessions on the importance of data accuracy, we found that field personnel still 

do not understand the impact of inaccurate data on the model’s staffing 

determinations. This is further hindered by FAA’s limited training of office 

managers, which currently consists only of an online course. FAA Flight 

Standards Headquarters officials cited the initial instructor-led training offered 

to field office managers in 2009 as adequate. However, since then, managers 

promoted after the 2009 class are forced to rely on the online course, which is 

less comprehensive.

ASSESSING THE NUMBER AND EXPERIENCE LEVEL OF 

INSPECTORS CONTINUES TO BE A CHALLENGE FOR FAA

Without a reliable staffing model, FAA’s process for determining the appropriate 

number of inspectors to assign to Part 121 air carriers remains largely unchanged 

from prior ineffective methods and varies across offices. Similarly, FAA has not 

yet established a method for determining how many analysts are needed to support 
12

the inspector workforce. Finally, levels and types of experience for inspectors 

varied between mainline and large cargo and regional and other Part 121 air 

carriers.

FAA’s Processes for Assigning Inspectors Are Inconsistent and 

Largely Unchanged From Past Efforts

Due to shortcomings in the staffing model’s ability to project accurate staffing 

needs, FAA’s process for determining the appropriate number of inspectors to 

assign to Part 121 air carriers remains largely unchanged from prior years. FAA 

regions and field offices vary in the extent to which they use regional staffing 

committees and older models and guides. As a result, FAA cannot be assured that 

it is consistently targeting inspector resources where they are most needed.

FAA’s primary staffing allocation decisions are made by the Director of Flight 

Standards after consulting with a national committee of FAA managers known as

the Human Capital Council (HCC). The HCC meets to determine regional staffing 

needs, and sends its recommendations to the Flight Standards Director. The 

Director establishes and disseminates staffing levels for FAA regions to regional 

office managers. Regional Managers then place inspectors in individual offices 

using processes that vary by region, which can lead to subjective staffing 

decisions. HCC members and regional office managers told us they use the 

staffing model results as a guide to compare overall numbers across regions and 

offices, but were not using the results to determine where to place inspectors due 

to a lack of confidence in the model’s projections.

At 15 of 28 field offices we visited, managers were still using unreliable processes 

to determine staffing needs. These included referencing the 1825 Position 

Classification Guide, commonly called the complexity guide,19 a document that 

FAA Headquarters officials told us should no longer be used to determine staffing 

needs.20 In 1995, FAA determined that the complexity guide methodology was 

ineffective and subsequently created the ASAM model. However, according to the 

2006 NRC report, ASAM was deficient as well. Thus, FAA’s reliance on previous 

methods of staffing allocation limits the Agency’s awareness of its staffing needs 

and consequently the outcomes of its staffing decisions. 

 19 Complexity Guide: A position classification guide that determines grade levels for FAA safety inspectors and 

justifies inspector grades based on specific characteristics ('complexity') of the assigned carrier. 20 The NRC concluded that FAA determined in 1995 that using complexity points to make staffing decisions was not 

effective.
13

FAA Inspectors’ Experience Levels Varied Between Regional and

Mainline Air Carriers

FAA has been able to attract candidates with aviation industry experience to fill 

inspector positions for both mainline21 and regional carriers.22 Both types of 

carriers reported that candidates had a high degree of Part 121 experience prior to 

joining FAA.23 Yet, we noted differences in FAA experience levels (i.e., tenure 

with the Agency) between those offices responsible for oversight of mainline air 

carriers compared with those responsible for regional or other air carriers. 

Although the impact of these differences is unclear, we found that over 86 percent 

of Principal Inspectors24 of mainline air carriers have more than 10 years of 

experience with FAA. In comparison, only 58 percent of Principal Inspectors of 

regional and other air carriers have a similar level of Agency experience. In the 

overall inspector ranks, 35 percent of inspectors at mainline air carriers have over 

10 years of FAA experience. In contrast, at regional and other air carriers, the 

number of inspectors with over 10 years of FAA experience is 14 percent.25

One reason for the difference in FAA experience levels is that FAA inspectors 

have greater promotion potential at offices that oversee mainline air carriers. For 

example, these offices generally have Supervisory Principal Inspector positions 

that offer a higher pay grade than Principal Inspector positions at oversight offices 

for regional and other air carriers. FAA CMO officials told us that it can be 

difficult to retain highly experienced non-principal inspectors for smaller carriers

when promotional opportunities arose at the larger mainline CMOs, as smaller 

carrier certificates often do not meet complexity guide requirements to have 

inspectors with the highest pay grades. Our interviews did not disclose concerns 

on the part of office managers regarding this difference in FAA experience.

Nonetheless, ensuring that regional carriers have adequate coverage with 

inspectors experienced in FAA requirements is critical because regional airlines 

account for over half of all scheduled commercial passenger flights carrying 

approximately 160 million passengers annually.

 21 Mainline Part 121 air carriers typically operate aircraft with more than 100 passenger seats or operate large numbers 

of cargo aircraft.

22 Regional and other Part 121 air carriers typically operate aircraft with fewer than 100 passenger seats or small 

numbers of large passenger or cargo aircraft.

23 Sixty percent of inspectors we surveyed who are assigned to an office with oversight responsibility for a mainline 

Part 121 air carrier had also been employed by a mainline Part 121 air carrier for 10 or more years prior to their 

employment with FAA. Likewise, 59 percent of inspectors assigned to offices with oversight responsibility for regional 

or other Part 121 carriers said they had 10 or more years of service with a mainline Part 121 carrier prior to joining 

FAA.

24 For each air carrier, FAA assigns three lead inspectors, called principal inspectors (one for each of the three major 

areas of specialization--operations, maintenance, and avionics). Within FAA, principal inspectors have the primary 

responsibility for ensuring their assigned air carrier complies with the Federal Aviation Regulations.

25 Our results are based on a survey of Part 121 air carrier inspectors. The survey was distributed to 816 inspectors at 

the 28 FAA offices we visited or contacted. We received 501 responses for a response rate of 61.3 percent.
14

FAA Lacks a Formal Process for Assigning and Training Analysts 

FAA has also not established a formal method to determine staffing allocations for 

analysts, who are assigned to support FAA safety inspectors of all Part 121 air 

carriers by analyzing air carrier and inspection data to identify risk areas and 

trends. Analysts for large Part 121 air carriers are typically dedicated to that 

carrier, while analysts for small Part 121 air carriers are usually a shared resource 

among more than one FAA safety inspector team. We found a single analyst can 

be responsible for up to 18 carriers, while other analysts were only assigned a 

single air carrier and had assistance from other analysts. These disparities may 

result in some analysts not being properly utilized in their safety oversight roles. 

In addition, FAA does not have a formal training program for analysts, which 

hinders their ability to create and distribute advanced reports, including predictive 

analysis identifying potential hazards and trends. All of the 19 analysts we 

interviewed expressed concern with their training, including delays in training 

after being hired, a limited number of classes, and a lack of training on ATOS. For 

example, one analyst we interviewed did not receive any training for over 

6 months after being hired, and two others had only taken two classes in 2 1⁄2 

years. This affects analysts’ ability to assist inspectors to their fullest capabilities 

and hampers analyst development as the aviation industry they help to oversee is 

constantly evolving.

FAA’S SUPPLEMENTAL SURVEILLANCE PROCESSES CAN 

ENHANCE OVERSIGHT IF USED EFFECTIVELY

In 2010, FAA reinstated a geographic surveillance program to provide a greater 

level of oversight of air carriers’ airport locations served by Part 121 and 13526

operators that have had minimal surveillance coverage in the past. FAA offices 

can accomplish additional oversight through the geographic surveillance program 

by using their own inspectors or by requesting assistance from other offices. 

Eighty percent of the inspections performed in the initial phase of the program for 

Part 121 air carriers were conducted by inspectors assigned to the responsible 

oversight office, while 20 percent of the inspections were conducted by inspectors 

outside of the responsible oversight office. 

Although supplemental inspection programs can be worthwhile, it is too soon to 

determine if the new geographic surveillance program will be successful and 

improve oversight. At 16 of 28 offices we visited, FAA inspectors and managers 

raised concerns about the program, such as a lack of air carrier specific training for 

inspectors outside their office. Air carrier-specific training for inspectors is

 26 14 CFR Part 135, On-Demand, Operating Requirements: Commuter and On Demand Operations and Rules 

Governing Persons On Board Such Aircraft. 
15

necessary because each Part 121 operator has unique procedures and operations. 

Without a full understanding of the unique specifications of an operator, outside 

inspectors may not be able to effectively identify safety deficiencies. Similar 

issues negatively impacted FAA’s prior geographic program, and could put the 

new program at risk if they are not carefully managed. For example:

• Training—While FAA headquarters officials told us that training 

requirements are in place to ensure outside inspectors are aware of air carrierspecific requirements, these requirements are either unclear to inspectors or 

have been poorly communicated. In fact, of the 16 offices where inspectors 

expressed concerns with the geographic program, the most frequently cited 

issue by inspectors and managers was reluctance to use outside inspectors to 

perform surveillance work on their assigned carrier. Only 4 of the 28 offices 

we visited had requested assistance from an inspector outside of their office 

through the program. 

• Workload—Even inspectors from those offices that did request assistance 

found the program lacking and had concerns about continuing to support it. 

Specifically, inspectors and managers were concerned that geographic 

inspection requests from other offices would add to their workload or the 

requested work would not be completed. For example, one office requested 

assistance on 75 inspections, but less than half of the inspections were 

completed.

• Request Process—FAA’s process for requesting assistance is cumbersome. 

All requests for surveillance assistance require the requesting office manager to 

compose a memo that must then go through three separate offices (see 

figure 3). This lengthy process makes it particularly difficult for inspectors for 

nonscheduled air carriers to use the program because there is not enough notice 

of where the aircraft to be inspected will be located. 
16

Figure 3. Geographic Surveillance Program Request Process

Source: OIG analysis of FAA data.

FAA has created an automated tool to facilitate requests for assistance under the 

new geographic program. Although automated, inspectors expressed concerns 

because the underlying issues remain the same. 

FAA also bolsters its inspection program through the use of en-route inspections. 

En-route inspections involve an inspector conducting surveillance aboard an 

aircraft while in operation. All the inspectors we spoke with agreed that en route 

inspections—which FAA has always used to supplement its inspector resources—

are a very valuable tool for surveillance. This surveillance method allows 

inspectors to observe how documented procedures are actually implemented into 

airline operations. It also allows inspectors to continue to work while traveling to 

distant locations to conduct other scheduled inspections. While there were some 

minor concerns over air carrier procedural training and administrative issues for 

en-route inspectors, we did not find significant concerns regarding Part 121 

inspectors performing en-route inspections. 
17

CONCLUSION

FAA’s inspector workforce plays a fundamental role in ensuring the safety of our 

nation’s aviation system. Accordingly, a reliable staffing model that best targets 

risk and identifies where inspectors are needed most is key to FAA’s sustained 

safety record. Correcting shortcomings with the model will be critical to address 

staffing vulnerabilities that could impede effective oversight. 

RECOMMENDATIONS

To enhance the effectiveness of FAA’s staffing model and its geographic 

surveillance program, we recommend that FAA:

1. Develop a plan with milestones to address the model’s shortcomings and 

regularly report progress relative to plan milestones.

2. Conduct and document a variance analysis of each model’s results and assess 

staffing at field offices where the on-board staffing level varies widely from 

the current model projection to verify if immediate staffing action is needed in 

the interest of safety.

3. Verify inspectors are following existing guidance to update and maintain the 

accuracy of databases prior to running iterations of the staffing model.

4. Conduct a comprehensive assessment of the staffing model as compared to the 

NRC recommendations, assess the quality of the data in the model, and 

identify the steps needed to make the staffing model more viable.

5. Implement comprehensive and recurrent training for managers and inspectors 

on the staffing model.

6. Establish a comprehensive analyst training program with guidance clarifying 

their roles, responsibilities, and training needs, and establish a method to 

determine an appropriate number of air carriers per analyst.

7. Clarify requirements and develop a process to ensure completion of training on 

specific air carrier policies and procedures for inspectors who participate in the 

geographic surveillance program.
18

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL 

RESPONSE 

We provided a draft of this report to FAA on March 27, 2013, and received its

response on June 14, 2013, which is included as an appendix to this report. FAA 

concurred with six of our seven recommendations and partially concurred with 

one. FAA’s proposed actions are generally responsive to our recommendations, 

and we consider recommendations 2 through 6 resolved but open pending 

completion of planned actions. Additionally, while we consider recommendation 5 

resolved, we will assess results of the training review scheduled for January 2014 

to determine whether improvements meet the intent of our recommendation. 

However, we are requesting that the Agency provide additional information for 

recommendation 1 and reconsider its response to recommendation 7, as detailed 

below.

Specifically, FAA requested that recommendation 1 be closed, stating that its 

ASTARS plan meets the intent of our recommendation. Although FAA attached 

an overview of this plan with its response, the attachment does not include a 

detailed schedule with milestones, and instead states that the schedule for 

completion is contained on an internal FAA Web site. Accordingly, we are 

requesting that FAA provide information on the schedule in order for us to 

validate that the plan fully meets the intent of our recommendation.

For recommendation 7, FAA cited the development of a new oversight system—

the Safety Assurance System (SAS)—which the Agency states will enable 

requesting offices to provide inspectors with supporting documentation that will 

enhance their geographic inspection activities. We agree that these new 

capabilities will meet the intent of our recommendation. However, FAA does not 

plan to deploy SAS until 2015. Given the concerns with geographic inspector 

training and workload that we identified in our review, we believe FAA should 

issue interim guidance to inspectors who participate in the geographic surveillance 

program prior to the deployment of SAS. Therefore, we request that FAA 

reconsider its response and provide additional planned interim actions to address 

our concerns.

ACTIONS REQUIRED 

We consider recommendations 2 through 6 resolved but open pending completion 

of planned actions. For recommendations 1 and 7, we request that FAA either

provide additional information or reconsider its position as described above. In 

accordance with Department of Transportation Order 8000.1C, we request that 

FAA provide this additional information within 30 days of this report.
19

We appreciate the courtesies and cooperation of FAA representatives during this 

audit. If you have any questions concerning this report, please call me at 

(202) 366-0500 or Robin Koch, Program Director, at (404) 562-3770.

#

cc: DOT Audit Liaison, M-1

 FAA Audit Liaison, AAE-100
20

Exhibit A. Scope and Methodology

EXHIBIT A. SCOPE AND METHODOLOGY

We conducted this audit in accordance with generally accepted Government 

auditing standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that the 

evidence obtained provides a reasonable basis for our findings and conclusions 

based on our audit objectives. We conducted this review between May 2011 and 

March 2013 using the following methodology.

To obtain information about FAA’s staffing model and to determine its status, we 

met with FAA Flight Standards officials including the manager of FAA’s staffing 

model program office as well as FAA budget and finance officials. We also 

interviewed analysts at the John A. Volpe National Transportation Systems Center 

in Cambridge, MA, to obtain information regarding its role in the development of 

the staffing model. We examined the NRC’s 2006 report and recommendations for 

FAA’s staffing model and compared those to what had been developed.

We interviewed 50 certificate management teams across 28 FAA field offices and 

8 regional offices. We selected a random sample of 30 out of 44 FAA field offices. 

However, during the course of our audit, oversight responsibilities for two air 

carriers were transferred to other offices in our sample resulting in the need to visit 

only 28 offices. We spoke with inspectors, principal inspectors, office managers, 

and analysts to obtain their insight into staffing and hiring policy and procedures, 

inspector experience levels, the staffing model, and use of other surveillance 

processes that supplement regular oversight. We compared and contrasted the 

staffing needs expressed at the field office level to the staffing projections for that 

office generated by the model. We also interviewed regional office managers 

regarding these same issues.

In addition, we collected examples of data at each office that fed into staffing 

model, and analyzed differences in how they were updated across offices and thus 

affecting model outputs. We examined the amount of oversight work done at each 

carrier and investigated whether the model was consistent in recognizing needs 

across offices overseeing different carrier certificates. We also compared the 

organizational structure and staffing methods used across offices and regions.

With the assistance of Team SAI, an air transportation consulting firm, we 

conducted a survey of FAA inspectors at the 28 field offices we interviewed 

assessing inspector grade, education, and experience levels. We distributed the 

survey to 816 inspectors responsible for oversight of Part 121 air carriers at the 28 

FAA offices we visited or contacted. We received 501 responses for a response 

rate of 61.3 percent.
21

Exhibit A. Scope and Methodology

We also met with representatives from the Professional Airways Safety Specialists 

(PASS) to discuss their views on the staffing model, and FAA Human Resources 

and Aviation Careers personnel to obtain information about FAA hiring processes. 
22

Exhibit B. Organizations Visited or Contacted

EXHIBIT B. ORGANIZATIONS VISITED OR CONTACTED

Federal Aviation Administration

Headquarters

FAA Headquarters Washington, DC

Mike Monroney Aeronautical Center Oklahoma City, OK

Certificate Management Offices

Airtran CMO Orlando, FL

Alaska CMO SeaTac, WA

American Airlines CMO Forth Worth, TX

Atlanta CMO Hapeville, GA

Delta Air Lines CMO Atlanta, GA

Denali CMO Anchorage, AK

FedEx CMO Memphis, TN

Honolulu CMO Honolulu, HI

Phoenix CMO Phoenix, AZ

Portland CMO Hillsboro, OR

South Florida CMO Miramar, FL

Southwest Airlines CMO Irving, TX

United Airlines CMO Daly City, CA

UPS CMO Louisville, KY

U.S. Airways CMO Coraopolis, PA

Flight Standards District Offices

Albany FSDO Latham, NY

Baltimore FSDO Glen Burnie, MD

Boston FSDO Lexington, MA

Charlotte FSDO Charlotte, NC

Chicago FSDO Des Plaines, IL

Dallas/Ft. Worth FSDO Forth Worth, TX

Indianapolis FSDO Plainfield, IN

Las Vegas FSDO Las Vegas, NV

Memphis FSDO Memphis, TN

Minneapolis FSDO Minneapolis, MN

Sacramento FSDO Sacramento, CA 

St. Louis FSDO St. Ann, MO

Washington FSDO Herndon, VA
23

Exhibit B. Organizations Visited or Contacted

Regional Offices

Alaska Region Anchorage, AK

Central Region Kansas City, MO

Eastern Region Jamaica, NY

Great Lakes Region Des Plaines, IL

Northwest Mountain Region Renton, WA

Southern Region College Park, GA

Southwest Region Forth Worth, TX

Western-Pacific Region Lawndale, CA

Other Organizations / Administrations

John A. Volpe Center Cambridge, MA

Professional Airways Safety Specialists Washington, DC
24

Exhibit C. Major Contributors to This Report

EXHIBIT C. MAJOR CONTRIBUTORS TO THIS REPORT

Name Title

Robin Koch Program Director

Marshall Jackson Project Manager

Mark Perrill Senior Analyst

R. Andrew Farnsworth Analyst

Ruth Foyere Analyst

James Ovelmen Analyst

Megha Joshipura Statistician

Audre Azuolas Writer-Editor
APPENDIX. AGENCY COMMENTS 25

Appendix. Agency Comments

Federal Aviation

Administration

Memorandum

Date: June 14, 2013

To: Jeffrey B. Guzzetti, Director, Assistant Inspector General for Aviation and 

Special Program Audits

From: H. Clayton Foushee, Director, Office of Audit and Evaluation, AAE-1

Subject: Federal Aviation Administration’s (FAA) Response to Office of Inspector 

General (OIG) Draft Report on FAA’s Inspector Staffing Model 

The FAA fully understands the limitations of the current Aviation Safety Staffing Tool and 

Reporting System (ASTARS) identified in the draft report and has been working to address 

them. When the agency first created ASTARS, it planned to use the work count data contained 

in the Vital Information System (VIS) and the Program Tracking and Reporting System (PTRS) 

combined with times recorded in the Labor Distribution Reporting (LDR) database. 

Unfortunately, experience has shown these data lacked sufficient fidelity for an inspector 

staffing model. To address the data limitations Flight Standards (AFS) used a statistical 

methodology for predicting the frequency of work performed. 

This is the modeling effort that was reviewed by the OIG. During the review, the FAA 

repeatedly stressed to the OIG team that the model under review was only an initial effort, and 

was intended to be refined as the project matured. The OIG’s identification of incomplete, 

inaccurate, and outdated data aligned with the issues FAA identified prior to the OIG’s review. 

FAA had already initiated ongoing efforts to improve the accuracy of the pertinent FAA 

databases. FAA is continuing its efforts to improve data quality, modify the databases for easier 

data collection, and improve the guidance for keeping the databases as current and accurate as 

possible.

The AFS portion of the model has been running for over three years with iterations completed 

approximately every six months. We now have sufficient knowledge of the current model and 

how it behaves and to implement further improvements in the accuracy of the databases feeding 

the model. The next enhanced version of ASTARS is expected to be completed in December 

2013. 

While ASTARS is a primary tool, it is not the sole determinant of the number and placement of 

inspectors. The FAA uses ASTARS for macro-level guidance, which must be further refined 

with expertise and judgment from field, division, executive management, subject matter experts, 

and the Human Capital Council (HCC) to finalize staffing decisions. While we agree that the 

model reviewed by OIG has well-known and identified limitations, it is useful for developing 
26

Appendix. Agency Comments

useful input at the national and regional levels and will continue to improve over time as data 

quality and model methodology evolves. 

RECOMMENDATIONS AND RESPONSES

Recommendation 1: Develop a plan with milestones to address the model’s shortcomings and 

regularly report progress relative to plan milestones.

FAA Response: Concur. The FAA has published an ASTARS plan which establishes: 1) 

short and long-term goals (including model improvement; 2) tracks progress of completed 

action items; 3) and sets an implementation schedule. Specifically, the plan addresses the data 

quality issues that the OIG also found. The model results (quantitatively) are reviewed and 

validated by subject matter experts and approved by management. This plan is available to all 

ASTARS users and resides on a SharePoint site within Aviation Safety (AVS). The ASTARS 

SharePoint site also includes information on aspects of the model where further improvement 

opportunities have been identified, as well as proposed steps to make those improvements. 

These steps are documented in minutes of the weekly development team meetings.

Regular reporting of our progress relative to the plan milestones is being accomplished through 

review by AVS and Flight Standards Service (AFS) leadership at monthly business plan 

reviews. A copy of the current ASTARS plan is attached to this response. The FAA believes it 

has met the intent of this recommendation and requests that it be closed.

Recommendation 2: Conduct and document a variance analysis of each model’s results and

assess staffing at field offices where the on-board staffing level varies widely from the

current model projection to verify if immediate staffing action is needed in the interest of safety.

FAA Response: Concur. The ASTARS model runs semi-annually, typically in October and 

April. Each time the model runs for a 90-day cycle. At the end of each cycle, a variance 

analysis is conducted, documented and reported. Steps within each cycle include:

1) Policy changes, legislative mandates, industry requests, and organizational changes 

are obtained and incorporated into the model prior to the release of the Online 

Manual Data Report (OMDR);

2) AFS Field Office Managers accomplish an OMDR by validating on-board staff 

and forwarding it to their region for review; 

3) Regional Office and HQ Office Managers validate and forward OMDRs to the 

ASTARS Manager;

4) The ASTARS Manager has 45 days to approve all OMDRs, run the model, 

conduct additional data validation (as needed), review preliminary model results, 

and publish official model results. 

5) Beginning with the June 2013 model results, the ASTARS workgroup will conduct 

a variance analysis comparing each model run with historical models to identify 

trends.
27

Appendix. Agency Comments

6) Each variance analysis will include an in-depth assessment of field offices where 

the on-board staffing level varies widely from the current model projection. 

Findings and recommendations regarding immediate staffing actions will be 

reported to the Director of Flight Standards by the ASTARS Manager. Results 

from the variance analyses will be analyzed to determine needed modifications to 

model algorithms and parameters and additional data quality improvements. 

Findings and their associated fixes are incorporated into enhancements of the 

ASTARS model. These are documented and managed by the Configuration 

Control Board. These findings will be documented and maintained on the 

SharePoint site.

Recommendation 3: Verify inspectors are following existing guidance to update and

maintain the accuracy of databases prior to running iterations of the staffing model.

FAA Response: Concur. To ensure inspectors are following existing guidance, the FAA has 

various data quality review mechanisms to assist with verification: 

1) For the Air Transportation Oversight System, the Data Evaluation Program 

Manager (DEPM) ensures data is of appropriate quality prior to being placed in the 

data repository. The DEPM conducts an ongoing audit of these data and works 

with Aviation Safety Inspectors (ASI) to correct or complete entries;

2) National Program Guidelines Order 1800.56 requires internal audits for Federal 

Aviation Regulations (FAR) 135 and FAR 145 task completion data. Each region 

utilizes the standardized job task guidance created by the Flight Standards Quality 

Assurance Division to implement annual internal data quality audits; and

3) Program Tracking and Reporting Subsystem and Operations Specifications 

Databases have internal automated data error flags that alert the ASI regarding 

some data inconsistencies.

Together, these activities provide useful verification of conformity with existing guidance. AVS 

will further reemphasize the guidance and the need to ensure inspectors are in compliance with 

this guidance in the briefings referenced below in response to recommendation 5.

Additionally, the FAA has various Orders in place to provide guidance to managers and 

inspectors regarding the data quality expectations of each database of record. The following 

excerpts are provided as examples:

“ASIs should report any errors, including instances of missing, inaccurate, or incomplete data, as 

well as broken or improperly functioning SPAS capabilities to the AVS National IT Service 

Desk.”

[FAA Order 8900.1 Vol. 1, Chapter 3, Section 1, E. Error and Problem Reporting]

“[D]ata quality should be a primary focus for all offices. Accurate reporting in VIS is essential... 

Managers and supervisors should establish and maintain an office environment conducive to 

effective data quality control of the VIS.”

[FAA Order 1380.54, Vital Information Subsystem]
28

Appendix. Agency Comments

The FAA believes it has met the intent of this recommendation and will seek closure once the 

briefings have been completed in January 2014.

Recommendation 4: Conduct a comprehensive assessment of the staffing model as compared

to the recommendations of the NRC, assess the quality of the data in the model, and identify

the steps needed to make the staffing model more viable.

FAA Response: Concur. The FAA understands the need to document how ASTARS 

compares with the National Research Council (NRC) recommendations. During the initial 

development of ASTARS, AVS conducted a comprehensive page-by-page and paragraph-byparagraph review of the NRC’s report and recommendations. AVS used the NRC analyses to 

build the requirements document for ASTARS, a part of the AVS SDLC process. 

In order to ensure responsiveness to the recommendations of the NRC, the FAA Office of 

Finance and Management ordered an independent assessment to review the ASTARS program’s 

progress toward meeting the NRC recommendations. This assessment is underway and is 

expected to be completed by December 31, 2013. The FAA will provide a copy of the 

assessment and the plan to address the findings 90 days after receiving the assessment.

Recommendation 5: Implement comprehensive and routine training for managers and 

inspectors on the staffing model.

FAA Response: Partially concur. Initial training for managers and other ASTARS authorized 

users on the staffing model has been developed and is available on the FAA Electronic Learning 

Management System (eLMS), course number 27100119. This training provides initial, entrylevel knowledge for users of the ASTARS system. The course is required to be completed by 

all ASTARS users prior to receiving system access. The completion certificate is submitted to 

the ASTARS Manager with an access request, and eLMS maintains the training completion 

record. This course was last reviewed and revised in May 2011. The next review of this course 

is scheduled for January 2014. 

Currently, in-depth instructions are provided within the ASTARS program with tutorials located 

in the help section of the system. The ASTARS tutorials are comprehensive, and ASTARSauthorized users are encouraged to review them as often as necessary to refresh their 

knowledge. The FAA will explore the feasibility of requiring users to visit the applicable 

tutorials on a regular basis. Additionally, data quality training is part of FAA Course 21423; 

Flight Standards Automation Tools. This course discusses data entry quality and examples of 

bad data entry.

The FAA only partially concurs with this recommendation because it does not agree that routine 

training is necessary for inspectors since they do not have access to ASTARS. ASTARS is a 

management tool and only management and their support staff have access to the system. 

However, inspectors play a critical role in ensuring data quality and must understand this part of 

their role in the daily performance of their duties. AFS intends to increase awareness of this 

issue by briefing our employees. AFS expects to complete these briefings by 

December 31, 2013.
29

Appendix. Agency Comments

Recommendation 6: Establish a comprehensive analyst training program with guidance

clarifying their roles, responsibilities, and training needs, and establish a method to

determine an appropriate number of air carriers per analyst.

FAA Response: Concur. The FAA recognizes the need for a comprehensive analyst training 

program for the Operations Research Analyst (ORA) position. The ORA is a non-management 

position with the responsibility for reviewing inspection results and completing system safety 

analyses. To ensure that multiple areas such as roles, responsibilities, and training needs were 

included in the new ORA Course design, AFS established a training course development 

workgroup. The first workgroup meeting was held in November 2012. The final course delivery 

is scheduled for March 2014, after the completion of the Safety Assurance System (SAS) 

automation (software) development in 2013.

By January 31, 2014, ASTARS will begin formulation of a method to determine an appropriate 

number of air carriers per ORA. Once defined, this requirement will be incorporated into a 

future version of the ASTARS model by December 31, 2014. 

Recommendation 7: Clarify requirements and develop a process to ensure completion of

training on specific air carrier policies and procedures for inspectors who participate in the 

geographic surveillance program.

FAA Response: Concur. Formal training for Inspectors begins with Initial Indoctrination at 

the FAA Academy in Oklahoma City and continues via On-the-Job training and continuing online training in eLMS. Policy documents outside of the ASTARS program stipulate inspector 

training requirements for the geographic surveillance program. 

The air carrier specific training requirement is intended for those ASIs assigned to a single 

certificate holder to enhance their knowledge of all aspects of their assigned air carrier. This is 

necessary to ensure that ASIs are capable of assessing the impact of changes within the air 

carriers programs and procedures. This differs from the geographic program where ASIs are 

assigned to accomplish en-route, spot, and ramp inspections. These inspections focus on 

general regulatory and safety compliance, not on operational aspects unique to a specific carrier. 

Air carrier specific training is not required to accomplish these inspections. 

With the implementation of SAS, the scope of the geographic program will be extended. All 

ASIs assigned to Volume 14 of the Code of Federal Regulations (CFR) Parts 121, 135, and 145 

certificate holders will be trained in SAS and will be able to accomplish additional inspection 

activities for these CFR parts. The SAS automation contains additional functionality to allow a 

requesting office to attach supporting documentation, manual excerpts, etc., to enhance the 

Geographic ASIs ability to accomplish the requested inspection activity. SAS Phase IIA key 

site testing is scheduled to begin in 2014 and full operational deployment for 14 CFR Parts 121, 

135, 145 is anticipated before September 30, 2015.

Attachment
30

Appendix. Agency Comments

ASTARS

FSDO/CMO/IFO Staffing Model

The Next Evolution

Background

The FAA developed the AVS (Aviation Safety) Staffing Tool and Reporting System (ASTARS) 

to meet the Congressional mandate for AVS to implement and incorporate appropriate staffing 

standards and in response to the needs identified in the National Academy of Sciences report, 

Staffing Standards for Aviation Safety Inspectors. FAA Flight Standards (AFS) assembled an 

integrated project team to develop this new demand-driven sufficiency staffing model. This team 

included FAA representatives and the contractor support team. This team concept continues to 

this day. The goals for the project are:

 Design a demand-driven staffing model for AVS using a combination of analytical 

supply/demand modeling, business process improvement, change management, and 

project management methods

 Establish an on-board supply baseline for AFS and eventually AVS

 Implement a staffing tool that automates the new staffing model and determines 

appropriate staffing requirements for AVS

 Introduce a new way to plan for the effective utilization of AVS staffing resources to 

ensure proper safety oversight

Version One of ASTARS modeled only Flight Standards District Offices (FSDO), Certificate 

Management Offices (CMO) and International Field Offices (IFO). It began operating on 

October 1, 2009. Since then ASTARS has been expanded to include forecasts for:

 FAA Safety Team (FAASTeam) offices

 Aircraft Evaluation Group (AEG) offices

 Aviation Safety Inspectors assigned to the Aircraft Certification Service (AIR)

ASTARS has a goal of continuing improvement. Now that ASTARS has been operating for 

almost three years, it is time to take it to the next evolution.

Areas for Improvement

 Original modeling technique used was heavily dependent upon use of statistics for 

predicting the frequency of work performed based upon configuration based demand 

drivers. This was because the primary databases ASTARS needed to use did not 

originally contain data of sufficient quality to make use of a more dynamic and datacentric modeling approach
31

Appendix. Agency Comments

 Demand drivers were derived by experimentation in an effort to establish statistically 

significant relationships for 130+ CFR Part/peer group, work type and specialty 

combinations

 The resulting demand driver equations were calibrated using certificate holder 

configuration and work activity data that turned out to have numerous underlying issues

 Nominal times were derived from surveys since reliability of the LDR data was suspect 

due to overly complex charge structures and inconsistent use of the proper charge codes 

for work performed

 Model results occasionally exhibit excessive sensitivity to variations in configuration data

 Model results occasionally yield highly counter intuitive results based upon changes to 

the configuration data due to non-causality of the statistically significant relationship

 The model is not fully transparent. It is difficult to explain model results to users in terms 

they can understand

 The impact of other work performed by nonsupervisory ASIs is not forecast and is but 

poorly accounted for in the model 

 AFS organization is considering making numerous changes to how work is to be 

performed by nonsupervisory ASIs. These possible changes include, but are not limited 

to:

 Implementation of the anticipated Safety Analysis System (SAS)

 Redesigning the FAASTeam concept and purpose

AFS FSDO/CMO/IFO Model Enhancement Goals

 Balance underlying principles with tractability and simplicity, while striving for 

appropriate usage of data, management judgment and feedback

 Use a combination of statistical, deterministic and judgment modeling techniques to 

increase accuracy

 Make use of improved historical LDR data from Fiscal Years 2011 and 2012 

 Derive for nominal times and ranges from real data

 Inherently provide for trending to aid future projections in a manner similar to the model

used for AIR

 Allow for significant modifications in the way FSDOs, CMOs and IFOs plan work based 

upon changes in business processes 

 Leverage managerial knowledge of data in CSOP for initial certification efforts and 

subsequent impact on certificate oversight efforts

 Account for Geographic and National Resource Inspector programs

 Provide for leveraging inspectors resources to perform inspector work

 Determine the ability to incorporate indices of possible understaffing in particular offices, 

such as:

 Backlogged work

 Applicants waiting for certification. (CSOP and elsewhere) 

 Work that was not performed due to having insufficient resources

 Overtime and compensatory hours charges by ASIs

 Administrative work performed by inspectors

 Inspector work performed by managers

 Provide for other items that consume a significant percentage of an inspectors 

availability, such as:
32

Appendix. Agency Comments

 On demand and other non-safety related work

 Travel

 Training

 Leave

 Provide for known or projected supply side staffing transitions and retention adjustments

 Individuals who are onboard and basically charged full time to training but will be 

available in subsequent years to perform as an ASI

 Fractional year contributions of ASIs

 Retirement/transfers

 Anticipate incorporating organizational plans under consideration for the following: 

 Appropriate number of PIs vs. ASIs

 Specialty generalization: OPS and AW vs. OPS, MX and AV

 AC vs. GA

 Continue to include business rules for personnel other than nonsupervisory ASIs

 Organizational minimums, e.g. one manager per office, CFR Part 121 dedicated 

principals, etc.

 Policy exceptions, e.g. large HEMS, protected positions, etc.

 Management/supervisory span of control

 Other support, e. g. ASA, administrative, etc.

 Allow for fundamental variations in behavioral characteristics of different types of 

offices, i.e. FSDO vs. CMO vs. IFO

 Consider including the concept of sufficiency as a computation proxy for quality of work 

performed

Schedule for Completion of this Evolution

The schedule is a living document. It resides on the AFS ASTARS Library SharePoint site. The 

URL is: 

https://avssharepoint.faa.gov/afs/100/160/staf/ASAM_Library/Large%20File%20Transfer%20to%20and

%20from%20Volpe/Forms/DispForm.aspx?ID=135&Source=https%3A%2F%2Favssharepoint%2Efaa%2

Egov%2Fafs%2F100%2F160%2Fstaf%2FASAM%5FLibrary%2FLarge%2520File%2520Transfer%2520to

%2520and%2520from%2520Volpe%2FForms%2FAllItems%2Easpx%3FRootFolder%3D%252fafs%252f

100%252f160%252fstaf%252fASAM%255fLibrary%252fLarge%2520File%2520Transfer%2520to%2520a

nd%2520from%2520Volpe%252fASTARS%2520Nominal%2520Time%2520Reference%2520Documentat

ion%26View%3D%257bF28F0026%252dAB4B%252d4311%252dAD02%252dEFA073926D9F%257d&Ro

otFolder=%2fafs%2f100%2f160%2fstaf%2fASAM%5fLibrary%2fLarge%20File%20Transfer%20to%20a

nd%20from%20Volpe%2fASTARS%20Nominal%20Time%20Reference%20Documentation