Office of Inspector General

Audit Report

FAA NEEDS TO STRENGTHEN ITS RISK 

ASSESSMENT AND OVERSIGHT APPROACH FOR 

ORGANIZATION DESIGNATION AUTHORIZATION AND 

RISK-BASED RESOURCE TARGETING PROGRAMS 

Federal Aviation Administration

Report Number: AV-2011-136

Date Issued: June 29, 2011
U.S. Department of

Memorandum

Transportation

Office of the Secretary

of Transportation

Office of Inspector General

Subject: INFORMATION: FAA Needs To Strengthen Its 

Risk Assessment and Oversight Approach for 

Organization Designation Authorization and RiskBased Resource Targeting Programs

Federal Aviation Administration

Report Number AV-2011-136

Date: June 29, 2011

From: Jeffrey B. Guzzetti

Assistant Inspector General 

 for Aviation and Special Program Audits

Reply to 

Attn. of: JA-10

To: Federal Aviation Administrator

The Federal Aviation Administration (FAA) is responsible for overseeing 

numerous aviation activities designed to ensure the safety of the flying public. 

Recognizing that it is not possible for FAA employees to personally oversee every 

facet of aviation, public law1

FAA created the Organization Designation Authorization (ODA) program in 2005 

to standardize its oversight of organizational designees.

allows FAA to delegate certain functions, such as 

approving new aircraft designs, to private individuals or organizations. Designees 

perform a substantial amount of critical work on FAA’s behalf—for example, at 

one aircraft manufacturer, they made about 90 percent of the regulatory 

compliance determinations for a new aircraft design.

2

Representative Daniel Lipinski requested that we review these two aircraft 

certification initiatives. He expressed concern that ODA allows companies to

 ODA was fully 

implemented in November 2009 when FAA required all delegated organizations to 

transition to the new ODA policy. FAA also deployed its Risk Based Resource 

Targeting (RBRT) system in 2007 to assist FAA engineers in deciding which 

aircraft certification projects represent the highest risk and therefore need more 

FAA oversight. 

 1 49 U.S.C. § 44702. 2 Organizational designees are companies (e.g., aircraft manufacturers) that FAA has approved to perform certain 

functions on its behalf, such as determining compliance with aircraft certification regulations. The organization is 

responsible for overseeing the employees who perform the delegated functions.
2

choose individuals who perform work on behalf of FAA3 with little or no FAA 

oversight and that RBRT precludes certification engineers from reviewing projects 

that are deemed low or medium risk. He was also concerned that these two 

programs were not in compliance with current laws and regulations.

4

We conducted this review between August 2009 and May 2011 in accordance with 

government auditing standards prescribed by the Comptroller General of the 

United States. We visited aircraft manufacturers and FAA Aircraft Certification 

Offices (ACO)

 

Accordingly, our audit objectives were to determine (1) the role FAA plays in the 

selection process for individuals who perform work under the Agency’s ODA 

program, (2) the adequacy of FAA’s oversight of the program, and (3) the 

effectiveness of FAA’s RBRT program. 

5

RESULTS IN BRIEF

for both small and transport category airplanes. Due to the nature 

of Representative Lipinski’s request, we focused on delegated organizations that 

approve new aircraft designs (type certification) and changes to existing designs 

(supplemental type certificates). An air transportation consultant assisted us in 

assessing the effectiveness of ODA and RBRT. Exhibit A details our scope and 

methodology. Exhibit B lists the entities we visited or contacted.

Under the ODA program, FAA has significantly reduced its role in approving

individuals who perform work on FAA’s behalf by further delegating this 

approval to private companies (e.g., aircraft manufacturers). Under previous 

forms of organizational delegation, FAA approved each appointment of personnel 

working for these companies. Now, once the Agency approves the company’s 

selection process, ODA company representatives select these personnel, known as 

unit members, without FAA concurrence. While FAA maintains some 

involvement with the selection process during an ODA holder’s first 2 years, it is 

unclear how FAA will be involved beyond that timeframe. Because FAA has not 

provided clear, written guidance on how to oversee unit member appointments, 

there are inconsistencies in how ACOs interpret FAA’s role and track unit 

members and in how manufacturers select those personnel. For example, only 

three of the five ACOs we visited consulted an FAA database to pre-screen 

prospective unit members’ performance histories, and FAA engineers in the field 

expressed confusion about whether this check would continue beyond an ODA’s 

first 2 years. With less FAA involvement in the selection process, there is also 

potential risk that an ODA company could appoint unit members with inadequate 

qualifications or a history of poor performance to approve certification projects. 

 3 Under ODA, these individuals are known as unit members. 4 Based on our review, we did not find any conflict with 49 U.S.C. § 44702 and 14 C.F.R. Part 21. 5 FAA offices responsible for engineering oversight of aircraft manufacturers.
3

While the ODA program is relatively new, we identified potential vulnerabilities 

in FAA’s oversight and training. Beyond the change in the unit member selection 

process, FAA’s ODA oversight methods (e.g., initial project review, site visits,

and technical evaluations) are similar to those used for past forms of 

organizational delegations. Therefore, FAA will likely face many of the same 

challenges with ODA. Past FAA audits discovered “after the fact” that delegated 

organizations had either neglected a critical rule or did not properly demonstrate 

compliance, calling into question how adequately FAA reviews new engineering 

project plans submitted by delegated organizations. For example, during initial 

project review, an FAA engineer failed to detect that a manufacturer’s certification 

plan did not demonstrate compliance with specific aviation regulations governing 

design and construction of aircraft flight controls. Under ODA, FAA engineers 

will also have expanded enforcement responsibilities, but the Agency has not 

ensured that they are adequately trained to perform these duties. As a result, FAA 

engineers may not detect and enforce all regulatory noncompliances.

Finally, RBRT has not been effective for measuring risk and directing FAA 

engineers’ oversight efforts to high-risk projects because it relies on subjective 

input from engineers, does not contain detailed data, and has experienced repeated

technical difficulties. Engineers reported numerous problems with the system, 

including a tendency to identify projects as low risk regardless of inputs that 

suggested higher risk factors, such as a lack of company experience with the 

design. FAA removed RBRT from active use in August 2009 to undergo revisions

identified during its pilot phase. FAA plans to reintroduce RBRT in late fiscal 

year (FY) 2011. In the interim, FAA engineers will continue to determine which 

projects to review using subjective judgment. 

We are making a series of recommendations to FAA to improve its oversight of 

ODA programs and the RBRT system.

BACKGROUND 

Historically, FAA has relied on a variety of organizational or individual designee

programs to meet its responsibility to hold the aviation industry accountable to its 

safety standards. A designee can be a person or an organization who witnesses

inspections or tests on FAA's behalf. For example, one type of designee is known 

as a Designated Engineering Representative (DER).6

 6 DERs are non-FAA employees authorized to approve information related to aircraft structure, engines, power plants, 

propellers, flight characteristics, systems, and equipment on behalf of FAA.

 To gain FAA approval of a 

new aircraft design, a manufacturer must demonstrate compliance to hundreds of 

detailed Federal Aviation Regulations. FAA has the option to thoroughly examine 

test data, accompanying analysis, and conclusions of DER approvals; spot check a 

few calculations or data points; or simply accept the report at face value. 
4

Organization Designation Authorization: Since 1956, FAA has developed 

various forms of organizational delegation to meet specific needs. In 2005, FAA 

created the ODA program to consolidate these types of organizational delegations 

under one program and standardize oversight.7

Figure 1. Development of Organizational Delegation

 By November 2009, all

67 companies that had applied for ODA had completed the transition as required 

by FAA. As of November 2010, there were 84 authorizations at air carriers, 

aircraft manufacturers, repair and maintenance facilities, and aircraft modification 

companies. Figure 1 illustrates the development of organizational delegation.

Source: OIG analysis of various FAA documents

By implementing ODA, FAA has reduced the numbers of both organizational 

delegations and individual designees. According to FAA, managing an 

organization is more efficient than managing the activity of many individual 

designees. Table 1 details changes in the number of individual and organizational 

designees since 2004.

 7 The ODA program is authorized in the Code of Federal Regulations, 14 C.F.R. Part 183, and FAA steps for 

approving ODA programs are governed by internal FAA policies. 

Organization 

Delegation 

Authorization 

(ODA)

DAS allowed 

manufacturers, repair 

stations, or air carriers 

to change aircraft or 

component designs 

and return them to 

service on FAA’s 

behalf.

DOA allowed 

approved 

manufacturers to 

conduct certification 

functions on FAA’s 

behalf.

1956

Designated 

Alteration Station 

(DAS)

1965

Delegation 

Option 

Authorization 

(DOA)

1978 2005-2009

Special Federal 

Aviation 

Regulation 36 

(SFAR 36)

SFAR 36 allowed a 

repair station, air 

carrier, or commercial 

operator to develop 

major repairs not 

specifically approved

by FAA.

ODA combined these prior 

forms of delegated authority 

under one program. The ODA 

transition was completed in 

November 2009.
5

Table 1. Change in the Number of Engineering and Manufacturing Designees

Individual Designees May 2004 November 2010

Designated Engineering Representatives 2,725 2,052

Designated Manufacturing Inspection 

Representatives 1,249 1,181

Designated Airworthiness Representatives 359 422

TOTAL 4,333 3,655

Organizational Designees May 2004 November 2010

Organizational Designated Airworthiness 

Representatives 86 0

Designated Alteration Station 31 0

Delegation Option Authorization 6 0

Special Federal Aviation Regulations No. 36 12 0

Organization Designation Authorization 0 84

TOTAL 135 84

Source: OIG analysis of FAA data

Risk Based Resource Targeting: In September 2007, as another way to leverage 

limited FAA engineering resources, FAA implemented a policy to allow RBRT in 

deciding which new engineering projects to review. RBRT is a process that 

evaluates the risk associated with non-compliance with FAA regulations that 

govern six “business processes” FAA oversees, including aircraft or aircraft 

component design.

Using a series of assessment questions, FAA inspectors and engineers rate the 

likelihood and severity of the risk of the organization failing to comply with FAA 

regulations. This risk assessment is expressed as a number from 1 to 5 and 

characterized as low, medium, or high. The higher the risk, the more direct FAA 

oversight of the project or activity is recommended. 

UNDER ODA, FAA HAS DELEGATED A SIGNIFICANT PORTION 

OF ITS PERSONNEL SELECTION AUTHORITY TO PRIVATE 

COMPANIES 

FAA’s role in selecting unit members will gradually decline under the 2009 ODA 

policy, as ODA holders can start selecting unit members without FAA’s input 

2 years after receiving their approval. Also, we identified inconsistencies in how 

ACOs interpret policy on unit member screening requirements and in how they 

track unit members. Further, at one of the three manufacturers we visited, where 

FAA allowed an ODA company to proceed with self-selection, problems arose 

that demonstrate the challenges FAA may face as its role in unit member selection 

declines. 
6

FAA’s Role in Selecting Personnel Who Perform Work on Its Behalf 

Will Decline Over Time 

Under the ODA program, FAA has significantly reduced its role in approving 

individuals who perform work on its behalf by delegating this approval to private 

companies (e.g., aircraft manufacturers). Under previous forms of organizational 

delegation, FAA approved each appointment of personnel working for these 

companies. Now, ODA company representatives select these personnel, known as 

unit members, without FAA concurrence once the Agency approves the 

company’s selection process. FAA’s ODA policy calls for a 2-year period before 

the ODA holder can self-select unit members. However, it also permits FAA to 

allow an ODA holder to proceed with self-selection sooner if the company has 

demonstrated a capability to do so.

Implementation and Interpretation of FAA’s Role in the Unit Member 

Selection Process Is Inconsistent Across FAA

FAA’s ACO personnel are not consistently pre-screening prospective unit 

members’ performance histories and have different interpretations of how long to 

continue pre-screening after the ODA holder is granted approval. We found this is 

largely due to a lack of clear guidance on FAA’s ODA policy.

8

Finally, FAA’s ACOs are not consistently tracking (i.e., collecting and 

maintaining data on) unit member performance because there is no FAA policy 

requiring them to do so. One ACO we visited tracks ODA unit members by name

in the DIN, while others did not track them at all. FAA Headquarters officials 

state that they would not allow an individual designee with poor past performance

to serve as a designee or ODA unit member. However, if these ODA employees

are not tracked by name, it is unclear how FAA or the ODA will know if a 

prospective unit member has a poor performance history. This lack of visibility 

into the background of prospective unit members will further diminish FAA’s 

FAA engineers 

and managers at the five ACOs we reviewed expressed confusion over the need 

for pre-screening. First, only three of the five ACOs used FAA’s Designee 

Information Network (DIN) to pre-screen unit members’ performance histories. 

The DIN is a system for tracking all aircraft certification designees and 

delegations so ACOs can look for any prior negative experience with the 

individual in question and share the information with the ODA holder. Second, 

some FAA program managers we met with asserted that pre-screening will 

continue beyond the 2-year phase while FAA engineers in the field stated that prescreening is only required for the first 2 years. If proposed unit members are not 

screened, an individual with a history of poor performance as a DER could be 

authorized to perform critical aircraft certification functions. 

 8 FAA Order 8100.15 contains the policy guidance for implementing ODA.
7

ability to conduct effective oversight of ODA companies and their unit member 

selection process.

FAA’s Process Does Not Prevent ODAs from Selecting Unit Members

With Past Performance Problems

While FAA has the authority to remove unit members based on performance 

issues, its guidance does not clearly define how individual offices should do so. 

We identified instances in which FAA did not act quickly to remove questionable 

unit members and appointed an individual to a key ODA position despite FAA 

engineers' objections. FAA is losing the direct supervisory connection it 

historically relied on with DERs; as a result, there is a greater risk that individuals 

with a history of poor performance could continue to carry out critical aircraft 

certification functions.

For example, FAA engineers cited troubling performance issues with a unit 

member and insisted that the manufacturer remove his authority to perform work 

under the ODA. FAA engineers were concerned with the unit member’s integrity 

because he was advocating a position that directly opposed FAA rules on an 

aircraft fuel system in favor of the manufacturer. Specifically, the unit member 

determined that an aircraft type was in compliance with FAA fuel system rules but

ignored other language in the rules that, according to FAA engineers, made it clear 

that the aircraft type did not comply. According to ODA regulations, when acting 

as a representative of FAA, the ODA is required to perform in a manner consistent 

with FAA policies, guidelines, and directives. When performing a delegated 

function, designees are legally distinct from and act independent of the 

organizations that employ them. The manufacturer disagreed with the FAA 

engineers’ assertions but, after nearly a year of discussions, temporarily suspended 

the unit member’s authority. During that year, the unit member continued to 

approve certification data on FAA’s behalf. After our visit, FAA took action to 

permanently remove the unit member.

In another instance, the manufacturer sought input from FAA on whom to appoint 

to a key ODA position. However, according to FAA engineers, the person that 

FAA managers ultimately approved was the one who received the most objections 

from the FAA engineering staff. 

The manufacturer also promoted a prior DER to a first-level ODA management 

position despite a considerable record of negative feedback from FAA. FAA 

performance evaluation records noted that he showed a consistent lack of integrity, 

unsound judgment, and an uncooperative attitude towards FAA. FAA noted that 

this person should never be considered for appointment as a representative or 

authority in any assignment on behalf of FAA or within the company’s delegated 

organization. Yet, according to FAA engineers we spoke with, the company 
8

hoped to eventually promote this individual to ODA Administrator—the company 

employee with overall responsibility for the ODA and its unit members.

9

FAA HAS NOT ADDRESSED OVERSIGHT OR TRAINING 

WEAKNESSES THAT COULD IMPACT THE ODA PROGRAM 

 

Weaknesses in FAA’s oversight of past forms of delegated authority and in ODA 

transition training could prevent successful ODA implementation. Our review 

indicated that FAA audits10

OIG Review of Prior FAA Audit Findings Show Lapses in Initial

Oversight of Aircraft Design and Modification

of prior forms of FAA’s delegated authority (before 

ODA) revealed lapses in FAA’s initial review of engineering projects. Inadequate

ODA transition training resulted in FAA engineers being unprepared to carry out 

their expanded oversight and enforcement responsibilities under ODA. As a 

result, FAA engineers may not detect and enforce all regulatory noncompliances. 

FAA’s past audits of manufacturers that held prior forms of delegated authority—

which are now ODA holders—identified instances of non-compliance with 

regulations after these companies had certified aircraft components as compliant. 

Our analysis of FAA audit findings from 2005 to 2008 disclosed 45 instances that 

indicated FAA had not carefully reviewed the certification plan in advance. This 

upfront review of the certification plan is a key component of FAA oversight of 

engineering projects. For example:

• A manufacturer did not have evidence that critical tests on a new aircraft 

engine component were ever performed. 

• A manufacturer’s certification plan did not indicate that it complied with 

specific aviation regulations governing design and construction of aircraft 

flight controls. 

• A manufacturer did not comply with regulations addressing the supplemental 

oxygen system for passengers and crew or emergency evacuation and escape 

routes. 

These “after the fact” findings raise questions about the effectiveness of FAA’s 

initial oversight of certification plans that the ODA holder submits. FAA’s ODA 

policy requires engineers to perform one supervisory visit to companies they 

oversee per year and a more comprehensive technical audit every 2 years. 

 9 As of January 2011, this individual has not been appointed as the ODA Administrator. 10 FAA completed the transition to ODA in November 2009. Since technical audits of ODA performance are 

conducted every 2 years, we did not have enough examples under ODA from which we could draw conclusions 

about the quality of FAA oversight.
9

Therefore, it can be a year or longer before FAA engineers perform additional 

oversight through an annual visit or a technical audit.

FAA Has Not Adequately Prepared Its Engineers To Carry Out Their 

Expanded Enforcement Responsibilities Under ODA

FAA has not ensured that its engineers are adequately trained to manage oversight 

of the ODA program. Under ODA, FAA’s certification engineers have increased 

capability to cite companies with violations of the Federal Aviation Regulations 

when an ODA holder does not comply with its FAA-approved ODA procedures 

manual. FAA engineers can now initiate an enforcement action for noncompliance with the procedures manual, which could lead to civil penalties. This 

is an important change from prior forms of delegated authority in which most noncompliance issues related to the procedures manual were not regulatory violations. 

However, FAA managers and engineers cited concerns to us that they never 

received training or that FAA’s training course does not fully inform them of their 

responsibilities under ODA. For example, engineers stated that they had no past 

experience in compliance and enforcement activities and were unsure of their role 

in any enforcement activities. One engineer even stated that he does not have any 

compliance and enforcement responsibility under ODA. Also FAA’s training is 

geared toward Flight Standards and manufacturing inspectors—not engineers, who 

will be overseeing ODA holders. Engineers working within FAA’s new oversight 

organization for large ODA holders with multiple certification locations also 

expressed concerns. These engineers, as well as ACO engineers, told us that this 

new oversight concept has been poorly communicated, leaving them uncertain

about their role. 

RBRT HAS NOT BEEN EFFECTIVE IN ASSESSING 

ENGINEERING PROJECT RISK

RBRT does not contain detailed data, such as accidents, to assess the risk of 

noncompliances with regulations, and engineers do not accept the RBRT process

due to numerous technical difficulties. RBRT contains risk assessment formulas 

based on experts’ opinions rather than an objective, automated analysis of accident 

or incident data. As a result, RBRT has not been effective for measuring risk and 

directing engineers’ oversight efforts to higher risk projects. Even if FAA is able 

to successfully address these shortcomings, it still must train and prepare engineers 

to effectively use RBRT.
10

RBRT Does Not Include Objective, Detailed Data

RBRT does not meet seven of nine risk assessment principles in FAA’s own risk 

assessment policy (see table 2).11

Table 2. FAA Risk Assessment Principles

 One principle is for the system to include all 

relevant data available. However, there is no automated mechanism to leverage 

data outside of the user’s potentially subjective knowledge. For example, RBRT 

does not contain any data related to actual aviation accidents resulting from 

manufacturing defects. A fully developed risk-based system would be able to 

retrieve data directly from FAA’s own accident and incident databases. 

FAA Risk Assessment 

Principles

RBRT Consistent 

with Principle Comment

Scientifically Objective No RBRT in its current state is purely subjective

Unbiased Yes If the user is unbiased

Include All Relevant Data 

Available No No automated mechanism to leverage data outside users’ personal knowledge

Use Default/Conservative 

Assumptions Only if Situation 

Specific Info Not Available

No

The same risk level is assigned to all 

regulations with no differentiation for those 

with direct and substantial impact to safety

Reasonably Detailed and 

Accurate No

An accurate assessment is not possible 

given that currently all regulations are 

assigned the same risk rating

Address Both Severity and 

Likelihood Yes none

Quantitative to the Maximum 

Extent Possible No

RBRT represents a quantitative processing 

of subjective judgment; however, there is 

little or no data-driven assessment capability

Flexible No

The RBRT tool requires the same tedious 

data input for all projects regardless of 

relevance to safety

Assumptions Documented No Users are clearly not aware of assumptions 

fundamental to RBRT design

 Source: OIG and consultant analysis of FAA data

Our analysis as well as that of our external consultant concluded that RBRT has 

data shortcomings. In addition to our determination that RBRT is driven by 

subject matter experts’ opinion rather than objective data, our consultant found 

that RBRT risk assessments are of limited value in differentiating projects by 

safety risk for resource targeting. For example, when originally introduced, the 

system did not differentiate the potential safety impacts of noncompliance with 

various regulations.12

 11 FAA Order 8040.4 defines FAA’s principles for risk assessment.

 Therefore, RBRT treated the potential impact of noncompliance with the regulation governing design of critical flight controls the 

same as non-compliance with the regulation requiring installation of a no-smoking 

decal. In another example, we reviewed a RBRT risk assessment that was rated as 

12 According to FAA officials, the system that will be reintroduced in late FY 2011 will have this capability.
11

low risk by RBRT even though the company involved in the assessment was a 

new company that would require a higher level of FAA oversight.

Further, FAA’s risk assessment policy calls for a plan that identifies specific

hazards that may be encountered in the overall certification process, analyzes the 

likelihood of their occurrence, and determines their severity. However, RBRT 

uses vague hazard statements rather than describing specific hazards that could 

endanger an aircraft, such as sudden depressurization or lightning strike.13

RBRT Has Experienced Significant Technical Difficulties

 For 

example, the hazard statement for aircraft design is, “If the [aircraft design] 

process is not effectively completed, it may result in a non-compliant design of an 

aviation product (aircraft, engine, or propeller) that may contribute to the cause of 

a fatal accident.” While identifying all hazards that put an aircraft at risk is an 

enormous task, it would greatly enhance RBRT’s effectiveness. Although FAA 

has acknowledged that a risk assessment tool based upon objective data is superior 

to the current subjective model, it does not expect to populate RBRT with more 

objective data before late 2014 to 2015 at the earliest. According to FAA officials, 

their plan to prioritize regulations in the next release of RBRT will better define 

risks and hazards.

FAA did not ensure that the RBRT tool was fully functional before requiring its 

use. FAA mandated the use of RBRT for all certification activity in August 2008;

however, the Agency poorly executed its deployment. Once it was in widespread 

use by certification engineers, the RBRT tool experienced a number of technical

difficulties, including slow system functionality and system 

“freezes.14

 13 FAA defines safety hazards as a condition, event, or circumstance that could lead to or contribute to an unplanned or 

an undesired event.

” Engineers stated it could take weeks to months to complete an RBRT 

risk assessment. After being removed and reintroduced, RBRT continued to 

experience technical difficulties, frustrating FAA engineers and causing FAA to 

take the system back offline to undergo modifications. RBRT is currently not in 

use, and FAA plans to reintroduce RBRT in late FY 2011. Until FAA deploys 

RBRT, FAA engineers will continue to determine which projects to review using 

only their subjective judgment. Figure 2 describes the timeline of RBRT’s 

deployment in greater detail.

14 According to FAA officials, the issues with slow functionality and system freezes were largely due to the lack of 

Information Technology (IT) capacity within FAA. Until this issue is resolved at the Agency level, IT programs will 

not run at optimum efficiency.
12

Figure 2. Timeline of RBRT Deployment

Source: OIG analysis of FAA data

FAA Has Not Effectively Prepared Engineers To Use the RBRT Tool

FAA did not effectively train engineers and managers on RBRT, which resulted in 

confusion among engineers we interviewed on how to implement RBRT. For 

example, FAA’s intent is to allow low-risk projects to be approved without a 

required data review—a concept known as “applicant showing only.” However, 

FAA did not adequately train engineers on this concept. The initial training given 

to engineers using RBRT consisted of briefing slides that did not fully address the 

engineers’ concerns regarding the level of involvement expected for each risk 

level. Engineers told us that they would never accept a project proposal without 

reviewing data.

CONCLUSION 

As the aviation industry continues to expand, FAA must continue to adapt its role 

in oversight efforts, including the use of designees to perform work on its behalf. 

While FAA’s effort to reduce the number of individual designees is a good step

toward efficiency, decreasing its involvement in selecting unit members is not 

without risk. Unless FAA has the necessary training and tools in place to conduct 

effective oversight, it cannot be assured that ODA organizations are fully 

complying with FAA’s safety requirements or that unit members are qualified to 

perform critical safety tasks. To best target limited oversight resources to the 

highest risk projects, FAA must continue efforts to develop a sound risk 

assessment process and inform personnel of how to utilize the system. 

RECOMMENDATIONS 

We recommend that FAA:

1. Revise its ODA policy to require a full 2-year transition for unit member selfselection.

FAA pilots 

RBRT in four 

offices. 

Engineers report 

continued 

difficulty.

FAA mandates 

use of RBRT 

for aircraft 

certification 

activity.

FAA makes use 

of RBRT optional 

due to technical 

difficulties.

FAA removes 

RBRT from use 

due to technical 

problems.

FAA again removes 

RBRT from use to 

make changes and 

reintroduce in 2011.

2008 Aug Sep Oct 2009 Apr Aug
13

2. Develop explicit guidance on the process to remove an ODA unit member in a 

timely fashion and require all ODA holders to include this standardized 

removal process in their approved ODA procedures manual.

3. Track unit member appointments in its Designee Information Network (DIN) 

database or another method in order to identify unit members with known 

performance issues and require engineers to cross-check names with the 

database beyond the first 2-year required timeframe.

4. Develop enforcement training and guidance that is pertinent to the unique 

requirements of the certification engineering discipline. 

5. Improve the new oversight structure for large ODA holders by: 

a. developing training for FAA engineers and disseminating comprehensive 

procedures on the new oversight structure for large ODA holders.

b. assessing the effectiveness of the new oversight structure before 

implementing it at other large ODA holders.

6. Improve the RBRT tool by: 

a. enhancing the risk assessment process so that it uses more automated data, 

such as accidents resulting from manufacturing defects, to accurately

differentiate higher risk projects that likely pose the most safety risk.

b. thoroughly testing and validating it to ensure that it is fully functional. 

c. ensuring that engineers are properly trained before requiring its use and

relying upon its results.
14

AGENCY COMMENTS AND OFFICE OF INSPECTOR GENERAL

RESPONSE

We provided FAA with our draft report on May 3, 2011, and received its response 

on June 1, 2011. FAA’s response is included in its entirety as an appendix to this 

report. FAA concurred or partially concurred with all of our recommendations, 

and its response meets the intent of most of them. However, we are concerned 

with the timeframe for issuing new guidance on the process for approving and 

tracking ODA unit members. Additionally, we are requesting that FAA expand its 

planned actions for recommendation 3 and clarify information regarding its

actions planned for recommendation 5a. 

Specifically, in response to recommendations 1, 2, and 3, FAA proposes to revise 

its guidance regarding the ODA transition period, procedures for unit member 

removal, and the requirement to cross-check names with FAA databases by 

September 30, 2012. However, because these recommendations affect the core 

tenets of the ODA program, we believe that FAA should issue interim guidance to 

immediately enhance its oversight until the Agency is able to issue permanent 

changes to its guidance. Additionally, for recommendation 3, FAA stated that it is 

impractical to maintain all ODA unit members in an FAA database but agreed to 

track unit members that have been removed due to performance-related issues. 

However, as we reported, unit members can experience performance issues for a 

prolonged period before removal. Given that poor performing unit members could 

leave on their own before removal, FAA should expand its action by including 

ODA unit members that are experiencing performance issues, but have not yet 

been removed. Accordingly, we are requesting that FAA provide additional 

planned actions and target dates for recommendations 1, 2, and 3 and consider 

these recommendations open and unresolved. 

For recommendation 5a, FAA states that it plans to revise its policy to clarify that 

new offices are responsible for developing and conducting training for their staff. 

However, the target date for this action is not clear. Additionally, given that FAA 

is making individual offices responsible for enhancing training and procedures, we 

request that the Agency clarify how it will ensure consistency in training and 

procedures across these new offices. We are requesting that FAA likewise 

provide additional planned actions and target dates for this recommendation and 

consider it open and unresolved as well. 

ACTIONS REQUIRED

FAA’s planned actions and target dates for recommendations 4, 5b, and 6 are 

responsive, and we consider these recommendations resolved but open pending 

completion of planned actions. The remaining recommendations remain 
15

unresolved pending further action by FAA. We request that FAA provide, within 

30 days of this report, additional actions to resolve recommendations 1, 2, and 3, 

along with estimated target completion dates. Also, we request that FAA clarify 

its target date and actions to enhance training and procedures for recommendation 

5a. We appreciate the courtesies and cooperation of FAA representatives during 

this audit. If you have any questions concerning this report, please call me at 

(202) 366-0500 or Robin Koch, Program Director, at (404) 562-3770.

#

cc: FAA Associate Administrator for Aviation Safety

Director, Aircraft Certification Service

Anthony Williams, AAE-001

Martin Gertel, M-1
16

Exhibit A. Scope and Methodology

EXHIBIT A. SCOPE AND METHODOLOGY 

We conducted this audit in accordance with generally accepted government 

auditing standards prescribed by the Comptroller General of the United States. 

Those standards require that we plan and perform the audit to obtain sufficient, 

appropriate evidence that provides a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that the evidence obtained 

provides a reasonable basis for our findings and conclusions based on our audit 

objectives. We conducted this review between August 2009 and May 2011. We

used the following scope and methodology in conducting this review.

To determine the role FAA plays in the selection process for individuals who 

perform work under the Agency’s ODA program, assess the adequacy of FAA’s 

oversight of the program, and evaluate the effectiveness of FAA’s RBRT program, 

we performed audit work at 5 of 10 FAA Aircraft Certification Offices (ACOs) 

and manufacturers for both small and transport category airplanes. Due to the 

nature of Representative Lipinski’s request, we focused on delegated organizations 

that approve new aircraft designs (type certification) and changes to existing 

designs (supplemental type certificates). 

We interviewed aircraft certification managers and engineers at FAA 

Headquarters, the Mike Monroney Aeronautical Center in Oklahoma City, and 

ACOs. We met with management officials responsible for administration of their 

respective ODA programs at Bombardier-Learjet, Cessna Aircraft, and the Boeing 

Company and interviewed individual unit members who perform work under each 

company’s ODA. We analyzed unit member qualifications and ODA procedures 

manuals for these companies. We obtained documents to analyze FAA’s policies 

and procedures for ODA and RBRT, assessed the adequacy of completed RBRT 

risk assessments, and reviewed the qualifications of ODA unit members. 

To assist us in assessing the effectiveness of ODA and RBRT, we contracted with 

TeamSAI, an air transportation consulting firm. TeamSAI assisted in our analysis 

and participated in meetings with several ACOs and all three aircraft 

manufacturers.

We met with the national certification engineering representative of the National 

Air Traffic Controllers Association (NATCA) to obtain the labor union’s 

perspectives on ODA and RBRT. We also engaged OIG legal and statistical 

representatives for assistance in evaluating ODA and RBRT.
17

Exhibit B. Entities Visited or Contacted

EXHIBIT B. ENTITIES VISITED OR CONTACTED

Federal Aviation Administration

FAA Headquarters Washington, DC

Mike Monroney Aeronautical Center Oklahoma City, OK

Wichita ACO Wichita, KS

Chicago ACO Chicago, IL

Atlanta ACO Atlanta, GA

Los Angeles ACO Los Angeles, CA

Seattle ACO Seattle, WA

Transport Airplane Directorate Seattle, WA

Aircraft Manufacturers

Cessna Aircraft Wichita, KS

Bombardier-Learjet Wichita, KS

The Boeing Company Seattle, WA

Other Organizations

National Air Traffic Controllers Association

(Aircraft Certification Engineers) Chicago, IL
18

Exhibit C. Major Contributors to This Report

EXHIBIT C. MAJOR CONTRIBUTORS TO THIS REPORT 

Robin Koch Program Director

Name Title

Marshall Jackson Project Manager

Mark Perrill Senior Analyst

Taniesha Snell Analyst

Ruth Foyere Analyst

Andrea Nossaman Writer/Editor
19

Appendix. Agency Comments

APPENDIX. AGENCY COMMENTS

Federal Aviation

Administration

Memorandum

Date: 

To: Jeffrey B. Guzzetti, Assistant Inspector General for Aviation and 

Special Program Audits 

From: Clay Foushee, Director, Office of Audit and Evaluations, AAE-1 

Subject: OIG Final Report: FAA Needs to Strengthen its Risk Assessment and 

Oversight Approach for Organization Designation Authorization and 

Risk Based Resources Targeting Programs 

While organizational delegation is not new to the FAA or the aviation industry, FAA is 

continuing its efforts to strengthen the program, provide meaningful and consistent 

oversight, and ensure it addresses the most important issues relating to aviation safety. In 

the process of improving this program, FAA is seeking to make the most efficient use of 

its resources, by redeploying assets previously devoted to less constructive oversight 

activities, and apply a risk based framework for directing the activities of its Designees. 

FAA recognizes that its efforts are a work in process, and has efforts underway that will 

provide better training, ongoing program evaluations, and policy and information 

technology (IT) improvements. 

 

RECOMMENDATIONS AND RESPONSES

Recommendation 1: Revise its ODA policy to require a full 2-year transition for unit 

member self selection.

FAA Response: Concur. The FAA will revise its guidance by September 30, 2012 to 

require the organization management team (OMT) to review selections made in the first 

two years or longer, if necessary. We will continue to communicate the role of the OMT 

in performing oversight of an ODA’s unit member selection to ensure the processes are 

being properly established and exercised. 

Recommendation 2: Develop explicit guidance on the process to remove an ODA unit 

member in a timely fashion and require all ODA holders to include this standardized 

removal process in their approved ODA procedures manual. 
20

Appendix. Agency Comments

FAA Response: Concur. Although the FAA believes that unit member (UM) removal 

clearly falls under the established corrective action procedures, the FAA will add policy 

and procedures to specifically address UM removal. This information will be included in 

Change 1 to 8100.15A by September 30, 2012.

Recommendation 3: Track unit member appointments in its Designee Information 

Network (DIN) database or another method in order to identify unit members with known 

performance issues and require engineers to cross-check names with the database beyond 

the first 2-year required timeframe.

FAA Response: Concur in part. It is impractical for the FAA to maintain all ODA UM 

names in an FAA database. However, since an ODA holder is responsible for their unit 

member management and we already require ODA holders to report the names of UMs 

removed for performance reasons that constitute misconduct, we could track only those 

unit members who have been removed due to performance-related issues. We propose to 

issue policy that will require these removed UMs to be tracked in DIN so that they may 

be precluded from future designee or UM selection. Order 8100.15 currently requires 

that proposed UMs are cross-checked with existing FAA databases beyond the initial 2-

year timeframe. This is being clarified in Order 8100.15A. The proposed change will be 

included in Change 1 to 8100.15A by September 30, 2012. 

Recommendation 4: Develop enforcement training and guidance that is pertinent to the 

unique requirements of the certification engineering discipline.

FAA Response: Concur. Aviation Safety (AVS) will ensure that pertinent portions of 

the existing FAA Academy Compliance & Enforcement course address ODA certificate 

management. AVS will also provide more information in the designee management 

course lesson on ODA oversight by December 31, 2012.

Recommendation 5: Improve the new oversight structure for large ODA holders by:

A. developing training for FAA engineers and disseminating comprehensive

procedures on the new oversight structure for large ODA holders.

B. assessing the effectiveness of the new oversight structure before implementing it 

at other large ODA holders.

FAA Response: Concur. The establishment of the Boeing Aviation Safety Oversight 

Office (BASOO) and Gulfstream Aviation Safety Oversight Office (GASOO) constitutes 

a new organizational structure, not a new oversight structure. Oversight roles and 

processes remain unchanged but may be divided among more staff members in these 

offices. AVS will revise its policy to clarify that these new offices are responsible for 

developing and conducting training for their staff and the people that interact with them. 

In addition, AVS will develop guidance defining best practices for proper oversight of 

ODAs by June 2012. We will then assess the effectiveness of the existing oversight 

offices (BASOO and GASOO) against this guidance before we implement any future 

organizational changes of a similar nature. We will have a plan developed to assess the 
21

Appendix. Agency Comments

existing oversight offices by September 2012. We will then perform the assessment and 

document any proposed changes, to policy or structure, in fiscal year (FY) 2013.

Recommendation 6: Improve the RBRT tool by:

A. Enhancing the risk assessment process so that it uses more automated data, such 

as accidents resulting from manufacturing defects, to accurately differentiate higher 

risk projects that likely pose the most safety risk.

B. Thoroughly testing and validating it to ensure that it is fully functional.

C. Ensuring that engineers are properly trained before requiring its use and relying 

upon its results.

FAA Response to 6.A: Concur. It has been the intention of the FAA to continuously 

improve the RBRT process. The implementation of the AVS Safety Management 

System (SMS) will establish a methodology to collect appropriate data to transition from 

a qualitative to a quantitative risk management system. There are several process 

initiatives within AVS SMS to provide data to support RBRT in the future. To get to that 

point though, we also need to begin using RBRT in a qualitative form to standardize the 

risks on which our workforce should focus. We have revised the RBRT tool to use the 

regulation prioritization data that differentiates and prioritizes the severity of all the 

regulations. This will help the workforce better use the tool to focus on specific 

regulations that are higher risk. New process and IT tools being developed to support the 

transition of RBRT to a fully quantitative process include All Lessons Learned (accident 

database) expected in mid-FY 2012; Aircraft Certification Audit Information System, 

expected by the end of FY 2012; the Engineering Design Approval process, expected in 

FY 2013; the Designee Management System, expected by the end of FY 2013; 

Manufacturers SMS; and potential rulemaking for Part 21 SMS beginning in FY 2013. 

All of these processes currently under development will collect the objective data needed 

to support RBRT in the future. 

FAA Response to 6.B: Concur. The new IT solution is currently in the user-acceptance 

testing phase. This new solution is being tested more vigorously than the last, based on 

the lessons learned from the prototype tool. Validation of the process has occurred 

multiple times in the past four years. The process was initially challenged by a large user 

community, after which several changes were made. The improved process and tool 

were then used in a prototype environment, during which we found additional changes 

and learned about the IT limitations. We will continue to take full advantage of testing 

and validation practices, as necessary, to deploy future data tools mentioned in 6A, and 

anticipate implementation by September 30, 2014. Additionally, there are targeted 

reviews scheduled by the process owner to review the user feedback being collected 

along with management recommendations, which will drive continuous improvement of 

the tool. 

FAA Response to 6.C: Concur. New training is being developed to provide the work 

force with the necessary knowledge to use the RBRT IT tool. This training will take 

many forms. First, we are developing a computer-based learning package that will be 

available to all employees. This computer-based package will be available two months 
22

Appendix. Agency Comments

before planned implementation to allow adequate time for the work force to access the 

training. There is also a plan to conduct an orientation presentation to all field offices, 

detailing the process and demonstrating the IT tool. These orientations will take place in 

the month before projected launch, so as to make sure the information on how to interface 

with and use the tool stays fresh in the minds of the field employees. To support the use 

of RBRT, we will also be issuing policy and Quality Management System work 

instructions, detailing when and how to use the tool for those who may have missed the 

orientation sessions. This information will be published and have an effective date no 

later than September 30, 2011. Additionally, AVS has been conducting a “road show” 

for the workforce on our discretionary function authority and what it means. This 

training is also in support of RBRT in that it lays the foundation for the management 

options RBRT provides. These discretionary function briefings are scheduled to be 

completed by December 31, 2011.