Aviation Safety: Certification and Approval Processes Are Generally Viewed as Working Well, but Better Evaluative Information Needed to Improve Efficiency
Published: 2010-10-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Among its responsibilities for aviation safety, the Federal Aviation Administration (FAA) issues thousands of certificates and approvals annually. These certificates and approvals, which FAA bases on its interpretation of federal standards, indicate that such things as new aircraft, the design and production of aircraft parts and equipment, and new air operators are safe for use in the national airspace system. Past studies and industry spokespersons assert that FAA's interpretations produce variation in its decisions and inefficiencies that adversely affect the industry. GAO was asked to examine the (1) extent of variation in FAA's interpretation of standards for certification and approval decisions and (2) views of key stakeholders and experts on how well these processes work. To perform the study, GAO reviewed industry studies and reports and FAA documents and processes; convened a panel of aviation experts; and interviewed officials from various industry sectors, senior FAA officials, and unions representing FAA staff.
section_title: What GAO Found
paragraphs: Studies, stakeholders, and experts indicated that variation in FAA's interpretation of standards for certification and approval decisions is a long-standing issue, but GAO found no evidence that quantified the extent of the problem in the industry as a whole. Ten of the 13 industry group and company officials GAO interviewed said that they or members of their organization had experienced variation in FAA certification and approval decisions on similar submissions. In addition, experts on GAO's panel, who discussed and then ranked problems with FAA's certification and approval processes, ranked inconsistent interpretation of regulations, which can lead to variation in decisions, as the first and second most significant problem, respectively, with these processes for FAA's Flight Standards Service (which issues certificates and approvals for individuals and entities to operate in the national airspace system) and Aircraft Certification Service (which issues approvals to the designers and manufacturers of aircraft and aircraft parts and equipment). According to industry stakeholders, variation in FAA's interpretation of standards for certification and approval decisions is a result of factors related to performance-based regulations, which allow for multiple avenues of compliance, and the use of professional judgment by FAA staff and can result in delays and higher costs. Industry stakeholders and experts generally agreed that FAA's certification and approval processes contribute to aviation safety and work well most of the time, but negative experiences have led to costly delays for the industry. Industry stakeholders have also raised concerns about the effects of process inefficiencies on the implementation of the Next Generation Air Transportation System (NextGen)--the transformation of the U.S. national airspace system from a ground-based system of air traffic control to a satellite-based system of air traffic management. They said that the processes take too long and impose costs that discourage aircraft operators from investing in NextGen equipment. FAA has taken actions to improve the certification and approval processes, including hiring additional inspectors and engineers and increasing the use of designees and delegated organizations--private persons and entities authorized to carry out many certification activities. Additionally, FAA is working to ensure that its processes are being followed and improved through a quality management system, which provides a mechanism for stakeholders to appeal FAA decisions. However, FAA does not know whether its actions under the quality management system are achieving the intended goals of reducing inconsistencies and increasing consistency and fairness in the agency's application of regulations and policies because FAA does not have outcome-based performance measures and a continuous evaluative process that would allow it to determine progress toward these goals. Without ongoing information on results, FAA managers do not know if their actions are having the intended effects.
section_title: What GAO Recommends
paragraphs: GAO recommends that FAA develop a continuous evaluative process with measurable performance goals to determine the effectiveness of the agency's actions to improve its certification and approval processes. DOT did not comment on the recommendations but provided technical comments, which were included as appropriate.

section_title: Letter
paragraphs: The Federal Aviation Administration (FAA) is responsible for aviation  safety, in part by issuing certificates for new air operators, new aircraft,  and aircraft parts and equipment, as well as granting approvals for such  things as changes to air operations and aircraft, and the design and  production of aircraft parts and equipment. FAA issues certificates and  approvals based on the evaluation of aviation industry submissions against  standards set forth in federal aviation regulations and related FAA  guidance documents. Studies published over the last 14 years have  asserted that inconsistencies or variation in FAA’s interpretation and  application of the regulations and guidance hinders the efficiency of the  certification and approval processes. More recently, several aviation  industry groups have asserted that FAA’s processes for carrying out these  functions continue to result in variation in decisions and inefficiencies,  which can result in delays and higher costs for their members.
You asked us to examine FAA’s processes for the certification and  approval of national airspace users and products. To do so, we addressed  (1) the extent of variation in FAA’s interpretation of standards with regard  to the agency’s certification and approval decisions and (2) key  stakeholder and expert views on how well the certification and approval  processes work.
To fulfill these objectives, we reviewed relevant studies, reports, and FAA  documents and processes and, with the assistance of the National  Academy of Sciences, convened a panel of aviation industry and other  experts on December 16, 2009. Selected with the goal of obtaining a  balance of perspectives, the panel included FAA senior managers; officials  representing large and small air carriers, aircraft and aerospace product  manufacturers, aviation services firms, repair stations, and aviation  consultants; and academicians specializing in aviation and organization  theory. We also interviewed trade groups and certificate and approval  holders of various sizes that represented a broad range of aviation industry  sectors—including air carriers, repair stations, and manufacturers. (See  app. I for more information on our objectives, scope, and methodology.)
We conducted this performance audit from July 2009 to October 2010, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: Located in FAA’s Office of Aviation Safety, the Aircraft Certification  Service (Aircraft Certification) and Flight Standards Service (Flight  Standards) issue certificates and approvals for the operators and aviation  products used in the national airspace system based on standards set forth  in federal aviation regulations. FAA inspectors and engineers working in  Aircraft Certification and Flight Standards interpret and implement the  regulations governing certificates and approvals via FAA policies and  guidance, such as orders, notices, and advisory circulars.
subsections: section_title: Aircraft Certification
paragraphs: Aircraft Certification’s approximately 950 engineers and inspectors in 38  field offices issue approvals to the designers and manufacturers of aircraft  and aircraft engines, propellers, parts, and equipment, including the  avionics and other equipment required for the Next Generation Air  Transportation System (NextGen)—a federal effort to transform the U.S.  national airspace system from a ground-based system of air traffic control  to a satellite-based system of air traffic management. These approvals are  issued in three areas: (1) design—including type certificates for new  aircraft, engine, or propeller designs, amended type certificates (issued  only to the type certificate holder) for derivative models, and  supplemental type certificates for major changes to existing designs by  either the type certificate holder or someone other than the original type  certificate holder; (2) production—including production certificates,  which certify a manufacturer’s ability to build an aircraft, engine, or  propeller in accordance with an FAA-approved design, and parts  manufacturer approvals for spare and replacement parts; and (3) flight  approval—original airworthiness certificates and approvals for newly  manufactured aircraft, engines, propellers, and parts. Aircraft  Certification, along with Flight Standards, provides a safety performance  management system intended to assure the continued operational safety  of all aircraft operating in the national airspace system and of U.S.-built  aircraft operating anywhere in the world. Aircraft Certification is also  responsible for the appointment and oversight of designees and delegated  organizations that play a critical role in acting on behalf of FAA to perform  many certification and approval activities, such as the issuance of design  and airworthiness approvals for aircraft parts.
Since 2005, Aircraft Certification has used project sequencing to prioritize  certification submissions on the basis of available resources. Projects are  evaluated against several criteria, including safety attributes and their  impact on the air transportation system. In fiscal year 2009, Aircraft  Certification issued 4,248 design approvals, 2,971 production approvals,  and 508 airworthiness certificates. Figure 1 shows the Aircraft  Certification approvals issued for fiscal years 2005 through 2009. As of  June 2010, according to FAA, Aircraft Certification had a backlog of 47  projects. (According to a senior FAA official, the number of approvals  decreased from fiscal year 2006 to fiscal year 2007 because Aircraft  Certification implemented a new data collection system in fiscal year 2007  that improved data collection definitions and processes.) Figure 2 contains  key information about Aircraft Certification’s organization, and figure 3  indicates key phases in Aircraft Certification’s product approvals process.
section_title: Flight Standards
paragraphs: Flight Standards’ nearly 4,000 inspectors issue certificates allowing  individuals and entities to operate in the national airspace system. Flight  Standards also issues approvals for programs, such as training and  minimum equipment lists. Flight Standards field office managers in over  100 field offices use the Certification Services Oversight Process to initiate  certification projects within their offices. According to FAA, the field  offices are also assisted by a headquarters-based office that provides  experts on specific aircraft and airlines. Accepted projects are processed  on a first-in, first-out basis within each office once FAA determines that it  has the resources to oversee an additional new certificate holder. Flight  Standards issued 599 air operator and air agency certificates in fiscal year  2009. These include certificates to commercial air carriers under 14 C.F.R.  part 121, operators of smaller commercial aircraft under 14 C.F.R. part  135, repair stations under 14 C.F.R. part 145, and pilot schools and training  centers under 14 C.F.R. parts 141 and 142, respectively. According to its  Director, Flight Standards also issues over 6,000 approvals daily. Figure 4  shows the number of air operator and air agency certificates issued by  Flight Standards in fiscal years 2005 through 2009.
FAA officials noted that certification projects within and among the  categories of air operators and air agencies require various amounts of  FAA resources. For example, FAA indicated that an agricultural operator  certification requires fewer FAA resources than a repair station  certification. Additionally, certifications of small commercial aircraft  operations that are single pilot, single plane require a different set of  resources than operations that are dual pilot and/or fly more aircraft. As of  July 2010, Flight Standards had 1,142 certifications in process and a  backlog of 489 applications. According to an FAA official, Flight  Standards has more wait-listed applications than Aircraft Certification  because it receives numerous requests for certificates, and its  certifications are substantially different in nature from those issued by  Aircraft Certification.
Flight Standards is also responsible for assuring the continued operational  safety of the national airspace system by overseeing certificate holders,  monitoring (along with Aircraft Certification) operators’ and air agencies’  operation and maintenance of aircraft, and overseeing designees and  delegated organizations. Flight Standards inspectors were tasked with  overseeing 13,089 air operators and air agencies, such as repair stations, as  of March 2010. Unless assigned to a large commercial air carrier issued a  certificate under part 121, a Flight Standards inspector is typically  responsible for overseeing several entities that often perform different or  several functions within the system—including transporting passengers,  repairing aircraft, and training pilots.
Figures 5 and 6 contain key information about Flight Standards’  organization and certification process for air operators and air agencies.
section_title: Extent of Variation in Interpretation Is Unknown but Potentially Stems from Factors Related to Performance-Based Regulations and FAA’s Processes
subsections: section_title: Extent of Variation in FAA’s Interpretation of Standards for Certification and Approval Decisions Is Unknown, but Stakeholders and Experts Indicate That Serious Problems Occur Infrequently
paragraphs: Studies we reviewed and aviation stakeholders and experts we spoke with  indicated that variation in FAA’s interpretation of standards for  certification and approval decisions is a long-standing issue that affects  both Aircraft Certification and Flight Standards, but the extent of the  problem has not been quantified in the industry as a whole. Inconsistent or  variant FAA interpretations have been noted in studies published over the  last 14 years. A 1996 study by Booz Allen & Hamilton, conducted at the  request of the FAA Administrator to assess challenges to the agency’s  regulatory and certification practices, reported that, for air carriers and  other operators, the agency’s regulations are often ambiguous; subject to  variation in interpretation by FAA inspectors, supervisors, and policy  managers; and in need of simplification and consistent implementation. A  1999 task force, convened at the request of the FAA Administrator to  assess FAA’s certification process, found that the agency’s requirements  for the various approvals—such as type certificates and supplemental type  certificates—varied substantially because of differences in standards and  inconsistent application of those standards by different FAA field offices.  While FAA has put measures in place since these two reports appeared, a  2008 Independent Review Team, which was commissioned by the  Secretary of Transportation to assess FAA’s safety culture and approach to  safety management, found that a wide degree of variation in “regulatory  ideology” among FAA staff continues to create the likelihood of wide  variation in decisions within and among field offices.
Industry officials and experts representing a broad range of large and  small aviation businesses told us that variation in interpretation and  subsequent decisions occurs in both Aircraft Certification and Flight  Standards, but we found no evidence that quantified the extent of the  problem in the industry as a whole. Specifically, 10 of the 13 industry  group and individual company representatives we interviewed said that  they or members of their organization experienced variation in FAA’s  certification and approval decisions on similar submissions; the remaining  3 industry representatives did not raise variation in interpretations and  decisions as an issue. For example, an official from one air carrier told us  that variation in decisions occurs regularly when obtaining approvals from  Flight Standards district offices, especially when dealing with inspectors  who are newly hired or replacing a previous inspector. He explained that  new inspectors often task air carriers to make changes to previously  obtained minimum equipment lists or conformity approvals for an  aircraft. The official further noted that inspector assignments often  change for reasons such as transfers, promotions, or retirement and that  four different principal operations inspectors were assigned to his  company during the past 18 months.
Experts on our panel and most industry officials we interviewed indicated  that, though variation in decisions is a long-standing, widespread problem,  it has rarely led to serious certification and approval process problems.  Experts on our panel generally noted that serious problems with the  certification and approval processes occur less than 10 percent of the  time. However, when we asked them to rank certification and approval  process problems we summarized from their discussion, they chose  inconsistent interpretation of regulations, which can lead to variation in  decisions, as the most significant problem for Flight Standards and as the  second most significant problem for Aircraft Certification. Panelists’  concerns about variation in decisions included instances in which  approvals are reevaluated and sometimes revised or revoked in FAA  jurisdictions other than those in which they were originally granted.  Industry officials we interviewed, though most had experienced it, did not  mention the frequency with which variation in decisions occurred.  However, 8 of the 13 said that their experiences with FAA’s certification  and approval processes were generally free of problems compared with 3  who said they regularly experienced problems with the process.
FAA’s Deputy Associate Administrator for Aviation Safety and union  officials representing FAA inspectors and engineers acknowledged that  variation in certification and approval decisions occurs. The Deputy  Associate Administrator noted that variation in interpretation and  certification and approval decisions occurs in both Aircraft Certification  and Flight Standards. He acknowledged that a nonstandardized process  for approvals exists and has been a challenge for, and a long-term criticism  of, the agency. Furthermore, he explained that efforts were being made to  address the issue, including the establishment of (1) an Office of Aviation  Safety quality management system (QMS) to standardize processes across  Aircraft Certification and Flight Standards, (2) a process for industry to  dispute FAA decisions, and (3) standardization offices within Aircraft  Certification directorates. The first two efforts are discussed in greater  detail later in this report.
section_title: Industry Stakeholders Noted That Variation in Decisions Occurs as a Consequence of Performance-Based Regulations and FAA’s Exercise of Professional Judgment
paragraphs: Variation in FAA’s interpretation of standards and certification and  approval decisions occurs as a result of factors related to performance- based regulations and the use of professional judgment by FAA inspectors  and engineers, according to industry stakeholders. FAA uses performance- based regulations, which identify a desired outcome and are flexible about  how the outcome is achieved. For example, performance-based  regulations on aircraft braking would establish minimum braking  distances for aircraft but would not call for a particular material in the  brake pads or a specific braking system design. According to officials in  FAA’s rulemaking office, about 20 percent of FAA’s regulations are  performance-based. Performance-based regulations, which are issued  governmentwide, provide a number of benefits, according to literature on  the regulatory process. By focusing on outcomes, for example,  performance-based regulations give firms flexibility in achieving the stated  level of performance; such regulations can accommodate technological  change in ways that prescriptive regulations that focus on a specific  technology generally cannot. For those certifications and approvals that  relate to performance-based regulations, variation in decisions is a  consequence of such regulations, according to one air carrier, since  performance-based regulations allow the applicant multiple avenues to  comply with regulations and broader discretion by FAA staff in making  certification and approval decisions. According to senior FAA officials,  performance-based regulations allow innovation and flexibility while  setting a specific safety standard. The officials added that the benefits of  performance-based regulations outweigh the potential for erroneous  interpretation by an individual inspector or engineer. While agreeing with  this statement, a panel member pointed out that the potential for  erroneous interpretation also entails a risk of inconsistent decisions.
In addition, FAA oversees a large, diverse industry, and its certification  and approval processes rely, in part, on FAA staffs’ exercise of  professional judgment in the unique situations they encounter. In the  opinion of senior FAA officials, some differences among inspectors may  be due to situation-specific factors that industry stakeholders may not be  aware of. According to officials from Flight Standards, because differences  may exist among regions and district offices, operators changing locations  may encounter these differences.
section_title: Key Stakeholders and Experts Said the Certification and Approval Processes Generally Work Well, but When They Do Not, It Can Be Costly for Industry
subsections: section_title: Stakeholders and Experts Said the Certification and Approval Processes Contribute to System Safety and Work Well Most of the Time
paragraphs: Many industry stakeholders and experts stated that FAA’s certification and  approval processes contribute positively to the safety of the national  airspace system. For example, industry stakeholders who participated in  our expert panel ranked the office’s safety culture and record as the  greatest strength of Flight Standards’ certification and approval processes  and the third greatest strength of Aircraft Certification’s processes.
Industry stakeholders and experts also noted that the certification and  approval processes work well most of the time because of FAA’s long- standing collaboration with industry, flexibility within the processes, and  committed, competent FAA staff. In most instances, stakeholders and  experts said, when industry seeks certifications and approvals, its  experiences with FAA’s processes are positive. For example, two aviation  manufacturers and an industry trade association with over 400,000  members noted that most of their experiences or their members’  experiences were positive. Seventeen of 19 panelists indicated positive or  very positive experiences with Aircraft Certification, and 9 of 19 panelists  indicated positive experiences with Flight Standards. Panelists ranked  FAA’s collaboration with applicants highly—as the second greatest  strength of both Aircraft Certification and Flight Standards. In addition,  representatives of two trade associations representing over 190 aviation  companies said that the processes provide flexibility for a large, diverse  industry. Additionally, panelists ranked FAA inspectors’ and engineers’  expertise as the greatest strength of Aircraft Certification and the third  greatest strength of Flight Standards, while officials from two industry  trade groups cited the inspectors’ and engineers’ competence and high  level of expertise.
section_title: Industry Stakeholders Said Negative Experiences Are Infrequent but Can Cause Costly Delays
paragraphs: Industry stakeholders and experts noted that negative certification and  approval experiences, although infrequent, can result in costly delays for  them, which can disproportionately affect smaller operators. While  industry stakeholders indicated that negative experiences occur in  dealings with both Aircraft Certification and Flight Standards, experts on  our panel noted that negative experiences are more likely to occur with  Flight Standards than with Aircraft Certification. For example, three  experts noted that, overall, industry’s experience in obtaining  certifications and approvals from Flight Standards has been negative or  very negative, while no experts thought industry’s experience with Aircraft  Certification was negative. The panelists indicated that negative  experiences occur during the processing of certifications and approvals  and as applicants wait for FAA resources to become available to  commence their certification or approval projects. For example, an  aviation industry representative reported that his company incurred a  delay of over 5 years and millions of dollars in costs when it attempted to  obtain approvals from Aircraft Certification and Flight Standards field  offices. Another industry representative indicated that it abandoned an  effort to obtain an operating certification after spending $1.2 million and  never receiving an explanation from FAA as to why the company’s  application was stalled. One panelist indicated that the negative  experiences focus more on administrative aspects of the certification and  approval processes and not on safety-related items.
The processing of original certifications and approvals in Aircraft  Certification and Flight Standards involves progressing through a schedule  of steps or phases. Responsibilities of both FAA and the applicant are  delineated. However, even with this framework in place, industry  stakeholders noted that the time it takes to obtain certifications and  approvals can differ from one FAA field office to another because of  differences in office resources and expertise. In some cases, delays may be  avoided when FAA directs the applicant to apply at a different field office.
Nevertheless, applicants who must apply to offices with fewer resources  can experience costly delays in obtaining certifications or approvals.
Delays also occur when FAA wait-lists certification submissions because it  does not have the resources to begin work on them. Aircraft Certification  meets weekly to review certification project submissions. If it determines  that a submission is to be wait-listed, the applicant is sent a 90-day delay  letter and if, after the initial 90 days, the submission is still wait-listed, the  applicant is sent another letter. Additionally, Aircraft Certification staff  and managers periodically contact applicants to advise them of the status  of their submissions. Flight Standards also notifies applicants when their  certification submissions are wait-listed, and Flight Standards staff are  encouraged to communicate with applicants regularly about the status of  their submissions. However, according to an FAA notice, staff are advised  not to provide an estimate of when an applicant’s submission might be  processed. While Aircraft Certification tracks in a national database how  long individual submissions are wait-listed, Flight Standards does not.  Without data on how long submissions are wait-listed, Flight Standards  cannot assess the extent of wait-listing delays or reallocate resources to  better meet demand. Further, industry stakeholders face uncertainty with  respect to any plans or investments that depend on obtaining a certificate  in a timely manner.
Industry stakeholders have also raised concerns about the effects of  inefficiencies in the certification and approval processes on the  implementation of NextGen. As NextGen progresses, operators will need  to install additional equipment on their aircraft to take full advantage of  NextGen capabilities, and FAA’s certification and approval workload is  likely to increase substantially. According to our October 2009 testimony  on NextGen, airlines and manufacturers said that FAA’s certification  processes take too long and impose costs on industry that discourage  them from investing in NextGen equipment. We reported that this  inefficiency in FAA’s processes constitutes a challenge to delivering  NextGen benefits to stakeholders and that streamlining FAA’s processes  will be essential for the timely implementation of NextGen. FAA is  working to address the certification issues that may impede the adoption  and acceleration of NextGen capabilities. Flight Standards has identified  NextGen-dedicated staff in each of its regional offices to support the  review and approval of NextGen capabilities within each region. Aircraft  Certification has created a team of experts from different offices to  coordinate NextGen approvals and identify specialists in Aircraft  Certification offices with significant NextGen activity. FAA also plans a  number of other actions to facilitate the certification and approval of  NextGen-related technology, including new procedures and criteria for  prioritizing certifications, updating policy and guidance, developing  additional communication mechanisms, and developing training for  inspectors and engineers. Since many of these actions have either just  been implemented or have not yet been completed, it is too early to tell  whether they will increase the efficiency of FAA’s certification and  approval processes and reduce unanticipated delays and costs for the  industry.
section_title: Industry Stakeholders and Experts Told Us That the Efficiency of FAA’s Processes Is Hampered by Several Shortcomings; FAA Has Taken Some Actions to Remedy Them
paragraphs: Industry stakeholders also noted that the efficiency of the certification and  approval processes was hampered by a lack of sufficient staff resources to  carry out certifications and approvals and a lack of effective  communication mechanisms for explaining the intent of the regulations to  both FAA staff and industry. The stakeholders said that these  inefficiencies have resulted in costly delays for them.
Stakeholders and experts said that, at some FAA offices, delays in  obtaining certifications and approvals were due to heavy staff workloads,  a lack of staff, or a lack of staff with the appropriate expertise. Staff and  managers at one FAA field office told us that in the past a lack of staff had  contributed to delays in completing certifications. The relative priority of  certifications and approvals within FAA’s overall workload also affects the  availability of staff to process certifications and approvals. According to  FAA, its highest priority is overseeing the continued operational safety of  the people and products already operating within the national airspace  system, but the same staff who provide this oversight are also tasked  with the lower-priority task of processing new certifications and  approvals. Additionally, Flight Standards field staff we contacted said that  the system under which their pay grades are established and maintained  provides a disincentive for inspectors to perform certification work  because the system allocates no credit toward retention of their pay  grades for doing certification work. Flight Standards headquarters officials  pointed out that there is an incentive for field office inspectors to perform  initial certifications because once certificated the new entities add points  to an inspector’s complexity calculation, which is used to determine his or  her pay grade.
FAA has addressed staff resource issues by increasing the number of  inspectors and engineers. Over the past 3 years, FAA has steadily  increased its hiring of Aircraft Certification engineers and Flight Standards  inspectors, thereby reducing the risk of certification delays. According to  agency data, FAA’s hiring efforts since fiscal year 2007 have resulted in an  8.8 percent increase in the number of Aircraft Certification engineers and  a 9.4 percent increase in the number of Flight Standards inspectors on  board. FAA hired 106 engineers in Aircraft Certification and 696 inspectors  in Flight Standards from the beginning of fiscal year 2007 to March 15,  2010. FAA also hired 89 inspectors in Aircraft Certification from fiscal year  2007 through August 2010. In addition, Flight Standards headquarters  staff are available to assist field staff with the certification of part 121 air  carriers—an average of 35 of these staff were available for this assistance  annually from 2005 through 2009, and they helped with 16 certification  projects.
Furthermore, FAA delegates many certification activities to individuals  and organizations (called designees) to better leverage its resources. As  we previously reported, FAA’s designees perform more than 90 percent of  FAA’s certification activities. We have reported that designees generally  conduct routine certification functions, such as approvals of aircraft  technologies that the agency and designees already have experience with,  allowing FAA staff to focus on new and complex aircraft designs or design  changes. Panelists ranked the expanded use of designees second and  fifth, respectively, among actions that we summarized from their  discussions that would have the most positive impact on improving  Aircraft Certification’s and Flight Standards’ certification and approval  processes. FAA is increasing organizational delegations under its  organization designation authorization (ODA) program and expects the  ODA program will allow more effective use of its resources over time.
Stakeholders pointed to a lack of effective communication mechanisms as  another problem with the certification and approval processes, especially  deficiencies in the guidance FAA issues and a lack of additional  communication mechanisms for sharing information on the interpretation  of regulations. Stakeholders said that the lack of effective communication  mechanisms can lead to costly delays when, for example, methods or  guidance for complying with regulations is not clear. Stakeholders and  experts had several issues with the FAA guidance that interprets the  regulations and provides supplemental information to the industry.  Stakeholders said there are sometimes discrepancies between the  guidance and the regulations. For example, one stakeholder reported  informing an FAA training course instructor that a particular piece of  guidance contradicted the regulations. The instructor agreed that the  contradiction existed but told the stakeholder that FAA teaches to the  guidance, not the regulations. One employee group representing some  FAA inspectors was concerned that not all guidance has been included in  an online system that FAA has established to consolidate regulations,  policy, and guidance. FAA acknowledged that it is working to further  standardize and simplify the online guidance in the Flight Standards  information management system.
Stakeholders also identified a lack of opportunities for sharing information  about the interpretation of regulations and guidance. An industry expert  noted that FAA lacks a culture that fosters communication and discussion  among peer groups. Moreover, an industry group with over 300 aviation  company members suggested that FAA should support and promote more  agencywide and industrywide information sharing in less formal, less  structured ways to enhance communication. Finally, according to an  official of an employee group representing some FAA inspectors, because  their workloads tend to be heavy, inspectors are less able to communicate  with the companies they oversee, and the reduced level of communication  contributes to variation in the interpretation of regulations. FAA officials  disagreed with these assertions and indicated that FAA staff participate in  numerous committees and conferences, share methods of compliance in  technical areas via forums with stakeholders, and communicate  resolutions to problems in various formats, such as by placing legal  decisions online.
section_title: FAA Has Taken Other Actions That Might Address Process Shortcomings but Lacks Performance Data to Assess the Actions’ Effectiveness
paragraphs: Other FAA actions could identify and potentially address some of the  shortcomings in the agency’s certification and approval processes as  follows:  In 2004, FAA’s Office of Aviation Safety introduced QMS, which is  intended to ensure that processes are being followed and improved and to  provide a methodology to standardize processes. QMS is expected to help  ensure that processes are followed by providing a means for staff to report  nonconformance with FAA procedures or processes and was established  as part of the office’s effort to achieve certification by the International  Organization for Standardization (ISO). Any employee can submit a  report and check the status of an issue that has been reported. From  October 2008 to March 2009, approximately 900 reports were submitted,  and 46 internal audits were completed. For example, in July 2009, an FAA  staffer noted that a required paragraph on aging aircraft inspection and  records review was missing from a certificate holder’s operations  specifications. The issue was resolved and closed in August 2009 when  the missing paragraph was issued to the certificate holder. Some FAA staff  told us that QMS has helped improve the processes because it requires  management action to respond to report submissions.
To provide industry stakeholders with a mechanism for appealing  certification and other decisions, the Office of Aviation Safety  implemented the Consistency and Standardization Initiative (CSI) in  2004. Appeals must begin at the field office level and can eventually be  taken to FAA headquarters. CSI requires that FAA staff document their  safety decisions and that stakeholders support their positions with specific  documentation. Within Aircraft Certification and Flight Standards, CSI  cases at each appeal level are expected to be processed within 30 working  days. The total length of the CSI process depends on how many levels of  appeal the stakeholder chooses. Aircraft Certification has had over 20 CSI  cases, and Flight Standards has had over 300. Most CSI cases in Aircraft  Certification involved clarification of a policy or an approved means of  complying with a regulation, while most of those submitted to Flight  Standards involved policy or method clarification, as well as scheduling  issues, such as delays in addressing a stakeholder’s certification, approval,  or other issue. The large discrepancy between the number of cases filed  for the two services, according to FAA officials, may be due to the fact that  Aircraft Certification decisions are the result of highly interactive,  deliberative processes, which are not typical in granting approvals in  Flight Standards, where an inspector might find the need to hand down a  decision without prolonged discussion or deliberation. Stakeholders told  us that CSI lacks agencywide buy-in and can leave stakeholders who use  the program potentially open to retribution from FAA staff. However,  others noted that CSI is beneficial because it requires industry  stakeholders to use the regulations as a basis for their complaints, which  often leads to resolution. According to one of our panelists,  inconsistencies occur when FAA does not start with the regulations as the  basis for decisions.
Although QMS and CSI are positive steps toward identifying ways to make  the certification and approval processes more efficient, FAA does not  know whether the programs are achieving their stated goals because it has  not established performance measures for determining program  accomplishments. One of the goals for QMS is to reduce inconsistencies  and increase standardization. A QMS database documents the reports  submitted and, through information in these reports, FAA says it has  identified instances of nonconformance and initiated corrective action to  prevent recurrence; revised orders to ensure they are consistent with  actual practice; and improved its processes to collect feedback from  stakeholders and take action on trends. However, FAA does not know  whether its actions have reduced inconsistencies because its measures  describe the agency’s output—for example, number of audits conducted— rather than any outcomes related to reductions in process inconsistencies.  FAA officials described CSI goals as promoting early resolution of  disagreements and consistency and fairness in applying FAA regulations  and policies. They provided us with data on the number of CSI cases in  both Aircraft Certification and Flight Standards, the types of complaints,  and the percentage of resolutions that upheld FAA’s original decision, but  as with the overall QMS program, we could find no evidence that FAA has  instituted CSI performance measures that would allow it to determine  progress toward program outcomes, such as consistency and fairness in  applying regulations and policies. Outcome-based performance measures  would also allow QMS and CSI program managers to determine where to  better target program resources to improve performance.
section_title: Conclusions
paragraphs: FAA has taken actions to address variation in decisions and inefficiency in  its certification and approval processes, although the agency does not  have outcome-based performance measures and a continuous evaluative  process to determine if these actions are having their intended effects.  Because the number of certification and approval applications is likely to  increase for NextGen technologies, achieving more efficiency in these  processes will help FAA better manage this increased workload, as well as  its current workload. In addition, while both Aircraft Certification and  Flight Standards notify applicants whether resources are available to begin  their projects, Flight Standards does not monitor how long applicants are  wait-listed and is therefore unaware how long projects are wait-listed and  unable to reallocate resources to better meet demand for certification  services.
section_title: Recommendations for Executive Action
paragraphs: To ensure that FAA actions contribute to more consistent decisions and  more efficient certification and approval processes, we recommend that  the Secretary of Transportation direct the Administrator of FAA to take  the following two actions:
Determine the effectiveness of actions to improve the certification and  approval processes by developing a continuous evaluative process and use  it to create measurable performance goals for the actions, track  performance toward those goals, and determine appropriate process  changes. To the extent that this evaluation of agency actions identifies  effective practices, consider instituting those practices agency wide.
Develop and implement a process in Flight Standards to track how long  certification and approval submissions are wait-listed, the reasons for  wait-listing them, and the factors that eventually allowed initiation of the  certification process. Use the data generated from this process to assess  the extent of wait-listing delays and to reallocate resources, as  appropriate, to better meet demand.
section_title: Agency Comments
paragraphs: We provided a copy of a draft of this report to the Department of  Transportation (DOT) for its review and comment. DOT provided  technical comments, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 21 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Transportation, the  Administrator of FAA, and other interested parties. The report also will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff members have any questions or would like to discuss  this work, please contact me at (202) 512-2834 or dillinghamg@gao.gov.  Contact points for our offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  key contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report provides information on the Federal Aviation Administration’s  (FAA) processes for granting certifications and approvals to air operators,  air agencies such as repair stations, and designers and manufacturers of  aircraft and aircraft components. It describes the processes and discusses  (1) the extent of variation in FAA’s interpretation of standards with regard  to the agency’s certification and approval decisions and (2) key  stakeholder and expert views on how well the certification and approval  processes work. To address these objectives, we reviewed relevant  studies, reports, and FAA documents and processes; convened a panel of  aviation industry and other experts; and interviewed aviation industry  members, an expert, and FAA officials. We did not address FAA processes  for issuing certifications to individuals, such as pilots and mechanics.
subsections: section_title: Expert Panel
paragraphs: We contracted with the National Academy of Sciences (the Academy) to  convene a panel on FAA’s certification and approval processes on  December 16, 2009. The panel was selected with the goal of obtaining a  balance of perspectives and included FAA senior managers; officials  representing large and small air carriers, aircraft and aerospace product  manufacturers, aviation services firms, repair stations, geospatial firms,  and aviation consultants; and academics specializing in aviation and  organization theory. (See table 1.)
In the first session, FAA and industry officials presented their  organizations’ perspectives on these processes and responded to  questions. The presenters then departed and did not participate in the  remaining sessions. In the next three discussion sessions, the panelists— led by a moderator—shared their views on various aspects of FAA’s  certification and approval processes. After the first two discussion  sessions, panelists voted in response to questions posed by GAO. (See app.  II for the questions and responses.) The views expressed by the panelists  were their own and do not necessarily represent the views of GAO or the  Academy. We shared a copy of an earlier draft of this report with all of the  presenters and panelists for their review and to ensure that we correctly  captured information from their discussions and, on the basis of their  comments, made technical corrections to the draft as necessary.
section_title: Industry Interviews
paragraphs: We interviewed aviation industry certificate and approval holders, trade  groups, an industry expert, officials of unions that represent FAA  inspectors and engineers, and FAA staff in Aircraft Certification and Flight  Standards (see table 2). The industry and trade groups were selected to  provide a range of large and small companies and a variety of industry  sectors (e.g., aircraft and parts manufacturers, air carriers, and repair  stations). The interviews were conducted to gain an understanding of the  extent of variation in FAA’s certification and approval decisions and  interviewees’ views on FAA’s certification and approval processes. The  FAA interviews provided an understanding of the key aspects of FAA’s  certification and approval processes, information on data collection and  analysis related to the processes, and current and planned process  improvement efforts. In addition to using information from the individual  interviews, as relevant throughout the report, we analyzed the content of  the interviews to identify and quantify the key issues raised by the  interviewees.
section_title: Appendix II: Summary of Responses from GAO Expert Panel
paragraphs: This appendix summarizes the responses the panelists provided to  questions we posed at the close of their discussion sessions. The response  options were based on the contents of their discussions. To develop the  rankings in questions 1, 2, and 12, we asked the panelists, in a series of  three questions, to vote for the option he or she believed was the first,  second, and third greatest, most significant, or most positive. To rank  order the items listed for these questions, we assigned three points to the  option identified as greatest, most significant, or most positive; two points  to the second greatest, most significant, or most positive; and one point to  the third greatest, most significant, or most positive option. We then  summed the weighted values for each option and ranked the options from  the highest number of points to the lowest.  1. What is the greatest strength of the certification and approval  processes?
2. What is the most significant problem with the certification and approval  processes?
3. What leading factor has contributed to problems with the certification  and approval processes?
Leading factor in process problems  FAA’s prioritization system for managing certifications and  approvals  FAA’s rulemaking process and development of guidance  (e.g., amount of time required to develop or change  regulations, etc.)
Culture of FAA (e.g., stove-piping, resistance to change,  etc.)
Organizational structure of FAA (e.g., decentralization,  varying procedures among local offices, etc.)
4. How often do serious problems occur each year with the certification  and approval processes?
5. Overall, how positive or negative do you think industry’s experience has  been in obtaining certifications and approvals from Aircraft Certification  and Flight Standards?
6. How would you assess the overall impact of the certification and  approval processes on the safety of the national airspace system?
7. Overall, how would you characterize efforts to improve the certification  and approval processes?
8. Overall, how would you characterize efforts to prioritize certifications  and approvals?
9. Overall, how would you characterize efforts to improve dispute  resolution through the Consistency and Standardization Initiative (CSI)?
10. Regarding efforts to improve dispute resolution through CSI, what is  the key factor hindering the progress of efforts?
11. What should be done to mitigate the effects of this factor?
FAA should establish support for efforts  FAA should improve data collection and analysis related  to efforts  Do not believe efforts are ineffective  Do not know/no basis to judge  This response option was not available to the panelists.
12. What action will have the most positive impact on improving the  certification and approval processes?
Expand use of designees/organization  designation authorizations (ODA)
Expand use of  designees/organization  designation authorizations (ODA)
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Gerald L. Dillingham, Ph.D., (202) 512-2834 or dillinghamg@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Teresa Spisak (Assistant  Director), Sharon Dyer, Bess Eisenstadt, Amy Frazier, Brandon Haller,  Dave Hooper, Michael Silver, and Pamela Vines made key contributions to  this report.