Aviation Safety: Status of Recommendations to Improve FAA's Certification and Approval Processes
Released: 2013-10-30T00:00:00

Letter

Background

FAA’s Aircraft Certification Service (Aircraft Certification) and Flight Standards Service (Flight Standards) issue certificates and approvals for the operators and aviation products used in the national airspace system based on standards set forth in federal aviation regulations. FAA inspectors and engineers working in Aircraft Certification and Flight Standards interpret and implement the regulations governing certificates and approvals via FAA policies and guidance, such as orders, notices, and advisory circulars. (See fig. 1.)

Aircraft Certification’s approximately 950 engineers and inspectors in 42 field offices issue approvals to the designers and manufacturers of aircraft and aircraft engines, propellers, parts, and equipment. Since 2005, Aircraft Certification has used project sequencing to prioritize certification submissions on the basis of available resources. Projects are evaluated against several criteria, including safety attributes and their impact on the air transportation system. Figure 2 outlines the key phases in Aircraft Certification’s approval process.

In Flight Standards, approximately 4,000 inspectors issue certificates allowing individuals and entities to operate in the National Airspace System (NAS). These include certificates to commercial air carriers, operators of smaller commercial aircraft, repair stations, and pilot schools and training centers. Flight Standards also issues approvals for programs, such as training. Flight Standards field office managers in over 100 field offices use the Certification Services Oversight Process to initiate certification projects within their offices. Delays occur when FAA wait-lists certification submissions because it does not have the resources to begin work on them. Once FAA determines that it has the resources to oversee an additional new certificate holder, accepted projects are processed on a first-in, first-out basis within each office. Figure 3 illustrates the key steps in the Flight Standards certification process.

Responsibility for the continued operational safety of the NAS is shared by Aircraft Certification and Flight Standards, which oversee certificate holders, monitor operators’ and air agencies’ operation and maintenance of aircraft, and oversee designees and delegated organizations (known as organization designation authorizations or ODA).

FAA’s Certification and Approval Processes

In 2010, we reported that many of FAA’s certification and approval processes contribute positively to the safety of the NAS, according to industry stakeholders and experts.and approval processes work well most of the time because of FAA’s long-standing collaboration with industry, flexibility within the processes, and committed, competent FAA staff. Industry stakeholders and experts noted that negative certification and approval experiences, such as duplication of approvals, although infrequent, can result in costly delays for them, which can disproportionately affect smaller operators. We made two recommendations to improve the efficiency of the certification and approval processes. FAA addressed one recommendation and partially addressed the other. We found that while FAA had taken actions to improve the efficiency of its certification and approval processes, it lacked outcome-based performance measures and a continuous evaluative process to determine if these actions were having the intended effects. To address these issues, we recommended that FAA develop a continuous evaluative process and use it to create measurable performance goals for the actions, track performance toward those goals, and determine appropriate process changes. To the extent that this evaluation of agency actions identifies effective practices, we further recommended that FAA consider instituting those practices agency wide, i.e., in Aircraft Certification and Flight Standards. In response to our recommendation, FAA implemented new metrics that provide the ability to track process performance and product conformity to standards. These metrics would allow FAA to set measurable performance goals necessary to determine the effectiveness of the certification and approval processes and assist FAA in deciding on necessary and appropriate actions to address systemic issues that could negatively impact agency processes and their outcomes. These actions addressed the intent of our recommendation. We also recommended that FAA develop and implement a process in They also noted that the certification Flight Standards to track how long certification and approval submissions are wait-listed, the reasons for wait-listing them, and the factors that eventually allowed initiation of the certification process. As of October 2013, FAA had partially addressed this recommendation by altering the software in its Flight Standards’ Certification Service Oversight Process database to designate when certification submissions are wait-listed. The database now tracks how long certification submissions are wait-listed. As a result, FAA now has the capability to track how long certification submissions are wait-listed and reallocate resources, if appropriate, to better meet demand.

In April 2012, as required by Section 312 of the Act, FAA established the Aircraft Certification Process Review and Reform Aviation Rulemaking Committee (certification process committee). Its role is to make recommendations to the director of FAA’s Aircraft Certification Service to streamline and reengineer the certification process. The committee considered guidance and current certification issues—including methods for enhancing the use of delegation and the training of FAA staff in safety management systems—and assessed the certification process. It developed six recommendations, which called for FAA to develop comprehensive implementation plans for certification process improvement initiatives, including measuring the effectiveness of the implementation and benefits of improvements as well as developing a means to track and monitor initiatives and programs; continue to improve the effectiveness of delegation programs; develop an integrated, overarching vision of the future state for certification procedures; update Part 21 certification procedures to reflect a systems approach develop and implement a comprehensive change management plan to prepare the workforce for its new responsibilities in a systems safety approach to certification and oversight; and review continued operational safety and rulemaking processes and implement reforms to improve efficiency.

We found these recommendations to be relevant, clear, and actionable. In response to the committee’s recommendations, FAA developed a plan that includes 14 initiatives to implement the committee’s recommendations and publicly reported the plan in July 2013.

We believe that the committee took a reasonable approach in assessing FAA’s aircraft certification process and developing recommendations by assessing the status of previous recommendations from 19 reports related to the certification process, reviewing certification guidance and processes as well as major initiatives, and reviewing other areas that it believed required consideration when making recommendations for improving efficiencies in the certification process. FAA has many initiatives and programs underway that it believes will respond to the committee’s recommendations to improve efficiency and reduce costs related to certifications. For example, FAA and two industry groups had already developed an ODA action plan to address the effectiveness of the ODA process. We found these initiatives were generally relevant to the recommendations and clear and measurable. However, FAA’s initiatives and programs to implement the recommendations do not contain some of the elements essential to a performance measurement process. For example, the certification process committee recommended that FAA develop an integrated roadmap and vision for certification process reforms, including an integrated overarching vision of the future state for certification procedures. While FAA has outlined a vision in AIR: 2018, it has not yet developed a roadmap. FAA is planning to roll out its roadmap, which is to include information on major change initiatives and a scaled change management process, concurrently with or following implementation of many of its certification process improvement initiatives. This calls into question FAA’s ability to use the roadmap to guide the initiatives.

FAA has developed milestones for each initiative and deployed a tracking system to track and monitor the implementation of all certification-related initiatives. However, FAA has not yet developed performance measures to track the success of most of the initiatives and programs. The agency plans to develop these measures of effectiveness after it has implemented its initiatives. Without early performance measures, FAA will not be able to gather the appropriate data to evaluate the success of current and future initiatives and programs. In addition, in response to the certification process committee’s recommendation to review rulemaking processes and implement reforms to improve efficiency, FAA plans to expedite the rulemaking process by implementing a new rulemaking prioritization model. However, this model will have no effect on the duration of the rulemaking process since it only prioritizes potential rulemaking projects for submission to the rulemaking process and makes no changes to the rulemaking process per se.

Consistency of Regulatory Interpretation

In 2010, we reported that variation in FAA’s interpretation of standards for certification and approval decisions is a long-standing issue that can result in delays and higher costs for industry. For example, a 1996 study found that, for air carriers and other operators, FAA’s regulations are often ambiguous; subject to variation in interpretation by FAA inspectors, supervisors, or policy managers; and in need of simplification and consistent implementation.officials we interviewed for our 2010 report indicated that although variation in decisions is a long-standing, widespread problem, it has rarely led to serious certification and approval process problems, and experts on our panel generally noted that serious problems occur less than 10 Experts on our panel and most industry percent of the time. Nonetheless, when such occasions occur, experts on our panel ranked inconsistent interpretation of regulations, which can lead to variation in decisions, as the most significant problem for Flight Standards and as the second most significant problem for Aircraft Certification. Panelists’ concerns about variation in decisions included instances in which approvals are reevaluated and sometimes revised or revoked in FAA jurisdictions other than those in which they were originally granted. Such situations can result in delays and higher costs for industry but also may catch legitimate safety concerns. According to industry stakeholders we spoke with, variation in FAA’s interpretation of standards for certification and approval decisions is a result of factors related to performance-based regulations, which allow for multiple avenues of compliance, and the use of professional judgment by FAA staff. FAA’s Deputy Associate Administrator for Aviation Safety and union officials representing FAA inspectors and engineers acknowledged that variation in certification and approval decisions occurs and that FAA has taken actions to address the issue, including the establishment of a quality management system to standardize processes across offices.

A second FAA-industry committee—the Consistency of Regulatory Interpretation Aviation Rulemaking Committee (regulatory consistency committee)—established to respond to Section 313 of the Act, identified three root causes of inconsistent interpretation of regulations—(1) unclear regulatory requirements; (2) inadequate and nonstandard FAA and industry training in developing regulations, applying standards, and resolving disputes; and (3) a culture that includes a general reluctance by both industry and FAA to work issues of inconsistent regulatory application through to a final resolution and a “fear of retribution.” The root causes are consistent with issues raised in our 2010 review and those raised by industry during that review. To address the root causes, the committee made six recommendations to promote clearer regulations and guidance, more standardized application of rules, a consolidation and cross-reference of guidance and rules, and improved communication between FAA and industry. In priority order, those recommendations called for developing a single master source for guidance organized by Title 14 of the Code of Federal Regulations (which covers commercial aviation); developing instructions for FAA staff with policy development responsibilities; reviewing FAA and industry training priorities and curriculums; setting up a board to provide clarification to industry and FAA on improving the clarity in final rules issued by FAA; and creating a communications center to act as a central clearinghouse to assist FAA staff with queries about interpretation of regulations.

We found that the committee took a reasonable approach in identifying these root causes and developing its recommendations. It compiled and reviewed case studies involving issues of regulatory application, obtained additional information by surveying industry stakeholders, and reviewed FAA regulatory guidance material. The recommendations are relevant to the root causes, actionable, and clear. The committee also considered the feasibility of the recommendations by identifying modifications to existing efforts and programs and prioritizing the recommendations.

FAA reported on July 19, 2013, that it is determining the feasibility of implementing these recommendations. The agency told us that it expected to develop an action plan to address the recommendations and metrics to measure implementation by December 2013. We note that measuring implementation may provide useful information, however, FAA is not intending to measure outcomes. Measuring outcomes can help in understanding if an action is having the intended effect.

Challenges Moving Forward

FAA’s certification and approval processes generally work well. However, when the certification and approval processes do not work well, the result can be costly for industry and FAA. Inconsistent interpretation of regulations can lead to rework by FAA and industry. Likewise, inefficient processes can require extra time and resources. FAA faces challenges in implementing the committees’ recommendations and further improving its certification and approval processes. FAA’s certification and approval workload is expected to grow over the next 10 years because of activities such as the introduction of new technologies and materials, such as composite materials used in airplanes, according to one industry committee report. Additional work will be needed to establish new means of compliance and establish new standards. In addition, FAA’s certification and approval workload is likely to increase substantially as the Next Generation Air Transportation System (NextGen) progresses and operators will need to install additional equipment on their aircraft to take full advantage of NextGen capabilities. Having certification and approval processes that work well will allow FAA to better meet these increasing workload demands and better ensure aviation safety in an era of limited resources.

To its credit, FAA has taken steps toward improving the efficiency of its certification and approval processes. It will be critical for FAA to follow through with its plans for implementing the key recommendations to achieve the intended efficiencies and streamlining. However, making fundamental changes to the certification and approval processes can require a cultural change by its workforce and resistance to change can cause delays. Some improvements to the processes, such as those requiring new rulemakings, will likely take years to implement and, therefore, will require a sustained commitment as well as congressional oversight.

Chairman LoBiondo, Ranking Member Larsen, and members of the Subcommittee, this concludes my prepared statement. I would be pleased to answer any questions at this time.

GAO Contact and Staff Acknowledgements

For further information on this testimony, please contact Gerald L. Dillingham, Ph.D., at (202) 512-2834 or dillinghamg@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony include Teresa Spisak (Assistant Director), Pamela Vines, Melissa Bodeau, David Hooper, Sara Ann Moessbauer, Josh Ormond, and Jessica Wintfeld.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.