Aviation Safety: Additional Oversight Planning by FAA Could Enhance Safety Risk Management
Published: 2014-06-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. aviation system is one of the safest in the world, reflecting the work of FAA, industry, and others to continually improve safety. To further enhance safety, in 2005, FAA began adopting a proactive, data-driven, risk-based approach to managing safety, referred to as SMS, and has proposed rules that would require SMS implementation for certain segments of the aviation industry.
GAO was asked to review SMS implementation in the aviation industry. This report addresses (1) the status of SMS implementation at FAA and in the aviation industry; (2) key challenges that FAA and industry face in implementing SMS; and (3) actions aviation stakeholders believe FAA could take to improve SMS implementation. GAO reviewed FAA documents and interviewed FAA officials. GAO also interviewed representatives from 20 selected aviation stakeholders, including commercial air carriers, certificated airports, repair stations, and design and manufacturing firms. Because the stakeholders were non-statistically selected based on their size, SMS implementation, and the industry segment represented, their views cannot be generalized to the industry or any industry segment.
section_title: What GAO Found
paragraphs: The Federal Aviation Administration's (FAA) Air Traffic Organization completed Safety Management System (SMS) implementation in 2010, and five other FAA organizations are implementing it now. SMS is an approach to collect and analyze safety data to identify hazards, manage risks, and take corrective action before an accident occurs. FAA's implementation activities include developing internal SMS guidance and procedures and using them to, among other things, identify hazards in the aviation system and provide oversight of the aviation industry. For example, FAA's Flight Standards Service is developing an SMS-based oversight system for the commercial air carriers it oversees. Although SMS is not yet required for commercial air carriers, airports, or any other industry segment, some are voluntarily implementing SMS as part of several FAA pilot projects. Of the 83 commercial air carriers, 77 are in the process of implementing SMS. FAA anticipates publishing a final rule in September 2014 requiring commercial air carriers to implement SMS. To a lesser extent, other industry segments are voluntarily implementing SMS. For example, according to FAA, 9 of the nation's largest airports are implementing SMS. FAA issued a proposed rule for airport SMS implementation, but development of a final rule has been delayed, and FAA has not yet determined if it will propose rules for other industry segments. Stakeholders and FAA officials speculated that the other industry segments may be waiting to implement SMS until FAA issues additional guidance or a final rule.
According to FAA officials, completing the rulemaking processes for commercial air carriers and airports has been a primary challenge to industry SMS implementation. Officials stated that one reason for delay has been difficulty in developing the benefit-cost analyses required for significant regulatory action. However, FAA is revisiting these analyses through the ongoing rulemaking process. Uncertainty about FAA plans for SMS oversight was among the key challenges for aviation industry SMS implementation. Although some inspector training has been provided, representatives from 9 of the 20 stakeholders GAO interviewed cited concerns that FAA inspectors may not be adequately trained to oversee industry SMS activities, and 6 expressed concerns that inspectors throughout FAA may not consistently interpret SMS regulations. However, FAA has not completed plans for its SMS oversight activities, including inspector training, and officials stated that they would not do so until the final rule is published. Without adequate planning of oversight and training of inspectors, FAA could find itself unprepared to meet its oversight responsibilities when final SMS rules are published.
Twelve of the 20 aviation stakeholders GAO spoke with identified additional FAA actions that could improve their SMS implementation efforts. For example, 4 stakeholders stated that providing SMS training to their employees was a challenge, and 2 suggested that FAA could assist by providing them access to FAA's SMS training. FAA indicated that it is considering industry stakeholder training needs and provided training through the pilot projects. Fourteen stakeholders were pleased with FAA's collaboration and communication, but 6 of them stated that this effort could be broadened. FAA updates its SMS website information, and FAA's most significant industry SMS forum is focusing more on SMS implementation.
section_title: What GAO Recommends
paragraphs: GAO recommends that FAA develop a plan for overseeing industry SMS implementation that includes providing guidance and training for FAA inspectors by the time final rules are published. GAO provided DOT with a draft of this report for comment. DOT provided technical corrections which were incorporated as appropriate.

section_title: Letter
paragraphs: The U.S. aviation system is one of the safest in the world with an  extremely low accident rate. This record reflects the efforts of the Federal  Aviation Administration (FAA), air carriers, airports, aviation  manufacturers, the National Transportation Safety Board (NTSB), and  others to continually improve aviation safety. Focus on safety continues to  be important, particularly in light of projected air traffic growth over the  next 20 years. In keeping with this anticipated growth, FAA is seeking to  further enhance safety by shifting to a proactive, data-driven, risk-based  approach to safety oversight referred to as a safety management system  (SMS).  SMS is a formalized process that involves collecting and  analyzing data on aviation operations to identify emerging safety  problems, determining risk severity, and mitigating that risk to an  acceptable level.  SMS is becoming the standard throughout the global  aviation industry. The International Civil Aviation Organization (ICAO)  requires SMS for the management of safety risk in air operations,  maintenance organizations, air traffic services, and airports as well as for  certain flight-training operations and organizations that design and  manufacture aircraft.  Further, ICAO has published safety management  requirements for its member countries that mandate that civil aviation  authorities—such as FAA—establish an SMS. FAA’s internal SMS  approach is designed to meet ICAO requirements. ICAO also directs civil  aviation authorities to require that certain types of aviation service  providers implement SMS. The adoption of SMS is important because the  low accident rate increases the importance of analyzing incident and  other data for root causes and indicators of potential hazards. According  to FAA, SMS can also address organizational and management issues  that have been identified as contributing to past accidents.
To this end, FAA is (1) engaged in implementing SMS internally across  selected organizations within the agency, (2) has undertaken SMS pilot  projects for certain segments of the aviation industry, and (3) has  developed proposed regulations that would require commercial air  carriers and certificated airports to implement SMS.  In 2012, we  reported on FAA’s internal implementation of SMS.  For this report, you  asked us to focus on the status of SMS implementation in the aviation  industry.  We examined the status of SMS implementation in four industry  segments—commercial air carriers, certificated airports, design and  manufacturing firms, and repair stations. We also examined the current  status of FAA’s internal SMS implementation efforts and FAA’s  development of SMS regulations. Specifically, this report addresses the  following questions:  1.  What is the status of SMS implementation within FAA and for key  segments of the aviation industry, including certificated airports,  commercial air carriers, repair stations, and design and manufacturing  firms?   2.  What are the key challenges FAA and the aviation industry face in  implementing SMS?   3.  What additional actions do aviation stakeholders believe FAA could  take to improve SMS implementation and potential effectiveness?
To assess the status of SMS implementation within FAA and four key  segments of the aviation industry, we reviewed FAA orders, advisory  circulars, and pilot program reports as well as FAA guidance for internal  SMS implementation and for implementation in the aviation industry. We  also interviewed FAA officials and industry trade group representatives.  To determine the key challenges FAA and the aviation industry face in  implementing SMS, we interviewed FAA officials and representatives  from 20 selected aviation stakeholders—commercial air carriers,  certificated airports, repair stations, and design and manufacturing  firms—that participated in SMS pilot projects. These 20 stakeholders, 5  from each industry segment, were non-statistically selected based on  size, SMS implementation, and industry sub-segment represented.   (Results of these interviews cannot be projected to all companies in any  of the given industry segments). We also reviewed documents from  FAA’s rulemaking projects for commercial air carriers and certificated  airports.  To identify additional actions FAA could take to improve SMS  implementation and potential effectiveness, we interviewed the industry  stakeholders described above and foreign aviation authorities that are  overseeing SMS implementation, including Australia’s Civil Aviation  Safety Authority, the European Aviation Safety Agency, and Transport  Canada. In addition, we interviewed two Canadian airlines and one  Canadian airport that have implemented SMS.  We also interviewed FAA  officials to obtain their perspectives on potential actions.  See appendix I  for a more detailed description of our scope and methodology.
We conducted this performance audit from May 2013 to June 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: When fully implemented, SMS provides a continuous approach to  managing safety risk, which FAA expects will improve aviation safety.  SMS is not an additional safety program that is distinct from existing  activities that accomplish an entity’s safety mission, such as quality  management, quality assurance, or similar activities. Rather, SMS  provides a set of decision-making processes and procedures to plan,  organize, direct, and control business activities in a manner that  enhances safety and ensures compliance with regulatory standards.  According to FAA, the overarching goal of SMS is to improve safety by  helping ensure that the outcomes of any management or system activity  incorporate informed, risk-based decision making.
SMS consists of four key components: (1) safety policy, (2) safety risk  management, (3) safety assurance, and (4) safety promotion.  (See fig.1.)   Together, these four components are intended to provide a systematic  approach to achieving acceptable risk levels. FAA provides its personnel  with guidance on the principles underpinning these components in its  official orders and other internal FAA guidance. To the industry, FAA  currently provides SMS guidance via advisory circulars, an SMS  newsletter, focus groups, and a dedicated page for the SMS program  office on the FAA website.
FAA is undertaking the transition to SMS in coordination with the  international aviation community, working with ICAO to adopt applicable  global standards for safety management.   ICAO first mandated SMS  worldwide for air traffic service providers in 2001.  ICAO later specified  that member states should mandate SMS implementation for airports, air  carriers, and others by 2009.  FAA began SMS implementation in 2005,  but FAA officials informed ICAO that the agency and industry would not  be able to meet the 2009 deadline. The United States filed a “difference”  with ICAO—indicating that it does not yet completely comply with the  standard—with the understanding that implementation is under way and  that FAA is in the midst of a rulemaking to require SMS for commercial air  carriers and certificated airports. There have been other actions within the  United States to encourage SMS implementation. For instance, in 2007,  NTSB recommended that FAA require all commercial air carriers to  establish an SMS and in 2011 added SMS for all modes of transportation  to its Most Wanted List. NTSB identified SMS as one of the most critical  changes needed to reduce the number of accidents and save lives.
FAA’s implementation of SMS will affect how the agency oversees the  aviation industry.  Historically, FAA oversight of airlines, airports, and  other regulated entities has involved oversight of such things as  operations and maintenance.  Once SMS requirements for industry are in  place, FAA will continue this oversight, but will also apply SMS principles  to its processes for oversight.  Specifically, the agency will provide  oversight of the safety management systems of service providers, such  as commercial air carriers and certificated airports, to help ensure that  they are managing safety within their operations through SMS.
Internally, FAA has directed the Air Traffic Organization, Airports  Organization, Aviation Safety Organization (Aviation Safety), the Office of  Commercial Space Transportation, the Next Generation Air  Transportation System (NextGen) Office, and the Office of Security and  Hazardous Materials Safety to implement SMS.  Within Aviation Safety,  the Aircraft Certification Service (Aircraft Certification), and the Flight  Standards Service (Flight Standards), among others, are also  implementing SMS.
Oversight of the aviation industry’s implementation of SMS will be  conducted by FAA’s inspector workforce, some of whom have been  involved in SMS pilot programs. The inspector workforce includes  approximately 4,000 Flight Standards aviation safety inspectors around  the country who oversee certificate holders, including commercial air  carriers and repair stations, among others. Approximately 950 aircraft  certification engineers and inspectors are responsible for overseeing firms  that design and manufacture aircraft, aircraft engines, propellers, and  other parts and equipment.  The nation’s certificated airports are  overseen by approximately 40 airport certification inspectors assigned to  regional offices.
In 2009, FAA issued an Advance Notice of Proposed Rulemaking  (ANPRM) to solicit comments on establishing a regulatory framework to  require SMS for various sectors of the aviation industry, including  commercial air carriers, repair stations, and design and manufacturing  firms.  FAA received public comments on the ANPRM but withdrew it  following a 2010 congressional mandate for FAA to issue a rule requiring  air carriers to implement SMS.  FAA subsequently issued a Notice of  Proposed Rulemaking (NPRM) in November 2010 that would require  SMS for commercial air carriers.  FAA simultaneously developed a  proposed rule for certificated airports and issued an NPRM in October  2010 that would also require SMS at those facilities.  The status of  rulemaking efforts for repair stations and design and manufacturing firms  are discussed below.
section_title: FAA’s Internal Implementation of SMS Continues, While Aviation Industry Segments Are in Various Stages of Voluntary Implementation
subsections: section_title: FAA Organizations Continue SMS Implementation
paragraphs: Five FAA organizations are proceeding with SMS implementation. A sixth  organization, Air Traffic, completed implementation in 2010. Figure 2  shows the FAA organizations that are implementing SMS as well as  industry segments included in this report that these selected  organizations oversee.
The Air Traffic Organization completed SMS implementation in 2010 and  now uses SMS-based processes to identify hazards, enact mitigations,  and assess the extent to which the mitigations are working. Since we  last reported on their progress in 2012, the other organizations—Aviation  Safety, the Airports Organization, the NextGen Office, and the Office of  Commercial Space Transportation—have continued SMS implementation  by developing SMS guidance and establishing internal SMS procedures,  and an additional organization, the Office of Security and Hazardous  Materials Safety has begun SMS implementation.
Aviation Safety—including two of its components, Flight Standards  and Aircraft Certification—are developing or updating SMS guidance  and plans, according to FAA officials. For example, Flight Standards  is developing for an SMS-based oversight system for commercial air  carriers.
The Airports Organization established procedures for conducting  safety risk management for certain airport actions that require FAA  approval.
At the time of our last report, the NextGen Office and the Office of  Commercial Space Transportation were implementing SMS in line  with FAA’s strategic goal in its 2009-2013 Flight Plan to implement  SMS policy in all appropriate FAA organizations. In May 2013, FAA  issued a revised order that directed these two organizations to  implement SMS.  In addition, an FAA order directed another organization, the Office of  Security and Hazardous Materials Safety, to implement SMS. That  organization is updating its inspections and investigations manual to  be more consistent with SMS principles and is coordinating with other  FAA organizations’ SMS efforts to identify hazards in the transport of  hazardous materials.
section_title: Although Not Currently Required, Some Aviation Stakeholders Are Voluntarily Implementing SMS
paragraphs: Voluntary implementation of SMS varies by industry segment, while FAA  continues rulemaking to require SMS implementation by commercial air  carriers and certificated airports. Beginning in 2007, FAA initiated  various pilot projects for commercial air carriers, certificated airports,  repair stations, and design and manufacturing firms, among others, to  encourage voluntary SMS implementation and to guide rulemaking  activities. The pilot projects varied among FAA organizations. The  Airports Organization and Aircraft Certification pilot projects were  conducted for fixed periods of time, while the Flight Standards pilot  project, which began in 2007, is ongoing. Flight Standards officials said  that the pilot project for commercial air carriers would end after the final  rule is published and that they hope to transition the pilot project for the  remaining certificate holders (e.g., repair stations) into a more formal  program. Participant selection for the pilots also varied. FAA selected the  participants for the Aircraft Certification pilot for design and manufacturing  firms, while the Flight Standards and Airports Organization pilot project  participants volunteered to participate. The pilot projects are more fully  described in forthcoming sections of this report.
While most commercial air carriers—the first industry segment for which  SMS implementation will be mandatory—are moving forward with  implementation, a small proportion of certificated airports, repair stations,  and design and manufacturing firms have begun SMS implementation.  For example, less than 1 percent of the more than 4,700 repair stations  are currently implementing SMS. These three industry segments,  according to FAA officials and industry stakeholders, may be waiting for  additional guidance or rules, which may not be finalized for a number of  years, if the time frames for finalizing the air carrier and airport SMS rules  are any indication.
subsections: section_title: Commercial Air Carriers
paragraphs: Seventy-seven of the 83 U.S. commercial air carriers are at varying levels  of SMS implementation as FAA nears completion of the final SMS rule.  Flight Standards developed four SMS implementation levels to  benchmark the progress of commercial air carriers (see table 1 below for  additional details on the Flight Standards SMS implementation levels). In  general, Flight Standards officials work with commercial air carriers to  determine their SMS implementation level. Flight Standards reports the  SMS level for each participating stakeholder in its monthly SMS  newsletter. According to the March 2014 SMS newsletter, of the 77  commercial air carriers implementing SMS, the vast majority of them (58)  were at SMS level one or two, 10 were at level three, and 9 were at the  final continuous improvement level.
The Airline Safety and Federal Aviation Administration Extension Act of  2010 stipulated that FAA issue a final SMS rule for commercial air  carriers. In November of 2010, FAA published an NPRM that would  require all commercial air carriers to implement an SMS that meets the  requirements of the new regulation and is acceptable to FAA within 3  years of the effective date of the final rule.  According to FAA, it did not  meet the statutory deadline of August 1, 2012, to issue the final rule due  to the lengthy internal DOT review process and complexity of the required  benefit-cost analysis. The DOT Significant Rulemaking Report for June  2014 indicated that FAA anticipates publishing the final rule in September  2014. (See fig. 3.)  Commercial air carriers that have already begun to  implement SMS must ensure that their efforts comply with the  requirements of the final rule.
While FAA’s rulemaking to require SMS at all 545 certificated airports has  been delayed, FAA officials estimated that 9 of the nation’s largest  certificated airports were voluntarily implementing SMS as of April 2014.  Between 2008 and 2012, the Airports Organization conducted three  voluntary SMS pilot studies to provide airports opportunities to gain  knowledge about SMS and provide feedback to FAA in its rulemaking  efforts. The pilot studies also allowed certificated airports to share their  SMS implementation practices with other airports. Thirty-one certificated  airports of various sizes participated in at least one of the pilot studies.  Twenty-one of the 22 airports in the first pilot study and 1 airport that did  not participate in any of the pilot studies received a grant to develop an  SMS plan. To further encourage SMS implementation, in August 2013,  the Airports Organization published guidance that provides some  certificated airports access to federal funds for certain SMS activities,  including making SMS management software eligible for funding through  the Airport Improvement Program. FAA officials indicated that another  goal of the guidance was to ensure that airports would be aware of the  basic elements required for SMS implementation plans and manuals.
FAA began the SMS rulemaking process for certificated airports in July  2008 and in October 2010 published an NPRM to require that all  certificated airports develop and maintain an SMS that is approved by  FAA. According to FAA officials, the nature of the comments received on  the NPRM led them to significantly modify the proposed rule, and provide  another period for public comment. As a result, in December 2012, FAA  announced that it would amend the next scheduled step in the rulemaking  process and issue a supplemental notice of proposed rulemaking  (SNPRM), which was originally scheduled for publication in December  2013, rather than publish a final rule. FAA is considering changes in the  proposed rule’s applicability and to some proposed requirements,  including SMS implementation options for various sizes of certificated  airports.  FAA has not yet issued the SNPRM, but in June 2014 FAA  estimated that the SNPRM would be published for public comment in  October 2014. With FAA’s decision to issue an SNPRM, it is likely that  required SMS implementation for certificated airports will not occur for  some time. Figure 4 summarizes key dates related to SMS rulemaking for  certificated airports, as indicated in the DOT Significant Rulemaking  Report for June 2014.
Few repair stations are currently implementing SMS, and FAA has not yet  determined if it will conduct rulemaking to require SMS at repair stations.  As of March 2014, just 15 repair stations (of more than 4,700) were  implementing SMS through the Flight Standards SMS pilot project,  according to an FAA newsletter. As it does with air carriers, Flight  Standards uses the four SMS implementation levels to benchmark and  determine the progress of repair station SMS implementation. Thirteen  were at SMS level one or two. Our interviews with 5 repair stations that  participated in the pilot program suggested that there may be fewer than  14 repair stations currently implementing SMS because representatives  we interviewed from 4 of the repair stations indicated that they had either  ceased or never actually initiated their SMS efforts. Among the reasons  these repair station representatives cited for not moving forward included  a lack of FAA support. For example, representatives from one repair  station told us that FAA did not respond to their questions on SMS policy  changes and did not provide feedback on the gap analysis they  performed.  FAA officials indicated that there are reasons that more repair  stations may want to implement SMS. Flight Standards officials said that  once the commercial air carrier SMS rule is finalized, repair stations might  want to implement their own SMS so that it could be recognized by the  commercial air carriers they may contract with. A senior FAA official noted  that because the commercial air carrier SMS rule will include identifying  risks posed by repair stations, it would have an impact on repair station  operations. Additionally, as other countries complete SMS rulemaking,  some repair stations may want to implement SMS for international  business reasons.
section_title: Design and Manufacturing Firms
paragraphs: Several design and manufacturing firms were implementing SMS as of  April 2014, while an FAA rulemaking committee studies SMS  incorporation into the design and manufacturing environment of  approximately 3,000 firms. Aircraft Certification conducted an SMS pilot  project for design and manufacturing firms from 2010 to 2012 with 11  participants. The participants were selected to represent the diversity of  the industry in terms of size and types of products designed and/or  manufactured. According to an FAA official, approximately 6 pilot project  participants were continuing SMS implementation as of April 2014. In our  discussions with several design and manufacturing firms that participated  in the pilot, we found that 2—Honeywell and Boeing—were planning to  complete SMS implementation in 2015. Honeywell representatives  indicated that they expect to complete SMS implementation by mid-2015.  Boeing representatives said their company was implementing an SMS as  defined to ICAO standards and also expect to complete SMS  implementation by mid-2015.
In October 2012, FAA chartered a 2-year aviation rulemaking committee  (ARC) to, among other things, study SMS incorporation into the design  and manufacturing firm environment. According to a January 2014 FAA  plan, once the committee has completed its work, FAA will begin a  rulemaking process that will result in a change to the current rule for  certificating design and manufacturing firms. The plan anticipates the  release of an NPRM by January 2016 and publication of a revised final  rule by June 2017 to require, among other things, design and  manufacturing firms to implement SMS. However, an FAA official stated  in April 2014 that FAA had not yet determined when an SMS requirement  might be proposed for design and manufacturing firms.
section_title: FAA and Stakeholders Cited Rulemaking, Data Issues, and Stakeholder Uncertainty as Key Challenges to Implementing SMS
subsections: section_title: FAA Cited Challenges in Completing SMS Rulemaking
paragraphs: FAA officials stated that a primary challenge in developing SMS rules for  commercial air carriers and certificated airports is developing the benefit- cost analyses. FAA officials explained that estimating the expected  benefits for SMS rules for air carriers and airports relies on analysis of the  economic cost of past accidents that might not have occurred had SMS  been in place.  However, determining which past accidents might not  have occurred based on SMS implementation either by air carriers or  airports or both is complex and somewhat subjective, and this  determination can be affected by other safety improvements that have  occurred in recent years or that are being proposed. For example, if  subsequent to a particular accident, some other safety improvements  have been implemented that would have reduced its likelihood, only a  portion of the value of the damage and harm related to that accident can  be included in the current benefit calculation for future SMS since the  likelihood of the same accident occurring has already been somewhat  mitigated. Additionally, an airport industry trade organization expressed  concerns in its public comments on the airport NPRM that the proposed  rule incorrectly assumed that certain accidents would be mitigated by an  airport SMS because those occurrences happened on airport grounds but  in areas controlled by air carriers.  The trade organization reviewed 53 of  the 89 accidents included in the FAA analysis.  In the trade organization’s  view, 37 of these 53 accidents had little or limited connection to safety  actions that an airport might take within its SMS. The likelihood of the  accident or incident occurring could therefore be mitigated by the air  carrier’s SMS implementation.  Some accidents might also be avoided  based on joint implementation of SMS systems by both the air carriers  and airports.  In such cases, expected benefits have to be apportioned  across the benefit-cost analyses of the two separate SMS systems—that  is, the SMS for air carriers and the SMS for airports—otherwise expected  benefits would be double counted.  In addition, a number of comments on  the proposed rules for both the air carrier and airports questioned the  estimated costs imposed on aviation stakeholders related to the  implementation of the SMS systems.  FAA, as part of its rulemaking  process, revisited the benefit-cost analysis after reviewing public  comments on the proposed rules.
Another challenge FAA faces in implementing SMS rules is planning for  how data collected for proactive safety efforts will be used. Over the  years, we have identified numerous challenges related to FAA data and  made a number of recommendations, some of which FAA has addressed,  to improve the use of data that FAA collects. One recommendation of  particular significance to SMS implementation, however, has yet to be  fully implemented.  In 2010, we recommended that FAA develop a  comprehensive plan that addresses how data fits into FAA’s  implementation of a proactive approach to safety oversight.  We indicated  that the plan should fully describe the relevant data challenges, analytical  approaches to be used, staffing requirements, and the efforts to address  them. Because SMS relies on data for maximum benefit, we concluded  that a comprehensive data plan by FAA would help expand its capability  to improve aviation oversight. DOT concurred with this recommendation,  and although FAA has taken actions such as implementing data quality  controls consistent with our standards for data quality, it has not yet  completed the recommended plan. For example, it has not outlined how it  will analyze or use data gathered under SMS, assessed the staffing  requirements it will need to conduct the data analysis, and indicated a  timeline to complete a plan.  GAO believes this recommendation remains  valid.
section_title: Aviation Stakeholders Cited Uncertainty over FAA Oversight, Implementation Costs, Data Protection Issues, and Final Rule Requirements among Their Key Challenges
paragraphs: SMS implementation challenges have been identified during FAA’s SMS  pilot projects and the rulemaking processes for commercial air carriers  and certificated airports.  We also identified some of these challenges in  our prior SMS work. We selected five frequently cited challenges to  discuss with aviation stakeholders: data sharing and protection, resource  constraints, implementation costs, FAA oversight, and collaboration with  other aviation stakeholders.  Our discussions with 20 selected aviation  stakeholders that participated in at least one FAA SMS pilot project  confirmed that these challenges still exist for some stakeholders.  Based  on these discussions, we found that a majority of the selected  stakeholders identified uncertainties about FAA’s oversight plans (17  stakeholders); uncertainties about SMS implementation costs (14  stakeholders); and data protection (16 stakeholders) as challenges. In  addition, we found that uncertainty about the FAA’s final rule  requirements raised some additional concerns with the stakeholders we  interviewed, and half of the stakeholders said that of the four SMS  components, the safety risk management is the most challenging to  implement because of the time and analyses required to identify and  mitigate risks.
subsections: section_title: Uncertainty about FAA’s SMS Oversight Plans
paragraphs: Seventeen stakeholders identified their uncertainty about FAA’s plans  and preparation for oversight of industry’s SMS implementation as a  challenge, with 12 identifying it as a great or very great challenge.  Specifically, they reported a lack of information about FAA’s intended  oversight plans, training for FAA inspectors responsible for oversight, and  the potential for inconsistent regulatory interpretation because FAA has  not shared that information with stakeholders. Just as the move to SMS  will change how stakeholders manage risk, FAA’s oversight approach will  also need to undergo some changes. (See table 2.) According to FAA,  while its inspectors will continue to check for compliance with safety  regulations and retain enforcement authority, oversight will also include  reviewing the SMS guidance and records and ensuring that processes for  identifying and mitigating risks are being carried out.
Internal control standards, an FAA rulemaking committee, and the ICAO  Safety Management Manual all state the need to establish plans to  ensure that goals and objectives are met. Additionally, we have  previously found that communicating internal control efforts on a timely  basis to internal and external stakeholders, such as FAA inspectors and  aviation industry stakeholders, helps ensure that effective oversight can  take place. A 2009 SMS aviation rulemaking committee recommended  that FAA ensure that sufficient planning, policy and guidance, and  workforce training are in place prior to SMS implementation to  accommodate efficient, timely, objective and consistent oversight. Internal  control standards also state that personnel should be provided with the  right training and tools to perform their duties, and the ICAO Safety  Management Manual notes the significance of training to successful SMS  implementation and indicates that priority for training must be given to  personnel involved in implementation or oversight of SMS.    While  acknowledging that its oversight role will be changing, FAA has not yet  completed plans for overseeing industry’s implementation of SMS,  including plans to develop guidance and train inspectors about their SMS  oversight functions and responsibilities. Nine of the aviation industry  stakeholders and three industry trade groups we interviewed expressed  concerns that FAA inspectors will not be prepared to oversee industry  implementation of SMS.  These concerns related primarily to a lack of  SMS training for FAA inspectors.  FAA officials told us that they plan to  finalize SMS rules before addressing oversight issues.
Based on the NPRM, commercial air carriers would have 3 years after the  effective date of the final rule to comply with SMS implementation  requirements.  When the final rule for commercial air carriers becomes  effective, FAA inspectors will assume oversight responsibility to ensure  that commercial air carriers comply with the final rule. Included with this  responsibility is approving commercial air carriers’ SMS plans. According  to Flight Standards data, as of November 2013, over 2,700 of  approximately 3,900 of their inspectors had completed initial training in  SMS activities. This includes inspectors who oversee commercial air  carriers.  However, even though the publication of the final SMS rule for  air carriers is expected before the end of the year—currently planned for  September 2014—Flight Standards has not yet established guidance for  inspectors for overseeing commercial air carriers’ implementation of SMS  or updated its inspector training program to incorporate such guidance. In  addition, even though a large number of inspectors have received initial  SMS training, stakeholders remain concerned about inspectors’  knowledge of SMS as well as the potential for inconsistent interpretation  of SMS requirements.
At the same time, the 77 commercial air carriers that have been  voluntarily implementing SMS will need to review their SMS to ensure  compliance with the final rule and may request FAA guidance during that  process to determine whether and what modifications need to be made.   Given that most commercial air carriers already have started  implementing SMS, the demand for inspector guidance or assistance may  come quickly.  For example, inspectors might be asked to review a  commercial air carrier’s SMS to ensure that all required components are  included. Without taking proactive preparations for oversight before  requiring SMS, FAA could find itself with oversight responsibilities for  which it is not fully prepared.  Consequently, the intended benefits of  implementing this proactive approach to managing risk could be limited.
For other industry segments, notably airports and design and  manufacturing firms, that are likely to be next to face an SMS  requirement, FAA has taken initial steps to train inspectors working in  those areas.  In 2008, the Airports Organization provided 3 days of basic  SMS instruction to airport safety inspectors during the organization’s  annual recurrent training.  Officials in the Airports Organization  acknowledged that, should FAA finalize an SMS rule for certificated  airports, up-to-date training will need to be developed and delivered to the  approximately 40 airport safety inspectors, including training in detailed  inspection procedures for oversight of an airport’s SMS.  Likewise,  Aircraft Certification plans for some of its staff to review the Flight  Standards initial SMS training course to determine what type of SMS  training is necessary for some of its estimated 950 aircraft certification  engineers and inspectors.
In addition to training, six aviation stakeholders we interviewed expressed  concerns about potentially inconsistent regulatory and other  interpretations by FAA inspectors.  This concern was identified by smaller  stakeholders we interviewed as well as those that hold multiple  certificates overseen by different FAA organizations. For example, one  stakeholder we interviewed indicated that the company holds certificates  for multiple facilities in multiple FAA districts, and is concerned that  inspectors from different districts may have different interpretations of the  SMS rules once they are implemented. We found in 2010 that ensuring  consistency in regulatory interpretations has been a long-standing issue  for FAA. This was also identified in the SMS ARC final report as an issue  that FAA should address prior to SMS implementation.  In 2012, FAA  formed another aviation rulemaking committee to address consistency in  regulatory interpretation by Flight Standards and Aircraft Certification. The  committee made six recommendations to improve the consistency of  regulatory interpretation and improve communication with industry. In July  2013, FAA provided a preliminary implementation plan to address the  recommendations and indicated a detailed implementation plan with  milestones would be developed, but provided no time frames for when  that detailed plan would be released.
Similarly, two stakeholders that operate both domestically and overseas  expressed concerns about whether FAA’s SMS regulations will be  consistent with the ICAO framework that the regulations are designed to  meet and harmonized with the SMS requirements of other countries.   These stakeholders are concerned that potential inconsistencies could  pose problems if different standards exist in different countries.  For  example, one repair station with overseas operations we spoke to said  that any SMS they may be required to implement to meet FAA regulations  must also comply with the SMS requirements in other countries where  this repair station operates. The need for international acceptance of a  service provider’s SMS was also identified as an issue for FAA to address  in the SMS ARC’s final report in 2010.
section_title: Uncertainty about SMS Implementation Costs
paragraphs: Aviation stakeholders we interviewed largely had not conducted cost  assessments of their SMS efforts (only 5 of 20 had done so), and the  difficulty in assessing these costs could present a challenge to those  considering voluntary implementation of SMS. Fourteen of the 20  stakeholders told us that identifying and assessing SMS costs was a  challenge. Further complicating cost estimates, three stakeholders  indicated that implementation costs are spread throughout the  organization and that isolating those specifically related to SMS would be  difficult.  For example, one stakeholder indicated that because the  computer systems it purchased to support SMS also support other  operational objectives, it has not attempted to separate out SMS costs. In  addition, seven stakeholders said that the absence of a final rule  specifying SMS requirements made assessing costs difficult.
Though stakeholders we spoke with were generally unable to determine  the cost of implementing SMS, they were able to identify some types of  related expenditures.  We asked each of them about specific costs  related to SMS implementation, and they reported incurring the following  types of costs:
14 trained employees on SMS or SMS concepts,
10 incurred recordkeeping costs,
10 purchased computer systems, and
7 hired new employees.
Other costs identified by stakeholders included those related to safety  promotion and mitigating hazards identified through SMS implementation.
section_title: Concerns about Collecting and Sharing SMS Data
paragraphs: Industry stakeholders we spoke with had concerns about sharing and  protecting their safety data; 16 of the 20 stakeholders we interviewed  identified this as a challenge, with 7 identifying this challenge as great or  very great. In March 2010, the SMS ARC final report identified this as an  issue and recommended that prior to the promulgation of an SMS rule,  protections be put in place to ensure that safety information and  proprietary data are protected from disclosure and use for other  purposes, such as enforcement actions.
The 2012 FAA Modernization and Reform Act (the 2012 Act) included a  provision that placed a limitation on the disclosure of safety information  under the federal Freedom of Information Act (FOIA) for information  gathered for the purposes of developing and implementing an SMS.    However, information provided by or to publicly owned airports is also  subject to state FOIA laws, which are not covered under the protections in  the 2012 Act. Commercial air carriers, repair stations, and design and  manufacturing firms are privately owned and not directly subject to state  FOIA laws. However, any data airports collect and any data shared with  airports could, according to FAA officials and industry experts, be subject  to state FOIA laws, because most certificated airports in the U.S. are  owned by a public entity such as a state, city, port, or other local or  regional government body. According to officials in the Airports  Organization, a federal legislative resolution to override state laws is not a  feasible option. These officials reported that in some locations, airports  have begun to work with state legislative bodies to address the issue of  data protection and disclosure.
Without legal protections to prevent data disclosure, stakeholders told us  they feel at risk in a variety of potential scenarios related to SMS  implementation.
Litigation for damage (real or perceived) caused by incidents  disclosed through SMS—Eight stakeholders we spoke with, as well as  the 2010 SMS ARC, indicated that protections are needed not only to  prevent the disclosure of safety data through the state FOIA process,  but also from disclosure as part of litigation.  For example, a  certificated airport must meet the regulatory requirements of 14 C.F.R.  Part 139.  One standard included in Part 139 is that holes in airport  pavement must not exceed 3 inches in depth.  An airport  representative we met with indicated that if a hole less than 3 inches  deep is noted through a reporting system implemented under SMS,  the airport may postpone that repair until a later time as it is within the  maximum depth allowed under the regulations and the risk is  acceptable. Additionally, it would not likely negatively impact the  operations of a large aircraft on the runway.  However, it could pose a  hazard for smaller general aviation aircraft.  If the airport does not  make that repair and the hole factors into an accident involving a  general aviation aircraft, the extent of the airport’s responsibility and  legal liability is uncertain and may become the subject of a lawsuit.   Because the intent of SMS is to correct hazardous conditions before  an accident occurs, stakeholders want protections that address their  vulnerability to potential litigation.
Misinterpretation of safety data by a third party—If safety data is  released to the public under a state FOIA, the aviation industry is  exposed to potential misinterpretation, misuse, and further  dissemination of the data by third parties. For example, one  stakeholder expressed concern that safety data could be obtained,  reviewed, analyzed, or disseminated by a third party without the  proper context, including how and why the data were collected and  the limitations of their use.  Without the proper context, this  stakeholder stated that dissemination would lead to misinterpretation,  which may then be further disseminated by others such as policy  makers or the media.
Enforcement actions by FAA or other regulatory bodies—As we  discussed earlier, the safety risk management component of SMS  includes systems that allow individuals to identify potential safety  hazards, assess the risks arising from those hazards, and develop  mitigation plans to reduce or eliminate those risks. ICAO guidance  indicates that in an SMS environment, procedures should be in place  to ensure information obtained under SMS will not be used for  enforcement actions. However, four stakeholders expressed concern  that data about potential hazards could be used beyond SMS.  For  example, a repair station’s SMS would address identified hazards  and, if necessary, develop a mitigation plan to address those hazards.  However, if an identified hazard also constitutes a regulatory violation,  the company might be subject to FAA enforcement action. The SMS  ARC recommended in 2010 that FAA should establish a policy or  regulation that provides limits on enforcement actions applicable to  information that is identified or produced by SMS.  Currently, FAA  foregoes civil penalty actions when violations are promptly disclosed  to FAA under a voluntary disclosure program, subject to some  limitations.  FAA believes that the open sharing of apparent violations  and a cooperative approach to solving problems will enhance and  promote aviation safety.  Current FAA policy covers commercial air  carriers, repair stations, and design and manufacturing firms, but not  certificated airports.  It is unclear if FAA will extend this coverage to  information provided as part of SMS efforts not currently covered.
In 2012, we also found the protection of data to be a concern and  recommended that FAA consider strategies to address concerns that may  adversely affect data collection and data sharing—essential to realizing  the benefits of SMS.  FAA began discussions with airports and airport  associations in 2013 to determine what type of SMS data should be  considered for protection.  FAA also received and considered comments  from airports and airport organizations about data collection, protection,  and sharing in response to the NPRM for airports.  We concluded in our  September 2012 report that the success of SMS relies heavily on the  sharing of safety data and that without appropriate protections, the  willingness of the aviation industry to share safety data will likely be  jeopardized.  Although protections exist for safety data from commercial  air carriers, FAA has determined that a similar legislative approach is not  feasible for data from certificated airports.  However, according to FAA,  some airports are working to address data protections under state laws  through outreach to state agencies and legislators.
section_title: Uncertainty about Final Rule Requirements
paragraphs: Four of the stakeholders we spoke with felt that delays in the rulemaking  process were detrimental to their implementation of SMS while others  were confident that their SMS implementation would meet forthcoming  requirements.  Representatives from two industry stakeholders we spoke  with told us that they did not want to move forward in implementing SMS  until rulemaking is complete, largely because they first want to know what  the requirements will be. They explained that they were concerned about  the time and resources required to rework the SMS should it not meet the  final rule requirements.
Fourteen of the stakeholders we interviewed were concerned that the  SMS requirement would not be scalable and flexible, that is, one which is  sufficient to apply to a broad range of organizations from small operators  to large ones with multiple facilities and certifications.  Some of the  concerns were based on the stakeholders’ views that FAA tends to  fashion one-size-fits-all regulations or regulations that focus on one  subset of a stakeholder group.  For example, stakeholders cited a  concern that FAA will establish regulatory requirements that are  achievable for larger operators but may require more significant efforts by  small operators. The NPRM for the commercial air carrier SMS  requirement states that the proposed regulation is designed to be  performance-based. FAA explained that because the regulation would be  performance-based, it would allow commercial air carriers to comply  through a variety of methods (suggesting flexibility) and to accommodate  a variety of business sizes and models (suggesting scalability). The  NPRM for the SMS requirement at certificated airports states that FAA  envisions SMS as an adaptable and scalable system. FAA explained that  an SMS could be developed by an organization to meet its unique  operating environment.  Accordingly, FAA stated in the NPRM that it  would prescribe only the general framework for an SMS.
Several stakeholders also expressed concerns about how SMS  regulations would affect their current efforts to improve safety. For  example, one design and manufacturing firm we spoke with reported that  it began using system safety techniques similar to those used in SMS in  the 1990s.  Although this stakeholder participated in an FAA pilot, the firm  has developed an SMS approach in house based on the ICAO model.   Fourteen stakeholders reported that prior to implementing SMS they had  undertaken their own proactive safety or quality assurance efforts.  FAA’s  SMS ARC recommended in 2010 that SMS regulations allow for the  incorporation of existing safety management efforts already in place in  order to prevent duplicative safety efforts.  FAA indicated that such  programs could be used to build an operational SMS. Other stakeholders  expressed concern that the content of the final rule may require them to  adopt practices that may not improve safety in their company while  imposing additional burdens.  For example, one stakeholder hoped it  would not be required to document every hazard identified and mitigated,  stating that such documentation did not necessarily improve safety.
section_title: Safety Risk Management Challenge
paragraphs: We also asked all 20 industry stakeholders which of the four components  of SMS would be the most challenging to implement, and more  stakeholders (10) cited safety risk management than any other  component. As previously discussed, safety risk management is designed  to examine a company’s operational functions and operational  environment to identify hazards and analyze associated risks.  The intent  of this process is to focus on the areas of greatest risk from a safety  perspective and on mitigation, taking into account such factors as  complexity and operational scope.  Specifically, stakeholders indicated  that conducting a risk analysis can be time consuming.  A robust analysis  includes identifying a wide range of risks, and determining which would  require mitigation—a process that some stakeholders were concerned  may keep stakeholder staff away from other duties.  A number of  stakeholders also noted the difficulty in identifying risks in some cases,  including industry changes that may have an impact on risk.
section_title: Stakeholders Identified Additional Actions FAA Could Take to Improve SMS Effectiveness and Address Implementation Challenges
paragraphs: Although 12 of the 20 aviation stakeholders we interviewed are continuing  with their SMS implementation, they identified actions FAA could take to  improve SMS effectiveness and help address the challenges of  implementation. These suggestions target SMS training, guidance, and  collaboration and communication.
subsections: section_title: SMS Training for Industry
paragraphs: Although we did not ask specifically about any internal training programs  developed by the 20 aviation stakeholders we spoke with, four  stakeholders cited some difficulties with obtaining needed SMS training  for their staff, including allocating time and resources for the training and  finding training that was specific to their segment of the industry. Further,  a 2012 study by the Transportation Research Board of FAA’s airport pilot  studies found that pilot participants identified time restrictions and funding  as obstacles to developing an SMS training plan for airport personnel.   ICAO recommends that civil aviation authorities, like FAA, facilitate the  SMS education or training of its stakeholders where feasible or  appropriate. To that end, stakeholders suggested an action that FAA  could take to mitigate stakeholder difficulties in this area. Specifically, two  stakeholders we interviewed felt that allowing their staffs to attend the  same training as FAA personnel could be beneficial in addressing  differences in regulatory interpretation and for increasing industry’s  understanding of FAA’s definitions and concepts as they relate to SMS.  According to FAA officials, they have considered the training needs of  industry stakeholders and provided some training through the pilot  projects.
section_title: SMS Guidance
paragraphs: Although, as required by ICAO, FAA has provided, as well as updated,  guidance to help stakeholders to implement SMS, some stakeholders  saw a need for additional guidance and updates.  For example, one  stakeholder noted that FAA should create one risk matrix to be used by  all organizations.  Two stakeholders suggested standardizing SMS  definitions across FAA.  Another stakeholder suggested that FAA create a  single document on the SMS framework.  FAA has already responded to  these concerns with an April 2012 order designed to establish common  language standards for some SMS terminology; the order also provides a  single risk matrix for the agency.  Stakeholders also suggested that FAA  develop additional guidance to aid them in documenting work processes,  assessing organizational hazards, and estimating SMS costs. FAA plans  to disseminate additional SMS guidance for air carriers and airports when  final rules for these industry segments are published. The final rule for air  carriers is expected to be published in September 2014.
section_title: Collaboration and Communication
paragraphs: Fourteen of the 20 stakeholder representatives we interviewed were  pleased with their collaboration and communication with FAA during the  pilot projects; six stakeholders, however, noted that FAA could take  additional actions to improve collaboration and communication in line with  ICAO’s guidance for SMS. This guidance recommends establishing an  appropriate communication platform to facilitate SMS implementation,  particularly for SMS requirements and guidance material.  FAA has used  numerous methods to communicate about and encourage collaboration  regarding SMS. These methods include the SMS newsletter, SMS focus  groups, participation in the Safety Management International  Collaboration Group, and the various SMS pilot projects. However,  some stakeholders suggested that additional opportunities exist for the  agency to share lessons learned. For example, one stakeholder was not  aware of any FAA communications on its progress regarding the data  protection and sharing issue, and three others thought that FAA needed  to encourage the industry to share lessons learned through existing  forums. Two stakeholders suggested using the Commercial Aviation  Safety Team  as an example of, and the Aviation Safety Information  Analysis and Sharing system as a potential tool for, improved SMS  collaboration between FAA and commercial air carriers. Two stakeholders  also recommended broadening current FAA collaboration and  communication efforts by opening membership in SMS focus groups and  other working groups to additional industry members, such as foreign air  carriers; increasing the number of conferences and meetings where  information-sharing can take place; and increasing FAA attendance at  these forums.  The Airports Organization noted that it continually updates  its SMS website, and Flight Standards indicated that it is allowing its most  significant forum with the aviation industry, InfoShare, to focus more on  SMS implementation and management.
section_title: Conclusions
paragraphs: Maintaining a high level of safety in the U.S. aviation industry is a shared  responsibility among FAA, air carriers, airports, and other stakeholders.  FAA continues to implement SMS internally, in accordance with ICAO  requirements, and industry stakeholders such as commercial airlines,  certificated airports and others have begun to voluntarily adopt SMS in  advance of regulations requiring its implementation. The successful  implementation of SMS, with its proactive, risk-based approach to  maintaining aviation safety, could help ensure the continued safety of the  U.S. aviation system.  However, the lack of a comprehensive plan for  using data, which we recommended in 2010, may limit the effectiveness  of SMS once it is implemented.  Although FAA has taken steps to  implement data quality controls that meet our standards, it has not  disclosed its plans for using data collected or to ensure that it has the  requisite staffing to make use of the data to be collected from commercial  air carriers, certificated airports, and others.  We continue to believe that  the success of SMS relies on data for maximum benefit and that such a  data plan will strengthen FAA’s capability to improve aviation oversight. In  addition, the SMS ARC recommended in March 2010 that, among other  things, FAA must ensure sufficient planning, guidance, and workforce  training are in place prior to SMS implementation.  Without providing the  necessary training and guidance to its inspector workforce, FAA may not  be adequately prepared to ensure the benefits of SMS as industry sectors  are required to implement it.  Although FAA continues to encourage  voluntary implementation of SMS, it also must be prepared to exercise its  oversight functions, once final rules requiring SMS implementation are in  place, to ensure that industry is developing and utilizing processes to  identify, document, and mitigate safety risks.  This includes providing  clarification and guidance to air carriers, airports and others as they  develop their own SMS’s once final rules for each industry segment are in  place.
section_title: Recommendation for Executive Action
paragraphs: To maximize the effectiveness and potential benefits of SMS  implementation, we recommend that the Secretary of Transportation  direct the FAA Administrator to take the following action:
Develop a plan to provide oversight of industry implementation of  SMS, a plan that includes providing guidance and training to the  relevant FAA inspectors by the time final SMS rules for industry  sectors (commercial air carriers, certificated airports, repair stations,  design and manufacturing firms) are published.
section_title: Agency Comments
paragraphs: We provided DOT with a draft of this report for review and comment.   DOT provided technical corrections and clarifications, which we  incorporated as appropriate.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Transportation, the FAA Administrator, and  other interested parties. In addition, the report is available at no charge on  the GAO website at http://www.gao.gov.
If you or any of your staff members have any questions about this report,  please contact me on (202) 512-2834 or at dillinghamg@gao.gov.   Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report.  Key contributors to  this report are listed in appendix II.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objective was to review the implementation of safety management  systems (SMS) in the U.S. aviation industry and provide an update on  SMS implementation within the Federal Aviation Administration (FAA). To  do so, we addressed the following questions:  1.  What is the status of SMS implementation within FAA and for key  segments of the aviation industry, including certificated airports,  commercial air carriers, repair stations, and design and manufacturing  firms?   2.  What are the key challenges FAA and the aviation industry face in  implementing SMS?   3.  What additional actions do aviation stakeholders believe FAA could  take to improve SMS implementation and potential effectiveness?
To assess the status of SMS implementation within FAA and for key  segments of the aviation industry, we reviewed FAA orders and advisory  circulars on SMS and reports on its SMS pilot projects for commercial air  carriers, certificated airports, repair stations, and design and  manufacturing firms and from the SMS rulemaking projects for certificated  airports and commercial air carriers. We also reviewed FAA information  on industry SMS implementation status. In addition, we interviewed FAA  officials and industry trade group representatives. To update the status of  FAA’s implementation of SMS, we reviewed FAA guidance,  implementation plans, and other documents published or revised since  our 2012 report and interviewed FAA officials and trade group  representatives.
To determine the key challenges FAA and the aviation industry face in  implementing SMS as well as additional actions FAA and other  stakeholders may take to improve the implementation and potential  effectiveness of SMS, we interviewed officials from FAA, FAA employee,  and industry groups. To obtain an international perspective on the  challenges and additional actions, we interviewed representatives from  foreign aviation authorities, specifically the European Aviation Safety  Agency, Transport Canada, and the Civil Aviation Safety Authority of  Australia.  In addition, we also interviewed two Canadian airlines and one  Canadian airport that have implemented SMS. Table 3 lists industry and  international interviewees.
To obtain the industry stakeholder perspective for each of our objectives,  we collected and analyzed information through structured interviews of a  total of 20 industry stakeholders—5 airlines, 5 airports, 5 repair stations,  and 5 design and manufacturing firms—that participated in FAA’s SMS  pilot projects. We selected the certificated airports based on their hub  size, geographic location, and whether we interviewed them for our prior  study. We did not include certificated airports that we interviewed from  that study. We chose the commercial air carriers based on carrier type  (mainline or regional; cargo or charter), the level of SMS implementation  they had reached by the time of our interview, and whether we  interviewed them for the prior study; two of the air carriers we selected  were interviewed for our 2012 study.  For the repair stations, we choose  five firms based on level of SMS implementation and the aircraft category  (e.g., transport, commuter, or acrobatic) to which they provide services.  We chose this selection factor as a result of research completed by the  Center for Aviation Research, which found that SMS compliance for  repair stations could be based on aircraft category. For design and  manufacturing firms, we chose five firms based on the extent of SMS  implementation, recommendations from industry stakeholders, and types  of products they design or manufacture, or both.  We conducted all  stakeholder interviews with a standardized data collection instrument to  maintain consistency across the interviews. To ensure that our interview  questions were clear and reliable, we conducted two pretests with  knowledgeable individuals and refined the interview questions based on  those results. Because these 20 stakeholders comprise a non- representative sample, the results from these interviews cannot be  projected to the universe of these industry segments. Table 4 lists these  selected industry stakeholders.
We conducted this performance audit from May 2013 to June 2014 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions, based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Heather MacLeod (Assistant  Director), Amy Abramowitz, James Geibel, David Hooper, Christopher  Jones, Delwen Jones, Brooke Leary, Josh Ormond, Pamela Vines, and  Elizabeth Wood made key contributions to this report.
section_title: Related GAO Products
paragraphs: Aviation: Status of DOT’s Actions to Address the Future of Aviation  Advisory Committee’s Recommendations. GAO-13-657. Washington,  D.C.: July 25, 2013.
Aviation Safety: Status of Recommendations to Improve FAA’s  Certification and Approval Processes. GAO-14-142T. Washington, D.C.:  October 30, 2013.
Department of Transportation: Key Issues and Management Challenges,  2013. GAO-13-402T. Washington, D.C.: March 14, 2013.
General Aviation Safety: Additional FAA Efforts Could Help Identify and  Mitigate Safety Risk. GAO-13-36. Washington, D.C.: October 4, 2012.
Aviation Safety: Additional FAA Efforts Could Enhance Safety Risk  Management. GAO-12-898. Washington, D.C.: September 12, 2012.
Aviation Safety: FAA Is Taking Steps to Improve Data, but Challenges for  Managing Safety Risks Remain. GAO-12-660T. Washington, D.C.: April  25, 2012.
Aviation Safety: Enhanced Oversight and Improved Availability of Risk- Based Data Could Further Improve Safety. GAO-12-24. Washington,  D.C.: October 5, 2011.
Aviation Safety: Certification and Approval Processes Are Generally  Viewed as Working Well, but Better Evaluative Information Needed to  Improve Efficiency. GAO-11-14. Washington, D.C.: October 7, 2010.
Aviation Safety: Improved Data Quality and Analysis Capabilities Are  Needed as FAA Plans a Risk-Based Approach to Safety Oversight.  GAO-10-414. Washington, D.C.: May 6, 2010.