Aviation Safety: Additional FAA Efforts Could Enhance Safety Risk Management
Published: 2012-09-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The nation's aviation system is one of the safest in the world, but with air travel projected to increase over the next 20 years, efforts to ensure the continued safety of aviation are increasingly important. The FAA is seeking to further enhance safety by shifting to a data-driven, risk-based safety oversight approach--referred to as SMS. SMS implementation is required for FAA and several of its business lines and the agency is taking steps to require industry implementation.
As requested, this report addresses (1) the status of FAA's implementation of SMS, (2) the extent to which FAA's SMS efforts have been consistent with key practices for successful planning and implementation of a new program, and (3) challenges FAA faces in implementing SMS. To address these issues, GAO reviewed FAA SMS documents, compared FAA efforts to key practices, and interviewed agency and industry officials.
section_title: What GAO Found
paragraphs: The Federal Aviation Administration (FAA) and its business lines and offices are in different stages of their implementation of Safety Management Systems (SMS). FAA finalized its agency-wide implementation plan in April 2012, and the Air Traffic Organization (ATO) has completed its SMS implementation, but other FAA SMS efforts are in the early stages. FAA business lines, such as the Aviation Safety Organization (AVS) and the Office of Airports (ARP), have SMS guidance and plans largely in place and have begun to integrate related practices into their operations, but many implementation tasks remain incomplete, and officials and experts project that full SMS implementation could take many years.
There are a number of key practices that can help agencies plan for and efficiently implement new projects, including large scale transformations such as FAA's SMS implementation, and FAA has many in place. For example, FAA has support from top leadership and a clear project mission. However, FAA has only partially addressed other key practices such as developing a project plan to track SMS implementation, and FAA has not addressed performance-related practices such as establishing SMS performance measures or links between employees' performance standards and SMS.
Several challenges remain that may affect FAA's ability to effectively implement SMS. FAA is taking steps to address some challenges and stakeholder concerns, but challenges related to data sharing and data quality; capacity to conduct SMS-based analyses and oversight; and standardization of policies and procedures could negatively affect FAA's efforts to implement SMS in a timely and efficient manner. Further, FAA officials stated that SMS implementation will require some skills that agency employees do not have, but FAA has not yet assessed the skills of its workforce to identify specific gaps in employee expertise. In addition, while existing federal law protects any data collected for SMS, any data airports collect could be subject to state-specific Freedom of Information Act laws, a gap that could create a disincentive for airports to fully participate in SMS implementation.
section_title: What GAO Recommends
paragraphs: GAO recommends that FAA develop systems to: track SMS implementation, evaluate employee performance as it relates to SMS, and assess whether SMS meets its goals and objectives; conduct a workforce analysis for SMS; and consider strategies to address airports' data concerns. The Department of Transportation agreed to consider the recommendations and provided clarifying information about SMS, which GAO incorporated.

section_title: Letter
paragraphs: The U.S. airspace system is one of the safest in the world, with no main  line air carrier passenger fatalities in the U.S. in more than 10 years. This  record reflects the efforts of the Federal Aviation Administration (FAA),  airlines, airports, manufacturers, the National Transportation Safety Board  (NTSB), and others to continually improve aviation safety. However, with  air travel projected to increase over the next 20 years, efforts to ensure  the continued safety of aviation are increasingly important. FAA is  attempting to further enhance aviation safety, in part, by shifting to a data- driven, risk-based safety oversight approach. This approach is becoming  the standard throughout the global aviation industry and is recognized by  aviation leaders such as the International Civil Aviation Organization  (ICAO), the Joint Planning and Development Office (JPDO), and others  as the next step in the evolution of safety.
FAA is overseeing implementation of this new approach—called a safety  management system (SMS) approach—both within FAA and throughout  the U.S. aviation industry and is coordinating these efforts with the  international aviation community. Safety management systems represent  a proactive approach to safety and are intended to continually monitor all  aspects of aviation operations and collect appropriate data to identify  emerging safety problems before they result in death, injury, or significant  property damage. Under SMS, FAA will use the aviation safety data it  collects to identify conditions that could lead to aviation accidents or  incidents and to address such conditions through changes in the FAA’s  organization, processes, management, and culture. SMS adoption and  implementation is one of the biggest cultural and procedural  transformations in FAA history and will likely involve years of continuous  effort on the part of agency and industry officials.
You asked us to assess FAA’s implementation of SMS. To do so, we  addressed the following questions:   (1)  What is the status of FAA’s implementation of SMS?   (2)  To what extent have FAA’s SMS efforts been consistent with key  practices for successful planning and implementation of a new  program?   (3)  What challenges does FAA face in implementing SMS?
To determine the status of FAA’s implementation of SMS, we reviewed  FAA’s SMS orders and pilot project guidance, implementation plans, and  Notices of Proposed Rulemaking for Part 121 air carriers and Part 139  airports. We interviewed FAA SMS program managers across FAA  business lines and offices. We also reviewed international and FAA  guidance and SMS and NTSB recommendations to FAA related to SMS.  To assess the extent to which FAA’s efforts have been consistent with  key practices, we reviewed our reports and other literature on successful  project planning and implementation, particularly for large-scale  transformative projects, and identified key practices applicable to FAA’s  SMS implementation. We then assessed FAA’s actions against the key  practices by examining such documents as FAA’s guidance and  implementation plans, and interviewing FAA officials. We determined  whether each key practice was addressed, partially addressed, or not  addressed by using criteria developed from prior GAO reports. For  example, we considered a practice partially addressed if FAA had  partially implemented, taken steps toward, or started but not completed  implementing it (see app. I for a discussion of our assessment). To  identify challenges FAA faces in implementing SMS, we reviewed our  prior work on long-standing FAA challenges, such as those related to  training and data, and interviewed aviation industry experts and FAA  officials. We also reviewed our prior work on performance measurement  and workforce analysis, and NTSB recommendations related to SMS. To  obtain industry views on challenges, we interviewed representatives from  airports and air carriers selected for size and geographic dispersion. We  also interviewed representatives of aviation industry associations and  reviewed written comments submitted by aviation stakeholders on two  FAA-issued Notices of Proposed Rulemaking that would require SMS for  some airports and air carriers. See appendix I for a more detailed  description of our scope and methodology.
We conducted this performance audit from September 2011 to  September 2012 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: SMS provides a top-down approach to managing safety risk, which FAA  expects will improve aviation safety. SMS is not an additional safety  program that is distinct from existing activities that accomplish an entity’s  safety mission, but rather, a process for safety management that  incorporates systematic procedures, practices, and policies. According to  FAA, the overarching goal of SMS is to improve safety by helping ensure  that the outcomes of any management or system activity incorporate  informed, risk-based decision making. We reported in 2010 that FAA  officials believe that successfully implementing SMS is critical to meeting  the challenges of a rapidly changing and expanding aviation system. To  achieve a higher level of safety in an already very safe system, FAA  requires a more forward-thinking approach, which SMS provides, by  addressing cultural and organizational problems that lead to safety  hazards, identifying system-wide trends in aviation safety, and managing  emerging hazards before they result in incidents or accidents.
SMS implementation should bring about a fundamental shift in aviation  safety oversight. For decades, the aviation industry and federal  regulators, including FAA, have used data reactively to identify the  causes of aviation accidents and incidents and take actions to prevent  their recurrence. While FAA plans to continue to use data to analyze past  safety events, it is also working to use data proactively to search for risks.  FAA’s shift to the proactive approach of SMS is important because, as  accidents have become increasingly rare, less information is available for  reactive analyses of their causes. As a result, information that can be  used to help identify accident and incident precursors has become more  critical for accident prevention. Thus, the open sharing of safety  information among aviation stakeholders and how FAA’s policies and  procedures govern the reporting of safety information are essential to the  success of SMS.
SMS consists of four key components: (1) safety policy, (2) safety risk  management, (3) safety assurance, and (4) safety promotion (see fig.1).  Together, these four components are intended to provide a systematic  approach to achieving acceptable levels of risk. FAA provides to its  personnel detailed guidance on the principles underpinning these  components and the application of these components to aviation  oversight in its official orders and other internal FAA guidance. To the  industry, FAA provides this SMS guidance via advisory circulars and a  dedicated page for the SMS program office on the FAA website.
FAA is undertaking the transition to SMS in coordination with the  international aviation community, working with ICAO to adopt applicable  global standards for safety management. ICAO requires SMS for the  management of safety risk in air operations, maintenance organizations,  air traffic services, and airports as well as certain flight-training operations  and for organizations that design or manufacture aircraft. Further, ICAO  has published safety management requirements for its member countries  that mandate that civil aviation authorities—such as FAA—establish  SMS. ICAO first mandated SMS worldwide for air traffic service  providers, such as air carriers and certified aerodromes, in 2001. ICAO  later specified that member states should mandate SMS implementation  for  airports, air carriers, and others by 2009. FAA began SMS  implementation in 2005, but FAA officials informed ICAO that the agency  and industry would not be able to meet the 2009 deadline. ICAO is  allowing FAA to take additional time in its efforts to implement SMS, with  the understanding that implementation is under way and that FAA is in  the midst of a rulemaking to require SMS for commercial air carriers.  ICAO officials stated that the United States is one of the leading  implementers of SMS worldwide and acknowledged that SMS  implementation in the U.S. aviation system may be more complicated  than in other countries because of the size and complexity of the U.S.  aviation industry. ICAO has not specified a date by which FAA is  expected to comply with the requirements to implement SMS in the  aviation system. There have also been actions within the United States to  encourage implementation of SMS. For instance, in 2007, NTSB  recommended that FAA require all commercial air carriers to establish an  SMS and, in 2011, added SMS for all modes of transportation to the  NTSB’s Most Wanted List, identifying SMS as one of the most critical  changes needed to reduce the number of accidents and save lives.
Partially in response to the ICAO requirement, FAA added goals related  to SMS implementation to its 2009-2013 Flight Plan. a requirement to implement SMS in three of FAA’s business lines—the  Air Traffic Organization (ATO), the Aviation Safety Organization (AVS),  and the Office of Airports (ARP)—and a goal to implement SMS policy in  all appropriate FAA organizations, which include the Office of Commercial  Space Transportation (AST) and the Office of NextGen (ANG). FAA is in  the process of implementing SMS within these business lines and offices  as well as in industry through rulemakings to require airports and  commercial air carriers to implement SMS. FAA designated AVS as the  lead for SMS implementation in September 2008. Within AVS, the Office  of Accident Investigation and Prevention’s (AVP) Safety Management and  Research Planning Division coordinates and manages SMS  implementation and operation across the agency, and so AVP serves as  the official SMS lead for the agency.
Federal Aviation Administration, 2009-2013 Flight Plan is the agency’s strategic plan.
Administrators, their deputies, and other high-level FAA officials from  each business line or office (see fig. 2). Within some of the business  lines, there are offices devoted to specific aviation oversight functions that  are responsible for overseeing detailed implementation of SMS for those  functions. For example, the Flight Standards Service (AFS), a division of  AVS that provides safety oversight of commercial air carriers and others,  is taking steps to require SMS implementation by commercial air carriers  and is also working to integrate SMS into its internal activities. In addition,  the Aircraft Certification Service (AIR), a division of AVS that provides  safety oversight to aviation design and manufacturing firms, is leading  agency efforts to encourage SMS implementation for that industry sector,  while ARP is leading agency efforts to require SMS implementation for  certificated airports.
SMS implementation will require changes to many of FAA’s operations.  As the agency and industry implement SMS, shifts will be necessary in  both the skills of FAA and industry staff and the tools that the agency  uses to monitor safety. FAA’s integration of SMS into its business  practices will also affect how the agency provides air navigation services  and oversees the aviation industry. Historically, FAA oversight of airlines,  airports, and other regulated entities has involved oversight of such things  as operations and maintenance. FAA will continue this oversight, but will  also apply SMS principles to its processes for oversight. The agency will  provide oversight of the safety management systems of service providers  such as air carriers and airports to help ensure that they are managing  safety within their operations through SMS. For example, AFS currently  provides oversight of the operations, maintenance, and safety data of  commercial air carriers and others. Once SMS is fully implemented, AFS  will continue to provide this oversight and will also conduct oversight of  the safety management systems that commercial air carriers and others  put in place.
section_title: FAA and Its Business Lines Are at Different Stages of SMS Implementation
paragraphs: ATO completed its implementation of SMS, but FAA and several of its  other business lines and offices are in the early stages of implementation.  Most FAA business lines and offices have guidance and plans for SMS  implementation in place and have begun to integrate SMS-related  practices into their operations, but many tasks remain and aviation  officials and experts with whom we spoke project that full SMS  implementation will take many years.
subsections: section_title: FAA Recently Finalized Its Agency-Wide Implementation Plan, but Full SMS Implementation Is Likely to Take Many Years
paragraphs: FAA finalized its agency-wide plan for SMS implementation in April 2012.  The plan provides a road map for SMS implementation across the agency  and describes the activities that FAA business lines and offices will need  to complete by the end of 2015 to integrate SMS into their operations.  These activities will lead to outcomes including:  revising and standardizing safety policies and safety risk  management methodologies across FAA to ensure SMS  principles are consistently addressed;   improving organizational processes so that FAA business lines and  offices can share safety data and information more easily; and coordinating communications to ensure a common understanding  of SMS across the agency.
FAA began its agency-wide SMS implementation efforts in 2008, and in  September of that year issued a policy for implementation of a common  SMS within FAA. Among other things, the policy sets forth management  principles to guide all of FAA in safety management and safety oversight  activities and requires AVS, ARP, and ATO to develop and execute  business line-specific plans for SMS implementation. In late 2008, FAA  formed the agency-wide FAA SMS Committee to coordinate  implementation efforts across FAA business lines and offices. Overall, the  agency has taken a bottom-up approach to implementation, with some  individual business lines and offices beginning implementation prior to  agency-wide efforts. FAA has also taken steps to ensure that its plans for  SMS implementation and policies align with international and  government-wide requirements and technical guidance on SMS  implementation, including ICAO’s Standards and Recommended  Practices, the ICAO Safety Management Manual, and the JPDO SMS  Standards. For instance, officials stated that they consulted international  and government-wide guidance on SMS implementation when drafting  agency implementation plans. (See fig. 3 for more information on  alignment of FAA requirements with international and government-wide  requirements and guidance on SMS.)
Although FAA has made progress, completion of SMS implementation  across FAA is likely to take many years. FAA’s agency-wide SMS  implementation plan includes tasks with estimated completion dates  through 2015, and some implementation tasks may take even longer to  complete. For instance, a project plan that AVS officials developed to  track status of AVS SMS implementation tasks contained in its  implementation plan includes task completion dates through 2016.  According to FAA, the overall SMS implementation effort is an  evolutionary process that will not have a specific completion date. The  current implementation time frame is consistent with experts’ estimates of  how long it may take to implement SMS and with other large-scale  organizational transformations. For example, representatives from The  MITRE Corporation, which manages a federally funded research center  for FAA and assisted FAA in selected SMS implementation efforts, stated  that organizational transformations like SMS can take from 6 to 10 years.
section_title: ATO Completed SMS Implementation in 2010
paragraphs: ATO is the only entity among FAA and its business lines to have  completed SMS implementation. ATO issued its internal SMS guidance in  March 2007 and finalized both its SMS implementation plan and its  updated SMS Manual in 2008. According to ATO officials, ATO  completed SMS implementation in March 2010, and the FAA Air Traffic  Safety Oversight Service validated that ATO’s implementation of SMS  was complete. Officials stated that implementation within ATO was  simpler, in part, because it is the only branch of FAA that is considered an  aviation service provider and therefore did not have to conduct a  rulemaking for external entities as part of its SMS implementation.the implementation phase complete, ATO is currently in the continuous  improvement phase of SMS. This means that ATO will continuously use  the SMS-based processes now in place to identify hazards, enact  strategies to mitigate the risks associated with those hazards, and assess  the extent to which the mitigations are working effectively. In addition,  FAA officials stated that ATO is working to improve its SMS operations,  will update guidance on SMS, and plans to perform audits of its SMS  functions on a regular basis. ATO officials added that they are working to   With  share lessons learned from their implementation efforts with other FAA  business lines and to develop SMS tools and processes that can be  commonly implemented across all FAA business lines.
section_title: Most FAA Business Lines and Offices Are in Early Stages of Implementation
paragraphs: With the exception of ATO, most FAA business lines and offices are in  the early stages of implementation, either in terms of integrating SMS into  their internal processes or in terms of their efforts to prepare to provide  oversight for proposed requirements for industry implementation of SMS.  To date, much of the work of the FAA business lines has focused on  efforts to draft implementation policies and guidance, train employees,  and create tools for applying safety analyses and risk-based decision- making to safety oversight. (See fig. 4 for more information on the status  of key SMS implementation efforts across FAA.)
AVS began its SMS implementation efforts in August 2006 and finalized  its SMS implementation plan in January 2012, which was then  incorporated into FAA’s overall plan for SMS implementation. Since 2006,  AVS and its seven services and offices have issued orders and other  guidance on SMS implementation; developed SMS training courses;  conducted voluntary pilot projects and rulemaking efforts on SMS  implementation for industry; and worked to begin integrating elements of  SMS into their operations. For example, AIR officials, who provide  oversight of aviation design and manufacturing firms, have developed a  central database that provides standard criteria for analyzing service data  in a risk-based manner. This should allow AIR inspectors and engineers  to rate the risk of potential safety issues and prioritize oversight to high  risk issues.
Some services and offices within AVS are in the midst of efforts to require  SMS for industry and are also operating voluntary pilot programs to  promote SMS implementation within industry. A final rule to require SMS  for commercial air carriers is expected to be issued in September 2012.  In 2007, AFS launched a pilot program to encourage voluntary  implementation of SMS by industry.
According to FAA officials, as part of its rulemaking efforts for commercial  air carriers, FAA and AVS are developing a new part in the Code of  Federal Regulations (CFR)—Part 5—that will describe SMS  implementation requirements for Part 121 certificate holders. In the future,  FAA may conduct rulemakings to require additional sectors of the aviation  industry to meet Part 5 requirements (see fig. 5). AVS officials stated that  efforts to establish SMS requirements more broadly across the aviation  industry will likely take many years.
Though FAA has not yet required SMS for air carriers or other parts of  industry, FAA has acted to encourage SMS implementation by industry  through voluntary pilot projects, and some aviation stakeholders have  chosen to implement SMS in advance of any federal requirement. Some  sectors of the aviation industry are farther along in their implementation of  SMS than others. For instance, FAA officials stated that a large majority  of commercial air carriers are in the process of implementing SMS. As  of June 2012, over 90 percent of commercial air carriers operating under  Part 121 were participating in the AFS pilot program, which provides air  carriers with direct implementation support from FAA officials under a  more relaxed implementation time frame than is anticipated under an  eventual implementation regulation. Of these air carriers, three have  reached the final stage of SMS implementation. However, most small air  carriers have not yet begun implementing SMS. In contrast to AFS, AIR is  at an earlier stage in its efforts to require SMS for the approximately  3,000 design and manufacturing firms it oversees. AIR began a voluntary  pilot project for SMS implementation by design and manufacturing firms in  2011 and has 11 pilot project participants. AIR officials stated that they  are in the process of launching a second aviation rulemaking committee  to continue to explore options to require SMS for design and  manufacturing firms. Officials also noted that AFS and AIR are working  together to share lessons learned and assist one another in their  implementation efforts.
ARP is in the early stages of working to integrate SMS principles into its  oversight of airports, and recently took steps to reduce the scope of that  oversight. ARP initially planned to apply SMS-based oversight to all  certificated airports. Officials stated that ARP is currently limiting its SMS- based oversight to large hub airports because of budget constraints and  will reassess its capacity to expand oversight to smaller airports in 2013.  ARP began its SMS implementation in 2010 and issued an internal order  to provide a basis for the integration of SMS into its operations later that  year. The office finalized its SMS implementation plan in September 2011  and has begun to make changes to its oversight. For instance, In June  2011, ARP began to apply SMS-based oversight to construction projects  at the 29 large hub airports in the United States. Under this new  oversight framework, ARP staff assess proposed airport construction  projects using risk-based SMS principles, and airports need to  incorporate strategies to mitigate identified risks into their construction  plans prior to receiving ARP’s approval for the project. Like AVS, ARP is  also in the midst of a rulemaking to require SMS for all certificated  airports and has completed three voluntary SMS pilot projects for airports  from 2008 to 2011. Thirty-one airports participated in at least one of  ARP’s SMS pilot projects. ARP is using information gathered through the  pilot projects to inform a planned advisory circular that will provide  additional guidance to airports on SMS implementation. The pilot projects  also allowed airports to share their SMS implementation practices with  other airports. The final rule to require SMS for Part 139 certificated  airports is expected to be issued in April 2013 and, if implemented as  proposed, would require over 500 airports to implement SMS.
Other FAA business lines are in varying stages of implementation. AST is  not currently required to implement SMS; however, AST is taking initial  steps toward integrating SMS into an existing set of safety management  processes. ANG is farther along in its implementation of SMS because of  its previous status as a part of ATO. basing its implementation of SMS on policies and processes established  during ATO’s implementation of SMS. The officials stated that since ANG  will provide the systems and components that will be used by ATO to  manage air traffic, it made sense for ANG to develop its SMS based on  policies, processes, and systems established by ATO. Officials stated that  ANG completed its implementation plan in June 2012 and estimated that  ANG’s SMS implementation is about 70 percent complete.
section_title: FAA’s SMS Approach Is Consistent with Many but Not All Key Practices for Successful Project Implementation
paragraphs: There are a number of key practices and implementation steps that can  help agencies successfully plan for and implement new projects, including  large scale transformative ones, such as FAA’s implementation of SMS.  As we have previously reported, addressing these key practices can help  an agency improve its efficiency, effectiveness, and accountability. FAA  currently has many of these key factors in place, such as established  support from top leadership and a clear project mission; however, it has  only partially addressed other key practices, such as providing needed  expertise and technology, and has yet to establish SMS performance  measures (see fig. 6).
In 2011, FAA reorganized some of its offices and, as part of the reorganization,  separated NextGen efforts from ATO.
subsections: section_title: Many Key Practices Are in Place
paragraphs: FAA has instituted many key practices that will help it prepare for and  implement SMS across its business lines and offices.
Top leadership: Top leaders from each FAA business line provide  support for and actively participate in SMS implementation. As  previously mentioned, FAA established the SMS Executive Council, a  group of high-ranking FAA officials that provides executive-level  guidance and conflict resolution for SMS-related issues across the  agency. In accordance with our key practices, the SMS Executive  Council has the authority to make resource allocation decisions, but  also confers decision-making authority where appropriate to the FAA  SMS Committee.
For instance, FAA officials told us that the SMS Executive Council  retains the authority to make final decisions about changes to FAA’s  implementation plan that affect policies or procedures for multiple  business lines; the FAA SMS Committee has the authority to make  decisions that relate to daily concerns that fall within the purview of its  members. For example, committee members settled a disagreement  between ATO and airport officials over whether an airport should  conduct certain components of a safety risk management panel. At  the time, FAA had not yet issued its safety risk management policy  clarifying terms and requirements, so the airport and ATO each had  its own distinct safety risk management definitions and processes.  Working with ARP and ATO officials, committee members identified a  compromise in which ATO protocols were followed, but any  disagreements on terms or procedures were documented. ARP  officials told us that FAA’s safety risk management policy, issued in  April 2012, should help prevent this type of disagreement from  occurring.
Clear project mission: FAA’s internal order requiring SMS  implementation for ARP, ATO, and AVS clearly describes that FAA’s  mission is to improve aviation safety and that implementing SMS and  its components supports that mission. Each business line also has its  own internal order requiring SMS implementation that mirrors this  mission and goals.
Implementation team: AVP’s safety management division and the  FAA SMS Committee, function jointly as FAA’s dedicated SMS  implementation team. The team’s structure and actions align with our  criteria for a strong and stable team because it is composed of  senior-level program managers from each business line, all of whom  had received SMS training according to FAA officials. Despite some  recent departures, its membership has been largely stable.
Leading practices: FAA shares information across business lines to  identify lessons learned related to SMS implementation. For example,  ATO assembled lessons learned from its SMS implementation into a  presentation for the other business lines, and included tips such as  encouraging others to implement a training program and monitor  mitigations. According to FAA’s implementation plan, the agency  plans to systematize the sharing of lessons learned by creating a  central repository to collect and communicate safety lessons learned  among its business lines and offices by September 30, 2013.
Troubleshooting: FAA has processes in place to manage SMS  implementation across FAA, including troubleshooting unexpected  problems. For example, the FAA SMS Committee meets monthly and  manages agency-wide SMS implementation and any challenges that  arise, and regularly briefs the SMS Executive Council, a briefing that  includes a discussion of any issues or unexpected problems that  could not be resolved at the committee level. For instance, when the  Air Traffic Manager at an airport disagreed with airport officials  regarding how to handle a potential safety issue with planes that were  taking off on runways that were temporarily closed, the FAA SMS  Committee elevated the issue to the SMS Executive Council, which  resolved it. As we have previously reported, instituting practices like  these can help an agency become more results-oriented, customer- focused, and collaborative.
Although FAA is still in the process of finalizing new requirements for  airports and air carriers to implement SMS, it has already taken some  steps to institute key practices for those efforts. For example, FAA  officials stated that the agency has taken steps to identify leading  practices during pilot projects by soliciting information from participating  airports and air carriers, and FAA officials told us they plan to incorporate  these lessons learned into rulemaking and guidance. ARP officials  reported that they encouraged pilot project participants to share lessons  learned directly with one another through studies and roundtable  discussions, and incorporated some of the lessons learned into FAA  advisory circulars. FAA has also made efforts to troubleshoot and  manage unexpected problems with pilot participants through meetings,  calls, and conferences with airport and air carrier officials to understand  their experiences. For example, AFS officials reported that they helped  officials from air carriers to understand when certain safety risk  management documentation and processes are necessary, and how they  could be adapted for a variety of changes made to carrier operations,  including smaller day-to-day changes. However, despite this assistance,  officials from some airports that participated in pilot projects reported that  they could have benefited from additional assistance from ARP, such as  clarification on the safety risk management component of SMS. In  addition, an official at one airport told us that he would have liked FAA to  facilitate conversations between airports of similar size to help them share  lessons learned.
section_title: FAA Efforts to Create a Project Plan and Consult and Assist Stakeholders Partially Align with Key Practices
paragraphs: Other steps FAA has taken in its SMS implementation efforts partially  align with key practices for implementing a new program.
Project plan: Currently, the agency-wide project plan for SMS  implementation is a single page of high-level milestones, which AVP  officials monitor and report on to the SMS Executive Council. Also,  AVS has a detailed project plan for its own SMS implementation and  elements of agency-wide implementation for which AVP, as the  agency SMS lead, has responsibility. Officials stated that they have  plans to develop a system to monitor and track the progress of  activities needed to implement SMS, but FAA does not currently have  a system for tracking agency-wide SMS implementation, a key  practice particularly important during the initial planning phase of  project implementation. However, given the scope and complexity of  SMS, a detailed, agency-wide project plan could help FAA track and  monitor the interim steps of SMS implementation across the agency.
Without such a plan, it may be more difficult for FAA to identify  problems or deviations from planned activities, putting both the  timeliness and effectiveness of SMS implementation at risk.
Consulting with stakeholders: FAA has made efforts to consult with  employees and stakeholders regarding its SMS implementation, but it  has not yet developed a communications plan. Agencies should  involve employees in planning, and incorporate employee feedback  into new policies and procedures. FAA involved its business line  program managers and some of the managers’ staff by assigning  them responsibility for the day-to-day tasks related to implementing  SMS across the agency. FAA has involved other employees by  soliciting questions and comments on SMS in town hall meetings and  the online DOT site called “IdeaHub,” and by offering SMS training  through each business line. ATO, ARP, and AVS all offer introductory  SMS courses for their staff as well as additional related courses, such  as an SMS course specifically for managers and ATO’s safety risk   management course.
FAA has been working to implement SMS for the last 4 years, but the  agency does not have a communications plan or strategy for ensuring  that the SMS messages communicated to staff are consistent across the  agency. Instead, FAA relies on a more informal communications structure  in which each program manager staffed to the implementation team  communicates relevant information back to their respective business line.  The implementation team does not communicate any information directly  to employees, which could hinder the team’s ability to ensure consistency  in its message across FAA. ATO officials reported experiencing this  challenge at the beginning of ATO’s SMS implementation, when a lack of  clear requirements for communicating SMS information resulted in  variation in staff’s understanding of guidance. We have previously  reported that a communication plan or strategy can ensure consistency of  message, provide information to meet the specific needs of employees,  encourage two-way communication, and build trust. FAA plans to begin  working on a communications plan in September 2012, and is scheduled  to issue the plan at the end of February 2013. FAA officials also said they  are in the process of developing an internal SMS website for employees  to share information and ideas, which could enhance SMS  communications. However, until the communications plan is developed  and implemented, FAA’s employees may not receive timely or consistent  information on SMS or be as invested in its implementation as they might  otherwise be.
FAA’s approach to overseeing industry SMS implementation allowed for  additional two-way communication. For example, FAA solicited views on  SMS implementation from airport and air carrier officials through voluntary  pilot projects described previously, and learned more about industry  perspectives through the formal rulemaking process—whereby an agency  issues a Notice of Proposed Rulemaking and is required to notify the  public and give them an opportunity to submit comments.
Providing technology and expertise: FAA has provided some SMS  training and tools to its employees; however, it has not yet provided  other tools important for SMS implementation. FAA officials reported  that each business line has provided SMS training to staff. In addition,  FAA recently developed a standardized Safety Risk Management  (SRM) policy, which will assist employees across FAA by  standardizing SRM terminology and clarifying confusion on the  conduct of SRM across the agency. FAA plans to create a simple  version of an agency-wide hazard-tracking system in the next 3 to 6  months, but does not have plans to create a more complex system  until August 2015, according to FAA’s SMS implementation plan. The  simple version will draw from hazard-tracking systems already in  place in some business lines, and summarize information from them  to highlight broader hazards such as those that would affect multiple  business lines. For instance, FAA officials stated that if ATO wanted  to make a change to its operations at a particular airport, then ATO  would be responsible for identifying associated hazards, risks, and  risk mitigations and would also be responsible for assuming  responsibility for the risk. However, if ATO determined that the airport  was better equipped to mitigate the identified risks, then the airport  and ARP would become more involved in designing risk mitigations  and overseeing their implementation. FAA’s efforts to provide tools to  help in SMS implementation are affected by differences in how data  are collected and assessed across the agency. For example, these  differences have held back agency efforts to model how changes to  the national airspace system, such as increases to air travel, can  affect safety. We have previously reported on and made  recommendations related to FAA’s data challenges, and also  discuss them later in this report. These data challenges mean that  FAA is not always able to perform comparisons across databases, a  challenge that that limits the usefulness of the data in identifying  possibly dangerous hazards. Identifying, monitoring, and mitigating  hazards is a key tenet of SMS, and without the proper technologies  and tools, FAA may not be able to do this as effectively.
section_title: FAA Has Yet to Integrate SMS into Employee Performance Plans and Establish Performance Measures
paragraphs: FAA’s efforts do not align with two key practices for implementing a new  program.
Integrating SMS into employee performance plans: FAA does not  consistently evaluate employees’ performance on SMS-related tasks.  We have previously reported that effective performance management  systems create a clear linkage between individual performance and  organizational success, and include aligning individual performance  expectations with organizational goals. FAA’s organizational mission  and goal, and that of SMS, is to improve safety, yet FAA officials told  us that the agency does not require employee performance plans to  include SMS-related tasks. Although officials reported that some  employees’ performance plans explicitly include SMS items, such as  providing SMS training or developing SMS policy, it is left to the  discretion of each business line whether SMS items are included. FAA  officials told us that SMS principles and methodologies will be  included in the performance plans of employees involved in writing  SMS policy and revising SMS processes, and will be incorporated into  the tasks of others once SMS implementation reaches those  individuals. However, currently, none of the business lines require  this. As such, FAA does not have a system for assessing the extent to  which staff are effectively supporting SMS, and FAA may not be able  to determine if staff are completing tasks and responsibilities  necessary for the successful implementation of SMS.
Measuring performance: FAA does not have performance measures  in place to assess whether the SMS goals of improving safety are  being achieved. FAA has broader safety-related performance  measures, such as tracking rates of runway incursions and losses of  separation, but SMS-related performance measures could address  intermediate safety issues, such as precursors to incursions or  incidents. Such measures could help FAA track progress toward its  broader safety measures. FAA officials told us that AVS is a  member of the Safety Management International Collaboration Group,  a group formed in 2009 to address safety management-related topics,  including performance measures. Most recently, FAA formed an  agency-wide working group to study performance metrics for SMS  implementation, and FAA’s implementation plan states that such  metrics will be finalized in October 2014. However, FAA officials we  spoke with acknowledged that they are at the very beginning phase of  this process and, although already in the process of implementing  SMS, have not yet identified metrics to measure safety results under  an SMS system. We have previously reported that performance  information is critical for achieving results and maximizing the return  on federal funds. Performance measures should help FAA identify  the extent to which SMS implementation will contribute to increased  aviation safety—FAA’s stated overall goal for SMS—as well as help  identify what changes could be made to improve SMS performance  over time.
section_title: Addressing Key Implementation Practices and Other Challenges Could Enhance SMS Effectiveness
paragraphs: As previously mentioned, FAA has taken steps to address many of the  practices associated with planning and implementing a new program.  However, we identified six challenges that could negatively affect FAA’s  efforts to implement SMS in a timely and efficient manner:   1)  the large scope and complexity of SMS implementation,   2)  resource and capacity constraints,   3)  standardization of policies and processes,   4)  data sharing and protection,  5)  data quality and usefulness, and   6)  development of performance measures to evaluate SMS  effectiveness.
subsections: section_title: Large Scope and Complexity of SMS
paragraphs: Implementing SMS is one of several major initiatives FAA has under way,  and its sheer scope and complexity could affect, or be affected by,  concurrent FAA efforts such as NextGen or Unmanned Aircraft  Systems. SMS requires changes in many of FAA’s operations: from the  way the agency tracks hazards to the way it oversees industry. SMS will  also require a transformation of FAA’s and the aviation industry’s safety  culture to one in which information and safety data are shared openly,  and errors are addressed through whatever action is necessary to  prevent them from happening in the future. FAA is making efforts to move  toward this new approach to safety, for instance by using data-sharing  systems that are protected from public disclosure to encourage voluntary  reporting of safety issues and enable more robust analysis of safety data  among FAA and air carriers. Moreover, as previously stated, each of  FAA’s business lines has its own role in implementing SMS that must be  coordinated across the agency. This is particularly challenging because  the business lines are at different stages of implementation and,  according to FAA officials, have historically operated independently.
The scope and complexity of SMS implementation may also be a  challenge for the aviation industry, and some stakeholders expressed  concerns both in interviews and in official comments on FAA’s Notices of  Proposed Rulemaking that eventual FAA requirements to implement SMS  need to allow for variation in airport and air carrier operations. For  example, officials from some smaller airlines and airports noted that SMS  implementation could require additional resources, such as staff and  software, which may not be readily available. In addition, officials from  some airports and air carriers were concerned that FAA’s final  requirements would be too prescriptive to allow entities to implement an  SMS program that best fit their organizational type, management  practices, and resources. Most stakeholders and experts we interviewed  stated that FAA could design SMS requirements for airports and airlines  that are scalable and flexible to accommodate this variation, which would  address these concerns. For instance, airport officials from smaller  airports told us that staff size limits their ability to assign a dedicated SMS  employee or safety director, while some officials at larger airports said  they were able to hire a SMS safety director or already had an  established safety director in place. Also, FAA’s SMS implementation pilot  project for airports found that 35 percent of participants planned to hire  additional staff to support SMS and 15 percent were not sure. FAA  officials have noted that they understand these scalability concerns, and  are taking them into consideration as they develop final SMS rules for  industry.
section_title: Resources and Capacity
paragraphs: SMS implementation across FAA will require some skills that agency  employees currently do not have, yet FAA has not formally assessed the  skills of its workforce to identify any gaps in the expertise required to  implement SMS or determined how to fill those gaps. In addition, FAA  officials stated that existing staff may not be able to be trained to fill SMS  implementation needs in all cases. For instance, FAA officials noted that  SMS implementation will require some engineers and other technical  employees to understand certain terminologies and have certain  knowledge, skills, and abilities, such as an enhanced ability to perform  complex modeling and analysis of aviation safety data to identify potential  safety hazards. AVS officials stated that to implement SMS, additional  employees with skills in analyzing data for hazards and associated risks  would be needed, along with additional training for existing staff. ARP  officials stated that the office might need program analysts with specific  data analysis skills to implement SMS.not expect to receive significantly more resources and, as previously  mentioned, have already had to reduce the scope of the office’s SMS- based oversight because of insufficient staff. Stakeholders and experts  also questioned whether FAA currently has the resources and capacity  needed to fully implement SMS. For example, experts noted that FAA  may not have the requisite engineers and other staff to participate in  safety risk management efforts, or FAA inspectors to oversee individual  airport and air carrier SMS programs.
ARP officials stated that they do  Despite these concerns, FAA has not yet conducted a strategic workforce  assessment to accurately determine the skills and staffing levels it needs  to manage SMS. Although FAA’s SMS implementation plan recommends  that business lines create such staffing analyses, none have done so. Nor  has FAA conducted an agency-wide workforce assessment for SMS. Our  internal control standards state that agencies should ensure that skill  needs are continually assessed to ensure workforces have the skills  necessary to help the agency meet its goals. We have reported that  strategic workforce planning is an integral part of human capital  management and helps an agency, among other things, determine the  critical skills and competencies that will be needed to achieve current and  future programmatic results, and then develop strategies tailored to  address any gaps identified.
A workforce analysis could help FAA determine how to best address its  most critical needs in ways that account for budget limitations, such as  through retraining or shifting staff, rather than hiring additional employees.  Without conducting an agency-wide SMS workforce analysis, FAA cannot  be sure that it has sufficient staff, skills, or competencies to implement  SMS, thus putting its SMS implementation efforts at risk.
section_title: Standardization of SMS Policies and Processes
paragraphs: SMS standardization across FAA business lines and offices is central to  implementation success, yet developing common systems for distinct  FAA business lines and offices has proved challenging. For example,  FAA realizes that the agency needs a common hazard-tracking system in  order to maximize SMS effectiveness, yet FAA officials and stakeholders  stated that it is difficult to develop such a system because each of FAA’s  business lines uses different hazard-related terms and definitions, and  often different data systems. These differences, in turn, prevent the  agency from performing simple comparisons across databases and have  delayed advances in using data analysis to proactively identify potential  safety hazards. FAA officials stated that the agency has recently taken  steps to make its databases interoperable, and also recently issued a  standardized policy for the safety risk management component of SMS.  Both of these steps may enhance FAA’s hazard-tracking and analysis  capabilities. The agency is also working with ICAO to address issues  related to standardization, such as adopting a collaborative approach to  increase the sharing of safety information internationally.
Industry officials are also concerned that FAA inspectors and certificate  management offices may have different interpretations of SMS and  other regulations. We and others have previously reported that variation  in FAA’s interpretation of standards for certification and approval  decisions is a long-standing issue. Industry stakeholders we interviewed  expressed concerns that a similar result could occur once final rules are  issued requiring airports and air carriers to implement SMS, and could  lead to airports or air carriers of similar size being held to different  standards of SMS implementation. FAA officials acknowledged that this is  a challenge for the agency and noted that the agency plans to provide  additional training to inspectors related to oversight of SMS. Additionally,  based on our 2010 recommendation, recent legislation directs FAA to  establish an advisory body of government and industry representatives to  address the issue of inconsistent interpretation of regulations.
FAA’s organizational structure for SMS implementation may pose  challenges to standardization as well. For example, as previously  mentioned, AVP’s safety management division is the lead for SMS, and  AVP and the FAA SMS Committee share responsibility for implementing  SMS across the agency. Despite AVP’s role as lead for SMS  implementation, it does not have any additional authority compared to the  other business lines’ committee representatives, something that AVP  officials noted can make SMS implementation difficult. This could slow  decision-making, particularly around issues that require business lines to  come to a single decision, such as how to standardize policies.  Nevertheless, FAA officials acknowledged that having to collaborate to  implement an agency-wide SMS has improved communication among the  business lines. FAA will likely continue to face challenges standardizing  its policies and processes as standardization of this scale is not  something the agency has previously undertaken, and the need to  negotiate solutions across FAA business lines could take time.
section_title: Data Sharing and Protection
paragraphs: Airport officials’ concerns about sharing and protecting their safety data  may reduce SMS effectiveness by limiting the ability of airports and FAA  to analyze safety data and identify trends. Although FAA has some data  protections in place, such as those established by the FAA Modernization  and Reform Act of 2012, which protects data that airports and air carriers  submit to FAA for SMS from federal Freedom of Information Act (FOIA)  requests, any data airports collect and any data air carriers share with  airports could be subject to state-specific FOIA laws. Most certificated  U.S. airports are either owned by a state, a subdivision of a state, or a  local government body, and thus are subject to state laws, including state  FOIA laws. This means that data airports collect and submit to FAA for  SMS—such as information on hazards or other safety data—is protected  from federal FOIA public disclosure requests, but, according to officials  and experts, may be subject to public disclosure under state FOIA laws.  Air carriers are not directly subject to state FOIA laws because they are  privately owned. Nevertheless, officials and experts stated that these laws  could affect air carriers because any data they choose to share with  airports could then be subject to state FOIA laws. As a result, air carrier  officials told us they may be less likely to share safety information with  airports. Airport and airline officials’ primary concern is that the public  disclosure of such information could result in negative publicity or expose  them to legal liability in the event of an incident or accident. FAA officials  said that data protection and legal liability are two of the major concerns  throughout the aviation industry that could hinder the implementation of  SMS.
FAA officials told us that they intend to continue to promote and expand  safety information sharing efforts, but that airports could find ways to  structure their SMS implementation so that they realized safety benefits  while limiting the public release of air carrier safety information. In FAA’s  official response to comments on two Notices of Proposed Rulemaking,  FAA stated that airport officials are best situated to understand how to  comply with state laws. Nonetheless, we found consensus among NTSB  and many aviation stakeholders that FAA should seek congressional  action regarding the protection of airport data from state FOIA laws.
Data sharing can also be challenging within FAA. In 2011, we  recommended that FAA improve information sharing among its programs  because not doing so could limit the ability of FAA and others to analyze  safety data and understand safety trends. The Department of  Transportation agreed that it must continue to promote and expand safety  information sharing efforts and safety practices in order to maximize the  effectiveness of safety data mining to analyze trends and prioritize  safety efforts to address hazards before they lead to incidents or  accidents. However, our recommendation remains open. According to  officials, ICAO has also formed the Safety Information Exchange Study  Group to help enhance data protection and identify potential international  solutions.
section_title: Data Quality and Usefulness
paragraphs: Long-standing issues with data quality and usefulness could negatively  affect FAA’s understanding of aspects of the safety of the aviation  industry and, consequently, affect SMS’s effectiveness. Obtaining  relevant data and understanding how to analyze those data to identify  potential hazards are major challenges that FAA will need to overcome. In  recent GAO reports, we commented on FAA’s lack of data to effectively  assess aviation trends for certain types of events and the safety  performance of certain industry sectors. For instance, in April 2012, we  reported that for such events as runway excursions (when an aircraft  veers off or overruns a runway) and ramp accidents (incidents or injuries  that occur off the runway), a shortage of FAA data exists for analysis.  The Department of Transportation concurred with this and our  recommendations, and stated that the agency has taken steps to improve  to its data quality and usefulness. For example, the FAA SMS Committee  directed a working group to determine what safety data the agency is  going to collect and track and to recommend what kind of system will be  needed. However, FAA has not yet fully implemented several of our  recommendations aimed at improving its capability to use data for  aviation safety oversight, or several data-related NTSB recommendations  from recent years. For example, we recommended that FAA extend  standard quality controls, as appropriate, to the databases that support  aviation safety oversight to ensure that the data are as reliable and valid  as possible. By not fully addressing these challenges and  recommendations, FAA’s ability to comprehensively and accurately  assess and manage hazards and risk will be compromised, reducing the  ability of SMS to prevent incidents and accidents.
section_title: SMS Performance Measures
paragraphs: The aviation community has widely acknowledged that developing SMS  performance measures is difficult, but without them, FAA will not be able  to gauge the direct impact of SMS on aviation safety. Some stakeholders  told us about ways in which SMS improved their organization’s  operations, and these examples could provide insight into possible SMS  performance measures. For instance, some airports and air carriers that  participated in FAA’s SMS pilot projects reported that SMS  implementation improved communication across their organizations,  helped them identify organizational gaps—such as those in internal  auditing and training—and decreased employees’ injuries, aircraft  damages, and insurance costs. Officials from the Flight Safety  Foundation recommended that the extent to which SMS informs  management decision making, such as by redirecting resources or  shifting priorities, may be one way to measure SMS effectiveness. An  FAA official suggested that performance measures could be directed to  specific components of SMS, for instance tracking the number of risks  mitigated as a measure of safety risk management efficacy. We have  previously reported that agencies need to set quantifiable outcome-based  performance measures for significant agency activities, such as SMS, to  demonstrate how they intend to achieve their program goals and measure  the extent to which they have done so. Performance measures allow an  agency to track its progress in achieving intended results, which can be  particularly important in the implementation stage of a new program such  as SMS. In our prior work we recommended that agencies develop  methods to accurately evaluate and measure the progress of  implementation, and develop contingency plans if the agency does not  meet its milestones to complete tasks. FAA has established a working  group to study the issue and participates on two international  performance measures work groups: the Safety Management  International Collaboration Group and the aforementioned Safety  Information Exchange Study Group.
section_title: Conclusions
paragraphs: FAA is making progress implementing SMS, both within the agency and  for the aviation industry. However, SMS implementation represents a  significant cultural and procedural shift in how the agency will conduct  business internally and provide oversight to aviation stakeholders such as  air carriers and airports, and by all estimates, this transformation will take  many years to complete. Going forward, if FAA is to attain the full benefits  of SMS, it will be important for the agency to remain committed to fully  implementing SMS across its business lines. FAA has taken a number of  steps that align with practices we identified as important to successful  project planning and implementation, but has not addressed or has only  partially addressed other key practices. These practices are important for  large-scale transformative projects such as SMS, which require a  dramatic shift in FAA’s approach to safety oversight and management. In  the absence of these key practices, it may be difficult for FAA to prioritize  projects or monitor SMS implementation and progress toward improving  safety.
Aviation safety is a shared responsibility among FAA, air carriers,  airports, and others in the aviation industry, and efforts to improve safety  will require the agency to overcome several challenges. The magnitude of  SMS’s potential impact on aviation oversight and the complexity of  implementation are both a benefit and a drawback for FAA, as SMS  implementation could help ensure the continued safety of the U.S.  aviation system, but could also affect implementation time frames for  other large initiatives as the agency works in a resource-limited  environment. FAA officials believe that SMS implementation will require  some skills that employees do not currently have; however, FAA has not  conducted an agency-wide workforce assessment. With agency  resources and capacity in great demand, it will be important for the  agency to maximize the efficiency of SMS implementation, both through  efficient use of its workforce and creation of policies and systems that  standardize and streamline implementation. In addition, data protection  concerns from airport officials and others could prevent aviation  stakeholders from fully embracing SMS implementation, thus hindering its  effectiveness. Without assurance of protection from state FOIA laws,  some aviation stakeholder may choose to collect only the bare minimum  of safety-related data or may choose to limit the extent to which collected  information is shared among aviation stakeholders. The agency also  lacks sufficient data to effectively assess aviation trends for some events  as well as the safety performance of certain industry sectors. The ability  of FAA to identify safety risks, develop mitigation strategies, and measure  outcomes is hindered by limited access to complete and meaningful data.
section_title: Recommendations for Executive Action
paragraphs: To enhance the effectiveness of efforts to implement SMS and maximize  the positive impact of SMS implementation on aviation safety, we  recommend that the Secretary of Transportation direct the FAA  Administrator to take the following five actions:  1.  To better evaluate the effectiveness of the agency’s efforts to  implement SMS, develop a system to assess whether SMS meets its  goals and objectives by identifying and collecting related data on  performance measures.  2.  To align strategic goals with employee efforts, develop a system to  evaluate employees’ performance as it relates to SMS.  3.  To better manage implementation, develop a system to track and  report on SMS implementation across business lines.  4.  To better leverage existing resources and facilitate SMS  implementation, conduct a workforce analysis to inventory existing  employee skills and abilities and develop strategies for addressing  any SMS-related gaps identified.  5.  To maximize the positive impact of SMS implementation on aviation  safety, consider strategies to address airports’ concerns that may  negatively affect data collection and data sharing, including asking  Congress to provide additional protections for SMS data collected by  public entities.
section_title: Agency Comments
paragraphs: We provided the Department of Transportation and NTSB with a draft of  this report for review and comment. DOT and NTSB officials provided  technical comments, which we incorporated as appropriate and DOT  agreed to consider the recommendations. In addition, DOT officials stated  there is a need for FAA to have a common hazard-tracking system. FAA  has taken initial steps towards standardization by publishing FAA Order  8040.4A, Safety Risk Management Policy, which identifies terms and  definitions used for safety risk management. DOT also reinforced its  dedication to the success of SMS and noted its continued efforts to  improve its implementation plans with a measured, structured approach  to implementation.
We are sending copies of this report to the appropriate congressional  committees, DOT, NTSB, and interested parties, and others. In addition,  the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me on (202) 512-2834 or at dillinghamg@gao.gov. Contact points  for our Office of Congressional Relations and Public Affairs may be found  on the last page of this report. Key contributors to this report are listed in  appendix II.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objective was to assess the Federal Aviation Administration’s (FAA)  implementation of Safety Management Systems (SMS) and provide  information on potential implementation challenges. To do so, we  addressed the following questions:   (1)  What is the status of FAA’s implementation of SMS?   (2)  To what extent have FAA’s SMS efforts been consistent with key  practices for successful planning and implementation of a new  program?   (3)  What challenges does FAA face in implementing SMS?
To perform our review, we focused primarily on FAA’s implementation of  SMS for its business lines as well as its preliminary efforts to require and  oversee SMS implementation by industry. We conducted background  research to identify literature related to SMS in aviation, and any  challenges that agencies might face when implementing SMS. We also  attended parts of a safety risk management panel on runway status lights  conducted by FAA’s Air Traffic Organization (ATO) at Seattle-Tacoma  International Airport in March 2012 as a means of learning more about  SMS and related processes. During the data collection and drafting  phases of this report, FAA was in the midst of rulemaking efforts to  require SMS of Part 121 air carriers and Part 139 airports, so we did not  comment on any draft or proposed regulatory guidance.
To determine the status of FAA’s implementation of SMS, we reviewed  FAA’s SMS orders and pilot project guidance, implementation plans, and  Notices of Proposed Rulemaking for Part 121 air carriers and Part 139  airports. We also reviewed international and FAA guidance on SMS  issued by the International Civil Aviation Organization (ICAO) and the  Joint Planning and Development Office (JPDO), respectively, and  National Transportation Safety Board (NTSB) recommendations to FAA  related to SMS. Finally, we interviewed FAA SMS program managers  across FAA business lines and offices; industry experts we identified  based on their knowledge and experience in industry, recommendations  from aviation industry officials, and a search of SMS literature; and ICAO  and NTSB officials.
To assess the extent to which FAA’s efforts have been consistent with  key practices, we reviewed our reports and other literature on successful  project planning and implementation, particularly for large-scale  transformative projects, and condensed the resulting list to eliminate  duplication and overlap. To do this, we reviewed previous GAO reports  that highlighted practices associated with successful planning and  implementation of a new program. We removed or consolidated any  duplicate items across the reports to create a single list of 10 criteria. We  then identified FAA’s actions related to these practices by reviewing FAA  guidance and agency documentation such as its SMS implementation  plans, conducting interviews with FAA officials across its business lines,  and using that information to assess the extent to which FAA had  addressed each practice. We determined whether each key practice was  addressed, partially addressed, or not addressed by using criteria  developed for prior GAO reports. As such, we considered a practice  “addressed” if FAA had instituted the practice; “partially addressed” if FAA  had shown some progress toward instituting, or started but not completed  the practice; and “not addressed” if FAA had made minimal or no  progress toward instituting the practice. The team made these coding  decisions together, with two analysts making initial judgments and team  management reviewing and confirming them.
To identify challenges FAA faces in implementing SMS, we reviewed our  prior work on long-standing FAA challenges, such as those related to  training and data, and interviewed aviation industry experts and FAA  officials mentioned above. We reviewed prior GAO work on performance  measurement and workforce analysis, Department of Transportation  Inspector General reports and NTSB recommendations related to SMS.  To obtain industry views on challenges, we interviewed officials from  selected airports and air carriers, industry associations representing  airports, air carriers, and pilots, and individuals with SMS experience  described above. We also reviewed and analyzed documents, including  language in the FAA Modernization and Reform Act of 2012 related to  data protection, and associated scholarly work. To supplement comments  received from the individuals we interviewed, we also reviewed comments  made by aviation stakeholders on the two Notices of Proposed  Rulemaking related to SMS.
To obtain industry views on both SMS implementation practices and  associated challenges, we interviewed officials from selected airports and  air carriers, which we selected for diversity in size, location, participation  in FAA SMS pilot projects, and submission of comments on FAA’s two  Notices of Proposed Rulemaking related to SMS. (See table 1 for a list of  selected airports.)
We also interviewed officials from six air carriers: Delta, GoJet, United,  Pinnacle, Southwest, and US Airways. Finally, we interviewed officials  with SMS knowledge and expertise, including experts from the Flight  Safety Foundation, Embry-Riddle Aeronautical University, John A. Volpe  National Transportation Systems Center, and MITRE Corporation.
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Heather MacLeod (Assistant  Director); Elizabeth Curda; Leia Dickerson; Sarah Farkas; David Hooper;  Delwen Jones; Brooke Leary; Josh Ormond; Larry Thomas; and Elizabeth  Wood made key contributions to this report.