Aviation Safety: Additional FAA Efforts Could Enhance Safety Risk Management
Released: 2012-09-12T00:00:00

Letter

Background

SMS provides a top-down approach to managing safety risk, which FAA expects will improve aviation safety. SMS is not an additional safety program that is distinct from existing activities that accomplish an entity’s safety mission, but rather, a process for safety management that incorporates systematic procedures, practices, and policies. According to FAA, the overarching goal of SMS is to improve safety by helping ensure that the outcomes of any management or system activity incorporate informed, risk-based decision making. We reported in 2010 that FAA officials believe that successfully implementing SMS is critical to meeting the challenges of a rapidly changing and expanding aviation system. To achieve a higher level of safety in an already very safe system, FAA requires a more forward-thinking approach, which SMS provides, by addressing cultural and organizational problems that lead to safety hazards, identifying system-wide trends in aviation safety, and managing emerging hazards before they result in incidents or accidents.

SMS implementation should bring about a fundamental shift in aviation safety oversight. For decades, the aviation industry and federal regulators, including FAA, have used data reactively to identify the causes of aviation accidents and incidents and take actions to prevent their recurrence. While FAA plans to continue to use data to analyze past safety events, it is also working to use data proactively to search for risks. FAA’s shift to the proactive approach of SMS is important because, as accidents have become increasingly rare, less information is available for reactive analyses of their causes. As a result, information that can be used to help identify accident and incident precursors has become more critical for accident prevention. Thus, the open sharing of safety information among aviation stakeholders and how FAA’s policies and procedures govern the reporting of safety information are essential to the success of SMS.

SMS consists of four key components: (1) safety policy, (2) safety risk management, (3) safety assurance, and (4) safety promotion (see fig.1). Together, these four components are intended to provide a systematic approach to achieving acceptable levels of risk. FAA provides to its personnel detailed guidance on the principles underpinning these components and the application of these components to aviation oversight in its official orders and other internal FAA guidance. To the industry, FAA provides this SMS guidance via advisory circulars and a dedicated page for the SMS program office on the FAA website.

FAA is undertaking the transition to SMS in coordination with the international aviation community, working with ICAO to adopt applicable global standards for safety management. ICAO requires SMS for the management of safety risk in air operations, maintenance organizations, air traffic services, and airports as well as certain flight-training operations and for organizations that design or manufacture aircraft. Further, ICAO has published safety management requirements for its member countries that mandate that civil aviation authorities—such as FAA—establish SMS. ICAO first mandated SMS worldwide for air traffic service providers, such as air carriers and certified aerodromes, in 2001. ICAO later specified that member states should mandate SMS implementation for airports, air carriers, and others by 2009. FAA began SMS implementation in 2005, but FAA officials informed ICAO that the agency and industry would not be able to meet the 2009 deadline. ICAO is allowing FAA to take additional time in its efforts to implement SMS, with the understanding that implementation is under way and that FAA is in the midst of a rulemaking to require SMS for commercial air carriers. ICAO officials stated that the United States is one of the leading implementers of SMS worldwide and acknowledged that SMS implementation in the U.S. aviation system may be more complicated than in other countries because of the size and complexity of the U.S. aviation industry. ICAO has not specified a date by which FAA is expected to comply with the requirements to implement SMS in the aviation system. There have also been actions within the United States to encourage implementation of SMS. For instance, in 2007, NTSB recommended that FAA require all commercial air carriers to establish an SMS and, in 2011, added SMS for all modes of transportation to the NTSB’s Most Wanted List, identifying SMS as one of the most critical changes needed to reduce the number of accidents and save lives.

Partially in response to the ICAO requirement, FAA added goals related to SMS implementation to its 2009-2013 Flight Plan. a requirement to implement SMS in three of FAA’s business lines—the Air Traffic Organization (ATO), the Aviation Safety Organization (AVS), and the Office of Airports (ARP)—and a goal to implement SMS policy in all appropriate FAA organizations, which include the Office of Commercial Space Transportation (AST) and the Office of NextGen (ANG). FAA is in the process of implementing SMS within these business lines and offices as well as in industry through rulemakings to require airports and commercial air carriers to implement SMS. FAA designated AVS as the lead for SMS implementation in September 2008. Within AVS, the Office of Accident Investigation and Prevention’s (AVP) Safety Management and Research Planning Division coordinates and manages SMS implementation and operation across the agency, and so AVP serves as the official SMS lead for the agency.

Federal Aviation Administration, 2009-2013 Flight Plan is the agency’s strategic plan.

Administrators, their deputies, and other high-level FAA officials from each business line or office (see fig. 2). Within some of the business lines, there are offices devoted to specific aviation oversight functions that are responsible for overseeing detailed implementation of SMS for those functions. For example, the Flight Standards Service (AFS), a division of AVS that provides safety oversight of commercial air carriers and others, is taking steps to require SMS implementation by commercial air carriers and is also working to integrate SMS into its internal activities. In addition, the Aircraft Certification Service (AIR), a division of AVS that provides safety oversight to aviation design and manufacturing firms, is leading agency efforts to encourage SMS implementation for that industry sector, while ARP is leading agency efforts to require SMS implementation for certificated airports.

SMS implementation will require changes to many of FAA’s operations. As the agency and industry implement SMS, shifts will be necessary in both the skills of FAA and industry staff and the tools that the agency uses to monitor safety. FAA’s integration of SMS into its business practices will also affect how the agency provides air navigation services and oversees the aviation industry. Historically, FAA oversight of airlines, airports, and other regulated entities has involved oversight of such things as operations and maintenance. FAA will continue this oversight, but will also apply SMS principles to its processes for oversight. The agency will provide oversight of the safety management systems of service providers such as air carriers and airports to help ensure that they are managing safety within their operations through SMS. For example, AFS currently provides oversight of the operations, maintenance, and safety data of commercial air carriers and others. Once SMS is fully implemented, AFS will continue to provide this oversight and will also conduct oversight of the safety management systems that commercial air carriers and others put in place.

FAA and Its Business Lines Are at Different Stages of SMS Implementation

ATO completed its implementation of SMS, but FAA and several of its other business lines and offices are in the early stages of implementation. Most FAA business lines and offices have guidance and plans for SMS implementation in place and have begun to integrate SMS-related practices into their operations, but many tasks remain and aviation officials and experts with whom we spoke project that full SMS implementation will take many years.

FAA Recently Finalized Its Agency-Wide Implementation Plan, but Full SMS Implementation Is Likely to Take Many Years

FAA finalized its agency-wide plan for SMS implementation in April 2012. The plan provides a road map for SMS implementation across the agency and describes the activities that FAA business lines and offices will need to complete by the end of 2015 to integrate SMS into their operations. These activities will lead to outcomes including: revising and standardizing safety policies and safety risk management methodologies across FAA to ensure SMS principles are consistently addressed; improving organizational processes so that FAA business lines and offices can share safety data and information more easily; and coordinating communications to ensure a common understanding of SMS across the agency.

FAA began its agency-wide SMS implementation efforts in 2008, and in September of that year issued a policy for implementation of a common SMS within FAA. Among other things, the policy sets forth management principles to guide all of FAA in safety management and safety oversight activities and requires AVS, ARP, and ATO to develop and execute business line-specific plans for SMS implementation. In late 2008, FAA formed the agency-wide FAA SMS Committee to coordinate implementation efforts across FAA business lines and offices. Overall, the agency has taken a bottom-up approach to implementation, with some individual business lines and offices beginning implementation prior to agency-wide efforts. FAA has also taken steps to ensure that its plans for SMS implementation and policies align with international and government-wide requirements and technical guidance on SMS implementation, including ICAO’s Standards and Recommended Practices, the ICAO Safety Management Manual, and the JPDO SMS Standards. For instance, officials stated that they consulted international and government-wide guidance on SMS implementation when drafting agency implementation plans. (See fig. 3 for more information on alignment of FAA requirements with international and government-wide requirements and guidance on SMS.)

Although FAA has made progress, completion of SMS implementation across FAA is likely to take many years. FAA’s agency-wide SMS implementation plan includes tasks with estimated completion dates through 2015, and some implementation tasks may take even longer to complete. For instance, a project plan that AVS officials developed to track status of AVS SMS implementation tasks contained in its implementation plan includes task completion dates through 2016. According to FAA, the overall SMS implementation effort is an evolutionary process that will not have a specific completion date. The current implementation time frame is consistent with experts’ estimates of how long it may take to implement SMS and with other large-scale organizational transformations. For example, representatives from The MITRE Corporation, which manages a federally funded research center for FAA and assisted FAA in selected SMS implementation efforts, stated that organizational transformations like SMS can take from 6 to 10 years.

ATO Completed SMS Implementation in 2010

ATO is the only entity among FAA and its business lines to have completed SMS implementation. ATO issued its internal SMS guidance in March 2007 and finalized both its SMS implementation plan and its updated SMS Manual in 2008. According to ATO officials, ATO completed SMS implementation in March 2010, and the FAA Air Traffic Safety Oversight Service validated that ATO’s implementation of SMS was complete. Officials stated that implementation within ATO was simpler, in part, because it is the only branch of FAA that is considered an aviation service provider and therefore did not have to conduct a rulemaking for external entities as part of its SMS implementation.the implementation phase complete, ATO is currently in the continuous improvement phase of SMS. This means that ATO will continuously use the SMS-based processes now in place to identify hazards, enact strategies to mitigate the risks associated with those hazards, and assess the extent to which the mitigations are working effectively. In addition, FAA officials stated that ATO is working to improve its SMS operations, will update guidance on SMS, and plans to perform audits of its SMS functions on a regular basis. ATO officials added that they are working to With share lessons learned from their implementation efforts with other FAA business lines and to develop SMS tools and processes that can be commonly implemented across all FAA business lines.

Most FAA Business Lines and Offices Are in Early Stages of Implementation

With the exception of ATO, most FAA business lines and offices are in the early stages of implementation, either in terms of integrating SMS into their internal processes or in terms of their efforts to prepare to provide oversight for proposed requirements for industry implementation of SMS. To date, much of the work of the FAA business lines has focused on efforts to draft implementation policies and guidance, train employees, and create tools for applying safety analyses and risk-based decision- making to safety oversight. (See fig. 4 for more information on the status of key SMS implementation efforts across FAA.)

AVS began its SMS implementation efforts in August 2006 and finalized its SMS implementation plan in January 2012, which was then incorporated into FAA’s overall plan for SMS implementation. Since 2006, AVS and its seven services and offices have issued orders and other guidance on SMS implementation; developed SMS training courses; conducted voluntary pilot projects and rulemaking efforts on SMS implementation for industry; and worked to begin integrating elements of SMS into their operations. For example, AIR officials, who provide oversight of aviation design and manufacturing firms, have developed a central database that provides standard criteria for analyzing service data in a risk-based manner. This should allow AIR inspectors and engineers to rate the risk of potential safety issues and prioritize oversight to high risk issues.

Some services and offices within AVS are in the midst of efforts to require SMS for industry and are also operating voluntary pilot programs to promote SMS implementation within industry. A final rule to require SMS for commercial air carriers is expected to be issued in September 2012. In 2007, AFS launched a pilot program to encourage voluntary implementation of SMS by industry.

According to FAA officials, as part of its rulemaking efforts for commercial air carriers, FAA and AVS are developing a new part in the Code of Federal Regulations (CFR)—Part 5—that will describe SMS implementation requirements for Part 121 certificate holders. In the future, FAA may conduct rulemakings to require additional sectors of the aviation industry to meet Part 5 requirements (see fig. 5). AVS officials stated that efforts to establish SMS requirements more broadly across the aviation industry will likely take many years.

Though FAA has not yet required SMS for air carriers or other parts of industry, FAA has acted to encourage SMS implementation by industry through voluntary pilot projects, and some aviation stakeholders have chosen to implement SMS in advance of any federal requirement. Some sectors of the aviation industry are farther along in their implementation of SMS than others. For instance, FAA officials stated that a large majority of commercial air carriers are in the process of implementing SMS. As of June 2012, over 90 percent of commercial air carriers operating under Part 121 were participating in the AFS pilot program, which provides air carriers with direct implementation support from FAA officials under a more relaxed implementation time frame than is anticipated under an eventual implementation regulation. Of these air carriers, three have reached the final stage of SMS implementation. However, most small air carriers have not yet begun implementing SMS. In contrast to AFS, AIR is at an earlier stage in its efforts to require SMS for the approximately 3,000 design and manufacturing firms it oversees. AIR began a voluntary pilot project for SMS implementation by design and manufacturing firms in 2011 and has 11 pilot project participants. AIR officials stated that they are in the process of launching a second aviation rulemaking committee to continue to explore options to require SMS for design and manufacturing firms. Officials also noted that AFS and AIR are working together to share lessons learned and assist one another in their implementation efforts.

ARP is in the early stages of working to integrate SMS principles into its oversight of airports, and recently took steps to reduce the scope of that oversight. ARP initially planned to apply SMS-based oversight to all certificated airports. Officials stated that ARP is currently limiting its SMS- based oversight to large hub airports because of budget constraints and will reassess its capacity to expand oversight to smaller airports in 2013. ARP began its SMS implementation in 2010 and issued an internal order to provide a basis for the integration of SMS into its operations later that year. The office finalized its SMS implementation plan in September 2011 and has begun to make changes to its oversight. For instance, In June 2011, ARP began to apply SMS-based oversight to construction projects at the 29 large hub airports in the United States. Under this new oversight framework, ARP staff assess proposed airport construction projects using risk-based SMS principles, and airports need to incorporate strategies to mitigate identified risks into their construction plans prior to receiving ARP’s approval for the project. Like AVS, ARP is also in the midst of a rulemaking to require SMS for all certificated airports and has completed three voluntary SMS pilot projects for airports from 2008 to 2011. Thirty-one airports participated in at least one of ARP’s SMS pilot projects. ARP is using information gathered through the pilot projects to inform a planned advisory circular that will provide additional guidance to airports on SMS implementation. The pilot projects also allowed airports to share their SMS implementation practices with other airports. The final rule to require SMS for Part 139 certificated airports is expected to be issued in April 2013 and, if implemented as proposed, would require over 500 airports to implement SMS.

Other FAA business lines are in varying stages of implementation. AST is not currently required to implement SMS; however, AST is taking initial steps toward integrating SMS into an existing set of safety management processes. ANG is farther along in its implementation of SMS because of its previous status as a part of ATO. basing its implementation of SMS on policies and processes established during ATO’s implementation of SMS. The officials stated that since ANG will provide the systems and components that will be used by ATO to manage air traffic, it made sense for ANG to develop its SMS based on policies, processes, and systems established by ATO. Officials stated that ANG completed its implementation plan in June 2012 and estimated that ANG’s SMS implementation is about 70 percent complete.

FAA’s SMS Approach Is Consistent with Many but Not All Key Practices for Successful Project Implementation

There are a number of key practices and implementation steps that can help agencies successfully plan for and implement new projects, including large scale transformative ones, such as FAA’s implementation of SMS. As we have previously reported, addressing these key practices can help an agency improve its efficiency, effectiveness, and accountability. FAA currently has many of these key factors in place, such as established support from top leadership and a clear project mission; however, it has only partially addressed other key practices, such as providing needed expertise and technology, and has yet to establish SMS performance measures (see fig. 6).

In 2011, FAA reorganized some of its offices and, as part of the reorganization, separated NextGen efforts from ATO.

Many Key Practices Are in Place

FAA has instituted many key practices that will help it prepare for and implement SMS across its business lines and offices.

Top leadership: Top leaders from each FAA business line provide support for and actively participate in SMS implementation. As previously mentioned, FAA established the SMS Executive Council, a group of high-ranking FAA officials that provides executive-level guidance and conflict resolution for SMS-related issues across the agency. In accordance with our key practices, the SMS Executive Council has the authority to make resource allocation decisions, but also confers decision-making authority where appropriate to the FAA SMS Committee.

For instance, FAA officials told us that the SMS Executive Council retains the authority to make final decisions about changes to FAA’s implementation plan that affect policies or procedures for multiple business lines; the FAA SMS Committee has the authority to make decisions that relate to daily concerns that fall within the purview of its members. For example, committee members settled a disagreement between ATO and airport officials over whether an airport should conduct certain components of a safety risk management panel. At the time, FAA had not yet issued its safety risk management policy clarifying terms and requirements, so the airport and ATO each had its own distinct safety risk management definitions and processes. Working with ARP and ATO officials, committee members identified a compromise in which ATO protocols were followed, but any disagreements on terms or procedures were documented. ARP officials told us that FAA’s safety risk management policy, issued in April 2012, should help prevent this type of disagreement from occurring.

Clear project mission: FAA’s internal order requiring SMS implementation for ARP, ATO, and AVS clearly describes that FAA’s mission is to improve aviation safety and that implementing SMS and its components supports that mission. Each business line also has its own internal order requiring SMS implementation that mirrors this mission and goals.

Implementation team: AVP’s safety management division and the FAA SMS Committee, function jointly as FAA’s dedicated SMS implementation team. The team’s structure and actions align with our criteria for a strong and stable team because it is composed of senior-level program managers from each business line, all of whom had received SMS training according to FAA officials. Despite some recent departures, its membership has been largely stable.

Leading practices: FAA shares information across business lines to identify lessons learned related to SMS implementation. For example, ATO assembled lessons learned from its SMS implementation into a presentation for the other business lines, and included tips such as encouraging others to implement a training program and monitor mitigations. According to FAA’s implementation plan, the agency plans to systematize the sharing of lessons learned by creating a central repository to collect and communicate safety lessons learned among its business lines and offices by September 30, 2013.

Troubleshooting: FAA has processes in place to manage SMS implementation across FAA, including troubleshooting unexpected problems. For example, the FAA SMS Committee meets monthly and manages agency-wide SMS implementation and any challenges that arise, and regularly briefs the SMS Executive Council, a briefing that includes a discussion of any issues or unexpected problems that could not be resolved at the committee level. For instance, when the Air Traffic Manager at an airport disagreed with airport officials regarding how to handle a potential safety issue with planes that were taking off on runways that were temporarily closed, the FAA SMS Committee elevated the issue to the SMS Executive Council, which resolved it. As we have previously reported, instituting practices like these can help an agency become more results-oriented, customer- focused, and collaborative.

Although FAA is still in the process of finalizing new requirements for airports and air carriers to implement SMS, it has already taken some steps to institute key practices for those efforts. For example, FAA officials stated that the agency has taken steps to identify leading practices during pilot projects by soliciting information from participating airports and air carriers, and FAA officials told us they plan to incorporate these lessons learned into rulemaking and guidance. ARP officials reported that they encouraged pilot project participants to share lessons learned directly with one another through studies and roundtable discussions, and incorporated some of the lessons learned into FAA advisory circulars. FAA has also made efforts to troubleshoot and manage unexpected problems with pilot participants through meetings, calls, and conferences with airport and air carrier officials to understand their experiences. For example, AFS officials reported that they helped officials from air carriers to understand when certain safety risk management documentation and processes are necessary, and how they could be adapted for a variety of changes made to carrier operations, including smaller day-to-day changes. However, despite this assistance, officials from some airports that participated in pilot projects reported that they could have benefited from additional assistance from ARP, such as clarification on the safety risk management component of SMS. In addition, an official at one airport told us that he would have liked FAA to facilitate conversations between airports of similar size to help them share lessons learned.

FAA Efforts to Create a Project Plan and Consult and Assist Stakeholders Partially Align with Key Practices

Other steps FAA has taken in its SMS implementation efforts partially align with key practices for implementing a new program.

Project plan: Currently, the agency-wide project plan for SMS implementation is a single page of high-level milestones, which AVP officials monitor and report on to the SMS Executive Council. Also, AVS has a detailed project plan for its own SMS implementation and elements of agency-wide implementation for which AVP, as the agency SMS lead, has responsibility. Officials stated that they have plans to develop a system to monitor and track the progress of activities needed to implement SMS, but FAA does not currently have a system for tracking agency-wide SMS implementation, a key practice particularly important during the initial planning phase of project implementation. However, given the scope and complexity of SMS, a detailed, agency-wide project plan could help FAA track and monitor the interim steps of SMS implementation across the agency.

Without such a plan, it may be more difficult for FAA to identify problems or deviations from planned activities, putting both the timeliness and effectiveness of SMS implementation at risk.

Consulting with stakeholders: FAA has made efforts to consult with employees and stakeholders regarding its SMS implementation, but it has not yet developed a communications plan. Agencies should involve employees in planning, and incorporate employee feedback into new policies and procedures. FAA involved its business line program managers and some of the managers’ staff by assigning them responsibility for the day-to-day tasks related to implementing SMS across the agency. FAA has involved other employees by soliciting questions and comments on SMS in town hall meetings and the online DOT site called “IdeaHub,” and by offering SMS training through each business line. ATO, ARP, and AVS all offer introductory SMS courses for their staff as well as additional related courses, such as an SMS course specifically for managers and ATO’s safety risk management course.

FAA has been working to implement SMS for the last 4 years, but the agency does not have a communications plan or strategy for ensuring that the SMS messages communicated to staff are consistent across the agency. Instead, FAA relies on a more informal communications structure in which each program manager staffed to the implementation team communicates relevant information back to their respective business line. The implementation team does not communicate any information directly to employees, which could hinder the team’s ability to ensure consistency in its message across FAA. ATO officials reported experiencing this challenge at the beginning of ATO’s SMS implementation, when a lack of clear requirements for communicating SMS information resulted in variation in staff’s understanding of guidance. We have previously reported that a communication plan or strategy can ensure consistency of message, provide information to meet the specific needs of employees, encourage two-way communication, and build trust. FAA plans to begin working on a communications plan in September 2012, and is scheduled to issue the plan at the end of February 2013. FAA officials also said they are in the process of developing an internal SMS website for employees to share information and ideas, which could enhance SMS communications. However, until the communications plan is developed and implemented, FAA’s employees may not receive timely or consistent information on SMS or be as invested in its implementation as they might otherwise be.

FAA’s approach to overseeing industry SMS implementation allowed for additional two-way communication. For example, FAA solicited views on SMS implementation from airport and air carrier officials through voluntary pilot projects described previously, and learned more about industry perspectives through the formal rulemaking process—whereby an agency issues a Notice of Proposed Rulemaking and is required to notify the public and give them an opportunity to submit comments.

Providing technology and expertise: FAA has provided some SMS training and tools to its employees; however, it has not yet provided other tools important for SMS implementation. FAA officials reported that each business line has provided SMS training to staff. In addition, FAA recently developed a standardized Safety Risk Management (SRM) policy, which will assist employees across FAA by standardizing SRM terminology and clarifying confusion on the conduct of SRM across the agency. FAA plans to create a simple version of an agency-wide hazard-tracking system in the next 3 to 6 months, but does not have plans to create a more complex system until August 2015, according to FAA’s SMS implementation plan. The simple version will draw from hazard-tracking systems already in place in some business lines, and summarize information from them to highlight broader hazards such as those that would affect multiple business lines. For instance, FAA officials stated that if ATO wanted to make a change to its operations at a particular airport, then ATO would be responsible for identifying associated hazards, risks, and risk mitigations and would also be responsible for assuming responsibility for the risk. However, if ATO determined that the airport was better equipped to mitigate the identified risks, then the airport and ARP would become more involved in designing risk mitigations and overseeing their implementation. FAA’s efforts to provide tools to help in SMS implementation are affected by differences in how data are collected and assessed across the agency. For example, these differences have held back agency efforts to model how changes to the national airspace system, such as increases to air travel, can affect safety. We have previously reported on and made recommendations related to FAA’s data challenges, and also discuss them later in this report. These data challenges mean that FAA is not always able to perform comparisons across databases, a challenge that that limits the usefulness of the data in identifying possibly dangerous hazards. Identifying, monitoring, and mitigating hazards is a key tenet of SMS, and without the proper technologies and tools, FAA may not be able to do this as effectively.

FAA Has Yet to Integrate SMS into Employee Performance Plans and Establish Performance Measures

FAA’s efforts do not align with two key practices for implementing a new program.

Integrating SMS into employee performance plans: FAA does not consistently evaluate employees’ performance on SMS-related tasks. We have previously reported that effective performance management systems create a clear linkage between individual performance and organizational success, and include aligning individual performance expectations with organizational goals. FAA’s organizational mission and goal, and that of SMS, is to improve safety, yet FAA officials told us that the agency does not require employee performance plans to include SMS-related tasks. Although officials reported that some employees’ performance plans explicitly include SMS items, such as providing SMS training or developing SMS policy, it is left to the discretion of each business line whether SMS items are included. FAA officials told us that SMS principles and methodologies will be included in the performance plans of employees involved in writing SMS policy and revising SMS processes, and will be incorporated into the tasks of others once SMS implementation reaches those individuals. However, currently, none of the business lines require this. As such, FAA does not have a system for assessing the extent to which staff are effectively supporting SMS, and FAA may not be able to determine if staff are completing tasks and responsibilities necessary for the successful implementation of SMS.

Measuring performance: FAA does not have performance measures in place to assess whether the SMS goals of improving safety are being achieved. FAA has broader safety-related performance measures, such as tracking rates of runway incursions and losses of separation, but SMS-related performance measures could address intermediate safety issues, such as precursors to incursions or incidents. Such measures could help FAA track progress toward its broader safety measures. FAA officials told us that AVS is a member of the Safety Management International Collaboration Group, a group formed in 2009 to address safety management-related topics, including performance measures. Most recently, FAA formed an agency-wide working group to study performance metrics for SMS implementation, and FAA’s implementation plan states that such metrics will be finalized in October 2014. However, FAA officials we spoke with acknowledged that they are at the very beginning phase of this process and, although already in the process of implementing SMS, have not yet identified metrics to measure safety results under an SMS system. We have previously reported that performance information is critical for achieving results and maximizing the return on federal funds. Performance measures should help FAA identify the extent to which SMS implementation will contribute to increased aviation safety—FAA’s stated overall goal for SMS—as well as help identify what changes could be made to improve SMS performance over time.

Addressing Key Implementation Practices and Other Challenges Could Enhance SMS Effectiveness

As previously mentioned, FAA has taken steps to address many of the practices associated with planning and implementing a new program. However, we identified six challenges that could negatively affect FAA’s efforts to implement SMS in a timely and efficient manner: 1) the large scope and complexity of SMS implementation, 2) resource and capacity constraints, 3) standardization of policies and processes, 4) data sharing and protection, 5) data quality and usefulness, and 6) development of performance measures to evaluate SMS effectiveness.

Large Scope and Complexity of SMS

Implementing SMS is one of several major initiatives FAA has under way, and its sheer scope and complexity could affect, or be affected by, concurrent FAA efforts such as NextGen or Unmanned Aircraft Systems. SMS requires changes in many of FAA’s operations: from the way the agency tracks hazards to the way it oversees industry. SMS will also require a transformation of FAA’s and the aviation industry’s safety culture to one in which information and safety data are shared openly, and errors are addressed through whatever action is necessary to prevent them from happening in the future. FAA is making efforts to move toward this new approach to safety, for instance by using data-sharing systems that are protected from public disclosure to encourage voluntary reporting of safety issues and enable more robust analysis of safety data among FAA and air carriers. Moreover, as previously stated, each of FAA’s business lines has its own role in implementing SMS that must be coordinated across the agency. This is particularly challenging because the business lines are at different stages of implementation and, according to FAA officials, have historically operated independently.

The scope and complexity of SMS implementation may also be a challenge for the aviation industry, and some stakeholders expressed concerns both in interviews and in official comments on FAA’s Notices of Proposed Rulemaking that eventual FAA requirements to implement SMS need to allow for variation in airport and air carrier operations. For example, officials from some smaller airlines and airports noted that SMS implementation could require additional resources, such as staff and software, which may not be readily available. In addition, officials from some airports and air carriers were concerned that FAA’s final requirements would be too prescriptive to allow entities to implement an SMS program that best fit their organizational type, management practices, and resources. Most stakeholders and experts we interviewed stated that FAA could design SMS requirements for airports and airlines that are scalable and flexible to accommodate this variation, which would address these concerns. For instance, airport officials from smaller airports told us that staff size limits their ability to assign a dedicated SMS employee or safety director, while some officials at larger airports said they were able to hire a SMS safety director or already had an established safety director in place. Also, FAA’s SMS implementation pilot project for airports found that 35 percent of participants planned to hire additional staff to support SMS and 15 percent were not sure. FAA officials have noted that they understand these scalability concerns, and are taking them into consideration as they develop final SMS rules for industry.

Resources and Capacity

SMS implementation across FAA will require some skills that agency employees currently do not have, yet FAA has not formally assessed the skills of its workforce to identify any gaps in the expertise required to implement SMS or determined how to fill those gaps. In addition, FAA officials stated that existing staff may not be able to be trained to fill SMS implementation needs in all cases. For instance, FAA officials noted that SMS implementation will require some engineers and other technical employees to understand certain terminologies and have certain knowledge, skills, and abilities, such as an enhanced ability to perform complex modeling and analysis of aviation safety data to identify potential safety hazards. AVS officials stated that to implement SMS, additional employees with skills in analyzing data for hazards and associated risks would be needed, along with additional training for existing staff. ARP officials stated that the office might need program analysts with specific data analysis skills to implement SMS.not expect to receive significantly more resources and, as previously mentioned, have already had to reduce the scope of the office’s SMS- based oversight because of insufficient staff. Stakeholders and experts also questioned whether FAA currently has the resources and capacity needed to fully implement SMS. For example, experts noted that FAA may not have the requisite engineers and other staff to participate in safety risk management efforts, or FAA inspectors to oversee individual airport and air carrier SMS programs.

ARP officials stated that they do Despite these concerns, FAA has not yet conducted a strategic workforce assessment to accurately determine the skills and staffing levels it needs to manage SMS. Although FAA’s SMS implementation plan recommends that business lines create such staffing analyses, none have done so. Nor has FAA conducted an agency-wide workforce assessment for SMS. Our internal control standards state that agencies should ensure that skill needs are continually assessed to ensure workforces have the skills necessary to help the agency meet its goals. We have reported that strategic workforce planning is an integral part of human capital management and helps an agency, among other things, determine the critical skills and competencies that will be needed to achieve current and future programmatic results, and then develop strategies tailored to address any gaps identified.

A workforce analysis could help FAA determine how to best address its most critical needs in ways that account for budget limitations, such as through retraining or shifting staff, rather than hiring additional employees. Without conducting an agency-wide SMS workforce analysis, FAA cannot be sure that it has sufficient staff, skills, or competencies to implement SMS, thus putting its SMS implementation efforts at risk.

Standardization of SMS Policies and Processes

SMS standardization across FAA business lines and offices is central to implementation success, yet developing common systems for distinct FAA business lines and offices has proved challenging. For example, FAA realizes that the agency needs a common hazard-tracking system in order to maximize SMS effectiveness, yet FAA officials and stakeholders stated that it is difficult to develop such a system because each of FAA’s business lines uses different hazard-related terms and definitions, and often different data systems. These differences, in turn, prevent the agency from performing simple comparisons across databases and have delayed advances in using data analysis to proactively identify potential safety hazards. FAA officials stated that the agency has recently taken steps to make its databases interoperable, and also recently issued a standardized policy for the safety risk management component of SMS. Both of these steps may enhance FAA’s hazard-tracking and analysis capabilities. The agency is also working with ICAO to address issues related to standardization, such as adopting a collaborative approach to increase the sharing of safety information internationally.

Industry officials are also concerned that FAA inspectors and certificate management offices may have different interpretations of SMS and other regulations. We and others have previously reported that variation in FAA’s interpretation of standards for certification and approval decisions is a long-standing issue. Industry stakeholders we interviewed expressed concerns that a similar result could occur once final rules are issued requiring airports and air carriers to implement SMS, and could lead to airports or air carriers of similar size being held to different standards of SMS implementation. FAA officials acknowledged that this is a challenge for the agency and noted that the agency plans to provide additional training to inspectors related to oversight of SMS. Additionally, based on our 2010 recommendation, recent legislation directs FAA to establish an advisory body of government and industry representatives to address the issue of inconsistent interpretation of regulations.

FAA’s organizational structure for SMS implementation may pose challenges to standardization as well. For example, as previously mentioned, AVP’s safety management division is the lead for SMS, and AVP and the FAA SMS Committee share responsibility for implementing SMS across the agency. Despite AVP’s role as lead for SMS implementation, it does not have any additional authority compared to the other business lines’ committee representatives, something that AVP officials noted can make SMS implementation difficult. This could slow decision-making, particularly around issues that require business lines to come to a single decision, such as how to standardize policies. Nevertheless, FAA officials acknowledged that having to collaborate to implement an agency-wide SMS has improved communication among the business lines. FAA will likely continue to face challenges standardizing its policies and processes as standardization of this scale is not something the agency has previously undertaken, and the need to negotiate solutions across FAA business lines could take time.

Data Sharing and Protection

Airport officials’ concerns about sharing and protecting their safety data may reduce SMS effectiveness by limiting the ability of airports and FAA to analyze safety data and identify trends. Although FAA has some data protections in place, such as those established by the FAA Modernization and Reform Act of 2012, which protects data that airports and air carriers submit to FAA for SMS from federal Freedom of Information Act (FOIA) requests, any data airports collect and any data air carriers share with airports could be subject to state-specific FOIA laws. Most certificated U.S. airports are either owned by a state, a subdivision of a state, or a local government body, and thus are subject to state laws, including state FOIA laws. This means that data airports collect and submit to FAA for SMS—such as information on hazards or other safety data—is protected from federal FOIA public disclosure requests, but, according to officials and experts, may be subject to public disclosure under state FOIA laws. Air carriers are not directly subject to state FOIA laws because they are privately owned. Nevertheless, officials and experts stated that these laws could affect air carriers because any data they choose to share with airports could then be subject to state FOIA laws. As a result, air carrier officials told us they may be less likely to share safety information with airports. Airport and airline officials’ primary concern is that the public disclosure of such information could result in negative publicity or expose them to legal liability in the event of an incident or accident. FAA officials said that data protection and legal liability are two of the major concerns throughout the aviation industry that could hinder the implementation of SMS.

FAA officials told us that they intend to continue to promote and expand safety information sharing efforts, but that airports could find ways to structure their SMS implementation so that they realized safety benefits while limiting the public release of air carrier safety information. In FAA’s official response to comments on two Notices of Proposed Rulemaking, FAA stated that airport officials are best situated to understand how to comply with state laws. Nonetheless, we found consensus among NTSB and many aviation stakeholders that FAA should seek congressional action regarding the protection of airport data from state FOIA laws.

Data sharing can also be challenging within FAA. In 2011, we recommended that FAA improve information sharing among its programs because not doing so could limit the ability of FAA and others to analyze safety data and understand safety trends. The Department of Transportation agreed that it must continue to promote and expand safety information sharing efforts and safety practices in order to maximize the effectiveness of safety data mining to analyze trends and prioritize safety efforts to address hazards before they lead to incidents or accidents. However, our recommendation remains open. According to officials, ICAO has also formed the Safety Information Exchange Study Group to help enhance data protection and identify potential international solutions.

Data Quality and Usefulness

Long-standing issues with data quality and usefulness could negatively affect FAA’s understanding of aspects of the safety of the aviation industry and, consequently, affect SMS’s effectiveness. Obtaining relevant data and understanding how to analyze those data to identify potential hazards are major challenges that FAA will need to overcome. In recent GAO reports, we commented on FAA’s lack of data to effectively assess aviation trends for certain types of events and the safety performance of certain industry sectors. For instance, in April 2012, we reported that for such events as runway excursions (when an aircraft veers off or overruns a runway) and ramp accidents (incidents or injuries that occur off the runway), a shortage of FAA data exists for analysis. The Department of Transportation concurred with this and our recommendations, and stated that the agency has taken steps to improve to its data quality and usefulness. For example, the FAA SMS Committee directed a working group to determine what safety data the agency is going to collect and track and to recommend what kind of system will be needed. However, FAA has not yet fully implemented several of our recommendations aimed at improving its capability to use data for aviation safety oversight, or several data-related NTSB recommendations from recent years. For example, we recommended that FAA extend standard quality controls, as appropriate, to the databases that support aviation safety oversight to ensure that the data are as reliable and valid as possible. By not fully addressing these challenges and recommendations, FAA’s ability to comprehensively and accurately assess and manage hazards and risk will be compromised, reducing the ability of SMS to prevent incidents and accidents.

SMS Performance Measures

The aviation community has widely acknowledged that developing SMS performance measures is difficult, but without them, FAA will not be able to gauge the direct impact of SMS on aviation safety. Some stakeholders told us about ways in which SMS improved their organization’s operations, and these examples could provide insight into possible SMS performance measures. For instance, some airports and air carriers that participated in FAA’s SMS pilot projects reported that SMS implementation improved communication across their organizations, helped them identify organizational gaps—such as those in internal auditing and training—and decreased employees’ injuries, aircraft damages, and insurance costs. Officials from the Flight Safety Foundation recommended that the extent to which SMS informs management decision making, such as by redirecting resources or shifting priorities, may be one way to measure SMS effectiveness. An FAA official suggested that performance measures could be directed to specific components of SMS, for instance tracking the number of risks mitigated as a measure of safety risk management efficacy. We have previously reported that agencies need to set quantifiable outcome-based performance measures for significant agency activities, such as SMS, to demonstrate how they intend to achieve their program goals and measure the extent to which they have done so. Performance measures allow an agency to track its progress in achieving intended results, which can be particularly important in the implementation stage of a new program such as SMS. In our prior work we recommended that agencies develop methods to accurately evaluate and measure the progress of implementation, and develop contingency plans if the agency does not meet its milestones to complete tasks. FAA has established a working group to study the issue and participates on two international performance measures work groups: the Safety Management International Collaboration Group and the aforementioned Safety Information Exchange Study Group.

Conclusions

FAA is making progress implementing SMS, both within the agency and for the aviation industry. However, SMS implementation represents a significant cultural and procedural shift in how the agency will conduct business internally and provide oversight to aviation stakeholders such as air carriers and airports, and by all estimates, this transformation will take many years to complete. Going forward, if FAA is to attain the full benefits of SMS, it will be important for the agency to remain committed to fully implementing SMS across its business lines. FAA has taken a number of steps that align with practices we identified as important to successful project planning and implementation, but has not addressed or has only partially addressed other key practices. These practices are important for large-scale transformative projects such as SMS, which require a dramatic shift in FAA’s approach to safety oversight and management. In the absence of these key practices, it may be difficult for FAA to prioritize projects or monitor SMS implementation and progress toward improving safety.

Aviation safety is a shared responsibility among FAA, air carriers, airports, and others in the aviation industry, and efforts to improve safety will require the agency to overcome several challenges. The magnitude of SMS’s potential impact on aviation oversight and the complexity of implementation are both a benefit and a drawback for FAA, as SMS implementation could help ensure the continued safety of the U.S. aviation system, but could also affect implementation time frames for other large initiatives as the agency works in a resource-limited environment. FAA officials believe that SMS implementation will require some skills that employees do not currently have; however, FAA has not conducted an agency-wide workforce assessment. With agency resources and capacity in great demand, it will be important for the agency to maximize the efficiency of SMS implementation, both through efficient use of its workforce and creation of policies and systems that standardize and streamline implementation. In addition, data protection concerns from airport officials and others could prevent aviation stakeholders from fully embracing SMS implementation, thus hindering its effectiveness. Without assurance of protection from state FOIA laws, some aviation stakeholder may choose to collect only the bare minimum of safety-related data or may choose to limit the extent to which collected information is shared among aviation stakeholders. The agency also lacks sufficient data to effectively assess aviation trends for some events as well as the safety performance of certain industry sectors. The ability of FAA to identify safety risks, develop mitigation strategies, and measure outcomes is hindered by limited access to complete and meaningful data.

Recommendations for Executive Action

To enhance the effectiveness of efforts to implement SMS and maximize the positive impact of SMS implementation on aviation safety, we recommend that the Secretary of Transportation direct the FAA Administrator to take the following five actions: 1. To better evaluate the effectiveness of the agency’s efforts to implement SMS, develop a system to assess whether SMS meets its goals and objectives by identifying and collecting related data on performance measures. 2. To align strategic goals with employee efforts, develop a system to evaluate employees’ performance as it relates to SMS. 3. To better manage implementation, develop a system to track and report on SMS implementation across business lines. 4. To better leverage existing resources and facilitate SMS implementation, conduct a workforce analysis to inventory existing employee skills and abilities and develop strategies for addressing any SMS-related gaps identified. 5. To maximize the positive impact of SMS implementation on aviation safety, consider strategies to address airports’ concerns that may negatively affect data collection and data sharing, including asking Congress to provide additional protections for SMS data collected by public entities.

Agency Comments

We provided the Department of Transportation and NTSB with a draft of this report for review and comment. DOT and NTSB officials provided technical comments, which we incorporated as appropriate and DOT agreed to consider the recommendations. In addition, DOT officials stated there is a need for FAA to have a common hazard-tracking system. FAA has taken initial steps towards standardization by publishing FAA Order 8040.4A, Safety Risk Management Policy, which identifies terms and definitions used for safety risk management. DOT also reinforced its dedication to the success of SMS and noted its continued efforts to improve its implementation plans with a measured, structured approach to implementation.

We are sending copies of this report to the appropriate congressional committees, DOT, NTSB, and interested parties, and others. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me on (202) 512-2834 or at dillinghamg@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

Our objective was to assess the Federal Aviation Administration’s (FAA) implementation of Safety Management Systems (SMS) and provide information on potential implementation challenges. To do so, we addressed the following questions: (1) What is the status of FAA’s implementation of SMS? (2) To what extent have FAA’s SMS efforts been consistent with key practices for successful planning and implementation of a new program? (3) What challenges does FAA face in implementing SMS?

To perform our review, we focused primarily on FAA’s implementation of SMS for its business lines as well as its preliminary efforts to require and oversee SMS implementation by industry. We conducted background research to identify literature related to SMS in aviation, and any challenges that agencies might face when implementing SMS. We also attended parts of a safety risk management panel on runway status lights conducted by FAA’s Air Traffic Organization (ATO) at Seattle-Tacoma International Airport in March 2012 as a means of learning more about SMS and related processes. During the data collection and drafting phases of this report, FAA was in the midst of rulemaking efforts to require SMS of Part 121 air carriers and Part 139 airports, so we did not comment on any draft or proposed regulatory guidance.

To determine the status of FAA’s implementation of SMS, we reviewed FAA’s SMS orders and pilot project guidance, implementation plans, and Notices of Proposed Rulemaking for Part 121 air carriers and Part 139 airports. We also reviewed international and FAA guidance on SMS issued by the International Civil Aviation Organization (ICAO) and the Joint Planning and Development Office (JPDO), respectively, and National Transportation Safety Board (NTSB) recommendations to FAA related to SMS. Finally, we interviewed FAA SMS program managers across FAA business lines and offices; industry experts we identified based on their knowledge and experience in industry, recommendations from aviation industry officials, and a search of SMS literature; and ICAO and NTSB officials.

To assess the extent to which FAA’s efforts have been consistent with key practices, we reviewed our reports and other literature on successful project planning and implementation, particularly for large-scale transformative projects, and condensed the resulting list to eliminate duplication and overlap. To do this, we reviewed previous GAO reports that highlighted practices associated with successful planning and implementation of a new program. We removed or consolidated any duplicate items across the reports to create a single list of 10 criteria. We then identified FAA’s actions related to these practices by reviewing FAA guidance and agency documentation such as its SMS implementation plans, conducting interviews with FAA officials across its business lines, and using that information to assess the extent to which FAA had addressed each practice. We determined whether each key practice was addressed, partially addressed, or not addressed by using criteria developed for prior GAO reports. As such, we considered a practice “addressed” if FAA had instituted the practice; “partially addressed” if FAA had shown some progress toward instituting, or started but not completed the practice; and “not addressed” if FAA had made minimal or no progress toward instituting the practice. The team made these coding decisions together, with two analysts making initial judgments and team management reviewing and confirming them.

To identify challenges FAA faces in implementing SMS, we reviewed our prior work on long-standing FAA challenges, such as those related to training and data, and interviewed aviation industry experts and FAA officials mentioned above. We reviewed prior GAO work on performance measurement and workforce analysis, Department of Transportation Inspector General reports and NTSB recommendations related to SMS. To obtain industry views on challenges, we interviewed officials from selected airports and air carriers, industry associations representing airports, air carriers, and pilots, and individuals with SMS experience described above. We also reviewed and analyzed documents, including language in the FAA Modernization and Reform Act of 2012 related to data protection, and associated scholarly work. To supplement comments received from the individuals we interviewed, we also reviewed comments made by aviation stakeholders on the two Notices of Proposed Rulemaking related to SMS.

To obtain industry views on both SMS implementation practices and associated challenges, we interviewed officials from selected airports and air carriers, which we selected for diversity in size, location, participation in FAA SMS pilot projects, and submission of comments on FAA’s two Notices of Proposed Rulemaking related to SMS. (See table 1 for a list of selected airports.)

We also interviewed officials from six air carriers: Delta, GoJet, United, Pinnacle, Southwest, and US Airways. Finally, we interviewed officials with SMS knowledge and expertise, including experts from the Flight Safety Foundation, Embry-Riddle Aeronautical University, John A. Volpe National Transportation Systems Center, and MITRE Corporation.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Heather MacLeod (Assistant Director); Elizabeth Curda; Leia Dickerson; Sarah Farkas; David Hooper; Delwen Jones; Brooke Leary; Josh Ormond; Larry Thomas; and Elizabeth Wood made key contributions to this report.