Aviation Security: Federal Air Marshal Service Has Taken Actions to Fulfill Its Core Mission and Address Workforce Issues, but Additional Actions Are Needed to Improve Workforce Survey
Released: 2009-02-13T00:00:00

Letter

Background

Originally established as the Sky Marshal program in the 1970s to counter hijackers, the Aviation and Transportation Security Act expanded FAMS’s mission and workforce in response to the September 11, 2001, terrorist attacks and mandated the deployment of federal air marshals on high- security risk flights. Within the 10-month period immediately following September 11, 2001, the number of air marshals grew significantly. Also, during years following the 2001 attacks, FAMS underwent various organizational transfers. Initially, FAMS was transferred within the Department of Transportation from the Federal Aviation Administration to the newly created TSA. In March 2003, FAMS moved, along with TSA, to the newly established Department of Homeland Security. In November 2003, FAMS was transferred to U.S. Immigration and Customs Enforcement (ICE). Then, about 2 years later, FAMS was transferred back to TSA in the fall of 2005.

FAMS is one layer among multiple layers of aviation security. For example, prospective passengers are prescreened against applicable records in the Terrorist Screening Center’s consolidated watch list. Passengers and baggage are also physically screened. Air marshals generally are characterized as being the last line of defense within the layered aviation-security framework. In this regard, FAMS officials stressed that air marshals constitute the only in-flight security layer deployed on the basis of risk.

FAMS deploys thousands of federal air marshals to a significant number of daily domestic and international flights. In carrying out this core mission of FAMS, air marshals are deployed in teams to various passenger flights.

Such deployments are based on FAMS’s concept of operations, which guides the agency in its selection of flights to cover. Once flights are selected for coverage, FAMS officials stated that they must schedule air marshals based on their availability, the logistics of getting individual air marshals in position to make a flight, and applicable workday rules.

At times, air marshals may have ground-based assignments. On a short- term basis, for example, air marshals participate in Visible Intermodal Prevention and Response (VIPR) teams, which provide security nationwide for mass transit systems other than aviation. Also, air marshals participate in Joint Terrorism Task Forces led by the Federal Bureau of Investigation.

Good marksmanship is considered a necessity for air marshals, particularly given the unique environment of the core mission—the relatively tight confines of an airplane, coupled with the presence of numerous passengers (“bystanders”) and the possibility of air turbulence that creates an unstable “shooting platform” (see fig. 1). Thus, according to TSA, air marshals have the highest marksmanship standard in the federal government and also must be recertified on their firearm every quarter.

To preserve their anonymity on covered flights, air marshals are to blend in with other passengers by dressing appropriately and performing their duties discreetly without drawing undue attention.

The Federal Air Marshal Service’s Operational Approach to Achieving Its Core Mission Is Based on Risk-Related Factors

FAMS’s operational approach (concept of operations) is based on risk- related factors, such as assessments of threat, vulnerability, and consequences. FAMS is guided by the provisions of the Aviation and Transportation Security Act that specify the deployment of federal air marshals on flights presenting high-security risks, such as nonstop, long- distance flights targeted on September 11, 2001. FAMS seeks to maximize coverage of high-risk flights by establishing coverage goals for 10 targeted critical flight categories. In order to reach these coverage goals, FAMS uses a scheduling process to determine the most efficient flight combinations that will allow air marshals to cover the desired flights. FAMS management officials stressed, however, that the overall coverage goals and the corresponding flight schedules of air marshals are subject to modification at any time based on changing threat information and intelligence.

Risk-Related Factors Are Considered in Assigning Air Marshals to Flights

Following the attacks of September 11, 2001, FAMS developed a risk- based concept of operations for deploying air marshals on U.S. commercial passenger air carriers. Because there are many more U.S. air carrier flights each day than can be covered by air marshals, FAMS relies on the methodology outlined in its concept of operations to assign air marshals to flights with the highest security risks. Under this approach, FAMS considers the following risk-related factors to categorize each of the approximately 29,000 domestic and international flights operated daily by U.S. commercial passenger air carriers into risk categories—high risk or lower risk:

Threat (intelligence): Available strategic or tactical information affecting aviation security is considered.

Vulnerabilities: Although FAMS’s specific definition is deemed to be sensitive security information, DHS defines “vulnerability” as a physical feature or operational attribute that renders an entity open to exploitation or susceptible to a given hazard.

Consequences: FAMS recognizes that flight routes over certain geographic locations involve more potential consequences than other routes.

FAMS attempts to assign air marshals to provide an on-board security presence on as many of the flights in the high-risk category as possible. However, other considerations can make covering only high-risk flights impractical from a scheduling perspective and potentially predictable to an adversary. Therefore, for purposes of scheduling efficiency and adversary uncertainty, FAMS may deploy some air marshals on lower-risk flights.

Air Marshals Are Deployed on Flights to Meet Coverage Goals for Targeted Critical Flight Categories; the Concept of Operations Provides Flexibility for Responding to Changing Threats

FAMS has established a scheduling process intended to maximize the coverage of high-risk flights and meet the agency’s desired coverage goals for 10 targeted critical flight categories. FAMS’s Domestic Planning Branch (within the Systems Operation Control Division) is responsible for scheduling air marshals to domestic missions. During the course of a year, the Domestic Planning Branch must prepare schedules for 13 roster periods of 28 days each. According to FAMS officials, each 28-day schedule takes approximately 3 weeks to prepare. The Domestic Planning Branch prepares each domestic schedule using an automated scheduling tool. As part of the scheduling process, each FAMS field office is responsible for making available a specific percentage of their air marshals on a daily basis to cover targeted critical flights (both domestic and international flights) in the roster periods. FAMS utilizes the automated scheduling tool to determine the most efficient flight “pairings” of departure and return flights that will bring an air marshal back to his or her starting point and that would be within the parameters for mission assignment and rest for the air marshal. FAMS officials also perform other checks on the fairness or appropriateness of the schedules, such as ensuring that certain flights are not being covered repeatedly by the same air marshals.

FAMS officials noted that the schedules for deploying air marshals are altered as needed to cover specific, high-threat flights. For example, in August 2006, FAMS increased its coverage of international flights in response to the discovery, by authorities in the United Kingdom, of specific terrorist threats directed at flights from Europe to the United States. However, the officials added that a shift in resources of this type can have consequences because of the limited number of air marshals. FAMS officials noted that international missions require more resources than domestic missions partly because the trips are of longer duration.

An Independent Assessment Concluded That the Federal Air Marshal Service’s Approach for Achieving Its Core Mission Was Reasonable; Recommendations for Enhancing the Approach Are Being Implemented

In its 2003 PART review of FAMS, OMB concluded that an independent evaluation should be conducted to assess FAMS’s performance related to aspects of the agency’s concept of operations—particularly aspects involving flight coverage risk categories, the distribution of covered flights, and target levels of coverage. The Homeland Security Institute, a federally funded research and development center, performed this evaluation and issued a final report in July 2006. The report concluded that FAMS’s approach for achieving its core mission of providing an onboard security presence for flights, as detailed in the agency’s concept of operations, was reasonable and made several recommendations for enhancements. The Homeland Security Institute recommended, for example, that FAMS increase randomness or unpredictability in selecting flights and otherwise diversify the coverage of flights within various risk categories. As of October 2008, FAMS had implemented or had ongoing efforts to implement the recommended enhancements.

In a July 2006 report, the Homeland Security Institute specifically noted the following regarding FAMS’s overall approach to flight coverage:

FAMS applies a structured, rigorous approach to analyzing risk and allocating resources.

The approach is reasonable and valid.

No other organizations facing comparable risk-management challenges apply notably better methodologies or tools.

As part of its evaluation methodology, the Homeland Security Institute examined the conceptual basis for FAMS’s approach to risk analysis. Also, the institute examined FAMS’s scheduling processes and analyzed outputs in the form of “coverage” data reflecting when and where air marshals were deployed on flights. Further, the Homeland Security Institute developed and used a model to study the implications of alternative strategies for assigning resources. We reviewed the Homeland Security Institute’s evaluation methodology and generally found it to be reasonable.

In a 2008 PART reassessment of FAMS, OMB also reported that the Homeland Security Institute’s evaluation employed quality evaluation methods and was comprehensive in scope. Further, OMB noted that an interagency steering group—which was convened by the Homeland Security Institute and met in conference in April 2006—also had reviewed FAMS’s concept of operations and considered it to be reasonable. In addition to FAMS and Homeland Security Institute participants, the interagency steering group consisted of representatives from various law enforcement and counterterrorism agencies, which included the Federal Bureau of Investigation, U.S. Customs and Border Protection, the Transportation Security Administration, the Federal Aviation Administration, the Homeland Infrastructure Threat and Risk Assessment Center, U.S. Northern Command/North American Aerospace Defense command, and the National Counterterrorism Center.

In its July 2006 report, the Homeland Security Institute made several recommendations for enhancing FAMS’s approach for deploying air marshals on flights. As presented in table 1, FAMS had implemented or had ongoing efforts to implement all of the recommended enhancements, as of October 2008.

In reference to the core mission of FAMS, the Homeland Security Institute’s recommendations regarding two processes—the filtering process for selecting flights and the allocation process for assigning air marshals to flights—are particularly important. To address the institute’s recommendations, FAMS officials stated that a broader approach to filtering flights has been implemented—an approach that opens up more flights for potential coverage, provides more diversity and randomness in flight coverage, and extends flight coverage to a variety of airports.

The Homeland Security Institute’s ongoing work has also resulted in two reports delivered to FAMS in July 2008. One of the reports detailed the institute’s analysis regarding requirements for a next-generation-mission scheduling tool for FAMS, and the other report presented the institute’s benchmark analysis that compared FAMS’s workday rules and practices against those of similar occupations involving frequent air travel and the related operational challenges, including fatigue and other human factors. Also, in September 2008, the Homeland Security Institute provided FAMS a classified report assessing the deterrent effects of the agency’s approach to flight coverage. Further, based on its continuing work, the institute expects to provide FAMS one additional final report by the end of calendar year 2008—a report regarding potential enhancements to performance measures.

The Federal Air Marshal Service Has Taken Positive Actions to Address Issues Affecting Its Workforce and to Help Ensure Continued Progress

To identify and address issues affecting the ability of its workforce to successfully carry out its mission, FAMS has implemented various communication-oriented processes or initiatives—including 36 issue- specific working groups—that have produced some positive results. For instance, FAMS has revised and documented certain policies—including the policy related to aircraft check-in and boarding procedures—to better protect air marshals’ anonymity. In addition, FAMS has modified its mission scheduling processes and implemented a voluntary lateral transfer program to address certain issues regarding air marshals’ quality of life— and has plans to further address health issues associated with varying work schedules and frequent flying. As an additional initiative to help determine the effectiveness of management’s actions to address issues affecting air marshals, FAMS conducted a workforce satisfaction survey of all staff in late fiscal year 2007. A majority (79 percent) of the respondents to the survey indicated that there had been positive changes from the prior year, although the overall response rate (46 percent) constituted less than half of the workforce. The 46 percent response rate was substantially less than the 80 percent rate encouraged by OMB in its guidance for federal surveys that require its approval. According to the OMB guidance, a high response rate increases the likelihood that the views of the target population are reflected in the survey results. Obtaining a higher response rate to FAMS’s future surveys, which the agency plans to conduct every 2 years, and modifying the structure of some questions, could enhance the surveys’ potential usefulness by, for instance, providing a more comprehensive basis for assessing employees’ attitudes and perspectives. All 67 of the air marshals we interviewed in 11 field offices attributed progress under these efforts largely to the “tone at the top,” particularly the commitment exhibited by the former FAMS Director who served in his position from March 2006 to June 2008. To reinforce a shared vision for workforce improvements and sustain forward progress, the current FAMS Director has expressed a commitment to continuing applicable processes and initiatives.

The Federal Air Marshal Service Implemented Various Processes or Initiatives to Address Issues Affecting Its Workforce

Our prior work has shown that leading organizations commonly sought their employees’ input on a periodic basis—by, for example, establishing working groups or task forces, convening focus groups, and conducting employee satisfaction surveys—and used that input to adjust their human capital approaches. Starting in March 2006, the then-serving FAMS Director implemented several communication processes or initiatives to better understand and address issues facing the agency’s workforce. Chief among these processes or initiatives were issue-specific working groups established to study, analyze, and address a variety of issues ranging from mission, organizational, and operational topics to workforce satisfaction and quality-of-life concerns. Initially, based on his knowledge of issues facing the organization when he assumed the leadership position in March 2006, the FAMS Director established 12 working groups. Subsequently, based on feedback from these initial groups and other sources regarding issues of concern, the number of working groups expanded to 36 (see app. V).

Each working group typically included a special agent-in-charge, a subject matter expert, air marshals, and mission support personnel from the field and headquarters. FAMS management directed working group members to define each group’s purpose, analyze specific issues, develop short- and long-term recommendations and determine their financial feasibility. As a final product, FAMS management expected each working group to submit a report, including recommendations, to the FAMS executive staff for managerial consideration. According to FAMS management, the working groups typically disband after submitting a final report. FAMS management stressed, however, that applicable groups could be reconvened or new groups established as needed to address relevant issues.

In addition to the working groups, other processes or initiatives implemented by FAMS management to address workforce issues or otherwise improve management-workforce communication include the following:

Field office focus groups—Each of the 21 FAMS field offices organized a local focus group composed of representatives from the respective office’s air marshal squads and at least one mission support staff. All members serve on a rotating basis, and the groups are to meet at least quarterly to discuss issues of concern to the local workforce and bring these issues to the attention of the applicable field office’s special agent-in-charge.

Field office visits by the FAMS Director— In 2006, the FAMS Director began visiting field offices and holding informal gatherings with air marshals, outside the presence of local managers, to discuss their questions and concerns.

Listening sessions—FAMS senior management established forums to allow direct communication between FAMS senior management and various personnel. In 2006, the FAMS Director and Deputy Directors conducted these sessions weekly in headquarters and the field offices with a total of 10 to14 staff selected for each meeting. In 2007, this format changed from weekly to monthly sessions and included larger groups of FAMS personnel.

Dinners with the Director—In 2006, the FAMS Director began holding weekly dinners to meet with air marshals transiting through the Washington, D.C., area. These dinners provide an opportunity for air marshals to speak personally with the director about any questions or concerns. The FAMS Deputy Director and one assistant director also attend these dinners with selected air marshals.

Director’s e-mail in-box—FAMS established an e-mail in-box for agency personnel to provide feedback to the FAMS Director. At any time, air marshals—whether at headquarters, in a field office, or deployed on mission—can send their insights, ideas, suggestions, and solutions to the FAMS Director.

Anonymous Web site—FAMS established an internal Web site for agency personnel to provide anonymous feedback to FAMS management on any topic.

Ombudsman Position—FAMS management assigned an air marshal to the position of Ombudsman in October 2006. According to FAMS management, the Ombudsman provides confidential, informal, and neutral assistance to employees to address workplace-related problems, issues, and concerns. FAMS reported that, in fiscal year 2007 (the first full year of the position), the Ombudsman handled 67 cases, and, through the first three quarters of fiscal year 2008, an additional 54 cases.

FAMS officials estimated that, as of October 2008, more than one-fourth of the agency’s employees had participated in one or more of these activities which encompass the various working groups and other processes and initiatives.

Based on input provided by the working groups and information obtained through the other processes and initiatives, FAMS has taken or is planning to take actions to address issues that affect the ability of air marshals to carry out the agency’s mission. As discussed in the following sections, these actions address operational issues, such as check-in and boarding procedures that affect air marshals’ anonymity as well as quality-of-life and health issues.

To Help Preserve the Anonymity of Air Marshals, the Federal Air Marshal Service Revised Its Policy and Procedures for Check-In and Boarding of Aircraft

To preserve their anonymity on covered flights, federal air marshals are to blend in with other passengers by dressing appropriately and performing their duties discreetly without drawing undue attention. In past years, air marshals frequently asserted that the check-in and boarding policy and procedures established by FAMS compromised their anonymity by requiring repeated interactions with airline personnel. In September 2005, we reported that the full extent of incidents that air marshals encounter was unknown because FAMS lacked adequate management controls for ensuring that such incidents were recorded, tracked, and addressed.

Accordingly, to facilitate management of incidents that affect air marshals’ ability to operate discreetly during their missions, our September 2005 report recommended that FAMS take the following four actions:

Develop a means for recording all incidents reported to the Mission Operations Center that affect air marshals’ ability to operate discreetly and criteria for determining which incidents require federal air marshals to complete a mission report.

Develop a means for tracking and retrieving data on mission reports to enable FAMS to analyze and monitor reported and systemic incidents.

Establish written policies and procedures for reviewing and addressing reported incidents.

Establish a means for providing feedback on the status and outcome of FAMS mission reports to the federal air marshals who submit them.

FAMS has taken steps to address all four of these recommendations and also address the related feedback received from air marshals through various working groups and other initiatives. In October 2005, FAMS issued a written directive establishing policies and procedures for reporting and managing mission incidents. In November 2005, we reported that we had reviewed the directive and believed that it addressed two of our recommendations—the first and the third recommendations. More recently, in March 2008, FAMS issued an addendum to its written directive establishing a means for providing feedback on the status and outcome of FAMS mission reports to the federal air marshals who submit them (fourth recommendation).

Additionally, FAMS revised its policy and procedures regarding interaction with airline personnel during the check-in and boarding processes in order to better protect the anonymity of air marshals in mission status. To help ensure effective implementation, the new policy and procedures have been incorporated into TSA’s Aircraft Operator Standard Security Program, which specifies requirements that domestic passenger air carriers must implement as part of their TSA-approved security programs. According to FAMS officials, the recent update constitutes the first time that the Aircraft Operator Standard Security Program guidance specifically includes a section regarding the boarding of federal air marshals.

Through use of a database created in fiscal year 2006 to track mission incidents, FAMS senior executive staff noted that analysis and monitoring are conducted daily of reported incidents, including those that could compromise the ability of air marshals to operate discreetly (second recommendation). The first management report detailing overall incident patterns and trends was produced in July 2008. Going forward, FAMS officials stated that reports would be produced quarterly to allow management to review patterns or trends regarding mission incidents and the effectiveness of the new policy and procedures.

The Federal Air Marshal Service Modified Its Dress Code and Hotel Policies to Further Protect Air Marshals’ Anonymity

To further protect the anonymity of air marshals while on missions, and in response to air marshals’ feedback and the working groups’ recommendations, FAMS management revised the dress code policy and the hotel policy for air marshals in August 2006 and February 2007, respectively. The revisions allow air marshals greater discretion in selecting appropriate attire to wear on missions and choosing hotels for overnight trips. Before the revisions, air marshals reported that the dress code policy was too restrictive and forced them to dress too formally for certain flights, such as those to vacation-oriented destinations. According to the air marshals, this restrictive policy resulted in their standing out from the other passengers, a situation that compromised their anonymity. Similarly, before being revised, FAMS’s hotel policy directed air marshals to stay at certain hotels on overnight missions so that they could be located easily by management in an emergency. Additional considerations of FAMS management for restricting the hotel selection were to ensure that air marshals were able to stay at hotels within per diem rates and would have ready access to transportation between the hotel and the airport. Air marshals expressed concerns that repeatedly staying at the same hotels risked exposing their anonymity. The revised policy allows air marshals to select their own hotels, provided the hotels are within per diem rates and have adequate transportation options. To alleviate concerns of FAMS management about being able to contact air marshals in an emergency, the revised policy requires air marshals to report their hotel locations via the FAMS intranet.

All 67 of the air marshals we interviewed in the 11 field offices we visited said that the revised dress code and hotel policies adequately addressed their concerns.

The Federal Air Marshal Service Is Planning to Replace Its Inventory of Personal Digital Assistant Communication Devices by Early Fiscal Year 2009

FAMS has described the agency’s personal digital assistant (PDA) communication device as being a lifeline for air marshals. The current device carried by air marshals is intended to function as a cell phone and personal computer and allow users to place phone calls, access the Internet, send e-mails, pull up basic Microsoft Word documents, store documents, and submit reports. However, the findings of FAMS’s applicable working groups indicated that the current PDA communication device has proven unreliable. Similarly, all 67 of the air marshals we interviewed in 11 field offices stated that they had experienced problems with their PDA device while on missions. Examples of problems reported by air marshals included dropped calls or lost signals in certain geographical areas, limited audio quality and durability, and lack of ability to send certain required documents (such as time and attendance reports). Another reported problem was the frequent freezing or locking of the PDA device, which then necessitated use of a cumbersome reset process. As a result of such problems, air marshals reported that the PDA device has hindered their ability to communicate effectively with management while in mission status. Additionally, the air marshals we interviewed commented that the current PDA device is relatively large and bulky, which potentially contributes to loss of anonymity.

In response to air marshals’ feedback and the working groups’ recommendations, FAMS is taking steps to procure new PDA communication devices and distribute them to air marshals. Furthermore, according to FAMS officials, the procurement contract for the new PDA devices will provide for a 2-year replacement cycle. In the interim, to improve voice communication capabilities pending arrival of the new devices, FAMS officials reported that it issued new cell phones to air marshals in June 2008. The officials noted, however, that air marshals still must rely on the current PDA device for non-voice functions, such as sending and receiving e-mail messages and documents, until the new PDA devices are available.

To Address Quality-of-Life and Health Issues, the Federal Air Marshal Service Has Implemented Changes to Mission Scheduling and Enlisted the Assistance of Outside Experts

In reference to quality-of-life and health issues, mission scheduling constitutes the most significant concern of air marshals, according to feedback that FAMS management received from working groups and other communication processes and initiatives. To be fully effective, air marshals must be healthy, fit, and alert. However, FAMS’s Medical Issues Working Group reported that air marshals have experienced various types of health issues—poor physical fitness as well as musculoskeletal injuries and upper respiratory infections—that may potentially be attributable to frequent flying and the overall nature of their jobs. The working group noted various challenges to ensuring that air marshals have adequate sleep, exercise, and recovery time. A contributing factor noted is that the agency’s automated scheduling tool historically has lacked the capability to consistently program an air marshal’s daily start and end times throughout a roster period, which makes normal sleep patterns difficult to maintain and often results in fatigue. For instance, an air marshal may have been scheduled to begin some days at 5 a.m. and other days at 10 a.m., with unpredictable ending times because of flight delays. In addition to inconsistent shifts, the Medical Issues Working Group noted that air marshals are subject to long hours—including arriving home late on a Friday and then having to depart early the following Monday morning. These types of schedules, according to the working group, make allowing adequate time for workouts and maintaining healthy eating habits difficult and also limit the amount of time available to take care of family and personal needs.

To address these scheduling issues, FAMS has implemented or is planning to implement various changes:

Mission exchange program—This program, which FAMS initially piloted in 2006 and is now available to all 21 field offices, allows air marshals within the respective field office to exchange mission days based on a demonstrated need, such as medical issues or family- related issues. For instance, an air marshal with an 8 a.m. mission start time and a 9 a.m. medical appointment could exchange shifts with another air marshal for a later mission start time. The program is intended to reduce the amount of unscheduled leave taken by air marshals and otherwise mitigate the hardships or other effects associated with FAMS’s current policy of requiring air marshals to submit requests for annual leave 38 to 66 days in advance.

Preset ending time and 60-hour rule—In September 2006, FAMS instituted a change to its mission-scheduling policy. The change is designed to help ensure that air marshals complete their mission flights by a preset time on the day before a regular day off (or the day before scheduled annual leave) and not begin a new mission until receiving a minimum of 60 hours rest. For example, if an air marshal’s regular days off are Saturday and Sunday, and this individual’s mission ended on Friday evening, the next mission assignment (on Monday) would begin no earlier than Monday morning.

Limit on number of flight days—In April 2007, FAMS implemented another change in mission-scheduling policy designed to distribute flight days equitably and improve the balance between work and personal life for air marshals. Specifically, under the new policy, each air marshal’s total flight days are targeted to not exceed 18 days per roster period and 200 days annually.

More rest time after completing extended international missions—Also in April 2007, FAMS issued guidance to field offices to make every attempt at increasing rest time for air marshals after completing an extended international mission. Under this guidance, air marshals returning from an international mission are to be given a non-flight day as their next duty day when any one of the following three conditions apply: (1) the return flight exceeds 10 hours in the air, (2) the flight crossed the international date line, or (3) the overall mission (round- trip flights plus overnight stays) was 4 days or longer in duration. Depending on an air marshal’s schedule, a non-flight day could be a training day, a regular day off, or a non-mission status day.

More consistent start times—FAMS is currently developing a modification to its scheduling tool to provide a consistent, defined scheduling window (encompassing, for example, 3 hours) for air marshals in mission status to report for duty during a 7-day period. Under the planned modification, for instance, FAMS schedulers would assign an air marshal to flights departing during 5:00 a.m. to 8:00 a.m., 9:00 a.m. to 12:00 p.m., or other 3-hour window during the week. FAMS officials stated that this modification, which is intended to provide more consistent start times for each air marshal throughout the applicable week, should be completed and ready for pilot testing by the middle of calendar year 2009.

The 67 air marshals we interviewed in 11 field offices generally expressed satisfaction with the various enhancements to mission scheduling, although most (43) mentioned that implementation of the mission exchange program was still evolving.

To more specifically address the health implications of flight scheduling, several efforts were recently completed or are planned. For instance, the Homeland Security Institute conducted a benchmark analysis and assessment of fatigue issues related to air marshals and issued a report to FAMS in July 2008. In its analysis, the institute compared FAMS’s workday rules against other occupations—largely in the aviation realm— that face challenges involving frequent travel, jet lag, long work hours, rotating shifts, and the stress of maintaining a schedule across multiple flights and airports. The Homeland Security Institute noted that although no other occupation is identical to that of air marshals, meaningful comparisons were made with similar occupations, such as commercial airline pilots and cargo pilots and law enforcement officers working in aviation (e.g., U.S. Marshals Service aviation enforcement officers responsible for transporting prisoners). In its July 2008 report, the Homeland Security Institute noted that while stress and fatigue issues are a part of all organizations and cannot be entirely eliminated, air marshals are provided considerable blocks of rest within their schedules, when assessed against similar occupations. Overall, the institute reported that the results of the benchmark analysis showed that air marshals are provided above-average time to recuperate from duty days.

Further, the institute noted that FAMS has taken various steps, including implementation of the mission exchange program, to improve aspects of mission scheduling. In addition, in October 2008, FAMS officials informed us that the agency has funded a contract with the National Institute of Justice to implement FAMS-specific research regarding mission scheduling, work-rest cycles, fatigue, and performance.

To Further Address Quality-of-Life Issues, the Federal Air Marshal Service Has Implemented a Voluntary Lateral Transfer Program

According to FAMS officials, air marshals frequently cited the need for a voluntary lateral transfer program during listening group sessions and dinners with the director. FAMS working groups that examined quality-of- life issues also reported that the agency would benefit from implementing a transfer program for air marshals to express interest in relocating to another field office. Thus, in October 2006, FAMS management implemented a voluntary lateral transfer program. Under the program, an air marshal in good standing may request a transfer for up to three field offices, ranked by order of preference, and FAMS management is to make decisions based on the number of vacancies in each office and the seniority of air marshals who apply for transfer.

In December 2006, FAMS announced that 176 air marshals had been selected, during the first phase of the program, for transfer to new offices within 60 days. In the second phase, which occurred in the spring of 2007, FAMS management made transfer offers to 40 air marshals—all of whom accepted. In the third and most recent phase, which occurred in the spring of 2008, FAMS management made offers to 48 air marshals—of whom 45 accepted. FAMS expects to continue offering voluntary transfer opportunities during open seasons in the spring of each year.

The Federal Air Marshal Service Conducted a Workforce Satisfaction Survey in 2007; the Potential Usefulness of Future Surveys Could Be Enhanced by Improving the Response Rate and Redesigning Certain Questions

In late fiscal year 2007, FAMS conducted a workforce satisfaction survey of all staff—not just air marshals—to help determine issues affecting the ability of agency personnel to perform their jobs—and to obtain feedback on the effectiveness of measures already taken by management to address relevant issues. The 2007 survey questionnaire consisted of a total of 60 questions that covered 13 topics—senior leadership; supervisor/management; resources and technology; training and education; career development; policies and procedures; employee involvement and autonomy; rewards and recognition; communication; safety, health, and medical issues; work and family life; organizational commitment; and job satisfaction. According to FAMS management officials, the survey provided useful information on quality-of-life and other issues affecting the ability of air marshals and other agency personnel to perform their jobs. In addition, the officials reported that survey results indicated that employees generally were pleased with the policy changes and other actions implemented by management to address relevant issues. For example, although the 2007 workforce satisfaction survey had an overall response rate (46 percent) that constituted less than half of the FAMS workforce, 79 percent of the respondents indicated that there had been positive changes from the prior year. Regarding future plans, FAMS expects to administer a workforce satisfaction survey every 2 years. FAMS officials stated that a purpose of the initial workforce satisfaction survey was to establish a baseline for use in comparing the results of future surveys.

In reviewing the 2007 survey’s implementation and results, we made several observations that are important for enhancing the potential usefulness of future surveys. First, as noted previously, the overall response rate was 46 percent. FAMS officials expressed satisfaction with this response rate given the highly mobile nature of their workforce. The FAMS officials also noted that the 46 percent response rate was similar to the response rates for other federal workforce satisfaction surveys. However, the 46 percent response rate was substantially less than the 80 percent rate OMB encourages for federal surveys that require its approval. Although internal workforce surveys such as the one conducted by FAMS do not require OMB approval, we believe the OMB standards and guidance provide relevant direction on planning, designing, and implementing high- quality surveys—including the need to obtain a high response rate to increase the potential that survey responses will accurately represent the views of the survey population.

Specifically, the OMB guidance stipulates that agencies must design surveys to achieve the highest practical rates of response to ensure that the results are representative of the target population and that they can be used with confidence as input for informed decision-making. OMB encourages agencies to obtain at least an 80 percent response rate, and its guidance states that response rates are an important indicator of the potential for nonresponse bias, which could affect the accuracy of a survey’s results. For instance, survey estimates may be biased if the individuals who choose to participate (respondents) differ substantially and systematically in some way from those who choose not to participate (nonrespondents). In general, a higher response rate increases the likelihood that any bias problem is decreased, resulting in the views and characteristics of the target population being more accurately reflected in the survey’s results. Thus, for any federal survey that must be approved by OMB, applicable guidelines stipulate that an analysis for possible nonresponse bias must be conducted if the final response rate is less than 80 percent.

Regarding the 46 percent response rate for the 2007 survey, FAMS management reported that an analysis of potential nonresponse bias was conducted by comparing various demographic data provided by the respondents to the FAMS workforce as a whole. Based on the analysis of the available demographic data, FAMS concluded that nonresponse bias did not exist as the respondents were representative of the entire workforce. Although the analysis conducted by FAMS was a useful effort, the potential for a nonresponse bias still exists given that over half of the FAMS workforce did not respond to the survey.

As noted previously, concerns about nonresponse bias could be avoided or mitigated by obtaining a higher response rate. FAMS employees were given 3 weeks (August 23 through September 14, 2007) to complete the 2007 workforce satisfaction survey. According to FAMS management, even though all employees (not just nonrespondents) were sent four messages reminding them of the deadline for completing the voluntary survey, the final overall response rate was 46 percent. We believe that other widely acknowledged methods, outlined in OMB guidance, could improve the response rate of future FAMS surveys. These methods include, for example, promoting awareness of the survey through outreach efforts with groups of prospective respondents and extending the cut-off date for responding to the survey. Also, monitoring questionnaire returns and targeting extra follow-up efforts to air marshals in particular field locations that have comparatively low response levels could help.

Additional observations we made in reviewing the 2007 workforce satisfaction survey’s questionnaire involve the sentence structure of certain questions and the response options. Generally, any question that combines two or more issues—but does not provide for separate or respective answers—can cause uncertainty about how to respond if the answer to each issue is different. Table 2 lists the seven 2007 workforce satisfaction survey questions that used these types of sentence structures. For instance, regarding the senior leadership of FAMS, question 3 cites two concepts (“visions” and “initiatives”), as well as two actions (“shared” and “supported”) associated with these concepts. However, the response options did not account for the fact that experiences could be different with each of these concepts and actions. Similarly, question 10 addresses the reliability of equipment used by agency personnel and cites four different devices. However, the response options did not account for the fact that experiences could be different with each of these devices.

Also, none of the 60 questions in the 2007 workforce satisfaction survey provided for response options such as “not applicable” or “no basis to judge”—responses that would be appropriate when respondents had little or no familiarity with the topic in question. Not providing response options such as “not applicable” or “no basis to judge” could lead to potentially misleading question responses. In the interest of being compliant, respondents might be compelled to give a response, such as “neutral,” to a question when they actually have no opinion due to either non- applicability or lack of familiarity with the question topic. While it might be assumed that all individuals being surveyed should be familiar with the topic of all questions, this might not be the case and will not be known unless the questionnaire contains the relevant response options. For example, question 39 (see app. VI), reads as follows: “I am satisfied that the work-related concerns I address with management are addressed appropriately.” As written, this sentence assumes that every employee has raised work-related concerns with management. If a respondent had never expressed work-related concerns with management, this individual might not know how to respond, given the question’s existing response options.

Thus, based on the sentence structure of certain questions and the response options, the results from the 2007 survey may provide an incomplete assessment of employees’ perspectives and attitudes to FAMS management. Regarding our observations on the design of survey questions and response options, FAMS officials stated that limited personnel resources precluded investing more time in development of the survey questionnaire and that the survey had served a useful purpose in providing information on issues of concern to be more fully explored through other communication processes or initiatives. Nonetheless, in developing future survey instruments, designing questions to avoid these types of ambiguities could provide FAMS management with information that is more focused and complete. Although we recognize that FAMS has a variety of other processes and initiatives—in addition to the customer satisfaction survey—for identifying and addressing workforce issues, customer satisfaction surveys can be particularly useful given that they are distributed to all employees and provide for anonymity of respondents. Further, the design considerations that we discussed involve relatively minor technical aspects that could be addressed with a minimal investment of personnel resources.

The Federal Air Marshal Service Has Committed to Addressing Workforce Issues by Sustaining Applicable Processes and Initiatives

As highlighted in our prior work, agency leaders in best practice organizations view people as an important enabler of agency performance and recognize the need for sustained commitment to strategically manage human capital. In developing approaches to managing the workforce, leaders of best practice agencies seek out the views of employees at all levels. Involving employees in the planning process helps agencies to develop goals and objectives that incorporate frontline insights and perspectives about operations. Further, such involvement can also serve to increase employees’ understanding and acceptance of organization goals and objectives and improve motivation and morale. Our work has shown that leading organizations commonly sought their employees’ input on a periodic basis and used that input to adjust their human capital approaches. Among other means, the organizations collected feedback by convening focus groups, providing opportunities for employees to participate in working groups or task forces, and conducting employee satisfaction surveys.

As discussed earlier in this report, FAMS has implemented a variety of processes and initiatives to address workforce issues by soliciting the views of front-line staff across the agency. Several key improvements in FAMS policies and procedures have resulted from these efforts. Among other improvements, for example, FAMS amended its policy for flight check-in and boarding procedures to better ensure the anonymity of air marshals in mission status. Also, the various processes and initiatives have helped to improve agency morale, according to the federal air marshals we interviewed. Moreover, agency officials noted that the processes and initiatives represented a significant commitment in management time and resources.

In our view, fostering continued progress in addressing workforce issues at FAMS is important. The current FAMS Director, after being designated in June 2008 to head the agency, issued a broadcast message to all employees, expressing a commitment to continue applicable processes and initiatives, including the working group process, listening sessions, field office visits, and the internal Web site for agency personnel to provide anonymous feedback to management on any topic. More recently, in response to our inquiry, FAMS’s Chief of Staff reported in October 2008 that the various communications processes and initiatives “have become an institutionalized and positive aspect” of the agency’s culture. Also, the Chief of Staff noted that FAMS was in the process of establishing an agencywide national advisory council—with representatives from headquarters and all field offices—to further enhance communication and outreach efforts, promote greater job satisfaction, and improve organizational effectiveness through cooperative problem solving and replication of best practices.

Conclusions

Federal air marshals are an important layer of aviation security. Thus, it is incumbent upon FAMS management to have sound management processes in place for identifying and addressing the challenges associated with sustaining the agency’s operations and addressing workforce quality-of-life issues. FAMS, to its credit, has established a number of processes and initiatives—including a workforce satisfaction survey—to address various operational and quality-of-life issues that affect the ability of air marshals and other FAMS personnel to perform their aviation-security mission. Consistent with the human capital practices of leading organizations, the current FAMS Director has expressed a commitment to continuing relevant processes and initiatives for identifying and addressing workforce concerns, maintaining open lines of communications, and sustaining forward progress. Although the workforce satisfaction survey is only one of a number of processes or initiatives used by FAMS to identify and address workforce issues, such surveys play an important role given their agencywide scope and the provision for anonymous responses. A higher response rate and more clearly structured questions and response options could add to the usefulness of this effort.

Recommendation for Executive Action

To facilitate continued progress in identifying and addressing issues that affect the ability of FAMS personnel to perform the agency’s aviation- security mission, we recommend that the FAMS Director take appropriate actions to increase the usefulness of the workforce satisfaction surveys that FAMS plans to conduct biennially. Such actions could include, for example, ensuring that the survey questions and the answer options are clearly structured and unambiguous and that additional efforts are considered for obtaining the highest possible response rates.

Agency Comments

We provided a draft of our restricted report for comment to the Department of Homeland Security and TSA. In November 2008, in written comments, the Department of Homeland Security and TSA agreed with our recommendation and noted that FAMS was in the initial stages of formulating the next workforce satisfaction survey, which included plans to implement the recommendation. Also, the Department of Homeland Security and TSA commented that our key findings and recommendation will facilitate continued progress in identifying and addressing issues that affect the ability of FAMS personnel to perform the agency’s aviation security mission. The full text of the department’s and TSA’s written comments is reprinted in appendix VII.

As arranged with your office, we plan no further distribution until 30 days after the date of this report. At that time, we will send copies of this report to interested congressional committees and subcommittees. We will also make copies available to others upon request.

If you or your staff have any questions about this report or wish to discuss the matter further, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other key contributors to this report are listed in appendix VIII.

Appendix I: Objectives, Scope, and Methodology

Objectives

This report addresses the following three principal questions:

What is the Federal Air Marshal Service’s operational approach for achieving its core mission of providing an onboard security presence for flights operated by U.S. commercial passenger air carriers?

To what extent has the Federal Air Marshal Service’s operational approach for achieving its core mission been independently assessed?

To what extent does the Federal Air Marshal Service have processes and initiatives in place to address issues that affect the ability of its workforce to carry out its mission?

Scope and Methodology

Initially, to obtain contextual and overview perspectives regarding the principal questions, we reviewed information available on the Web sites of relevant federal entities—the Department of Homeland Security, the Transportation Security Administration (TSA), and the Federal Air Marshal Service (FAMS). To obtain additional perspectives regarding FAMS’s mission and operations—and issues affecting its workforce—we conducted a literature search to identify relevant reports or studies and other publicly available information, including news media articles. In particular, we focused on reviewing congressional studies, Inspector General reports, and our previous reports. These included the following:

U.S. House of Representatives, Committee on the Judiciary, Plane Clothes: Lack of Anonymity at the Federal Air Marshal Service Compromises Aviation and National Security (Washington, D.C.: May 25, 2006).

GAO, Aviation Security: Federal Air Marshal Service Could Benefit from Improved Planning and Controls, GAO-05-884SU (Washington, D.C.: Sept. 29, 2005). The report is restricted (not available to the public) because it contains sensitive security information. The public version of the report is GAO-06-203 (Nov. 28, 2005).

U.S. Department of Homeland Security, Office of Inspector General, Review of Alleged Actions by Transportation Security Administration to Discipline Federal Air Marshals for Talking to the Press, Congress, or the Public, OIG-05-01 (Washington, D.C.: Nov. 2004).

GAO, Budget Issues: Reprogramming of Federal Air Marshal Service Funds in Fiscal Year 2003, GAO-04-577R (Washington, D.C.: Mar. 31, 2004).

GAO, Aviation Security: Federal Air Marshal Service Is Addressing Challenges of Its Expanded Mission and Workforce, but Additional Actions Needed, GAO-04-242 (Washington, D.C.: Nov. 19, 2003).

This report is the public version of a restricted report that we provided to congressional requesters in December 2008. Further details about the scope and methodology of our work regarding each of the three principal questions are presented in the following sections, respectively.

The Federal Air Marshal Service’s Operational Approach for Achieving Its Core Mission

In addressing this topic, we reviewed relevant legislation regarding FAMS’s mission and organizational structure. In particular, we reviewed a provision of the Aviation and Transportation Security Act that requires the deployment of federal air marshals on passenger airline flights and specifically requires the deployment of federal air marshals on every flight determined to present high security risks.

We analyzed FAMS documentation regarding the agency’s strategy and concept of operations for carrying out its mission. Also, we reviewed the results of an evaluation conducted in 2003 by the Office of Management and Budget (OMB), which utilized its Program Assessment Rating Tool (PART) to assess the management and performance of FAMS and concluded that key aspects of program design needed to be independently assessed. Further, we reviewed the follow-on PART-related reassessment of FAMS that OMB conducted in 2008 (see app. II).

Independent Assessment of the Federal Air Marshal Service’s Operational Approach

We reviewed the July 2006 classified report prepared by the Homeland Security Institute based on its independent evaluation of FAMS’s concept of operations. Our engagement team included a social science analyst and an economist with experience in risk assessment, who used generally accepted social science research standards in reviewing the institute’s report. Also, we interviewed applicable Homeland Security Institute officials to enhance our understanding of the evaluation’s scope, methodology, findings, and recommendations. Based on our review and discussion, we determined this report to be sufficiently reliable for the purposes of our work. Further, we reviewed FAMS documentation—and interviewed the Director of FAMS and other senior officials at the agency’s headquarters—regarding the status of efforts to address recommendations made by the Homeland Security Institute and any related initiatives involving strategic planning and the agency’s concept of operations.

We also reviewed two additional Homeland Security Institute reports, which FAMS provided to us in September 2008. One of the reports detailed the institute’s analysis regarding requirements for a next-generation mission scheduling tool for FAMS, and the other report presented the institute’s benchmark analysis that compared FAMS’s workday rules and practices against those of similar occupations involving frequent air travel and the related operational challenges, including fatigue and other human factors.

The Federal Air Marshal Service’s Processes and Initiatives for Addressing Issues That Affect Its Workforce

Regarding operational or tactical issues as well as quality-of-life issues that affect the ability of air marshals to carry out the agency’s mission, we reviewed published reports, including our September 2005 report (GAO-05-884SU) as well as news media accounts of relevant issues. We also reviewed FAMS documentation regarding various working groups (see app. V) and other initiatives that FAMS had established to address issues that affect the ability of air marshals to carry out the agency’s mission. In particular, we reviewed the final report (if available) produced by the respective working group. For criteria in reviewing the agency’s documentation regarding these efforts, we drew on our prior work regarding leading organizations and the best practices for strategically managing human capital.

Further, we interviewed the Director of FAMS and other senior officials at agency headquarters, and we visited 11 of the agency’s 21 field offices, where we interviewed managers and a total of 67 air marshals. We selected the 11 field offices and the 67 air marshals based on nonprobability sampling. In selecting the 11 field offices, we considered various factors, such as geographic location of the offices and the involvement of local management in agencywide working groups to address issues affecting air marshals.

At each of the 11 field offices, we first reviewed available work-related information about individual air marshals, such as their starting dates with FAMS and their involvement in ground-based assignments or any agencywide working groups. Based on these factors, we selected and interviewed 6 to 7 air marshals at each of the 11 field offices. Specifically, we selected 6 air marshals at each of 10 field offices and 7 air marshals at the remaining office. Our selections were made to encompass a variety of experience levels. Also, at each field office, rather than meeting separately with each individual, we conducted the interviews of the selected air marshals in group settings to encourage a wide array of perspectives, whether corroborating or contradictory. We conducted our interviews at the field offices during a 7-month time period, July 2007 through January 2008.

Because we selected a nonprobability sample of FAMS field offices to visit and air marshals to interview, the information we obtained in these visits and interviews cannot be generalized either to all 21 field locations or to all air marshals in the offices we visited. However, the visits and interviews provided us a broad overview of issues important to air marshals.

We reviewed documentation regarding the implementation and results of a workforce satisfaction survey that FAMS conducted in 2007. Our engagement team, which included social science analysts with extensive survey research experience, reviewed the questionnaire used in the survey for clarity and the related response options for appropriateness (see app. VI). Also, we discussed with FAMS officials the extent to which efforts were made to obtain an overall response rate as high as possible. As criteria to guide our review of the survey results, we used the following OMB guidance:

Standards and Guidelines for Statistical Surveys (September 2006).

Questions and Answers When Designing Surveys for Information Collections (Jan. 20, 2006).

We conducted this performance audit from April 2007 to December 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: OMB’s Program Assessment Rating Tool Results in 2003 and 2008 for the Federal Air Marshal Service

The Office of Management and Budget’s (OMB) Program Assessment Rating Tool (PART) consists of a standard series of 25 questions intended to determine the strengths and weaknesses of federal programs. The 25 total questions cover 4 broad topics—(1) program purpose and design, (2) strategic planning, (3) program management, and (4) program results/accountability.

This appendix provides an overview of OMB’s PART-based assessments of the Federal Air Marshal Service (FAMS) conducted in 2003 and 2008. Additionally, for each of the 25 questions used in the PART-based assessments, the appendix compares OMB’s 2003 and 2008 answers and summarizes OMB’s narrative findings (explanation and evidence). Also, when the answers in 2003 and 2008 differed for a particular question, the appendix briefly explains the basis for the respective answer.

More detailed information regarding the 2003 and 2008 OMB PART assessments of FAMS can be found on OMB’s Web site: www.ExpectMore.gov.

Overview of the 2003 and 2008 Assessments of the Federal Air Marshal Service

OMB’s PART assessments of federal agencies provide performance ratings that indicate how effectively tax dollars are spent. Following an assessment, OMB assigns an agency one of five possible overall ratings:

Effective. Programs rated “effective” set ambitious goals, achieve results, are well-managed, and improve efficiency.

Moderately effective. A “moderately effective” rating indicates a program that sets ambitious goals and is well-managed but needs to improve its efficiency or address other problems in the programs’ design or management in order to achieve better results.

Adequate. An “adequate” rating describes a program that needs to set more ambitious goals, achieve better results, improve accountability, or strengthen its management practices. Ineffective. An “ineffective” rating indicates a program that fails to use tax dollars effectively and is unable to achieve results because of a lack of clarity regarding the program’s purpose or goals, poor management, or some other significant weakness.

Results not demonstrated. A “results not demonstrated” rating indicates that a program has been unable to develop acceptable performance goals or collect data to determine whether it is performing.

In OMB’s 2003 PART assessment, FAMS received a rating of “results not demonstrated” because at that time FAMS did not have measurable results. Additionally, OMB cited strategic planning deficiencies that included the absence of baselines, targets, and time frames associated with performance goals and performance measurements. OMB further noted the absence of a second long-term outcome measure, proxy measures with respect to deterrence, and an efficiency measure.

In OMB’s 2008 PART assessment, FAMS received a rating of “moderately effective.” Regarding the improved rating, OMB recognized the Homeland Security Institute’s independent evaluation, which endorsed FAMS’s concept of operations. Also, OMB noted that FAMS had addressed other deficiencies by developing the following performance measures:

A second long-term outcome measure—the level of public confidence in air marshals’ ability to promote aviation security—which is reflective of FAMS’s purpose.

Proxy measures of deterrence, such as air marshals’ average annual rate of accuracy in firearms requalification testing.

Efficiency measures, such as (a) cost per flight per air marshal and (b) percentage of air marshals meeting the targeted number of flying days per year.

Section-by-Section Details of the 2003 and 2008 Assessments of the Federal Air Marshal Service

The results of PART’s 25 questions in reference to the 2003 and 2008 assessments of FAMS are presented in tables 3 through 6, specifically:

The 5 questions in table 3 cover program purpose and design.

The 8 questions in table 4 cover strategic planning.

The 7 questions in table 5 cover program management.

The 5 questions in table 6 cover program results/accountability.

As presented in tables 3 through 6, the narrative discussion (explanation and evidence) is our summary of OMB’s key points. If needed for purposes of clarifying the respective topic or ensuring accuracy, we used additional or alternative wording to summarize OMB’s findings. Also, in a few instances, we updated the information as appropriate.

Appendix III: Excerpts from the Homeland Security Institute’s July 2006 Report

The Homeland Security Institute, a federally funded research and development center, was established to assist the Department of Homeland Security in addressing relevant issues that require scientific, technical, and analytical expertise. The institute—after conducting an evaluation of the Federal Air Marshal Service’s approach for achieving the agency’s core mission of providing an onboard security presence for flights operated by U.S. commercial passenger air carriers—issued its final report in July 2006.

This appendix presents quoted excerpts that substantially replicate the executive summary in the Homeland Security Institute’s July 2006 report.

Background and Tasking for an Independent Evaluation

“The Federal Air Marshal Service (FAMS) challenge to reduce risk in the aviation domain is daunting. U.S. commercial passenger carriers make roughly 28,000 domestic and international flights each day. These flights canvas the globe and originate, terminate, or fly in proximity to thousands of critical facilities. The FAMS must evaluate which flights it will defend and to what extent. It cannot cover every flight. “In response to an Office of Management and Budget (OMB) direction, the FAMS asked HSI for an independent evaluation of its methods for analyzing risk and allocating resources. In particular, it asked HSI to determine if its risk management processes and the application of its concept of operations (CONOPS) to scheduled commercial flights were valid.”

Evaluation Approach Used by the Homeland Security Institute

“We defined ‘validation’ as a test of whether or not the FAMS risk management processes and the outcome of those processes are reasonable and consistent externally with stated guidance and internally with its own CONOPS. Our analysis involved three tasks. First, we examined the conceptual basis for the FAMS approach to risk analysis. Second, we examined the FAMS scheduling process and analyzed the output of that process in the form of ‘coverage’ data, i.e., when and where air marshals were deployed on flights. Third, we developed and employed a basic quantitative model to study the implications of alternative strategies for assigning resources.”

Findings Reported by the Homeland Security Institute

“Based on our analysis, we find that the FAMS applies a valid approach to analyzing risk and allocating resources. In particular, its approach is reasonable given the scarcity of resources and the guidance it has received. It assesses risk as a function of threat, vulnerability, and consequence and employs a filtering process along with an allocation tool to optimize resource allocation. Moreover, the FAMS seeks to strengthen risk management processes by improving its scheduling tools and analytical techniques. We did not find any other organizations that face a similar challenge and apply significantly better methodologies or tools.”

Additional Considerations Reported by the Homeland Security Institute

“During our analysis, we identified five issues that the FAMS should address itself or in conjunction with the Transportation Security Administration (TSA), the Department of Homeland Security (DHS), and the broader intelligence and security communities.”

Vulnerability

“The FAMS definition of vulnerability … is inconsistent with traditional risk-based definitions, which focus on the probability that an attack will succeed. It shifts the focus away from other potential vulnerabilities. We recommend that the FAMS reconsider its approach to vulnerability and engage the aviation security community on this issue.” (The emphasis is in the original.)

Filtering Process

“The FAMS understanding of consequence and its subsequent ‘filtering’ process … bias its allocation decision. To focus limited resources, the FAMS filters flights according to … . “Guidance in the form of legislation and departmental memoranda following 9/11 directed FAMS to focus on flights that present ‘high security risks.’ But, ultimately, that guidance was ambiguous and could be outdated. These fundamental assumptions concerning risk, on which it allocates resources, warrant interagency review by the broader intelligence and security community.” (The emphasis is in the original.)

Allocation Process

“The FAMS filtering process defines ‘high risk’ and directs its efforts toward flights fitting those characteristics. Its allocation process—a modified version of the SABRE software used by airlines to schedule flight crews—attempts to cover the maximum number of high risk flights within fixed resources. … The scheduling tool requires manual involvement to recognize and modify scheduling solutions, which may not be consistent with effective risk reduction. “Our analysis of one month of FAMS coverage data reveals … [some concerns.] To compensate for a lack of resources and deny predictability, the FAMS should integrate randomness into its allocations.” (The emphasis is in the original.) “Contrary to the popular use of the term ‘random,’ allocating resources in such a way does not mean choosing them haphazardly or without a plan. The overall probability distribution for a group of comparable aircraft can be chosen based on risk analysis. For instance, the FAMS may choose to cover flights in and out of [a particular geographic region]. But the tactical allocation decision concerning a specific flight must be random and converge around the overall category average over time. A terrorist group may be able to discern the overall category average through effective, long-term surveillance but will never know conclusively whether or not the flight it plans to hijack will be covered on a particular day.”

Performance Measures

“The FAMS primary performance measure—average coverage rates—can mask weaknesses in coverage patterns. In particular, they can mask a situation in which certain flights within a category of comparable flights are heavily covered while others are rarely if ever covered. Accordingly, the FAMS should develop performance measures to track coverage consistency. One example involves tracking coverage deviation, defined as the average difference between the individual coverage rates of each flight in a comparable category and the overall category coverage rate.” (The emphasis is in the original.)

Decision Support Tools

“During the course of our analysis, we noted that FAMS decision makers did not have a tool for evaluating the resource implications of different threat scenarios and alternative coverage schemes. The SABRE scheduler is not flexible enough to support quick-response analyses. The FAMS should build a simple decision-support tool, along the lines of the model we developed based on risk balancing, to facilitate a system- wide view of resource decisions.” (The emphasis is in the original.) “Such a tool would augment, not replace its scheduling tool, by allowing decision makers to look across the entire aviation system and investigate the resource implications of alternative allocation scenarios. In other words, how easily could the FAMS adapt to a different threat environment? Could it significantly increase the number of marshals aboard specific flights? Where might those resources come from? How would these changes affect coverage elsewhere?”

Appendix IV: Performance Measures for Assessing the Federal Air Marshal Service

In 2003, the Office of Management and Budget (OMB) used its Program Assessment Rating Tool (PART) to assess the management and performance of the Federal Air Marshal Service (FAMS). At that time, a key performance measure for assessing FAMS was based on coverage of targeted critical flights under various risk categories. This measure is still applicable currently; however, its designation has been changed from an output measure to an outcome measure. Moreover, this performance measure—the coverage of targeted critical flights—is now considered a proxy indicator regarding air marshals’ ability to defeat an attempted attack. Also, in further response to the findings of OMB’s 2003 assessment, FAMS established two additional outcome measures, one of which serves as another proxy indicator of air marshals’ ability to defeat an attempted attack: The additional proxy outcome measure is the average annual rate of accuracy in air marshals’ firearms requalification testing.

The additional, non-proxy outcome measure is based on a national survey of households to determine the level of public confidence in air marshals’ ability to promote aviation security.

These updated measures have been approved by the Department of Homeland Security—and also were approved in 2008 by OMB during its PART-based reassessment of FAMS. An overview of FAMS’s updated performance measures is presented in table 7.

Appendix V: Working Groups Formed by the Federal Air Marshal Service to Address Various Issues

In March 2006, the Director of the Federal Air Marshal Service (FAMS) communicated to employees his intention to establish working groups to examine a variety of issues ranging from mission, organizational, and operational topics to workforce satisfaction and quality-of-life concerns. Two months later, in May 2006, the director communicated to FAMS employees that 12 working groups had been established, with each chaired by a field office special agent-in-charge (SAC) and that subject matter experts from the field and headquarters were available to assist in an advisory role.

Subsequently, the number of working groups increased to a total of 36. Table 8 categorizes the 36 working groups and briefly summarizes the purposes of each.

Also, regarding the status of the 36 working groups as of October 1, 2008, FAMS officials reported the following (see notes to table 8): 18 working groups (table note a): Each of these working groups had completed its work and given a final report to FAMS management. Each report had been reviewed by FAMS executives and then distributed to agency employees via a broadcast message from the FAMS Director. If applicable, the broadcast message also presented management’s responses to any recommendations made by the respective report. 9 working groups (table note b): Each of these working groups had completed its work and given a final report to FAMS management. The reports were undergoing review by FAMS executives. 5 working groups (table note c): Each of these working groups had yet to complete its work and give a final report to FAMS management. 4 working groups (note d): Each of these working groups is to remain ongoing. As such, final reports are not expected to be issued; rather, each group will present its findings when applicable and by appropriate means.

Appendix VI: FAMS Workforce Satisfaction Survey

Appendix VI: FAMS Workforce Satisfaction Survey On behalf of the Director and the Office of Law Enforcement /FAMS Office of Workforce Planning and Management, thank you for taking the time to complete this survey. The purpose of this survey is to capture information regarding workforce satisfaction at OLE/FAMS. The information obtained by this survey will be used by OLE/FAMS leadership to assess the current levels of workforce satisfaction for the purposes of planning, policy development, and program enhancement. In addition, the data obtained by this survey will be used to evaluate current OLE/FAMS workforce satisfaction initiatives and strategies. Please take the time to carefully complete this survey. Your input will remain confidential and is vital to making OLE/FAMS a premier law enforcement organization. Thank you for your help.

For each item, choose the response that best reflects your experience at OLE/FAMS.

SA A N D SD SA A N D SD 9. I have adequate equipment, supplies, and materials to accomplish my duties. 10. Generally, the equipment I use (e.g., firearm, computer, cell phone, PDA , etc.) to perform my job works properly. 11. The equipment I use is sufficiently easy to operate. 12. I am generally satisfied with the quality of OLE/FAMS physical facilities (e.g., workspaces, training facilities, 10/30/2008 physical fitness areas, firearms ranges, etc.) 13. I am generally satisfied with the availability of OLE/FAMS physical facilities (e.g., workspaces, training facilities, physical fitness areas, firearms ranges, etc.)

SA A N D SD 14. I receive the training I need to do my job. 15. I am satisfied with the frequency/amount of training I receive in my office. 16. Generally, I am satisfied with the content and variety of job-related training I receive in my office. 17. I am satisfied with the continuing education opportunities offered by my job. 18. OLE/FAMS supports continuing education opportunities relevant to my job. 19. My work schedule affords me the opportunity to pursue continuing education. 20. I am encouraged by my supervisors and managers to seek training and educational opportunities. Career Development 21. Overall, I am satisfied with the progress I have made toward my career goals. 22. There are sufficient opportunities for career advancement at OLE/FAMS. 23. In my present position, I have a clearly understood career path. 24. Promotions to supervisory levels in OLE/FAMS are based on merit. 25. In OLE/FAMS, the selection criteria for promotion are clear. Policies & Procedures 26. OLE/FAMS’ written policies support (and do not hinder) mission accomplishment. 27. The local policies and procedures of my office support mission accomplishment. 28. I am able to stay updated and am informed about the latest policies and procedures.

Employee Involvement & Autonomy 29. I have effective channels to voice my opinion regarding work-related issues (e.g., working groups, listening sessions, e-mail suggestion box, etc.). 30. I am empowered to use my professional discretion in daily execution of my duties. 31. I am provided sufficient opportunities to participate in important decisions affecting my work. Rewards & Recognition 32. Outstanding performance is recognized in my office. 33. I am satisfied with the promotion practices of OLE/FAMS. 34. In my office, monetary rewards (i.e., cash awards, in- position increases, etc.) are tied to performance. 35. I am generally satisfied with my pay. Communication 36. OLE/FAMS’ policies and procedures are clearly communicated and easy to understand. 37. I am satisfied with communication within my office (e.g., FO , branch, division, directorate). 38. There are mechanisms in place which allow me to freely express my comments, concerns, and suggestions without fear of retaliation. 39. I am satisfied that the work-related concerns I address with management are addressed appropriately. 40. I have enough information to do my job well. Safety, Health & Medical Issues 41. I feel that my job-related stress is manageable. 42. I am generally satisfied with OLE/FAMS programs related to employee safety, health, and wellness. 43. I am satisfied that OLE/FAMS management is concerned for the health and safety of employees and is working continuously to offer improved services. 44. I have been provided information and resources to take personal responsibility for my health and wellness as it relates to my job (e.g., proper diet, fitness, sufficient rest). 45. I feel that medical information relevant to my job is communicated to me. Work & Family Life 46. I am able to effectively balance my work with my personal/family life. 47. My family is supportive of my career with OLE/FAMS.

SA A N D SD SA A N D SD SA A N D SD SA A N D SD SA A N D SD 48. Current initiatives (e.g., Voluntary Lateral Transfer Program, new office openings, etc.) have a positive effect on quality of work life/family life.

10/30/2008 49. OLE/FAMS leadership has implemented positive changes affecting scheduling. 50. I am satisfied that OLE/FAMS is exploring initiatives to improve quality of life/family life. 51. I have seen improvement in quality of work life and family life as a result of the recommendations from the Director’s Working Groups. 52. I have seen positive changes made in OLE/FAMS in the last year. Organizational Commitment 53. I am proud to work for OLE/FAMS. 54. I find my values are similar to OLE/FAMS values. 55. I feel a sense of loyalty to OLE/FAMS. 56. I am likely to stay at OLE/FAMS for the next 12 months. Job Satisfaction 57. The work I do is important. 58. I find my work challenging and interesting. 59. Generally speaking, I am very satisfied with my job. 60. I like the kind of work I do (e.g., my current duties and assignment).

Comments Section Please use this section to provide more specific information for any of the above questions.

Reasons for Staying with OLE/FAMS Indicate the importance of each of the following factors in your reasons for staying with OLE/FAMS.

Appendix VII: Comments from the Department of Homeland Security

Appendix VIII: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the contacts named above, Danny Burton and John Hansen (Assistant Directors) and Michael Harmond (Analyst-in-Charge) managed this assignment.

David Alexander, Chuck Bausell, Arturo Cornejo, Wendy Dye, Stuart Kaufman, and Courtney Reid made significant contributions to the work.

Tom Lombardi provided legal support.

Katherine Davis provided assistance in report preparation.