Federal Air Marshal Service: Additional Actions Needed to Ensure Air Marshals' Mission Readiness
Released: 2016-09-14T00:00:00

Letter

Background

Created in 1961 to counter hijackers, the organization that is now FAMS was expanded in response to the September 11, 2001, terrorist attacks. On September 11, 2001, 33 air marshals were operating on U.S. flights. In accordance with the Aviation and Transportation Security Act (ATSA), enacted in November 2001, TSA is authorized to deploy federal air marshals on every passenger flight of a U.S. air carrier and is required to deploy federal air marshals on every flight determined by the Secretary of Homeland Security to present high security risks—with nonstop, long distance flights, such as those targeted on September 11, 2001, considered a priority. Since the enactment of ATSA, FAMS staff grew significantly and, as of July 2016, FAMS employed thousands of air marshals. FAMS received an increase in appropriations each fiscal year from 2002 through 2012—peaking at an appropriation of approximately $966 million in fiscal year 2012. However, since 2012, FAMS has experienced a reduction in amounts appropriated. Specifically, FAMS received appropriations amounting to approximately $908 million in fiscal year 2013, $819 million in fiscal year 2014, and $790 million in fiscal year 2015. Of these appropriations, TSA expenditures for FAMS training were about $1.7 million, $4.4 million, $6 million, and $4.8 million in fiscal years 2012, 2013, 2014 and 2015, respectively. According to FAMS officials, due in part to reductions in its appropriations, FAMS hired no new air marshals during fiscal years 2012 through 2015. However, FAMS received appropriations amounting to $805 million for fiscal year 2016 (an increase of about $15 million over fiscal year 2015) and hired new air marshals in fiscal year 2016.

Training Roles and Responsibilities

FAMS and TSA’s OTD share responsibility for providing training to federal air marshals. OTD is primarily responsible for designing, developing, and evaluating all the training courses that air marshals receive. In addition, OTD delivers the training programs that are offered at TSATC and oversees training instructors assigned there. These training programs include, among others, FAMTP, discussed later in this report, and the field office training instructor program that is taught at TSATC. FAMS, in collaboration with OTD, develops training requirements for air marshal candidates and incumbent air marshals, serves as a subject matter expert to OTD in developing and evaluating new or proposed training courses, as well as operates and oversees the FAMS recurrent training program, which is taught by training instructors within each FAMS field office.

Federal Air Marshal Training Program

To ensure that air marshals are fully trained and can effectively carry out FAMS’s mission, TSA established FAMTP. Air marshal candidates are required to successfully complete 16 and one-half weeks of training. After an initial one-week orientation at TSATC, air marshal candidates complete FAMTP in two phases. FAMTP-I is a seven week course in which new hires learn basic law enforcement skills at the Federal Law Enforcement Training Center in Artesia, New Mexico. On completing FAMTP-I, FAMS candidates complete FAMTP-II—an eight and one-half- week course at TSATC that is intended to teach air marshal candidates the knowledge, skills, and abilities necessary to prepare them for their roles as federal air marshals.

Once air marshal candidates graduate from FAMTP-II, they report for duty at their assigned field office. As incumbent air marshals, they are required to complete 160 hours of recurrent training courses annually. FAMTP-II courses serve as the core of the recurrent training courses and incumbent air marshals receive these courses from training instructors in training facilities in or near their respective field offices. These recurrent training courses are intended to ensure air marshals maintain and enhance perishable tactical skills that are deemed critical to the success of FAMS’s mission. FAMS recurrent training includes both mandatory refresher courses that all air marshals must complete every year as well as a broad set of courses within several disciplines, as shown in figure 1, that field offices must ensure are incorporated into their annual or quarterly training plans. The mandatory courses include use of force, off-range safety, fire extinguisher use, and baton use. The remainder of air marshals’ annual recurrent training hours must include courses within each of the FAMS training disciplines, such as defensive measures, firearms, mission tactics, and physical fitness. FAMS also requires air marshals to pass quarterly firearms qualifications, complete biannual fitness assessments, and pass periodic medical exams, which are discussed later in this report.

Overview of Training Evaluation

In October 2010, DHS issued its Learning Evaluation Guide to help the department’s learning and development community evaluate the effectiveness of its training activities. Among other things, the guidance identifies the Kirkpatrick model—a commonly accepted training evaluation model that is endorsed by the Office of Personnel Management in its training evaluation guidance—as a best practice. This model is commonly used in the federal government. The Kirkpatrick model consists of a four- level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. The following is a description of what each level within the Kirkpatrick model is to accomplish:

Level 1: The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training.

Level 2: The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course.

Level 3: The third level measures how training affects changes in behavior on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on the job.

Level 4: The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before and after a training modification was made.

TSA Assesses Air Marshals’ Training Needs by Considering Several Information Sources, but Opportunities Exist to Obtain More Feedback from Air Marshals

TSA Assesses Its Federal Air Marshal Training Program Using Multiple Information Sources

TSA’s primary method for assessing air marshals’ training needs is by holding Curriculum Development Conferences (CDC) and Curriculum Review Conferences (CRC). Specifically, OTD holds CDCs to determine whether to approve proposals to develop new training courses, and convenes CRCs to evaluate the effectiveness of existing FAMTP courses and, if appropriate, to make recommendations to address any identified shortcomings. These conferences are composed of OTD officials responsible for developing and implementing FAMTP courses and relevant subject matter experts, such as training instructors, field office Supervisory Air Marshals-in-Charge (SACs), SFAMs, and air marshals. According to OTD guidance and consistent with Federal Law Enforcement Training Accreditation Board standards, CDCs are to be held prior to the development of new training programs and CRCs are to be held no less than every three years or sooner if directed by FAMS management. CDCs can also be held in response to directives from OTD or FAMS management and to requests for additional training from FAMS personnel.

As part of the CDCs and CRCs, OTD conducts assessments to determine the extent that existing FAMTP courses are current with existing or planned FAMS policy, procedures, or new equipment or technology, and address the known threat environment, and air marshals’ training needs. When doing so, OTD considers various sources of information including, among others, its job task analysis; training-related concerns raised by field office focus groups; feedback from air marshal candidates, training instructors and other subject matter experts; and intelligence. According to OTD guidance, this information is primarily to be gathered as described below.

TSATC Training Evaluation Surveys: OTD’s student critique review program evaluates FAMTP training courses delivered at TSATC consistent with Kirkpatrick levels one and three. Under this program, OTD solicits and reviews feedback from air marshal candidates on the quality of the FAMTP courses that they complete at TSATC and from newly graduated air marshals on the extent that these courses effectively prepare them for their duties. Specifically, consistent with Kirkpatrick level 1, OTD requires air marshal candidates to complete a course evaluation on the effectiveness of the course and the quality of the instructor and facility immediately after completing the course. Further, consistent with Kirkpatrick level 3, TSATC surveys newly graduated air marshals 10 to 12 months after they have graduated from FAMTP-II, and their supervisors within 12 months of their graduation, to obtain their feedback on the extent that the training adequately prepared the FAMTP graduates to successfully perform their mission. In addition, the program provides OTD with feedback from air marshal candidates and newly graduated air marshals on the effectiveness of FAMTP curriculum, instructor performance, and TSATC facility or safety, or other related issues. This feedback is used by CDCs and CRCs to identify training gaps and determine how to appropriately address them. However, as described later in this report, response rates by air marshals on these surveys have been low.

TSATC Examinations and Simulations: Consistent with Kirkpatrick level 2, OTD requires air marshal candidates to pass written exams or job simulations in order to advance through FAMTP. Specifically, air marshal candidates must demonstrate that they possess the knowledge, cognitive, or physical skills that classroom courses are intended to impart by passing examinations. OTD has developed evaluation tools, such as checklists, that TSATC training instructors must use to objectively determine air marshal candidates’ proficiency in law enforcement tactics and techniques such as marksmanship, defensive tactics, arrest procedures, and decision-making. OTD collects and analyzes the data on newly hired air marshals’ performance to determine the extent to which air marshal candidates have mastered the learning objectives with each FAMTP course and to identify any areas in the curriculum that may need revision. For example, OTD officials stated that they may revise examination questions in response to a relatively high number of air marshal candidates failing a question or series of questions due to poor wording. Furthermore, OTD uses these data to identify and address any training needs not met by the existing FAMTP curriculum when carrying out CRCs and CDCs.

In addition to using surveys and examinations when evaluating FAMTP curriculum provided at TSATC, OTD officials noted additional information sources they use when evaluating FAMTP curriculum, including field office training assessment teams and quarterly training teleconferences.

Field Office Training Assessment Teams: OTD established field office assessment teams, which consist of TSATC instructors, to assess field office training programs and their instructors. As described earlier, field office training programs primarily provide the recurrent training that incumbent air marshals are required to fulfill each year. In advance of the assessment team’s visit, TSATC sends surveys to supervisors and air marshals in the field with questions on the effectiveness of the field office’s training program, including its training instructors and facilities, as well as FAMS’s training curriculum. According to OTD officials, when conducting assessments at the field offices, team members are to observe field office trainers in class to ensure that FAMTP courses are taught uniformly across all FAMS field offices. They also are to review the field office’s training records and policies and procedures to ensure the field office’s training program is in compliance with OTD and FAMS policies, and when necessary, to make recommendations for improvement. For example, OTD officials told us that an assessment team discovered a field office whose training staff were using unapproved “dynamic fighting” tactics to teach air marshals how to fend off multiple attackers when cornered, which had resulted in many severe injuries. In this case, the assessment team halted use of the unapproved scenarios and provided approved lesson plans that taught air marshals to counter multiple attackers. OTD officials stated that training assessment team visits also provide opportunities for TSATC trainers to engage directly with field office trainers and air marshals to share new best practices and identify any unmet training needs. However, as we discuss later in this report, OTD has not sent assessment teams to evaluate field office training programs since March 2013.

Quarterly Training Teleconferences: OTD holds quarterly conference calls between TSATC staff, FAMS headquarters training staff, and field office training staff to discuss service-wide training issues. According to OTD officials, these teleconferences provide opportunities to elicit feedback from trainers on unmet training needs and any challenges in delivering training, and to share best practices among the field offices.

OTD Could Improve its Collection of Feedback from Air Marshals to Fully Identify and Address Training Needs

OTD Does Not Systematically Collect Training Feedback from Incumbent Air Marshals on Field-Based Training

OTD conducts surveys to obtain feedback from air marshal candidates and newly graduated air marshals on the effectiveness of FAMTP courses they complete at TSATC and the quality of TSATC trainers and facilities, consistent with Kirkpatrick levels 1 and 3. However, OTD does not also obtain such feedback from incumbent air marshals after they complete their recurrent training courses at their respective field offices.

Our previous work on federal training programs, as well as DHS’s Learning Evaluation Guide, has found that implementing a balanced multi-level systematic approach to evaluate and develop training, such as the Kirkpatrick model, can provide agencies with varied data and perspectives on the effectiveness of training efforts necessary to identify problems and improve training and development programs as needed. In addition, our work has also shown that agencies should ensure that they incorporate a wide variety of stakeholder perspectives in assessing the impact of training on employee and agency performance.

OTD officials stated that conducting level 1 and 3 evaluations for air marshal candidates and newly graduated air marshals has provided sufficient feedback to reliably identify all air marshals’ training needs because the agency has taken steps to ensure that the content and quality of training for air marshals candidates is identical to that of recurrent training for incumbent air marshals. However, FAMS did not hire any new air marshals from fiscal years 2012 through 2015. As a result, TSA has not systematically gather feedback on the effectiveness of FAMTP training curriculum from air marshals for approximately four years. Over this time period, OTD has revised the training curriculum, such as adding a course on personal security when overseas and expanding the number of courses within the legal and investigative discipline to cover all transportation modes. Moreover, while the minimum skill requirements may be the same for both air marshal candidates and incumbent air marshals in the field, the training needs for both groups may not necessarily be identical. With greater experience in carrying out missions, incumbent air marshals may have a better idea of their training needs than air marshal candidates or newly graduated air marshals, which could result in more experienced incumbent air marshals providing different feedback on the quality of the training. Further, although incumbent air marshals take many of the same training courses as air marshal candidates, they do so at different facilities and with different instructors.

OTD officials also stated that field office training assessments and quarterly training teleconferences provide additional opportunities to both ensure that the training all air marshals receive is standardized across the service and to obtain incumbent air marshal feedback. However, OTD has not sent assessment teams to evaluate field office training programs since March 2013 due, in part, to a lack of resources. OTD officials reported that they plan to resume field office training assessments during fiscal year 2017 and conduct assessments at 10 FAMS field offices per year if sufficient funding is available. These officials also reported that OTD plans to increase the frequency of training teleconferences between TSATC and field office training programs from a quarterly to monthly basis and invite field office leadership—SACs and Assistant Supervisory Air Marshals-in-Charge (ASACs)—to participate in these meetings.

Nevertheless, our review suggests that OTD could benefit from broadening its efforts to gather feedback on recurrent training courses. First, field office staff we interviewed at the seven field offices we visited stated that improvements to training could better prepare them for their roles. For example, SFAMs and training staff in four of the seven field offices we visited stated that the training curriculum is overly focused on the training needs of air marshal candidates and newly graduated air marshals. Staff from five of the seven field offices also identified advanced training courses beyond those currently provided that they believed should be offered to incumbent air marshals, in areas such as firearms, defensive, or medical training. Second, field office staff at all seven field offices we visited identified training that should be revised, expanded, or added, to include topics such as active shooter response, counter surveillance and behavior detection techniques, training on improvised explosive devices and other explosives, and expanded legal and investigative training, among others. These sources also told us that the curriculum did not adequately address changes in their responsibilities over time, which include a broader set of current threats such as improvised explosive devices or FAMS-specific training on active shooters. OTD officials stated that they believed the current FAMTP curriculum adequately addresses the types of additional training that field office staff identified and that the curriculum has been designed to meet the needs of air marshals at all experience levels and may be consistently and safely delivered to the entire workforce. However, without a mechanism to systematically collect and incorporate feedback on field- based training for incumbent air marshals, consistent with Kirkpatrick level 1 and 3, OTD could miss important opportunities to identify problems and improve overall training and development.

Response Rates for Air Marshal Training Surveys Have Been Consistently Low

When OTD administered surveys to obtain feedback on the FAMTP-II and field-based training, the response rates were substantially lower than the 80 percent rate OMB encourages for federal surveys that require its approval. Specifically, about 19 to 38 percent of air marshals that graduated from FAMTP-II and their supervisors responded to the surveys that TSATC administered from 2009 through 2011—the last 3 full years in which FAMS hired air marshals. Additionally, according to OTD officials, the combined response rates for the surveys that training assessment teams conducted from June 2012 through March 2013 was about 16 percent. OTD staff acknowledged that the response rate to these surveys have been consistently low, but stated that the low response rates have not significantly affected the usefulness of the surveys. According to OTD staff, with regard to the FAMTP-II surveys, they received a sufficient number of responses to successfully evaluate the extent that FAMTP courses have met all air marshals’ training needs. However, OMB guidance stipulates that agencies must design surveys to achieve the highest practical rates of response to ensure that the results are representative of the target population and that they can be used with confidence as input for informed decision-making. The guidance also states that response rates are an important indicator of the potential for a bias called nonresponse bias, which could affect the accuracy of a survey’s results. In general, as a survey’s response rate increases, the likelihood of a bias problem decreases, and, therefore, the views and characteristics of the target population are more accurately reflected in the survey’s results.

OMB guidance also describes the methods agencies can use to improve the response rate of future surveys, including conducting outreach to groups of prospective respondents, ensuring that the survey is well- designed and brief, providing alternative modes to provide responses, conducting nonresponse follow-up efforts, and extending cut-off dates for survey completion. OTD officials reported that they have taken several of these actions to improve the response rates of the FAMTP-II surveys, but have had little success in improving their response rate. Specifically, officials stated that TSATC instructors and staff discussed the surveys and their importance to improving future course offerings in class. In addition, OTD officials reported designing the survey to be as brief as possible, making it accessible via the internet and air marshals’ handheld devices, sending out follow-up reminders to survey respondents via e-mail and telephone, and contacting non-respondents’ field office supervisors. OTD officials told us that the low response rates may be attributable to “survey fatigue” given the high number of surveys that TSA employees are asked to complete and stated that there was little more that they could do to persuade air marshals to respond.

Although OTD officials reported taking several of the actions that OMB recommends for agencies to improve survey response rates, additional actions could improve the response rate of future OTD surveys, including those administered to the air marshals FAMS hired this year. For example, monitoring future survey response rates by field office could help OTD identify and then target extra follow-up efforts to air marshals and their supervisors in field locations that have comparatively low response levels. Further, extending the cut-off date for air marshals and their supervisors to respond to the survey, or requiring survey respondents to complete the surveys, could help improve response rates to future surveys. Until OTD achieves sufficient response rates, OTD cannot be reasonably assured that the feedback it received represents the full spectrum of views held by air marshals or their supervisors. Achieving an adequate response rate is important, particularly as FAMS’s CRC and CDC processes rely, in part, on the survey results to identify training gaps and determine how to appropriately address them.

TSA Has Established Recurrent Training Requirements to Ensure Mission Readiness, but Opportunities for Improvement Exist

FAMS relies on its recurrent training program to help ensure incumbent air marshals’ mission readiness, but additional actions could strengthen FAMS’s ability to do so. First, FAMS does not have complete and timely data on the extent to which air marshals have fulfilled their recurrent training requirements. Second, FAMS evaluates incumbent air marshals’ proficiency in some, but not all, key skills using tools such as examinations or checklists. In addition, FAMS has established a new health, fitness, and wellness program as part of its recurrent training program—in part to address recent concerns with air marshals’ fitness and injury rates—but it is too early to gauge the program’s effectiveness.

FAMS Lacks Complete and Timely Data to Ensure Air Marshals’ Compliance with Recurrent Training Requirements

As shown in figure 2, FAMS requires air marshals to complete certain recurrent training requirements on a regular basis to ensure that air marshals maintain their proficiency in the knowledge, skills, and abilities that are needed to successfully carry out FAMS’s mission. However, FAMS does not have complete and timely data to ensure air marshals’ compliance with these training requirements.

Senior OTD and FAMS officials responsible for developing and overseeing the recurrent training program, as well as field office SFAMs, training instructors, and air marshals at the field offices we visited, identified the importance of the FAMS training program to ensuring air marshals’ mission readiness. These personnel stated that air marshals are unique among their fellow law enforcement officers because air marshals lack regular on-the-job opportunities to actively utilize the knowledge, skills, and abilities they develop in training courses to address a key aspect of FAMS’s mission—defeating terrorist or other criminal hostile acts. Therefore, according to OTD and FAMS officials, FAMS ensures air marshals’ mission readiness by monitoring the extent to which they have completed their recurrent training requirements.

According to FAMS policy, field office SACs or their designees are responsible for ensuring that air marshals assigned to them have completed their recurrent training requirements and that the completion of these requirements is recorded in FAMS’s database—Federal Air Marshal Information System (FAMIS)—no later than 5 days after an air marshal has completed a training requirement. FAMS headquarters personnel within the Field Operations Division (Field Operations) generate reports in FAMIS detailing the extent to which air marshals have passed the practical pistol course, participated in physical fitness assessments, and completed their requisite number of recurrent training hours on a quarterly and annual basis. According to Field Operations staff, these personnel contact field office SACs or their designees when these reports identify air marshals that have not met their recurrent requirements. If field office staff report that the air marshal(s) have completed a requirement(s), but have not entered this information in FAMIS, Field Operations is to request appropriate documentation and update FAMIS. Field Operations officials stated they discuss with field offices why any air marshals have not completed their training requirements, such as illnesses, injuries, or scheduling issues, and, if necessary, the field office SAC is to take appropriate action.

In addition, FAMS policy allows for air marshals to be exempted from training requirements when certain conditions, such as illness, injury, or military leave, are met and defines the process by which exemptions are to be requested and granted. Specifically, FAMS policy states that SACs must prepare a letter to the appropriate regional director to request approval of the exemption no later than 5 days after the end of a quarter. Field Operations officials reported that a FAMS headquarters staff person records the exemption into FAMIS once a regional director has approved the request.

FAMS has processes for field office SACs to monitor which air marshals have completed their required recurrent training each year, as well as those who have received exemptions from such training. However, we found that the data used to track this information were not complete or readily available for purposes of tracking air marshals’ compliance with these requirements when we requested these data in March 2015. We reviewed training data from FAMIS’s training module for calendar year 2014 to determine the extent that air marshals have met their recurrent training requirements. Although we were ultimately able to determine that almost all of the air marshals met their training requirements or received an appropriate exemption in calendar year 2014, it was difficult to do so because data on both approved exemptions and training completions were missing or had not been entered in a timely manner. We found that nearly one-third of all training exemptions granted to air marshals in calendar year 2014 had not been entered into FAMIS. Specifically, at least 299 training exemptions granted to about 2 percent of air marshals had not been entered into FAMIS when we received the data in March 2015—nearly three months after the calendar year had ended. Additionally, we found that nearly one-quarter of all training records for calendar year 2014 had been entered into FAMIS more than 5 days after an air marshal had completed the training.

FAMS headquarters officials responsible for reconciling recurrent training service-wide stated that these exemptions were not entered into FAMIS until July 2015—seven months after the calendar year ended. These officials told us that the delay was partly because FAMS took the database offline for three weeks in September 2014 to allow for an upgrade of the system. As a result, the staff person responsible for entering exemptions had become backlogged and later entered the backlogged exemptions into the database, in part, to reconcile the missing exemptions that were identified through our analysis of the 2014 training data. Additionally, FAMS officials responsible for reconciling completion of recurrent training service-wide reported that each quarter there are a significant number of air marshals for whom field office staff have not entered training records. According to these officials, at the end of every quarter, FAMS Field Operations staff must contact staff from several field offices to remind them to review and enter missing training records—a process that officials described as labor-intensive. In December 2015, FAMS officials provided us with the updated records for the air marshals whose exemptions had been entered into FAMIS as a result of our audit work to demonstrate that the air marshals’ 2014 recurrent training data had been corrected and were complete.

TSA Office of Inspection (OOI) reports have found similar problems with monitoring, or timely and accurate recording, of air marshals’ training records. Specifically, OOI inspections of FAMS’s field offices completed during 2010 through 2015 found that three field offices had not accurately recorded air marshals’ training data or done so in a timely manner— issues FAMS had not identified through its training monitoring process.

FAMS processes for monitoring the extent that air marshals service-wide have completed their recurrent training requirements have not ensured that air marshals’ training data are entered in a timely manner. These processes, as defined in FAMS policy, lack effective controls to ensure accountability. Specifically, FAMS has not specified in policy who has oversight responsibility at the headquarters level for ensuring that each field office has entered recurrent training data in a timely manner. Additionally, FAMS has not specified in policy who has oversight responsibility at the headquarters level for ensuring that headquarters personnel have entered air marshals exemptions into FAMIS within a defined timeframe.

Federal Standards for Internal Control states that agencies should ensure that transactions and events are completely and accurately recorded in a timely manner, and are readily available for examination. Federal regulations require that agencies establish policies governing employee training including the assignment of responsibility to ensure the training goals are achieved. In addition, internal control standards state that in a good control environment, areas of authority and responsibility are clearly defined and appropriately documented through its policies and procedures, and appropriate lines of reporting are established.

Given the number of training records that we found were incomplete or not entered into FAMIS in a timely manner, as well as the ongoing challenges that FAMS has faced in ensuring accurate and timely input of training and exemptions data as described in the OOI findings, policies that specify who is responsible at the headquarters level for overseeing these activities could help FAMS ensure its data on air marshals’ recurrent training are consistently accurate and up to date. Complete and readily available training and exemptions data would enable FAMS to more effectively determine the extent that air marshals service-wide have met their training requirements and are mission ready.

FAMS Requires Air Marshals to Demonstrate Their Proficiency in Marksmanship, but Does Not Require That They Do So for Other Key Skills

Air marshals must demonstrate their proficiency in marksmanship by taking the practical pistol course on a quarterly basis and achieving a minimum score of 255 out of 300—the highest qualification standard for any federal law enforcement agency, according to FAMS officials. However, for the remainder of air marshals’ required recurrent training courses, FAMS does not assess air marshals against a similarly identified level of proficiency, such as by requiring examinations to evaluate air marshals’ knowledge in classroom-based courses or by using checklists or other objective tools to evaluate air marshals’ performance during simulation-based courses, such as mission tactics. For instance, FAMS’s recurrent training includes both mandatory refresher courses that all air marshals must complete annually as well as a broad set of courses within several disciplines that field offices must ensure are incorporated into their annual or quarterly training plans. However, FAMS does not require air marshals to take an examination for any course within these disciplines.

Federal Standards for Internal Control states that agencies should establish expectations of competence for key roles, such as federal air marshals, to help the entity achieve its objectives, and that all personnel need to possess and maintain the relevant knowledge, skills, and abilities that allow them to accomplish their assigned duties. Additionally, GAO’s prior work on training and development states that in some cases, agencies may identify critical skills and competencies that are important to mission success, and require that employees meet requirements to ensure they possess needed knowledge and skills. Further, DHS’s Learning Evaluation Guide identifies testing or skill checklists as tools agencies can use to determine whether students have the knowledge and can perform the skills classes are designed to teach. The guide also states that learning activities that are skill-based, such as FAMS courses on tactical and defensive techniques, may require the development of skill checklists to determine the level of trainee proficiency.

Field Operations officials said that it is not necessary to use examinations for recurrent training courses because air marshals are continuously evaluated by field office training instructors and SFAMs who participate in their training. In addition, officials stated that air marshals demonstrate their proficiency in the various cognitive or physical skills they must possess during simulations conducted as part of FAMS’s recurrent training program. As a result, according to officials, FAMS can be assured that any gaps in air marshals’ performance are identified and addressed in a timely manner. Field Operations officials also stated that checklists are also unnecessary because training instructors do not evaluate air marshals’ performance solely on whether their actions were appropriate and the air marshal correctly applied the relevant principles or tactics taught by course simulations. Rather, air marshals must also articulate why their actions were appropriate and how they applied the relevant principles or tactics. For example, to evaluate air marshals’ performance in mission tactics simulations, Field Operations officials stated that training instructors observe air marshals’ actions in response to various simulated threats ranging from verbal or physical assaults to the crew by passengers to suicide bombers. According to FAMS officials, training instructors evaluate the extent the actions taken by air marshals resulted in positive outcomes (i.e., protected the plane, passengers, and crew) and were carried out in accordance with applicable authorities, policies, procedures, and principles. Officials stated that part of this assessment is based on air marshals’ explanation for why their actions appropriately addressed the simulated threat and applied relevant FAMS principles and tactics. In addition, TSA has established a training instructor training program, which, according to FAMS Field Operations officials, ensures that training instructors are highly trained and certified; and, therefore, can assess air marshals’ performance in a reasonably objective manner.

As previously discussed, for training courses taught at TSATC, OTD requires air marshal candidates and incumbent air marshals to demonstrate that they possess the knowledge or cognitive skills that classroom courses are intended to impart by passing examinations for training courses taught at TSATC. Additionally, when evaluating the performance of air marshal candidates and incumbent air marshals in courses taught at TSATC, OTD requires training instructors to use evaluation tools, including checklists. For example, TSATC training instructors must use these tools when evaluating air marshal candidates’ performance in defensive measures and mission tactics simulations as part of FAMTP-II. TSATC staff reported that they require TSATC training instructors to use such checklists because doing so better ensures air marshals are evaluated in an objective, fair, and consistent manner. Further, a field office SAC reported that given the absence of an objective tool for assessing air marshals’ performance in field-based training, such as defensive measures and tactics, there are air marshals who have not fully demonstrated the requisite level of proficiency, but still “passed” these courses and continued to fly missions. According to the SAC, air marshals are flying missions with colleagues they do not view as mission ready in part due to their performance in training courses—a concern raised by air marshals in 3 of the 7 field offices we visited. Finally, field office trainers in 3 field offices reported that a standardized tool for evaluating air marshals during training would help them to identify and address trainee deficiencies.

FAMS Field Operations officials also noted that standardized examinations or checklists during trainings are not necessary because SFAMs have opportunities to continually assess their air marshals’ mission readiness by flying with their squads or attending training. However, we found that SFAMs infrequently attend training with their squads or accompany them on flying missions, although they are not necessarily required to do so. The July 2014 FAMS Advisory Council minutes state that the council unanimously agreed that a large population of SFAMs do not fully participate in their air marshals’ required training. Air marshals from 6 of 22 field offices raised similar concerns based on our review of the minutes from field office focus groups conducted in fiscal year 2014. In addition, SFAMs in all 7 field offices we visited reported that they rarely fly with their squads, i.e., once per quarter or less. Further, SFAMs in 6 of the 7 field offices stated that they rely on air marshals’ self-assessments and factors unrelated to mission readiness, such as quality of administrative paperwork (i.e., travel vouchers and timecards), and completion of OLC training to assess air marshals’ performance.

Standardized methods for determining whether incumbent air marshals are mission ready, such as required examinations or evaluation tools, in key training courses could help provide better assurance that air marshals service-wide are mission ready. Objective and standardized methods of evaluating incumbent air marshals performance would better enable FAMS to assess air marshals’ proficiency in key skills and also more effectively target areas for improvement.

FAMS Developed a New Physical Fitness Program, but It Is Too Early to Gauge its Effectiveness

In 2015, FAMS developed a new physical fitness program—the Health, Fitness, and Wellness Program—in part to address recent concerns with air marshals’ fitness and injury rates, but it is too early to gauge the program’s impact. Over the period 2008 to 2015, FAMS commissioned two studies to evaluate air marshals’ health and fitness, as well as a third study to evaluate air marshal fatigue and sleeplessness. FitForce, a consulting group, which conducted the first evaluation of air marshals’ fitness in 2009, found that nearly 32 percent of the FAMS who participated in the study exercised less than three times per week and almost 7 percent did not exercise at all. FitForce also concluded that physical fitness is a necessity for air marshals to be able to perform the essential functions of their job, and stated that FAMS should make a commitment to address the fitness needs of air marshals. Additionally, a 2012 sleep study conducted by Harvard University concluded that more than half of the air marshals who responded to the study’s survey were overweight and nearly one-third were obese, and, therefore, may suffer a variety of health issues that could directly impact mission readiness. Furthermore, FAMS conducted its own review of air marshals’ fitness from 2012 through 2013 and concluded that air marshals suffered from high injury rates and declining overall health and wellness, which FAMS officials attributed in part to the increasing age of air marshals. Specifically, the review found that the injuries that occurred while air marshals took their physical fitness assessment from 2010 through 2013 had resulted in approximately 8,060 lost or restricted work days and 12,896 lost mission opportunities and Office of Workers’ Compensation Program claims totaling over $1 million.

We analyzed the scores that air marshals achieved in calendar year 2014 when taking the quarterly Mission Readiness Assessment (MRA)—the health evaluation program that FAMS had in place at that time. We found that, with the exception of the 1.5-mile run, the majority of air marshals who took the MRA met or exceeded each of the MRA component test goals. In quarters 2 through 4 of calendar year 2014, 84 to almost 90 percent of air marshals who participated in the MRA failed to meet the 1.5-mile run goal, as shown in figure 3. Moreover, about 5 percent of the air marshals did not meet the performance goals for any of the component tests in quarters 2 through 4 of calendar year 2014.

To address the impact that air marshals’ declining health and fitness may pose to FAMS’s ability to carry out its mission within TSA, as well as air marshals’ injury rates, FAMS has developed the Health, Fitness, and Wellness Program, which went into effect in April 2016. According to FAMS policy, this program will include a revised fitness assessment—the Health and Fitness Assessment (HFA)—and a general health and wellness program. FAMS officials reported that air marshals are to complete the HFA on a biannual basis, but will not be required to meet performance goals for any of the HFA’s four components: cardiorespiratory endurance, muscular strength, muscular endurance, and flexibility. Rather, FAMS will use the results of air marshals’ first HFA to establish a fitness baseline and to take appropriate action to improve the performance of those who do not maintain their fitness levels or show improvement. According to FAMS officials, the agency decided not to require air marshals to meet the performance goals for the HFA tests because the results of the HFA cannot reliably determine the extent that an air marshal is physically capable of carrying out FAMS’s mission. Officials explained that FAMS had originally intended to require incumbent air marshals to meet a physical fitness standard similar to the HFA, but did not do so because of concerns raised by TSA’s Office of Human Capital and Office of the Chief Counsel. Specifically, according to FAMS officials, advice was provided regarding whether the proposed physical fitness standard could reliably predict an air marshal’s physical ability to carry out FAMS job-related mission, as well as if FAMS could demonstrate the business necessity (mission-related) of the standard. Because of these concerns, FAMS’s leadership decided to implement the Health, Wellness, and Fitness Program with a focus on reducing the incidence of air marshals’ injuries, reducing the number of exemptions air marshals needed to request from taking the HFA, increasing program participation, and improving air marshals’ overall health and wellness instead. FAMS officials stated that in addition to general improvement of air marshals’ health and fitness, a key benefit of the new program will be that air marshals will request and receive fewer exemptions because the HFA will allow air marshals to demonstrate their fitness through alternative means of testing. FAMS officials reported that, when taking the HFA, air marshals may choose one of three exercises to perform for five of the six subsets within the four components. For example, when taking the upper body subset of the muscular strength component, air marshals may choose to perform pull-ups, assisted pull-ups, or lateral pulldowns. According to FAMS officials, because multiple exercises will be available for each HFA component, FAMS will no longer grant air marshals exemptions from taking the HFA unless an injury prevents them from performing any of the HFA exercises.

FAMS has established a goal for the Health, Wellness, and Fitness program—to provide the opportunity, resources, and education necessary to enhance mission readiness and promote workplace wellness, but it is too early to know if the program is achieving its intended goal. FAMS and OTD officials responsible for developing this program told us that FAMS plans to collect and analyze data on air marshals’ performance on the HFA over a period of about 12 to 18 months—two or three assessment periods. These officials stated that after FAMS had collected and analyzed sufficient data and established a baseline, the agency would be better positioned to collaborate with OTD to establish performance measures for the program. In the interim, FAMS plan to monitor data such as injury rates and the results of periodic physical exams.

Conclusions

Given the unique operating environment of air marshals, it is vital that TSA ensure that air marshals’ training needs are identified and addressed, and that air marshals are mission ready. TSA does not systematically obtain feedback on the extent to which FAMTP courses meet incumbent air marshals’ training needs because officials state that they collect sufficient information from air marshal candidates on their training programs. However, by regularly collecting incumbent air marshals’ feedback on the recurrent training they receive in the field offices, OTD would better ensure it considers the input and experience of incumbent air marshals when assessing and refining their training programs. Also, by taking additional steps to improve the response rates for the training surveys it administers to air marshal candidates, incumbent air marshals, and their supervisors, OTD could be more reasonably assured that the feedback it receives represents the full spectrum of views held by its air marshal workforce.

FAMS has established recurrent training requirements to ensure that air marshals maintain the knowledge, skills and abilities needed to carry out their mission. However, because FAMS processes have not ensured the timely and complete recording of training data—an ongoing challenge for FAMS—FAMS has been hindered in its ability to ensure air marshals’ compliance with training requirements. Specifying in policy who has oversight responsibility at the headquarters level for ensuring that each field office has entered air marshals’ training data in a timely manner and that headquarters personnel have entered air marshals’ exemptions into FAMIS could help FAMS better ensure its data on air marshals’ recurrent training are consistently complete and up to date. Such a policy could also enable FAMS to more effectively determine the extent that air marshals service-wide have met their training requirements and are mission ready. Additionally, by developing and implementing more objective and standardized methods of determining, in the course of their recurrent training, whether incumbent air marshals continue to be mission ready, FAMS could better assess their skills and also more effectively target areas for improvement.

Recommendations for Executive Action

To ensure effective evaluation of air marshal training, we recommend that the TSA Administrator direct OTD to take the following two actions: implement a mechanism for regularly collecting and incorporating incumbent air marshals’ feedback on the training they receive from field office programs, and take additional steps to improve the response rates of the training surveys it conducts.

To provide reasonable assurance that air marshals are complying with recurrent training requirements and have the capability to carry out FAMS’s mission, we recommend the TSA Administrator direct FAMS to take the following three actions: specify in policy who at the headquarters level has oversight responsibility for ensuring that field office SACs or their designees meet their responsibilities for ensuring that training completion records are entered in a timely manner, specify in policy who at the headquarters level is responsible for ensuring that headquarters personnel enter approved air marshals’ training exemptions into FAMIS, and define the timeframe for doing so, and develop and implement standardized methods, such as examinations and checklists, for determining whether incumbent air marshals continue to be mission ready in key skills.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for comment. In its written comments, reproduced in appendix II, DHS concurred with the five recommendations and described actions under way or planned to address them. DHS also provided technical comments that we incorporated, as appropriate.

With regard to the first recommendation to implement a mechanism for regularly collecting and incorporating incumbent air marshals' feedback on the training they receive from field office programs, DHS concurred and stated that TSA has developed a survey to measure the effectiveness of air marshal training curriculum, field office training personnel, and training facilities. DHS also stated that this survey will be added to the TSA On-Line Learning Center where it can be distributed to air marshals and supervisors on a regular basis. According to DHS, TSA implemented the survey in the On-Line Learning Center in July 2016 and, beginning in October 2016, will send the survey to air marshals and supervisors after they complete a course at TSATC. TSA also plans for curriculum development and review committees to use the feedback from these surveys to improve courses offered at TSATC. These actions, if implemented effectively, should address the intent of our recommendation.

With regard to the second recommendation to take additional steps to improve the response rates of the training surveys it conducts, DHS concurred and stated that future surveys of FAMTP graduates and their supervisors will be distributed to personnel through the On-Line Learning Center. DHS stated that the capabilities of the On-Line Learning Center will provide a tracking mechanism for program managers to ensure that personnel complete and submit the survey. According to DHS, survey reports will be compiled and sent to TSATC in a manner that maintains the anonymity of the respondent. TSA anticipates that this process will significantly improve response rates. These actions, if implemented effectively, should address the intent of our recommendation.

DHS concurred with our third and fourth recommendations that FAMS specify in policy (1) who at the headquarters level has oversight responsibility for ensuring that field office SACs or their designees meet their responsibilities for ensuring that training completion records are entered in a timely manner, and (2) who at the headquarters level is responsible for ensuring that headquarters personnel enter approved air marshals' training exemptions into FAMIS and define the timeframe for doing so. In response to our recommendations, FAMS updated its policy on recurrent training requirements for air marshals to assign Regional Directors, who are based in headquarters, the responsibility for ensuring that field office SACs or their designees adhere to FAMS’s procedures for recording training completion. The updated policy also requires that FAMS’s Field Operations Division, Tactical Support Section, verify that FAMIS entries are made for all training exemptions within five business days of the approval of the exemptions. These actions, if implemented effectively, should address the intent of our recommendations.

With regard to the fifth recommendation to develop and implement standardized methods, such as examinations and checklists, for determining whether air marshals continue to be mission ready in key skills, DHS concurred and stated that FAMS and OTD established a joint Integrated Project Team/Development Committee, which met in June 2016 to develop an assessment process that will be used to determine air marshals’ mission readiness. According to DHS, the joint Integrated Project Team/Development Committee consisted of representatives from seven FAMS field offices and FAMS headquarters as well as instructors and instructional design specialists from TSATC. DHS stated that the Integrated Project Team is drafting recommendations and that approved readiness measures will be implemented beginning in fiscal year 2018. This action, if implemented effectively, could address the intent of our recommendation. However, it is not clear to what extent this assessment process will include standardized methods for determining whether incumbent air marshals continue to be mission ready. We will continue to monitor TSA’s efforts.

We are sending copies of this report to appropriate congressional committees, the Secretary of Homeland Security, the TSA Administrator, and other interested parties. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7141 or groverj@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report addresses the following questions:

How does the Transportation Security Administration (TSA) assess the training needs of air marshal candidates and incumbent air marshals, and what opportunities exist, if any, to improve this assessment?

To what extent does the Federal Air Marshal Service (FAMS) ensure that incumbent air marshals are mission ready?

This report is a public version of the prior sensitive report that we provided to you. TSA deemed some of the information in the report as sensitive security information, which must be protected from public disclosure. Therefore, this report omits this information, such as specific numbers of air marshals and specific types of training that air marshals reported needed to be added to FAMS’s training curriculum to address changes in air marshals’ responsibilities. Although the information provided in this report is more limited in scope in that it excludes such information, it addresses the same questions as the sensitive security information report and the methodology used for both reports is the same.

To address the first objective, we reviewed TSA directives, guidance, and other relevant documentation describing TSA’s processes for developing and evaluating Federal Air Marshal Training Program (FAMTP) training curriculum to determine how TSA evaluates existing courses and develops new courses within FAMTP and other relevant training programs. We interviewed senior officials responsible for these efforts in TSA’s Office of Training and Development (OTD). We also analyzed documentation on the results of training curriculum assessments OTD conducted to identify recommendations made to improve training and the extent to which OTD implemented the recommendations. OTD conducted these assessments from May 2007 through April 2014. We compared OTD’s training development and evaluation processes to key principles identified in DHS guidance on training evaluation, and GAO’s prior work on training and development, specifically the Guide for Assessing Strategic Training and Development Efforts in the Federal Government. We also reviewed the minutes of the quarterly teleconferences held in fiscal years 2014 through 2015—the most recent time period for which the meeting minutes were available—between Transportation Security Administration Training Center (TSATC) staff, FAMS headquarters staff, and field office training staff to determine the types of issues discussed during these meetings. Additionally, we obtained the available response rates for surveys OTD conducted of FAMTP graduates and their supervisor’s on the effectiveness of FAMTP courses for calendar years 2009 through 2011—the last three full years that FAMS hired air marshals. In addition, we met with OTD officials to discuss the actions that had been taken to improve these response rates, and compared these actions to Office of Management and Budget standards and guidance for conducting surveys. Further, we visited the TSATC in Atlantic City, New Jersey and 7 of FAMS 22 field offices, which we selected, in part, to reflect a range in size (as determined by the number of air marshals assigned to the office) and geographic dispersion. At TSATC, we interviewed TSATC management and training instructors and toured the facility. At the field offices, we interviewed field office management, Supervisory Federal Air Marshals (SFAM), air marshals, and training instructors to obtain their views on the current training curriculum. The results of these interviews cannot be generalized to all field offices, but provide insight into the extent to which TSA is addressing air marshals’ training needs and ensuring their mission readiness.

To address the second objective, we assessed FAMS directives that set forth training requirements for incumbent air marshals, and analyzed air marshals training data for calendar year 2014, which is the most recent year for which training data were available, to determine the extent to which air marshals met these requirements. We interviewed senior FAMS officials to understand how FAMS uses this information to ensure that air marshals are mission ready. We compared the results of our analyses to Standards for Internal Control in the Federal Government, the DHS Learning Evaluation Guide, and GAO’s prior work on training and development. We assessed the reliability of the 2014 training data by (1) reviewing documentation on the processes for entering air marshals’ training records into the Federal Air Marshal Information System (FAMIS), (2) performing electronic testing for obvious anomalies and comparing FAMIS data to FAMIS-generated reports on training completion, and (3) interviewing knowledgeable officials about training records and exemptions entered into FAMIS. Although the data FAMS originally provided were not complete or entered in a timely manner, over the course of our audit we identified missing data that FAMS corrected in response to our inquiries. Therefore, we found the data were reliable for the purposes of our report. Additionally, as previously discussed, we interviewed TSATC training instructors and FAMS field office personnel to obtain their perspectives on FAMS methods for ensuring that air marshals are mission ready. We also reviewed the most recent Management Assessment Program inspection report completed by TSA’s Office of Inspections for all 22 field offices to identify the training-related findings. Additionally, we reviewed the FAMS Advisory Council minutes and the field offices’ focus group minutes for fiscal year 2014—the most recent full year of information available at the time of our request—to identify the training related issues that FAMS personnel raised to their leadership. Finally, we reviewed the studies that FAMS conducted or commissioned to inform its development of its physical fitness program and assessment—a component of air marshals training. We interviewed FAMS and OTD officials responsible for developing and implementing FAMS’s health, wellness, and fitness program to determine how TSA plans to measure the effectiveness of the program.

We conducted this performance audit from October 2014 to September 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Maria Strudwick (Assistant Director) and Michael C. Lenington (Analyst-in-Charge) managed this assignment. Jonathan Bachman, Claudia Becker, Juli Digate, Michele Fejfar, Imoni Hampton, Eric Hauswirth, Susan Hsu, Thomas Lombardi, and Minette Richardson made key contributions to this report.