Federal Air Marshal Service: Additional Actions Needed to Ensure Air Marshals' Mission Readiness
Published: 2016-09-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: FAMS, within TSA, is the federal entity responsible for promoting confidence in the nation's aviation system through deploying air marshals to protect U.S. air carriers, airports, passengers, and crews.
GAO was asked to assess FAMS's training program for federal air marshals. This report examines (1) how TSA assesses the training needs of air marshal candidates and incumbent air marshals, and any opportunities that exist to improve this assessment, and (2) the extent to which FAMS ensures that incumbent air marshals are mission ready.
GAO analyzed FAMS training data for calendar year 2014, the last year of available data, reviewed TSA, OTD and DHS guidance and policies on FAMS's air marshal training program, interviewed TSA and FAMS headquarters officials, and visited the TSA Training Center and 7 of FAMS 22 field offices selected based on size and geographic dispersion.
section_title: What GAO Found
paragraphs: The Transportation Security Administration's (TSA) Office of Training and Development (OTD) assesses air marshals' training needs using several information sources, but opportunities exist to obtain more feedback from air marshals on whether the training courses they must take met their needs. OTD primarily assesses air marshals' training needs by holding curriculum development and review conferences composed of OTD officials, training instructors, and other subject matter experts. In assessing courses, conference participants use, among other things, the results of surveys that some air marshals complete on the effectiveness of their training. However, while OTD administers these surveys for air marshal candidates and newly graduated air marshals, it does not use them to obtain feedback from incumbent air marshals on the effectiveness of their annual recurrent training courses. Systematically gathering feedback from incumbent air marshals would better position OTD to fully assess whether the training program is meeting air marshals' needs. Additionally, among the training surveys that OTD does currently administer to air marshals, the response rates have been low. For example, among newly hired air marshals and their supervisors from 2009 through 2011—the last three full years in which the Federal Air Marshal Service (FAMS) hired air marshals—the survey response rates ranged from 16 to 38 percent. Until OTD takes steps to achieve sufficient response rates, OTD cannot be reasonably assured that the feedback it receives represents the full spectrum of views held by air marshals.
FAMS relies on its annual recurrent training program to ensure incumbent air marshals' mission readiness, but additional actions could strengthen FAMS's ability to do so. First, FAMS does not have complete and timely data on the extent to which air marshals have completed their recurrent training. For example, nearly one-quarter of all training records for calendar year 2014 had not been entered into FAMS's training database within the required time period. Policies that specify who is responsible at the headquarters level for overseeing these activities could help FAMS ensure its data on air marshals' recurrent training are accurate and up to date. Second, FAMS requires air marshals to demonstrate proficiency in marksmanship by achieving a minimum score of 255 out of 300 on the practical pistol course every quarter. However, for the remaining recurrent training courses FAMS does not assess air marshals' knowledge or performance in these courses against a similarly identified level of proficiency, such as by requiring examinations or by using checklists or other objective tools. More objective and standardized methods of determining incumbent air marshals' mission readiness, as called for by the Department of Homeland Security's (DHS) Learning Evaluation Guide, could help FAMS better and more consistently assess air marshals' skills and target areas for improvement. Additionally, in 2015 FAMS developed a health, fitness, and wellness program to improve air marshals' overall health and wellness, but it is too early to gauge the program's effectiveness. This is a public version of a sensitive report that GAO issued in June 2016. Information that TSA deems “Sensitive Security Information” has been removed.
section_title: What GAO Recommends
paragraphs: GAO recommends that OTD implement a mechanism for regularly collecting incumbent air marshals' feedback on their recurrent training, and take steps to improve the response rates of training surveys it conducts. GAO also recommends that FAMS specify in policy who at the headquarters level has oversight responsibility for ensuring that recurrent training records are entered in a timely manner, and develop and implement standardized methods to determine whether incumbent air marshals continue to be mission ready in key skills. DHS concurred with all of the recommendations.

section_title: Letter
paragraphs: The successful bombing of Metrojet’s flight 9268 on October 31, 2015, as  well as attempts or plots by other terrorist organizations or individuals to  detonate explosives onboard commercial aircraft, demonstrate that  terrorists are continuing to target the aviation sector. To help ensure the  security of the nation’s civil aviation system addresses such threats, the  Office of Law Enforcement/Federal Air Marshal Service (FAMS), an Office  within the Department of Homeland Security’s (DHS) Transportation  Security Administration (TSA), is tasked with promoting confidence in the  nation’s aviation system through the deployment of federal air marshals to  protect U.S. air carriers, airports, passengers, and crews. While  deployed, these air marshals’ mission is to deter, detect, and defeat  terrorist or other criminal hostile acts. TSA established the Federal Air  Marshal Training Program (FAMTP) to provide air marshals with the  knowledge, skills, and abilities necessary to recognize and respond  effectively, including through the use of deadly force, to threats they may  encounter while deployed on a mission to protect the traveling public. The  unique operating environment of air marshals—the relatively tight  confines of an airplane, the possibility of air turbulence, and the presence  of numerous passengers and crew—highlights the importance of ensuring  that air marshals are well trained and capable of successfully carrying out  their duties.
You asked us to assess the training that air marshals receive to ensure  they effectively respond to the incidents that they encounter. This report  addresses the following questions:
How does TSA assess the training needs of air marshal candidates  and incumbent air marshals, and what opportunities exist, if any, to  improve this assessment?
To what extent does FAMS ensure that incumbent air marshals are  mission ready?
This report is a public version of the prior sensitive report that we  provided to you. TSA deemed some of the information in the report as  sensitive security information, which must be protected from public  disclosure. Therefore, this report omits this information, such as specific  numbers of air marshals and specific types of training that air marshals  reported needed to be added to FAMS’s training curriculum to address  changes in air marshals’ responsibilities. Although the information  provided in this report is more limited in scope in that it excludes such  information, it addresses the same questions as the sensitive report and  the methodology used for both reports is the same.
To address the first objective, we reviewed TSA guidance and other  relevant documents describing TSA’s processes for evaluating the  existing curriculum and developing new courses for air marshal  candidates and incumbent air marshals, and interviewed senior Office of  Training and Development (OTD) officials responsible for these efforts.  We also analyzed documentation on the results of training curriculum  assessments OTD conducted to identify recommendations made to  improve training and the extent to which OTD implemented the  recommendations. OTD conducted these assessments from May 2007  through April 2014. We then compared OTD’s training development and   evaluation processes to key principles identified in DHS guidance on  training evaluation, and GAO’s prior work on training and development.
We also obtained the available response rates for surveys that OTD  conducted of FAMTP graduates and their supervisors on the  effectiveness of FAMTP courses for calendar years 2009 through 2011— the last three full years that FAMS hired air marshals. In addition, we met  with senior OTD officials to discuss the actions that had been taken to  improve these response rates, and compared these actions to Office of  Management and Budget (OMB) standards and guidance for conducting  surveys. Further, we visited the TSA Training Center (TSATC) in Atlantic  City, New Jersey and 7 of 22 FAMS field offices, which we selected to  reflect a range in size (as determined by the number of air marshals  assigned to the office) and geographic dispersion. At TSATC, we  interviewed TSATC management and training instructors and toured the  facility. At the field offices, we interviewed field office management,  supervisory federal air marshals (SFAM), air marshals, and training  instructors to obtain their views on the current training curriculum. The  results of these interviews cannot be generalized to all field offices, but  provide insight into the extent to which TSA is addressing air marshals’  training needs and ensuring their mission readiness.
To address the second objective, we assessed FAMS directives that set  forth training requirements for incumbent air marshals, analyzed air  marshals’ training data for calendar year 2014—the most recent full year  for which training data were available—to determine the extent to which  air marshals met these requirements. We interviewed senior FAMS  officials to understand how FAMS uses this information to ensure that air  marshals are mission ready. We compared the results of our analyses to  Standards for Internal Control in the Federal Government and GAO’s  prior work on training and development. We assessed the reliability of  the 2014 training data by (1) reviewing documentation on the processes  for entering air marshals’ training records into the Federal Air Marshal  Information System (FAMIS), (2) performing electronic testing for obvious  anomalies and comparing FAMIS data to FAMIS-generated reports on  training completion, and (3) interviewing knowledgeable officials about  training records and exemptions entered into FAMIS. We discuss our  findings about the reliability of these data later in this report. Additionally,  as previously discussed, we interviewed TSATC personnel and FAMS  field office personnel to obtain their perspectives on FAMS methods for  ensuring that air marshals are mission ready. Further, we reviewed the  studies that FAMS conducted or commissioned to inform its development  of its physical fitness program and assessment—a component of air  marshals’ training. We interviewed FAMS and OTD officials responsible  for developing and implementing the new FAMS physical fitness program  to determine how TSA plans to measure the effectiveness of the program.  See appendix I for a more detailed discussion of our objectives, scope,  and methodology.
We conducted this performance audit from October 2014 to September  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Created in 1961 to counter hijackers, the organization that is now FAMS  was expanded in response to the September 11, 2001, terrorist attacks.  On September 11, 2001, 33 air marshals were operating on U.S. flights.  In accordance with the Aviation and Transportation Security Act (ATSA),  enacted in November 2001, TSA is authorized to deploy federal air  marshals on every passenger flight of a U.S. air carrier and is required to  deploy federal air marshals on every flight determined by the Secretary of  Homeland Security to present high security risks—with nonstop, long  distance flights, such as those targeted on September 11, 2001,  considered a priority. Since the enactment of ATSA, FAMS staff grew  significantly and, as of July 2016, FAMS employed thousands of air  marshals. FAMS received an increase in appropriations each fiscal year  from 2002 through 2012—peaking at an appropriation of approximately  $966 million in fiscal year 2012. However, since 2012, FAMS has  experienced a reduction in amounts appropriated. Specifically, FAMS  received appropriations amounting to approximately $908 million in fiscal  year 2013, $819 million in fiscal year 2014, and $790 million in fiscal year  2015. Of these appropriations, TSA expenditures for FAMS training were  about $1.7 million, $4.4 million, $6 million, and $4.8 million in fiscal years  2012, 2013, 2014 and 2015, respectively. According to FAMS officials,  due in part to reductions in its appropriations, FAMS hired no new air  marshals during fiscal years 2012 through 2015. However, FAMS  received appropriations amounting to $805 million for fiscal year 2016 (an  increase of about $15 million over fiscal year 2015) and hired new air  marshals in fiscal year 2016.
subsections: section_title: Training Roles and Responsibilities
paragraphs: FAMS and TSA’s OTD share responsibility for providing training to federal  air marshals. OTD is primarily responsible for designing, developing,  and evaluating all the training courses that air marshals receive. In  addition, OTD delivers the training programs that are offered at TSATC  and oversees training instructors assigned there. These training  programs include, among others, FAMTP, discussed later in this report,  and the field office training instructor program that is taught at TSATC.  FAMS, in collaboration with OTD, develops training requirements for air  marshal candidates and incumbent air marshals, serves as a subject  matter expert to OTD in developing and evaluating new or proposed  training courses, as well as operates and oversees the FAMS recurrent  training program, which is taught by training instructors within each FAMS  field office.
section_title: Federal Air Marshal Training Program
paragraphs: To ensure that air marshals are fully trained and can effectively carry out  FAMS’s mission, TSA established FAMTP. Air marshal candidates are  required to successfully complete 16 and one-half weeks of training. After  an initial one-week orientation at TSATC, air marshal candidates  complete FAMTP in two phases. FAMTP-I is a seven week course in  which new hires learn basic law enforcement skills at the Federal Law  Enforcement Training Center in Artesia, New Mexico. On completing  FAMTP-I, FAMS candidates complete FAMTP-II—an eight and one-half-  week course at TSATC that is intended to teach air marshal candidates  the knowledge, skills, and abilities necessary to prepare them for their  roles as federal air marshals.
Once air marshal candidates graduate from FAMTP-II, they report for  duty at their assigned field office. As incumbent air marshals, they are  required to complete 160 hours of recurrent training courses annually.  FAMTP-II courses serve as the core of the recurrent training courses and  incumbent air marshals receive these courses from training instructors in  training facilities in or near their respective field offices. These recurrent  training courses are intended to ensure air marshals maintain and  enhance perishable tactical skills that are deemed critical to the success  of FAMS’s mission. FAMS recurrent training includes both mandatory  refresher courses that all air marshals must complete every year as well  as a broad set of courses within several disciplines, as shown in figure 1,  that field offices must ensure are incorporated into their annual or  quarterly training plans. The mandatory courses include use of force,  off-range safety, fire extinguisher use, and baton use. The remainder of  air marshals’ annual recurrent training hours must include courses within  each of the FAMS training disciplines, such as defensive measures,  firearms, mission tactics, and physical fitness. FAMS also requires air  marshals to pass quarterly firearms qualifications, complete biannual  fitness assessments, and pass periodic medical exams, which are  discussed later in this report.
section_title: Overview of Training Evaluation
paragraphs: In October 2010, DHS issued its Learning Evaluation Guide to help the  department’s learning and development community evaluate the  effectiveness of its training activities. Among other things, the guidance  identifies the Kirkpatrick model—a commonly accepted training evaluation  model that is endorsed by the Office of Personnel Management in its  training evaluation guidance—as a best practice. This model is commonly  used in the federal government. The Kirkpatrick model consists of a four- level approach for soliciting feedback from training course participants  and evaluating the impact the training had on individual development,  among other things. The following is a description of what each level  within the Kirkpatrick model is to accomplish:
Level 1: The first level measures the training participants’ reaction to,  and satisfaction with, the training program. A level 1 evaluation could  take the form of a course survey that a participant fills out immediately  after completing the training.
Level 2: The second level measures the extent to which learning has  occurred because of the training effort. A level 2 evaluation could take  the form of a written exam that a participant takes during the course.
Level 3: The third level measures how training affects changes in  behavior on the job. Such an evaluation could take the form of a  survey sent to participants several months after they have completed  the training to follow up on the impact of the training on the job.
Level 4: The fourth level measures the impact of the training program  on the agency’s mission or organizational results. Such an evaluation  could take the form of comparing operational data before and after a  training modification was made.
section_title: TSA Assesses Air Marshals’ Training Needs by Considering Several Information Sources, but Opportunities Exist to Obtain More Feedback from Air Marshals
subsections: section_title: TSA Assesses Its Federal Air Marshal Training Program Using Multiple Information Sources
paragraphs: TSA’s primary method for assessing air marshals’ training needs is by  holding Curriculum Development Conferences (CDC) and Curriculum  Review Conferences (CRC). Specifically, OTD holds CDCs to determine  whether to approve proposals to develop new training courses, and  convenes CRCs to evaluate the effectiveness of existing FAMTP courses  and, if appropriate, to make recommendations to address any identified  shortcomings. These conferences are composed of OTD officials  responsible for developing and implementing FAMTP courses and  relevant subject matter experts, such as training instructors, field office  Supervisory Air Marshals-in-Charge (SACs), SFAMs, and air marshals.  According to OTD guidance and consistent with Federal Law  Enforcement Training Accreditation Board standards, CDCs are to be  held prior to the development of new training programs and CRCs are to  be held no less than every three years or sooner if directed by FAMS  management. CDCs can also be held in response to directives from  OTD or FAMS management and to requests for additional training from  FAMS personnel.
As part of the CDCs and CRCs, OTD conducts assessments to determine  the extent that existing FAMTP courses are current with existing or  planned FAMS policy, procedures, or new equipment or technology, and  address the known threat environment, and air marshals’ training  needs. When doing so, OTD considers various sources of information  including, among others, its job task analysis; training-related concerns  raised by field office focus groups; feedback from air marshal  candidates, training instructors and other subject matter experts; and  intelligence. According to OTD guidance, this information is primarily to  be gathered as described below.
TSATC Training Evaluation Surveys: OTD’s student critique review  program evaluates FAMTP training courses delivered at TSATC  consistent with Kirkpatrick levels one and three. Under this program,  OTD solicits and reviews feedback from air marshal candidates on the  quality of the FAMTP courses that they complete at TSATC and from  newly graduated air marshals on the extent that these courses  effectively prepare them for their duties. Specifically, consistent with  Kirkpatrick level 1, OTD requires air marshal candidates to complete a  course evaluation on the effectiveness of the course and the quality of  the instructor and facility immediately after completing the course.  Further, consistent with Kirkpatrick level 3, TSATC surveys newly  graduated air marshals 10 to 12 months after they have graduated  from FAMTP-II, and their supervisors within 12 months of their  graduation, to obtain their feedback on the extent that the training  adequately prepared the FAMTP graduates to successfully perform  their mission. In addition, the program provides OTD with feedback  from air marshal candidates and newly graduated air marshals on the  effectiveness of FAMTP curriculum, instructor performance, and  TSATC facility or safety, or other related issues. This feedback is  used by CDCs and CRCs to identify training gaps and determine how  to appropriately address them. However, as described later in this  report, response rates by air marshals on these surveys have been  low.
TSATC Examinations and Simulations: Consistent with Kirkpatrick  level 2, OTD requires air marshal candidates to pass written exams or  job simulations in order to advance through FAMTP. Specifically, air  marshal candidates must demonstrate that they possess the  knowledge, cognitive, or physical skills that classroom courses are  intended to impart by passing examinations. OTD has developed  evaluation tools, such as checklists, that TSATC training instructors  must use to objectively determine air marshal candidates’ proficiency  in law enforcement tactics and techniques such as marksmanship,  defensive tactics, arrest procedures, and decision-making. OTD  collects and analyzes the data on newly hired air marshals’  performance to determine the extent to which air marshal candidates  have mastered the learning objectives with each FAMTP course and  to identify any areas in the curriculum that may need revision. For  example, OTD officials stated that they may revise examination  questions in response to a relatively high number of air marshal  candidates failing a question or series of questions due to poor  wording. Furthermore, OTD uses these data to identify and address  any training needs not met by the existing FAMTP curriculum when  carrying out CRCs and CDCs.
In addition to using surveys and examinations when evaluating FAMTP  curriculum provided at TSATC, OTD officials noted additional information  sources they use when evaluating FAMTP curriculum, including field  office training assessment teams and quarterly training teleconferences.
Field Office Training Assessment Teams: OTD established field  office assessment teams, which consist of TSATC instructors, to  assess field office training programs and their instructors. As  described earlier, field office training programs primarily provide the  recurrent training that incumbent air marshals are required to fulfill  each year. In advance of the assessment team’s visit, TSATC sends  surveys to supervisors and air marshals in the field with questions on  the effectiveness of the field office’s training program, including its  training instructors and facilities, as well as FAMS’s training  curriculum. According to OTD officials, when conducting assessments  at the field offices, team members are to observe field office trainers  in class to ensure that FAMTP courses are taught uniformly across all  FAMS field offices. They also are to review the field office’s training  records and policies and procedures to ensure the field office’s  training program is in compliance with OTD and FAMS policies, and  when necessary, to make recommendations for improvement. For  example, OTD officials told us that an assessment team discovered a  field office whose training staff were using unapproved “dynamic  fighting” tactics to teach air marshals how to fend off multiple  attackers when cornered, which had resulted in many severe injuries.  In this case, the assessment team halted use of the unapproved  scenarios and provided approved lesson plans that taught air  marshals to counter multiple attackers. OTD officials stated that  training assessment team visits also provide opportunities for TSATC  trainers to engage directly with field office trainers and air marshals to  share new best practices and identify any unmet training needs.  However, as we discuss later in this report, OTD has not sent  assessment teams to evaluate field office training programs since  March 2013.
Quarterly Training Teleconferences: OTD holds quarterly  conference calls between TSATC staff, FAMS headquarters training  staff, and field office training staff to discuss service-wide training  issues. According to OTD officials, these teleconferences provide  opportunities to elicit feedback from trainers on unmet training needs  and any challenges in delivering training, and to share best practices  among the field offices.
section_title: OTD Could Improve its Collection of Feedback from Air Marshals to Fully Identify and Address Training Needs
subsections: section_title: OTD Does Not Systematically Collect Training Feedback from Incumbent Air Marshals on Field-Based Training
paragraphs: OTD conducts surveys to obtain feedback from air marshal candidates  and newly graduated air marshals on the effectiveness of FAMTP  courses they complete at TSATC and the quality of TSATC trainers and  facilities, consistent with Kirkpatrick levels 1 and 3. However, OTD does  not also obtain such feedback from incumbent air marshals after they  complete their recurrent training courses at their respective field offices.
Our previous work on federal training programs, as well as DHS’s  Learning Evaluation Guide, has found that implementing a balanced  multi-level systematic approach to evaluate and develop training, such as  the Kirkpatrick model, can provide agencies with varied data and  perspectives on the effectiveness of training efforts necessary to identify  problems and improve training and development programs as needed. In  addition, our work has also shown that agencies should ensure that they  incorporate a wide variety of stakeholder perspectives in assessing the  impact of training on employee and agency performance.
OTD officials stated that conducting level 1 and 3 evaluations for air  marshal candidates and newly graduated air marshals has provided  sufficient feedback to reliably identify all air marshals’ training needs  because the agency has taken steps to ensure that the content and  quality of training for air marshals candidates is identical to that of  recurrent training for incumbent air marshals. However, FAMS did not hire  any new air marshals from fiscal years 2012 through 2015. As a result,  TSA has not systematically gather feedback on the effectiveness of  FAMTP training curriculum from air marshals for approximately four  years. Over this time period, OTD has revised the training curriculum,  such as adding a course on personal security when overseas and  expanding the number of courses within the legal and investigative  discipline to cover all transportation modes. Moreover, while the minimum  skill requirements may be the same for both air marshal candidates and  incumbent air marshals in the field, the training needs for both groups  may not necessarily be identical. With greater experience in carrying out  missions, incumbent air marshals may have a better idea of their training  needs than air marshal candidates or newly graduated air marshals,  which could result in more experienced incumbent air marshals providing  different feedback on the quality of the training. Further, although  incumbent air marshals take many of the same training courses as air  marshal candidates, they do so at different facilities and with different  instructors.
OTD officials also stated that field office training assessments and  quarterly training teleconferences provide additional opportunities to both  ensure that the training all air marshals receive is standardized across the  service and to obtain incumbent air marshal feedback. However, OTD  has not sent assessment teams to evaluate field office training programs  since March 2013 due, in part, to a lack of resources. OTD officials  reported that they plan to resume field office training assessments during  fiscal year 2017 and conduct assessments at 10 FAMS field offices per  year if sufficient funding is available. These officials also reported that  OTD plans to increase the frequency of training teleconferences between  TSATC and field office training programs from a quarterly to monthly  basis and invite field office leadership—SACs and Assistant Supervisory  Air Marshals-in-Charge (ASACs)—to participate in these meetings.
Nevertheless, our review suggests that OTD could benefit from  broadening its efforts to gather feedback on recurrent training courses.  First, field office staff we interviewed at the seven field offices we visited  stated that improvements to training could better prepare them for their  roles. For example, SFAMs and training staff in four of the seven field  offices we visited stated that the training curriculum is overly focused on  the training needs of air marshal candidates and newly graduated air  marshals. Staff from five of the seven field offices also identified  advanced training courses beyond those currently provided that they  believed should be offered to incumbent air marshals, in areas such as  firearms, defensive, or medical training. Second, field office staff at all  seven field offices we visited identified training that should be revised,  expanded, or added, to include topics such as active shooter response,  counter surveillance and behavior detection techniques, training on  improvised explosive devices and other explosives, and expanded legal  and investigative training, among others. These sources also told us that  the curriculum did not adequately address changes in their  responsibilities over time, which include a broader set of current threats  such as improvised explosive devices or FAMS-specific training on active  shooters. OTD officials stated that they believed the current FAMTP  curriculum adequately addresses the types of additional training that field  office staff identified and that the curriculum has been designed to meet  the needs of air marshals at all experience levels and may be consistently  and safely delivered to the entire workforce. However, without a  mechanism to systematically collect and incorporate feedback on field- based training for incumbent air marshals, consistent with Kirkpatrick  level 1 and 3, OTD could miss important opportunities to identify  problems and improve overall training and development.
section_title: Response Rates for Air Marshal Training Surveys Have Been Consistently Low
paragraphs: When OTD administered surveys to obtain feedback on the FAMTP-II  and field-based training, the response rates were substantially lower than  the 80 percent rate OMB encourages for federal surveys that require its  approval. Specifically, about 19 to 38 percent of air marshals that  graduated from FAMTP-II and their supervisors responded to the surveys  that TSATC administered from 2009 through 2011—the last 3 full years in  which FAMS hired air marshals. Additionally, according to OTD officials,  the combined response rates for the surveys that training assessment  teams conducted from June 2012 through March 2013 was about 16  percent. OTD staff acknowledged that the response rate to these surveys  have been consistently low, but stated that the low response rates have  not significantly affected the usefulness of the surveys. According to OTD  staff, with regard to the FAMTP-II surveys, they received a sufficient  number of responses to successfully evaluate the extent that FAMTP  courses have met all air marshals’ training needs. However, OMB  guidance stipulates that agencies must design surveys to achieve the  highest practical rates of response to ensure that the results are  representative of the target population and that they can be used with  confidence as input for informed decision-making. The guidance also  states that response rates are an important indicator of the potential for a  bias called nonresponse bias, which could affect the accuracy of a  survey’s results. In general, as a survey’s response rate increases, the  likelihood of a bias problem decreases, and, therefore, the views and  characteristics of the target population are more accurately reflected in  the survey’s results.
OMB guidance also describes the methods agencies can use to improve  the response rate of future surveys, including conducting outreach to  groups of prospective respondents, ensuring that the survey is well- designed and brief, providing alternative modes to provide responses,  conducting nonresponse follow-up efforts, and extending cut-off dates for  survey completion. OTD officials reported that they have taken several  of these actions to improve the response rates of the FAMTP-II surveys,  but have had little success in improving their response rate. Specifically,  officials stated that TSATC instructors and staff discussed the surveys  and their importance to improving future course offerings in class. In  addition, OTD officials reported designing the survey to be as brief as  possible, making it accessible via the internet and air marshals’ handheld  devices, sending out follow-up reminders to survey respondents via   e-mail and telephone, and contacting non-respondents’ field office  supervisors. OTD officials told us that the low response rates may be  attributable to “survey fatigue” given the high number of surveys that TSA  employees are asked to complete and stated that there was little more  that they could do to persuade air marshals to respond.
Although OTD officials reported taking several of the actions that OMB  recommends for agencies to improve survey response rates, additional  actions could improve the response rate of future OTD surveys, including  those administered to the air marshals FAMS hired this year. For  example, monitoring future survey response rates by field office could  help OTD identify and then target extra follow-up efforts to air marshals  and their supervisors in field locations that have comparatively low  response levels. Further, extending the cut-off date for air marshals and  their supervisors to respond to the survey, or requiring survey  respondents to complete the surveys, could help improve response rates  to future surveys. Until OTD achieves sufficient response rates, OTD  cannot be reasonably assured that the feedback it received represents  the full spectrum of views held by air marshals or their supervisors.  Achieving an adequate response rate is important, particularly as FAMS’s  CRC and CDC processes rely, in part, on the survey results to identify  training gaps and determine how to appropriately address them.
section_title: TSA Has Established Recurrent Training Requirements to Ensure Mission Readiness, but Opportunities for Improvement Exist
paragraphs: FAMS relies on its recurrent training program to help ensure incumbent  air marshals’ mission readiness, but additional actions could strengthen  FAMS’s ability to do so. First, FAMS does not have complete and timely  data on the extent to which air marshals have fulfilled their recurrent  training requirements. Second, FAMS evaluates incumbent air marshals’  proficiency in some, but not all, key skills using tools such as  examinations or checklists. In addition, FAMS has established a new  health, fitness, and wellness program as part of its recurrent training  program—in part to address recent concerns with air marshals’ fitness  and injury rates—but it is too early to gauge the program’s effectiveness.
subsections: section_title: FAMS Lacks Complete and Timely Data to Ensure Air Marshals’ Compliance with Recurrent Training Requirements
paragraphs: As shown in figure 2, FAMS requires air marshals to complete certain  recurrent training requirements on a regular basis to ensure that air  marshals maintain their proficiency in the knowledge, skills, and abilities  that are needed to successfully carry out FAMS’s mission. However,  FAMS does not have complete and timely data to ensure air marshals’  compliance with these training requirements.
Senior OTD and FAMS officials responsible for developing and  overseeing the recurrent training program, as well as field office SFAMs,  training instructors, and air marshals at the field offices we visited,  identified the importance of the FAMS training program to ensuring air  marshals’ mission readiness. These personnel stated that air marshals  are unique among their fellow law enforcement officers because air  marshals lack regular on-the-job opportunities to actively utilize the  knowledge, skills, and abilities they develop in training courses to address  a key aspect of FAMS’s mission—defeating terrorist or other criminal  hostile acts. Therefore, according to OTD and FAMS officials, FAMS  ensures air marshals’ mission readiness by monitoring the extent to which  they have completed their recurrent training requirements.
According to FAMS policy, field office SACs or their designees are  responsible for ensuring that air marshals assigned to them have  completed their recurrent training requirements and that the completion of  these requirements is recorded in FAMS’s database—Federal Air  Marshal Information System (FAMIS)—no later than 5 days after an air  marshal has completed a training requirement. FAMS headquarters  personnel within the Field Operations Division (Field Operations)  generate reports in FAMIS detailing the extent to which air marshals have  passed the practical pistol course, participated in physical fitness  assessments, and completed their requisite number of recurrent training  hours on a quarterly and annual basis. According to Field Operations  staff, these personnel contact field office SACs or their designees when  these reports identify air marshals that have not met their recurrent  requirements. If field office staff report that the air marshal(s) have  completed a requirement(s), but have not entered this information in  FAMIS, Field Operations is to request appropriate documentation and  update FAMIS. Field Operations officials stated they discuss with field  offices why any air marshals have not completed their training  requirements, such as illnesses, injuries, or scheduling issues, and, if  necessary, the field office SAC is to take appropriate action.
In addition, FAMS policy allows for air marshals to be exempted from  training requirements when certain conditions, such as illness, injury, or  military leave, are met and defines the process by which exemptions are  to be requested and granted. Specifically, FAMS policy states that  SACs must prepare a letter to the appropriate regional director to request  approval of the exemption no later than 5 days after the end of a quarter.  Field Operations officials reported that a FAMS headquarters staff person  records the exemption into FAMIS once a regional director has approved  the request.
FAMS has processes for field office SACs to monitor which air marshals  have completed their required recurrent training each year, as well as  those who have received exemptions from such training. However, we  found that the data used to track this information were not complete or  readily available for purposes of tracking air marshals’ compliance with  these requirements when we requested these data in March 2015. We  reviewed training data from FAMIS’s training module for calendar year  2014 to determine the extent that air marshals have met their recurrent  training requirements. Although we were ultimately able to determine  that almost all of the air marshals met their training requirements or  received an appropriate exemption in calendar year 2014, it was difficult  to do so because data on both approved exemptions and training  completions were missing or had not been entered in a timely manner.  We found that nearly one-third of all training exemptions granted to air  marshals in calendar year 2014 had not been entered into FAMIS.  Specifically, at least 299 training exemptions granted to about 2 percent  of air marshals had not been entered into FAMIS when we received the  data in March 2015—nearly three months after the calendar year had  ended. Additionally, we found that nearly one-quarter of all training  records for calendar year 2014 had been entered into FAMIS more than 5  days after an air marshal had completed the training.
FAMS headquarters officials responsible for reconciling recurrent training  service-wide stated that these exemptions were not entered into FAMIS  until July 2015—seven months after the calendar year ended. These  officials told us that the delay was partly because FAMS took the  database offline for three weeks in September 2014 to allow for an  upgrade of the system. As a result, the staff person responsible for  entering exemptions had become backlogged and later entered the  backlogged exemptions into the database, in part, to reconcile the  missing exemptions that were identified through our analysis of the 2014  training data. Additionally, FAMS officials responsible for reconciling  completion of recurrent training service-wide reported that each quarter  there are a significant number of air marshals for whom field office staff  have not entered training records. According to these officials, at the end  of every quarter, FAMS Field Operations staff must contact staff from  several field offices to remind them to review and enter missing training  records—a process that officials described as labor-intensive. In  December 2015, FAMS officials provided us with the updated records for  the air marshals whose exemptions had been entered into FAMIS as a  result of our audit work to demonstrate that the air marshals’ 2014  recurrent training data had been corrected and were complete.
TSA Office of Inspection (OOI) reports have found similar problems with  monitoring, or timely and accurate recording, of air marshals’ training  records. Specifically, OOI inspections of FAMS’s field offices completed  during 2010 through 2015 found that three field offices had not accurately  recorded air marshals’ training data or done so in a timely manner— issues FAMS had not identified through its training monitoring process.
FAMS processes for monitoring the extent that air marshals service-wide  have completed their recurrent training requirements have not ensured  that air marshals’ training data are entered in a timely manner. These  processes, as defined in FAMS policy, lack effective controls to ensure  accountability. Specifically, FAMS has not specified in policy who has  oversight responsibility at the headquarters level for ensuring that each  field office has entered recurrent training data in a timely manner.  Additionally, FAMS has not specified in policy who has oversight  responsibility at the headquarters level for ensuring that headquarters  personnel have entered air marshals exemptions into FAMIS within a  defined timeframe.
Federal Standards for Internal Control states that agencies should ensure  that transactions and events are completely and accurately recorded in a  timely manner, and are readily available for examination. Federal  regulations require that agencies establish policies governing employee  training including the assignment of responsibility to ensure the training  goals are achieved. In addition, internal control standards state that in a  good control environment, areas of authority and responsibility are clearly  defined and appropriately documented through its policies and  procedures, and appropriate lines of reporting are established.
Given the number of training records that we found were incomplete or  not entered into FAMIS in a timely manner, as well as the ongoing  challenges that FAMS has faced in ensuring accurate and timely input of  training and exemptions data as described in the OOI findings, policies  that specify who is responsible at the headquarters level for overseeing  these activities could help FAMS ensure its data on air marshals’  recurrent training are consistently accurate and up to date. Complete and  readily available training and exemptions data would enable FAMS to  more effectively determine the extent that air marshals service-wide have  met their training requirements and are mission ready.
section_title: FAMS Requires Air Marshals to Demonstrate Their Proficiency in Marksmanship, but Does Not Require That They Do So for Other Key Skills
paragraphs: Air marshals must demonstrate their proficiency in marksmanship by  taking the practical pistol course on a quarterly basis and achieving a  minimum score of 255 out of 300—the highest qualification standard for  any federal law enforcement agency, according to FAMS officials.  However, for the remainder of air marshals’ required recurrent training  courses, FAMS does not assess air marshals against a similarly identified  level of proficiency, such as by requiring examinations to evaluate air  marshals’ knowledge in classroom-based courses or by using checklists  or other objective tools to evaluate air marshals’ performance during  simulation-based courses, such as mission tactics. For instance, FAMS’s  recurrent training includes both mandatory refresher courses that all air  marshals must complete annually as well as a broad set of courses within  several disciplines that field offices must ensure are incorporated into  their annual or quarterly training plans. However, FAMS does not  require air marshals to take an examination for any course within these  disciplines.
Federal Standards for Internal Control states that agencies should  establish expectations of competence for key roles, such as federal air  marshals, to help the entity achieve its objectives, and that all personnel  need to possess and maintain the relevant knowledge, skills, and abilities  that allow them to accomplish their assigned duties. Additionally, GAO’s  prior work on training and development states that in some cases,  agencies may identify critical skills and competencies that are important  to mission success, and require that employees meet requirements to  ensure they possess needed knowledge and skills. Further, DHS’s  Learning Evaluation Guide identifies testing or skill checklists as tools  agencies can use to determine whether students have the knowledge and  can perform the skills classes are designed to teach. The guide also  states that learning activities that are skill-based, such as FAMS courses  on tactical and defensive techniques, may require the development of skill  checklists to determine the level of trainee proficiency.
Field Operations officials said that it is not necessary to use examinations  for recurrent training courses because air marshals are continuously  evaluated by field office training instructors and SFAMs who participate in  their training. In addition, officials stated that air marshals demonstrate  their proficiency in the various cognitive or physical skills they must  possess during simulations conducted as part of FAMS’s recurrent  training program. As a result, according to officials, FAMS can be assured  that any gaps in air marshals’ performance are identified and addressed  in a timely manner. Field Operations officials also stated that checklists  are also unnecessary because training instructors do not evaluate air  marshals’ performance solely on whether their actions were appropriate  and the air marshal correctly applied the relevant principles or tactics  taught by course simulations. Rather, air marshals must also articulate  why their actions were appropriate and how they applied the relevant  principles or tactics. For example, to evaluate air marshals’ performance  in mission tactics simulations, Field Operations officials stated that  training instructors observe air marshals’ actions in response to various  simulated threats ranging from verbal or physical assaults to the crew by  passengers to suicide bombers. According to FAMS officials, training  instructors evaluate the extent the actions taken by air marshals resulted  in positive outcomes (i.e., protected the plane, passengers, and crew)  and were carried out in accordance with applicable authorities, policies,  procedures, and principles. Officials stated that part of this assessment is  based on air marshals’ explanation for why their actions appropriately  addressed the simulated threat and applied relevant FAMS principles and  tactics. In addition, TSA has established a training instructor training  program, which, according to FAMS Field Operations officials, ensures  that training instructors are highly trained and certified; and, therefore,  can assess air marshals’ performance in a reasonably objective manner.
As previously discussed, for training courses taught at TSATC, OTD  requires air marshal candidates and incumbent air marshals to  demonstrate that they possess the knowledge or cognitive skills that  classroom courses are intended to impart by passing examinations for  training courses taught at TSATC. Additionally, when evaluating the  performance of air marshal candidates and incumbent air marshals in  courses taught at TSATC, OTD requires training instructors to use  evaluation tools, including checklists. For example, TSATC training  instructors must use these tools when evaluating air marshal candidates’  performance in defensive measures and mission tactics simulations as  part of FAMTP-II. TSATC staff reported that they require TSATC training  instructors to use such checklists because doing so better ensures air  marshals are evaluated in an objective, fair, and consistent manner.  Further, a field office SAC reported that given the absence of an objective  tool for assessing air marshals’ performance in field-based training, such  as defensive measures and tactics, there are air marshals who have not  fully demonstrated the requisite level of proficiency, but still “passed”  these courses and continued to fly missions. According to the SAC, air  marshals are flying missions with colleagues they do not view as mission  ready in part due to their performance in training courses—a concern  raised by air marshals in 3 of the 7 field offices we visited. Finally, field  office trainers in 3 field offices reported that a standardized tool for  evaluating air marshals during training would help them to identify and  address trainee deficiencies.
FAMS Field Operations officials also noted that standardized  examinations or checklists during trainings are not necessary because  SFAMs have opportunities to continually assess their air marshals’  mission readiness by flying with their squads or attending training.  However, we found that SFAMs infrequently attend training with their  squads or accompany them on flying missions, although they are not  necessarily required to do so. The July 2014 FAMS Advisory Council  minutes state that the council unanimously agreed that a large population  of SFAMs do not fully participate in their air marshals’ required training.  Air marshals from 6 of 22 field offices raised similar concerns based on  our review of the minutes from field office focus groups conducted in  fiscal year 2014. In addition, SFAMs in all 7 field offices we visited  reported that they rarely fly with their squads, i.e., once per quarter or  less. Further, SFAMs in 6 of the 7 field offices stated that they rely on air  marshals’ self-assessments and factors unrelated to mission readiness,  such as quality of administrative paperwork (i.e., travel vouchers and  timecards), and completion of OLC training to assess air marshals’  performance.
Standardized methods for determining whether incumbent air marshals  are mission ready, such as required examinations or evaluation tools, in  key training courses could help provide better assurance that air marshals  service-wide are mission ready. Objective and standardized methods of  evaluating incumbent air marshals performance would better enable  FAMS to assess air marshals’ proficiency in key skills and also more  effectively target areas for improvement.
section_title: FAMS Developed a New Physical Fitness Program, but It Is Too Early to Gauge its Effectiveness
paragraphs: In 2015, FAMS developed a new physical fitness program—the Health,  Fitness, and Wellness Program—in part to address recent concerns with  air marshals’ fitness and injury rates, but it is too early to gauge the  program’s impact. Over the period 2008 to 2015, FAMS commissioned  two studies to evaluate air marshals’ health and fitness, as well as a third  study to evaluate air marshal fatigue and sleeplessness. FitForce, a  consulting group, which conducted the first evaluation of air marshals’  fitness in 2009, found that nearly 32 percent of the FAMS who  participated in the study exercised less than three times per week and  almost 7 percent did not exercise at all. FitForce also concluded that  physical fitness is a necessity for air marshals to be able to perform the  essential functions of their job, and stated that FAMS should make a  commitment to address the fitness needs of air marshals. Additionally, a  2012 sleep study conducted by Harvard University concluded that more  than half of the air marshals who responded to the study’s survey were  overweight and nearly one-third were obese, and, therefore, may suffer a  variety of health issues that could directly impact mission readiness.  Furthermore, FAMS conducted its own review of air marshals’ fitness  from 2012 through 2013 and concluded that air marshals suffered from  high injury rates and declining overall health and wellness, which FAMS  officials attributed in part to the increasing age of air marshals.  Specifically, the review found that the injuries that occurred while air  marshals took their physical fitness assessment from 2010 through 2013  had resulted in approximately 8,060 lost or restricted work days and  12,896 lost mission opportunities and Office of Workers’ Compensation  Program claims totaling over $1 million.
We analyzed the scores that air marshals achieved in calendar year 2014  when taking the quarterly Mission Readiness Assessment (MRA)—the  health evaluation program that FAMS had in place at that time. We found  that, with the exception of the 1.5-mile run, the majority of air marshals  who took the MRA met or exceeded each of the MRA component test  goals. In quarters 2 through 4 of calendar year 2014, 84 to almost 90  percent of air marshals who participated in the MRA failed to meet the  1.5-mile run goal, as shown in figure 3. Moreover, about 5 percent of the  air marshals did not meet the performance goals for any of the  component tests in quarters 2 through 4 of calendar year 2014.
To address the impact that air marshals’ declining health and fitness may  pose to FAMS’s ability to carry out its mission within TSA, as well as air  marshals’ injury rates, FAMS has developed the Health, Fitness, and  Wellness Program, which went into effect in April 2016. According to  FAMS policy, this program will include a revised fitness assessment—the  Health and Fitness Assessment (HFA)—and a general health and  wellness program. FAMS officials reported that air marshals are to  complete the HFA on a biannual basis, but will not be required to meet  performance goals for any of the HFA’s four components:  cardiorespiratory endurance, muscular strength, muscular endurance,  and flexibility. Rather, FAMS will use the results of air marshals’ first  HFA to establish a fitness baseline and to take appropriate action to  improve the performance of those who do not maintain their fitness levels  or show improvement. According to FAMS officials, the agency decided  not to require air marshals to meet the performance goals for the HFA  tests because the results of the HFA cannot reliably determine the extent  that an air marshal is physically capable of carrying out FAMS’s mission.  Officials explained that FAMS had originally intended to require  incumbent air marshals to meet a physical fitness standard similar to the  HFA, but did not do so because of concerns raised by TSA’s Office of  Human Capital and Office of the Chief Counsel. Specifically, according to  FAMS officials, advice was provided regarding whether the proposed  physical fitness standard could reliably predict an air marshal’s physical  ability to carry out FAMS job-related mission, as well as if FAMS could  demonstrate the business necessity (mission-related) of the standard.  Because of these concerns, FAMS’s leadership decided to implement the  Health, Wellness, and Fitness Program with a focus on reducing the  incidence of air marshals’ injuries, reducing the number of exemptions air  marshals needed to request from taking the HFA, increasing program  participation, and improving air marshals’ overall health and wellness  instead. FAMS officials stated that in addition to general improvement of  air marshals’ health and fitness, a key benefit of the new program will be  that air marshals will request and receive fewer exemptions because the  HFA will allow air marshals to demonstrate their fitness through  alternative means of testing. FAMS officials reported that, when taking the  HFA, air marshals may choose one of three exercises to perform for five  of the six subsets within the four components. For example, when taking  the upper body subset of the muscular strength component, air marshals  may choose to perform pull-ups, assisted pull-ups, or lateral pulldowns.  According to FAMS officials, because multiple exercises will be available  for each HFA component, FAMS will no longer grant air marshals  exemptions from taking the HFA unless an injury prevents them from  performing any of the HFA exercises.
FAMS has established a goal for the Health, Wellness, and Fitness  program—to provide the opportunity, resources, and education necessary  to enhance mission readiness and promote workplace wellness, but it is  too early to know if the program is achieving its intended goal. FAMS and  OTD officials responsible for developing this program told us that FAMS  plans to collect and analyze data on air marshals’ performance on the  HFA over a period of about 12 to 18 months—two or three assessment  periods. These officials stated that after FAMS had collected and  analyzed sufficient data and established a baseline, the agency would be  better positioned to collaborate with OTD to establish performance  measures for the program. In the interim, FAMS plan to monitor data such  as injury rates and the results of periodic physical exams.
section_title: Conclusions
paragraphs: Given the unique operating environment of air marshals, it is vital that  TSA ensure that air marshals’ training needs are identified and  addressed, and that air marshals are mission ready. TSA does not  systematically obtain feedback on the extent to which FAMTP courses  meet incumbent air marshals’ training needs because officials state that  they collect sufficient information from air marshal candidates on their  training programs. However, by regularly collecting incumbent air  marshals’ feedback on the recurrent training they receive in the field  offices, OTD would better ensure it considers the input and experience of  incumbent air marshals when assessing and refining their training  programs. Also, by taking additional steps to improve the response rates  for the training surveys it administers to air marshal candidates,  incumbent air marshals, and their supervisors, OTD could be more  reasonably assured that the feedback it receives represents the full  spectrum of views held by its air marshal workforce.
FAMS has established recurrent training requirements to ensure that air  marshals maintain the knowledge, skills and abilities needed to carry out  their mission. However, because FAMS processes have not ensured the  timely and complete recording of training data—an ongoing challenge for  FAMS—FAMS has been hindered in its ability to ensure air marshals’  compliance with training requirements. Specifying in policy who has  oversight responsibility at the headquarters level for ensuring that each  field office has entered air marshals’ training data in a timely manner and  that headquarters personnel have entered air marshals’ exemptions into  FAMIS could help FAMS better ensure its data on air marshals’ recurrent  training are consistently complete and up to date. Such a policy could  also enable FAMS to more effectively determine the extent that air  marshals service-wide have met their training requirements and are  mission ready. Additionally, by developing and implementing more  objective and standardized methods of determining, in the course of their  recurrent training, whether incumbent air marshals continue to be mission  ready, FAMS could better assess their skills and also more effectively  target areas for improvement.
section_title: Recommendations for Executive Action
paragraphs: To ensure effective evaluation of air marshal training, we recommend that  the TSA Administrator direct OTD to take the following two actions:  implement a mechanism for regularly collecting and incorporating  incumbent air marshals’ feedback on the training they receive from  field office programs, and  take additional steps to improve the response rates of the training  surveys it conducts.
To provide reasonable assurance that air marshals are complying with  recurrent training requirements and have the capability to carry out  FAMS’s mission, we recommend the TSA Administrator direct FAMS to  take the following three actions:  specify in policy who at the headquarters level has oversight  responsibility for ensuring that field office SACs or their designees  meet their responsibilities for ensuring that training completion records  are entered in a timely manner,   specify in policy who at the headquarters level is responsible for  ensuring that headquarters personnel enter approved air marshals’  training exemptions into FAMIS, and define the timeframe for doing  so, and develop and implement standardized methods, such as examinations  and checklists, for determining whether incumbent air marshals  continue to be mission ready in key skills.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for comment. In its written  comments, reproduced in appendix II, DHS concurred with the five  recommendations and described actions under way or planned to  address them. DHS also provided technical comments that we  incorporated, as appropriate.
With regard to the first recommendation to implement a mechanism for  regularly collecting and incorporating incumbent air marshals' feedback  on the training they receive from field office programs, DHS concurred  and stated that TSA has developed a survey to measure the  effectiveness of air marshal training curriculum, field office training  personnel, and training facilities. DHS also stated that this survey will be  added to the TSA On-Line Learning Center where it can be distributed to  air marshals and supervisors on a regular basis. According to DHS, TSA  implemented the survey in the On-Line Learning Center in July 2016 and,  beginning in October 2016, will send the survey to air marshals and  supervisors after they complete a course at TSATC. TSA also plans for  curriculum development and review committees to use the feedback from  these surveys to improve courses offered at TSATC. These actions, if  implemented effectively, should address the intent of our  recommendation.
With regard to the second recommendation to take additional steps to  improve the response rates of the training surveys it conducts, DHS  concurred and stated that future surveys of FAMTP graduates and their  supervisors will be distributed to personnel through the On-Line Learning  Center. DHS stated that the capabilities of the On-Line Learning Center  will provide a tracking mechanism for program managers to ensure that  personnel complete and submit the survey. According to DHS, survey  reports will be compiled and sent to TSATC in a manner that maintains  the anonymity of the respondent. TSA anticipates that this process will  significantly improve response rates. These actions, if implemented  effectively, should address the intent of our recommendation.
DHS concurred with our third and fourth recommendations that FAMS  specify in policy (1) who at the headquarters level has oversight  responsibility for ensuring that field office SACs or their designees meet  their responsibilities for ensuring that training completion records are  entered in a timely manner, and (2) who at the headquarters level is  responsible for ensuring that headquarters personnel enter approved air  marshals' training exemptions into FAMIS and define the timeframe for  doing so. In response to our recommendations, FAMS updated its policy  on recurrent training requirements for air marshals to assign Regional  Directors, who are based in headquarters, the responsibility for ensuring  that field office SACs or their designees adhere to FAMS’s procedures for  recording training completion. The updated policy also requires that  FAMS’s Field Operations Division, Tactical Support Section, verify that  FAMIS entries are made for all training exemptions within five business  days of the approval of the exemptions. These actions, if implemented  effectively, should address the intent of our recommendations.
With regard to the fifth recommendation to develop and implement  standardized methods, such as examinations and checklists, for  determining whether air marshals continue to be mission ready in key  skills, DHS concurred and stated that FAMS and OTD established a joint  Integrated Project Team/Development Committee, which met in June  2016 to develop an assessment process that will be used to determine air  marshals’ mission readiness. According to DHS, the joint Integrated  Project Team/Development Committee consisted of representatives from  seven FAMS field offices and FAMS headquarters as well as instructors  and instructional design specialists from TSATC. DHS stated that the  Integrated Project Team is drafting recommendations and that approved  readiness measures will be implemented beginning in fiscal year 2018.  This action, if implemented effectively, could address the intent of our  recommendation. However, it is not clear to what extent this assessment  process will include standardized methods for determining whether  incumbent air marshals continue to be mission ready. We will continue to  monitor TSA’s efforts.
We are sending copies of this report to appropriate congressional  committees, the Secretary of Homeland Security, the TSA Administrator,  and other interested parties. In addition, this report will be available at no  charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7141 or groverj@gao.gov. Contact points for our Office  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff that made key contributions to this report  are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report addresses the following questions:
How does the Transportation Security Administration (TSA) assess  the training needs of air marshal candidates and incumbent air  marshals, and what opportunities exist, if any, to improve this  assessment?
To what extent does the Federal Air Marshal Service (FAMS) ensure  that incumbent air marshals are mission ready?
This report is a public version of the prior sensitive report that we  provided to you. TSA deemed some of the information in the report as  sensitive security information, which must be protected from public  disclosure. Therefore, this report omits this information, such as specific  numbers of air marshals and specific types of training that air marshals  reported needed to be added to FAMS’s training curriculum to address  changes in air marshals’ responsibilities. Although the information  provided in this report is more limited in scope in that it excludes such  information, it addresses the same questions as the sensitive security  information report and the methodology used for both reports is the same.
To address the first objective, we reviewed TSA directives, guidance, and  other relevant documentation describing TSA’s processes for developing  and evaluating Federal Air Marshal Training Program (FAMTP) training  curriculum to determine how TSA evaluates existing courses and  develops new courses within FAMTP and other relevant training  programs. We interviewed senior officials responsible for these efforts in  TSA’s Office of Training and Development (OTD). We also analyzed  documentation on the results of training curriculum assessments OTD  conducted to identify recommendations made to improve training and the  extent to which OTD implemented the recommendations. OTD conducted  these assessments from May 2007 through April 2014. We compared  OTD’s training development and evaluation processes to key principles  identified in DHS guidance on training evaluation, and GAO’s prior work  on training and development, specifically the Guide for Assessing  Strategic Training and Development Efforts in the Federal Government.  We also reviewed the minutes of the quarterly teleconferences held in  fiscal years 2014 through 2015—the most recent time period for which  the meeting minutes were available—between Transportation Security  Administration Training Center (TSATC) staff, FAMS headquarters staff,  and field office training staff to determine the types of issues discussed  during these meetings. Additionally, we obtained the available response  rates for surveys OTD conducted of FAMTP graduates and their  supervisor’s on the effectiveness of FAMTP courses for calendar years  2009 through 2011—the last three full years that FAMS hired air  marshals. In addition, we met with OTD officials to discuss the actions  that had been taken to improve these response rates, and compared  these actions to Office of Management and Budget standards and  guidance for conducting surveys. Further, we visited the TSATC in  Atlantic City, New Jersey and 7 of FAMS 22 field offices, which we  selected, in part, to reflect a range in size (as determined by the number  of air marshals assigned to the office) and geographic dispersion. At  TSATC, we interviewed TSATC management and training instructors and  toured the facility. At the field offices, we interviewed field office  management, Supervisory Federal Air Marshals (SFAM), air marshals,  and training instructors to obtain their views on the current training  curriculum. The results of these interviews cannot be generalized to all  field offices, but provide insight into the extent to which TSA is addressing  air marshals’ training needs and ensuring their mission readiness.
To address the second objective, we assessed FAMS directives that set  forth training requirements for incumbent air marshals, and analyzed air  marshals training data for calendar year 2014, which is the most recent  year for which training data were available, to determine the extent to  which air marshals met these requirements. We interviewed senior FAMS  officials to understand how FAMS uses this information to ensure that air  marshals are mission ready. We compared the results of our analyses to  Standards for Internal Control in the Federal Government, the DHS  Learning Evaluation Guide, and GAO’s prior work on training and  development. We assessed the reliability of the 2014 training data by (1)  reviewing documentation on the processes for entering air marshals’  training records into the Federal Air Marshal Information System (FAMIS),  (2) performing electronic testing for obvious anomalies and comparing  FAMIS data to FAMIS-generated reports on training completion, and (3)  interviewing knowledgeable officials about training records and  exemptions entered into FAMIS. Although the data FAMS originally  provided were not complete or entered in a timely manner, over the  course of our audit we identified missing data that FAMS corrected in  response to our inquiries. Therefore, we found the data were reliable for  the purposes of our report. Additionally, as previously discussed, we  interviewed TSATC training instructors and FAMS field office personnel to  obtain their perspectives on FAMS methods for ensuring that air marshals  are mission ready. We also reviewed the most recent Management  Assessment Program inspection report completed by TSA’s Office of  Inspections for all 22 field offices to identify the training-related findings.  Additionally, we reviewed the FAMS Advisory Council minutes and the  field offices’ focus group minutes for fiscal year 2014—the most recent full  year of information available at the time of our request—to identify the  training related issues that FAMS personnel raised to their leadership.  Finally, we reviewed the studies that FAMS conducted or commissioned  to inform its development of its physical fitness program and  assessment—a component of air marshals training. We interviewed  FAMS and OTD officials responsible for developing and implementing  FAMS’s health, wellness, and fitness program to determine how TSA  plans to measure the effectiveness of the program.
We conducted this performance audit from October 2014 to September  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Maria Strudwick (Assistant  Director) and Michael C. Lenington (Analyst-in-Charge) managed this  assignment. Jonathan Bachman, Claudia Becker, Juli Digate, Michele  Fejfar, Imoni Hampton, Eric Hauswirth, Susan Hsu, Thomas Lombardi,  and Minette Richardson made key contributions to this report.