Aviation Security: Flight and Cabin Crew Member Security Training Strengthened, but Better Planning and Internal Controls Needed
Published: 2005-09-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Training flight and cabin crew members to handle potential threats against domestic aircraft is an important element in securing our nation's aviation system. The responsibility for ensuring that crew members are prepared to handle these threats is a shared responsibility between the private sector--air carriers--and the federal government, primarily the Transportation Security Administration (TSA). This report addresses (1) actions TSA has taken to develop guidance and standards for flight and cabin crew member security training and to measure the effectiveness of the training, (2) how TSA ensures domestic air carriers comply with the training guidance and standards, and (3) efforts TSA has taken to develop and assess the effectiveness of its voluntary self-defense training program.
section_title: What GAO Found
paragraphs: Since the terrorist attacks of September 11, 2001, TSA enhanced guidance and standards for flight and cabin crew member security training with input from stakeholders. Specifically, TSA revised the guidance and standards to include additional training elements required by law and to improve the organization and clarity of the guidance and standards. Some stakeholders we interviewed and our own review generally found that the revised guidance and standards improved upon previous versions in terms of organization and clarity of the information provided. However, some stakeholders identified concerns about, for example, the reasonableness of applying parts of the guidance and standards to both flight and cabin crew members and the difficulty in implementing some of the standards without additional information or training tools from TSA. Additionally, TSA has not established strategic goals and performance measures for assessing the effectiveness of the training because it considers its role in the training program as regulatory. In this regard, TSA views the individual air carriers as responsible for establishing performance goals and measures for their training programs, but has not required them to do so. Without goals and measures, TSA and air carriers will be limited in their ability to fully assess accomplishments and target associated improvements. TSA recently took steps to strengthen its efforts to oversee air carriers' flight and cabin crew security training to ensure they are complying with the required guidance and standards. For example, in January 2005, TSA added staff with expertise in designing training programs to review air carriers' crew member security training curriculums and developed a standard form for staff to use to conduct their reviews. However, TSA lacks adequate controls for monitoring and reviewing air carriers' crew member security training, including written procedures for conducting and documenting these reviews. TSA plans to develop written procedures, but has not established a timeframe for completing this effort. TSA has developed an advanced voluntary self-defense training program with input from stakeholders and implemented the program in December 2004, as required by law. However, stakeholders and our own analysis identified concerns about the training design and delivery, such as the lack of recurrent training and the lack of a realistic training environment. Also, TSA has not yet established performance measures for the program or established a time frame for evaluating the program's effectiveness.

section_title: Letter
paragraphs: Training flight and cabin crew members (pilots and flight attendants) to  handle potential threats against domestic commercial aircraft is an  important element in securing our nation’s aviation system. These flight  and cabin crew members play a key role in ensuring the security of  commercial aircraft and the safety of airline passengers and are the last  line of defense in the event of an attempted terrorist attack onboard a  commercial aircraft. The responsibility for ensuring that flight and cabin  crew members are prepared to respond to these threats is a shared  responsibility between the private sector—air carriers—and the federal  government, primarily the Transportation Security Administration (TSA).  In an effort to strengthen aviation security, title 49 of the United States  Code, section 44918, requires that air carriers develop and deliver a basic  security training program for flight and cabin crew members in  accordance with training elements prescribed in the law. The law also  requires that TSA monitor and periodically review air carriers’ training  programs to ensure they adequately prepare crew members for potential  threat conditions. Furthermore, the law requires TSA to develop and  provide an advanced voluntary self-defense training program for flight and  cabin crew members. Although there has not been another successful  terrorist attack against commercial aircraft in the United States since  September 11, 2001, concerns remain about whether flight and cabin crew  members are prepared to handle a potential hijacking and other threats to  commercial aircraft.
To determine the progress TSA has made in developing and monitoring  flight and cabin crew security training, we examined TSA’s efforts to  develop guidance and standards for air carriers’ flight and cabin crew  security training, monitor air carriers’ compliance with the guidance and  standards, and develop and deliver advanced voluntary self-defense  training for crew members. Specifically, this report addresses the  following questions: (1) What actions has TSA taken to develop guidance  and standards for flight and cabin crew security training and to measure  the effectiveness of the training? (2) How does TSA ensure domestic air  carriers comply with required training guidance and standards? (3) What  efforts has TSA taken to develop, implement, and measure the  effectiveness of advanced voluntary self-defense training for flight and  cabin crew members?
In conducting our work, we reviewed TSA and Federal Aviation  Administration (FAA) documentation related to flight and cabin crew  member security training guidance and standards and advanced voluntary  crew member self-defense training. The security training guidance  provides detailed guidance from which air carriers must develop their  flight and cabin crew security training programs. The security training  standards set forth the specific requirements for crew member security  training, such as the training elements that must be included in air carriers’  basic and recurrent (refresher) crew member security training programs.  We also interviewed officials from 19 domestic air carriers, 2 air carrier  associations, and 6 crew member labor organizations regarding security  training guidance and standards, and advanced voluntary self-defense  training. We selected domestic air carriers based on whether they were  currently offering initial and/or recurrent security training and on the size  of the air carrier in an effort to include a mixture of various domestic air  carriers and air carriers of varying sizes. The size of an air carrier is based  on the annual operating revenues and the number of revenue passenger  boardings. We visited 8 of these air carriers to observe their training and to  interview flight and cabin crew members about the training. Because we  selected a nonprobability sample of domestic air carriers, the information  we obtained from these interviews and visits cannot be generalized to all  domestic air carriers. We also observed two sessions of TSA’s advanced  voluntary crew member self-defense training. Information obtained from  our interviews and visits cannot be generalized to all air carriers and  stakeholders. Additionally, we interviewed officials responsible for  developing security training guidance and standards and reviewing air  carriers security training programs at TSA headquarters. We also  interviewed officials at the FAA; Federal Bureau of Investigation (FBI);  Bureau of Immigration and Customs Enforcement, Federal Air Marshal  Service (FAMS); Department of Defense (DOD); crew member labor  organizations; and associations representing air carriers to obtain  information on their involvement in the development of crew member  security training guidance and standards and advanced voluntary crew  member self-defense training. A more detailed description of our scope  and methodology is contained in appendix I.
We conducted our work from June 2004 through August 2005 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: After the terrorist attacks of September 11, 2001, the President signed the  Aviation and Transportation Security Act (ATSA) into law on November  19, 2001, with the primary goal of strengthening the security of the nation’s  aviation system. ATSA created TSA as the agency responsible for securing  all modes of transportation, including aviation. The President also issued  the National Strategy for Homeland Security in July 2002. The National  Strategy for Homeland Security sets forth a plan to strengthen homeland  security through the cooperation of federal, state, local, and private-sector  organizations in various areas. The National Strategy for Homeland  Security aligns and focuses homeland security functions into six critical  mission areas: (1) intelligence and warning, (2) border and transportation  security, (3) domestic counterterrorism, (4) protecting critical  infrastructures and key assets, (5) defending against catastrophic threats,  and (6) emergency preparedness and response. A theme of the national  strategy is that homeland security is a shared responsibility among these  stakeholders, not solely the responsibility of the federal government. In  the case of flight and cabin crew member security training, air carriers and  TSA both play an important role. Air carriers are responsible for  developing and delivering security training programs for their crew  members. TSA (and previously FAA) is responsible for developing the  guidance and standards that air carriers are to use to design and deliver  their security training and for monitoring air carriers’ flight and cabin crew  member security training programs for compliance with the guidance and  standards. If TSA finds an air carrier to be noncompliant with developing  and conducting the required flight and cabin crew member security  training, TSA has a range of actions it can take, including imposing fines,  and in extreme circumstances, force the air carrier to shut down its  operations.
The Bureau of Transportation Statistics reported that 105 domestic  passenger air carriers were operating in the United States in 2004. Of the  105 air carriers, 12 (11 percent) are major air carriers that carried over   76 percent of the passengers in 2004. With a few exceptions for small  aircraft, every commercial flight in the United States has at least two flight  crew members and one cabin crew member onboard. These crew  members are viewed as the last line of defense in what TSA describes as  its layered security system, which includes perimeter security (e.g., airport  security fencing), 100 percent passenger and checked baggage screening,  hardened flight deck doors, armed federal air marshals, and armed pilots.  Figure 1 provides the number of domestic air carriers by carrier group  (major, national, and regional) and the percentage of passengers flown  domestically by carrier group during fiscal year 2004.
Federal guidance for air carriers to use to develop their flight and cabin  crew security training programs has been in place for over 20 years. FAA  developed the crew member security training guidance, referred to as  Common Strategy I, in the early 1980’s in response to numerous hijacking  incidents in the late 1970’s. Common Strategy I generally instructed air  carriers to develop training programs that called for flight and cabin crew  members to cooperate with threatening passengers or hijackers and slow  compliance with their demands. Based on this guidance, FAA also  developed corresponding security training standards that set forth the  requirements for flight and cabin crew member security training. Air  carriers were required to incorporate the guidance and standards into  their security training programs. FAA principal security inspectors and  principal operations inspectors were responsible for monitoring air  carriers’ compliance with the security training standards. The nature of the  terrorist attacks on September 11, 2001, however, demonstrated that the  philosophy of Common Strategy I—to cooperate with hijackers—was  flawed as it presumed that hijackers would not use aircraft as weapons of  mass destruction.
Following the events of September 11, 2001, section 107 of ATSA required  FAA, in consultation with TSA and other stakeholders, to develop detailed  guidance for flight and cabin crew security training programs within   60 days after the enactment of the act. FAA developed and issued security  training guidance, in accordance with the requirements of ATSA, on  January 19, 2002. In February 2002, TSA assumed responsibility for  monitoring air carriers’ security training for United States passenger air  carriers and the air carrier security inspections function was transferred  from FAA to TSA.
Following the enactment of ATSA, the President signed into law two acts  that amended the flight and cabin crew training requirements codified at  title 49 of the U.S. Code, section 44918—the Homeland Security Act of  2002 and Vision 100. The Homeland Security Act, enacted on November  25, 2002, amended the law by, among other things, mandating that, if TSA  updated training guidance, it must issue a rule to include elements of self- defense in the training programs. Vision 100, subsequently enacted on  December 12, 2003, amended the flight and cabin crew security training  law in its entirety to require that    air carriers providing scheduled passenger air transportation carry out a  training program that addresses the 10 elements listed in table 1;   TSA approve the air carrier’s training programs;    TSA, in consultation with FAA, monitor air carrier training programs and  periodically review an air carrier’s training program to ensure the program  is adequately preparing crew members for potential threat conditions;   TSA, in consultation with FAA, order air carriers to modify training  programs to reflect new or different security threats; and   TSA develop and provide an advanced voluntary self-defense training  program to provide both classroom and effective hands-on training in, at  least, the six training elements listed in table 2.
Table 1 lists the minimum training elements required by law, as enacted by  ATSA and as amended by Vision 100, for basic crew member security  training.
Table 2 lists the training elements that TSA must include in an advanced  voluntary self-defense training program for flight and cabin crew members  under the law, as amended by Vision 100.
Over the years, our work on best practices in training has found that  generally high-performing organizations follow certain key steps in  developing and measuring the effectiveness of training programs. These  steps include    planning—developing a strategic approach that establishes priorities and  leverages investments in training to achieve agency results and identify the  competencies—commonly referred to as knowledge, skills, abilities, and  behaviors—needed to achieve organizational missions and goals, and  measure the extent to which their employees possess these competencies;   design and development—identifying specific initiatives that the agency  will use, along with other strategies, to include individual and  organizational performance; implementation—ensuring effective and efficient delivery of training  opportunities in an environment that supports learning and change; and    evaluation—assessing the extent to which training efforts contribute to  improved performance and results.
section_title: TSA Has Enhanced Guidance and Standards for Flight and Cabin Crew Security Training with Input from Stakeholders, but Better Planning Is Needed
subsections: section_title: TSA Enhanced Flight and Cabin Crew Member Security Training Guidance and Standards with Input from Stakeholders, but Stakeholders Continue to Have Concerns
paragraphs: Building on the legislatively mandated guidance developed by FAA and the  corresponding standards, TSA enhanced crew member security training  guidance and standards with input from stakeholders in accordance with  the law, as amended by Vision 100. TSA policy and training officials stated  that they revised the guidance and standards for two main reasons. First,  the law required that air carriers include additional training elements in  their basic crew member security training programs to prepare flight and  cabin crew members for potential threat conditions. Second, TSA  determined that the guidance and standards needed to be better organized  and to more clearly define security training elements, in part due to  feedback from air carriers, flight and cabin crew member labor  organizations, and associations representing air carriers. For example,  stakeholders we interviewed and our own review found that the  organization of the previous security training standards was difficult to  follow in that several requirements were addressed in multiple sections of  the document rather than focused in a single section. During the summer  of 2003 and May 2004, TSA established two internal working groups  comprised of representatives of its policy, training, regulatory, and/or legal  offices. One working group was responsible for revising the security  training guidance, and the other working group was responsible for  revising the corresponding security training standards—the standards  from which air carriers must train their flight and cabin crew members.  TSA officials stated that these working groups determined the  reasonableness and appropriateness of the security training elements  contained in the existing guidance and standards in place at that time and  what additional training elements were needed.
During the development of the revised guidance and standards, TSA  provided external stakeholders with two opportunities to provide  comments. In July 2004, the first comment period, TSA convened a  meeting of external stakeholders to present an overview of the draft  revised guidance and standards and to provide copies of the documents  for their review and comment. TSA initially requested that stakeholders  provide comments on the draft revised guidance and standards within   2 weeks. However, in response to stakeholder concerns about the short  comment period, TSA extended the comment period for an additional   2 weeks. After consolidating all stakeholder comments, TSA’s internal  working group reviewed the comments to determine which to incorporate  in the guidance and standards. In August 2004, the second comment  period, TSA convened additional meetings with external stakeholders— one meeting with air carrier associations and another with crew member  labor organizations—to review each of the stakeholders’ comments and to  discuss changes made to the revised guidance and standards in response  to these comments. In September 2004, TSA provided the stakeholders  with a 30-day comment period on the revised guidance and standards.  After receiving comments and determining which of the suggested  changes to include in the revised guidance and standards, TSA issued the  finalized guidance and standards to air carriers on January 3, 2005.
Stakeholders we interviewed and our own analysis of revisions made to  the guidance and standards generally found the revised guidance and  standards to be better organized and to provide some additional clarity on  security training requirements for crew members. For example, we found  that the previous standards only implicitly addressed the requirement for  training on the psychology of terrorists and addressed it in multiple  sections in the document. In contrast, the revised standards organized  information on this requirement in a single section and clearly identified  the requirement as “psychology of terrorists.” Additionally, the previous  guidance did not define what constitutes life-threatening behavior,  whereas the revised guidance provides both a definition of this behavior  and examples.
Although TSA made these enhancements, stakeholders we interviewed  and stakeholders identified by TSA provided concerns about the  reasonableness of applying parts of the guidance and standards to both  flight and cabin crew members, the difficulty in implementing some of the  standards without additional information or training tools from TSA, and  the vagueness of some of the guidance and standards. Our interviews with  officials from 19 air carriers and 8 representatives from aviation  associations and crew member labor organizations, after the revised  guidance and standards were finalized in January 2005, also identified  similar concerns. Regarding the applicability of the standards to flight and  cabin crew members, officials from 9 of the 19 air carriers that we  interviewed stated that some of the training standards remained  generalized to both pilots and flight attendants, rather than targeted to  their specific job functions in responding to a security threat. For example,  TSA requires both pilots and flight attendants to have annual hands-on  training on how to use restraining devices. However, 2 of the 19 air  carriers we interviewed stated that training pilots annually on how to use  restraining devices is not necessary because pilots are trained to stay  inside the flight deck at all times, even when an incident occurs in the  aircraft cabin. TSA officials stated that all crew members need annual  hands-on training on how to use restraining devices because off-duty flight  crew members frequently fly, and if an incident occurs in the aircraft  cabin, they will know how to use the devices. One crew member labor  organization agreed with TSA’s position, stating that incidents could occur  in which pilots may need to apply the restraints. Additionally, the crew  member labor organization official stated since pilots in command are the  security coordinators on flights, they must be familiar with the strategies,  tactics, and techniques that flight attendants may use in defense of  themselves, the passengers, and the aircraft.
Additionally, some stakeholders expressed concerns about the difficulty in  implementing some of the standards without additional information or  training tools from TSA. For example, officials from 12 of the 19 air  carriers we interviewed stated that TSA had not provided sufficient  training materials or tools to enable them to deliver certain elements of the  security training. These air carriers stated that although they requested the  additional information or tools, TSA responded that air carriers were  responsible for identifying and providing the required tools needed to  deliver the security training. A labor union organization official stated that  relying on training organizations and air carriers to develop the training  materials “perpetuates the disparate quality and breadth of training  available throughout the industry, which does little to assure a common  strategy approach to securing United States skies.” Additionally, officials  from 4 of the 19 air carriers we interviewed expressed concerns that TSA  did not take into consideration that some air carriers do not have the  expertise and personnel to conduct the annual basic self-defense training.  TSA responded that basic self-defense training is legislatively required and  the Federal Air Marshal Service, FBI, and other agencies are willing to  work with the air carriers on their overall flight and cabin crew security  training. TSA officials further stated that the air carriers should have an  established line of communication with these agencies, but if the air  carriers are seeking a point of contact, TSA would provide agency contact  information. According to a Federal Air Marshal Service official, a Federal  Air Marshal Service liaison meets periodically with the air carriers and  aviation industry associations representing the air carriers and crew  members to discuss overall communications including flight and cabin  crew training issues.
Furthermore, 9 of 27 stakeholders (air carriers, associations representing  air carriers, and crew member labor organizations) we interviewed were  concerned about the lack of definition, guidance, and clarity for parts of  the revised security training guidance and standards. For example, the  crew member security training standards require that crew members  demonstrate proficiency in various security training elements, such as the  use of protective and restraining devices and proper conduct of a cabin  search. However, the standards do not define proficiency. Officials from a  crew member labor organization stated that without clear, measurable  training objectives for individual air carrier training departments to  determine crew member proficiency and training objectives, the likelihood  that training quality and content will vary from air carrier to air carrier  increases. TSA training officials stated that air carriers, in conjunction  with their training departments, are required to develop a method for  determining crew member proficiency in the required training elements.
TSA officials stated that the air carriers developed the training program,  not TSA, and are therefore in the best position to define proficiency. TSA  officials stated that their training staff’s review of the training materials  include verifying that there are opportunities built into the training for  flight and cabin crew members to demonstrate proficiency in the required  elements. TSA officials further stated that air carriers should have the  latitude to tailor their desired level of proficiency for the various standards  to their individual operations. We found, however, that without standards  for proficiency, which commonly serve as criteria for success in training  programs, TSA will only be able to document training activity, and not the  results of the training, i.e., whether the intended knowledge was in fact  transferred to the training participants at a level acceptable to TSA.
section_title: TSA Has Not Established Strategic Goals or Performance Measures for Crew Member Security Training
paragraphs: TSA has not established strategic goals or performance measures for flight  and cabin crew member security training, nor required air carriers to do  so. GPRA requires that agencies use outcome-oriented goals and measures  that assess results, effects, or impacts of a program or activity compared  to its intended purpose. GPRA also requires federal agencies to consult  with key stakeholders—those with a direct interest in the success of the  program—in developing goals and measures. Strategic goals explain the  results that are expected from a program and when to expect those  results. These goals should be expressed in a manner that could be used to  gauge success in the future. Performance measures (indicators used to  gauge performance) are meant to cover key facets of performance and  help decision makers assess program accomplishments and improve  program performance. With respect to flight and cabin crew security  training, strategic goals would represent the key outcomes that TSA  expects air carriers to achieve in providing flight and cabin crew member  security training, and performance measures would gauge to what extent  air carriers are achieving these outcomes.
TSA training officials stated that they decided not to develop strategic  goals or performance measures because they view their role in the crew  member security training program as purely regulatory—that is,  monitoring air carriers’ compliance with the training guidance and  standards established by TSA. In this regard, TSA is the regulatory agency  responsible for determining whether the security training program is  adequately preparing flight and cabin crew members for potential threat  conditions. TSA training officials also stated that due to the varying nature  of the air carriers’ training programs, TSA believes that it is the individual  air carriers that are responsible for establishing goals and performance  measures specific to their security training programs and for using the  results to make program improvements. However, without overall  strategic goals established by TSA in collaboration with air carriers, air  carriers do not have a framework from which to develop their individual  performance goals and measures. Furthermore, TSA has not explicitly  required air carriers to develop performance goals and measures or  provided them with guidance and standards for doing so. Without  guidance and standards, the 84 individual air carriers may establish  inconsistent performance goals and measures. Additionally, the absence of  performance goals and measures for flight and cabin crew security  training limits the ability of TSA and air carriers to fully assess the  accomplishments of the flight and cabin crew member security training  and to target program improvements.
section_title: Recent Progress Made to Strengthen Oversight of Flight and Cabin Crew Member Security Training Program, but Weaknesses Remain
subsections: section_title: TSA Has Recently Taken Steps to Strengthen Oversight Efforts of Air Carriers Security Training
paragraphs: TSA has recently taken steps to improve its oversight of air carriers’ crew  member security training. One step includes adding staff with training  expertise to review air carriers’ crew member security training  curriculums to determine whether there is evidence that each applicable  training standard is being met. When we began our review, TSA’s review of  air carriers’ crew member security training programs was solely the  responsibility of the principal security inspectors. These TSA inspectors  were responsible for conducting a regulatory review to determine whether  air carriers’ crew member security training curriculums met the  requirements set forth in the standards. Beginning in January 2005, TSA  began using training staff with expertise in designing training programs to  review the overall design of the air carriers’ crew member security training  curriculum, how the information is to be conveyed, the expected setting of  the practice environment, and the way in which the information is to be  presented—and to ensure that the security training curriculum satisfies  the required security training standards. TSA inspectors are responsible  for identifying which standards apply to each of the air carriers, based on  their knowledge of the air carrier’s flight operations, size of aircraft, and  presence or absence of international routes. TSA officials stated that  between January 2005, when the revised guidance and standards were  issued, and August 2, 2005, the training staff were involved in the review of  the 71 security training curriculums that had been submitted to TSA.
In January 2005, TSA took another step to strengthen its review of air  carriers’ flight and cabin crew member security training by developing a  standard form for TSA inspectors and training staff to use to conduct and  document their reviews of air carriers’ security training curriculums. Also,  TSA developed an internal memorandum, dated January 5, 2005, that  generally describes the review process TSA inspectors and training staff  are to use when reviewing air carriers’ crew member security training  curriculums. The standard form, which lists the required training elements,  is used by TSA inspectors to document the requirements stated in the  revised security training standards that apply to a particular air carrier,  and by the training staff to verify that air carrier’s initial and recurrent  training plans include the applicable requirements and to document their  comments. Prior to the development of this form, there were no  documented procedures for how the inspections were to be conducted or  a standard form for TSA inspectors to use to document their reviews of air  carriers’ crew member security training curriculums. Additionally, TSA  lacked complete documentation of its reviews of air carriers’ security  training. Specifically, although TSA officials stated that TSA inspectors  reviewed all 84 air carriers’ revised security training curriculums in  response to January 2002 guidance and the corresponding standards, TSA  was only able to provide us documentation related to 11 reviews. The  Comptroller General’s Standards for Internal Control in the Federal  Government states that agencies should document all transactions and  other significant events and should be able to make this documentation  readily available for examination. With the development of a standard  form for reviewing air carriers’ security training curriculums in January  2005, TSA was able to provide us with documentation for all 18 of the  reviews of air carriers’ security training curriculums that TSA inspectors  and training staff had conducted between January 2005 and April 20,  2005.
Additionally, in January 2005, TSA began requiring air carriers to obtain  participant feedback at the end of crew member security training.  According to our human capital work, participant feedback can be useful  in providing the agency with varied perspectives on the effect of the  training. However, TSA training officials stated that they are not certain  how, if at all, they will use the participant feedback in conducting  oversight of air carriers’ crew member security training programs. TSA  officials stated that it is the responsibility of the individual air carriers to  assess the results of participant feedback and to make changes to improve  the security training as necessary. In May 2005, TSA training officials  acknowledged that it would be useful for its inspectors to review  participant feedback on an annual basis to assess flight and cabin crew  members’ views of their air carriers’ security training programs and to  identify trends within and across air carriers. The official acknowledged  that these results could provide TSA inspectors information they could use  to prioritize their reviews of air carriers’ crew member security training.  However, the official stated that reviewing the participant feedback is a  resource intensive process that also requires a certain level of expertise  and is not feasible for TSA to undertake at this time. Without plans for  reviewing participant feedback, TSA is not making use of available  information on possible deficiencies in the quality of air carriers’ security  training programs or identifying best practices that could be shared.
Furthermore, TSA is taking steps to address a staffing shortage among its  TSA inspector workforce to enable greater monitoring of air carriers’ flight  and cabin crew member security training. Specifically, on April 1, 2005,  TSA reorganized its inspection staff into a newly created Office of  Compliance. TSA officials stated that this reorganization should help  address the staffing shortfalls that previously existed. TSA also issued  position announcements in an effort to fill vacant inspector positions. TSA  officials stated that they had about 23 TSA inspectors onboard when the  inspection function transferred from FAA to TSA in February 2002. As of  February 2005, TSA had 15 inspectors onboard, 5 of whom were in the  position for less than 5 months. Between January 2004 and September  2004, the TSA inspector workforce ranged from about 7 to 14 inspectors.  TSA officials stated that a number of these staff subsequently left TSA  because of advancement opportunities within the Department of  Homeland Security and personal reasons. As part of TSA’s monitoring  efforts, TSA inspectors periodically visit air carriers to observe classroom  delivery of flight and cabin crew member security training and to review  air carrier records documenting flight and cabin crew member completion  of required security training. TSA officials stated that with the existing  inspector workforce, they were only able to conduct observations of about  25 of air carriers’ classroom delivery of flight and cabin crew member  security training during fiscal year 2004. Although TSA is not required to  observe the classroom delivery of all air carriers’ flight and cabin crew  member security training on an annual basis, TSA officials stated that  these observations allow them to determine whether security training is  being delivered consistent with air carriers’ approved security training  curriculums and to identify potential problems with the training delivery.
section_title: TSA Has Not Fully Developed Procedures for Monitoring Air Carriers’ Crew Member Security Training
paragraphs: While TSA has taken steps to strengthen its oversight of air carriers’ crew  member security training, TSA has not fully developed procedures for  monitoring this training. TSA is required by law to monitor and  periodically review air carriers’ security training to ensure that the training  is adequately preparing crew members for potential threat conditions. The  Comptroller General’s Standards for Internal Control in the Federal  Government calls for controls generally to be designed to assure that  ongoing monitoring occurs during the course of normal operations and  that transactions and other significant events be documented clearly and  the documentation be readily available for examination. We identified  weaknesses in TSA’s controls in these areas with regard to monitoring and  reviewing air carriers’ flight and cabin crew security training.
First, although TSA recently developed a standard form for its inspectors  and training staff to use in reviewing air carriers’ flight and cabin crew  member security training, TSA has not developed procedures for  completing this form. TSA officials acknowledged that there are no  documented procedures or criteria for staff to use to complete the  standard form or for determining which standards apply to individual air  carriers and whether or not to approve an air carrier’s security training  curriculum. The lack of written procedures may result in inconsistent  assessments of the air carriers’ security training curriculums and  inconsistent application of the standards to air carriers. Formal  procedures for reviewing air carriers’ flight and cabin crew security  training could provide standardization when TSA inspectors and training  staff assess the air carriers’ security training curriculum.
Second, TSA does not have documented procedures for conducting and  documenting observations of air carriers’ classroom delivery of flight and  cabin crew member security training. During fiscal year 2004, according to  TSA officials, TSA inspectors visited about 25 air carriers to observe crew  member security training and review files, such as records documenting  crew members’ completion of required security training. TSA officials  stated that they did not have sufficient resources to visit all 84 air carriers  to observe their security training. We requested records documenting TSA  inspectors’ visits to air carriers to assess the completeness and  consistency of these reviews. However, TSA officials stated that they were  unable to provide us with the requested documentation. Without written  procedures to guide TSA inspectors in observing security training and  assessing the results of their observations, its inspectors may not conduct  comprehensive and consistent reviews. Additionally, without a mechanism  for documenting and maintaining TSA inspectors’ reviews of air carriers’  security training delivery in a standard format, TSA lacks the ability to  track the results of these reviews and identify patterns, including strengths  and weaknesses, in training delivery within and across air carriers. In June  2005, a TSA official stated that TSA inspectors will monitor at least one  flight and cabin crew member training class per year to ensure the  curriculum is being followed. TSA inspectors are to provide the results of  the monitoring to the principal operations inspector via memo or email.  TSA officials stated that inspections of monitoring crew member security  training will be maintained in a database, but TSA has not established a  time frame for completing this database or documented procedures for  this process.
Additionally, although the law requires TSA to consider complaints from  crew members in determining when to review air carriers’ flight and cabin  crew member security training programs, TSA does not have procedures  for considering such complaints. TSA inspection officials stated that they  were not aware of any instances in which crew members had complained  to TSA about security. However, in the event that TSA does receive  complaints from crew members in the future, it is important that TSA have  established and documented procedures to inform its inspectors of how to  consider the complaints in reviewing air carriers’ security training  programs. TSA officials stated that complaints from flight and cabin crew  members will be directed to their Office of Transportation Security Policy  for review and all decisions regarding flight and cabin crew member  security training program modifications or policy changes will be  evaluated and disseminated by this office. The officials also stated that if  the complaints involve the training delivery process, the TSA inspectors  may be required to increase the frequency of on-site inspections based  upon an evaluation of the seriousness of the complaints that are received.
TSA officials stated that they plan to develop a handbook for its inspectors  and guidance for its training staff to use in monitoring and reviewing air  carriers’ flight and cabin crew member security training to help provide  assurance that standardized monitoring occurs. However, TSA has not  established a time frame for completing these efforts.
section_title: TSA Developed and Implemented Crew Member Self-Defense Training, but Has Not Established Performance Measures or Evaluated Training Effectiveness
paragraphs: In December 2004, as required by law, TSA implemented an advanced  voluntary crew member self-defense training program for flight and cabin  crew members after obtaining stakeholder input. Participation in the  voluntary training course has been relatively low, with only 474 flight and  cabin crew members (39 percent of total capacity) attending the training  during the first 7 months of the program. TSA training officials attributed  the low participation to crew members having a difficult time in obtaining  3 consecutive days of leave to enable them to participate in the training.  Additionally, although TSA incorporated some stakeholder concerns into  the course design, some stakeholders, including individuals identified as  experts by TSA and our own analysis identified concerns regarding the  training design and delivery, including the training’s voluntary nature, the  setting’s lack of realism, the training’s lack of recurrence, and the  instructor’s lack of knowledge of crew members’ actual work  environment. TSA has not developed performance measures for the  program or established a time frame for evaluating the program’s overall  effectiveness, including the effectiveness of the training design and  delivery.
subsections: section_title: TSA Developed and Implemented Advanced Voluntary Self-Defense Training in Consultation with Stakeholders, but Participation in Training Remains Low
paragraphs: TSA developed and implemented an advanced voluntary self-defense  training program for flight and cabin crew members in consultation with  key stakeholders by December 12, 2004, as required by law. TSA consulted  with law enforcement personnel, security experts with expertise in self- defense training, representatives of air carriers, flight attendants, labor  organizations representing flight attendants, terrorism experts, Federal Air  Marshal Service officials, and educational institutions offering law  enforcement training programs, in developing the self-defense training  program and determining how to apply the training elements specified by  law.  According to TSA officials, in 2002, in anticipation of having to  develop a mandatory self-defense training program as required by the  Homeland Security Act, TSA established a working group comprised of  law enforcement experts, Federal Air Marshals, and other subject matter  experts, such as aviation security experts and self-defense/martial arts  training experts, to assess what elements should be included in the  training. This working group collaborated on the overall program design  and delivery, including the program goals and objectives and the course  content and delivery method. The working group’s efforts were placed on  hold in 2003 when TSA was advised that legislation would be enacted to  make the training a voluntary program to be provided by TSA, rather than  a mandatory training program to be delivered by individual air carriers.  After the enactment of Vision 100 in December 2003, TSA continued its  efforts to develop an advanced voluntary self-defense training program  until the program’s official implementation in December 2004, building on  the input of the initial working group. The overall goal of the advanced  voluntary crew member self-defense training, as defined by TSA, is to  enable crew members to develop a higher level of competency in self- defense tactics to prevent or reduce the possibility of injury or death to  one’s person or the takeover of an aircraft. TSA also established several  objectives for the training, including  recognize potential threats before an act of violence occurs; interpret behaviors that lead to potential hostile acts;   conclude appropriate courses of action crew members must take to avert  hostile actions intended to injure crew members or passengers or to take  over an aircraft; and   apply appropriate individual self-protection measures and self-defense  tactics to prevent or reduce the possibility of injury or death to one’s  person or the takeover of an aircraft.
Prior to implementing the voluntary training in December 2004, TSA  piloted the prototype advanced voluntary self-defense training in August  and September 2004 in five cities with major airline hubs and refined the  training based on comments from participants. The participants provided  positive feedback in four areas, including (1) the repetitive moves taught  throughout the course made the self-defense tactics easy to learn; (2) the  training prepared them mentally and physically to defend themselves and  provided a good foundation in self-defense; (3) the small class size and  instructor to student ratio of 1 to 8 was conducive to a productive learning  environment; and (4) the location of the training facility and lodging was  well received. TSA also received feedback on changes that could be made  to enhance the training. Table 3 provides a summary of the stakeholders’  concerns on TSA’s prototype advanced voluntary self-defense training and  actions taken by TSA in response to the concerns.
As of June 2005, a total of 474 crew members had participated in the  training in 51 classes. During the initial deployment of the advanced  voluntary crew member self-defense training in December 2004,  participation was only about 14 percent of the total capacity being utilized,  and about 38 percent of total enrolled participants actually attended the  training course. Participation increased in January through March 2005,  but declined in April through June 2005, with only 23 percent of total  capacity utilized in June 2005. TSA plans to offer 46 additional advanced  voluntary self-defense training courses during the remainder of fiscal year  2005 in 10 cities. According to TSA officials, TSA estimated that  approximately 21,700 crew members will participate in the training, based  on information obtained from air carrier associations and crew member  labor organizations. Table 4 provides information on flight and cabin crew  member participation in the advanced voluntary self-defense training from  December 2004 through June 2005.
TSA officials stated that the low participation rate in December 2004 was  largely due to the short advance notice they provided stakeholders  regarding the training. TSA announced the availability of the December  2004 training courses 6 days before the training was to take place. This  short notice significantly restricted the ability of flight and cabin crew  members to participate in the training because they generally submit their  scheduling bids 30 days prior to their work schedule. TSA attributed the  short notice provided to the length of time it took to reallocate funds from  other TSA programs to enable implementation of the advanced voluntary  self-defense training program. TSA provided crew members with more  than 30 days notice for the remaining sessions. However, TSA training  officials stated that participation continued to remain low in January 2005  due to crew members calling in sick, crew members being called for flight  duty at the last moment, and inclement weather. TSA training officials  stated that although they projected that crew member participation would  increase over time, crew member participation decreased in June 2005 due  to crew members having a difficult time obtaining 3 consecutive days of  leave to attend training. TSA training officials also stated that based on  their experience with the Federal Flight Deck Officer training program,  crew members’ ability to obtain leave for the purpose of attending training  may be associated with seasonal variances, with low participation usually  occurring during the spring and summer months. Stakeholders, including  air carriers and crew member labor organizations, attributed the low  participation to crew members having to attend the training on their own  time and pay the cost of travel, lodging, and meals. TSA training officials  stated that they were continuing to gather information from flight and  cabin crew members through the training contractor in an effort to  identify the causes for the low participation and, ultimately, to try to  address these causes.
section_title: Stakeholders Identified Concerns about the Self- Defense Training Design and Delivery
paragraphs: Stakeholders, including individuals that TSA identified as subject matter  experts, and our own analysis identified concerns with the design and  delivery of the advanced voluntary crew member self-defense training.  These concerns include the training’s voluntary nature, the setting’s lack  of realism, the training’s lack of recurrence, and the instructor’s lack of  knowledge of crew members’ actual work environment. These same  concerns were identified by stakeholders in response to the prototype self- defense training. As previously stated, our prior human capital work has  found that in implementing a training program, an agency should ensure  that implementation involves effective and efficient delivery of training— that is, the training should be conducted in a setting that approximates the  participants’ working conditions and taught by individuals who are  knowledgeable about the subject matter and work environment.
In the case of the advanced voluntary self-defense training program, 13 of  the 33 stakeholders expressed concerns about the voluntary nature of the  training and stated that the training should be mandatory. Six of these   13 stakeholders stated that the program’s voluntary nature is inconsistent  with TSA’s revised security training guidance that seeks to establish a  common strategy that would enable individuals involved in an incident  onboard an aircraft to know what others involved will be thinking and  doing. These same stakeholders stated that because the training is not  mandatory, if some crew members have had the self-defense training while  others have not, a breakdown in communication could occur. TSA training  officials stated that because the security training standards require all  crew members to receive training on how to communicate and coordinate  during a disturbance, they are not concerned about the voluntary nature of  the self-defense training program.
Additionally, 14 of the 33 stakeholders expressed concerns about the lack  of a realistic training setting during the delivery of advanced voluntary  crew member self-defense training. The self-defense techniques are taught  in an open-space setting, unlike the narrow aisles crew members have to  work within on an actual aircraft. In the two training sessions we observed  in two different cities, participants had to be constantly reminded by the  instructors of the restricted training space because participants repeatedly  made defensive moves, such as spins and wide kicks, which could not be  performed inside an aircraft cabin. Our prior human capital work has  found that for learning to be successful, the training environment— training facility and equipment—should be favorable to successful  learning. TSA officials stated they examined the possibility of purchasing  aircraft simulators for the self-defense training and found that it would  cost TSA about $100,000 per simulator. Officials stated that they have  advised the instructors to try to create a setting, using chairs, tape, or  other means, to simulate the narrow aisles on an aircraft. We informed  TSA that in the two training sessions we attended, instructors did not use  these techniques. TSA officials stated that they would follow up with the  instructors to ensure they use these techniques.
Fifteen of the 33 stakeholders also expressed concerns about the lack of  recurrent self-defense training given that self-defense skills are difficult to  sustain if not consistently practiced over time. Stakeholders stated that a  3-day, one-time self-defense training course would not enable crew  members to develop proficiency in self-defense. TSA officials responsible  for developing the advanced voluntary training program stated that the  self-defense training is not intended to make participants proficient in self- defense. Rather, the training is intended to enable crew members to  develop a higher level of competency in self-defense tactics by extending  their knowledge and skills in the use of self-defense techniques and  improvised weapons. TSA officials also stated that the key benefit of the  training is a change in the mindset of participants that enables a greater  awareness of threat conditions onboard an aircraft and in their daily lives.  Additionally, they stated that it is the responsibility of the individual  participants to practice the various self-defense techniques they were  taught. Furthermore, they stated that although TSA is not currently  offering a recurrent training program, it has not ruled out the possibility of  recurrent training in the future. Eleven of the 33 stakeholders also  expressed concerns that the self-defense training could give participants a  false sense of security. For example, two stakeholders stated that the false  sense of security lies in the participants who take the course once and  expect to be skilled and proficient using the self-defense techniques  without realizing that they may not be capable of following through when  an incident occurs.
Finally, 6 of the 33 stakeholders, including subject matter experts and  crew member labor organizations, identified concerns about the lack of  knowledge instructors had about the crew members’ actual working  environment. While some stakeholders commended the instructors for  their technical knowledge of crew members’ actual work environment,  others expressed concerns that some instructors lacked technical  knowledge and expertise of the aviation industry. For example, a training  participant we interviewed stated that the instructor did not understand  how safety devices onboard an aircraft operate. The instructor suggested  inflating an emergency raft while inflight to protect the flight deck.  However, according to the training participant, inflating an emergency raft  in flight could injure or kill passengers and crew members. Our prior  human capital work found that the use of instructors who are  knowledgeable of the subject matter and experienced in aviation industry  issues can help provide assurance that instructors can effectively transfer  these skills and knowledge to others. TSA officials stated that they advised  the training contractor to hire instructors with law enforcement, martial  arts/self-defense, and aviation backgrounds. Additionally, TSA provided  the instructors with training on aviation terminology so instructors could  better communicate with the students throughout the course.
TSA officials stated that they were aware of the stakeholder concerns  regarding the self-defense training course. The officials stated that their  ability to address these concerns is limited by funding constraints and  competing priorities. TSA officials further stated that they will continue to  work with the contractor that is delivering the training to obtain any  information that would be beneficial to the design and implementation of  the training program.
section_title: TSA Has Not Established Performance Measures or a Time Frame for Evaluating Training Effectiveness
paragraphs: TSA has not yet developed performance measures for the advanced  voluntary crew member self-defense training program or established a  time frame for evaluating the program’s overall effectiveness. Our prior  human capital work on best practices in training has found that generally,  high-performing organizations evaluate the effectiveness of their training  programs and use the results to target performance improvements. In  February 2005, TSA began conducting end-of-course evaluations— participant feedback—of the training and is planning to assess these  evaluations to ensure the training is consistently achieving results over  time. Additionally, TSA will use the results to modify the training, if  appropriate. Although these evaluations should enable TSA to assess  participants’ views on the training facilities, materials, and instructors,  they will not enable TSA to determine whether the training increased the  participants’ knowledge and skills. TSA officials stated that they recognize  the importance of developing performance measures and evaluating the  effectiveness of the program to ensure that it is consistently achieving the  goals and target performance improvements. Although TSA plans to  undertake these efforts, it has not established time frames for doing so.  TSA officials stated that the numerous internal process improvements  currently under way in TSA that compete for time and resources will affect  how soon the agency can establish performance measures and conduct an  evaluation of the training program. Without performance measures and an  evaluation of the program’s overall effectiveness, TSA will not have  meaningful information with which to determine whether the training  program is actually enabling crew members to develop a higher level of  competency in self-defense tactics—the intended goal of the training  program.
section_title: Conclusions
paragraphs: It has been less than 4 years since TSA assumed responsibility for aviation  security. During this period, TSA implemented numerous initiatives to  strengthen the various layers of security in commercial aviation. These  efforts have largely focused on passenger and checked-baggage  screening—among the first lines of defense in preventing terrorist attacks  on commercial aircraft. TSA has recently taken steps to ensure that flight  and cabin crew members—the last line of defense—are prepared to  handle potential threat conditions onboard commercial aircraft. The  revised guidance and standards TSA developed for air carriers to follow in  developing and delivering their flight and cabin crew member security  training is a positive step forward in strengthening security onboard  commercial aircraft. However, guidance and standards alone do not  provide assurances that the training delivered by air carriers is achieving  TSA’s intended results. TSA views its role in flight and cabin crew member  security training as regulatory, and that air carriers are responsible for  measuring the success of their individual training programs. We agree that  air carriers have responsibility for assessing the effectiveness of their  training programs. However, we believe that overall responsibility for  ensuring that flight and cabin crew members are prepared to respond to  terrorist threats must be shared between the air carriers and TSA. In  supporting this partnership, TSA should establish strategic goals for the  flight and cabin crew security training program so that air carriers can  develop their security programs, and measure the effectiveness of these  programs, based on desired results, or goals, clearly defined by TSA.  Without strategic goals to inform air carriers of what is expected from  their training programs, and in the absence of guidance and standards to  help ensure that air carriers establish consistent performance goals and  measures, it will be difficult for TSA and the air carriers to gauge the  success of training programs over time and to determine how to direct  improvement efforts most effectively.
Additionally, while we are encouraged by the recent steps TSA has taken  to improve its monitoring and review of air carriers’ security training  programs, without enhanced controls, such as written procedures for TSA  staff to follow in conducting and documenting their reviews, TSA lacks  reasonable assurance that its monitoring and review efforts will be  conducted in a consistent and complete manner. Furthermore, a key  source of information on the effectiveness of air carriers’ security training  is participant feedback on the training. TSA’s recent requirement that air  carriers obtain written feedback from flight and cabin crew members at  the end of security training is a step in the right direction. However,  without a process in place for considering this information during its  oversight efforts, TSA is not effectively utilizing available information that  could assist it in prioritizing and focusing its monitoring and review  activities.
Through developing and implementing the advanced voluntary self- defense training program, TSA took another step forward in its efforts to  prepare flight and cabin crew members to handle potential threat  conditions onboard commercial aircraft. However, TSA has not yet  established performance measures or a timeframe for evaluating the  effectiveness of the training program, including the training design and  delivery. Congress enacted the Government Performance and Results Act  of 1993 to focus the federal government on achieving results and providing  objective, results-oriented information to improve congressional decision  making. Without performance measures or a method for evaluating the  effectiveness of the training, TSA may not have information with which to  systematically assess the program’s strengths, weaknesses, and  performance. Performance measures and an evaluation of the program’s  effectiveness can assist TSA in focusing its improvement efforts and  provide Congress with information to assess the impact of an advanced  voluntary self-defense training program.
section_title: Recommendations
paragraphs: To help provide TSA management with reasonable assurance that its  security training guidance and standards for flight and cabin crew  members are preparing crew members for potential threat conditions, and  to enable TSA and air carriers to assess the accomplishments of the  security training and target program improvements, we recommend that  the Secretary of the Department of Homeland Security direct the Assistant  Secretary, Transportation Security Administration, to take the following  three actions:    establish strategic goals for the flight and cabin crew member security  training program, in collaboration with air carriers, and communicate  these goals to air carriers to explain the results that are expected from the  training;   develop guidance and standards for air carriers to use in establishing  performance goals and measures for their individual flight and cabin crew  member security training programs to help ensure consistency in the  development of goals and measures; and review air carriers’ goals and measures as part of its monitoring efforts to  help ensure that they are linked to strategic goals established by TSA and  to assess whether the training programs are achieving their intended  results.
To strengthen TSA’s internal controls and help ensure that air carriers are  complying with TSA’s guidance and standards, we also recommend that  the Assistant Secretary, Transportation Security Administration, establish  a time frame for finalizing written procedures for monitoring and  reviewing air carriers’ flight and cabin crew security training. These  procedures should address the process for completing flight and cabin  crew member curriculum review forms, determining which standards  apply to individual air carriers and whether or not to approve an air  carrier’s training curriculum, conducting and documenting observations of  air carriers’ classroom delivery of security training, reviewing air carriers’  security training goals and measures, and considering security related  complaints from flight and cabin crew members. As part of its efforts to  develop written procedures, TSA should examine ways to incorporate  participant feedback into its monitoring and review efforts.
In addition, to help ensure that the advanced voluntary crew member self- defense training is achieving its intended results, we recommend that the  Assistant Secretary, Transportation Security Administration, establish  performance measures for the advanced voluntary crew member self- defense training program and a time frame for evaluating the effectiveness  of the training, including the effectiveness of the training design and  delivery.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for review and comment. On  August 29, 2005, we received written comments on the draft report, which  are reproduced in full in appendix III. DHS generally concurred with the  findings and recommendations in the report, and agreed that efforts to  implement our recommendations are critical to a successful flight and  cabin crew member security training program. With regard to our  recommendations that TSA establish strategic goals for the flight and  cabin crew member security training program and develop guidance and  standards for air carriers to use in establishing performance goals and  measures for their individual flight and cabin crew member security  training programs, DHS stated that TSA has begun efforts to establish  strategic goals for the program and, air carriers would benefit from  additional guidance—that is, guidance in addition to the flight and cabin  crew security training standards—to use in establishing performance goals  and measures for their individual flight and cabin crew security training  programs. While TSA has established standards for air carriers’ to use in  developing their flight and cabin crew security training, these standards do  not include strategic goals for the training nor provide any guidance for  establishing performance goals and measures. In addition, at the time of  our review, TSA had not begun developing strategic goals for flight and  cabin crew security training.  Therefore, we cannot assess the extent to  which the goals TSA is currently developing satisfy our recommendation.  With respect to our recommendation that TSA establish a time frame for  finalizing written procedures for monitoring and reviewing air carriers’  flight and cabin crew security training, DHS stated that TSA is in the  process of developing a monitoring plan, to the extent that resources  permit, and a handbook for reviewing air carriers’ flight and cabin crew  member security training programs. DHS further stated that the handbook  is currently under development and will be completed and ready for  implementation in fiscal year 2006. Finally, regarding our recommendation  that TSA establish performance measures for the advanced voluntary crew  member self-defense training program and a time frame for evaluating the  effectiveness of the training, DHS stated that TSA is working with OMB to  establish performance measures for use in OMB’s Performance  Assessment Rating Tool for flight security training and will finalize these  measures using fiscal year 2005 data as the baseline. According to DHS,  these measures will provide TSA with information that can be used in  evaluating the effectiveness of the advanced voluntary crew member self- defense training. DHS also stated that TSA has begun to reach out to  stakeholders to obtain feedback on this training. TSA’s successful  completion of these ongoing and planned activities should address the  concerns we raised in this report. We also provided relevant sections of  this report to FAA, FBI, and DOD for their review, and incorporated their  technical comments into the report as appropriate.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 2 days from the  date of this report. At that time, we will send copies of this report to the  Secretary of the Department of Homeland Security and the Administrator  of the Transportation Security Administration and interested  congressional committees. We will also make copies available to others  upon request. In addition, the report will be made available at no charge  on GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-8777 or berrickc@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To determine the progress TSA has made in developing and monitoring  flight and cabin crew security training, we examined TSA’s efforts to  develop guidance and standards for air carriers’ flight and cabin crew  security training, monitor air carriers’ compliance with the guidance and  standards, and develop and deliver advanced voluntary self-defense  training for crew members. Specifically, this report addresses the  following questions: (1) What actions has TSA taken to develop guidance  and standards for flight and cabin crew security training and to measure  the effectiveness of the training? (2) How does TSA ensure domestic air  carriers comply with required training guidance and standards? (3) What  efforts has TSA taken to develop, implement, and measure the  effectiveness of advanced voluntary self-defense training for flight and  cabin crew members?
To determine the actions TSA has taken to develop guidance and  standards for flight and cabin crew security training and to measure the  effectiveness of the training as well as how TSA ensures domestic air  carriers comply with required training guidance and standards, we  obtained and analyzed relevant legislation, guidance, and standards  developed by TSA and FAA, and TSA records documenting its reviews of  air carriers’ security training programs. We reviewed the security training  guidance and standards to determine whether they contained the statutory  requirements for flight and cabin crew security training of 49 U.S.C. §  44918, as established by the Aviation and Transportation Security Act, and  as amended by the Homeland Security Act of 2002 and Vision 100— Century of Aviation Reauthorization Act. We also interviewed TSA  training, policy, and inspections officials to identify their reasons for  revising the guidance and standards and the process they used to revise  these documents and ensure air carriers’ compliance with the guidance  and standards. Additionally, we compared TSA’s process for monitoring  and reviewing air carrier compliance with flight and cabin crew security  training guidance and standards to standards for internal control in the  federal government. To assess stakeholder involvement in the  development of the guidance and standards and identify any stakeholder  concerns, we interviewed officials from the FAA, FBI, FAMS, DOD, crew  member labor organizations, and associations representing air carriers. At  eight domestic air carriers we visited, we interviewed air carrier officials  to obtain their views on the security training guidance and standards and  TSA’s efforts to ensure air carriers’ compliance with the guidance and  standards and to observe the flight and cabin crew initial or recurrent  (refresher) security training. We selected these domestic air carriers based  on whether they were currently offering initial and/or recurrent security  training and on the size of the air carrier in an effort to include a mixture  of various domestic air carriers and air carriers of varying sizes. The size  of an air carrier is based on the annual operating revenues and the number  of revenue passenger boardings. Finally, we conducted phone interviews  with representatives of 11 additional domestic air carriers—which we  selected using the same criteria we used to select the 8 air carriers to  visit—to obtain their views on the flight and cabin crew member guidance  and standards and TSA’s monitoring of air carriers’ compliance with these  standards. Because we selected a nonprobability sample of domestic air  carriers, the information we obtained from these interviews and visits  cannot be generalized to all domestic air carriers.
To determine the efforts TSA has taken to develop, implement, and  measure the effectiveness of advanced voluntary self-defense training for  flight and cabin crew members, we obtained and analyzed relevant  legislation, TSA’s course training manual for the self-defense training, and  feedback provided by flight and cabin crew members who participated in  the prototype training. We also interviewed TSA training officials  responsible for designing and implementing the voluntary advanced crew  member self-defense training program. Additionally, we observed the final  training in two cities. Furthermore, we interviewed relevant stakeholders,  including representatives of air carriers; labor organizations representing  flight attendants and pilots as well as individual flight attendants and  pilots; aviation industry associations representing air carriers; individuals  identified as subject matter experts or self-defense training experts; and  federal officials at the FBI, FAMS, and FAA to determine whether TSA  consulted them when developing the crew member self-defense training.  We identified subject matter experts or self-defense training experts based  on recommendations from TSA and crew member labor organizations. We  also interviewed representatives of the 19 domestic air carriers mentioned  above to obtain their views on the design and delivery of the advanced  voluntary crew member self-defense training. We assessed the extent to  which TSA incorporated stakeholder input into the training program, and  the basis for TSA’s decisions on which stakeholder input to incorporate  into the training. Finally, we assessed TSA’s efforts to develop the training  programs relative to our guidance for assessing training and development  efforts in the federal government.
We conducted our work from June 2004 through August 2005 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: TSA’s Deployment of an Advanced Voluntary Crew Member Self- Defense Training from December 2004 through September 2005, as of July 2005
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgements
paragraphs: In addition to the contact named above, Katherine Davis, Kimberly  Gianopoulos, Sally Gilley, Stan Kostyla, Tom Lombardi, Gary Malavenda,  Maria Strudwick, Carol Willett, and Su Jin Yon made key contributions to  this report.