Aviation Security: TSA's Managed Inclusion Process Expands Passenger Expedited Screening, But TSA Has Not Tested Its Security Effectiveness
Published: 2015-03-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2011, TSA began providing expedited screening to selected passengers and has expanded the availability of such screening to increasing numbers of passengers as part of its overall emphasis on risk-based security. Passengers who qualify for expedited screening enjoy varying levels of benefits, including not having to remove their shoes, light outerwear, jackets, belts, liquids, gels and laptops for X-ray screening at airport security checkpoints. By determining passenger risk prior to travel, TSA intended to focus its screening resources on higher-risk passengers while expediting screening for lower-risk passengers. Further, TSA developed the Managed Inclusion process, designed to provide expedited screening to passengers not deemed low risk prior to arriving at the airport.
This testimony addresses (1) how TSA assesses the risk of passengers to determine their eligibility to receive expedited screening and (2) the extent to which TSA determined the effectiveness of its Managed Inclusion process. This statement is based on a report GAO issued in December 2014 and selected updates from March 2015. Among other things, GAO analyzed TSA policies and procedures and interviewed TSA security officials.
section_title: What GAO Found
paragraphs: The Transportation Security Administration (TSA) implemented its expedited screening program—known as TSA Pre✓ TM —in 2011.TSA uses the following methods to assess whether a passenger is low risk and therefore eligible for expedited screening.
(1) Approved TSA Pre✓ TM lists of known travelers —These lists are comprised of individuals whom TSA has determined to be low risk by virtue of their membership in a specific group, such as active duty military members, or based on group vetting requirements.
(2) Automated TSA Pre✓ TM risk assessments of all passengers —Using these assessments, TSA assigns passengers scores based upon information available to TSA to identify low risk passengers eligible for expedited screening for a specific flight prior to the passengers' arrival at the airport.
(3) Real-time threat assessments through Managed Inclusion —These assessments use several layers of security, including procedures that randomly select passengers for expedited screening, behavior detection officers who observe passengers to identify high-risk behaviors, and either passenger screening canine teams or explosives trace detection devices to help ensure that passengers selected for expedited screening have not handled explosive material. TSA developed Managed Inclusion as a tool to improve the efficiency of dedicated TSA Pre✓ TM screening lanes as well as to help TSA reach its internal goal of providing expedited screening to at least 25 percent of passengers by the end of calendar year 2013.
TSA has tested the effectiveness of individual Managed Inclusion security layers and determined that each layer provides effective security. However, GAO has previously identified challenges in several of the layers used in the Managed Inclusion process, raising concerns regarding their effectiveness. For example, in November 2013, GAO found that TSA had not demonstrated that behavioral indicators can be used to reliably and effectively identify passengers who may pose a threat to aviation security. TSA is taking steps to revise and test the behavior detection program, but the issue remains open. In December 2014, GAO reported that TSA planned to begin testing Managed Inclusion as an overall system in October 2014 and TSA estimated that testing could take 12 to 18 months to complete. GAO has previously reported on challenges TSA has faced in designing studies to test the security effectiveness of other programs in accordance with established methodological practices such as ensuring an adequate sample size or randomly selecting items in a study to ensure the results can be generalizable—key features of established evaluation design practices. In March 2015, TSA officials noted that a pilot for testing behavior detection officers was scheduled to run from October 2014 through May 2015, and testing of canines was scheduled to begin in June 2015 and be completed in March 2016. Ensuring its planned testing of the Managed Inclusion process adheres to established evaluation design practices will help TSA provide reasonable assurance that the testing will yield reliable results.
section_title: What GAO Recommends
paragraphs: In its December 2014 report, GAO recommended that TSA take steps to ensure and document that its planned testing of the Managed Inclusion process adheres to established evaluation design practices. DHS concurred with GAO's recommendation and is taking action to address it.

section_title: Letter
paragraphs: Thank you for the opportunity to discuss our work on the Transportation  Security Administration’s (TSA) efforts in implementing expedited  screening, including TSA PreTM and the Managed Inclusion process. In  2011, TSA began developing new security procedures intended to  strengthen security and improve the passenger experience by shortening  lines and wait times. These new procedures applied risk-based,  intelligence-driven screening concepts and enhanced the use of  technology to determine passenger risk prior to travel. The procedures  were intended to allow TSA to devote more time and resources at the  airport to screening the passengers TSA determined to be higher or  unknown risk while providing expedited screening to those passengers  determined to pose a lower risk to the aviation system. Further, TSA  developed the Managed Inclusion process, designed to provide expedited  screening to passengers not deemed low risk prior to arriving at the  airport. In March 2015, TSA officials stated they provided expedited  screening at essentially all of the approximately 450 airports at which TSA  performs or oversees security screening, including 126 airports where  TSA offers expedited screening in dedicated TSA PreTM screening  lanes. The 126 airports where expedited screening is offered in dedicated  TSA PreTM screening lanes represent about 95 percent of total air  carrier enplanements based on Federal Aviation Administration calendar  year 2013 data.
My testimony today addresses (1) how TSA assesses the risk levels of  passengers to determine their eligibility to receive expedited screening  and (2) the extent to which TSA determined the effectiveness of its  Managed Inclusion process. This statement is based on a report we  issued in December 2014 and selected updates from March 2015. For  this report, we analyzed TSA documentation including expedited  screening and Managed Inclusion procedures, memorandums of  agreement, and decision memorandums, among other documents to gain  an understanding of how expedited screening operated, TSA’s risk  assessment methodologies, and TSA’s security assessment of the  Managed Inclusion process. In addition we visited six airports to observe  expedited screening and Managed Inclusion and interviewed TSA federal  security directors (FSD) and other TSA officials about how expedited  screening and Managed Inclusion were implemented at these airports.  Because we selected a nonprobability sample of airports to visit, the  information obtained cannot be generalized to all airports. However, the  site visits provided illustrative examples of how TSA PreTM and  Managed Inclusion operate. We also reviewed our prior work on the  TSA’s behavior detection and analysis program and Advanced Imaging  Technology to inform our understanding of the overall checkpoint  screening process and TSA’s past experience with evaluating and testing  security related programs. A more detailed methodology can be found in  the respective reports upon which this testimony is based.
The work upon which this testimony is based was conducted in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: When TSA began offering expedited screening at airports in the summer  of 2011, transportation security officers (TSO) initially provided such  screenings in standard lanes to passengers aged 12 and younger, and  subsequently extended expedited screening to certain flight crew  members and then to passengers aged 75 and older. In October 2011,  TSA began to expand the concept of expedited airport screening to more  of the flying public by piloting the TSA PreTM program. This pilot  program allowed certain frequent fliers of two air carriers to experience  expedited screening at four airports. These frequent fliers became  eligible for screening in dedicated expedited screening lanes, called TSA  PreTM lanes, because they had opted into the TSA PreTM program  through the air carrier with which they had attained frequent flier status.  TSA also allowed certain members of the U.S. Customs and Border  Protection’s (CBP) Trusted Traveler programs to experience expedited  screening as part of the TSA PreTM pilot. TSA provided expedited  screening in dedicated screening lanes to these frequent fliers and  eligible CBP Trusted Travelers during the TSA PreTM pilot program  because TSA used information available to it to determine that eligible  passengers in these groups were lower risk. When traveling on one of the  air carriers and departing from one of the airports participating in the pilot,  these passengers were eligible to be screened in dedicated TSA PreTM  screening lanes where the passengers were not required to remove their  shoes; divest light outerwear, jackets, and belts; or remove liquids, gels,  and laptops from carry-on baggage.
Since October 2011, TSA has further expanded the known traveler  populations eligible for expedited screening. After TSA piloted TSA  PreTM with certain passengers who are frequent fliers and members of  CBP’s Trusted Traveler programs, TSA established separate TSA PreTM  lists for additional low-risk passenger populations, including members of  the U.S. armed forces, Congressional Medal of Honor Society Members,  members of the Homeland Security Advisory Council, and Members of  Congress, among others.other agencies or entities, TSA created its own TSA PreTM list  composed of individuals who apply to be preapproved as low-risk  travelers through the TSA PreTM Application Program, an initiative  launched in December 2013. To apply, individuals must visit an  enrollment center where they provide biographic information (i.e., name,  date of birth, and address), valid identity and citizenship documentation,   In addition to TSA PreTM lists sponsored by  and fingerprints to undergo a TSA Security Threat Assessment. TSA  leveraged existing federal capabilities to both enroll and conduct threat  assessments for program applicants using enrollment centers previously  established for the Transportation Worker Identification Credential  Program, and existing transportation vetting systems to conduct applicant   Applicants must be U.S. citizens, U.S. nationals or  threat assessments.lawful permanent residents, and cannot have been convicted of certain  crimes. As of March 2015, about 7.2 million individuals were eligible,  through TSA PreTM lists, for expedited screening. Figure 1 shows the  populations for each TSA PreTM list.
In addition to passengers who are included on one of the TSA PreTM  lists, in October 2013, TSA continued to expand the opportunities for  expedited screening to broader groups of passengers through the TSA  PreTM Risk Assessment program and the Managed Inclusion process,  both of which are described in greater detail below. Figure 2 shows a  snapshot from February 25, 2015, through March 3, 2015, of the  percentage of weekly passengers receiving non-expedited screening and  expedited screening, and further shows whether known crew members  experienced expedited screening, and whether expedited screening  occurred in TSA PreTM lanes (for passengers designated as known  travelers or through the TSA PreTM Risk Assessment program, or  passengers chosen for expedited screening using Managed Inclusion), or  in standard lanes.
As noted in figure 2, during the week ending March 3, 2015, 28 percent of  passengers nationwide received expedited screening were issued TSA  PreTM boarding passes, but were provided expedited screening in a  standard screening lane, meaning that they did not have to remove their  shoes, belts, and light outerwear, but they had to divest their liquids, gels,  and laptops. TSA provides expedited screening to TSA PreTM -eligible  passengers in standard lanes when airports do not have dedicated TSA  PreTM screening lanes because of airport space constraints and limited  TSA PreTM throughput.
section_title: TSA Uses Three Methods to Assess Risk for Participation in Expedited Screening
paragraphs: As we found in 2014, TSA determines a passenger’s eligibility for or  opportunity to experience expedited screening at the airport using one of  three risk assessment methods. These include (1) inclusion on a TSA  PreTM list of known travelers, (2) identification of passengers as low risk  by TSA’s Risk Assessment algorithm, or (3) a real-time threat  assessment at the airport using the Managed Inclusion process.
subsections: section_title: Passenger Eligibility based on TSA PreTM Lists of Known Travelers
paragraphs: TSA has determined that the individuals included on the TSA PreTM lists  of known travelers are low risk by virtue of their membership in a specific  group or based on group vetting requirements. For example, TSA  determined that members of the Congressional Medal of Honor Society, a  group whose members have been awarded the highest U.S. award for  valor in action against enemy forces, present a low risk to transportation  security and are appropriate candidates to receive expedited screening.  In other cases, TSA determined that members of groups whose members  have undergone a security threat assessment by the federal government,  such as individuals working for agencies in the intelligence community  and who hold active Top Secret/Sensitive Compartmentalized Information  clearances, are low risk and can be provided expedited screening.  Similarly, TSA designated all active and reserve service members of the  United States armed forces, whose combined members total about 2  million people, as a low risk group whose members were eligible for  expedited screening. TSA determined that active duty military members  were low risk and appropriate candidates to receive expedited screening  because the Department of Defense administers common background  checks of its members.
Members of the list-based, low-risk populations who requested, or were otherwise  deemed eligible, to participate in TSA PreTM were provided a unique known traveler  number. Their personal identifying information (name and date of birth) along with the  known traveler number are included on lists used by Secure Flight for screening. To be  recognized as low risk by the Secure Flight system, individuals on TSA PreTM lists with  known traveler numbers must submit these numbers when making a flight reservation. maintain them by ensuring that individuals continue to meet the criteria for  inclusion and to update the lists as needed.
TSA also continues to provide expedited screening on a per-flight basis to  the almost 1.5 million frequent fliers who opted to participate in the TSA  PreTM program pilot. According to TSA, this group of eligible frequent  fliers met the standards set for the pilot based on their frequent flier status  as of October 1, 2011. According to TSA officials, TSA determined that  these frequent fliers were an appropriate population to include in the  program for several reasons, including the fact that frequent fliers are  vetted against various watchlists, such as the No-Fly list, each time they  travel to ensure that they are not listed as known or suspected terrorists,  and are screened appropriately at the checkpoint.
section_title: Passenger Eligibility Based On TSA PreTM Risk Assessments
paragraphs: As we found in December 2014, the TSA PreTM Risk Assessment  program evaluates passenger risk based on certain information available  for the passenger’s specific flight and determines the likelihood that  passengers will be designated as eligible to receive expedited screening  through TSA PreTM. Beginning in 2011, TSA piloted the process of using  the Secure Flight system to obtain Secure Flight Passenger Data from air  carriers and other data to assess whether the passenger is low risk on a  per-flight basis and thus eligible to receive a TSA PreTM designation on  his or her boarding pass to undergo expedited screening. In September  2013 after completing this pilot, TSA decided to explore expanding this  risk assessment approach to every traveler. In order to develop the set of  low-risk rules used to determine the passengers’ relative risk, TSA formed  an Integrated Project Team consisting of officials from the Offices of  Security Operations, Intelligence and Analysis, Security Capabilities, and  Risk-Based Security. The team used data from multiple sources,  including passenger data from the Secure Flight system from calendar  year 2012, to derive a baseline level of relative risk for the entire  passenger population. Our review of TSA’s documentation in our 2014  report showed that TSA considered the three elements of risk  assessment—Threat, Vulnerability, and Consequence—in its  development of the risk assessment. These three elements constitute the  framework for assessing risk as called for in the Department of Homeland  Security’s National Infrastructure Protection Plan. We found that TSA  worked with a contractor to evaluate the data elements taken from  information available for passengers’ specific flights and the proposed risk  model rules used to determine the baseline level of relative risk. In its  assessment of the algorithm used for the analysis, the contractor agreed  with TSA’s analysis of the relationship between the data elements and  relative risk assigned to the data elements. TSA officials stated that as  of March 2015, the agency is continuing to refine the algorithm to include  additional variables to help determine passenger risk.
As we found in December 2014, although TSA determined that certain  combinations of data elements in its risk-based algorithm are less likely to  include unknown potential terrorists, it also noted that designating  passengers as low risk based solely on the algorithm carries some risk.  To mitigate these risks, TSA uses a random exclusion factor that places  passengers, even those who are otherwise eligible for expedited  screening, into standard screening a certain percentage of the time. TSA  adjusts the level of random exclusion based on the relative risk posed by  the combinations of various data elements used in the algorithm. The  result is that passengers associated with some data combinations that  carry more risk are randomly excluded from expedited screening more  often than passengers associated with other data combinations. For  example, TSA’s assessment indicated that combinations of certain data  elements are considered relatively more risky than other data groups and  passengers who fit this profile for a given flight should seldom be eligible  for expedited screening, while combinations of other data on a given flight  pose relatively less risk and therefore passengers who fit these  combinations could be made eligible for expedited screening a majority of  the time. TSA developed a risk algorithm that scores each passenger on  each flight, and passengers with a high enough score receive a TSA  PreTM boarding pass designation making them eligible for expedited  screening for that trip.
section_title: Passenger Eligibility Based on Real Time Threat Assessments Using Managed Inclusion Process
paragraphs: As we found in December 2014, Managed Inclusion is designed to  provide expedited screening to passengers not deemed low risk prior to  arriving at the airport. TSA uses Managed Inclusion as a tool to direct  passengers who are not on a TSA PreTM list, or designated as eligible  for expedited screening via the TSA PreTM Risk Assessments, into the  expedited screening lanes to increase passenger throughput in these  lanes when the volume of TSA PreTM -eligible passengers is low. In  addition, TSA developed Managed Inclusion to improve the efficiency of  dedicated TSA PreTM screening lanes as well as to help TSA reach its  internal goal of providing expedited screening to at least 25 percent of  passengers by the end of calendar year 2013.
To operate Managed Inclusion, TSA randomly directs a certain  percentage of passengers not previously designated that day as eligible  for expedited screening to the TSA PreTM expedited screening lane. To  screen passengers who have been randomly directed into the expedited  screening lane, TSA uses real time threat assessments including  combinations of Behavior Detection Officers (BDOs), canine teams and  Explosives Trace Detection (ETD) devices to ensure that passengers do  not exhibit high-risk behaviors or otherwise present a risk at the airport.
According to TSA, it designed the Managed Inclusion process using a  layered approach to provide security when providing expedited screening  to passengers via Managed Inclusion. Specifically, these layers include  (1) the Secure Flight vetting TSA performs to identify high-risk  passengers required to undergo enhanced screening at the checkpoint  and to ensure these passengers are not directed to TSA PreTM  expedited screening lanes, (2) a randomization process that TSA uses to  include passengers into TSA PreTM screening lanes who otherwise were  not eligible for expedited screening, (3) BDOs who observe passengers  and look for certain high-risk behaviors, (4) canine teams and ETD  devices that help ensure that passengers have not handled explosive  materials prior to travel, and (5) an unpredictable screening process  involving walk-through metal detectors in expedited screening lanes that  randomly select a percentage of passengers for additional screening.
subsections: section_title: Managed Inclusion Process
paragraphs: When passengers approach a security checkpoint that is operating  Managed Inclusion, they approach a TSO who is holding a randomizer  device, typically an iPad that directs the passenger to the expedited or  standard screening lane. TSA officials stated that the randomization layer  of security is intended to ensure that passengers cannot count on being  screened in the expedited screening lane even if they use a security  checkpoint that is operating Managed Inclusion. FSDs can adjust the  percentage of passengers randomly sent into the Managed Inclusion lane  depending on specific risk factors. Figure 3 illustrates how these layers of  security operate when FSDs use Managed Inclusion lanes.
According to TSA, it designed the Managed Inclusion process to use  BDOs stationed in the expedited screening lane as one of its layers of  security when Managed Inclusion is operational to observe passengers’  behavior as they move through the security checkpoint queue. When  BDOs observe certain behaviors that indicate a passenger may be higher  risk, the BDOs are to refer the passenger to a standard screening lane so  that the passenger can be screened using standard or enhanced  screening procedures. In our November 2013 report on TSA’s behavior  detection and analysis program, we concluded that although TSA had  taken several positive steps to validate the scientific basis and strengthen  program management of behavior detection and analysis program, TSA  had not demonstrated that BDOs can reliably and effectively identify high- risk passengers who may pose a threat to the U.S. aviation system.  Further, we recommended that the Secretary of Homeland Security direct  the TSA Administrator to limit future funding support for the agency’s  behavior detection activities until TSA can provide scientifically validated  evidence that demonstrates that behavioral indicators can be used to  identify passengers who may pose a threat to aviation security. The  Department of Homeland Security did not concur with this  recommendation, in part, because it disagreed with GAO’s analysis of  TSA’s behavioral indicators. In February 2015, TSA officials told us that  they had revised the behavioral indicators, were conducting pilot tests on  the use of new BDO protocols, and anticipated concluding the testing at 5  airports in late 2015. At that time, TSA plans to make a determination  about whether the new protocols are ready for further testing, including an  operational test in 10 airports to determine the protocols’ effectiveness,  which has an estimated completion date in the latter half of 2016.
According to a TSA decision memorandum and its accompanying  analysis, TSA uses canine teams and ETD devices at airports as an  additional layer of security when Managed Inclusion is operational to  determine whether passengers may have interacted with explosives prior  to arriving at the airport. In airports with canine teams, passengers must  walk past a canine and its handler in an environment where the canine is  trained to detect explosive odors and to alert the handler when a  passenger has any trace of explosives on his or her person. For example,  passengers in the Managed Inclusion lane may be directed to walk from  the travel document checker through the passageway and past the  canine teams to reach the X-ray belt and the walk-through metal detector.  According to TSA documents, the canines, when combined with the other  layers of security in the Managed Inclusion process provide effective  security. According to TSA, it made this determination by considering  the probability of canines detecting explosives on passengers, and then  designed the Managed Inclusion process to ensure that passengers  would encounter a canine a certain percentage of the time.
Our prior work examined data TSA had on its canine program, what these  data showed, and to what extent TSA analyzed these data to identify  program trends. Further we analyzed the extent to which TSA deployed  canine teams using a risk-based approach and determined their  effectiveness prior to deployment. As a result of this work, we  recommended in January 2013, among other things, that TSA take  actions to comprehensively assess the effectiveness of canine teams.  The Department of Homeland Security concurred with this  recommendation and has taken steps to address it. Specifically, according to TSA canine test results, TSA has conducted work to assess  canine teams and to ensure they meet the security effectiveness  thresholds TSA established for working in the Managed Inclusion lane,  and the canines met these thresholds as a requirement to screen  passengers in Managed Inclusion lanes.
In those airports where canines are unavailable or not working, TSA uses  ETD devices as a layer of security when operating Managed Inclusion.  TSOs stationed at the ETD device are to select passengers to have their  hands swabbed as they move through the expedited screening lane.  TSOs are to wait for a passenger to proceed through the Managed  Inclusion queue and approach the device, where the TSO is to swab the  passenger’s hands with an ETD pad and place the pad in the ETD device  to determine whether any explosive residue is detected on the pad.  Once the passenger who was swabbed is cleared, the passenger then  proceeds through the lane to the X-ray belt and walk-through metal  detector for screening. TSA procedures require FSDs to meet certain  performance requirements when ETD devices are operating, such as  swabbing passengers at a designated rate, and TSA data from January 1,  2014, through April 1, 2014, show that these requirements were not  always met. Beginning in May 2014, TSA’s Office of Security Operations  began tracking compliance with the ETD swab requirements and  developed and implemented a process to ensure that the requirements  are met. In March 2015 TSA officials confirmed this process was still in  place.
According to TSA, it uses unpredictable screening procedures as an  additional layer of security after passengers who are using expedited  screening pass through the walk-through metal detector. This random  selection of passengers for enhanced screening occurs after they have  passed all security layers TSA uses for Managed Inclusion, and provides  one more chance for TSA to detect explosives on a passenger.
section_title: TSA Has Not Tested the Overall Effectiveness of the Managed Inclusion Process
paragraphs: As we reported in December 2014, according to TSA, it designed the  Managed Inclusion process using a layered approach to security when  providing expedited screening to passengers via Managed Inclusion.  Specifically, the Office of Security Capabilities’ proof of concept design  noted that the Managed Inclusion process was designed to provide a  more rigorous real-time threat assessment layer of security when  compared to standard screening or TSA PreTM screening. According to  the design concept, this real-time threat assessment, utilizing both BDOs  and explosives detection, allows TSA to provide expedited screening to  passengers who have not been designated as low risk without decreasing  overall security effectiveness. While TSA has tested the security  effectiveness of each of these layers of security, TSA has not yet tested  the security effectiveness of the overall Managed Inclusion process as it  functions as a whole.
subsections: section_title: Conducting Effectiveness Testing of the Managed Inclusion Process in Accordance with Evaluation Best Practices Would Enhance Reliability of the Process
paragraphs: GAO-14-159, GAO-10-763, GAO-13-239, GAO-14-695T, and GAO-11-740. addressing those recommendations but they have not yet been fully  implemented. TSA officials stated that they have not yet tested the  security effectiveness of the overall Managed Inclusion process as it  functions as a whole, as TSA has been planning for such testing over the  course of the last year. TSA documentation shows that the Office of  Security Capabilities recommended in January 2013 that TSA test the  security effectiveness of Managed Inclusion as a system. We reported in  2014 that according to officials, TSA anticipated that testing would begin  in October 2014 and estimated that testing could take 12 to 18 months to  complete. In March 2015, TSA officials provided us a schedule for the  development and completion of BDO and Canine testing supporting the  Managed Inclusion process. TSA scheduled a pilot for testing BDOs  which was set to begin October 2014 and run through May 2015. Further,  the schedule TSA provided indicates that a proof of concept for Canine  Covert Testing was scheduled for November 2014 and that operational  testing of canines was scheduled to begin in June 2015 and be  completed in March 2016. Testing the security effectiveness of the  Managed Inclusion process is consistent with federal policy, which states  that agencies should assess program effectiveness and make  improvements as needed.
We have previously reported on challenges TSA has faced in designing  studies and protocols to test the effectiveness of security systems and  programs in accordance with established methodological practices. For  example, in our March 2014 assessment of TSA’s acquisition of  Advanced Imaging Technology, we found that TSA conducted operational  and laboratory tests, but did not evaluate the performance of the entire  system, which is necessary to ensure that mission needs are met. element of evaluation design is to define purpose and scope, to establish  what questions the evaluation will and will not address.
See GAO-14-357. We recommend that TSA better measure the effectiveness of its  entire AIT system. TSA concurred with the recommendation. and analysis program. For example we found that TSA did not randomly  select airports to participate in the study, so the results were not  generalizeable across airports. In addition, we found that TSA collected  the validation study data unevenly and experienced challenges in  collecting an adequate sample size for the randomly selected  passengers, facts that might have further affected the representativeness  of the findings. According to established evaluation design practices, data  collection should be sufficiently free of bias or other significant errors that  could lead to inaccurate conclusions.
In our December 2014 report we concluded that ensuring the planned  effectiveness testing of the Managed Inclusion process adheres to  established evaluation design practices would help TSA provide  reasonable assurance that the effectiveness testing will yield reliable  results. The specific design limitations we identified in TSA’s previous  studies of Advanced Imaging Technology and behavior detection and  analysis program may or may not be relevant design issues for an  assessment of the effectiveness of the Managed Inclusion process, as  evaluation design necessarily differs based on the scope and nature of  the question being addressed. In general, evaluations are most likely to  be successful when key steps are addressed during design, including  defining research questions appropriate to the scope of the evaluation,  and selecting appropriate measures and study approaches that will permit  valid conclusions. As a result, we recommended that to ensure that TSA’s  planned testing yields reliable results, the TSA Administrator take steps to  ensure that TSA’s planned effectiveness testing of the Managed Inclusion  process adheres to established evaluation design practices. DHS  concurred with our recommendations and began taking steps to ensure  that its planned effectiveness testing of the Managed Inclusion process  adheres to established evaluation practices. Specifically, DHS stated that  TSA plans to use a test and evaluation process—which calls for the  preparation of test and evaluation framework documents including plans,  analyses, and a final report describing the test results—for its planned  effectiveness testing of Managed Inclusion.
Chairman Katko, Ranking Member Rice, and Members of the  Subcommittee, this completes my prepared statement. I would be  pleased to respond to any questions that you may have at this time.
section_title: GAO Contact and Staff Acknowledgements
paragraphs: For questions about this statement, please contact Jennifer Grover at  (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this statement  include Glenn Davis (Assistant Director), Brendan Kretzschmar, Ellen  Wolfe, David Alexander, Thomas Lombardi, Susan Hsu, Caroline  Neidhold, and Serena Epstein. Key contributors for the previous work that  this testimony is based on are listed in each product.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.