Aviation Security: Significant Management Challenges May Adversely Affect Implementation of the Transportation Security Administration's Secure Flight Program
Published: 2006-02-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: After the events of September 11, 2001, Congress created the Transportation Security Administration (TSA) and directed it to assume the function of passenger prescreening--or the matching of passenger information against terrorist watch lists to identify persons who should undergo additional security scrutiny--for domestic flights, which is currently performed by the air carriers. To do so, TSA is developing Secure Flight. This testimony covers TSA's progress and challenges in (1) developing, managing, and overseeing Secure Flight; (2) coordinating with key stakeholders critical to program operations; (3) addressing key factors that will impact system effectiveness; and (4) minimizing impacts on passenger privacy and protecting passenger rights. This testimony includes information on areas of congressional interest that GAO has previously reported on.
section_title: What GAO Found
paragraphs: TSA has made some progress in developing and testing the Secure Flight program. However, TSA has not followed a disciplined life cycle approach to manage systems development, or fully defined system requirements. Rather, TSA has followed a rapid development method intended to develop the program quickly. This process has been ad hoc, resulting in project activities being conducted out of sequence, requirements not being fully defined, and documentation containing contradictory information or omissions. Further, while TSA has taken steps to implement an information security management program for protecting information and assets, its efforts are incomplete. Finally, TSA is proceeding to develop Secure Flight without a program management plan containing program schedule and cost estimates. Oversight reviews of the program have also raised questions about program management. Without following a more rigorous and disciplined life cycle process, including defining system requirements, the Secure Flight program is at serious risk of not meeting program goals. Over the past year, TSA has made some progress in managing risks associated with developing Secure Flight, and has recently taken actions that recognize the need to install more rigor and discipline into the development process. TSA has also taken steps to collaborate with Secure Flight stakeholders whose participation is essential to ensuring that passenger and terrorist watch list data are collected and transmitted to support Secure Flight. However, key program stakeholders--including the U.S. Customs and Border Protection, the Terrorist Screening Center, and air carriers--stated that they need more definitive information about system requirements from TSA to plan for their support of the program. In addition, several activities that will affect Secure Flight's effectiveness are under way, or have not yet been decided. For example, TSA conducted name-matching tests, which compared passenger and terrorist screening database data, to evaluate the ability of the system to function. However, TSA has not yet made key policy decisions which could significantly impact program operations, including what passenger data it will require air carriers to provide and the name-matching technologies it will use. Further, Secure Flight's system development documentation does not fully explain how passenger privacy protections are to be met, and TSA has not issued the privacy notices that describe how it will protect passenger data once Secure Flight becomes operational. As a result, it is not possible to assess how TSA is addressing privacy concerns. TSA is also determining how it will provide for redress, as mandated by Congress, to provide aviation passengers with a process to appeal determinations made by the program and correct erroneous information contained within the prescreening process. However, TSA has not finalized its redress polices.

section_title: Letter
paragraphs: Thank you for inviting me to participate in today’s hearing on the  Transportation Security Administration’s (TSA) Secure Flight program.  The purpose of Secure Flight is to enable our government to protect the  public and strengthen aviation security by identifying and scrutinizing  individuals suspected of having ties to terrorism, or who may otherwise  pose a threat to aviation, in order to prevent them from boarding  commercial aircraft in the United States, if warranted, or by subjecting  them to additional security scrutiny prior to boarding an aircraft. The  program also aims to reduce the number of individuals unnecessarily  selected for secondary screening while protecting passengers’ privacy and  civil liberties. My testimony today presents information on the progress  TSA has made and the challenges it faces in (1) developing, managing, and  overseeing the Secure Flight program; (2) coordinating with federal and  private sector stakeholders who will play critical roles in Secure Flight  operations; (3) addressing key factors that will impact system  effectiveness; and (4) minimizing program impacts on passenger privacy  and protecting passenger rights.
My testimony is based on our past reviews of the Secure Flight program,  and on preliminary results from our ongoing review of 10 issues related to  the development and implementation of Secure Flight, as mandated by  Public Law 109-90, and as requested by eight congressional committees.  (See app. 1 for a description of the 10 issues.) My testimony today updates  information presented in our March 2005 report on the status of Secure  Flight’s development and implementation, including 9 of the 10 areas of  congressional interest. In March 2005, we reported that TSA had made  progress in developing and testing Secure Flight, but had not completed  key system testing, had not finalized system requirements or determined  how certain aspects of the program would operate (such as the basis on  which passengers would be selected for preflight scrutiny), and had not  clearly defined the privacy impacts of the program. At the time, we  recommended that TSA take several actions to manage the risks  associated with developing and implementing Secure Flight, including  finalizing system requirements and test plans, privacy and redress  requirements, and program cost estimates.
Today, I present information that suggests that, 3 years after TSA began  developing a program to provide passenger prescreening, significant  challenges remain in developing and implementing the Secure Flight  program. The results I am presenting are based on our review of available  documentation on Secure Flight’s systems development and oversight,  policies governing program operations, and our past reports on the  program, and interviews with Department of Homeland Security (DHS)  officials, TSA program officials and their contractors, and other federal  officials who are key stakeholders in the Secure Flight program. We  reviewed TSA’s System Development Life Cycle Guidance for developing  information technology systems, and other federal reports describing best  practices in developing and acquiring these systems. We also reviewed  draft TSA documents containing information on the development and  testing of Secure Flight, including concept of operations, requirements,  test plans, and test results. My testimony is based on TSA documents  received, but does not necessarily reflect all documentation that was only  recently made available. In addition to the TSA documents we have  reviewed, we also reviewed reports from the U.S. Department of Justice  Office of the Inspector General (DOJ-OIG), which reviewed the Secure  Flight program, and reports from two oversight groups that provided  advisory recommendations for Secure Flight: DHS’s Privacy and Data  Integrity Advisory Committee and TSA’s Aviation Security Advisory  Committee Secure Flight Working Group. We interviewed senior-level TSA  officials, including representatives from the Office of Transportation  Threat Analysis and Credentialing, which is responsible for Secure Flight,  and the Office of Transportation Security Redress (OTSR), to obtain  information on Secure Flight’s planning, development, testing, and policy  decisions. We also interviewed representatives from the U.S. Customs and  Border Protection (CBP) and Terrorist Screening Center (TSC) to obtain  information about stakeholder coordination. We also interviewed officials  from an air carrier and representatives from aviation trade organizations  regarding issues related to Secure Flight’s development and  implementation. In addition, we attended conferences on name-matching  technologies sponsored by MITRE (a federally funded research and  development corporation) and the Office of the Director of National  Intelligence. Our work was conducted from April 2005 to February 2006 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: TSA is responsible for securing all modes of transportation while  facilitating commerce and the freedom of movement for the traveling  public. Passenger prescreening is one program among many that TSA uses  to secure the domestic aviation sector. The process of prescreening  passengers—that is, determining whether airline passengers might pose a  security risk before they reach the passenger-screening checkpoint—is  used to focus security efforts on those passengers that represent the  greatest potential threat. Currently, U.S. air carriers conduct passenger  prescreening by comparing passenger names against government-supplied  terrorist watch lists and applying the Computer-Assisted Passenger  Prescreening System rules, known as CAPPS rules.
subsections: section_title: Development of Legacy Passenger Prescreening Systems
paragraphs: Following the events of September 11, and in accordance with the  requirement set forth in the Aviation and Transportation Security Act that  a computer-assisted passenger prescreening system be used to evaluate all  passengers before they board an aircraft, TSA established the Office of  National Risk Assessment to develop and maintain a capability to  prescreen passengers in an effort to protect U.S. transportation systems  and the public against potential terrorists. In March 2003, this office began  developing the second-generation computer-assisted passenger  prescreening system, known as CAPPS II, to provide improvements over  the current prescreening process, and to screen all passengers flying into,  out of, and within the United States.
Based in part on concerns about privacy and other issues expressed by us  and others, DHS canceled the development of CAPPS II in August 2004  and shortly thereafter announced that it planned to develop a new  passenger prescreening program called Secure Flight. In contrast to  CAPPS II, Secure Flight, among other changes, will only prescreen  passengers flying domestically within the United States, rather than  passengers flying into and out of the United States. Also, the CAPPS rules  will not be implemented as part of Secure Flight, but rather the rules will  continue to be applied by commercial air carriers. Secure Flight will  operate on the Transportation Vetting Platform (TVP)—the underlying  infrastructure (hardware and software) to support the Secure Flight  application, including security, communications, and data management;  and, the Secure Flight application is to perform the functions associated  with receiving, vetting, and returning requests related to the determination  of whether passengers are on government watch lists. This application is  also to be configurable—meaning that it can be quickly adjusted to reflect  changes to workflow parameters. Aspects of Secure Flight are currently  undergoing development and testing, and policy decisions regarding the  operations of the program have not been finalized.
section_title: Overview of Secure Flight Operations
paragraphs: As currently envisioned, under Secure Flight, when a passenger makes  flight arrangements, the organization accepting the reservation, such as  the air carrier’s reservation office or a travel agent, will enter passenger  name record (PNR) information obtained from the passenger, which will  then be stored in the air carrier’s reservation system. While the  government will be asking for only portions of the PNR, the PNR data can  include the passenger’s name, phone number, number of bags, seat  number, and form of payment, among other information. Approximately  72 hours prior to the flight, portions of the passenger data contained in the  PNR will be sent to Secure Flight through a network connection provided  by DHS’s CBP. Reservations or changes to reservations that are made less  than 72 hours prior to flight time will be sent immediately to TSA through  CBP.
Upon receipt of passenger data, TSA plans to process the passenger data  through the Secure Flight application running on the TVP. During this  process, Secure Flight is to determine if the passenger data match the data  extracted daily from TSC’s Terrorist Screening Database (TSDB)—the  information consolidated by TSC from terrorist watch lists to provide  government screeners with a unified set of terrorist-related information. In  addition, TSA will screen against its own watch list composed of  individuals who do not have a nexus to terrorism but who may pose a  threat to aviation security.
In order to match passenger data to information contained in the TSDB,  TSC plans to provide TSA with an extract of the TSDB for use in Secure  Flight, and provide updates as they occur. This TSDB subset will include  all individuals classified as either selectees (individuals who are selected  for additional security measures prior to boarding an aircraft) or no-flys  (individuals who will be denied boarding unless they are cleared by law  enforcement personnel). To perform the match, Secure Flight is to  compare the passenger, TSDB, and other watch list data using automated  name-matching technologies. When a possible match is generated, TSA  and potentially TSC analysts will conduct a manual review comparing  additional law enforcement and other government information with  passenger data to determine if the person can be ruled out as a possible  match. TSA is to return the matching results to the air carriers through  CBP. Figure 1 illustrates how Secure Flight is intended to operate.
As shown in figure 1, when the passenger checks in for the flight at the  airport, the passenger is to receive a level of screening based on his or her  designated category. A cleared passenger is to be provided a boarding pass  and allowed to proceed to the screening checkpoint in the normal manner.  A selectee passenger is to receive additional security scrutiny at the  screening checkpoint. A no-fly passenger will not be issued a boarding  pass. Instead, appropriate law enforcement agencies will be notified. Law  enforcement officials will determine whether the individual will be  allowed to proceed through the screening checkpoint or if other actions  are warranted, such as additional questioning of the passenger or taking  the passenger into custody.
section_title: TSA Has Not Followed a Disciplined Life Cycle Approach or Fully Defined System Requirements, Schedule, and Costs
paragraphs: TSA has not followed a disciplined life cycle approach in developing  Secure Flight, in accordance with best practices for large-scale  information technology programs. Following a disciplined life cycle,  activities and related documentation are to be developed in a logical  sequence. TSA also has not finalized and documented functional and  system requirements that fully link to each other and to source documents.  Without adequately defined requirements, TSA cannot finalize a system  security plan or develop a reliable program schedule or life cycle cost  estimates. In addition to these concerns, other reviews that have been  conducted of Secure Flight have raised questions about the management  of the program.
subsections: section_title: TSA Has Not Followed a Disciplined Life Cycle Process or Fully Defined System Requirements but Plans to Address These Issues
paragraphs: Based on evaluations of major federal information technology programs  like Secure Flight, and research by others, following a disciplined life cycle  management process in which key activities and phases of the project are  conducted in a logical and orderly process and are fully documented,  helps ensure that programs achieve intended goals within acceptable  levels of cost and risk. Such a life cycle process begins with initial concept  definition and continues through requirements determination to final  testing, implementation, and maintenance. TSA has established a System  Development Life Cycle (SDLC) that defines a series of orderly phases and  associated steps and documentation. The SDLC serves as the mechanism  to ensure that systems are effectively managed and overseen. Figure 2  provides a description of TSA’s SDLC phases and related documentation.
TSA has not followed its SDLC in developing and managing Secure Flight.  Rather, program officials stated that they have used a rapid development  method that was intended to enable them to develop the program more  quickly. However, these officials could not provide us with details on how  this approach was implemented. As a result, our analysis of steps  performed and documentation developed indicates that Secure Flight has  not been pursued within the context of a logical, disciplined, system  development methodology. Rather the process has been ad hoc, with  project activities conducted out of sequence. For example, program  officials declared that the program’s design phase was completed before  system requirements had been adequately detailed, and key activities have  yet to be adequately performed, such as program planning and defining  system requirements. TSA officials acknowledged that problems arose  with Secure Flight as a result of using this approach. As a result, it is  currently unclear what Secure Flight capabilities are to be developed, by  when, at what cost, and what benefits are to accrue from the program.  Without clarification on these decision points, the program is at risk of  failure.
Defining and documenting system requirements is integral to life cycle  development. Based on best practices and our prior work in this area, the  expected capabilities of a system such as Secure Flight should be defined  in terms of requirements for functionality (what the system is to do),  performance (how well the system is to execute functions), data (what  data are needed by what functions, when, and in what form), interface  (what interactions with related and dependent systems are needed), and  security. Further, system requirements should be unambiguous, consistent  with one another, linked (that is, traceable from one source level to  another), verifiable, understood by stakeholders, and fully documented.
TSA has prepared certain Secure Flight requirements documents, and  officials stated that they are now reviewing those requirements  documents. We support these review efforts because we found, in the  requirements documents we reviewed, inconsistencies and ambiguities in  requirements documentation for system functions, performance, data, and  security—and that these documents were not always complete. For  example, according to TSA’s SDLC guidance and best practices for  developing information technology systems, systems like Secure Flight  should have a comprehensive concept of operations covering all aspects  of the program during the planning phase (see fig. 2). We reported in our  March 2005 report that TSA had not yet finalized a concept of operations,  which would describe conceptually the full range of Secure Flight  operations and interfaces with other systems, and we recommended that it  develop one. Since March 2005, TSA documents refer to numerous  concept of operations, such as a long concept of operations, a short  concept of operations, and an initial operational capability concept of  operations. TSA provided a June 2005 concept of operations for our  review, but this document does not contain key system requirements, such  as the high-level requirements for security and privacy.
In addition, we found that Secure Flight requirements were unclear or  missing. For example, while the requirements that we reviewed state that  the system be available 99 percent of the time, this only covers the TVP  and Secure Flight application. It does not include requirements for the  interfacing systems critical for Secure Flight operations. Thus, the  availability requirements for all of the components of the Secure Flight  system are not yet known. Some data requirements are also vague or  incomplete; for example, one data requirement is that the data is current,  but the meaning of current is not defined. In addition, only some system  security requirements are identified in the security document provided to  us for the TVP, and sections in TSA’s Systems Requirements Specification  contain only placeholder notes—“to be finalized”—for security and  privacy requirements.
TSA officials acknowledged that it is important that requirements be  traceable to ensure that they are consistently, completely, and correctly  defined, implemented, and tested. To help accomplish this, TSA officials  stated that they use a requirements tracking tool for Secure Flight that can  align related requirements to different documents, and thus establish  traceability (e.g., it can map the Systems Requirements Specification to a  functional requirements document). According to program officials, this  tool can also be used for aligning and tracing requirements to test cases  (i.e., scenarios used to determine that the system is working as intended).  We found, however, that requirements for Secure Flight have not been  fully traced. For example, we were not able to trace system capabilities in  contractual documents to the concept of operations and then to the  various requirement documents, to design phase use cases, and to test  cases. In addition, contractor staff we interviewed stated that they were  unable to use this tool to align or trace necessary requirements without the  aid of supplemental information. Without internal alignment among  system documentation relating to requirements, there is not adequate  assurance that the system produced will perform as intended.
In addition, we found that available Secure Flight requirements documents  did not define the system’s boundaries, including interfaces, for each of  the stakeholders—that is, the scope of the system from end to end, from  an air carrier to CBP, to TSA, to TSC, and back to TSA, then again to CBP  and air carriers (refer to fig. 1 for an overview of this process). Defining a  system’s boundaries is important in ensuring that system requirements  reflect all of the processes that must be executed to achieve a system’s  intended purpose. According to TSA’s SDLC guidance, a System Boundary  Document is to be developed early in the system life cycle. However, in its  third year of developing a passenger prescreening system, TSA has not yet  prepared such a document. Although the System Boundary Document was  not available, the program’s Systems Security Document does refer to an  “accreditation boundary,” which defines the Secure Flight system from the  standpoint of system security accreditation and certification. According to  this definition of what Secure Flight includes, those systems that are  needed to accomplish Secure Flight program goals (e.g., those of  commercial air carriers, CBP, and TSC) are not part of Secure Flight. If the  boundary documents, and thus the requirements, do not reflect all system  processes and connections that need to be performed, the risk is increased  that the system will not achieve Secure Flight’s intended purpose.  Moreover, until all system requirements have been defined, TSA will not  be able to stress-test Secure Flight in an operational, end-to-end mode. In  our March 2005 report, we recommended that TSA finalize its system  requirements documents and ensure that these documents address all  system functionality. Although TSA agreed with our recommendations, the  requirements documentation that we reviewed showed that the agency has  not yet completed these activities.
Our evaluations of major federal information technology programs, and  research by others, has shown that following a disciplined life cycle  management process decreases the risks associated with acquiring  systems. The steps and products in the life cycle process each have  important purposes, and they have inherent dependencies among  themselves. Thus, if earlier steps and products are omitted or deficient,  later steps and products will be affected, resulting in costly and time- consuming rework. For example, a system can be effectively tested to  determine whether it meets requirements only if these requirements have  already been fully defined. Concurrent, incomplete, and omitted activities  in life cycle management exacerbate the program risks. Life cycle  management weaknesses become even more critical as the program  continues, because the size and complexity of the program will likely only  increase, and the later problems are found, the harder and more costly  they will likely be to fix.
In October 2005, Secure Flight’s director of development stated in a  memorandum to the assistant TSA administrator responsible for Secure  Flight that by not following a disciplined life cycle approach, in order to  expedite the delivery of Secure Flight, the government had taken a  calculated risk during the requirements definition, design, and  development phases of the program’s life cycle development. The director  stated that by prioritizing delivery of the system by a specified date in lieu  of delivering complete documentation, TSA had to lower its standards of  what constituted acceptable engineering processes and documentation.  Since then, TSA officials stated that the required system documentation  associated with each phase of the TSA life cycle is now being developed to  catch up with development efforts. In addition, TSA recognized that it  faces challenges preparing required systems documentation, and to help in  this regard it has recently hired a certified systems program manager to  manage systems development. In January 2006, this program manager  stated that as Secure Flight moves forward, TSA’s SDLC would be  followed in order to instill greater rigor and discipline into the system’s  development. In addition, TSA plans to hire a dedicated program director  for Secure Flight to manage program activities, schedules, milestones,  costs, and program contractors, among other things.
section_title: Comprehensive System Security Management Program Has Not Yet Been Established in Accordance with Federal Guidance
paragraphs: TSA has taken steps to implement an information system security  management program for protecting Secure Flight information and assets.  Secure Flight’s security plans and the related security review, which TSA  developed and conducted to establish authority to operate, are important  steps in the system’s development. However, the steps related to system  security TSA has taken to date are individually incomplete, and  collectively fall short of a comprehensive system security management  program. Federal guidance and industry best practices describe critical  elements of a comprehensive information system security management  program. Without effective system security management, it is unlikely that  Secure Flight will, for example, be adequately protected against  unauthorized access and use, disruption, modification, and destruction.
According to National Institute of Standards and Technology (NIST) and  Office of Management and Budget (OMB) guidance under the Federal  Information Security Management Act, as well as industry best practices, a  comprehensive system security management program includes   (1) conducting a system wide risk assessment that is based on system  threats and vulnerabilities, (2) developing system security requirements  and related policies and procedures that govern the operation and use of  the system and address identified risks, (3) certifying that the system is  secure based on sufficient review and testing to demonstrate that the  system meets security requirements, and (4) accrediting the system as  secure in an operational setting.
TSA has developed two system security plans—one for the TVP and one  for the Secure Flight application. However, neither of these plans nor the  security activities that TSA has conducted to date are complete. For  example, while security threats and vulnerabilities were assessed in the  documentation and risks were identified in risk assessments, requirements  to address these risks were only partially defined in the security plan for  the TVP, and they were not included at all in the plan for the Secure Flight  application. In addition, the sections on security requirements and privacy  requirements in the System Requirements Specification document read “to  be finalized” with no further description.
Moreover, we also found that the security systems plans did not reflect the  current level of risk designated for the program. For example, although the  July 15, 2005, System Security Plan for the TVP arrived at an overall  assessment of its exposure to risks as being “medium,” an August 23, 2005,  requirements document found that the security risk level for the TVP was  “high.” As a system moves from a medium to a high level of risk, the  security requirements become more stringent. TSA has not provided us  with an updated System Security Plan for the TVP that addressed this  greater level of risk by including additional NIST requirements for a high- risk system. In addition, this TVP System Security Plan included only  about 40 percent of the NIST requirements associated with a medium-risk  system. Without addressing all NIST requirements, in addition to those  required for a high-risk system, TSA may not have proper controls in place  to protect sensitive information.
According to federal guidance and requirements, the determination and  approval of the readiness of a system to securely operate is accomplished  via a certification and accreditation process. On September 30, 2005, the  TSA assistant administrator responsible for Secure Flight formally granted  authority, based on certification and accreditation results, for the TVP and  the Secure Flight application to operate. However, the team performing  the certification found that TSA was unsure whether they tested all  components of the security system for the TVP and the Secure Flight  application, because TSA lacked an effective and comprehensive inventory  system. Therefore the certification team could not determine whether its  risk assessments were complete or accurate. This team also documented  62 security vulnerabilities for the Secure Flight application and 82 security  vulnerabilities for the TVP. The certification team recommended authority  to operate on the condition that corrective action or obtaining an  exemption for the identified vulnerabilities would be taken within 90 days  or the authority to operate would expire. TSA officials stated that these  vulnerabilities had been addressed except for three that are being  reviewed in a current security audit.
section_title: Program Management Plan and Supporting Schedules and Cost Estimates for Secure Flight Have Not Been Maintained
paragraphs: TSA has proceeded with Secure Flight development over the past year  without a complete and up-to-date program management plan, and  without associated cost and schedule estimates showing what work will  be done by whom, at what cost, and when. A program management plan  can be viewed as a central instrument for guiding program development.  Among other things, the plan should include a breakout of the work  activities and products that are to be conducted in order to deliver a  mission capability to satisfy stated requirements and produce promised  mission results. This information, in turn, provides the basis for  determining the time frames and resources needed for accomplishing this  work, including the basis for milestones, schedules, and cost estimates.  TSA has not provided us with either the complete and up-to-date program  management plan, or an estimated schedule and costs for Secure Flight.  According to a TSA official, an updated program management plan is  currently being developed and is about 90 percent complete.
In lieu of a program management plan with a schedule and milestones,  TSA has periodically disclosed program milestones. However, the basis for  and meaning of these milestones have not been made clear, and TSA’s  progress in meeting these milestones has not been measured and  disclosed. TSA’s SDLC and OMB guidance require that programs like  Secure Flight provide risk-adjusted schedule goals, including key  milestones, and that programs demonstrate satisfactory progress toward  achieving their stated performance goals. In March 2005, we reported that  the milestone that TSA set for achieving initial operating capability for  Secure Flight had slipped from April 2005 to August 2005. TSA officials  stated that TSA revised this milestone to state that instead of achieving  initial operating capability, it would begin operational testing. This new  milestone subsequently slipped first to September 2005, then to November  2005. Since that time, the program has not yet begun operational testing or  initial operations, and TSA has not yet produced an updated schedule  identifying when program operations will begin or when other key  milestones are to be achieved to guide program development and  implementation. Further, while agency officials stated that they are now  planning for operational testing of an unspecified capability, no milestone  date has been set for doing so.
TSA officials stated that they have not maintained an updated program  schedule for Secure Flight in part because the agency has not yet  determined the rulemaking approach it will pursue for requiring  commercial air carriers to submit certain passenger data needed to  operate Secure Flight, among other things. Specifically, TSA officials  stated that a schedule with key milestones, such as operational testing,  cannot be set until after air carriers have responded to the rulemaking and  provided their plans and schedules for participating in Secure Flight. The  rulemaking has been pending since the spring of 2005, and the rule  remains in draft form and is under review, according to TSA officials.  Once the rule has been issued, TSA officials stated that air carriers will be  given time to respond with their plans and schedules. TSA officials further  stated that until this occurs, and a decision is made as to how many air  carriers will participate in a yet-to-be-defined initial phase of the program  (they are expected to begin incrementally), a program schedule cannot be  set.
Further, TSA has not yet established cost estimates for developing and  deploying either an initial or a full operating capability for Secure Flight,  and it has not developed a life-cycle cost estimate (estimated costs over  the expected life of a program, including direct and indirect costs and  costs of operation and maintenance). TSA also has not updated its  expenditure plan—plans that generally identify near-term program  expenditures—to reflect the cost impact of program delays, estimated  costs associated with obtaining system connectivity with CBP, or  estimated costs expected to be borne by air carriers. Program and life  cycle cost estimates are critical components of sound program  management for the development of any major investment. Developing  cost estimates is also required by OMB guidance and can be important in  making realistic decisions about developing a system. Expenditure plans  are designed to provide lawmakers and other officials overseeing a  program’s development with a sufficient understanding of the system  acquisition to permit effective oversight, and to allow for informed  decision making about the use of appropriated funds.
In our March 2005 report, we recommended that TSA develop reliable life  cycle cost estimates and expenditure plans for the Secure Flight program,  in accordance with guidance issued by OMB, in order to provide program  managers and oversight officials with the information needed to make  informed decisions about program development and resource allocations.  Although TSA agreed with our recommendation, it has not yet provided  this information. TSA officials stated that developing program and life  cycle cost estimates for Secure Flight is challenging because no similar  programs exist from which to base cost estimates and because of the  uncertainties surrounding Secure Flight requirements. Further, they stated  that cost estimates cannot be accurately developed until after system  testing is completed and policy decisions have been made regarding  Secure Flight requirements and operations. Notwithstanding these  statements, TSA officials stated that they are currently assessing program  and life cycle costs as part of their rebaselining and that this new baseline  will reflect updated cost, funding, scheduling, and other aspects of the  program’s development.
While we recognize that program unknowns introduce uncertainty into the  program-planning process, including estimating tasks, time frames, and  costs, uncertainty is a practical reality in planning all programs and is not  a reason for not developing plans, including cost and schedule estimates,  that reflect known and unknown aspects of the program. In program  planning, assumptions need to be made and disclosed in the plans, along  with the impact of the associated uncertainty on the plans and estimates.  As more information becomes known over the life of the program, these  plans should be updated to recognize and reflect the greater confidence in  activities that can be expressed with estimates.
Program management plans and related schedules and cost estimates— based on well-defined requirements—are important in making realistic  decisions about a system’s development, and can alert an agency to  growing schedule or cost problems and the need for mitigating actions.  Moreover, best practices and related federal guidance emphasize the need  to ensure that programs and projects are implemented at acceptable costs  and within reasonable and expected time frames. Investments such as  Secure Flight are approved on the expectation that programs and projects  will meet certain commitments to produce certain capabilities and  benefits (mission value) within the defined schedule and cost. Until an  updated program management plan and related schedules and cost  estimates and expenditure plans, are prepared for Secure Flight—which  should be developed despite program uncertainties, and updated as more  information is gained—TSA and Congress will not be able to provide  complete oversight over the program’s progress in meeting established  commitments.
section_title: Oversight Reviews of Secure Flight Have Been Conducted and Raised Questions about Program Management
paragraphs: DHS and TSA have executive and advisory oversight mechanisms in place  to oversee Secure Flight. As we reported in March 2005, the DHS  Investment Review Board (IRB)—designed to review certain programs at  key phases of development to help ensure they meet mission needs at  expected levels of costs and risks—reviewed the TVP from which Secure  Flight will operate, in January 2005. As a result of this review, the board  withheld approval for the TVP to proceed from development and testing  into production and deployment until a formal acquisition plan, a plan for  integrating and coordinating Secure Flight with other DHS people- screening programs, and a revised acquisition program baseline (cost,  schedule, and performance parameters) had been completed. Since that  time, TSA has not yet addressed these conditions and has not obtained  approval from the IRB to proceed into production. DHS officials stated  that an IRB review is scheduled to be held in March 2006—14 months after  the IRB last met to examine Secure Flight—to review Secure Flight and  other people-screening programs, including international prescreening  conducted by CBP. Specifically, the board will review the acquisition  strategy and progress for each program, focusing, in part, on areas of  potential duplication. According to TSA officials, the agency intends to  establish a new program cost, schedule, and capability baseline for Secure  Flight, which will be provided to the IRB for review.
DHS’s Data Privacy and Integrity Advisory Committee also reviewed  Secure Flight during the last year. Committee members have diverse  expertise in privacy, security, and emerging technology, and come from  large and small companies, the academic community, and the nonprofit  sector. In December 2005, the committee issued five recommendations on  key aspects of the program, including recommendations designed to  minimize data collection and provide an effective redress mechanism to  passengers who believe they have been incorrectly identified for  additional security scrutiny. TSA officials stated that they are considering  the advisory committees’ findings and recommendations as part of their  rebaselining efforts.
In September 2004, TSA appointed an independent working group within  the Aviation Security Advisory Committee, composed of government  privacy and security experts, to review Secure Flight. The working group  issued a report in September 2005 that concluded, among other things,  that TSA had not produced a comprehensive policy document for Secure  Flight that could define oversight or governance responsibilities, nor had it  provided an accountability structure for the program. The group attributed  this omission to the lack of a program-level policy document issued by a  senior executive, which would clearly state program goals. The working  group also questioned Secure Flight’s oversight structure and stated that it  should focus on the effectiveness of privacy aspects of the program and, in  doing so, consider oversight regimes for federal law enforcement and U.S.  intelligence activities.
In addition to oversight reviews initiated by DHS and TSA, the DOJ-OIG  issued a report in August 2005 reviewing TSC’s role in supporting Secure  Flight.  In its report, the DOJ-OIG reported that TSC faced several key  factors that were unknown with respect to supporting Secure Flight,  including when the program will begin, the volume of inquiries it will  receive, the number of TSC resources required to respond to these  inquiries, and the quality of the data it will have to analyze. In light of these  findings, the DOJ-OIG report recommended that, among other things, TSC  better prepare itself for future needs related to Secure Flight by  strengthening its budgeting and staffing processes and by improving  coordination with TSA on data exchange standards. In June 2005, a DOJ- OIG report recommended that TSC conduct a record-by-record review of  the TSDB to improve overall data quality and integrity. TSC agreed with all  recommendations made.
section_title: TSA Has Made Progress in Coordinating with Critical Stakeholders but More Work Remains
paragraphs: TSA has drafted policy and technical guidance to help inform air carriers  of their Secure Flight responsibilities, and has begun coordinating with  CBP and TSC on Secure Flight requirements and broader issues of  integration and interoperability between Secure Flight and other people- screening programs. However, TSA has not yet provided information and  technical requirements that all stakeholders need to finalize their plans to  support the program’s operations, and to adequately plan for the resources  needed to do so.
subsections: section_title: TSA Has Begun Collaborating with Key Stakeholders, but Their Participation Will Be Limited Until System Requirements Have Been Finalized
paragraphs: As we reported in March 2005, key federal and commercial stakeholders— CBP, TSC, and commercial air carriers—will play a critical role in the  collection and transmission of data needed for Secure Flight to operate  successfully. Accordingly, TSA will need to ensure that requirements for  each stakeholder are determined. For instance, TSA will need to define  how air carriers are to connect to CBP and what passenger data formats  and structures will be used. Although more remains to be done, TSA has  worked to communicate and coordinate requirements with stakeholders.  For example, TSA has maintained weekly communications with CBP and  TSC regarding their roles and responsibilities related to Secure Flight  operations.
TSA has also begun to address air carriers’ questions about forthcoming  Secure Flight requirements. For example, TSA Officials have produced  draft air carrier guidance, known as the Secure Flight Data Transmission  Plan Guidance (DTPG). The final DTPG is to include guidance to air  carriers addressing the following areas: Secure Flight’s mission overview  and objectives, project planning phases, aircraft operator operations and  airport procedures, technical data requirements, aircraft operator  application development, Secure Flight operations, and system  maintenance and support. According to TSA officials, air carriers have  received copies of a partial draft DTPG, and some air carriers have  submitted feedback to Secure Flight’s Airline Implementation and  Operations Team that TSA says it is working to address.
In addition to drafting guidance, TSA has conducted preliminary network  connectivity testing between TSA and federal stakeholders. For example,  messages have been transmitted from CBP to TSA and back. However,  such tests included only dummy data. According to CBP officials, no real- time passenger data have been used in this testing, and system stress  testing has not yet been conducted. Without real-time passenger data, the  official said, CBP cannot estimate total capacity or conduct stress testing  to ensure the system operates effectively. Further, according to a TSC  official, testing has been conducted to show that a data exchange between  the TSC and TSA is functioning, but the system has not been stress-tested  to determine if it can handle the volume of data traffic that will be required  to operate Secure Flight. According to this official, TSA has not specified  what these data volume requirements will be. TSA officials acknowledged  that they have not yet made this determination and stated that they will  not be able to do so until they (1) issue the rule, and (2) have received the  air carrier plans for participating in Secure Flight based on requirements  identified in the rule.
Although CBP, TSC, and air carrier officials we interviewed acknowledged  TSA’s outreach efforts, they cited several areas where additional  information was needed from TSA before they could fully support Secure  Flight. Several CBP officials stated, for example, that they cannot proceed  with establishing connectivity with all air carriers until DHS publishes the  rule—the regulation that will specify what type of information is to be  provided for Secure Flight—and the air carriers provide their plans for  providing this information. Similarly, a TSC official stated that TSC cannot  make key decisions on how to support Secure Flight until TSA provides  estimates of the volume of potential name matches that TSC will be  required to screen, as identified above. The TSC official stated that  without this information, TSC cannot make decisions about required  resources, such as personnel needed to operate its call center. As we  reported in March 2005, air carriers also expressed concerns regarding the  uncertainty of the Secure Flight system and data requirements, and the  impact these requirements may have on the airline industry and traveling  public. Air carriers will not be able to begin to modify their passenger data  systems to record the data attributes—such as full name and date of birth,  which Secure Flight will use to conduct name matching—until TSA  determines and communicates which specific data attributes are to be  used.
Oversight groups that have reviewed Secure Flight agreed that additional  work was needed to improve the flow of information to, and coordination  with, program stakeholders. In its December 2005 report on Secure Flight,  the DHS Data Privacy and Integrity Advisory Committee stated that TSA  needs to be clear with air carriers about what information it needs now  and what information it may consider requesting in the future, to enable  air carriers to avoid sequential revisions of data-handling systems. Also, in  September 2005, the Aviation Security Advisory Committee working group  expressed concerns about the lack of clarity regarding how Secure Flight  will interact with other screening programs.
Further, in its August 2005 audit of TSC’s support of Secure Flight, the  DOJ-OIG reported that TSC officials believed that their ability to prepare  for the implementation of Secure Flight has been hampered by TSA’s  failure to make, communicate, and comply with key program and policy  decisions in a timely manner, such as the launch date and volume of  screening to be conducted during initial implementation. In addition, the  report noted that because TSA is unsure about how many air carriers will  participate in the initial phase of the program, neither TSA nor TSC can  know how many passenger records will be screened, and cannot project  the number of watch list hits that will be forwarded to the TSC for action.  Finally, the DOJ-OIG report concluded that the shifting of critical  milestones—including TSA’s schedule slippages over the past year—has  affected TSC’s ability to adequately plan for its role in Secure Flight.
Despite TSA’s outreach efforts, stakeholder participation in Secure Flight  is dependent on TSA’s effort to complete its definition of requirements and  describe these in the rule. Because TSA has not fully defined system  requirements, key stakeholders have not been able to fully plan for or  make needed adjustments to their systems. In our March 2005 report, we  recommended that TSA develop a plan for establishing connectivity  among the air carriers, CBP, and TSC to help ensure the secure, effective,  and timely transmission of data for use in Secure Flight operations.  Although TSA has continued to coordinate with these key stakeholders, at  present the agency has still not completed the plans and agreements  necessary to ensure the effective support of Secure Flight.
section_title: Ongoing Coordination of Prescreening and Name-Matching Initiatives Can Impact How Secure Flight Is Implemented
paragraphs: In January 2006, TSA officials stated that they are in the early stages of  coordinating with CBP on broader issues of integration and  interoperability related to other people-screening programs. These  broader coordination efforts, which are focused on minimizing duplicative  efforts that may exist between the agencies that screen individuals using  watch list data and achieving synergies and efficiencies, are important  because they may affect how Secure Flight will operate initially and in the  future. Specifically, TSA Officials stated that they are coordinating more  closely with CBP’s international prescreening initiatives for passengers on  flights bound for the United States. The Air Transport Association and the  Association of European Airlines—organizations representing air  carriers—had requested, among other things, that both domestic and  international prescreening function through coordinated information  connections and avoid unnecessary duplication of communications,  programming, and information requirements.
In response to air carrier concerns, and the initiatives of DHS to minimize  duplicative efforts, officials from both CBP and TSA explained that they  are beginning to work together to ensure that air carriers have a single  interface with the government for prescreening both domestic and  international passengers. TSA and CBP officials further stated that they  will try to use CBP’s network to transmit domestic and international  passenger data to and from the air carriers, thus providing the air carriers  with a single interface for sending and receiving information. TSA and  CBP officials also stated that air carriers should receive a common  notification about whether a passenger—domestic or international— requires normal processing, additional screening, or is not permitted to  board a plane. However, according to these officials, TSA and CBP have  not yet resolved other system differences—such as the fact that their  prescreening systems use different passenger data elements,  documentation, and name matching technologies—that could lead to  conflicting notifications that would instruct air carriers to handle a  passenger differently for an international than for a domestic flight. Both  TSA and CBP officials agreed that additional coordination efforts are  needed to resolve these differences, and stated that they plan to work  closely together in developing a prescreening capability for both domestic  and international passengers. Decisions made as a result of further  coordination could result in changes to the way that Secure Flight is  implemented.
In addition to coordinating with CBP on international prescreening, TSA  faces additional coordination challenges working with TSC. Specifically,  according to TSC officials, TSC has an initiative under way to, among  other things, better safeguard watch list data. Currently, TSC exports  watch list data to other federal agencies, such as TSA and the State  Department, for use in these agencies’ screening efforts or processes for  examining documents and records related to terrorism. However, TSC is  currently developing a new system whereby watch list data would not be  exported, but rather would be maintained by TSC. This system, called  Query, is to serve as a common shared service that will allow agencies to  directly search the TSDB using TSC’s name matching technology for their  own purposes. TSC has conducted limited testing of the system. If TSC  chooses to use Query, TSA will be required to modify the system  architecture for Secure Flight in order to accommodate the new system.  According to a TSC official, this effort could be costly. While TSA  acknowledged in its draft concept of operations plan in June 2005 that  Secure Flight would need to be modified to accommodate TSC’s Query “as  necessary,” the agency has not made adjustments to its system  requirements or conducted a cost analysis of expected impacts on the  Secure Flight program. Rather, TSA has decided that it will continue  developing the Secure Flight application, which includes TSA’s name- matching technologies. Thus, TSC will need to export watch list data to  TSA to support Secure Flight, once it becomes operational.
section_title: Key Factors That Will Influence the Effectiveness of Secure Flight Have Not Been Finalized or Resolved
paragraphs: Several activities are under way, or are to be decided, that will affect  Secure Flight’s effectiveness, including how operational testing is  conducted, and how data requirements and data accuracy are determined.  TSA has been testing and evaluating name-matching technologies for  determining what type of passenger data will be needed to match against  the TSDB. These tests have been conducted thus far in a controlled, rather  than real-world environment, using historical data, and additional testing  is needed. In addition, TSA has not made key decisions regarding how the  name-matching technologies to be used by Secure Flight will operate or  which data will be used to conduct name matching. While TSA is not  responsible for ensuring the accuracy of passenger data, the agency must  nonetheless advise stakeholders on data accuracy and quality  requirements. Another factor that could impact the effectiveness of Secure  Flight in identifying known or suspected terrorists is the system’s inability  to identify passengers who assume the identity of another individual by  committing identity theft, or passengers who use false identifying  information. Secure Flight is neither intended to nor designed to address  these vulnerabilities.
subsections: section_title: Tests of Name-Matching Capability Are Under Way, but Full System Testing Has Not Yet Been Conducted
paragraphs: TSA has tested—and continues to test—the effectiveness of one aspect of  the Secure Flight system, namely name-matching technologies. These  name-matching tests will help TSA determine what passenger data will be  needed for the system to match most effectively passenger records with  information contained in the TSDB. These tests are critical to defining data  requirements and making decisions about how to configure the name- matching technologies. Additional tests will need to be conducted in an  operational, real-world environment to fully understand how to configure  the system effectively. This is because the name-matching tests conducted  to date were conducted in a controlled, rather than real-world,  environment—that is, under controlled, or simulated, conditions. For  example, TSA used historic air carrier passenger data from June 2004 and  historic and simulated watch list data to test the functionality and  effectiveness of Secure Flight’s name-matching technologies that match air  carrier passenger records with potential terrorists in the TSDB.
Additional testing beyond name-matching also needs to be conducted,  after TSA rebaselines its program, defines system requirements, and  begins adhering to its SDLC. For example, stress and operational testing  would help determine whether Secure Flight can process the volume of  data expected and operate as intended in an operational environment. As  we reported in March 2005, TSA had planned to conduct a series of  operational tests consisting of increasingly larger increments of the  system’s functionality until the complete system was tested. These tests  were to begin in June 2005. However, due to program delays, TSA has not  yet conducted this end-to-end testing needed to verify that the entire  system, including any interfaces with external systems, functions as  intended in an operational environment. TSA also has not yet conducted  the stress testing needed to measure the system’s performance and  availability in times of particularly heavy (i.e., peak) loads. Recently, TSA  documented its overall strategy for conducting these tests and developed  draft test plans. TSA officials stated that information about its plans for  future testing will be included in its rebaselined program plan. Until this  testing is complete, it will not be possible to determine whether Secure  Flight will function as intended in an operational environment.
section_title: Key Policy Decisions That Will Impact System Effectiveness Have Not Been Made
paragraphs: Key policy decisions that will influence the effectiveness of Secure Flight  in identifying passengers who should undergo additional security scrutiny  have not yet been made. These policy decisions include (1) determining  the passenger information that air carriers will be required to collect and  provide for vetting, (2) the name-matching technologies that will be used  to vet passenger data against data contained in the TSDB, and (3) the  thresholds that will be set to determine when a passenger will be  identified as a potential match against the TSDB. These three decisions,  discussed below, are all critical to ensuring that Secure Flight identifies  potential terrorist threats as effectively as possible while minimizing the  number of potential matches that will require further review by TSA and  TSC analysts.  (1) Determining the passenger information that air carriers will be  required to collect and provide for vetting: TSA needs to decide which  data attributes air carriers will be required to provide in passenger data to  be used to match against data contained in the TSDB, such as full first,  middle, and last name plus other discrete identifiers, such as date of birth.
Using too many data attributes can increase the difficulty of matching,  since the risk of errors or mismatches increases. Using too few attributes  can create an unnecessarily high number of incorrect matches due to,  among other things, the difficulty of differentiating among similar common  names without using further information. Initial TSA test results have  shown that the use of name and date of birth alone might not be sufficient  for decreasing the number of false positives—that is, passengers  inappropriately matched against data contained in the TSDB.  (2) Selecting name-matching technologies used to vet passenger names  against the TSDB: TSA must determine what type or combination of  name-matching technologies to acquire and implement for Secure Flight,  as these different technologies have different capabilities. For example,  TSA’s PNR testing showed that some name-matching technologies are  more capable than others at detecting significant name modifications,  which allows for the matching of two names that contain some variation.  Detecting variation is important because passengers may intentionally  make alterations to their names in an attempt to conceal their identity.  Also, unintentional variations can result from different translations of  nonnative names or data entry errors. For example, some name-matching  technologies might correctly discriminate between “John Smith” and  “John Smythe,” others may not. However, name matching technologies  that are best at detecting name variations may also increase the number of  potential matches that will have to be further reviewed, which could be  offset using a combination of name matching technologies. TSA officials  stated in November 2005 that it planned to continuously evaluate the best  name-matching technologies or combination of technologies to enhance  the system in future iterations. TSA officials recently stated that they had  made, but not yet documented, an initial determination regarding the  name-matching technologies that will be used for Secure Flight and that  they plan to conduct continuous reviews of the name-matching  technologies to address circumstances as they arise.  (3) Selecting thresholds for determining when a possible name match  has occurred: TSA has discretion to determine what constitutes a possible  match between a passenger’s data and a TSDB record. For each name  that is matched, the name-matching tool will assign a numeric score that  indicates the strength of the potential match. For example, a score of   95 out of 100 would indicate a more likely match than a score of 85. If TSA  were to set the threshold too high, many names may be cleared and  relatively few flagged as possible matches—that is, there is a possibility  that terrorists’ names may not be matched. Conversely, if the threshold  were set too low, passengers may be flagged unnecessarily, and relatively  few cleared through the automated process. As an example of the  importance of setting thresholds, during one of the PNR tests conducted,  TSA set the name-matching threshold at 80, which resulted in over   60 percent of passengers requiring manual review. Alternatively, when  TSA set the threshold at 95, less than 5 percent of the same group of  passenger records were identified as requiring further review. With about  1.8 million passengers traveling domestically per day, having a threshold  that is too low could produce an unmanageable number of matches— possibly leading to passenger delays—while setting the threshold too high  could result in the system missing potential terrorists. Although TSA will  not decide how the thresholds should be set until it conducts additional  evaluations, it has indicated that the threshold might be adjusted to reflect  changes in the terrorist threat level. This would result in Secure Flight  flagging more names for potential manual review in order to ensure  greater scrutiny in response to changing conditions.
TSA plans to finalize decisions on these factors as system development  progresses. However, until these decisions are made, requirements will  remain unsettled and key stakeholders—in particular air carriers—will not  have the information they need to assess and plan for changes to their  systems necessary for interfacing with Secure Flight. Air carriers and  reservation companies will also not know which additional data attributes  they may be required to collect from passengers, to support Secure Flight  operations, as reservations are made. These decisions will also directly  influence the number of analysts that TSA and TSC will need to manually  review potential matches to the TSDB. Accordingly, stakeholders have  expressed concern that they have not been provided information about  what these decisions are. They stated that they are awaiting additional  information from TSA in order to move forward with their plans to  interface with and support Secure Flight.
section_title: Efforts to Improve Data Quality and Accuracy Are Under Way, but Additional Work Remains
paragraphs: Two additional factors that will impact the effectiveness of Secure Flight  are (1) the accuracy and completeness of data contained in TSC’s TSDB  and in passenger data submitted by air carriers, and (2) the ability of TSA  and TSC to identify false positives and resolve possible mistakes during  the data matching process, in order to minimize inconveniencing  passengers. According to TSA and TSC officials, the data attributes that  Secure Flight will require for name matching need to be included in both  the passenger data and the TSDB in order for the automated system to  effectively match names between the two lists. As we reported in March  2005, while the completeness and accuracy of data contained in the TSDB  can never be certain—given the varying quality of intelligence information  gathered, and changes in this information over time—TSC has established  some processes to help ensure the quality of these data. However, the  DOJ-OIG, in its June 2005 review of TSC, found that that the TSC could  not ensure that the information contained in its databases was complete or  accurate. According to a TSC official, since the time of the DOJ-OIG  review, TSC has taken several steps to improve the quality of TSDB  records, including conducting a record-by-record review, updating  procedures for a daily review of each new or modified record, and using  automated rules to check the completeness of records received from other  agencies. According to this official, TSA and TSC plan to enter into a  letter of agreement that will describe the TSDB data elements that TSC  will produce for TSA, among other things, to be used for Secure Flight.   However, these data requirements have not yet been determined.
In order to obtain accurate and complete passenger data from air carriers,  TSA plans to describe the required data attributes that must be contained  in passenger data provided to TSA in the forthcoming rule. TSA also plans  to issue a final and complete DTPG to specify the data formats and other  transmission requirements. However, the accuracy and completeness of  the information contained in the passenger data record will still be  dependent on the air carriers’ reservations systems and passengers, and  the air carriers’ modifications of their systems for transmitting the data in  the proper format. These steps are not trivial, as indicated by the June  2004 historical passenger data provided by the air carriers for TSA’s name- matching tests. For these tests, many passenger data records submitted by  air carriers were found to be inaccurate or incomplete, creating problems  during the automated name-matching process. For example, some  passenger data included invalid characters or prefixes, such as “Mr.” and  “Mrs.,” in the name fields. Other inaccuracies included invalid characters  or prefixes, spelling errors, and inverted birth date information.  Additionally, some of the records had omitted or incomplete data elements  necessary for performing the automated match or were in an unusable  format.
In a related effort to address accuracy, TSA and TSC plan to work together  to identify false positives as passenger data are matched against data in  the TSDB and to resolve mistakes to the extent possible before  inconveniencing passengers. The agencies will use intelligence analysts  during the actual matching of passenger data to data contained in the  TSDB to increase the accuracy of data matches. As indicated in figure 1,  when TSA’s name-matching technologies indicate a possible match, TSA  analysts are to manually review all of the passenger data and other  information to determine if the passenger can be ruled out as a match to  the TSDB. If a TSA analyst cannot rule out a possible match, the record  will be forwarded to a TSC analyst to conduct a further review using  additional information. According to a TSC official, TSA and TSC analysts  participated in a tabletop exercises to test the consistency of their  respective manual reviews, and found that the matching logic used by both  groups of analysts was consistent. This official stated that TSA and TSC  also tested their operational procedures, and found gaps in their  procedures that are now being addressed. According to this official, TSA  and TSC plan to conduct additional joint exercises. Completing these  exercises will be important to further understanding the effectiveness of  using intelligence analysts to clear misidentified passengers during Secure  Flight operations.
section_title: False Identifying Information and Identity Theft Could Impact the Security Benefits of Secure Flight
paragraphs: Another factor that could affect Secure Flight’s effectiveness in identifying  known or suspected terrorists is the system’s inability to identify  passengers who falsify their identifying information or who commit  identity theft. TSA Officials stated that the program is not intended to or  designed to protect against the use of falsified identities or to detect  identity theft. However, TSA officials stated that the use of commercial  data during the name-matching process may help identify situations in  which a passenger submits fictitious information such as a false address.  In the spring of 2005, a TSA contractor tested the use of commercial data  composed of personally identifiable information (such as name and  address) to determine, among other things, if such data could be used to  increase Secure Flight’s effectiveness in identifying false or stolen  identities. However, according to the DHS Data Privacy and Integrity  Advisory Committee report, testing performed to date does not provide a  reasonable case for utilizing commercial data as part of Secure Flight. TSA  officials are not currently pursuing the use of commercial data to support  Secure Flight because the fiscal year 2006 DHS appropriations act  prohibits TSA from using data or databases obtained from or that remain  under the control of a non-federal entity, effectively terminating this type  of testing for the duration of fiscal year 2006. Further, TSA officials stated  that incorporating biometrics—technologies that can automate the  identification of people by one or more of their distinct physical or  behavioral characteristics—is not currently envisioned for Secure Flight.  As noted in our previous work, biometric technologies, such as fingerprint  recognition, are being used in other TSA screening programs. Moreover,  the current prescreening process of matching passenger names against no- fly and selectee lists implemented by air carriers also does not protect  against identity theft or the use of fictitious identities.
section_title: Secure Flight Privacy Notices and Passenger Redress Process Cannot Be Finalized Until Program Requirements Are More Fully Defined
paragraphs: TSA is aware of, and plans to address, the potential for Secure Flight to  adversely affect travelers’ privacy and impact their rights. However, TSA,  as part of its requirements development process, has not yet clearly  identified the privacy impacts of the planned system or the full actions it  plans to take to mitigate them. Nor has the agency completed its  assessment of the potential impact on passenger privacy of the system in  an operational environment or defined its redress process for Secure  Flight because, in part, the operational plans and system requirements for  Secure Flight have not been finalized. TSA officials stated that they are in  the process of reviewing new privacy notices that will be issued in  conjunction with a forthcoming rule making prior to proceeding with its  initial operating capability, and that these notices will also address certain  aspects of Secure Flight’s redress process. Until TSA finalizes system  requirements and notices, however, privacy protections and impacts  cannot be assessed.
subsections: section_title: Privacy Cannot Be Fully Assessed Because System Development Documentation Does Not Fully Address Privacy Requirements
paragraphs: The Privacy Act and the Fair Information Practices—a set of  internationally recognized privacy principles that underlie the Privacy  Act—limit the collection, use, and disclosure of personal information by  federal agencies. While TSA has reiterated its commitment to meet the  requirements of the Privacy Act and the Fair Information Practices, it is  not yet evident how this will be accomplished. To begin with, TSA has  not decided what data attributes from the PNR it plans to collect, or how  such data will be provided by airlines, through CBP, to TSA. Further,  according to TSA officials, the agency is in the process of developing but  has not issued the system of records notice, which is required by the  Privacy Act, or the privacy impact assessment, which is required by the  E-Government Act, that would describe how TSA considered privacy in  the development of the system and how it will protect passenger data once  the system becomes operational.
Moreover, privacy requirements were not incorporated into the Secure  Flight system development process in such a way that would explain  whether personal information will be collected and maintained in the  system in a manner that complies with statutory requirements and TSA’s  SDLC guidance. One requirement of the privacy impact assessment is that  privacy be addressed in the systems development documentation. In  addition, TSA’s SDLC guidance acknowledges that privacy protections  should be planned for and carried out as part of the system development  process. In our review of Secure Flight’s system requirements, we found  that privacy concerns were broadly addressed in Secure Flight’s functional  requirements, but had not been translated into specific system  requirements. For example, the functional requirements stated that the  Privacy Act must be considered in the development of the system, but the  system requirements documents do not reflect how privacy protections  will be supported by the system. Rather, system requirements documents  state that privacy requirements are “yet to be finalized.” TSA’s Privacy  Officer stated that she has been collaborating with the system  development team, but this is not evident in the documents we reviewed.
Without taking steps to ensure that privacy protections are built into the  system requirements, TSA cannot be assured that it will be in compliance  with the Privacy Act once operational, and it runs the risk of repeating  problems it experienced last spring. We reported in July 2005 that TSA’s  initially issued privacy notices for the Secure Flight data-processing tests  did not meet Privacy Act requirements because personal information was  used in testing in ways that the agency had not disclosed to the public.  We explained that in its fall 2004 notices, TSA had informed the public of  its plans to use personal information during Secure Flight testing,  including the use of commercial data in a limited manner. However, these  initial notices did not fully describe how personal information would be  collected, used, and stored for commercial data testing as it was carried  out. As a result, individuals were not fully informed that their personal  information was being collected and used, nor did they have the  opportunity to comment on this or become informed on how they might  exercise their rights of access to their information. Although TSA did not  fully disclose its use of personal information prior to beginning Secure  Flight commercial data testing, the agency issued revised privacy notices  in June 2005 to more fully disclose the nature of the commercial tests and  address the issues disclosed by us.
As we reported in March 2005, until TSA fully defines its operational plans  for Secure Flight and addresses international privacy concerns, it will  remain difficult to determine whether the planned system will offer  reasonable privacy protections to passengers who are subject to  prescreening or mitigate potential impacts on passengers’ privacy. At that  time, we recommended that TSA finalize privacy policies and issue  associated documentation prior to Secure Flight achieving initial operating  capability. TSA acknowledged that it needs to publish new privacy notices  to cover the collection, use, and storage of personal data for Secure  Flight’s initial and full operating capability, before beginning operational  testing. TSA officials stated that these privacy notices are currently being  reviewed by TSA and DHS and will be released in conjunction with the  forthcoming rulemaking.
section_title: TSA Has Not Determined Secure Flight’s Redress Process
paragraphs: Congress mandates that Secure Flight include a process whereby aviation  passengers determined to pose a threat to aviation security may appeal  that determination and correct erroneous information contained within  the prescreening system. TSA currently has a process in place that allows  passengers who experience delays, under the current process run by air  carriers, to submit a passenger identity verification form to TSA and  request that the agency place their names on a cleared list. If, upon review,  TSA determines that the passenger’s identity is distinct from the person on  a watch list, TSA will add the passenger’s name to its cleared list, and will  forward the updated list to the air carriers. TSA will also notify the  passenger of his or her cleared status and explain that in the future the  passenger may still experience delays. Recently, TSA has automated the  cleared list process, enabling the agency to further mitigate inconvenience  to travelers on the cleared list.
The Intelligence Reform and Terrorism Prevention Act, enacted in  December 2004, directs TSA to include certain elements in its Secure  Flight redress policy. Specifically, it requires the establishment of a  timely and fair process for individuals identified as a threat to appeal the  determination to TSA and correct any erroneous information. It further  requires that TSA establish a method for maintaining a record of air  passengers who have been misidentified and have corrected erroneous  information. To prevent repeated delays of misidentified passengers, this  record must contain information determined by TSA to authenticate the  identity of such a passenger. In January 2006, TSA officials stated that no  final decisions have been made regarding how TSA will address the  relevant requirements for redress found in the Intelligence Reform and  Terrorism Prevention Act requirements. However, OTSR officials stated  that a cleared list will be part of the process. The June 2005 concept of  operations describes a process where individuals that are frequently  misidentified as being on the TSDB and TSA selectee list can request to be  placed on a list of individuals who have been cleared.
In our March 2005 report, we recommended that TSA finalize its Secure  Flight redress policies and procedures prior to achieving its initial  operating capability. Information concerning aspects of the redress  process will be published before operational tests or full implementation  of the Secure Flight process, and will be contained within the privacy  notices that TSA officials stated will be released in conjunction with the  forthcoming rulemaking. Moving forward, TSA has assigned a manager to  serve as liaison with DHS on privacy and redress issues.
section_title: Concluding Observations
paragraphs: TSA has continued its development and testing of Secure Flight, but has  made limited progress in addressing longstanding issues related to system  development and testing, program management, and privacy and redress  protections. To make and demonstrate progress on any large-scale  information technology program, such as Secure Flight, an agency must  first adequately define what program capabilities, such as requirements  related to performance, security, privacy, and data content and accuracy,  are to be provided. These requirements can then in turn be used to  produce reliable estimates of what these capabilities will cost, when they  will be delivered, and what mission value or benefits will accrue as a  result. For Secure Flight, well-defined requirements would provide a guide  for developing the system and a baseline to test the developed system to  ensure that it delivers necessary capabilities, and would help to ensure  that key program areas—such as security, system connectivity, and  privacy and redress protections—are appropriately managed.
When we reported on Secure Flight in March 2005, TSA had committed to  take action on our recommendations to manage the risks associated with  developing and implementing Secure Flight, including finalizing the  concept of operations, system requirements and test plans; completing  formal agreements with CBP and air carriers to obtain passenger data;  developing life cycle cost estimates and a comprehensive set of critical  performance measures; issuing new privacy notices; and putting a redress  process in place. Over the past 11 months, TSA has made some progress  on all of these areas, including conducting further testing of factors that  could influence system effectiveness and corroborating with key  stakeholders. However, TSA has not completed any of the actions it had  scheduled to accomplish. In particular, TSA has not yet developed  complete system requirements or conducted important system testing  (including stress testing), fully established security measures, made key  decisions that will determine system effectiveness, developed a program  management plan and a schedule for accomplishing program goals, or  published updated privacy and redress notices. Taken as a whole, this lack  of progress indicates that the program has not been effectively managed  and is at risk of failure.
While we recognize that TSA faces program uncertainties that can directly  impact Secure Flight’s development and progress, uncertainty is a  component of most programs, and should not be used as a reason for not  defining requirements and developing plans and cost estimates, to manage  risk. We believe that Secure Flight, like all programs, can utilize best  practices to develop such plans to manage program uncertainties.
To its credit, TSA has recently taken actions that recognize the need to  instill more rigor and discipline into the development and management of  Secure Flight, including hiring a program manager with information  systems program management credentials. We also support TSA’s efforts  to rebaseline the program, including defining system requirements and  finalizing a program management plan, including the development of  schedules and cost estimates, before proceeding with program  development. In fact, proceeding with operational testing and completing  other key program activities should not be pursued until TSA puts in place  a more disciplined life cycle process and defines system requirements. In  the absence of this and other program information, such as requirements,  capabilities, and benefits, further investment in this program would be  difficult to justify.
We are also encouraged that DHS’s IRB—the executive decision making  authorities—has scheduled a review of Secure Flight and other   people-screening programs. Given the potential duplication with CBP’s  new initiatives for international prescreening, DHS, TSA, and CBP need to  assess alternative system solutions that should be factored into Secure  Flight’s rebaselined program and be the basis for IRB decisions regarding  Secure Flight’s future. Notwithstanding these efforts, however, much work  remains to be accomplished before Secure Flight is positioned to be  properly executed so that informed and prudent investment decisions can  be made.
Mr. Chairman, this concludes my prepared statement. I will be pleased to  respond to any questions that you or other members of the committee  have at the appropriate time.
section_title: GAO Contacts and Staff Acknowledgments
paragraphs: For further information about this testimony, please contact   Cathleen Berrick, at 202-512-3404 or at berrickc@gao.gov, or   Randolph C. Hite at 202-512-6256 or at hiter@gao.gov.
Other key contributors to this statement were David Alexander,   Amy Bernstein, Mona Nichols Blake, John de Ferrari, Christine Fossett,  Brent Helt, Richard Hung, Thomas Lombardi, C. James Madar,   Matthew Mohning, David Plocher, Karl Seifert, and William Wadsworth.
section_title: Appendix I: Legislatively Mandated Secure Flight Issues to be Certified by DHS and Reviewed by GAO
paragraphs: A system of due process exists whereby aviation passengers determined to pose a threat  are either delayed or prohibited from boarding their scheduled flights by TSA may appeal  such decisions and correct erroneous information contained in CAPPS II or Secure Flight  or other follow-on/successor programs.
The underlying error rate of the government and private databases that will be used to  both establish identity and assign a risk level to a passenger will not produce a large  number of false positives that will result in a significant number of passengers being  treated mistakenly or security resources being diverted.
TSA has stress-tested and demonstrated the efficacy and accuracy of all search  technologies in CAPPS II or Secure Flight or other follow-on/successor programs and  has demonstrated that CAPPS II or Secure Flight or other follow-on/successor programs  can make an accurate predictive assessment of those passengers who may constitute a  threat to aviation.
The Secretary of Homeland Security has established an internal oversight board to  monitor the manner in which CAPPS II or Secure Flight or other follow-on/successor  programs are being developed and prepared.
TSA has built in sufficient operational safeguards to reduce the opportunities for abuse.
Substantial security measures are in place to protect CAPPS II or Secure Flight or other  follow-on/successor programs from unauthorized access by hackers or other intruders.  7. Oversight of system use and operation  TSA has adopted policies establishing effective oversight of the use and operation of the  system.
There are no specific privacy concerns with the technological architecture of the system.
TSA has, in accordance with the requirements of section 44903 (j)(2)(B) of title 49, United  States Code, modified CAPPS II or Secure Flight or other follow-on/successor programs  with respect to intrastate transportation to accommodate states with unique air  transportation needs and passengers who might otherwise regularly trigger primary  selectee status.
Appropriate life-cycle cost estimates, and expenditure and program plans exist.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.