Aviation Security: TSA Has Completed Key Activities Associated with Implementing Secure Flight, but Additional Actions Are Needed to Mitigate Risks
Published: 2009-05-13T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To enhance aviation security, the Department of Homeland Security's (DHS) Transportation Security Administration (TSA) developed a program--known as Secure Flight--to assume from air carriers the function of matching passenger information against terrorist watch-list records. In accordance with a mandate in the Department of Homeland Security Appropriations Act, 2008, GAO's objective was to assess the extent to which TSA met the requirements of 10 statutory conditions related to the development of the Secure Flight program. GAO is required to review the program until all 10 conditions are met. In September 2008, DHS certified that it had satisfied all 10 conditions. To address this objective, GAO (1) identified key activities related to each of the 10 conditions; (2) identified federal guidance and best practices that are relevant to successfully meeting each condition; (3) analyzed whether TSA had demonstrated, through program documentation and oral explanation, that the guidance was followed and best practices were met; and (4) assessed the risks associated with not fully following applicable guidance and meeting best practices.
section_title: What GAO Found
paragraphs: As of April 2009, TSA had generally achieved 9 of the 10 statutory conditions related to the development of the Secure Flight program and had conditionally achieved 1 condition (TSA had defined plans, but had not completed all activities for this condition). Also, TSA's actions completed and those planned have reduced the risks associated with implementing the program. Although DHS asserted that TSA had satisfied all 10 conditions in September 2008, GAO completed its initial assessment in January 2009 and found that TSA had not demonstrated Secure Flight's operational readiness and that the agency had generally not achieved 5 of the 10 statutory conditions. Consistent with the statutory mandate, GAO continued to review the program and, in March 2009, provided a draft of this report to DHS for comment. In the draft report, GAO noted that TSA had made significant progress and had generally achieved 6 statutory conditions, conditionally achieved 3 conditions, and had generally not achieved 1 condition. After receiving the draft report, TSA took additional actions and provided GAO with documentation to demonstrate progress related to 4 conditions. Thus, GAO revised its assessment in this report. Related to the condition that addresses the efficacy and accuracy of search tools, TSA had not yet developed plans to periodically assess the performance of the Secure Flight system's name-matching capabilities, which would help ensure that the system is working as intended. GAO will continue to review the Secure Flight program until all 10 conditions are generally achieved.

section_title: Letter
paragraphs: The matching of airline passenger information against terrorist watch-list  records (watch-list matching) is a frontline defense against acts of  terrorism that target the nation’s civil aviation system. In general,  passengers identified by air carriers as a match to the No-Fly list are  prohibited from boarding a commercial flight, while those matched to the  Selectee list are required to undergo additional screening. Historically,  airline passenger prescreening has been performed by commercial air  carriers.
As required by the Intelligence Reform and Terrorism Prevention Act of  2004, the Transportation Security Administration (TSA) developed an  advanced passenger prescreening program known as Secure Flight that  will allow TSA to assume from air carriers the function of watch-list  matching. Since fiscal year 2004, GAO has been mandated to assess the  development and implementation of the Secure Flight program. Most  recently, in February 2008, we reported that TSA had instilled more  discipline and rigor into Secure Flight’s development, but that the program  continued to face challenges related to completing performance testing,  fully defining and testing security requirements, and establishing reliable  cost and schedule estimates. We made recommendations to address these  challenges and TSA generally agreed with them.
Section 522(a) of the Department of Homeland Security (DHS)  Appropriations Act, 2005, set forth 10 conditions related to the  development and implementation of the Secure Flight program that the  Secretary of Homeland Security must certify have been successfully met  before the program may be implemented or deployed on other than a test  basis (see table 1). On September 24, 2008, DHS certified that it had  satisfied all 10 conditions.
In accordance with section 513 of the Department of Homeland Security  Appropriations Act, 2008, our objective was to assess the extent to which  TSA met 10 statutory conditions and the associated risks of any shortfalls  in meeting the requirements. Our overall methodology included (1)  identifying key activities related to each condition; (2) identifying federal  guidance and related best practices, if applicable, that are relevant to  successfully meeting each condition (e.g., GAO’s Standards for Internal  Control in the Federal Government); (3) analyzing whether TSA has  demonstrated through verifiable analysis and documentation, as well as  oral explanation, that the guidance has been followed and best practices  have been met; and (4) assessing the risks associated with not fully  following applicable guidance and meeting best practices. Based on our  assessment, we categorized each condition as generally achieved,  conditionally achieved, or generally not achieved.
Generally achieved—TSA has demonstrated that it completed all key  activities related to the condition in accordance with applicable federal  guidelines and related best practices, which should reduce the risk of the  program experiencing cost, schedule, or performance shortfalls.
Conditionally achieved—TSA has demonstrated that it completed some  key activities related to the condition in accordance with applicable  federal guidelines and related best practices and has defined plans for  completing remaining key activities that, if effectively implemented as  planned, should result in a reduced risk that the program will experience  cost, schedule, or performance shortfalls.
Generally not achieved—TSA has not demonstrated that it completed all  key activities related to the condition in accordance with applicable  federal guidelines and related best practices and does not have defined  plans for completing the remaining activities, and the uncompleted  activities result in an increased risk of the program experiencing cost,  schedule, or performance shortfalls.
On January 7, 2009, we briefed staff of the Senate and House  Appropriations Committees’ Subcommittees on Homeland Security on the  results of our initial work, and reported that TSA had not demonstrated  Secure Flight’s operational readiness and that the agency had generally  not achieved 5 of the 10 statutory conditions. Our briefing also included  several recommendations for DHS to mitigate risks of Secure Flight cost,  schedule, or performance shortfalls and strengthen management of the  program. In addition, under this mandate, GAO is required to continue to  review the Secure Flight program until it determines that all 10 conditions  have been successfully met. In accordance with this requirement, we  conducted additional work from January through April 2009, which  included assessing information DHS provided after we submitted a copy of  our draft report to the department for formal agency comment. Based on  this additional work, we revised the status of several conditions and now  consider three of the recommendations we made in our draft report to be  met. This report contains information on our initial January 2009  assessment and recommendations, and related updates through April 2009.
We conducted this performance audit from May 2008 to May 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Appendix I presents more details about our  scope and methodology.
subsections: section_title: Background
subsections: section_title: Overview of Secure Flight
paragraphs: The prescreening of airline passengers who may pose a security risk  before they board an aircraft is one of many layers of security intended to  strengthen commercial aviation. In July 2004, the National Commission on  Terrorist Attacks Upon the United States, also known as the 9/11  Commission, reported that the current system of matching passenger  information to the No-Fly and Selectee lists needed improvements. The  commission recommended, among other things, that watch-list matching  be performed by the federal government rather than by air carriers.  Consistent with this recommendation and as required by law, TSA has  undertaken to develop a program—Secure Flight—to assume from air  carriers the function of watch-list matching. Secure Flight is intended to    eliminate inconsistencies in current passenger watch-list matching  procedures conducted by air carriers and use a larger set of watch-list  records when warranted,  reduce the number of individuals who are misidentified as being on the  No-Fly or Selectee list,  reduce the risk of unauthorized disclosure of sensitive watch-list  information, and  integrate information from DHS’s redress process into watch-list matching  so that individuals are less likely to be improperly or unfairly delayed or  prohibited from boarding an aircraft.
Statutory requirements govern the protection of personal information by  federal agencies, including the use of air passengers’ information by  Secure Flight. For example, the Privacy Act of 1974 places limitations on  agencies’ collection, disclosure, and use of personal information  maintained in systems of records. The Privacy Act requires agencies to  publish a notice—known as a System of Records Notice (SORN)—in the  Federal Register identifying, among other things, the type of data  collected, the types of individuals about whom information is collected,  the intended “routine” use of the data, and procedures that individuals can  use to review and correct personal information. Also, the E-Government  Act of 2002 requires agencies to conduct Privacy Impact Assessments  (PIA) that analyze how personal information is collected, stored, shared,  and managed in a federal system. Agencies are required to make their  PIAs publicly available if practicable.
section_title: Secure Flight Development and Watch-List Matching Process
paragraphs: According to TSA, the agency developed and is implementing Secure  Flight’s domestic watch-list matching function in 3 releases:    Release 1—Systems development and testing.
Release 2—First stages of parallel operations with airline operators during  which both Secure Flight and air carriers perform watch-list matching.
Release 3—Continued parallel operations with airline operators and  preparation for airline cutovers, in which Secure Flight will perform  passenger watch-list matching for domestic flights.
Under the Secure Flight watch-list matching process (see fig. 1), air  carriers submit passenger information, referred to as Secure Flight  Passenger Data, electronically through a DHS router or eSecure Flight, a  Web-based access system for air carriers that do not use automated  reservation systems to send and receive the data. Secure Flight Passenger  Data are matched automatically against watch-list records, with results  provided to air carriers through a Boarding Pass Printing Result.  Passengers are subject to three possible outcomes from the watch-list  matching process: cleared to fly, selected for additional screening, or  prohibited from flying. Individuals initially selected for additional  screening and those prohibited from flying undergo additional review,  which results in the final Boarding Pass Printing Result and may lead to  law enforcement involvement.
TSA is to use discretion to determine what constitutes a possible match  between passenger information and a watch-list record, based on  matching settings made in the system. The matching settings include   (1) the relative importance of each piece of passenger information (e.g.,  name versus date of birth); (2) the numeric threshold over which a  passenger will be flagged as a potential match (e.g., a scoring threshold of  95 would result in fewer matches than a scoring threshold of 85); and   (3) the criteria used to determine whether an element of passenger  information is a potential match to the watch list (e.g., the types of name  variations or the date-of-birth range that the system considers a match).  The Secure Flight matching system will use this information to assign each  passenger record a numeric score that indicates its strength as a potential  match to a watch-list record.
Raising the scoring threshold would result in more names cleared and  fewer names identified as possible matches, which would raise the risk of  the subject of a watch-list record being allowed to board an airplane  (false-negative matches). Conversely, lowering the scoring threshold  would raise the risk of passengers being mistakenly matched to the watch  list (false-positive matches). In October 2008, TSA issued the Secure Flight  Final Rule, which specifies requirements for air carriers to follow as TSA  implements and operates Secure Flight, including the collection of full  name and date-of-birth information from airline passengers to facilitate  watch-list matching.
In late-January 2009, TSA began to assume the watch-list matching  function for a limited number of domestic flights for one airline, and has  since phased in additional flights and airlines. TSA plans to complete  assumption of the watch-list matching function for all domestic flights in  March 2010 and to then assume from U.S. Customs and Border Protection  this watch-list-matching function for international flights departing to and  from the United States. According to TSA, since fiscal year 2004, it has  received approximately $300 million in appropriated funds for the  development and implementation of the Secure Flight program.
section_title: Related System Also Prescreens Airline Passengers
paragraphs: In addition to matching passenger information against terrorist watch-list  records, TSA requires air carriers to prescreen passengers using the  Computer-Assisted Passenger Prescreening System (CAPPS). Through  CAPPS, air carriers compare data related to a passenger’s reservation and  travel itinerary to a set of weighted characteristics and behaviors (CAPPS  rules) that TSA has determined correlate closely with the characteristics  and behaviors of terrorists. Passengers identified by CAPPS as exhibiting  these characteristics—termed selectees—must undergo additional  security screening. This system is separate from the Secure Flight watch- list matching process and thus Secure Flight has no effect on CAPPS  selection rates.
section_title: TSA Has Completed Key Activities Associated with Implementing Secure Flight, but Additional Actions Are Needed to Mitigate Risks
paragraphs: In a January 2009 briefing to congressional staff, we reported that TSA had  not demonstrated Secure Flight’s operational readiness and that the  agency had generally not achieved 5 of the 10 statutory conditions  (Conditions 3, 5, 6, 8, 10), although DHS asserted that it had satisfied all 10  conditions. Since then, TSA has made progress in developing the Secure  Flight program and meeting the requirements of the 10 conditions, and the  activities completed to date and those planned reduce the risks associated  with implementing the program. Table 2 shows the status of the 10  conditions as of April 2009.
subsections: section_title: TSA Has Generally Achieved 9 of the 10 Statutory Conditions, but Additional Actions Would Help Mitigate Future Risks
subsections: section_title: Condition 1: Redress
paragraphs: Condition 1 requires that a system of due process exist whereby aviation passengers  determined to pose a threat who are either delayed or prohibited from boarding their  scheduled flights by TSA may appeal such decisions and correct erroneous information  contained in the Secure Flight program.
TSA has generally achieved this condition. For the Secure Flight program,  TSA plans to use the existing redress process that is managed by the DHS  Traveler Redress Inquiry Program (TRIP). TRIP, which was established in  February 2007, serves as the central processing point within DHS for  travel-related redress inquiries. TRIP refers redress inquiries submitted by  airline passengers to TSA’s Office of Transportation Security Redress  (OTSR) for review. This process provides passengers who believe their  travels have been adversely affected by a TSA screening process with an  opportunity to be cleared if they are determined to be an incorrect match  to watch-list records, or to appeal if they believe that they have been  wrongly identified as the subject of a watch-list record. Specifically, air  travelers who apply for redress and who TSA determines pose no threat to  aviation security are added to a list that should automatically “clear” them  and allow them to board an aircraft (the “cleared list”), thereby reducing  any inconvenience experienced as a result of the watch-list matching  process. After a review of the passenger’s redress application, if OTSR  determines that an individual was, in fact, misidentified as being on the  No-Fly or Selectee list, it will add the individual to the cleared list. If OTSR  determines that an individual is actually on the No-Fly or Selectee list, it  will refer the matter to the Terrorist Screening Center, which determines  whether the individual is appropriately listed and should remain on the list  or is wrongly assigned and should be removed from the list.
Although Secure Flight will use the same redress process that is used by  the current air carrier-run watch-list matching process, some aspects of  the redress process for air travelers are to change as the program is  implemented. For example, individuals who apply for redress are issued a  redress number by TRIP that they will be able to submit during future  domestic air travel reservations that will assist in the preclearing process  before they arrive at the airport. TSA expects this will reduce the  likelihood of travel delays at check-in for those passengers who have been  determined to pose no threat to aviation security. According to TSA  officials, individuals who have applied for redress in the past and were  placed on the cleared list will need to be informed of their new ability to  use their redress number to preclear themselves under Secure Flight.  These officials stated that they intend to send mailings to past redress  applicants with information on this change.
TSA has also coordinated with key stakeholders to identify and document  shared redress processes and to clarify roles and responsibilities,  consistent with relevant GAO guidance for coordination and  documentation of internal controls. In addition, Secure Flight, TSA  OTSR, and TSA’s Office of Intelligence (OI) have jointly produced  guidance that clarifies how the entities will coordinate their respective  roles in the redress process, consistent with GAO best practices on  coordinating efforts across government stakeholders.
See GAO/GGD/AIMD-99-69. TSA OI is responsible for disseminating the cleared list. guidance clarifies the roles and responsibilities for each entity with  respect to reviewing potential watch-list matches.
Furthermore, TSA is developing performance measures to monitor the  timeliness and accuracy of Secure Flight redress, as we recommended in  February 2008. TRIP and OTSR’s performance goals are to process  redress applications as quickly and as accurately as possible. In February  2008, we reported that TRIP and OTSR track only one redress  performance measure, related to the timeliness of case completion. We  further reported that by not measuring all key defined program objectives,  TRIP and OTSR lack the information needed to oversee the performance  of the redress program. We recommended that DHS and TSA reevaluate  the redress performance measures and consider creating and  implementing additional measures, consistent with best practices that  among other things address all program goals, to include the accuracy of  the redress process.
In response to GAO’s recommendation, representatives from the TRIP  office are participating in a Redress Timeliness Working Group, with other  agencies involved in the watch-list redress process, to develop additional  timeliness measures. According to DHS officials, the TRIP office has also  established a quality assurance review process to improve the accuracy of  redress application processing and will collect and report on these data.
Secure Flight officials are developing additional performance measures to  measure new processes that will be introduced once Secure Flight is  operational, such as the efficacy of the system to preclear individuals who  submit a redress number.
section_title: Condition 2: Minimizing False Positives
paragraphs: Condition 2 requires that the underlying error rate of the government and private  databases that will be used both to establish identity and assign a risk level to a  passenger will not produce a large number of false-positives (mistakenly matched) that  will result in a significant number of passengers being treated mistakenly or security  resources being diverted.
TSA has generally achieved this condition by taking a range of actions that  should minimize the number of false-positive matches. For example, the  Secure Flight Final Rule requires air carriers to (1) collect date-of-birth  information from airline passengers and (2) be capable of collecting  redress numbers from passengers. Collecting date-of-birth information  should improve the system’s ability to correctly match passengers against  watch-list records since each record contains a date of birth. TSA  conducted a test in 2004 that concluded that the use of date-of-birth  information would reduce the number of false-positive matches. In  addition, airline passengers who have completed the redress process and  are determined by DHS to not pose a threat to aviation security can submit  their redress number when making a flight reservation. The submission of  redress numbers by airline passengers should reduce the likelihood of  passengers being mistakenly matched to watch list records, which in turn  should reduce the overall number of false-positive matches.
TSA has established a performance measure and target for the system’s  false-positive rate, which should allow the agency to track the extent to  which it is minimizing false-positive matches and whether the rate at any  point in time is consistent with the program’s goals. TSA officials stated  that they tested the system’s false-positive performance during Secure  Flight’s parallel testing with selected air carriers in January 2009 and  found that the false-positive rate was consistent with the established target  and program’s goals.
section_title: Condition 3: Efficacy and Accuracy of the System and Stress Testing
paragraphs: Condition 3 requires TSA to demonstrate the efficacy and accuracy of the search tools  used as part of Secure Flight and to perform stress testing on the Secure Flight  system.
We addressed efficacy and accuracy separately from stress testing  because they require different activities and utilize different criteria.
Efficacy and Accuracy of the System  TSA has generally achieved the part of Condition 3 that requires TSA to  demonstrate the efficacy and accuracy of the search tools used as part of  Secure Flight. According to TSA, as a screening system, Secure Flight is  designed to identify subjects of watch-list records without generating an  unacceptable number of false-positive matches. To accomplish this goal,  TSA officials stated that Secure Flight’s matching system and related  search parameters were designed to identify potential matches to watch- list records if a passenger’s date of birth is within a defined range of the  date of birth on a watch-list record. According to TSA officials, the  matching system and related search parameters were designed based on  TSA OI policy and in consultation with TSA OI, the Federal Bureau of  Investigation, and others.
TSA conducted a series of tests—using a simulated passenger list and a  simulated watch list created by a contractor with expertise in watch-list  matching—that jointly assessed the system’s false-negative and false- positive performance. However, in conducting these tests, the contractor  used a wider date-of-birth matching range than TSA used in designing the  Secure Flight matching system, which the contractor determined was  appropriate to test the capabilities of a name-matching system. The tests  showed that the Secure Flight system did not identify all of the simulated  watch-list records that the contractor identified as matches to the watch  list (the false-negative rate). Officials from TSA OI reviewed the test  results and determined that the records not matched did not pose an  unacceptable risk to aviation security. These officials further stated that  increasing the date-of-birth range would unacceptably increase the  number of false positives generated by the system.
Moving forward, TSA is considering conducting periodic reviews of the  Secure Flight system’s matching capabilities and results (i.e., false  positives and false negatives) to determine whether the system is  performing as intended. However, final decisions regarding whether to  conduct such reviews have not been made. Relevant guidance on internal  controls identifies the importance of ongoing monitoring of programs,  documenting control activities, and establishing performance measures to  assess performance over time. By periodically monitoring the system’s  matching criteria as well as documenting and measuring any results to  either (1) confirm that the system is producing effective and accurate  matching results or (2) modify the settings as needed, TSA would be able  to better assess whether the system is performing as intended. Without  such activities in place, TSA will not be able to assess the system’s false- negative rate, which increases the risk of the system experiencing future  performance shortfalls. Given the inverse relationship between false  positives and false negatives—that is, an increase in one rate may lead to a  decrease in the other rate—it is important to assess both rates  concurrently to fully test the system’s matching performance. In our  January 2009 briefing, we recommended that TSA periodically assess the  performance of the Secure Flight system’s matching capabilities to  determine whether the system is accurately matching watch-listed  individuals while minimizing the number of false positives. TSA agreed  with our recommendation.
Separate from the efficacy and accuracy of Secure Flight search tools, a  security concern exists. Specifically, passengers could attempt to provide  fraudulent information when making an airline reservation to avoid  detection. TSA officials stated that they are aware of this situation and are  taking actions to mitigate it.  We did not assess TSA's progress in taking  actions to address this issue or the effectiveness of TSA’s efforts as part of  this review.
The second part of Condition 3 requires TSA to perform stress testing on  the Secure Flight system. In our January 2009 briefing to the Senate and  House Appropriations Committees’ Subcommittees on Homeland Security,  we reported that TSA had generally not achieved this part of the condition  because despite provisions for stress testing in Secure Flight test plans,  such stress testing had not been performed at the time DHS certified that  it had met the 10 statutory conditions, or prior to the completion of our  audit work on December 8, 2008. However, TSA has since generally  achieved this part of the condition.
According to the Secure Flight Test and Evaluation Master Plan, the  system was to be stress tested in order to assess performance when  abnormal or extreme conditions are encountered, such as during periods  of diminished resources or an extremely high number of users. Further,  the Secure Flight Performance, Stress, and Load Test Plan states that the  system’s performance, throughput, and capacity are to be stressed at a  range beyond its defined performance parameters in order to find the  operational bounds of the system. In lieu of stress testing, program  officials stated that Release 2 performance testing included “limit testing”  to determine if the system could operate within the limits of expected  peak loads (i.e., defined performance requirements). According to the  officials, this testing would provide a sufficient basis for predicting which  system components would experience degraded performance and  potential failure if these peak loads were exceeded. However, in our view,  such “limit testing” does not constitute stress testing because it focuses on  the system’s ability to meet defined performance requirements only, and  does not stress the system beyond the requirements. Moreover, this “limit  testing” did not meet the provisions for stress testing in TSA’s own Secure  Flight test plans. Program officials agreed that the limit testing did not  meet the provisions for stress testing in accordance with test plans and  revised program test plans and procedures for Release 3 to include stress  testing.
Beyond stress testing, our analysis at the time of our January 2009 briefing  showed that TSA had not yet sufficiently conducted performance testing.  According to the Secure Flight Test and Evaluation Master Plan,  performance and load tests should be conducted to assess performance  against varying operational conditions and configurations. Further, the  Secure Flight Performance, Stress, and Load Test Plan states that each test  should begin within a limited scope and build up to longer runs with a  greater scope, periodically recording system performance results. These  tests also should be performed using simulated interfaces under real-world  conditions and employ several pass/fail conditions, including overall  throughput. However, Secure Flight Release 2 performance testing was  limited in scope because it did not include 10 of the 14 Secure Flight  performance requirements. According to program officials, these 10  requirements were not tested because they were to be tested as part of  Release 3 testing that was scheduled for December 2008. Moreover, 2 of  the 10 untested performance requirements were directly relevant to stress  testing. According to program officials, these 2 requirements were not  tested as part of Release 2 because the subsystems supporting them were  not ready at that time. Further, the performance testing only addressed the  4 requirements as isolated capabilities, and thus did not reflect real-world  conditions and demands, such as each requirement’s competing demands  for system resources. Program officials agreed and stated that they  planned to employ real world conditions in testing all performance  requirements during Release 3 testing.
In our January 2009 briefing, we recommended that TSA execute  performance and stress tests in accordance with recently developed plans  and procedures and report any limitations in the scope of the tests  performed and shortfalls in meeting requirements to its oversight board,  the DHS Investment Review Board. Since then, based on our analysis of  updated performance, stress, and load test procedures and results, we  found that TSA has now completed performance testing and significantly  stress tested the vetting system portion of Secure Flight. For example, the  stress testing demonstrated that the vetting system can process more than  10 names in 4 seconds, which is the system’s performance requirement. As  a result of the performance and stress testing that TSA has recently  conducted, we now consider this condition to be generally achieved and  the related recommendation we made at our January 2009 briefing to be  met.
section_title: Condition 4: Establishment of an Internal Oversight Board
paragraphs: Condition 4 requires the Secretary of Homeland Security to establish an internal  oversight board to monitor the manner in which the Secure Flight programs is being  developed and prepared.
TSA has generally achieved this condition through the presence of five  oversight entities that have met at key program intervals to monitor  Secure Flight. In accordance with GAO’s Standards for Internal Control in  the Federal Government, a system of internal controls should include,  among other things, an organizational structure that establishes  appropriate lines of authority, a process that tracks agency performance  against key objectives, and ongoing monitoring activities to ensure that  recommendations made were addressed. Consistent with these practices,  the internal oversight entities monitoring the Secure Flight program have  defined missions with established lines of authority, have met at key  milestones to review program performance, and have made  recommendations designed to strengthen Secure Flight’s development.  Our review of a selection of these recommendations showed that the  Secure Flight program addressed these recommendations.
The oversight entities for the Secure Flight program are the following:    DHS Steering Committee,    TSA Executive Oversight Board,    DHS Investment Review Board (IRB),    TSA IRB, and    DHS Enterprise Architecture Board (EAB).
The DHS Steering Committee and TSA Executive Oversight Board are  informal oversight entities that were established to provide oversight and  guidance to the Secure Flight program, including in the areas of funding,  and coordination with U.S. Customs and Border Protection (CBP) on  technical issues. According to TSA officials, the DHS Steering Committee  and TSA Executive Oversight Board do not have formalized approval  requirements outlined in management directives. The DHS IRB, TSA IRB,  and DHS EAB are formal entities that oversee DHS information technology  projects and focus on ensuring that investments directly support missions  and meet schedule, budget, and operational objectives. (App. III contains  additional information on these oversight boards.)
GAO has previously reported on oversight deficiencies related to the DHS  IRB, such as the board’s failure to conduct required departmental reviews  of major DHS investments (including the failure to review and approve a  key Secure Flight requirements document). To address these  deficiencies, GAO made a number of recommendations to DHS, such as  ensuring that investment decisions are transparent and documented as  required. DHS generally agreed with these recommendations. Moving  forward, it will be critical for these oversight entities to actively monitor  Secure Flight as it progresses through future phases of systems  development and implementation and ensure that the recommendations  we make in this report are addressed.
section_title: Conditions 5 and 6: Information Security
paragraphs: Conditions 5 and 6 require TSA to build in sufficient operational safeguards to reduce the  opportunities for abuse, and to ensure substantial security measures are in place to  protect the Secure Flight system from unauthorized access by hackers and other  intruders.
TSA has generally achieved the statutory requirements related to systems  information security based on, among other things, actions to mitigate  high- and moderate-risk vulnerabilities associated with Release 3. As of  completion of our initial audit work on December 8, 2008, which we  reported on at our January 2009 briefing, we identified deficiencies in  TSA’s information security safeguards that increased the risk that the  system will be vulnerable to abuse and unauthorized access from hackers  and other intruders.
Federal law, standards, and guidance identify the need to address  information security throughout the life cycle of information systems.
Accordingly, the guidance and standards specify a minimum set of security  steps needed to effectively incorporate security into a system during its  development. These steps include    categorizing system impact, performing a risk assessment, and  determining security control requirements for the system;    documenting security requirements and controls and ensuring that they  are designed, developed, tested, and implemented;    performing tests and evaluations to ensure controls are working properly  and effectively, and implementing remedial action plans to mitigate  identified weaknesses; and    certifying and accrediting the information system prior to operation.
To its credit, TSA had performed several of these key security steps for  Release 1, such as:    categorizing the system as high-impact, performing a risk assessment, and  identifying and documenting the associated recommended security control  requirements;    preparing security documentation such as a system security plan and  loading security requirements into the developer’s requirements  management tool;  testing and evaluating security controls for the Secure Flight system and  incorporating identified weaknesses in remedial action plans; and    conducting security certification and accreditation activities.
However, as of December 8, 2008, TSA had not taken sufficient steps to  ensure that operational safeguards and substantial security measures were  fully implemented for Release 3 of Secure Flight. This is important  because Release 3 is the version that is to be placed into production.  Moreover, Release 3 provides for (1) a change in the Secure Flight  operating environment from a single operational site with a “hot” backup  site to dual processing sites where each site processes passenger data  simultaneously, and (2) the eSecure Flight Web portal, which provides an  alternative means for air carriers to submit passenger data to Secure  Flight. While these changes could expose the Secure Flight program to  security risks not previously identified, TSA had not completed key  security activities to address these risks.
Further, we found that TSA had not completed testing and evaluating of  key security controls or performed disaster recovery tests for the Release  3 environment. These tests are important to ensure that the operational  safeguards and security measures in the production version of the Secure  Flight operating environment are effective, operate as intended, and  appropriately mitigate risks. In addition, TSA had not updated or  completed certain security documents for Release 3, such as its security  plan, disaster recovery plan, security assessment report, and risk  assessment, nor had it certified and accredited Release 3 of the Secure  Flight environment it plans to put into production. Further, TSA had also  not demonstrated that CBP had implemented adequate security controls  over its hardware and software devices that interface with the Secure  Flight system to ensure that Secure Flight data are not vulnerable to abuse  and unauthorized access.
Finally, TSA had not corrected 6 of 38 high- and moderate-risk  vulnerabilities identified in Release 1 of the Secure Flight program. For  example, TSA did not apply key security controls to its operating systems  for the Secure Flight environment, which could then allow an attacker to  view, change, or delete sensitive Secure Flight information. While TSA  officials assert that they had mitigated 4 of the 6 uncorrected  vulnerabilities, we determined the documentation provided was not  sufficient to demonstrate that the vulnerabilities were mitigated. As a  result of the security risks that existed as of December 8, 2008, we  recommended that TSA take steps to complete its security testing and  update key security documentation prior to initial operations.
After our January 2009 briefing, TSA provided documentation showing  that it had implemented or was in the process of implementing our  recommendation. For example, TSA had completed security testing of the  most recent release of Secure Flight (Release 3), updated security  documents, certified and accredited Release 3, received an updated  certification and accreditation decision from CBP for its interface with the  Secure Flight program, and mitigated the high- and moderate-risk  vulnerabilities related to Release 1. In addition, TSA had prepared plans of  actions and milestones (POA&M) for the 28 high-risk and 32 moderate-risk  vulnerabilities it identified during security testing of Release 3. The  POA&Ms stated that TSA would correct the high-risk vulnerabilities within  60 days and the moderate-risk vulnerabilities within 90 days. Based on  these actions, we concluded that TSA had conditionally achieved this  condition as of January 29, 2009.
Further, after we submitted our draft report to DHS for formal agency  comment on March 20, 2009, TSA provided us updated information that  demonstrated that it had completed the actions discussed above. Based on  our review of documentation provided by TSA on March 31, 2009, we  concluded that TSA had mitigated all 60 high- and moderate-risk  vulnerabilities associated with Release 3. Therefore, we concluded that  TSA had generally achieved the statutory requirements related to systems  information security and we consider the related recommendation to be  met.
section_title: Condition 7: Oversight of the Use and Operation of the System
paragraphs: Condition 7 requires TSA to adopt policies establishing effective oversight of the use and  operation of the Secure Flight system.
As of the completion of our initial audit work on December 8, 2008, TSA  had generally achieved this condition, but we nevertheless identified  opportunities for strengthening oversight and thus made a  recommendation aimed at doing so. According to GAO’s best practices for  internal control, effective oversight includes (1) the plans and procedures  used to meet mission goals and objectives, and (2) activities that ensure  the effectiveness and efficiency of operations, safeguard assets, prevent  and detect errors and fraud, and provide reasonable assurance that a  program is meeting its intended objectives. To its credit, TSA had  finalized the vast majority of key documents related to the effective  oversight of the use and operation of the system as of the completion of  our initial audit work on December 8, 2008. For example, TSA had  established performance measures to monitor and assess the effectiveness  of the Secure Flight program; provided training to air carriers on  transitioning their watch-list matching functions to TSA; developed a plan  to oversee air carriers’ compliance with Secure Flight program  requirements; and finalized key standard operating procedures. However,  TSA had not yet finalized or updated all key program documents or  completed necessary training, which was needed prior to the program  beginning operations. Accordingly, we recommended that TSA finalize or  update all key Secure Flight program documents—including the  agreement with the Terrorist Screening Center for exchanging watch-list  and passenger data and standard operating procedures—and complete  training before the program begins operations. In response, TSA finalized  its memorandum of understanding with the Terrorist Screening Center on  December 30, 2008, and completed program training in January 2009.  Based on these actions, we consider this recommendation to be met.  Appendix IV contains additional information on Condition 7.
section_title: Condition 8: Privacy
paragraphs: Condition 8 requires TSA to take action to ensure that no specific privacy concerns  remain with the technological architecture of the Secure Flight system.
TSA has generally achieved the statutory requirement related to privacy  based on progress the agency has made in establishing a privacy program  as well as recent actions taken to address security vulnerabilities related  to conditions 5 and 6. In our January 2009 briefing, we identified  deficiencies in TSA’s information security safeguards that posed a risk to  the confidentiality of the personally identifiable information maintained by  the Secure Flight system.
The Fair Information Practices, a set of principles first proposed in 1973  by a U.S. government advisory committee, are used with some variation by  organizations to address privacy considerations in their business practices  and are also the basis of privacy laws and related policies in many  countries, including the United States, Australia, and New Zealand, as well  as the European Union. The widely-adopted version developed by the  Organisation for Economic Co-operation and Development in 1980 is  shown in table 3.
At the time of our January 2009 briefing, TSA had established a variety of  programmatic and technical controls for Secure Flight, including  involving privacy experts in major aspects of Secure Flight development,    developing privacy training for all Secure Flight staff and incident  response procedures to address and contain privacy incidents,  tracking privacy issues and performing analysis when significant privacy  issues are identified,  instituting access controls to ensure that data are not accidentally or  maliciously altered or destroyed,  filtering unauthorized data from incoming data to ensure collection is  limited to predefined types of information,    establishing standard formats for the transmission of personally  identifiable information (PII) in order to reduce variance in data and  improve data quality, and    maintaining audit logs to track access to PII and document privacy  incidents.
In addition, TSA had issued required privacy notices—including a Privacy  Impact Assessment and System of Records Notice—that meet legal  requirements and address key privacy principles. These notices describe,  among other things, the information that will be collected from passengers  and airlines, the purpose of collection, and planned uses of the data.  Through its privacy program, TSA had taken actions to implement most  Fair Information Practice Principles. For information on the actions TSA  has taken to generally address Fair Information Practices, see appendix V.
However, at our January 2009 briefing, we also concluded that the  weaknesses in Secure Flight’s security posture—as described in our  earlier discussion of information security—created an increased risk that  the confidentiality of the personally identifiable information maintained by  the Secure Flight system could be compromised. As a result, we  recommended that TSA take steps to complete its security testing and  update key security documentation prior to initial operations.
After our January 2009 briefing, TSA provided documentation that it had  implemented or was in the process of implementing our recommendation  related to information security and we concluded that this condition had  been conditionally achieved as of January 29, 2009. Further, after we  submitted our draft report to DHS for formal agency comment on March  20, 2009, TSA provided us updated information that demonstrated that it  had completed the actions to implement our recommendation. Based on  our review of documentation provided by TSA on March 31, 2009, we  believe TSA has generally achieved the condition related to privacy.
section_title: Condition 9: CAPPS Rules
paragraphs: Condition 9 requires that TSA—pursuant to the requirements of section  44903(i)(2)(A) of title 49, United States Code—modify Secure Flight with respect to  intrastate transportation to accommodate states with unique air transportation needs and  passengers who might otherwise regularly trigger primary selectee status.
TSA has generally achieved this condition. TSA is developing the Secure  Flight program without incorporating the CAPPS rules and, therefore,  Secure Flight will have no effect on CAPPS selection rates. According to  TSA, the agency has modified the CAPPS rules to address air carriers  operating in states with unique transportation needs and passengers who  might otherwise regularly trigger primary selectee status. However, our  review found that TSA lacked data on the effect of its modifications on air  carrier selectee rates. We interviewed four air carriers to determine (1) the  extent to which the CAPPS modifications and a related security  amendment affected these carriers’ selectee rates and (2) whether TSA  had outreached to these carriers to assess the effect of the modifications  and amendment on their selectee rates. The carriers provided mixed  responses regarding whether the modifications and amendment affected  their selectee rates. Further, three of the four air carriers stated that TSA  had not contacted them to determine the effect of these initiatives.  According to GAO best practices for internal control, agencies should  ensure adequate means of communicating with, and obtaining information  from, external stakeholders that may have a significant effect on achieving  goals. Without communications with air carriers, and given the agency’s  lack of data on carrier selectee rates, TSA cannot ensure that the CAPPS  modifications and related security amendment have their intended effect.  In our January 2009 briefing, we recommended that TSA conduct outreach  to air carriers—particularly carriers in states with unique transportation  needs—to determine whether modifications to the CAPPS rules and  security amendment have achieved their intended effect. TSA agreed with  our recommendation.
section_title: TSA Has Conditionally Achieved 1 of the 10 Conditions, but Further Actions Are Needed to Mitigate the Risk of Cost and Schedule Overruns
subsections: section_title: Condition 10: Life-Cycle Cost and Schedule Estimates
paragraphs: Condition 10 requires the existence of appropriate life-cycle cost estimates and  expenditure and program plans.
TSA has conditionally achieved this statutory requirement based on our  review of its plan of action for developing appropriate cost and schedule  estimates and other associated documents submitted after we provided a  copy our draft report to DHS for formal comment on March 20, 2009. The  plan includes proposed activities and time frames for addressing  weaknesses that we identified in the Secure Flight program’s cost estimate  and schedule and was the basis for our reassessment of this condition.
At the time of our January 2009 briefing, we reported that this condition  had generally not been achieved. Specifically, while TSA had made  improvements to its life-cycle cost estimate and schedule, neither were  developed in accordance with key best practices outlined in our Cost  Assessment Guide. Our research has identified several practices that are  the basis for effective program cost estimating. We have issued guidance  that associates these practices with four characteristics of a reliable cost  estimate: comprehensive, well documented, accurate, and credible. The  Office of Management and Budget (OMB) endorsed our guidance as being  sufficient for meeting most cost and schedule estimating requirements. In  addition, the best practices outlined in our guide closely match DHS’s own  guidance for developing life-cycle cost estimates. Reliable cost and  schedule estimates are critical to the success of a program, as they provide  the basis for informed investment decision making, realistic budget  formulation, program resourcing, meaningful progress measurement,  proactive course correction, and accountability for results.
As we reported at our January 2009 briefing, Secure Flight’s $1.36 billion  Life Cycle Cost Estimate (LCCE) is well documented in that it clearly  states the purpose, source, assumptions, and calculations. However, it is  not comprehensive, fully accurate, or credible. As a result, the life-cycle  cost estimate does not provide a meaningful baseline from which to track  progress, hold TSA accountable, and provide a basis for sound investment  decision making. In our January 2009 briefing, we recommended that DHS  take actions to address these weaknesses. TSA agreed with our  recommendation.
The success of any program depends in part on having a reliable schedule  specifying when the program’s set of work activities will occur, how long  they will take, and how they relate to one another. As such, the schedule  not only provides a road map for the systematic execution of a program,  but it also provides the means by which to gauge progress, identify and  address potential problems, and promote accountability. As we reported in  January 2009, the November 15, 2008, TSA’s Integrated Master Schedule  (IMS) for Secure Flight—which provided supporting activities leading up  to the program’s initial operations in January 2009—was a significant  improvement over its February 2008 version. For example, after meeting  with GAO and its schedule analysis consultant, TSA took actions to  improve the Secure Flight schedule, including adding initial efforts for  domestic and international cutover activities, removing constraints that  kept its schedule rigid, and providing significant status updates.
Our research has identified nine practices associated with effective  schedule estimating, which we used to assess Secure Flight. These  practices are: capturing key activities, sequencing key activities,  establishing duration of key activities, assigning resources to key  activities, integrating key activities horizontally and vertically, establishing  critical path, identifying float time, performing a schedule risk analysis,  and distributing reserves to high risk activities. In assessing the  November 15, 2008, schedule against our best practices, we found that  TSA had met one of the nine best practices, but five were only partially  met and three were not met. Despite the improvements TSA made to its  schedule for activities supporting initial operational capability, the  remaining part of the schedule associated with implementing Secure Flight  for domestic and international flights was represented as milestones  rather than the detailed work required to meet milestones and events. As  such, the schedule was more characteristic of a target deliverable plan  than the work involved with TSA assuming the watch-list matching  function. Moreover, likely program completion dates were not being  driven by the schedule logic, but instead were being imposed by the  program office in the form of target dates. This practice made it difficult  for TSA to use the schedule to reflect the program’s status. Without fully  employing all key scheduling practices, TSA cannot assure a sufficiently  reliable basis for estimating costs, measuring progress, and forecasting  slippages. In our January 2009 briefing, we recommended that DHS take  actions to address these weaknesses. TSA agreed with our  recommendation.
In January 2009, TSA provided us with a new schedule, dated December  15, 2008. Our analysis showed that this new schedule continued to not  follow best practices, did not correct the deficiencies we previously  identified, and therefore could not be used as a reliable management tool.  For example, a majority of the scheduled activities did not have baseline  dates that allow the schedule to be tracked against a plan moving forward.  In addition, best practices require that a schedule identify the longest  duration path through the sequenced list of key activities—known as the  schedule’s critical path—where if any activity slips along this path, the  entire program will be delayed. TSA’s updated schedule did not include a  critical path, which prevents the program from understanding the effect of  any delays. Further, updating the Secure Flight program’s schedule is  important because of the significant cost and time that remains to be  incurred to cutover all domestic flights to operations as planned by March  2010 and to develop, test, and deploy the functionality to assume watch- list matching for international flights.
After we submitted a copy of our draft report to DHS for formal agency  comment on March 20, 2009, TSA provided us its plan of action, dated  April 2009, that details the steps the Secure Flight program management  office intends to carry out to address weaknesses that we identified in the  program’s cost and schedule estimates. With regard to the program’s cost  estimate, TSA’s plan has established a timeline of activities that, if  effectively implemented, should result in (1) a more detailed work  breakdown structure that would define the work necessary to accomplish  the program’s objectives; (2) the cost estimate and schedule work  breakdown structures being aligned properly; (3) an independent cost  estimate performed by a contractor; (4) an assessment of the life-cycle  cost estimate by the DHS Cost Analysis Division; and (5) cost uncertainty  and sensitivity analyses. In addition, TSA’s plan has estimated government  costs that were originally missing from its cost estimate. According to  TSA, these costs will be addressed in its life-cycle cost estimate  documentation.
With regard to the Secure Flight program’s schedule, TSA’s plan of action  has established a timeline of activities that, if effectively implemented,  should result in, most notably: (1) a sequenced and logical schedule that  will accurately calculate float time and a critical path; (2) a fully resource- loaded schedule based on subject-matter-expert opinion that does not  overburden resources; (3) a schedule that includes realistic activity  duration estimates; and (4) a schedule risk analysis that will be used by  TSA leadership to distribute reserves to high-risk activities. According to  TSA, this revised schedule will forecast the completion date for the project  based on logic, duration, and resource estimates rather than artificial date  constraints.
The plan of action provides the Secure Flight program management office  with a clearer understanding of the steps that need to be taken to address  our concerns regarding the Secure Flight life-cycle cost estimate and  schedule. Based on our review of the plan and the associated  documentation provided, we therefore now consider this legislative  requirement to be conditionally achieved and the related  recommendations that we made at our January 2009 briefing to be met. It  should be noted that a significant level of effort is involved in completing  these activities, yet the actions—with the exception of the independent  cost estimate—are planned to be completed by June 5, 2009. According to  TSA, the independent cost estimate is to be completed by October 2009.
While TSA’s ability to fully meet the requirements of Condition 10 does not  affect the Secure Flight system’s operational readiness, having reliable  cost and schedule estimates allows for better insight into the management  of program resources and time frames as the program is deployed. We will  continue to assess TSA’s progress in carrying out the plan of action to  address the weaknesses that we identified in the program’s cost estimate  and schedule and fully satisfying this condition. Appendix VI contains  additional information on our analysis of TSA’s efforts relative to GAO’s  best practices.
section_title: Conclusions
paragraphs: TSA has made significant progress in developing the Secure Flight  program, and the activities completed to date, as well planned, reduce the  risks associated with implementing the program. However, TSA is still in  the process of taking steps to address key activities related to testing the  system’s watch-list matching capability and cost and schedule estimates,  which should be completed to mitigate risks and to strengthen the  management of the program.
Until these activities are completed, TSA lacks adequate assurance that  Secure Flight will fully achieve its desired purpose and operate as  intended. Moreover, if these activities are not completed expeditiously, the  program will be at an increased risk of cost, schedule, or performance  shortfalls. Specifically, the system might not perform as intended in the  future if its matching capabilities and results (that is, false positives and  false negatives) are not periodically assessed. In addition, cost overruns  and missed deadlines will likely occur if reliable benchmarks are not  established for managing costs and the remaining schedule.
In addition to the issues and risks we identified related to the Secure  Flight program, our work revealed one other TSA prescreening-related  issue that should be addressed to mitigate risks and ensure that passenger  prescreening is working as intended. Specifically, the effect that  modifications to the CAPPS rules and a related security amendment have  had on air carriers—particularly carriers in states with unique  transportation needs—will remain largely unknown unless TSA conducts  outreach to these air carriers to determine the effect of these changes.
section_title: Recommendations for Executive Action
paragraphs: We are recommending that the Secretary of Homeland Security take the  following two actions:
To mitigate future risks of performance shortfalls and strengthen  management of the Secure Flight program moving forward, we  recommend that the Secretary of Homeland Security direct the  Assistant Secretary for the Transportation Security Administration  to periodically assess the performance of the Secure Flight  system’s matching capabilities and results to determine whether  the system is accurately matching watch-listed individuals while  minimizing the number of false positives—consistent with the  goals of the program; document how this assessment will be  conducted and how its results will be measured; and use these  results to determine whether the system settings should be  modified.
To ensure that passenger prescreening is working as intended, we  recommend that the Secretary of Homeland Security direct the  Assistant Secretary for the Transportation Security Administration  to conduct outreach to air carriers—particularly carriers in states  with unique transportation needs—to determine whether  modifications to the CAPPS rules and related security amendment  have achieved their intended effect.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for review and comment on  March 20, 2009. Subsequently, TSA provided us additional information  related to several of the conditions, which resulted in a reassessment of  the status of these conditions. Specifically, in the draft report that we  provided for agency comment, we had concluded that Conditions 5 and 6  (information security) and Condition 8 (privacy) were conditionally  achieved and Condition 10 (cost and schedule) was generally not  achieved. Based on our review of the additional documentation provided  by TSA, we are now concluding that Conditions 5, 6, and 8 are generally  achieved and Condition 10 is conditionally achieved.
In addition, in the draft report we provided to DHS for agency comment,  we made five recommendations, four of which were related to the Secure  Flight program. The fifth recommendation was related to Condition 9  (CAPPS rules), which is not related to the Secure Flight program. Based  on the additional information that TSA provided during the agency  comment period, we now consider three of these recommendations to be  met (those related to information security, the cost estimate, and the  program schedule). The other two recommendations have not been met  and, therefore, are still included in this report (those related to monitoring  the performance of the system’s matching capability and assessing the  effect of modifications on CAPPS rules). We provided our updated  assessment to DHS and on April 23, 2009, DHS provided us written  comments, which are presented in appendix VII. In its comments, DHS  stated that TSA concurred with our updated assessment.
We are sending copies of this report to the appropriate congressional  committees and other interested parties. We are also sending a copy to the  Secretary of Homeland Security. This report will also be available at no  charge on our Web site at http://www.gao.gov. Should you or your staff  have any questions about this report, please contact Cathleen A. Berrick at  (202) 512-3404 or berrickc@gao.gov; Randolph C. Hite at (202) 512-3439 or  hiter@gao.gov; or Gregory C. Wilshusen at (202) 512-6244 or  wilshuseng@gao.gov.
Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. Key contributors to  this report are acknowledged in appendix VIII.
section_title: Appendix I: Objectives, Scope, and Methodology
subsections: section_title: Objectives
paragraphs: In accordance with section 513 of the Department of Homeland Security  Appropriations Act, 2008, our objective was to assess the extent to which  the Transportation Security Administration (TSA) met the requirements of  10 statutory conditions related to the development and implementation of  the Secure Flight program and the associated risks of any shortfalls in  meeting the requirements. Specifically, the act requires the Secretary of  Homeland Security to certify, and GAO to report, that the 10 statutory  conditions have been successfully met before TSA implements or deploys  the program on other than a test basis. Pursuant to the act, after the  Department of Homeland Security (DHS) certified that it had satisfied all  10 conditions—which it did on September 24, 2008—we were required to  report within 90 days on whether the 10 conditions had been successfully  met. It further requires GAO to report periodically thereafter until it  determines that all 10 conditions have been successfully met.
section_title: Scope and Methodology
paragraphs: Our overall methodology included (1) identifying key activities related to  each condition; (2) identifying federal guidance and related best practices,  if applicable, that are relevant to successfully meeting each condition (e.g.,  GAO’s Standards for Internal Control in the Federal Government); (3)  analyzing whether TSA has demonstrated through verifiable analysis and  documentation, as well as oral explanation, that the guidance has been  followed and best practices have been met; and (4) assessing the risks  associated with not fully following applicable guidance and meeting best  practices. Based on our assessment, we categorized each condition as  generally achieved, conditionally achieved, or generally not achieved.
Generally achieved—TSA has demonstrated that it completed all key  activities related to the condition in accordance with applicable federal  guidelines and related best practices, which should reduce the risk of the  program experiencing cost, schedule, or performance shortfalls.
Conditionally achieved—TSA has demonstrated that it completed some  key activities related to the condition in accordance with applicable  federal guidelines and related best practices and has defined plans for  completing remaining key activities that, if effectively implemented as  planned, should result in reduced risk that the program will experience  cost, schedule, or performance shortfalls.
Generally not achieved—TSA has not demonstrated that it completed all  key activities related to the condition in accordance with applicable  federal guidelines and related best practices and does not have defined  plans for completing the remaining activities, and the uncompleted  activities result in an increased risk of the program experiencing cost,  schedule, or performance shortfalls.
In conducting this review, we worked constructively with TSA officials.  We provided TSA with our criteria for assessing each of the 10 conditions  and periodically met with TSA officials to discuss TSA’s progress and our  observations. To meet our 90-day reporting requirement, we conducted  audit work until December 8, 2008, which included assessing activities and  documents that TSA completed after DHS certified that it had met the 10  conditions. We reported the initial results of our review to the mandated  reporting committees in two restricted briefings, first on December 19,  2008, and then on January 7, 2009. Because we concluded that TSA had  not successfully met all 10 conditions, we conducted additional work from  January through April 2009, the results of which are also included in this  report. Further, after we submitted a copy of our draft report to DHS for  formal agency comment on March 20, 2009, TSA provided us additional  information related to Conditions 5, 6, 8, and 10 which resulted in our  reassessment of the status of these conditions. The report has been  updated to include the additional information and reassessments.
subsections: section_title: Condition 1: Redress
paragraphs: To assess Condition 1 (redress), we interviewed program officials and  reviewed and assessed agency documentation to determine how, once  Secure Flight becomes operational, the DHS redress process will be  coordinated with the Secure Flight program, based upon GAO best  practices for coordination; as well as whether the process was  documented, consistent with GAO best practices on documenting internal  controls. We also reviewed performance measures for the Secure Flight  redress process as well as TSA’s progress in addressing a February 2008  GAO recommendation that DHS consider creating and implementing  additional measures for its redress process.
section_title: Condition 2: Minimizing False Positives
paragraphs: To assess Condition 2 (minimizing false positives), we interviewed  program and TSA Office of Intelligence (OI) officials and reviewed and  assessed Secure Flight performance objectives, tests, and other relevant  documentation to determine the extent to which TSA’s activities  demonstrate that the Secure Flight system will minimize its false-positive  rate. Additionally, we interviewed program and TSA OI officials and  reviewed and assessed Secure Flight documentation to determine how the  program established performance goals for its false-positive and false- negative rates. We also interviewed a representative from the contractor  that designed a dataset that TSA used to test the efficacy and accuracy of  Secure Flight’s matching system to discuss the methodology of that  dataset. Our engagement team, which included a social science analyst  with extensive research methodology experience and engineers with  extensive experience in systems testing, reviewed the test methodologies  for the appropriateness and logical structure of their design and  implementation, any data limitations, and the validity of the results. Our  review focused on steps TSA is taking to reduce false-positive matches  produced by Secure Flight’s watch-list matching process, which is  consistent with TSA’s interpretation of the requirements of this condition.  We did not review the Terrorist Screening Center’s role in ensuring the  quality of records in the Terrorist Screening Database (TSDB).
section_title: Condition 3: Efficacy and Accuracy of the System and Stress Testing
paragraphs: To assess the first part of Condition 3 (efficacy and accuracy of the  system), we interviewed program and TSA OI officials and reviewed and  assessed Secure Flight performance objectives, tests, and other  documentation that address the type and extent of testing and other  activities that demonstrate that Secure Flight will minimize the number of  false positives while not allowing an unacceptable number of false  negatives. We also interviewed a representative from the contractor that  designed a dataset that TSA used to test the efficacy and accuracy of  Secure Flight’s matching system to discuss the methodology of that  dataset. Our engagement team, which included a social science analyst  with extensive research methodology experience and engineers with  extensive experience in systems testing, reviewed the test methodologies  for the appropriateness and logical structure of their design and  implementation and the validity of the results. However, we did not assess  the appropriateness of TSA’s definition of what should constitute a match  to the watch list. We did not assess the accuracy of the system’s predictive  assessment, as this is no longer applicable to the Secure Flight program  given the change in its mission scope compared to its predecessor  program CAPPS II (i.e., Secure Flight only includes comparing passenger  information to watch-list records whereas CAPPS II was to perform  different analyses and access additional data, including data from  commercial databases, to classify passengers according to their level of  risk).
To assess the second part of Condition 3, stress testing, we reviewed  Secure Flight documentation—including test plans, test procedures, and  test results—and interviewed program officials to determine whether TSA  has defined and managed system performance and stress requirements in  a manner that is consistent with relevant guidance and standards. We also  determined whether the testing that was performed included testing the  performance of Secure Flight search tools under increasingly heavy  workloads, demands, and conditions to identify points of failure. For  example, in January 2009, we met with the Secure Flight development  team and a program official to observe test results related to the 14 Secure  Flight performance and stress requirements. We walked through each of  the 14 requirements and observed actual test scenarios and results.
section_title: Condition 4: Establishment of an Internal Oversight Board
paragraphs: To assess Condition 4 (internal oversight), we interviewed DHS and TSA  program officials and reviewed and analyzed documentation related to  various DHS and TSA oversight boards—the DHS and TSA Investment  Review Boards, the DHS Enterprise Architecture Board, the TSA  Executive Oversight Board, and the DHS Steering Committee—to identify  the types of oversight provided to the Secure Flight program. We also  reviewed agency documentation to determine whether the oversight  entities met as intended and, in accordance with GAO’s Standards for  Internal Control in the Federal Government, the extent to which the  Secure Flight program has addressed a selection of recommendations and  action items made by the oversight bodies. We evaluated oversight  activities related to key milestones in the development of the Secure Flight  system.
section_title: Conditions 5 and 6: Information Security
section_title: Condition 7: Oversight of the Use and Operation of the System
paragraphs: In regard to Condition 7 (oversight of the system), for purposes of  certification, TSA primarily defined effective oversight of the system in  relation to information security. However, we assessed DHS’s oversight  activities against a broader set of internal controls for managing the  program, as outlined in GAO’s Standards for Internal Control in the  Federal Government, to oversee the Secure Flight system during  development and implementation. We interviewed Secure Flight program  officials and reviewed agency documentation—including policies,  standard operating procedures, and performance measures—to determine  the extent to which policies and procedures addressed the management,  use, and operation of the system. We also interviewed program officials at  TSA’s Office of Security Operations to determine how TSA intends to  oversee internal and external compliance with system security, privacy  requirements, and other functional requirements. We did not assess the  quality of documentation provided by TSA. Our methodology for assessing  information security is outlined under Conditions 5 and 6.
section_title: Condition 8: Privacy
paragraphs: To assess Condition 8 (privacy), we analyzed legally-required privacy  documentation, including systems-of-record notices and privacy impact  assessments, as well as interviewed Secure Flight and designated TSA  privacy officials to determine the completeness of privacy safeguards. In  addition, we assessed available systems development documentation to  determine the extent to which privacy protections have been addressed  based on the Fair Information Practices. We also assessed whether key  documentation had been finalized and key provisions, such as planned  privacy protections, had been clearly determined. We reassessed the  status of Condition 8 based on our review of documentation provided by  TSA on March 31, 2009, showing that it had mitigated all high- and  moderate-risk information security vulnerabilities associated with the  Secure Flight program’s Release 3.
section_title: Condition 9: CAPPS Rules
paragraphs: To assess Condition 9 (CAPPS rules), we reviewed TSA documentation to  identify modifications to the CAPPS rules and a related security program  amendment to address air carriers operating in states with unique  transportation needs and passengers who might otherwise regularly  trigger primary selectee status. In addition, we interviewed TSA officials to  determine the extent to which TSA assessed the effect of these activities  on air carriers’ selectee rates—either through conducting tests or by  communicating with and obtaining information from air carriers—in  accordance with GAO best practices for coordinating with external  stakeholders. We also interviewed officials from four air carriers to  obtain their views regarding the effect of CAPPS changes on the air  carriers’ selectee rates. These carriers were selected because they operate  in states with unique transportation needs or have passengers who might  otherwise regularly trigger primary selectee status as a result of CAPPS  rules.
section_title: Condition 10: Life-Cycle Cost and Schedule Estimates
paragraphs: To assess Condition 10 (cost and schedule estimates), we reviewed the  program’s life-cycle cost estimate, integrated master schedule, and other  relevant agency documentation against best practices, including GAO’s  Cost Estimating and Assessment Guide: Best Practices for Developing and  Managing Capital Program Costs.  We also interviewed key program  officials overseeing these activities and consulted with a scheduling expert  to identify risks to the integrated master schedule. We reassessed the  status of Condition 10, based on TSA’s plan of action provided to us on  April 3, 2009. The Plan of Action, dated April 2009, details the steps the  Secure Flight program management office intends to carry out to address  weaknesses that we identified in the program’s cost and schedule  estimates. Appendix VI contains additional information on our analysis of  TSA’s efforts relative to GAO’s best practices.
We conducted this performance audit from May 2008 to May 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Details on TSA’s Testing of the Efficacy and Accuracy of Secure Flight’s Matching System (Condition 3)
paragraphs: Appendix II: Details on TSA’s Testing of the  Efficacy and Accuracy of Secure Flight’s  Matching System (Condition 3)
The Transportation Security Administration (TSA) hired a contractor with  expertise in matching systems to construct a dataset against which to test  the Secure Flight matching system and assess the system’s false-positive  and false-negative performance. Given the inverse relationship between  false positives and false negatives—that is, a decrease in one may lead to  an increase in the other—it is important to assess both rates concurrently  to fully test the system’s matching performance. The contractor developed  the dataset specifically for Secure Flight using name-matching software  and expert review by analysts and linguists.
The dataset consisted of a passenger list and a watch list using name types  that were consistent with those on the actual No-Fly and Selectee lists.  Each record included a passenger name and date of birth. The passenger  list consisted of about 12,000 records, of which nearly 1,500 were “seeded”  records that represented matches to the simulated watch list. According  to the contractor, the seeded records were plausible variations to  passenger names and dates of birth based on the contractor’s analysis of  real watch-list records.
The passenger list was run through Secure Flight’s automated matching  system to determine its ability to accurately match the passenger records  against the simulated watch list. The system used name-matching criteria  outlined in the TSA No-Fly List security directive, and a defined date-of- birth matching criteria that TSA officials state was consistent with TSA  Office of Intelligence policy.
According to TSA, Secure Flight officials reviewed the test results to  determine whether the system was accurately applying its matching  criteria for passenger name and date of birth. TSA officials concluded that  all matches and nonmatches made by the system were in accordance with  these criteria. The test results for the system’s default matching rules  showed that the system produced a number of false-negative matches— that is, of the passenger records deemed by the contractor to be matches  to the watch list, Secure Flight did not match a number of those records.  TSA officials stated that the false-negative rate in the test was primarily  due to the Secure Flight system’s criteria for a date-of-birth match, which  differed from the contractor’s criteria.
TSA determined a criteria range for a date-of-birth match that was  consistent with TSA Office of Intelligence policy. According to TSA  officials, these matching criteria are consistent with Secure Flight’s  responsibilities as a screening program—that is, the system must process  high passenger volumes and quickly provide results to air carriers—and  that those responsibilities were considered when balancing the risk  presented by the system’s false-positive and false-negative rates. The  contractor’s date-of-birth criteria range, however, was wider than the  range used by TSA, which the contractor stated was established based on  expert analysis of an excerpt from the watch list.
According to TSA officials, officials from TSA’s Office of Intelligence  reviewed the test results and determined that the records identified as  false negatives by the contractor— that is, the records that were matched  by the contractor but not by the Secure Flight system—did not pose an  unacceptable risk and should not have been flagged, and that these  nonmatches were designated as such in accordance with Office of  Intelligence policies and TSA’s No Fly list security directive. These  officials further stated that increasing the date-of-birth range would  unacceptably increase the number of false positives generated by the  system.
TSA officials stated that the Secure Flight system’s matching setting could  be reconfigured in the future to adjust the system’s false-positive and false- negative matching results should the need arise—for example, due to  relevant intelligence information or improvements in the system’s  matching software.
section_title: Appendix III: Secure Flight’s Oversight Entities (Condition 4)
paragraphs: Appendix III: Secure Flight’s Oversight  Entities (Condition 4)
Table 4 shows the entities responsible for overseeing the development of  the Secure Flight program and a sample of activities that had been  completed.
section_title: Appendix IV: TSA’s Activities Related to the Effective Oversight of System Use and Operation (Condition 7)
paragraphs: Appendix IV: TSA’s Activities Related to the  Effective Oversight of System Use and  Operation (Condition 7)
The Transportation Security Administration (TSA) completed several  internal control activities related to the management, use, and operation of  the Secure Flight system. For example:    TSA developed 21 standard operating procedures related to Secure Flight’s  business processes. In addition, TSA incorporated additional  programmatic procedures into various plans and manuals that will provide  support for the program once it becomes operational. According to a  Secure Flight official, all 21 standard operating procedures were finalized  as of December 12, 2008.
TSA released its Airline Operator Implementation Plan, which is a written  procedure describing how and when an aircraft operator transmits  passenger and nontraveler information to TSA. The plan amends an  aircraft operator’s Aircraft Operator Standard Security Program to  incorporate the requirements of the Secure Flight program.
TSA finalized its plan to oversee air carrier compliance with Secure  Flight’s policies and procedures. All domestic air carriers and foreign  carriers covered under the Secure Flight rule will be required to comply  with and implement requirements set forth in the final rule.
The Airline Operator Implementation Plan and the Consolidated User  Guide will provide air carriers with the requirements for compliance  monitoring during the initial cutover phases.
The Airline Implementation Team, which assists air carriers’ transition to  Secure Flight, will ensure that air carriers are in compliance with program  requirements prior to cutover.
TSA developed performance measures to monitor and assess the  effectiveness of the Secure Flight program, such as measures to address  privacy regulations, training requirements, data quality and submission  requirements, and the functioning of the Secure Flight matching engine.  TSA will also use performance measures to ensure that air carriers are  complying with Secure Flight data requirements.
TSA developed written guidance for managing Secure Flight’s workforce,  including a Comprehensive Training Plan that outlines training  requirements for users and operators of the system and service centers.
According to TSA officials, TSA completed programmatic training, which  includes privacy and program-related training, for the entire Secure Flight  workforce.
TSA provided stakeholder training for covered U.S. air carriers and foreign  air carriers on the Secure Flight program. This training, while not required  of stakeholders, provided air carriers with information on changes to the  Secure Flight program after the Final Rule was released and technical and  operational guidance as outlined in the Consolidated User Guide. The  Airline Implementation, Communications, and Training Teams will  support requests from air carriers for additional training throughout  deployment.
According to TSA, the agency planned to pilot its operational training,  which is necessary for employees and contractors to effectively undertake  their assigned responsibilities, during the week of December 8, 2008. TSA  officials stated that piloting this training would allow them to make any  needed updates to Secure Flight’s standard operating procedures.  However, TSA officials said that updates to the Standard Operating  Procedures as a result of training were expected to be minimal and would  not have an effect on initial cutover in their view.
section_title: Appendix V: TSA’s Actions to Address Fair Information Practices (Condition 8)
paragraphs: Appendix V: TSA’s Actions to Address Fair  Information Practices (Condition 8)
The Transportation Security Administration (TSA) has taken actions that  generally address the following Fair Information Practices.
The Purpose Specification principle states that the purposes for a  collection of personal information should be disclosed before collection  and upon any change to that purpose. TSA addressed this principle by  issuing privacy notices that define a specific purpose for the collection of  passenger information. According to TSA privacy notices, the purpose of  the Secure Flight Program is to identify and prevent known or suspected  terrorists from boarding aircraft or accessing sterile areas of airports and  better focus passenger and baggage screening efforts on persons likely to  pose a threat to civil aviation, to facilitate the secure and efficient travel of  the public while protecting individuals’ privacy.
The Data Quality principle states that personal information should be  relevant to the purpose for which it is collected, and should be accurate,  complete, and current as needed for that purpose. TSA addressed this  principle through its planned use of the Department of Homeland  Security’s (DHS) Traveler Redress Inquiry Program (TRIP), collecting  information directly from passengers, and setting standard data formats.  More specifically, TSA is planning to use DHS TRIP as a mechanism to  correct erroneous data. TSA also believes that relying on passengers to  provide their own name, date of birth, and gender will further help ensure  the quality of the data collected. Moreover, TSA has developed a  Consolidated User Guide that provides standard formats for air carriers to  use when submitting passenger information to reduce variance and  improve data quality. We reported previously that the consolidated  terrorist watch list, elements of which are matched with passenger data to  make Secure Flight screening decisions, has had data- quality issues.  However, this database is administered by the Terrorist Screening Center  and is not overseen by TSA.
The Openness principle states that the public should be informed about  privacy policies and practices, and that individuals should have a ready  means of learning about the use of personal information. TSA addressed  this principle by publishing and receiving comments on required privacy  notices. TSA has issued a Final Rule, Privacy Impact Assessment, and  System of Records Notice that discuss the purposes, uses, and protections  for passenger data, and outline which data elements are to be collected  and from whom. TSA obtained and responded to public comments on its  planned measures for protecting the data a passenger is required to  provide.
The Individual Participation principle states that individuals should have  the following rights: to know about the collection of personal information,  to access that information, to request correction, and to challenge the  denial of those rights. TSA addressed this principle through its planned  use of DHS TRIP and its Privacy Act access and correction process. As  previously mentioned, TSA plans to use DHS TRIP in order to allow  passengers to request correction of erroneous data. Passengers can also  request access to the information that is maintained by Secure Flight  through DHS’s Privacy Act request process. As permitted by the Privacy  Act, TSA has claimed exemptions from the Privacy Act that limit what  information individuals can access about themselves. For example,  individuals will not be permitted to view information concerning whether  they are in the Terrorist Screening Database (TSDB). However, TSA has  stated that it may waive certain exemptions when disclosure would not  adversely affect law enforcement or national security.
The Use Limitation principle states that personal information should not  be used for other than a specified purpose without consent of the  individual or legal authority. TSA addressed this principle by identifying  permitted disclosures of data and establishing mechanisms to ensure that  disclosures are limited to those authorized. The Secure Flight system  design requires that data owners initiate transfers of information, a  provision that helps to assure that data is being used only for specified  purposes. According to TSA privacy notices, the Secure Flight Records  system is intended to be used to identify and protect against potential and  actual threats to transportation security through watch-list matching  against the No-Fly and Selectee components of the consolidated and  integrated terrorist watch list known as the Terrorist Screening Database.  TSA plans to allow other types of disclosures, as permitted by the Privacy  Act. For example, TSA is permitted to share Secure Flight data with  federal, state, local, tribal, territorial, foreign, or international agencies  responsible for investigating, prosecuting, enforcing, or implementing a  statute, rule, regulation, or order regarding a violation or potential  violation of civil or criminal law or regulation; and international and foreign governmental authorities in accordance with law  and formal or informal international agreements.
The Collection Limitation principle states that the collection of personal  information should be limited, should be obtained by lawful and fair  means, and, where appropriate, with the knowledge or consent of the  individual. TSA addressed this principle by conducting a data-element  analysis, developing a data retention schedule, and establishing technical  controls to filter unauthorized data and purge data. TSA has performed a  data element analysis to determine the least amount of personal  information needed to perform effective automated matching of  passengers with individuals on the watch list. As a result, TSA has limited  collection by only requiring that passengers provide their full name,  gender, and date of birth. In addition, TSA requires air carriers to request  other specific information, such as a passenger’s redress number, and to  provide TSA with other specific information in the airline’s possession,  such as the passenger’s passport information. TSA established a data- purging control to rid the system of data according to its data-retention  schedule. Further, TSA established technical controls to filter  unauthorized data to ensure that collection is limited to authorized data  fields. TSA is also developing a data-retention schedule which was issued  for public comment and is in accordance with the Terrorist Screening  Center’s National Archives and Records Administration (NARA)—- approved record-retention schedule for TSDB records.
The Accountability principle states that individuals controlling the  collection or use of personal information should be accountable for taking  steps to ensure the implementation of these principles. TSA addressed the  Accountability principle by designating a program privacy officer and a  team of privacy experts working on various aspects of the Secure Flight  program, and by planning to establish several oversight mechanisms:   TSA implemented a system for tracking privacy issues that arise  throughout the development and use of Secure Flight, and TSA is  conducting follow-up analysis of significant privacy issues and providing  resolution strategies for management consideration.
TSA developed privacy rules of behavior, which require that individuals  handling personally identifiable information (PII) only use it for a stated  purpose.
TSA is planning to maintain audit logs of system and user events to  provide oversight of system activities, such as access to PII and transfer of  PII in or out of the system.
TSA is planning to issue periodic privacy compliance reports, intended to  track and aggregate privacy concerns or incidents, but it has not finalized  the reporting process.
TSA developed general privacy training for all Secure Flight staff and is  developing role-based privacy training for employees handling PII.
While TSA has also taken steps related to the Security Safeguards  principle, this principle had not been fully addressed at the time of our  January 2009 briefing. The Security Safeguards principle states that  personal information should be protected with reasonable security  safeguards against risks such as loss or unauthorized access, destruction,  use, modification, or disclosure. TSA actions to address the Security  Safeguards principle include planning to prevent unauthorized access to  data stored in its system through technical controls including firewalls,  intrusion detection, encryption, and other security methods. Although TSA  had laid out a plan to protect the confidentiality of sensitive information  through various security safeguards, our security review—discussed in  more detail under conditions 5 and 6 on information security—identified  weaknesses in Secure Flight’s security posture that create an increased  risk that the confidentiality of the personally identifiable information  maintained by the Secure Flight system could be compromised. As a result  of the security risks we identified and reported on at our January 2009  briefing, and their corresponding effect on privacy, we recommended that  TSA take steps to complete its security testing and update key security  documentation prior to initial operations. TSA agreed with our  recommendation.
Since our January 2009 briefing, TSA provided documentation that it has  implemented our recommendation related to information security. In light  of these actions, we believe TSA has now generally achieved the condition  related to privacy and we consider the related recommendation we made  at the briefing to be met.
section_title: Appendix VI: GAO Analyses of Secure Flight’s Life-Cycle Cost Estimate and Schedule against Best Practices (Condition 10)
paragraphs: Appendix VI: GAO Analyses of Secure Flight’s  Life-Cycle Cost Estimate and Schedule  against Best Practices (Condition 10)
After submitting a copy of our draft report to the Department of Homeland  Security (DHS) for formal agency comment on March 20, 2009, the  Transportation Security Administration (TSA) provided us its plan of  action, dated April 2009, that details the steps the Secure Flight program  management office intends to carry out to address weaknesses that we  identified in the program’s cost and schedule estimates. We reviewed  TSA’s plan and associated documentation and reassessed the program  against our Cost and Schedule Best Practices. The following tables show  our original assessment and reassessment of TSA’s cost and schedule  against our best practices.
Table 5 summarizes the results of our analysis relative to the four  characteristics of a reliable cost estimate based on information provided  by TSA as of March 20, 2009.
Table 6 summarizes the results of our reassessment of the Secure Flight  program’s cost estimate relative to the four characteristics of a reliable  cost estimate based on information provided by TSA as of April 3, 2009.
Table 7 summarizes the results of our analysis relative to the nine  schedule-estimating best practices based on information provided by TSA  as of March 20, 2009.
Table 8 summarizes the results of our reassessment of the Secure Flight  program’s schedule relative to the nine schedule estimating best practices  based on information provided by TSA as of April 3, 2009.
section_title: Appendix VII: Comments from the Department of Homeland Security
section_title: Appendix VIII: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the contacts listed above, Idris Adjerid, David Alexander,  Mathew Bader, Timothy Boatwright, John de Ferrari, Katherine Davis, Eric  Erdman, Anthony Fernandez, Ed Glagola, Richard Hung, Jeff Jensen,  Neela Lakhmani, Jason Lee, Thomas Lombardi, Sara Margraf, Vernetta  Marquis, Victoria Miller, Daniel Patterson, David Plocher, Karen Richey,  Karl Seifert, Maria Stattel, Margaret Vo, and Charles Vrabel made key  contributions to this report.