Transportation Worker Identification Credential: Card Reader Pilot Results Are Unreliable; Security Benefits Need to Be Reassessed
Released: 2013-05-08T00:00:00

Letter

Background

TWIC Program History and Our Prior Related Work

Following the terrorist attacks of September 11, 2001, the Aviation and Transportation Security Act (ATSA) was enacted in November 2001 and required TSA to work with airport operators to strengthen access controls to secure areas, and to consider using biometric access control systems, or similar technologies, to verify the identity of individuals who seek to enter a secure airport area. In response, TSA established the TWIC program in December 2001. TWIC was originally envisioned as a nationwide transportation worker identity solution to be used by approximately 6 million credential holders across all modes of transportation, including seaports, airports, rail, pipeline, trucking, and mass transit facilities. In November 2002, MTSA further required DHS to issue a maritime worker identification card that uses biometrics to control access to secure areas of maritime transportation facilities and vessels.

TSA and USCG decided to implement TWIC initially in the maritime domain. Other transportation modes such as aviation have a preference for site-specific credentials.

As defined by DHS, and consistent with the requirements of MTSA, the purpose of the TWIC program is to design and field a common biometric credential for all transportation workers across the United States who require unescorted access to secure areas at MTSA-regulated maritime facilities and vessels. As stated in the TWIC mission needs statement, the TWIC program aims to meet the following mission needs: positively identify authorized individuals who require unescorted access to secure areas of the nation’s transportation system, determine the eligibility of individuals to be authorized unescorted access to secure areas of the transportation system by conducting a security threat assessment, ensure that unauthorized individuals are not able to defeat or otherwise compromise the access system in order to be granted permissions that have been assigned to an authorized individual, and identify individuals who fail to maintain their eligibility requirements subsequent to being permitted unescorted access to secure areas of the nation’s transportation system and immediately revoke the individual’s permissions.

In 2005, TSA conducted an analysis of alternatives and a cost-benefit analysis to identify possible options for addressing MTSA’s requirement to develop a biometric transportation security card that would also meet the related mission needs specified above. On the basis of these analyses, TSA determined that the best alternative was for the federal government to issue a single biometric credential that could be used across all vessels and maritime facilities, and for the government to manage all aspects of the credentialing process—enrollment, card issuance, and card revocation. TSA considered an alternative option based on a more decentralized and locally managed approach wherein MTSA-regulated facilities, vessels, and other port-related entities could issue their own credentials after individuals passed a TSA security threat assessment, but ultimately rejected the option (additional details are provided later in this report).

Transportation Security Administration. Transportation Worker Identification Credential (TWIC) Program Analysis of Alternatives Version 2.0. Feb. 15, 2005, and Transportation Worker Identification Credential (TWIC) Program Cost Benefit Analysis, Version 1.0. Aug. 31, 2005. environment.contractor’s report identified problems with the report, such as inaccurate and missing information. As a result, the independent assessment recommended that TSA not rely on the contractor’s final report on the TWIC prototype when making future decisions about the implementation of TWIC.

We found that an independent assessment of the testing In 2006, the SAFE Port Act amended MTSA and directed the Secretary of Homeland Security to, among other things, implement a TWIC reader pilot to test the technology and operational impacts of deploying card readers at maritime facilities and vessels. August 2008. This pilot was conducted with the voluntary participation of maritime port, facility, and vessel operators at 17 sites within the United States. In November 2009, we reported on the TWIC reader pilot design and planned approach, and found that DHS did not have a sound evaluation approach to ensure information collected through the TWIC reader pilot would be complete, accurate, and representative of deployment conditions. Among other things, we recommended that an evaluation plan and data analysis plan be developed to guide the remainder of the pilot and to identify how DHS would compensate for areas where the TWIC reader pilot would not provide the information needed to report to Congress and implement the TWIC card reader rule. DHS concurred with this recommendation. The status of TSA’s efforts to develop these plans is discussed later in this report. In addition, the Coast Guard Authorization Act of 2010 required that the findings of the pilot be included in a report to Congress, and that we assess the reported findings and recommendations.

Pub. L. No 109-347, § 104(a), 120 Stat. 1884, 1888 (codified at 46 U.S.C. § 70105(k)).

In May 2011, we reported that internal control weaknesses governing the enrollment, background checking, and use of TWIC potentially limited the program’s ability to provide reasonable assurance that access to secure areas of MTSA-regulated facilities is restricted to qualified individuals. We also reported that DHS had not assessed the TWIC program’s effectiveness at enhancing security or reducing risk for MTSA-regulated facilities and vessels. Further, we reported that DHS had not conducted a risk-informed cost-benefit analysis that considered existing security risks. We recommended, among other things, that DHS (1) assess TWIC program internal controls to identify needed corrective actions; (2) assess the TWIC program’s effectiveness; and (3) use the information from the assessment as the basis for evaluating the costs, benefits, security risks, and corrective actions needed to implement the TWIC program in a manner that will meet program objectives and mitigate existing security risks. DHS concurred with our recommendations and has taken steps to assess TWIC program internal controls. Appendix II summarizes key activities in the implementation of the TWIC program.

Over $23 million had been made available to pilot participants from two Federal Emergency Management Agency (FEMA) grant programs—the Port Security Grant Program and the Transit Security Grant Program. Of the $23 million, grant recipients agreed to spend nearly $15 million on the TWIC reader pilot. However, DHS is unable to validate the exact amount grant recipients spent on the TWIC reader pilot, as rules for allocating what costs would be included as TWIC reader pilot costs versus other allowable grant expenditures were not defined. Sixteen of the 17 participating pilot sites were funded using these grants. In addition, TSA obligated an additional $8.1 million of appropriated funds to support the pilot. proposed rulemaking published on March 22, 2013, estimated an additional cost of $234.2 million (undiscounted) to implement readers at 570 facilities and vessels that the TWIC reader currently targets. However, USCG does not rule out expanding reader requirements in the future. Appendix III contains additional program funding details.

TSA’s Pilot to Test Key TWIC-Related Access Control Technologies

The TWIC reader pilot was intended to test the technology, business processes, and operational impacts of deploying TWIC readers at secure areas of the marine transportation system. Accordingly, the pilot was to test the viability of using selected biometric card readers to read TWICs within the maritime environment. It was also to test the technical aspects of connecting TWIC readers to access control systems. The results of the pilot are to inform the development of a proposed rule requiring the use of electronic card readers with TWICs at MTSA-regulated vessels and facilities.

To conduct the TWIC reader pilot, TSA contracted with the Navy’s Space and Naval Warfare Systems Command (SPAWAR) to serve as the independent test agent to plan, analyze, evaluate, and report on all test events. Furthermore, the Navy’s Naval Air Systems Command (NAVAIR) conducted environmental testing of select TWIC readers. In addition, TSA partnered with the maritime industry at 17 pilot sites distributed across seven geographic locations within the United States.

See appendix IV for a complete listing of the pilot sites, locations, and types of maritime operation each represented. Levels of participation varied across the pilot sites. For example, at one facility, one pedestrian turnstile was tested out of 22 identified entry points. At another, the single vehicle gate was tested, but none of the seven pedestrian gates were tested. At a third facility with three pedestrian gates and 36 truck lanes, all three turnstiles and 2 truck lanes were tested. According to TSA, given the voluntary nature of the pilot, levels of participation varied across the pilot sites, and TSA could not dictate to the respective facilities and vessels specific and uniform requirements for testing.

The TWIC reader pilot, as initially planned, was to consist of three sequential assessments, with the results of each assessment intended to inform the subsequent ones. Table 1 highlights key aspects of the three assessments.

To address time and cost constraints related to using the results of the TWIC reader pilot to inform the TWIC card reader rule, two key changes were made to the pilot tests in 2008. First, TSA and USCG inserted an initial reader evaluation as the first step of the initial technical test. This evaluation was an initial assessment of each reader’s ability to read a TWIC. Initiated in August 2008, the initial reader evaluation resulted in a list of biometric card readers from which pilot participants could select for use in the pilot rather than waiting for the entire ITT to be completed. Further, the list of readers that passed the initial reader evaluation was used by TSA and USCG to help select a limited number of readers for full functional and environmental testing. Second, TSA did not require the TWIC reader pilot to be conducted in the sequence highlighted in table 1. Rather, pilot sites were allowed to conduct the early operational assessment and the system test and evaluation testing while ITT was under way.

Various reports were produced to document the results of each TWIC reader pilot assessment. An overall report was produced to document the ITT results conducted prior to testing at pilot sites. To document the results of testing at each of the 17 pilot sites, the independent test agent produced one EOA report and one ST&E report for each site. These reports summarized information collected from each of the pilot sites and trip reports documenting the independent test agent’s observations during visits to pilot sites.

On February 27, 2012, DHS conveyed the results of the TWIC reader pilot by submitting the TWIC Reader Pilot Program report to Congress. On March 22, 2013, USCG issued a notice of proposed rulemaking that would, if finalized, require owners and operators of certain MTSA- regulated vessels and facilities to use readers designed to work with TWICs.

TWIC Reader Pilot Results Are Not Sufficiently Complete, Accurate, and Reliable for Informing Congress and the TWIC Card Reader Rule

Challenges related to pilot planning, data collection, and reporting affect the completeness, accuracy, and reliability of the pilot test aimed at assessing the technology and operational impact of using TSA’s TWIC with card readers. Moreover, according to our review of the pilot and TSA’s past efforts to demonstrate the validity and security benefits of the TWIC program, the program’s premise and effectiveness in enhancing security are not supported.

Shortfalls in Planning Affected the Completeness, Accuracy, and Reliability of Data Collected during the Pilot

As we previously reported, TSA encountered challenges in its efforts to plan the TWIC reader pilot. In November 2009, we reviewed and reported on the TWIC reader pilot design and planned approach for collecting data at pilot sites. For example, we reported that the pilot test and evaluation documentation did not identify how individual pilot site designs and resulting variances in the information collected from each pilot site were to be assessed. This had implications for both the technology aspect of the pilot as well as the business and operational aspect. We further reported that pilot site test designs may not be representative of future plans for using TWIC because pilot participants were not necessarily using the technologies and approaches they intend to use in the future when TWIC readers are implemented at their sites.reported that there was a risk that the selected pilot sites and test methods would not result in the information needed to understand the impacts of TWIC nationwide. At the time, TSA officials told us that no specific unit of analysis, site selection criteria, or sampling methodology As a result, we was developed or documented prior to selecting the facilities and vessels to participate in the TWIC reader pilot.

As a result of these challenges, we recommended that DHS, through TSA and USCG, develop an evaluation plan to guide the remainder of the pilot that includes (1) performance standards for measuring the business and operational impacts of using TWIC with biometric card readers, (2) a clearly articulated evaluation methodology, and (3) a data analysis plan. We also recommended that TSA and USCG identify how they will compensate for areas where the TWIC reader pilot will not provide the necessary information needed to report to Congress and inform the TWIC card reader rule. DHS concurred with these recommendations.

While TSA developed a data analysis plan, TSA and USCG reported that they did not develop an evaluation plan with an evaluation methodology or performance standards, as we recommended. The data analysis plan was a positive step because it identified specific data elements to be captured from the pilot for comparison across pilot sites. If accurate data had been collected, adherence to the data analysis plan could have helped yield valid results. However, TSA and the independent test agent did not utilize the data analysis plan. According to officials from the independent test agent, they started to use the data analysis plan but stopped using the plan because they were experiencing difficulty in collecting the required data and TSA directed them to change the reporting approach. TSA officials stated that they directed the independent test agent to change its collection and reporting approach because of TSA’s inability to require or control data collection to the extent required to execute the data analysis plan. However, TSA and USCG did not fully identify how they would compensate for areas where the pilot did not provide the necessary information needed to report to Congress and inform the TWIC card reader rule. For example, such areas could include (1) testing to determine the impact of the business and operational processes put in place by a facility to handle those persons that are unable to match their live fingerprint to the fingerprint template stored on the TWIC and (2) requiring operators using a physical access control system in conjunction with a TWIC to identify how they are protecting personal identify information and testing how this protection affects the speed of processing TWICs. While USCG commissioned two studies to help compensate for areas where the TWIC reader pilot will not provide necessary information, the studies did not compensate for all of the challenges we identified in our November 2009 report. Such challenges included, for example, the impact of adding additional security protection on systems to prevent the disclosure of personal identity information and the related cost and processing implications.

In addition, our review of the TWIC reader pilot approach as implemented since 2009 and resulting pilot data identified some technology issues that affected the reliability of the TWIC reader pilot data. As DHS noted in its report to Congress, successful implementation of TWIC readers includes the development of an effective system architecture and physical access control system and properly functioning TWIC cards, among other things. However, not all TWIC card readers used in the TWIC reader pilot underwent both environmental and functional tests in the laboratory prior to use at pilot sites as originally intended. Because of cost and time constraints, TSA officials instead conducted an initial evaluation of all readers included in the pilot to determine their ability to read a TWIC. These initial evaluations resulted in a list of 30 biometric TWIC card readers from which pilot participants could select a reader for use. However, of these 30 readers, 8 underwent functional testing and 5 underwent environmental testing. None of the TWIC card readers underwent and passed all tests.

TSA and independent test agent summary test results note that ambiguities within the TWIC card reader specification—the documented requirements for what and how TWIC card readers are to function—may have led to different interpretations and caused failures of tested TWIC systems. According to TSA, the readers that underwent laboratory-based environmental and functional testing and were placed on the TSA list of acceptable readers did not have problems that would severely impact pilot site operations or prevent the collection of useful pilot data and therefore the readers were all available for use during the pilot. However, according to our review of the pilot documentation, TSA did not define what “severely impact” meant or performance thresholds for reader problems identified during laboratory-based environmental and functional testing that would severely impact pilot site operations or prevent the collection of useful pilot data. Further, according to TSA officials, TSA could not eliminate 1 of the readers that may have failed a test from the list of acceptable readers when other readers that had not been tested would be allowed on the list. According to TSA officials, doing so would have been an unfair disadvantage to the readers that were selected for the more rigorous laboratory-based environmental and functional testing. In addition, TSA did not provide pilot sites with the results of the laboratory-based environmental and functional testing. According to TSA, it signed confidentiality agreements with reader vendors, which prevented it from sharing this information. The results could have been used to help inform each pilot site’s selection of readers appropriate for its organization’s environmental and operational considerations. This may have hindered TSA’s efforts to determine if issues observed during the pilot were due to the TWIC, TWIC reader, or a facility’s access control system. Nonetheless, TSA determined that information collected during reader laboratory-based testing and at pilot sites was still useful for refining future TWIC reader specifications.

In addition, while TWIC cards are intended for use in the harsh maritime environment, the finalized TWIC cards did not undergo durability testing prior to testing at pilot sites. TSA selected card stock that had been tested in accordance with defined standards. However, TSA did not conduct durability tests of the TWIC cards after they were personalized with security features, such as the TWIC holder’s picture, or laminated. According to TSA, technology reasons that may render a TWIC card damaged include, among others, breakage to the antenna or the antenna’s connection to the card’s computing chip. Without testing the durability of personalized TWIC cards, the likelihood that TWIC cards and added security features can withstand the harsh maritime environment is unknown. According to TWIC program officials, each TWIC is tested to ensure it functions prior to being issued to an individual. However, the finalized TWIC card was not tested for durability to ensure that it could withstand the harsh maritime environment because doing so would be costly; TWIC is a fee-funded program, and the officials believed it would be unfair to pass on the cost to further test TWICs to consumers. However, testing TWIC credentials to ensure they can withstand the harsh maritime environment may prove to be more cost-effective, as it could minimize the time lost at access points and the TWIC holder’s need to pay a $60 replacement fee if the TWIC were to fail.

The importance of durability testing has been recognized by other government agencies and reported by GAO as a means to identify card failures before issuance. For example, the Department of Defense’s (DOD) common access card—also used in harsh environments such as Afghanistan and other areas with severe weather conditions—has, according to DOD officials, been tested after personalization to ensure that it remains functional and durable. DOD also assesses returned nonfunctioning common access cards to identify the potential cause of card failures. In addition, in June 2010, as part of our review of another credential program, we recommended that the Department of State fully test or evaluate the security features on its Border Crossing Cards, including any significant changes made to the cards’ physical construction, security features, or appearance during the development process. Thus, durability testing TWIC cards after personalization could have reduced the pervasiveness of problems encountered with malfunctioning TWIC cards during the pilot.

As a result of the noted planning and preparation shortfalls, including (1) the absence of defined performance standards for measuring pilot performance, (2) variances in pilot site testing approaches without compensating measures to ensure complete and comparable data were collected, and (3) inadequate testing to ensure that piloted readers and TWICs worked as intended, the data TSA and the independent test agent collected on the technology and operational impacts of using TWIC at pilot sites were not complete, accurate, and reliable.

Data Collection Challenges Were Encountered during the TWIC Reader Pilot

In addition to the pilot planning challenges discussed above, we found that the data collected through the pilot are also not generalizable because of certain pilot implementation and data collection practices we identified. As required by the SAFE Port Act of 2006, the pilot was to test the technology and operational impacts of deploying transportation security card readers at secure areas of the marine transportation system. In addition, as set forth in the TWIC test and evaluation master plan, the TWIC reader pilot was to provide accurate and timely information necessary to evaluate the economic impact of a nationwide deployment of TWIC card readers at over 16,400 MTSA-regulated facilities and vessels, and was to be focused on assessing the use of TWIC readers in contactless mode. However, data were collected and recorded in an incomplete and inconsistent manner during the pilot, further undermining the completeness, accuracy, and reliability of the data collected at pilot sites. Table 2 presents a summary of TWIC reader pilot data collection and supporting documentation reporting weaknesses that we identified that affected the completeness, accuracy, and reliability of the pilot data, which we discuss in further detail below. 1. Installed TWIC readers and access control systems could not collect required data on TWIC reader use, and TSA and the independent test agent did not employ effective compensating data collection measures. The TWIC reader pilot test and evaluation master plan recognizes that in some cases, readers or related access control systems at pilot sites may not collect the required test data, potentially requiring additional resources, such as on-site personnel, to monitor and log TWIC card reader use issues. Moreover, such instances were to be addressed as part of the test planning. However, the independent test agent reported challenges in sufficiently documenting reader and system errors. For example, in its monthly communications with TSA, the independent test agent reported that the logs from the TWIC readers and related access control systems were not detailed enough to determine the reason for errors, such as biometric match failure, an expired TWIC card, or that the TWIC was identified as being on the list of revoked credentials. The independent test agent further reported that the inability to determine the reason for errors limited its ability to understand why readers were failing, and thus it was unable to determine whether errors encountered were due to TWIC cards, readers, or users, or some combination thereof. As a result, according to the independent test agent, in some cases the TWIC readers and automated access control systems at various pilot sites were not capable of collecting the data required to assess pilot results. According to the independent test agent, this was primarily due to the lack of reader messaging standards—that is, a set of standard messages readers would display in response to each transaction type. Some readers used were newly developed by vendors, and some standards were not defined, causing inconsistencies in the log capabilities of some readers.manufacturers and system integrators—or individuals or companies that integrate TWIC-related systems—were not all willing to alter their systems’ audit logs to collect the required information, such as how long a transaction might take prior to granting access. Both TSA and the independent test agent agree that this issue limited their ability to collect the data needed for assessing pilot results.

The independent test agent noted that reader According to TSA officials, TSA allowed pilot participants to select their own readers and related access control systems and audit logs.

Consequently, TSA could not require that logs capable of meeting pilot data collection needs be used. In addition, TSA officials noted that determining the reason for certain errors, such as biometric match failures, could be made only while the independent test agent was present and had the time and ability to investigate the reason that a TWIC card had been rejected by a reader for access. On average, the independent test agent visited each pilot participant seven times during the early operational assessment and system test and evaluation testing period. TSA further noted that the development or use of alternative automated data collection methods would have been costly and would have required integration with the pilot site’s system. However, given that TSA was aware of the data needed from the pilot sites prior to initiating testing and the importance of collecting accurate and consistent data from the pilot, proceeding with the pilot without implementing adequate compensating mechanisms for collecting requisite data or adjusting the pilot design accordingly is inconsistent with the basic components of effective evaluation design and renders the results less reliable. 2. Reported transaction data did not match underlying documentation. A total of 34 pilot site reports were issued by the independent test agent. According to TSA, the pilot site reports were used as the basis for DHS’s report to Congress. We separately requested copies of the 34 pilot site reports from both TSA and the independent test agent. In comparing the reports provided, we found that 31 of the 34 pilot site reports provided to us by TSA did not contain the same information as those provided by the independent test agent. Differences for 27 of the 31 pilot site reports pertained to how pilot site data were characterized, such as the baseline throughput time used to compare against throughput times observed during two phases of testing: early operational assessment and systems test and evaluation. For example, TSA inserted USCG’s 6-second visual inspection estimate as the baseline throughput time measure for all pilot site access points in its amended pilot site reports instead of the actual throughput time collected and reported by the independent test agent during baseline data collection efforts. However, at two pilot sites, Brownsville and Staten Island Ferry, transaction data reported by the independent test agent did not match the data included in TSA’s reports. For example, of the 15 transaction data sets in the Staten Island Ferry ST&E report, 10 of these 15 data sets showed different data reported by TSA and the independent test agent. These differences were found in the weekly transactions and the sum total of valid and invalid transactions.

According to TSA officials, it used an iterative process to review and analyze pilot data as the data became available to it from the pilot participant sites. In addition, TSA officials noted that the independent test agent’s reports were modified in order to “provide additional context” and consistent data descriptions, and to present data in a more usable or understandable manner. Specifically, according to TSA officials, they and USCG officials believed that they had more knowledge of the data than the independent test agent and there was a need, in some cases, for intervening and changing the test reports in order to better explain the data. USCG officials further noted that the independent test agent’s draft reports were incomplete and lacked clarity, making revisions necessary to make the information more thorough. TSA also reported that it inadvertently used an earlier version of the report and not the final September 2011 site reports provided by the independent test agent to prepare the report to Congress.

In addition to differences found in the EOA and ST&E pilot site reports, we found differences between the data recorded during the independent test agent’s visits to pilot sites versus data reported in the EOA and ST&E pilot site reports. Data recorded during the independent test agent’s visits to pilot sites in trip reports were to inform final pilot site reports. The independent test agent produced 76 trip reports containing throughput data. We examined 34 of the 76 trip reports and found that all 34 trip reports contained data that were excluded or did not match data reported in the EOA and ST&E pilot site reports completed by the independent test agent. According to the independent test agent, the trip reports did not match the EOA and ST&E pilot site reports because the trip reports contained raw data that were analyzed and prepared for presentation in the participant EOA and ST&E pilot site reports. However, this does not explain why data reported by date in trip reports do not match related data in the EOA and ST&E pilot site reports. Having inconsistent versions of final pilot site reports, conflicting data in the reports, and data excluded from final reports without explanation calls into question the accuracy and reliability of the data. 3. Pilot documentation did not contain complete TWIC reader and access control system characteristics. Pilot documentation did not always identify which TWIC readers or which interface (e.g., contact or contactless interface) the reader used to communicate with the TWIC card during data collection. For example, at one pilot site, two different readers were tested. However, the pilot site report did not identify which data were collected using which reader. Likewise, at pilot sites that had readers with both a contact and a contactless interface, the pilot site report did not always identify which interface was used during data collection efforts. According to TSA officials, sites were allowed to determine which interface to use based on their business and operational needs. According to the independent test agent, it had no control over what interface pilot sites used during testing if more than one option was available. Consequently, pilot sites could have used the contactless interface for some transactions and the contact interface for others without recording changes. The independent test agent therefore could not document with certainty which interface was used during data collection efforts. Without accurate documentation of information such as this, an assessment of TWIC reader performance based on interface cannot be determined. This is a significant data reliability issue, as performance may vary depending on which interface is used, and in accordance with the TWIC reader pilot’s test and evaluation master plan, use of the contactless interface was a key element to be evaluated during the pilot. 4. TSA and the independent test agent did not record clear baseline data for comparing operational performance at access points with TWIC readers. Baseline data, which were to be collected prior to piloting the use of TWIC with readers, were to be a measure of throughput time, that is, the time required to inspect a TWIC card and complete access- related processes prior to granting entry. This was to provide the basis for quantifying and assessing any TWIC card reader impacts on the existing systems at pilot sites. Pilot documentation shows that baseline throughput data were collected for all pilot sites. However, it is unclear from the documentation whether acquired data were sufficient to reliably identify throughput times at truck, other vehicle, and pedestrian access points, which may vary. It is further unclear whether the summary baseline throughput data presented are based on a single access point, an average from all like access points, or whether the data are from the access points that were actually tested during later phases of the pilot. Further complicating the analysis of baseline data is that there was a TSA version of the baseline report and a separate version produced by the independent test agent, and facts and figures in each do not fully match. Where both documents present summary baseline throughput data for each pilot site, the summary baseline throughput data differ for each pilot site. For example, summary baseline throughput data at one pilot site is reported as 4 minutes and 10 seconds in one version of the report but is reported as 47 seconds in the other report. As a result, the accuracy and reliability of the available baseline data are questionable. Further, according to TSA, where summary throughput data were not included in the baseline report, the independent test agent’s later site reports did contain the data. 5. TSA and the independent test agent did not collect complete data on malfunctioning TWIC cards. TSA officials observed malfunctioning TWICs during the pilot, largely because of broken antennas. The antenna is the piece of technology needed for a contactless reader to communicate with a TWIC. If a TWIC with a broken antenna was presented for a contactless read, the reader would not identify that a TWIC had been presented, as the broken antenna would not communicate TWIC information to a contactless reader. In such instances, the reader would not log that an access attempt had been made and failed. Individuals holding TWICs with bad antennas had presented their TWICs at contactless readers; however, the readers did not document and report each instance that a malfunctioning TWIC was presented. Instead, as noted by pilot participants and confirmed by TSA officials, pilot sites generally conducted visual inspections when confronting a malfunctioning TWIC and granted the TWIC holder access. While in some cases the independent test agent used a card analysis tool to assess malfunctioning TWICs, TSA officials reported that neither they nor the independent test agent documented the overall number of TWICs with broken antennas or other damage. According to TSA officials, the number of TWICs with broken antennas or other damage was not tracked because failed TWIC cards could be tracked only if an evaluator was present, had access to a card analysis tool, and had the cooperation of the pilot participants to hold up a worker’s access long enough to confirm that the problem was the TWIC card and not some other factor. However, it is unclear why TSA was unable to provide a count of TWICs with broken antennas or other damage based on the TWIC cards that were analyzed with the card analysis tool.

While TSA could not provide an accounting of TWICs with broken antennas or other damage experienced during the pilot, pilot participants and other data collected provide additional context and perspective for understanding the nature and extent of TWIC card failure rates during the pilot. Officials at one pilot container facility told us that a 10 percent failure rate would be unacceptable and would slow down cargo operations. However, according to officials from two pilot sites, approximately 70 percent of the TWICs they encountered when testing TWICs against contactless readers had broken antennas or malfunctioned. Further, a separate 2011 report commissioned and led by USCG identified problems with reading TWICs in contactless mode during data collection. This report identified one site where 49 percent of TWICs could not be read in contactless (or proximity) mode, and two other sites where 11 percent and 13 percent of TWICs could not be read in contactless mode. Because TWIC cards malfunctioned, they could not be detected by readers. Accordingly, individuals may have made multiple attempts to get the TWIC reader to read the TWIC card; however, each attempt was not recorded and thus TSA does not have an accurate accounting of the number of attempts or time it may have taken to resolve resulting access issues. Consequently, assessments of the operational impacts of using TWIC with readers using the collected data alone should be interpreted cautiously as they may be based on inaccurate data.

In discussing these failure rates with TSA officials, the officials reported that TSA does not have a record of a pilot participant reporting a 70 percent failure rate. In addition, they believe that the failure rates reported by pilot sites and the separate USCG-commissioned report are imperfect because they did not have the card analysis tool necessary to confirm a failed TWIC card, and instances where a failed TWIC card was presented at a pilot site could be documented only when the independent test agent was present at the site with a card analysis tool. However, a contractor from TSA visited the facility where the USCG report notes that 49 percent of TWICs could not be read in contactless mode and found that 60 out of 110 of TWIC cards checked, or 54.5 percent, would not work in contactless mode. TSA officials agreed that TWIC card failure rates were higher than anticipated and stated that TSA is continuing to assess TWIC card failures to identify the root cause of the failures and correct for them. TSA is also looking to test the TWIC cards at facilities that have not previously used TWIC readers to get a better sense of how inoperable TWIC cards might affect a facility operationally. 6. Pilot participants did not document instances of denied access. Incomplete data resulted from challenges documenting how to manage individuals with a denied TWIC across pilot sites. The independent test agent reported that facility security personnel were unclear on how to process people who are denied access by a TWIC reader because of a biometric mismatch or other TWIC card issue. In these cases, pilot site officials would need to receive input from USCG as to whether to grant or deny access to an individual presenting a TWIC card that had been denied. According to TSA officials, during the pilot, if a TWIC reader denied access to a TWIC, the facility could visually inspect the TWIC, as allowed under current regulation, and grant the individual access. However, TSA and the independent test agent did not require pilot participants to document when individuals were granted access based on a visual inspection of the TWIC, or deny the individual access as may be required under future regulation. This is contrary to the TWIC reader pilot test and evaluation master plan, which calls for documenting the number of entrants “rejected” with the TWIC card reader system operational as part of assessing the economic impact. Without such documentation, the pilot sites were not completely measuring the operational impact of using TWIC with readers. 7. TSA and the independent test agent did not collect consistent data on the operational impact of using TWIC cards with readers. TWIC reader pilot testing scenarios included having each individual present his or her TWIC for verification; however, it is unclear whether this actually occurred in practice. For example, at one pilot site, the independent test agent did not require each individual to have his or her TWIC checked during throughput data collection.site noted that during testing, approximately 1 in 10 individuals was required to have his or her TWIC checked while entering the facility because of concerns about causing a traffic backup. They said that this approach was used because pilot site officials believed that reading each TWIC would have caused significant congestion. However, the report for the pilot site does not note this selective use of the TWIC card. In addition, officials from another pilot site reported that truck drivers could elect to go to other lanes that were not being monitored during throughput time collection. Officials at this pilot site noted that truck drivers, observing congestion in lanes where throughput time was being collected, used other lanes to avoid delays. This was especially the case when the tested truck lane was blocked to troubleshoot TWIC card and reader problems. However, the pilot site report did not record congestion issues or the avoidance of congestion issues by allowing truck drivers to use alternative lanes where TWIC readers were not being tested. TSA officials also noted that another pilot site would allow trucks entry without using a TWIC reader on an ad hoc basis during the pilot to prevent congestion, making it difficult to consistently acquire the data needed to Officials at the pilot accurately assess the operational impacts, such as the truck congestion resulting from TWIC cards with readers. Despite the noted deviations in test protocols, the reports for these pilot sites do not note that these deviations occurred.

In commenting on this issue, TSA officials noted that these deviations occurred most frequently at those facilities with multiple truck or pedestrian access points where readers were installed at a few access points. Most commonly these facilities were large container terminals. Because of the voluntary nature of the pilot, TSA elected to primarily use reader performance data from facilities that did not install and use readers at all access points. TSA officials further noted that the impact of readers on operations at these facilities necessarily was discounted in the final report to Congress. However, pilot documentation shows that container terminals held the largest population of individuals potentially requiring the use of a TWIC. Noting deviations such as those described above in each pilot site report would have provided important perspective by identifying the limitations of the data collected at the pilot site and providing context when comparing the pilot site data with data from other pilot sites. Further, identifying the presence of such deviations could have helped the independent test agent and TSA recognize the limitations of the data when using them to develop and support conclusions for the pilot report on the business and operational impact of using TWICs with readers. 8. Pilot site reports did not contain complete information about installed TWIC readers’ and access control systems’ design. TSA and the independent test agent tested the TWIC readers at each pilot site to ensure they worked before individuals began presenting their TWIC cards to the readers during the pilot. As part of this test, information on how each TWIC reader communicated with TWICs and related access control systems was to be documented. In accordance with TWIC test plans, this testing was to specify, among other things, whether the TWIC reader (1) was contactless or required contact with a TWIC, (2) communicated with a facility’s physical access control system(s) through a wired or wireless conduit, or (3) granted or denied access to a TWIC holder itself or relied on a centralized access system to make that determination. However, the data gathered during the testing were incomplete. For example, 10 of 15 sites tested readers for which no record of system design characteristics were recorded.reader information was identified for 4 pilot sites but did not identify the specific readers or associated software tested. Further, 1 pilot site report included reader information for another pilot site and none for its own. This limited TSA’s ability to assess performance results by various reader and access control system characteristics. The absence of this information is particularly important, as it was the only source of data recorded at pilot sites where reader and operational throughput performance could be assessed at a level of granularity that would allow for the consideration of the array of reader, system design, and entry process characteristics. According to TSA officials, collecting these data was the independent test agent’s responsibility, but the independent test agent did not record and provide all required data. The independent test agent maintains that the data are present. However, we reviewed the documentation, and we did not find the data.

As we have previously reported, the basic components of an evaluation design include identifying information sources and measures, data collection methods, and an assessment of study limitations, among other We further reported that care should be taken to ensure that things. collected data are sufficient and appropriate, and that measures are incorporated into data collection to ensure that data are accurate and reliable. Data may not be sufficiently reliable if (1) significant errors or incompleteness exists in some of or all the key data elements, and (2) using the data would probably lead to an incorrect or unintentional message. Moreover, in accordance with Standards for Internal Control in the Federal Government, controls are to be designed to help ensure the accurate and timely recording of transactions and events. Properly implemented control activities help to ensure that all transactions are completely and accurately recorded. Having measures in place to ensure collected data are complete, are not subject to inappropriate alteration, and are collected in a consistent manner helps ensure that data are accurate and reliable. However, as discussed in the examples above, TSA and the independent test agent did not take the steps needed to ensure the completeness, accuracy, and reliability of TWIC reader data collected at pilot sites, and the pilot lacked effective mechanisms for ensuring that transactions were completely and consistently recorded.

According to TSA, a variety of challenges prevented TSA and the independent test agent from collecting pilot data in a complete and consistent fashion. Among the challenges noted by TSA, (1) pilot participation was voluntary, which allowed pilot sites to stop participation at any time or not adhere to established testing and data collection protocols; (2) the independent test agent did not correctly and completely collect and record pilot data; (3) systems in place during the pilot did not record all required data, including information on failed TWIC card reads and the reasons for the failure; and (4) prior to pilot testing, officials did not expect to confront problems with nonfunctioning TWIC cards. Additionally, TSA noted that it lacked the authority to compel pilot sites to collect data in a way that would have been in compliance with federal standards. In addition to these challenges, the independent test agent identified the lack of a database to track and analyze all pilot data in a consistent manner as an additional challenge to data collection and reporting. The independent test agent, however, noted that all data collection plans and resulting data representation were ultimately approved by TSA and USCG. However, our review of pilot test results shows that because the resulting pilot data are incomplete, inaccurate, and unreliable, they should not be used to help inform the card reader rule. While TSA’s stated challenges may have hindered TWIC reader pilot efforts, planning and management shortfalls also resulted in TWIC reader pilot data being incomplete, inaccurate, and unreliable. The challenges TSA and the independent test agent confronted during the pilot limited their data collection efforts, which were a critical piece of the assessment of the technology and operational impacts of using TWIC at pilot sites that were to be representative of actual deployment conditions.

Issues with DHS’s Congressional Report on the Pilot and the Validity of the TWIC Security Premise Raise Concerns about the Effectiveness of the TWIC Program

DHS’s Report to Congress Presented Findings and Lessons Learned That Were Not Always Supported by the Collected Data

As required by the SAFE Port Act and the Coast Guard Authorization Act of 2010, DHS’s report to Congress on the TWIC reader pilot presented several findings with respect to technical and operational aspects of implementing TWIC technologies in the maritime environment. DHS reported the following, among other findings: 1. Despite facing a number of challenges, the TWIC reader pilot obtained sufficient data to evaluate reader performance and assess the impact of using readers at ports and maritime facilities. 2. A biometric match may take longer than a visual inspection alone but not long enough to cause access point throughput delays that would negatively impact business operations. 3. When designed, installed, and operated in manners consistent with the business considerations of the facility or vessel operation, TWIC readers provide an additional layer of security by reducing the risk that an unauthorized individual could gain access to a secure area.

In addition, the report noted a number of lessons learned. For example, TWIC cards were found to be sensitive to wet conditions, and users experienced difficulty reading messages on the screens of readers not shielded from direct sunlight, which prevented users from determining the cause of access denial, among other things. According to officials from TSA and DHS’s Screening Coordination Office, many of these lessons learned did not require a pilot in order to be identified, but the pilot did make a positive contribution by helping to validate these lessons learned. Additionally, officials from DHS’s Screening Coordination Office noted that they believe that the report to Congress included a comprehensive listing of the extent to which established metrics were achieved during the pilot program, as required by the Coast Guard Authorization Act of 2010.

However, according to our review, the findings and lessons learned in DHS’s report to Congress were based on incomplete or unreliable data, and thus should not be used to inform the development of the future regulation on the use of TWIC with readers. Specifically, incomplete TWIC cost data and unreliable access point throughput time data result in an inaccurate description of the impact of TWIC on MTSA-regulated facilities and vessels. Further, data on the security benefits of TWIC were not collected as part of the pilot and therefore the statements made in DHS’s report to Congress are not supported by the pilot data.

DHS’s report identified costs for implementing TWIC readers during the pilot. However, the costs reported by DHS do not represent the full costs of operating and maintaining TWIC readers and related systems within a particular site, or the cost of fully implementing TWIC at all sites. First, DHS’s reported costs for implementing TWIC with readers during the pilot did not consistently reflect the costs of implementing TWIC at all access points needed for each facility. For example, DHS’s report correctly notes that 2 container facilities did not implement TWIC readers at all access points and are therefore not reflective of full implementation. However, on the basis of our analysis and interviews with pilot site officials, at least 5 of the remaining pilot sites would need to make additional investments in readers, totaling 7 pilot sites requiring investments beyond reported expenditures. For example, officials at 2 pilot sites told us that they would need to invest in and install additional readers if reader use was required by regulation. Officials at 3 pilot sites told us that their investment in TWIC readers during pilot testing was not representative of how they would invest in TWIC if regulation required that an individual’s TWIC be checked with a reader at each entry. Second, we found that reported implementation costs did not match TSA’s supporting documentation for 4 of 17 pilot sites. TSA told us that this discrepancy may be due to having multiple versions of cost data available and relying on different cost documents when compiling the cost data in the DHS report to Congress. The lack of complete and accurate cost data limits the usefulness of the information provided to Congress and does not help inform the development of the future regulation on the use of TWIC with readers.

In addition, DHS reported that facilities and vessels that cease issuing site-specific badges and instead use the TWIC card as the only identification needed for access may benefit financially by reducing card management operational costs associated with identity vetting, card inventory, printing equipment, and issuance infrastructure. However, according to TSA, data in support of this finding are based on the statement of one pilot participant who anticipated utilizing the TWIC and not issuing facility badges for access control. Further, DHS’s Screening Coordination Office officials noted that the proximity and bar code cards that facilities currently use do not contain the same level of security features that the TWIC card does. However, a related March 2011 study on the use of TWIC with readers commissioned and led by USCG noted that there are significant reliability problems with using TWIC cards, which The report further cost $60 each to replace, in the contactless mode. notes that off-the-shelf industry standard proximity and bar code cards are already inexpensively produced and managed at various facilities; are considered much more functionally reliable than the TWIC; and provide better overall security, since the cards and associated access control systems—such as readers and centralized databases—are less prone to failure.

Systems Planning and Analysis, Inc. Survey of Physical Access Control System Architectures, Functionality, Associated Components, and Cost Estimates: Prepared for the U.S. Coast Guard Office of Standards Evaluation and Development (CG-523), (Alexandria, Virginia: March 31, 2011). of readers on business operations.comparisons presented in DHS’s report were not throughput times gathered at pilot sites, but reader response times gathered during laboratory testing. The differences between throughput time and reader response times can vary significantly. For example, as recorded during the pilot, throughput time at a facility using a TWIC card reader was 1 minute and 36 seconds, whereas reader response time at the facility was 11 seconds. As noted by DHS, throughput time accounts for conditions at a particular facility or access point, including individual processes. In addition, measuring throughput time with TWIC readers and related systems can also capture variances due to system connectivity (e.g., hardwired or wireless connections), installed readers and interfaces, weather, and integration with access control or other business-related systems—all representative of real-world experiences at a given location or type of access point.

However, the times and In contrast, reader response time, as reported by DHS, measures the amount of time a TWIC reader takes to determine whether a TWIC is valid in controlled laboratory settings. Measuring reader response time alone is valuable, as it can help a site determine what amount of increase or decrease in throughput time may be due to TWIC systems alone rather than business processes. However, DHS’s reporting of reader response time data was not based on a specific pilot site or group of sites. Instead, it was based on lab testing, which is not representative of the technology challenges sites may face in practice, such as time lags due to the distance between a reader and supporting computing system, types of infrastructure available to implement the TWIC system, or the various variables that could delay actual transaction times. Accordingly, DHS’s reporting of reader response time is not an effective measure of response time in a real-world environment and therefore is not an accurate representation of response times that might be experienced at maritime ports and facilities.

DHS’s report to Congress stated that “when designed, installed, and operated in manners consistent with the business considerations of the facility or vessel operations, TWIC readers provide an additional layer of security by reducing the risk that an unauthorized individual could gain access to a secure area.” Further, in a written statement by DHS officials presented before Congress on June 28, 2012, DHS officials stated that TWIC enhances port facility and vessel security and that the pilot operation also highlighted security and operational benefits associated with readers, including the automation of access control, so that regular users could use their TWICs for quick and easy processing into a port.

However, USCG told us that assessment of security benefits was outside the scope of the TWIC reader pilot. Further, TSA confirmed that data regarding the security enhancements provided by TWIC were not collected during the pilot because that was neither the goal nor the legislative mandate of the TWIC reader pilot. Such data might include, for example, data on the number of people turned away at pilot access points for security infractions, information from covert testing at pilot sites, or other types of data to show enhanced security resulting from the implementation of TWIC.

Systems Planning and Analysis, Inc. Survey of Physical Access Control System Architectures, Functionality, Associated Components, and Cost Estimates: Prepared for the U.S. Coast Guard Office of Standards Evaluation and Development (CG-523), (Alexandria, Virginia: March 31, 2011). security may be realized by allowing facilities and vessels to use a combination of traditional access control systems with the TSA background check, also known as a security threat assessment.

The findings of the study commissioned by USCG and the findings of our prior reviews of TSA’s efforts to demonstrate the validity and security benefits of the TWIC program, coupled with the cost of expanding the program to include the installation of TWIC readers at ports throughout the country, raise significant concerns about the program’s premise and effectiveness. While MTSA required the Secretary of Homeland Security to issue biometric transportation security cards to individuals for unescorted entry to secure areas of vessels or facilities, TSA did consider other models for implementing the TWIC program and enhancing security. However, we have found that key reasons for electing to proceed with a government-issued TWIC card have not been validated in practice. Specifically, in February 2005, TSA completed an analysis of alternatives that identified two viable models for implementing TWIC in accordance with MTSA requirements and worthy of additional consideration: (1) a federally managed option wherein the federal government would issue a credential and manage all aspects of the credentialing program except for making access control decisions at entry points to regulated operations, and (2) a federally regulated, decentralized option with a more limited federal role in which the federal government would conduct background checks and MTSA-regulated entities would be responsible for all other aspects of enrolling individuals and implementing a credential system that would comply with federal regulations.

The analysis of alternatives concluded that the federally managed option would best meet security needs and stated mission needs, including ensuring that (1) unauthorized individuals would be denied access to secure areas of the nation’s transportation system and (2) individuals failing to maintain their eligibility requirements would have their access permissions revoked, among others. In part, these conclusions were based on the premise that the federally managed TWIC option would first establish and verify an individual’s claimed identity; and once the individual’s identity has been verified, it would be checked against threat and background check information prior to issuing a TWIC; and once a TWIC was issued, cardholder eligibility would continue to be checked. However, in May 2011, we reported that the TWIC program was not meeting its four program goals, or mission needs, because of internal control weaknesses. Among other things, we reported that internal controls in the enrollment and background checking processes were not designed to provide reasonable assurance that only qualified individuals could acquire TWICs, or once issued TWICs, TWIC holders have maintained their eligibility.

In August 2005, TSA completed an additional analysis comparing the potential costs and benefits of the two alternatives, concluding that the federally managed solution was the most economical choice because the potential benefits outweigh the costs. As noted in the analysis, reasons for selecting the federally managed approach included assumptions such as the following:

The lack of a common credential across the industry could leave facilities open to a security breach with falsified credentials.

Under the decentralized federally regulated solution, each facility would have to perform its own background checks instead of leveraging a federal background check or security threat assessment.

The federally managed solution would eliminate security weaknesses in existing identification systems by, among other things, having built- in security features such as sponsorship from a trusted individual or company.

Transportation Security Administration. Transportation Worker Identification Credential (TWIC) Program Cost Benefit Analysis, Version 1.0. August 31, 2005. not include an assessment of each alternative’s technological maturity and readiness to be used as a security measure at MTSA-regulated entities without impeding commerce. However, as the TWIC reader pilot and the study commissioned by USCG demonstrate, TWIC cards and readers are not operating as envisioned.

Moreover, our reviews of the TWIC program using the federally managed option over several years, as well as other credentialing models used at airports and federal agencies, raise questions about the validity of the assumptions TSA made at the inception of the program. For example, in the airport credentialing model, the organization granting access to an individual leveraged the existing federal process for conducting background checks, and there is no requirement for a single federal security credential. The federal government is also able to recover some of the costs of the program through user fees as it is under other credentialing and endorsement models such as the Hazardous Materials endorsement for truck drivers, where applicants pay $89.25 to have their TSA security threat assessments conducted. American Association of Airport Executives and airport operators argue that maintaining their own site-specific credentials enhances security over a standard, centrally issued credential such as TWIC and best leverages the combined local and federal knowledge for determining access decisions. Likewise, federal agencies also issue their agency-specific credentials for controlling access. For example, unlike the currently implemented TWIC program, the airport and federal government’s own agency-specific credentialing models intrinsically rely on organizational sponsorship, such as sponsorship by an employer, to help validate an individual’s identity prior to conducting background checks to enhance security. In discussing these issues, TSA officials noted, however, that the statute as currently written requires the Secretary of Homeland Security to issue the biometric credential, and therefore decentralized issuance of the TWIC may be inconsistent with congressional intent.

Furthermore, one of the driving assumptions in the TWIC cost-benefit analysis was that the lack of a common credential across the industry could leave facilities open to a security breach with falsified credentials. However, the validity of this assumption is questionable. As we reported in May 2011, our investigators conducted a small number of covert tests to assess the use of TWIC as a means for controlling access to secure areas of MTSA-regulated facilities. During covert tests of TWIC at several selected ports, our investigators were successful in accessing ports using counterfeit TWICs, authentic TWICs acquired through fraudulent means, and false business cases (i.e., reasons for requesting access). However, our investigators did not gain unescorted access to a port where a secondary port-specific identification was required in addition to the TWIC. The investigators’ possession of TWIC cards provided them with the appearance of legitimacy and facilitated their unescorted entry into secure areas of MTSA-regulated facilities and ports at multiple locations across the country.

We have also reported that DHS had not assessed the effectiveness of TWIC at enhancing security or reducing risk for MTSA-regulated facilities and vessels. currently implemented and planned with readers, is more effective than prior approaches used to limit access to ports and facilities, such as using facility-specific identity credentials with business cases. To determine if the internal control weaknesses identified in our May 2011 report still exist, we conducted limited covert testing in late 2012. Our investigators again acquired an authentic TWIC through fraudulent means and were able to use this card and counterfeit TWIC cards to access areas of ports or port facilities requiring a TWIC for entry at four ports.

GAO-11-657. other things, TWIC pilot findings, USCG’s risk-based approach to categorizing vessels and facilities, and Maritime Security Risk Analysis Model (MSRAM) terrorist scenarios that could potentially be thwarted by using TWIC. However, we noted the following issues in the supporting analysis.

With regard to the TWIC pilot findings, as we previously noted, TSA did not collect data during the TWIC pilot regarding the security enhancements provided by TWIC. According to USCG, assessing security benefits was outside the scope of the TWIC pilot. We therefore cannot assess USCG’s claim in its NPRM that TWIC enhances maritime security.

The purpose of USCG’s analysis for categorizing vessels and facilities into risk categories was to allocate where to place readers, not to assess the effectiveness of TWIC or determine the extent to which, or if, use of TWIC with readers would enhance security, reduce risk, or address a specific threat. Rather, USCG assumed that TWIC would help reduce the risk of a terrorist attack at a maritime facility or vessel based on the security threat assessment, but did not consider whether use of the TWIC might introduce a security risk to MTSA-regulated facilities and vessels, or whether use of TWIC would enhance the security beyond efforts already in place.

USCG’s NPRM lists three MSRAM terrorist scenarios that, according to USCG, are most likely to be mitigated by the use of TWIC readers—truck bomb, terrorist assault team, and passenger/passerby explosives/improvised explosive device. According to USCG, because the function of the TWIC reader is to enhance access control, the deployment of TWIC readers would increase the likelihood of identifying and denying access to an individual attempting nefarious acts. However, USCG’s preliminary analysis notes that the use of TWIC with readers would not stop terrorists from detonating a truck at the perimeter of a facility, attempting to break through the gates or protective barriers at a facility, or obtaining a TWIC card using fraudulent documents as we did through covert means. As confirmed with USCG officials, its models for assessing the benefit of TWIC do not account for these known security weaknesses. Further, USCG’s draft regulatory impact analysis may lead to an overestimate (or mischaracterization) of the avoided consequences of using TWIC with readers. This is because the calculation is based on the use of TWIC with readers thwarting worst-case terrorist security incidents rather than a range of avoided consequence estimates, some of which would be lower than what was presented in the draft regulatory analysis.

While USCG has issued the TWIC-reader NPRM and has asserted benefits to be derived by using TWIC with electronic readers, USCG has not conducted an effectiveness assessment of the TWIC program, as we recommended in 2011; thus, it is unclear whether there will be sufficient time to complete the effectiveness assessment prior to the issuance of the rule. In November 2012, USCG officials reported that they are considering taking steps to assess the effectiveness of TWIC, but noted that given the complexity of the effort, the effectiveness assessment may be better suited for another organization, such as the Department of Homeland Security’s Centers of Excellence, to conduct. We continue to believe that the effectiveness assessment would help inform future requirements for using TWIC with biometric card readers if the study was completed and included as part of the TWIC reader regulatory analysis. Further, given USCG’s leading role in assessing and implementing security programs intended to enhance maritime security, we believe that USCG should continue to be involved in conducting this analysis.

Conclusions

With potentially billions of dollars needed to implement the TWIC program, it is important that DHS provide effective stewardship of taxpayer funds and avoid requiring the maritime industry to invest in a program that may not achieve its stated goals. DHS estimates that implementing the TWIC program could cost the federal government and the private sector a combined total of as much as $3 billion over a 10- year period. This does not include an additional estimated $234.2 million (undiscounted) to implement readers at 570 facilities and vessels that the TWIC reader NPRM currently targets. The TWIC reader pilot, conducted at a cost of approximately $23 million, was intended to test the technology and operational impacts of TWIC cards with readers in the maritime environment. However, as a result of weaknesses in the pilot’s planning, implementation, and reporting, data from the TWIC reader pilot cannot be relied upon to make decisions regarding the TWIC card reader rule or the future deployment of the TWIC program.

Additionally, the TWIC reader pilot report concluded that TWIC cards and readers provide a critical layer of security at our nation’s ports. However, 11 years after initiation, the TWIC program continues to be beset with significant internal control weaknesses and technology issues, and, as highlighted in our prior and ongoing work and a related USCG report, the security benefits of the program have yet to be demonstrated. The weaknesses we have identified suggest that the program as designed may not be able to fulfill the principal rationale for the program— enhancing maritime security. Correcting technological problems with the cards and readers alone will not address the security vulnerabilities identified in our previous work or the USCG reports. The depth and pervasiveness of the TWIC program’s planning and implementation challenges require a reassessment of DHS’s efforts to improve maritime security through the issuance of a U.S. government-sponsored TWIC card and card readers. It is important that this reassessment occur before the additional investment of funds is made to install TWIC readers at the nation’s ports, at considerable taxpayer expense.

Matter for Congressional Consideration

Given that the results of the pilot are unreliable for informing the TWIC card reader rule on the technology and operational impacts of using TWICs with readers, Congress should consider repealing the requirement that the Secretary of Homeland Security promulgate final regulations that require the deployment of card readers that are consistent with the findings of the pilot program. Instead, Congress should require that the Secretary of Homeland Security first complete an assessment that evaluates the effectiveness of using TWIC with readers for enhancing port security, as we recommended in our May 2011 report, and then use the results of this assessment to promulgate a final regulation as appropriate. Given DHS’s challenges in implementing TWIC over the past decade, at a minimum, the assessment should include a comprehensive comparison of alternative credentialing approaches, which might include a more decentralized approach, for achieving TWIC program goals.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS and DOD for review and comment. DHS provided written comments, which are printed in full in appendix V. DHS, as well as DOD, provided technical comments, which we incorporated as appropriate. In commenting on this report, DHS identified concerns with our findings and conclusions related to the use of the TWIC reader pilot results. For example, DHS asserted that the TWIC reader pilot did obtain data in sufficient quantity and quality to support the general findings and conclusions of the TWIC reader pilot report, and that the pilot obtained sufficient data to evaluate reader performance and assess the impact of using readers at maritime facilities. We disagree with this assertion. Specifically, as discussed in our report, and as confirmed by the supplemental technical comments provided by DHS, the pilot test’s results were incomplete, inaccurate, and unreliable for informing Congress and for developing a regulation about the readers. For example, as discussed in the report: Installed TWIC readers and access control systems could not collect required data, including reasons for errors, on TWIC reader use, and TSA and the independent test agent did not employ effective compensating data collection measures, such as manually recording reasons for errors in reading TWICs.

TSA and the independent test agent did not record clear baseline data for comparing operational performance at access points with TWIC readers.

TSA and the independent test agent did not collect complete data on malfunctioning TWIC cards.

Moreover, in its written comments, DHS confirmed that the voluntary nature of the pilot limited opportunities for random selection of pilot sites, as we noted in our report. Therefore, the results of the pilot cannot be generalized beyond the 17 sites participating in the pilot. Further, according to DHS, we asserted that the pilot data should have been assessed using the same data collection and reporting methods for “determining the reliability of computer-processed data.” We recognize that the voluntary nature of the pilot posed challenges to the department; however, we evaluated the TWIC pilot data against recognized federal guidance for designing evaluations, and Standards for Internal Control in the Federal Government in addition to assessing the reliability of computer-processed data.

Because of the significant issues we identified in this report concerning the reliability of the data collected during the pilot, when we sent the draft report to DHS for comment, we recommended that DHS not use the results collected at pilot sites on the operational impacts of using TWIC with readers to inform the upcoming TWIC card reader rule or the future deployment of the TWIC program. However, subsequent to sending the draft to DHS for comment, on March 22, 2013, USCG published the TWIC card reader NPRM, which included results from the TWIC card reader pilot. We subsequently removed the recommendation from the report, given that USCG moved forward with issuing the NPRM and incorporated the pilot results. DHS asserted that some of the perceived data anomalies we cited are not significant to the conclusions TSA reached during the pilot and that the pilot report was only one of multiple sources of information available to USCG in drafting the TWIC reader NPRM. We recognize that USCG had multiple sources of information available to it when drafting the proposed rule; however, the pilot was used as an important basis for informing the development of the NPRM. Thus, we believe that the NPRM is based on findings and conclusions that are inaccurate, and unreliable for informing Congress and for developing the TWIC Card Reader Rule. In its addendum to its agency comments, DHS provides explanations for some of the weaknesses that we identified in the pilot program. We acknowledge these challenges but believe that they support our conclusion that the results of the pilot program should not be used to inform the card reader rule.

Further, related to the security benefits of the program, in its written comments, DHS maintains that a common credential used across MTSA- regulated facilities and vessels enhances security. DHS further stated that comparing airport access to maritime port access is inappropriate because most airport workers only access one airport, whereas individuals accessing maritime ports and facilities are more likely to access several different facilities. We recognize the value of conducting the security threat assessment for all workers accessing port facilities; however, TSA has not assessed the security benefits, if any, resulting from use of a common credential versus a port-, facility-, or vessel-based credential. Moreover, we continue to believe, as discussed earlier in this report, that the original assumptions that TSA made when it decided to proceed with the use of TWIC as a common credential are questionable. Thus, a comprehensive comparison of alternative credentialing approaches, which could include a more decentralized approach, would provide the necessary assurance that DHS is pursuing the most effective option for enhancing maritime security.

We are sending copies of this report to the Secretaries of Homeland Security and Defense, the Assistant Secretary for the Transportation Security Administration, the Commandant of the United States Coast Guard, and appropriate congressional committees. In addition, this report is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix VI.

Appendix I: Objective, Scope, and Methodology

The Coast Guard Authorization Act of 2010 required that that the Transportation Worker Identification Credential (TWIC) reader pilot report include (1) the findings of the pilot program with respect to key technical and operational aspects of implementing TWIC technologies in the maritime sector; (2) a comprehensive listing of the extent to which established metrics were achieved during the pilot program; and (3) an analysis of the viability of those technologies for use in the maritime environment, including any challenges to implementing those technologies and strategies for mitigating identified challenges. The act further required that we conduct an assessment of the report’s findings and recommendations. To meet this requirement, we addressed the following question: To what extent were the results from the TWIC reader pilot sufficiently complete, accurate, and reliable for informing Congress and the TWIC card reader rule?

To evaluate the extent to which the results from the TWIC reader pilot were sufficiently complete, accurate, and reliable for informing Congress and the TWIC card reader rule, we assessed (1) TWIC reader pilot test planning and preparation activities, (2) pilot implementation and data collection practices, and (3) the findings reported in the Department of Homeland Security’s (DHS) February 2012 report to Congress on the results of the TWIC reader pilot against underlying pilot data.

TWIC Reader Pilot Test Planning and Preparation Activities

To identify and assess TWIC reader pilot test planning and preparation activities, we reviewed our prior reports and testimonies on the TWIC program issued from September 2003 through May 2011, and key documents related to the TWIC reader pilot. We reviewed the following pilot planning and testing documents to understand the pilot’s design and planned approach, and to assess the extent to which pilot test plans were updated and used since our November 2009 report on the subject matter.

TWIC Contactless Biometric Card and Reader Capability Pilot Test, Test and Evaluation Master Plan (TEMP), dated December 2007;

TWIC Pilot Concept of Operations Plan, signed February 19, 2009;

TWIC Pilot Test Reader Usage Scenarios, dated February 2, 2009;

TWIC Initial Technical Test (ITT) Plan, signed March 20, 2009;

TWIC Reader Functional Specification Conformance Test (F-SCT)

Plan, dated March 2009;

Naval Air (NAVAIR) Systems Command’s TWIC Card Reader Environmental and Electrical Test Plan, dated February 28, 2008;

TWIC Reader Environmental Specification Conformance Test (E-

Space and Naval Warfare Systems Command (SPAWAR), Systems SCT) Plan, dated March 23, 2009; Initial Capability Evaluation Scenarios, Version 1.5, dated June 2008; Center (SSC) Atlantic, TWIC Initial Capability Evaluation Test Plan, Draft Version 1.1, dated November 13, 2008;

TWIC Baseline Data Collection Plan, dated January 2009;

TWIC Early Operational Assessment (EOA) Test Plan, signed March 18, 2009; and

TWIC Reader Pilot Program System Test and Evaluation (ST&E) Test Plan, dated February 2010 (signed August 4, 2011).

We further reviewed the TWIC Reader Pilot Program Data Analysis Plan, dated October 2010. The plan was developed in response to our November 2009 recommendations to develop an evaluation plan and data analysis plan to identify pilot data to be collected and associated data collection approaches. We also recommended that the evaluation plan identify areas for which the TWIC reader pilot would not provide the information needed to report to Congress and implement the TWIC card reader rule, and document the compensating information to be collected and an approach for obtaining and evaluating the information obtained through this effort. We assessed the extent to which the TWIC Data Analysis Plan addressed our 2009 recommendations and the extent to which it was used during the pilot. We also reviewed the extent to which two studies commissioned by the U.S. Coast Guard (USCG) addressed our 2009 recommendations.

See GAO, Border Security: Improvements in the Department of State’s Development Process Could Increase the Security of Passport Cards and Border Crossing Cards, GAO-10-589 (Washington, D.C.: June 1, 2010). readiness of readers for use during the pilot. Specifically, we considered TSA’s modified approach for testing and assessing reader readiness prior to use at pilot sites as well as the results of the more detailed environmental and functional reader testing conducted. We further reviewed reader testing plans and results to identify and assess the performance criteria used to determine whether tested readers would severely impact pilot site operations or prevent the collection of useful pilot data.

TWIC Reader Pilot Data Collection Practices

To identify and assess the pilot as implemented, we reviewed relevant legislation, such as the Maritime Transportation Security Act of 2002 (MTSA), amendments to MTSA made by the Security and Accountability For Every Port Act of 2006 (SAFE Port Act), and the Coast Guard Authorization Act of 2010 to inform our review of requirements for TWIC and the TWIC reader pilot specifically. We further reviewed key TWIC reader pilot test documents, such as the TWIC reader pilot test and evaluation master plan and underlying test protocols, and compared planned pilot testing and data collection practices with the methods used to collect and analyze pilot data. In doing so, we reviewed and assessed the following documents where TWIC reader pilot results were recorded.

TWIC Reader Pilot Program Baseline Report, dated December 2010;

TWIC Initial Technical Test Report, dated September 2010;

TWIC Card Reader Environmental Specification Conformance and Evaluation Test, signed March 2, 2010;

TWIC Reader Pilot Program TWIC Early Operational Assessment Summary Report, signed February 6, 2012;

Early operational assessment reports (final reports) provided by TSA and the independent test agent for each of the 17 pilot sites;

TWIC Reader Pilot Program System Test and Evaluation Summary Report, signed February 6, 2012;

Systems test and evaluation reports (ST&E) (final reports) provided by TSA and the independent test agent for each of the 17 pilot sites;

117 pilot site trip reports where on-site observations were recorded against data recorded in final EOA and ST&E reports;

TWIC Reader Pilot Program Data Analysis Plan, dated October 2010;

46 TWIC Program Weekly and Monthly Status Reports provided by the independent test agent; and

TSA’s TWIC Reader Pilot Cost Summary Report by Participant.

We further assessed TWIC reader pilot data collection efforts against established practices for designing evaluations, assessing the reliability of computer-processed data, as well as internal control standards for collecting and maintaining records. To do so, we identified practices in place and assessed whether measures and internal controls were in place to ensure the resulting data were sufficiently complete, accurate, and reliable. We further interviewed officials representing 14 of the 17 participating pilot sites, the independent test agent (SPAWAR) and relevant agency officials that oversaw or contributed to the pilot results at TSA and USCG about pilot testing approaches, results, and challenges.

While information we obtained from the interviews with officials representing 14 of the 17 participating pilot sites may not be generalized across the maritime transportation industry as a whole, because we selected TWIC reader pilot participants located across the nation and representing varying maritime operations, the interviews provided us with information on the views of individuals and organizations that participated in the pilot and could be directly affected by TWIC reader use requirements. We also reviewed pilot site reports and underlying data to assess the extent to which data in these reports were collected and assessed in a consistent and complete manner, so as to ensure the data and the analysis thereof could result in accurate and reliable findings. TSA reported that it relied on each of the final EOA and ST&E reports for each of the 17 pilot sites—a total of 34 reports—as the basis of its report to Congress. Accordingly, we tested the data in each of the 34 reports as follows. 1. We requested that TSA and the independent test agent each provide us with final copies of each pilot site’s EOA and ST&E pilot site reports. We compared the 34 reports provided by TSA with the 34 reports provided by the independent test agent to validate whether the final reports provided by each entity were identical. We also reviewed the 117 pilot site trip reports provided by TSA and the independent test agent. Pilot site trip reports documented observations made by TSA or the independent test agent during visits to each pilot site and were to serve as input to the final EOA and ST&E pilot site reports. Of the 117 pilot site trip reports, 76 contained access point throughput data. We further reviewed 34 of 76 pilot site trip reports to identify the extent to which all collected observations and data were included in the final EOA and ST&E pilot site reports, and to determine if reasons for exclusions, if any, were documented. While information we obtained from our review of the 34 pilot site trip reports compared with the final EOA and ST&E pilot site reports cannot be generalized, the reports provided us with important insight on potential limitations present in reported pilot data. 2. We employed computer-based testing techniques, including the development of a database, to assess the completeness of collected data as well as the consistency of data collected across pilot sites. To do so, we used TWIC reader pilot data results recorded in the TWIC Reader Pilot Program Baseline Report and the 34 final EOA and ST&E pilot site reports. We linked results reported in the baseline report and each pilot site’s EOA or ST&E reports where data were present for a particular pilot site, access point, and reader. These techniques provided us with the following summary and comparative views of collected pilot data, among others, which in part served as the basis of our data analysis: compiled data by pilot site; compiled data on baseline population of users at each pilot site and reported access points; comparison of the total population at baseline to total population reported during the ST&E phase; view of pilot site access point and reader matches across testing results (baseline data, Systems Operational Verification Testing (SOVT) data, EOA data, and ST&E data); view of tested reader and access control system characteristics; comparison of baseline throughput times versus EOA and ST&E throughput times for access points with similar readers used; comparison of data across the pilot to identify trends, if any, in areas such as risk level, facility and vessel type, access point type, access decision location, testing mode throughput and transactions, reader hardware model and software version, reader types (fixed versus portable), interface type (contact versus contactless), communication protocol, whether or not registration was used, the enrollment process, the source of the biometric reference template, and canceled card list input frequency by site; comparison of the total number of access points identified during baseline data collection versus the total of access points tested during the EOA and ST&E phases of the pilot; comparison of the mean, median, and mode based on the ST&E number of throughput transactions; and assessment of testing duration during EOA and ST&E testing phases for both throughput and transaction data collection efforts.

We utilized the results of our above-noted testing techniques and data results recorded in the TWIC Reader Pilot Program Baseline Report and the 34 final EOA and ST&E pilot site reports to inform our analysis of the pilot data’s completeness, reliability, and accuracy. We further reviewed the data with TSA—the agency leading the TWIC reader pilot—and the independent test agent to better understand observed anomalies. We also considered input from pilot site officials regarding the testing operations and officials from USCG who contributed to the TWIC reader pilot or are to utilize the results of the pilot to inform their future implementation of TWIC. Last, we reviewed the two reports commissioned by USCG to inform the impending regulation on the use of TWIC cards with biometric readers in consideration of comparative data.

DHS’s TWIC Reader Pilot Report to Congress

We analyzed and compared the TWIC reader pilot data with DHS’s TWIC reader pilot report submitted to Congress to determine whether the findings identified in the report are based on sufficiently complete, accurate, and reliable evidence, and are supported by pilot documentation. In doing so, we leveraged our above-noted assessments of TWIC reader pilot planning and data collection practices. Since our assessment determined that pilot data on TWIC technology and operational performance at pilot sites were incomplete, inaccurate, or unreliable, we did not further report on differences between TWIC reader pilot data and DHS’s TWIC reader pilot report. We focused the remainder of our assessment on three areas that were not identified in our prior analysis: (1) reported costs and statements about cost savings, (2) reported entry times for accessing pilot sites versus reader response times, and (3) statements of enhanced security resulting from the use of TWIC with biometric readers.

Reported costs and cost savings. We sought to validate the cost data reported in DHS’s TWIC reader pilot report to Congress against cost data provided by TSA and the independent test agent. We reviewed cost data in the report and compared them with the cost schedule provided by TSA that, according to TSA, served as the central cost data document used in support of the data reported to Congress. We further compared the data in the report to Congress against the data held in individual pilot site reports. In addition, we compared the data in TSA’s central cost data document with cost data in each individual EOA and ST&E pilot site report to assess the extent to which cost data in each matched. We reviewed our prior work and received input from seven pilot participants regarding their planned implementation of TWIC readers and related systems. This enabled us to assess the extent to which costs reported in DHS’s report represented likely costs for fully implementing, operating, and maintaining the use of TWIC with readers at these pilot sites. Last, we reviewed available pilot documentation to identify data demonstrating that cost savings had been realized as a result of implementing the use of TWIC with biometric card readers. We further reviewed the results of a report commissioned by the Coast Guard to inform the impending regulation on the use of TWIC cards with biometric readers.

Reported entry time for accessing pilot sites versus reader response time. We reviewed DHS’s TWIC Reader Pilot Program report to Congress to assess the presentation of recorded time measurements. Specifically, we assessed the extent to which the report accurately conveyed entry time for accessing piloted sites, known as throughput time, versus reader response time, known as transaction time. We further assessed the reported time data to identify the extent to which, if at all, throughput time and transaction time data were used interchangeably, could be validated against data from the pilot, and representations made about the data could be validated by data collected during the pilot.

Enhanced security. We reviewed DHS’s TWIC Reader Pilot Program report to Congress and identified statements made about security enhancements based on pilot results. We examined available pilot documentation to identify data demonstrating that security at the piloted sites had been realized as a result of implementing the use of TWIC with biometric card readers. We further discussed the lack of supporting pilot data with TSA and DHS and provided opportunities for data to be provided. We also reviewed statements made by DHS officials during a hearing before Congress on the results of the pilot and the results of a report commissioned by USCG to inform the impending regulation on the use of TWIC cards with biometric readers. We further considered two key documents, the TWIC Program Analysis of Alternatives and the TWIC Program Cost Benefit Analysis, which were used to support the decision to execute the TWIC program to enhance security using common credential and biometric card readers. In doing so, we assessed the information presented in the documents and the operational cost and security benefits defined therein as having significant weight on the decision to implement the TWIC program through the use of a federally issued credential and biometric card readers. We then assessed the defined security benefits against our 2011 review of the TWIC program’s security as implemented and subsequent actions taken by TSA and USCG to address recommendations made in the product. Our investigators also conducted limited covert testing of TWIC program internal controls for acquiring and using TWIC at four maritime ports to update our understanding of the effectiveness of TWIC at enhancing maritime security since our work in May 2011. The information we obtained from covert testing efforts is not generalizable, but we believe that the information from our covert tests provided us with important additional perspective and context on the TWIC program. Finally, we reviewed and assessed the security benefits presented in the TWIC reader notice of proposed rulemaking (NPRM) issued March 22, 2013, to determine whether the effectiveness of the noted security benefits was presented.

We conducted this performance audit from January 2012 to May 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. We conducted our related investigative work in accordance with standards prescribed by the Council of the Inspectors General on Integrity and Efficiency.

Appendix II: Key TWIC Implementation Actions

Table 3 summarizes key TWIC program laws and milestones for implementing the program through November 2012.

Appendix III: TWIC Program Funding

From fiscal year 2002 through fiscal year 2012, the TWIC program had funding authority totaling $393.4 million, including $111.4 million in appropriated funds (including reprogramming and adjustments). An additional $151.3 million has been made available to maritime facility and vessel owners and operators through port and transportation security grants related to TWIC. Table 4 provides further funding details.

As reported by DHS, the TWIC reader pilot cost approximately $23 million and was funded by appropriated funds and federal security grant awards. In issuing the credential rule, DHS estimated that implementing the TWIC program could cost the federal government and the private sector a combined total of between $694.3 million and $3.2 billion over a 10-year period. However, these figures did not include costs associated with implementing and operating readers, as the credential rule did not require the installation or use of TWIC cards with readers. The notice of proposed rulemaking published on March 22, 2013, estimated an additional cost of $234.2 million (undiscounted) to implement readers at 570 facilities and vessels that the TWIC reader currently targets.

Appendix IV: TWIC Reader Pilot Sites, Locations, and Types of Maritime Operation or Industry Group

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, David Bruno (Assistant Director), Joseph P. Cruz (Analyst-in-Charge), David Alexander, Hiwotte Amare, Nabajyoti Barkakati, Chuck Bausell, Justin Fisher, Tracey King, James Lawson, Lara Miklozek, and Anna Maria Ortiz made key contributions to this report.