Transportation Worker Identification Credential: Card Reader Pilot Results Are Unreliable; Security Benefits Should Be Reassessed
Released: 2013-06-18T00:00:00

Letter

TWIC Reader Pilot Results Are Not Sufficiently Complete, Accurate, and Reliable for Informing Congress and the TWIC Card Reader Rule

Our review of the pilot test identified several challenges related to pilot planning, data collection, and reporting, which affected the completeness, accuracy, and reliability of the results.

Pilot Planning

DHS did not correct planning shortfalls that we identified in our November 2009 report. We determined that these weaknesses presented a challenge in ensuring that the pilot would yield information needed to inform Congress and the card reader rule and recommended that DHS components implementing the pilot—TSA and USCG—develop an evaluation plan to guide the remainder of the pilot and identify how it would compensate for areas where the TWIC reader pilot would not provide the information needed. DHS agreed with the recommendations; however, while TSA developed a data analysis plan, TSA and USCG reported that they did not develop an evaluation plan with an evaluation methodology or performance standards, as we recommended. The data analysis plan was a positive step because it identified specific data elements to be captured from the pilot for comparison across pilot sites. If accurate data had been collected, adherence to the data analysis plan could have helped yield valid results. However, TSA and the independent test agent did not utilize the data analysis plan. According to officials from the independent test agent, they started to use the data analysis plan but stopped using the plan because they were experiencing difficulty in collecting the required data and TSA directed them to change the reporting approach. TSA officials stated that they directed the independent test agent to change its collection and reporting approach because of TSA’s inability to require or control data collection to the extent required to execute the plan.

Data Collection

We identified eight areas where TWIC reader pilot data collection, supporting documentation, and recording weaknesses affected the completeness, accuracy, and reliability of the pilot data. 1. Installed TWIC readers and access control systems could not collect required data on TWIC reader use, and TSA and the independent test agent did not employ effective compensating data collection measures. The TWIC reader pilot test and evaluation master plan recognizes that in some cases, readers or related access control systems at pilot sites may not collect the required test data, potentially requiring additional resources, such as on-site personnel, to monitor and log TWIC card reader use issues. Moreover, such instances were to be addressed as part of the test planning. However, the independent test agent reported challenges in sufficiently documenting reader and system errors. For example, the independent test agent reported that the logs from the TWIC readers and related access control systems were not detailed enough to determine the reason for errors, such as biometric match failure, an expired TWIC card, or that the TWIC was identified as being on the list of revoked credentials. The independent test agent further reported that the inability to determine the reason for errors limited its ability to understand why readers were failing, and thus it was unable to determine whether errors encountered were due to TWIC cards, readers, or users, or some combination thereof. 2. Reported transaction data did not match underlying documentation. A total of 34 pilot site reports were issued by the independent test agent. According to TSA, the pilot site reports were used as the basis for DHS’s report to Congress. We separately requested copies of the 34 pilot site reports from both TSA and the independent test agent. In comparing the reports provided, we found that 31 of the 34 pilot site reports provided to us by TSA did not contain the same information as those provided by the independent test agent. Differences for 27 of the 31 pilot site reports pertained to how pilot site data were characterized, such as the baseline throughput time used to compare against throughput times observed during two phases of testing. However, at two pilot sites, Brownsville and Staten Island Ferry, transaction data reported by the independent test agent did not match the data included in TSA’s reports. Moreover, data in the pilot site reports did not always match data collected by the independent test agent during the pilot. 3. Pilot documentation did not contain complete TWIC reader and access control system characteristics. Pilot documentation did not always identify which TWIC readers or which interface (e.g., contact or contactless interface) the reader used to communicate with the TWIC card during data collection. For example, at one pilot site, two different readers were tested. However, the pilot site report did not identify which data were collected using which reader. 4. TSA and the independent test agent did not record clear baseline data for comparing operational performance at access points with TWIC readers. Baseline data, which were to be collected prior to piloting the use of TWIC with readers, were to be a measure of throughput time, that is, the time required to inspect a TWIC card and complete access-related processes prior to granting entry. However, it is unclear from the documentation whether acquired data were sufficient to reliably identify throughput times at truck, other vehicle, and pedestrian access points, which may vary. 5. TSA and the independent test agent did not collect complete data on malfunctioning TWIC cards. TSA officials observed malfunctioning TWIC cards during the pilot, largely because of broken antennas. If a TWIC with a broken antenna was presented for a contactless read, the reader would not identify that a TWIC had been presented, as the broken antenna would not communicate TWIC information to a contactless reader. In such instances, the reader would not log that an access attempt had been made and failed. 6. Pilot participants did not document instances of denied access.

Incomplete data resulted from challenges documenting how to manage individuals with a denied TWIC across pilot sites. Specifically, TSA and the independent test agent did not require pilot participants to document when individuals were granted access based on a visual inspection of the TWIC, or deny the individual access as may be required under future regulation. This is contrary to the TWIC reader pilot test and evaluation master plan, which calls for documenting the number of entrants “rejected” with the TWIC card reader system operational as part of assessing the economic impact. Without such documentation, the pilot sites were not completely measuring the operational impact of using TWIC with readers. 7. TSA and the independent test agent did not collect consistent data on the operational impact of using TWIC cards with readers. TWIC reader pilot testing scenarios included having each individual present his or her TWIC for verification; however, it is unclear whether this actually occurred in practice. For example, at one pilot site, officials noted that during testing, approximately 1 in 10 individuals was required to have his or her TWIC checked while entering the facility because of concerns about causing a traffic backup. Despite noted deviations in test protocols, the reports for these pilot sites do not note that these deviations occurred. Noting deviations in each pilot site report would have provided important perspective by identifying the limitations of the data collected at the pilot site and providing context when comparing the pilot site data with data from other pilot sites. 8. Pilot site records did not contain complete information about installed TWIC readers’ and access control systems’ design. TSA and the independent test agent tested the TWIC readers at each pilot site to ensure they worked before individuals began presenting their TWIC cards to the readers during the pilot. However, the data gathered during the testing were incomplete. For example, 10 of 15 sites tested readers for which no record of system design characteristics were recorded. In addition, pilot reader information was identified for 4 pilot sites but did not identify the specific readers or associated software tested.

According to TSA, a variety of challenges prevented TSA and the independent test agent from collecting pilot data in a complete and consistent fashion. Among the challenges noted by TSA, (1) pilot participation was voluntary, which allowed pilot sites to stop participation at any time or not adhere to established testing and data collection protocols; (2) the independent test agent did not correctly and completely collect and record pilot data; (3) systems in place during the pilot did not record all required data, including information on failed TWIC card reads and the reasons for the failure; and (4) prior to pilot testing, officials did not expect to confront problems with nonfunctioning TWIC cards. Additionally, TSA noted that it lacked the authority to compel pilot sites to collect data in a way that would have been in compliance with federal standards. In addition to these challenges, the independent test agent identified the lack of a database to track and analyze all pilot data in a consistent manner as an additional challenge to data collection and reporting. The independent test agent, however, noted that all data collection plans and resulting data representation were ultimately approved by TSA and USCG.

Reporting

As required by the SAFE Port Act and the Coast Guard Authorization Act of 2010, DHS’s report to Congress on the TWIC reader pilot presented several findings with respect to technical and operational aspects of implementing TWIC technologies in the maritime environment. However, DHS’s reported findings were not always supported by the pilot data, or were based on incomplete or unreliable data, thus limiting the report’s usefulness in informing Congress about the results of the TWIC reader pilot. For example, reported entry times into facilities were not based on data collected at pilot sites as intended. Further, the report concluded that TWIC cards and readers provide a critical layer of port security, but data were not collected to support this conclusion.

Because of the number of concerns that we identified with the TWIC pilot, in our March 13, 2013, draft report to DHS, we recommended that DHS not use the pilot data to inform the upcoming TWIC card reader rule. However, after receiving the draft that we sent to DHS for comment, on March 22, 2013, USCG published the TWIC card reader NPRM, which included results from the TWIC card reader pilot. We subsequently removed the recommendation from our final report, given that USCG had moved forward with issuing the NPRM and had incorporated the pilot results into the proposed rulemaking. In its official comments on our report, DHS asserted that some of the perceived data anomalies we cited were not significant to the conclusions TSA reached during the pilot and that the pilot report was only one of multiple sources of information available to USCG in drafting the TWIC reader NPRM. We recognize that USCG had multiple sources of information available to it when drafting the proposed rule; however, the pilot was used as an important basis for informing the development of the NPRM, and the issues and concerns that we identified remain valid.

Given that the results of the pilot are unreliable for informing the TWIC card reader rule on the technology and operational impacts of using TWIC cards with readers, we recommended that Congress consider repealing the requirement that the Secretary of Homeland Security promulgate final regulations that require the deployment of card readers that are consistent with the findings of the pilot program, and that Congress should consider requiring that the Secretary of Homeland Security complete an assessment that evaluates the effectiveness of using TWIC with readers for enhancing port security. This would be consistent with the recommendation that we made in our May 2011 report. These results could then be used to promulgate a final regulation as appropriate. Given DHS’s challenges in implementing TWIC over the past decade, at a minimum, the assessment should include a comprehensive comparison of alternative credentialing approaches, which might include a more decentralized approach, for achieving TWIC program goals.

Chairman Miller, Ranking Member Jackson-Lee, and members of the subcommittee, this concludes my prepared statement. I would be happy to respond to any questions that you may have.

GAO Contact and Staff Acknowledgments

For questions about this statement, please contact Steve Lord at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Dave Bruno, Assistant Director; Joseph P. Cruz; and James Lawson. Key contributors for the previous work that this testimony is based on are listed within each individual product.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.