Transportation Worker Identification Credential: Progress Made in Enrolling Workers and Activating Credentials but Evaluation Plan Needed to Help Inform the Implementation of Card Readers
Published: 2009-11-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Transportation Worker Identification Credential (TWIC) program, which is managed by the Department of Homeland Security's (DHS) Transportation Security Administration (TSA) and the U.S. Coast Guard, requires maritime workers who access secure areas of transportation facilities to obtain a biometric identification card to access these facilities. A federal regulation set a national compliance deadline of April 15, 2009. TSA is conducting a pilot program to test the use of TWICs with biometric card readers in part to inform the development of a second TWIC regulation. The Government Accountability Office (GAO) was asked to evaluate TSA's and the Coast Guard's progress and related challenges in implementing TWIC, and to evaluate the management challenges, if any, TSA, Coast Guard, and DHS face in executing the TWIC pilot test. GAO reviewed TWIC enrollment and implementation documents and conducted site visits or interviewed officials at the seven pilot program sites.
section_title: What GAO Found
paragraphs: TSA, Coast Guard, and the maritime industry took a number of steps to enroll 1,121,461 workers in the TWIC program, or over 93 percent of the estimated 1.2 million users, by the April 15, 2009, national compliance deadline, but experienced challenges that resulted in delays. TSA and the Coast Guard implemented a staggered compliance approach whereby each of 42 regions impacted by TWIC were required to meet TWIC compliance prior to the national compliance date. Further, based on lessons learned from its early experiences with enrollment and activation, and to prepare for an expected surge in TWIC enrollments and activations as compliance dates approached, TSA and its contractor increased the number of stations available for TWIC enrollment and activation. While 93 percent of users were enrolled in TWIC by the compliance date, TSA data shows that some workers experienced delays in receiving TWICs. Among reasons for the delays, a power failure in October 2008 occurred at the government facility that processes TWIC data. The power failure resulted in credential activations being halted until late November 2008, and the inability to set new personal identification numbers (PIN) on 410,000 TWICs issued prior to the power failure. While TSA officials stated that they are taking steps to develop a disaster recovery plan by next year and a system to support disaster recovery by 2012, until such a plan and system(s) are put in place, TWIC systems remain vulnerable to similar disasters. While the full cost of this power failure is unknown, based on TSA provided figures, it could cost the government and industry up to approximately $26 million to replace all affected TWIC cards. While TSA has made progress in incorporating management best practices to execute the TWIC pilot, TSA faces two management challenges in ensuring the successful execution of the pilot test aimed at informing Congress and the development of the second TWIC regulation. First, TSA has faced challenges in using the TWIC pilot schedule to guide the pilot and accurately identify the pilot's completion date. TSA has improved its scheduling practices in executing the pilot, but weaknesses remain, such as not capturing all pilot activities in the schedule, that may adversely impact the schedule's usefulness as a management tool and for communicating with pilot participants in the maritime industry. Second, shortfalls in TWIC pilot planning have hindered TSA and Coast Guard's efforts to ensure that the pilot is broadly representative of deployment conditions and will yield the information needed--such as information on the operational impacts of deploying biometric card readers and their costs--to accurately inform Congress and the second rule. This is in part because these agencies have not developed an evaluation plan that fully identifies the scope of the pilot and specifies how the information from the pilot will be analyzed. The current evaluation plans describe data collection methods but do not identify the evaluation criteria and methodology to be used in analyzing the pilot data once collected. A well-developed, sound evaluation plan would help TSA and the Coast Guard determine how the data are to be analyzed to measure the project's performance.

section_title: Letter
paragraphs: It is estimated that over 1 million workers, including longshoremen,  mechanics, truck drivers, and merchant mariners, access secure areas of  the nation’s estimated 4,000 maritime-related transportation facilities, such  as cargo container and cruise ship terminals, each day while performing  their jobs. Securing transportation systems and facilities requires  balancing security to address potential threats while facilitating the flow of  people and goods that are critical to the U.S. economy and necessary for  supporting international commerce. As we have previously reported, these  systems and facilities are vulnerable and difficult to secure given their size,  easy accessibility, large number of potential targets, and proximity to  urban areas.
Within the Department of Homeland Security (DHS), the Transportation  Security Administration (TSA) and the U.S. Coast Guard manage the  Transportation Worker Identification Credential (TWIC) program. The  TWIC program aims to protect the nation’s maritime transportation  facilities and vessels by requiring maritime workers to complete  background checks and obtain a biometric identification card in order to  gain unescorted access to the secure areas of these facilities and vessels.  Key aspects of the program include collecting biographic and biometric  information, such as fingerprints, to validate workers’ identities;  conducting background checks to ensure that workers do not pose a  security threat; and issuing tamper-resistant, biometric credentials for use  in granting workers unescorted access to secure areas.
TSA’s responsibilities include enrolling TWIC applicants, conducting  security threat assessments, and processing workers’ appeals to adverse  TWIC qualification decisions. The Coast Guard is responsible for  developing TWIC-related security regulations and ensuring that maritime  facilities and vessels are in compliance with these regulations. A federal  regulation (known as the credential rule) in January 2007 set a compliance  deadline, subsequently extended to April 15, 2009, whereby each maritime  worker was required to hold a TWIC in order to obtain unescorted access  to secure areas of Maritime Transportation Security Act of 2002 (MTSA)  regulated facilities and vessels. TSA and Coast Guard estimated that  approximately 1.2 million workers would use TWICs to access secure  areas of maritime vessels and facilities. In August 2008, a pilot was  initiated to test the use of TWICs with biometric card readers for granting  access to maritime facilities and vessels, and to inform the development of  the card reader rule (regulation) related to the use of these readers. In  September 2008, we reported that TSA, Coast Guard, and maritime  industry stakeholders have faced challenges in implementing the TWIC  program, including enrolling and issuing TWICs to a larger population than  was originally anticipated, ensuring that TWIC access control technologies  perform effectively in the harsh maritime environment, and balancing  security requirements with the flow of maritime commerce.
In response to your request, we evaluated TSA and Coast Guard’s overall  progress in implementing the TWIC program and addressed the following  questions: (1) To what extent did TSA, the Coast Guard, and the maritime  industry take steps to meet the TWIC compliance date and address related  challenges? and (2) What management challenges, if any, do TSA, Coast  Guard, and DHS face in executing the TWIC pilot test for informing  Congress and the card reader rule?
To identify the steps taken by TSA, the Coast Guard, and the maritime  industry to meet the April 15, 2009, TWIC compliance date, and address  related challenges, we reviewed program documentation on the status of  TWIC enrollment and activation as well as implementation efforts from  both TSA and the Coast Guard. Among other things, this documentation  includes compliance reports compiled by the Coast Guard from facility- gathered information, TSA’s TWIC communication plan for disseminating  information about the TWIC enrollment process and compliance  deadlines, and program management reviews on TWIC enrollment,  activation, and issuance. We analyzed pertinent information including key  statutes, such as MTSA, as amended by the Security and Accountability  For Every (SAFE) Port Act of 2006, and related regulations, policies, and  guidance setting out requirements for the TWIC program. We also  reviewed maritime industry documents, such as TWIC Stakeholder  Communication Committee meeting minutes and reports by the National  Maritime Security Advisory Committee, an advisory council to DHS. We  met with nine associations, the members of which are affected by the  implementation of TWIC, such as the American Association of Port  Authorities—a trade association that represents more than 160 public port  organizations—and the Independent Liquid Terminals Association—a  trade association representing companies with bulk liquid terminals and  above ground storage tank facilities. We also visited four TWIC enrollment  and activation centers and visited and met with officials of facilities and  vessels affected by TWIC across the country. While information we  obtained from these interviews and site visits may not be generalized  across the maritime transportation industry as a whole, because the  facilities, vessels, and enrollment centers we selected are representative of  high and low volume entities in the maritime industry and the enrollment  centers are representative of areas with high population density, the  locations we visited provided us with an overview of the general progress  of the TWIC program, as well as any potential implementation challenges  faced by MTSA-regulated facilities/vessels, transportation workers, and  mariners. Lastly, we interviewed TWIC program officials from the Coast  Guard and TSA—including the TWIC Program Director—regarding their  efforts to implement the TWIC program. To assess the extent to which  TSA planned for the potential failure of information technology systems  supporting the TWIC program in order to minimize the effects of potential  TWIC system failures, we reviewed TWIC program management reviews  and conducted interviews with TWIC program staff. We compared TSA’s  efforts with internal control standards and industry best practices for  contingency planning.
To identify and assess the management challenges TSA, the Coast Guard,  and DHS face in executing the TWIC pilot test for informing Congress and  the card reader rule, we reviewed prior GAO reports and testimonies on  the TWIC program issued from December 2004 through September 2008,  key documents related to the TWIC reader pilot, such as the TWIC Pilot  Test and Evaluation Master Plan, the Initial Technical Test Plan, and the  Early Operational Assessment Test Plan. We also reviewed relevant  legislation, such as the MTSA, as amended by the SAFE Port Act. In  addition, we met with various officials at DHS and the Department of  Defense to better understand stakeholder contributions and testing  approaches related to the TWIC pilot. To further inform our review, we  conducted site visits or interviews with officials at each of the seven TWIC  pilot sites, and met with local Coast Guard officials and representatives  from 15 stakeholder organizations, including associations and business  owners from industries impacted by TWIC, such as longshoremen and  truck drivers. While information we obtained from the interviews with  stakeholders may not be generalized across the maritime transportation  industry as a whole, because we selected stakeholders who either  represent national associations or who operate in or access the ports  where the TWIC reader pilot will be conducted, the interviews provided us  with information on the views of individuals and organizations that will be  directly impacted by the program. In assessing the TWIC pilot approach,  we reviewed the information obtained through these endeavors against  best practices we identified in program and project management as well as  program evaluation efforts that are relevant to the TWIC program pilot.  These practices were identified based on a review of (1) guidance issued  by the Office of Management and Budget (OMB); (2) our prior work on  results oriented government, program management and evaluation, and  regulatory analysis; and (3) literature on program management  principles. We also assessed the pilot schedule against relevant best  practices in our Cost Estimating and Assessment Guide to determine the  extent to which it reflects key estimating practices that are fundamental to  having a reliable schedule. In addition, we compared the TWIC Reader  Advanced Notice of Proposed Rulemaking (ANPRM) issued on March 27,  2009, for the card reader rule on using TWICs with biometric card readers  to the pilot’s test documentation to assess whether the pilot test is  considering the proposed characteristics contained in the ANPRM.  Appendix I contains more detailed information regarding our scope and  methodology.
We conducted this performance audit from July 2008 through November  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: TWIC Program History
paragraphs: The TWIC program was established in response to several pieces of  legislation and subsequent programming decisions. In November 2001, the  Aviation and Transportation Security Act (ATSA) was enacted, which  included a provision that requires TSA to work with airport operators to  strengthen access controls to secure areas, and to consider using  biometric access control systems, or similar technologies, to verify the  identity of individuals who seek to enter a secure airport area. In response  to ATSA, TSA established the TWIC program in December 2001. In  November 2002, MTSA required the Secretary of Homeland Security to  issue a maritime worker identification card that uses biometrics to control  access to secure areas of maritime transportation facilities and vessels.  TSA and Coast Guard decided to implement TWIC initially in the maritime  domain. In addition, the Security and Accountability For Every (SAFE)  Port Act of 2006 amended MTSA to direct the Secretary of Homeland  Security to, among other things, implement the TWIC pilot project.  Appendix II summarizes a number of key activities in the implementation  of the TWIC program.
In August 2006, DHS officials decided, based on significant industry  comment, to implement TWIC through two separate regulations, or rules,  the first of which directs the use of the TWIC as an identification  credential. The card reader rule, currently under development, is expected  to address how the access control technologies, such as biometric card  readers, are to be used for confirming the identity of the TWIC holder  against the biometric information on the TWIC. On March 27, 2009, the  Coast Guard issued an ANPRM for the card reader rule.
From fiscal year 2002 through 2009, the TWIC program had funding  authority totaling $286.9 million. Through fiscal year 2009, $111.5 million  in appropriated funds, including reprogramming and adjustments, has  been provided to TWIC (see table 1). An additional $151.8 million in  funding was authorized in fiscal years 2008 and 2009 through the  collection of TWIC enrollment fees by TSA, and $23.6 million had been  made available to pilot participants from the Federal Emergency  Management Agency (FEMA) grant programs—the Port Security Grant  Program and the Transit Security Grant Program. In addition, industry has  spent approximately $179.9 million to purchase 1,358,066 TWICs as of  September 24, 2009.
section_title: Key Components of the TWIC Program’s Enrollment, Activation, and Issuance Process
paragraphs: The TWIC program includes several key components:    Enrollment: Transportation workers are enrolled by providing  biographic information, such as name, date of birth, and address, and  then photographed and fingerprinted at enrollment centers.
Background checks: TSA conducts background checks on each  worker to ensure that individuals who enroll do not pose a known  security threat. First, TSA conducts a security threat assessment that  may include, for example, checks of terrorism databases or watch lists,  such as TSA’s no-fly list. Second, a Federal Bureau of Investigation  criminal history records check is conducted to determine whether the  worker has any disqualifying criminal offenses. Third, the worker’s  immigration status and prior determinations related to mental capacity  are checked. Workers are to have the opportunity to appeal negative  results of the threat assessment or request a waiver in certain  circumstances.
TWIC production: After TSA determines that a worker has passed the  background check, the worker’s information is provided to a federal  card production facility where the TWIC is personalized with the  worker’s information and sent to the appropriate enrollment center for  activation and issuance for each individual applicant.
Card activation and issuance: A worker is informed when his or her  TWIC is ready and must return to an enrollment center to select a  personal identification number (PIN) and obtain and activate his or her  card. Once a TWIC has been activated and issued, the worker may  present his or her TWIC to security officials when they seek to enter a  secure area, and in the future may use biometric card readers to verify  identity.
Once the card is issued, it is presented at MTSA-regulated facilities and  vessels in order to obtain access to secured areas of these entities. Current  regulation requires that the card at a minimum be presented for visual  inspection.
section_title: TSA Is Conducting a Pilot to Test Key TWIC-Related Access Control Technologies
paragraphs: In response to our 2006 recommendation and a SAFE Port Act  requirement, TSA initiated a pilot in August 2008 known as the TWIC  reader pilot, to test TWIC-related access control technologies. This pilot  is intended to test the technology, business processes, and operational  impacts of deploying TWIC readers at secure areas of the marine  transportation system. As such, the pilot is expected to test the viability of  selected biometric card readers for use in reading TWICs within the  maritime environment. It is also to test the technical aspects of connecting  TWIC readers to access control systems. After the pilot has concluded, the  results of the pilot are expected to inform the development of the card  reader rule requiring the deployment of TWIC readers for use in  controlling access at MTSA-regulated vessels and facilities. Based on the  August 2008 pilot initiation date, the card reader rule is to be issued no  later than 24 months from the initiation of the pilot, or by August 2010, and  a report on the findings of the pilot 4 months prior, or by April 2010.
To conduct the TWIC reader pilot, during the course of our review TSA  was partnering with the maritime industry at four ports as well as three  vessel operations that are receiving federal grant money for TWIC  implementation. The participating grantee pilot sites include the ports of  Los Angeles, California; Long Beach, California; Brownsville, Texas; and  the port authority of New York and New Jersey. In addition, vessel  operation participants include the Staten Island Ferry in Staten Island,  New York; Magnolia Marine Transports in Vicksburg, Mississippi; and  Watermark Cruises in Annapolis, Maryland. Of these seven grant  recipients, the four port grant recipients, with input from TSA and Coast  Guard, have identified locations at the port where the pilot is to be  conducted, such as public berths, facilities, and vessels.
The TWIC reader pilot, as initially planned, was to consist of three  sequential assessments, with the results of each assessment intended to  inform the subsequent ones. Table 2 below highlights key aspects of the  three assessments.
To address possible time constraints related to using the results of the  TWIC pilot to inform the card reader rule, two key changes were made to  the pilot test in 2008. First, TSA and Coast Guard inserted a round of  testing called the Initial Capability Evaluation (ICE) as the first step of the  ITT. The intent of the ICE was to conduct an initial evaluation of readers  and determine each reader’s ability to read a TWIC. Initiated in August  2008, the ICE testing resulted in a list of biometric card readers from  which pilot participants can select a reader for use in the pilot rather than  waiting for the entire ITT to be completed. Further, the ICE list has been  used by TSA and Coast Guard to help select a limited number of readers  for full functional and environmental testing. Second, TSA is no longer  requiring the TWIC reader pilot to be conducted in the sequence  highlighted in table 2. Pilot sites may conduct early operational  assessment and system test and evaluation testing while the initial  technical testing is still under way. Currently, ITT testing by TSA is  underway and pilot sites are concurrently executing Early Operational  Assessment (EOA) testing in varying degrees. Because of the concurrent  test approach, some pilot sites may complete ST&E testing while ITT  testing remains under way.
section_title: TSA, Coast Guard, and the Maritime Industry Implemented a Number of Measures to Facilitate Enrollment, Activation, and Compliance but Implementation Efforts Were Affected by the Lack of Planning for Potential System Failures
paragraphs: TSA, the Coast Guard, and the maritime industry took several steps to  meet the compliance date and address implementation related challenges  in an effort to avoid negatively impacting the flow of commerce, but  experienced challenges in enrolling transportation workers and activating  their TWIC cards. Planning for potential information technology system  failures could have helped address one challenge by minimizing the effect  of a system failure that affected TSA enrollment and activation efforts.  TSA reported enrolling 1,121,461 workers in the TWIC program, or over 93  percent of the estimated 1.2 million users, as of the April 15, 2009,  deadline. Although no major disruptions to port facilities or commerce  occurred, TSA data shows that some workers experienced delays in  receiving TWICs.
subsections: section_title: TSA Took Steps to Prepare for a Surge in TWIC Enrollment and Activations, but Experienced Challenges in Meeting the April 2009 Deadline, Including Those Related to Planning for Potential TWIC System Failures
paragraphs: TSA began enrolling maritime workers in the TWIC program in October  2007 through their network of enrollment centers which grew to 149  centers by September 2008. In September 2008 we reported that TSA had  taken steps to confront the challenge of enrolling and issuing TWICs in a  timely manner to a significantly larger population of workers than was  originally anticipated. For example, according to TSA officials, the TWIC  enrollment systems were tested to ensure that they would work effectively  and be able to handle the full capacity of enrollments during  implementation. To address issues with the TWIC help desk, such as calls  being abandoned and longer-than-expected call wait times, TWIC program  management reported that it worked with its contractor to add additional  resources at the help desk to meet call volume demand. Similarly, to  counter the lack of access or parking at enrollment centers at the Port of  Los Angeles, TSA’s contractor opened an additional enrollment facility  with truck parking access as well as extended operating hours. In addition,  TSA reported that it conducted a contingency analysis in coordination  with the Coast Guard to better identify the size of its target enrollee  population at major ports. For example, in preparation for meeting  enrollment demands at the Port of Houston, TWIC program officials  updated prior estimates of maritime workers requiring TWICs for access  to this port’s facilities. Lastly, TSA embarked in a series of communication  efforts designed to help inform and educate transportation workers about  TWIC requirements and encourage compliance with TWIC. TSA’s TWIC  communications plan outlines a series of efforts, such as the use of fliers,  Web media, and targeted presentations, to inform transportation workers  and MTSA-regulated facility/vessel operators. According to TSA officials,  the resulting communication efforts contributed to the high number of  TWIC enrollments and activations by the April 15, 2009, national  compliance date.
Based on lessons learned from its early experiences with enrollment and  activation, TSA and its contractor took steps to prepare for a surge in  TWIC enrollments and activations as local compliance dates approached.  For example, as identified in TWIC program documentation and by port  facility representatives, TSA and its contractor increased enrollment  center resources, such as increasing the number of trusted agents,  enrollment stations, and activation stations as needed to meet projected  TWIC user demands. TSA and its contractor also utilized mobile  enrollment centers and employed more flexible hours at enrollment  centers in order to accommodate TWIC applicants’ needs. For example, at  two of the nation’s largest ports, the Ports of Los Angeles and Long Beach,  TSA and its contractor opened a facility dedicated entirely to TWIC  activations in addition to providing additional trusted agents and  extending hours of operation at enrollment centers. As a result of these  efforts, TSA reported enrolling 1,121,461 workers in the TWIC program, or  over 93 percent of the estimated 1.2 million users, by the April 15, 2009,  deadline. On this date, the total number of TWIC cards activated and  issued reached 906,956, short of the 1,121,461 million enrollees by 214,505  individuals, or 19 percent. According to TSA officials, TWICs were  available for 129,090, or approximately 60 percent of these individuals, but  had not been picked up by the individual and activated. See figure 1  below for details.
Although no nationwide problem occurred due to TWIC implementation,  surges of activity occurred that challenged TWIC enrollment and  activation efforts at some locations. For example, at the Port of Baltimore,  Coast Guard and port officials stated that, despite multiple  communications with TSA about instituting a self-imposed early  compliance date, TSA and its contractors were not prepared to handle the  increased enrollment demand brought on by the early compliance. As a  result, the local fire marshal visited the enrollment center when the  number of enrollees exceeded the capacity of the center. In response, TSA  and its contractor enhanced its enrollment center operations in  Baltimore—opening an additional enrollment center at a nearby hotel on  the same day—to adapt to the surge in enrollment and activation. In  another case, representatives of the New York maritime industry reported  that the wait time for employees to receive their TWIC cards following  enrollment rose from 6 days to between 6 and 9 weeks as the March 23,  2009, local compliance date approached for Captain of the Port Zone New  York. TWIC users in New York also reported difficulty accessing records  in the online TWIC database designed as a means for facility operators to  verify enrollment in order to grant interim access to employees who had  enrolled in the TWIC program but who had not yet received their cards.  Furthermore, according to Port of Brownsville and local Coast Guard  officials, the lack of resources at the Brownsville enrollment center led to  long lines at the center once the local compliance date neared.  Additionally, the approach used to notify TWIC applicants that their  TWICs were ready for pick-up also proved problematic for Mexican  workers. Port of Brownsville officials noted that in many cases these  workers have no e-mail and, since many are Mexican citizens, most hold a  cell phone with an international phone number (from Mexico). As a result,  according to Port of Brownsville officials, many of these enrollees were  not adequately notified that their TWIC cards had arrived and were ready  for pick-up and activation.
In addition, thousands of TWIC enrollees experienced delays in receiving  their TWICs for varying reasons. According to TSA officials and contractor  reports, reasons for delayed TWIC issuance included, among others, TSA’s  inability to locate enrollment records, problems with information on the  TWIC cards, such as photo quality, problems with the quality of the  manufactured blank cards, and incomplete applicant information required  to complete the security threat assessment. Further, TWIC enrollees also  experienced delays in obtaining a TWIC because they were initially  determined to not be qualified for a TWIC. According to TSA records, as of  July 23, 2009, almost 59,000 TWIC applicants received initial  disqualification letters and over 30,000 of these applicants appealed the  decision questioning the basis for the initial disqualification decision.  Under TSA implementing regulations, an applicant may appeal an initial  determination of threat assessment if the applicant is asserting that he or  she meets the standards for the security threat assessment for which he or  she is applying. Almost 25,000 (approximately 42 percent of those  receiving initial disqualification letters) of the appeals resulted in an  approval upon subsequent review, which suggests that some of these  delays could have been avoided if additional or corrected data had been  available and reviewed during the original application process. In addition,  about 2,300 of the over 4,800 applicants who requested waivers from the  TWIC disqualifying factors were granted them upon subsequent review.
Advocacy groups, such as the National Employment Law Project (Law  Project), have reported that hundreds of individuals experienced delays in  receiving their TWICs and that individuals have been unable to work as a  result of processing delays at TSA. The Law Project has identified at least  485 transportation workers as of June 2009 who requested assistance from  it in requesting appeals or waivers from TSA following an initial  determination of disqualifying offenses based on TSA’s threat assessment.  According to officials at the Law Project, for the TWIC applications on  which they provided assistance and approvals were granted, it took an  average of 213 days between the applicant’s enrollment date and final  approval for a TWIC. Furthermore, Law Project officials noted that  applicants they assisted were out of work for an average of 69 days while  waiting for TWIC approval after their port passed the TWIC compliance  date. However, TSA could not confirm the figures presented by the Law  Project officials because TSA does not track this information in the same  format. For example, if a person is sent a disqualification letter and does  not respond within 60 days, TSA’s system does not continue to track the  enrollee’s file as an open enrollment waiting to be filled. Rather, TSA  closes the file and considers the person to not have passed the threat  assessment. According to agency officials, when an applicant contacts  TSA after the 60-day period passes, TSA routinely reopens their case,  though not required to do so, and handles the application until its  conclusion. These types of cases often take time to resolve. Similarly, for  those situations in which enrollees assert that they never received a  disqualification letter and include it as part of the wait time accounted for,  TSA’s numbers will differ as well because, according to TSA officials, they  have no way to track whether or not enrollees receive these letters.
Finally, a power failure on October 21, 2008, occurred at the TWIC data  center at Annapolis Junction, Maryland—a government facility that  processes TWIC data. The power outage caused a hardware component  failure in the TWIC enrollment and activation system for which no  replacement component was on hand. Consequently, data associated with  individual TWICs could not be accessed or processed. As a result of this  failure, (1) credential activations were halted until late November 2008 and  several TWIC compliance dates originally scheduled for October 31, 2008  were postponed; and (2) the failure affected TSA’s ability to reset the  PINs (i.e., provide users with new PINs) on 410,000 TWIC cards issued  prior to the power failure. Consequently, TSA will have to replace the  cards for cardholders who forget their PINs instead of resetting these  PINs. TSA does not know the full cost implications of the power failure at  the data center because it is unknown how many of the 410,000 TWIC  cards will need to be replaced. Moreover, TSA cannot determine how  many of the TWIC cards need to be replaced until all uses for PINs are  identified at facilities across the country. For example, one use that will  affect the number of TWICs TSA will need to replace is dependant on the  number of MTSA-regulated facilities and vessel operators that will require  the use of PINs to confirm an individual’s identity prior to integrating the  user’s TWIC into the facility’s or vessel’s access control system. Officials  from two ports we met with stated that the PIN reset problem had caused  delays in their system enrollment process, as several enrollees could not  remember their PINs and needed to request new TWICs. As of August 1,  2009, TSA reported that 1,246 individuals had requested that their TWIC  cards be replaced due to TSA’s inability to reset the PINs. While TSA  addressed the PIN reset issue by replacing TWICs free of charge, we  estimate that it could cost the government $24,920 to issue new cards to  these individuals and cost the industry $54,375 in lost personal and work  productivity because of time related to the pick-up and activation of the  new TWICs. If all 410,000 affected TWIC cards need to be replaced, it  could cost the government and industry up to approximately $26 million.
If TSA had planned for a potential TWIC system failure in accordance with  federal requirements in contingency planning and internal control  standards, it might have averted the system failure that occurred in  October 2008. Federal guidance includes having an information  technology contingency plan, disaster recovery plan, and supporting  system(s) in place. The type of system failure that TSA experienced  indicates that TSA did not meet federal requirements for minimal  protections for federal systems, which include applying minimum security  controls with regard to protecting the confidentiality, integrity, and  availability of federal information systems and the information processed,  stored, and transmitted by those systems. For example, TSA did not have  an information technology contingency plan or disaster recovery plan in  place to address a potential TWIC system failure. To minimize the effects  of losses resulting from system failures, such plans should provide  procedures and capabilities for recovering a major application or facilitate  the recovery of capabilities at an alternative site. Moreover, TSA did not  have the capabilities or supporting systems in place for recovering the  computer system that houses the TWIC data. Nor did TSA have an  alternate computer system in place to minimize the effects of a TWIC  system failure.
The lack of an approved contingency plan has been a longstanding  concern as identified by the DHS Office of Inspector General. In July 2006  the DHS Inspector General identified that a systems contingency plan for  TWIC had not been approved or tested. According to TWIC program  management officials, they did not previously implement an information  technology contingency plan or develop a disaster recovery plan or  supporting system(s) because they did not have funds to do so. Currently,  TSA has no effort underway for implementing a contingency plan.  However, according to TSA senior officials, they intend to initiate the  development of a disaster recovery plan at the beginning of fiscal year  2010. No documentation has been provided, however, to illustrate  progress in developing a disaster recovery plan. TSA has, however,  identified the lack of a system to support disaster recovery as a risk and  has plans to develop one by 2012. While preparing to initiate the  development of a disaster recovery plan in the next year and a system to  support disaster recovery by 2012 is a positive step, until such plans and  system(s) are put in place, TWIC systems remain vulnerable to similar  disasters.
section_title: Coast Guard Strategies Helped Meet Compliance Dates and Minimize Compliance-Related Issues
paragraphs: Coast Guard employed strategies to help the maritime industry meet the  TWIC national compliance date while not disrupting the flow of  commerce. The strategies utilized included using rolling compliance dates  and a TWIC temporary equivalency. The TWIC temporary equivalency  included allowing workers to gain entry to secure areas of MTSA-regulated  facilities/vessels for a limited time without a TWIC by showing proof of,  for example, TWIC enrollment and evidence that the individual requesting  access had passed the security threat assessment. Below are several  examples of the Coast Guard’s strategies.
Rolling Compliance Dates. To help ensure that all MTSA-regulated  facilities were in compliance by April 15, 2009, the Coast Guard  required affected facilities to comply with TWIC requirements ahead of  the national compliance date on a staggered basis. (See appendix III for  the TWIC compliance schedule.) According to officials from Coast  Guard, TSA, and DHS, in executing the rolling compliance approach,  Coast Guard required ports with a lower population of TWIC users to  comply first, expecting to learn from experiences at these ports prior to  requiring compliance at ports with larger populations. For example, the  first TWIC deadlines were established for ports in Northern New  England, Boston, and Southeastern New England, where Coast Guard  anticipated a lower population of TWIC users. The largest ports, which  TSA believed would present more of a challenge—the Port of New  York and New Jersey, and the Ports of Los Angeles and Long Beach— had TWIC program deadlines of March 23 and April 14, 2009,  respectively. Together, these three ports represent 46 percent of total  U.S. container volume.
TWIC Temporary Equivalency. In accordance with a policy decision,  Coast Guard allowed the use of a TWIC temporary equivalency—or  documentation other than an actual TWIC—for a limited time, prior to  the national compliance date, to allow TWIC applicants who had  passed the security threat assessment access to secure areas of MTSA- regulated facilities/vessels. For example, in Captain of the Port Zone  Corpus Christi, the local TWIC compliance enforcement date was  November 28, 2008. According to a local Coast Guard official, the  sector accepted either the TWIC card or proof that the individual met  the temporary equivalency criteria even though they had yet to receive  an actual TWIC. This approach was in line with the Coast Guard’s  desire to ease the administrative burden on maritime workers.  Similarly, in Captain of the Port Zone New York, the Coast Guard  authorized MTSA-regulated facilities to use a temporary equivalency at  their discretion for those individuals in the same situation. Individuals  meeting the criteria described above were eligible to continue to access  MTSA-regulated facilities until April 15, 2009. On April 1, 2009, the  Coast Guard published an update to the policy decision allowing  individuals who had enrolled in the TWIC program but had not  received their TWIC to be eligible for access to facilities in five Captain  of the Port Zones through May 2009 if they met the applicable criteria  described above, which includes passing the TSA background  investigation. Similarly, due to card issuance challenges and potential  activation back-logs for mariners, on May 28, 2009, the Coast Guard  published a new policy decision allowing all U.S.-credentialed mariners  eligibility for access to specified U.S. vessels and facilities until July 15,  2009, under similar criteria for the temporary equivalency described  above.
The Coast Guard and port strategies also helped to enroll workers in the  TWIC program by the national compliance date, and helped to minimize  compliance related issues through other strategies. For example, during  the first 3 days of compliance for Captain of the Port Corpus Christi, from  November 28 through 30, 2008, the Coast Guard conducted 25 spot checks  at various facilities, during which they inspected 550 workers. Of these,  430 (78 percent) had their TWIC cards and an additional 109 (20 percent)  workers were enrolled but had yet to receive their cards. No trucks or  employees were denied access for lack of a TWIC. Similarly, when Captain  of the Port Zones Miami, Key West, and St. Petersburg reached their local  compliance date on January 12, 2009, the Coast Guard conducted spot  checks of 890 workers from January 13 through January 15, 2009. Of the  890 workers, 709 (80 percent) possessed TWIC cards, and an additional  164 workers, or 18 percent, were enrolled but had not received their  cards.
In addition, during compliance inspections in Captain of the Port Zone  Miami, five cargo facilities were found to be noncompliant. Of the five, two  were brought into compliance immediately upon identification of the  compliance issue with no impact to operations, and three were ordered to  suspend MTSA-related operations until they complied with the TWIC  requirements. As a result of the suspensions, these facilities could not  accept any additional MTSA-regulated vessels until conditions required by  the Captain of the Port were met. The Coast Guard worked with the three  non-compliant facilities and all were cleared to resume MTSA operations  within 2 days. According to one port authority official, the small number of  workers and trucks turned away from ports and facilities on the various  compliance dates may have been attributable to various factors, such as  non-TWIC holders not attempting to enter port facilities, the impact of  reduced port traffic due to the downturn in the economy, or facilities  providing escorts for non-TWIC holders.
section_title: Individual Maritime Ports Employed Different Strategies for Meeting TWIC Compliance Deadlines
paragraphs: Maritime ports across the country also implemented different strategies  for meeting their respective TWIC compliance date. Strategies included,  among others, enacting compliance exercises ahead of the scheduled  compliance date to help identify and address any potential implementation  issues that would arise, and requiring a TWIC as part of meeting other  locally mandated requirements, such as obtaining a local credential that  confirms an individual’s eligibility to access a port’s facilities.
While the official local compliance date for Captain of the Port Zone  Baltimore was December 30, 2008, the Maryland Port Administration  announced that a TWIC would be required for unescorted access to all  Maryland Port Administration facilities beginning December 1, 2008, to  help sensitize workers to the need to obtain a TWIC. As a result,  Baltimore officials reported that most potential compliance issues were  addressed in advance of the official local compliance date.
As of January 15, 2009, the Port Authority of New York and New Jersey  made the possession of a TWIC a prerequisite for obtaining or  renewing a SeaLink Card—a local credential required by the port  authority to verify which drivers are eligible to access facilities under  the port authority’s jurisdiction. According to port authority officials,  by the port’s March 23, 2009, local compliance date, over 7,000 of the  estimated 8,000 truck drivers and International Longshoremen’s  Association members that conduct ongoing business at the port had  met the requirement. As a result, according to port authority officials,  New York did not experience an interruption to commerce on the  March 23, 2009, local compliance date.
At the Ports of Los Angeles and Long Beach, a clean truck program  required truckers doing business at the port to also obtain a TWIC by  October 1, 2008, in order to participate in the program. As a result, the  program requirement helped enroll truck drivers—a population of  concern for TWIC program officials—well ahead of the national April  15, 2009, compliance date for the two ports.
section_title: Challenges in Program Scheduling and Evaluation May Hinder the TWIC Reader Pilot’s Usefulness
paragraphs: Although TSA has made significant progress in incorporating best  practices into TWIC’s schedule for implementing the reader pilot program,  weaknesses continue that limit TSA’s ability to use the schedule as a  management tool to guide the pilot and accurately identify the pilot’s  completion date. Moreover, developing a sound evaluation approach for  collecting information on the pilot’s results could strengthen DHS’s  approach to help ensure the information collected is accurate and  representative of deployment conditions.
subsections: section_title: TSA Has Made Progress Incorporating Best Practices into the TWIC Pilot Schedule, but Weaknesses Exist That Limit Its Usefulness as a Management Tool For Guiding the Pilot and Identifying the Pilot’s Completion Date
paragraphs: As we have previously reported, the success of any program depends in  part on having a reliable schedule that defines, among other things, when  work activities will occur, how long they will take, and how they are  related to one another. As such, the schedule is to not only provide a  road map for the systematic execution of a program, but also provide the  means by which to gauge progress, identify and address potential  problems, and promote accountability. Among other things, best practice and related federal guidance call for a program schedule to be program- wide in scope, meaning that it should include the integrated breakdown of  the work to be performed by both the government and its contractors over the expected life of the program. management include sharing documents such as the schedule with  stakeholders to attain their buy-in and confirm that the schedule captures  the agreed upon activities, time estimates, and other scheduling elements  needed to meet project objectives. Best practices also call for the  schedule to expressly identify and define the relationships and  dependencies among work elements and the constraints affecting the sta and completion of work elements. A well-defined schedule also helps to Moreover, best practices in project  identify the amount of human capital and fiscal resources that are needed  to execute a program.
See, for example, GAO-09-3SP; and OMB, Capital Programming Guide V 2.0,  Supplement to Office of Management and Budget Circular A-11, Part 7: Planning,  Budgeting, and Acquisition of Capital Assets (Washington, D.C.: June 2006).
3.  assigning resources to all activities—identifying the resources needed  to complete the activities;  4.  establishing the duration of all activities—determining how lon 5. activity will take to execute;  integrating all activities horizontally and vertically—achieving  aggregated products or outcomes by ensuring that products and  outcome the right order, and dates for supporting tasks and subtasks are  aligned;  s associated with other sequenced activities are arranged in   critical path for all activities—identifying the path in  7. the schedule with the longest duration through the sequenced list of  key activities;  identifying float between activities—using information on the amount  of time that a predecessor activity can slip before the delay affects  successor activities;  ucting a schedule risk analysis—using statistical techniques to  predict the level of confidence in meeting a project’s completion date;  and  9.  updating the schedule using logic and durations to determine the dat tinuously updating the schedule to determine  ealistic start and completion dates for program activities based on  for all activities—con r current information.
See appendix IV for a more detailed explanation of each scheduling  anagement dated  practice. In a memo from the DHS Under Secretary for M July 10, 2008, DHS endorsed the use of these practices and noted that DHS  would be utilizing them as a “best practices” approach.
TSA has made significant progress during the course of our review in  incorporating best practices into the schedule for implementing the TWIC  pilot program, although weaknesses continue to exist. Specifically, in  response to limitations that we identified and shared with TSA’s program  office, the program office developed a new TWIC pilot integrated master  schedule in March 2009, and updated it in April 2009, and again in May  2009. As figure 2 illustrates, the pilot schedule went from not meeting any  of the nine scheduling best practices in September 2008 to fully addressing one of the practices, addressing seven practices to varying degrees, and  not addressing one practice. According to TSA program officials, prior to  GAO’s first review of the schedule in September 2008, they had not  followed best practices in schedule management because they did not  have enough staffing resources to meet these practices. However, pr officials had not developed a workforce plan to determine the numbe resources needed to carry out the pilot because, accordin g to these  officials, they knew that only two TSA employees and no additional  contract staff would be available to perform this work.
The four areas where TSA’s schedule made the most improvement towa addressing the technical aspects of scheduling best practices include (1) sequencing all activities; (2) integrating schedule activities horizont ally  and vertically; (3) establishing the critical path for all activities; and (4) rd  identifying float between activities. For example, in sequencing all  activities, the activities identified in the schedule were linked to a single  end milestone and pilot sites are no longer scheduled to finish submitting  pilot test data on a federal holiday, December 25, 2009—Christmas Day.  Furthermore, with regard to integrating the schedule horizontally and  vertically, activities contained at different levels of the schedule can now  be viewed in relation to each other. In addition, the schedule now  identifies a critical path, which is useful for determining which activities  are critical for meeting the pilot’s completion date. Finally, the float time  identified—or amount of time an activity can be delayed before affecting  the project finish date—improved, allowing for a better assessment of the  time that each activity can slip before the delay affects the project finish  date. For example, one activity in the schedule went from having 249 days  of float identified to 59.
While TSA has improved its technical application of program scheduling  practices on the TWIC reader pilot program, as of May 2009, weakne sses  remain that may adversely impact its usefulness as a m presenting clear insight as to the progress in each phase of the pilot  assessment. Weaknesses exist in the following areas:    Capturing all activities. The schedule does not accurately reflect all  key pilot activities. For the TWIC pilot, there is no centralized,  consolidated document, such as a statement of work, that captures all  key activities and can be referred to in order to help assure all intended activities are completed and outcomes achieved for each phase of the  pilot testing. While TSA officials acknowledge that each pilot site may  take different steps in preparing for and executing the pilot, they said  that the assumption applied in developing the schedule is that similar  steps are being taken at each site even though each pilot has adopted  varying approaches. Moreover, contrary to best practices in program  management, the schedule has not been shared with and reviewed by  key stakeholders at the pilot sites to capture the varying conditions, pilot related activities, at each site. Key stakeholders at the pilot sites would, for example, be able to (1) identify areas that did or did not  appropriately describe the full scope of their efforts; (2) identify how  the activities at their pilot sites would enable or hinder meeting the  activities identified by TSA; and (3) validate the activities identified by   or TSA and durations of the activities. For example, the schedule include iew. To  having each pilot site complete an environmental related rev ensure consistency with federal environmental and historic  preservation policies and laws, it is FEMA’s policy to require, for  example, environmental reviews of each pilot participant in order t receive federal grant funding. However, depending on the level of  review to be conducted, it may require more or less effort, or activities,  from each grant participant and FEMA to complete. However, the pilot  schedule does not account for the activities required to meet the FEMA required environmental reviews or consistently capture the amoun time such reviews would take relative to the level of review to be  conducted. Without capturing all activities, TSA’s schedule will be  inaccurate, thus, hindering its usefulness a guiding the pilot and measuring progress.  s a management tool for    Assigning resources to all activities. The current schedule does not  fully identify the resources needed to do the work or their availability. For example, the schedule does not identify the labor, material cos ts,  and other direct costs needed to complete key activities. Instead,  resources are assigned to activities at the organization level (e Vendor). TSA officials stated that they do not have complete  information on or control over the required resources because TSA  does not “own the resources” since pilot activities are completed by  non-DHS participants, and some funding is provided through FEMA Port Security and Transit Security Grant programs. However, th should not preclude the TWIC program office from gaining an  understanding of what the overall resource requirements are completing the work. Individual stakeholders, such as pilot  participants, could in part be the source of this information. Moreover,  while TSA expressed concern over their ability to identify resources  for  the pilot in the schedule, officials at pilot sites told us that they had  trouble planning for the pilot and allocating resources because they d not fully understand what the pilot was to entail, therefore making  it  difficult to effectively plan for and identify the needed resources.
Establishing the duration of all activities. The pilot schedule incl duration figures (that is, information on how long each activity is  expected to take to perform), but they may not be reliable. According  to TSA officials, target dates are discussed with participants for some  activities, such as when to start a phase of testing. However, since thepilot program implementation schedule, or relevant segmen schedule, and related updates are not shared with the pilot  participants, it is not clear if the durations TSA’s program office  associated to each activity are realistic or up-to-date. For example,  nearly 86 percent (259 of the 302 activities) of the activities identified  the schedule are based on a 7-day calendar that does not account for  weekends or holidays. While normal operations at pilot sites may occu on a 7-day schedule, resources for conducting pilot activities such a installing readers and associated infrastructure such as cables and  computers or analyzing the results of pilot data may not be available the weekend. By using a 7-day schedule, the schedule inaccurat represents approximately 28 percent more days per year being  available to conduct certain work than is actually available. Best  practices in project management include having stakeholders agree  with project plans, such as the schedule. Because the schedule is no shared with the individual pilots, responsible pilot officials have not  been afforded the opportunity to comment on the viability of the 7-day schedule given available resources. Therefore, pilot participants  not have the resources, such as employees available to work on  weekends, in order to meet pilot goals. As such, if an activity is define as taking 60 days, or approximately 2 months using a 7-day calendar,  the reality may be that participants work a 5-day work week and as a  result the activity takes  longer than scheduled.  approximately 3 months to complete—1 month  TSA program management officials told us that they believe the impact  tand  of using a 7-day versus 5-day calendar is minimal since they unders their key milestones and are committed to meeting the dates they  established. Moreover, according to TSA officials, while knowledge of  when a task would be completed is important to TSA’s managemen the pilot, the level of effort (e.g., number of hours) required by the  grantees or their contractors to complete the work is not. However, not  having a full understanding of how long activities will take to comple has already had an adverse impact on the resource allocation at the  Port of Brownsville pilot site. Port officials in Brownsville told us th to meet the date for initiating pilot testing, their contractors had t work unplanned hours to install electrical wiring and fiber optic  communication cable needed for the TWIC readers to work. T he  contractor stated that this required overtime pay, a resource  expenditure that was not planned. Therefore, although program  management officials may have insight into the schedule using the 7- day approach, the cumulative effect of planning multiple activities to  be completed on non-workdays increases the risk that activities will  not be completed on time with available resources. Since pilot  participants are working on a 5-day schedule, there is a greater risk that key program milestones will not be met, thereby perpetua ting  inaccuracies in the schedule, and reducing its usefulness as a  management and communica completed as TSA intended.  tion tool for ensuring that activities are    Conducting a schedule risk analysis. TWIC program officials have not  performed a schedule risk analysis for the pilot schedule because they do not believe it to be necessary. For the TWIC pilot, a schedule risk  analysis could enable the program to model “what if” scenarios as to  when and if locations such as Long Beach will complete their  preliminary work and the effects that schedule changes, if any, might  have on meeting the pilot reporting goal. A schedule risk analysis  could also help facilitate detailed discussions between the TWIC  program office at TSA and the individual pilot locations regarding task  durations and expected progress. This is especially relevant for the  TWIC pilot given that the schedule does not clearly articulate all of the  tasks that need to be completed to carry out the pilot, or changes that  may result due to the availability of funding. For example, according t TSA officials and one pilot participant, such changes included delays  in FEMA’s approval of pilot participants’ award contracts to allow the  grantees to expend grant funds. In any program that lacks a schedule  risk analysis, it is not possible to reliably determine a level of  confidence for meeting the completion date.
Updating the schedule using logic and durations to determine th dates for all key activities. The pilot schedule is missing several  elements needed to reliably use logic and durations to continuously  update the schedule and determine revised dates for all key activities.  Implementing this practice is reliant upon other scheduling practices,  such as capturing all activities, assigning resources to all activities,  establishing the duration of all activities. However, the TWIC pilot  schedule has not yet fully addressed each of these practices. Thu s,  schedule updates may not result in reliable dates. Moreover, the  current schedule includes date anomalies, such as identifying task to be started as already having started, and includes 18 activit scheduled in the past for which no actual start date has been  identified. For example, the schedule indicates that three activities at  the Staten Island pilot site have started on a future date yet to occur These anom schedule.  . alies indicate the presence of questionable logic in the  Contrary to best practices in program management, as of August, 2009,  TSA had not shared the pilot schedule, or at least relevant segments of the  schedule, with pilot participants—all key stakeholders whose buy-in—th at  is commitment and resources—is needed to ensure that pilot goals and  time frames are met. Benefits of sharing the schedule with stakeholde include, for example, confirming the activities needed to complete the  pilot, associated resources, activity durations, the viability of attaining  milestone dates, and potential risks for schedule slippages. Furtherm the schedule can serve as a valuable communication tool by helping stakeholders in their individual planning efforts. According to TSA  officials, they do not see the value in providing the schedule to pilot  participants because it contains too much information. Further, TSA  officials told us that they have not shared the schedule with pilot  participants due to concerns about sensitive information related to when  the pilot results will be provided for congressional review. Lastly, TSA is  uch as  also concerned that the pilot participants will not have the tools, s Microsoft Project, available to read and understand the schedule.  However, sharing the schedule with pilot participants in a format read by all can be accomplished using tools such as email or by providing  participants with a paper copy. Moreover, to overcome sensitivity TSA could provide participants with the segment of the schedule  applicable to the pilot participant and separately inform them of their  impact on the overall schedule. Furthermore, having pilot participants, as   issues,  stakeholders, confirm the viability of key dates and duration of activities,  and illustrating the impacts that schedule slippages on any one act have on meeting pilot goals and reporting deadlines, can enhance  collaboration and communication, help participants in their individual  planning efforts, and help minimize future schedule slippages. Without  doing so, TSA runs the risk of continuing to manage the program base an unreliable schedule, further delaying the development of the card  reader rule and implementation of the TWIC program with biometric card  readers. Since September 2008, TSA has revised its schedule for  completing the TWIC reader pilot from October 13, 2009, to a year later,  October 4, 2010. Consequently, TSA’s current schedule indicates that th will not meet the April 2010 deadline for reporting to Congress on the  results of the TWIC reader pilot.
section_title: DHS Does Not Hav Sound Evaluation Approach to Ensure Information Collected to the Inform Congress and Card Reader Rule Is Complete, Accurat Representative of Deployment Conditions
paragraphs: Shortfalls in TWIC pilot planning have presented a challenge for TS Coast Guard in ensuring that the pilot is broadly representative of  deployment conditions, and will yield the information needed to inform  Congress and a card reader rule aimed at defining how TWICs will be used  with biometric card readers. This is in part because an evaluation plan th at  fully identifies the scope of the pilot and the methodology for collecting  and analyzing the information resulting from the pilot has not been  developed. Agency officials told us that no such evaluation plan was  developed because they believe that the existing pilot documentation  coupled with subject matter expertise would be sufficient to guide the  pilot and no evaluation plan is needed. However, our review of the TWIC  pilot highlights weak nesses that could be rectified by the development of  an evaluation plan.
In informing the card reader rule, the TWIC reader pilot is to, among othe things, test the technology, business processes, and operational impacts  required to deploy card readers at secure areas of the marine  r  transportation system. Specifically, the testing is to assess how the T performs when used in conjunction with biometric card readers and  systems at maritime facilities and vessels, how the technology performs when used as part of the pilot sites’ normal business processes, and to  help identify the operational impacts of deploying biometric card readers  based on these locations. The pilot results are to help identify the actionsnecessary to ensure maritime facilities and vessels can comply with the  TWIC regulation that is currently being drafted known as the card rea rule. provide information needed for developing the regulatory analysi required by the Office of Management and Budget as part of the  rulemaking process. The regulatory analysis is to demonstrate that  examinations of the most efficient alternatives were considered and an evaluation of the costs and benefits—or impacts—to be borne by the  government, private sector, a regulation were considered.
The regulation requiring the use of a TWIC for accessing MTSA-regulated facilitie essels was issued on January 25, 2007. pilot and analysis of the results. At a minimum, a well-developed, sound evaluation plan contains several key elements, including (1) clear  objectives, (2) standards for pilot performance, (3) a clearly articulated  methodology, and (4) a detailed data analysis plan. Incorporating these  elements can help ensure that the implementation of a pilot generates  performance information needed to make effective management decisions.
In planning for and designing the TWIC pilot, DHS—including TSA, Coas Guard, and its Science and Technology Directorate—developed a t evaluation master plan consisting of several documents. Together,  the  TWIC pilot documents address key evaluation plan elements to varying  degrees. These documents are useful for identifying planned data  collection methods. However, addressing several shortfalls in their  planning efforts—such as omissions in the planning methodology a absence of a data analysis plan to help guide information collection  efforts—could strengthen the usefulness of the information collected  through the pilot. The following discusses the extent to which key  elements are addressed in the TWIC pilot program documentation.
Clear objectives. TWIC pilot documentation identified general program  objectives, referred to as the program goals. TWIC program objectives  include (1) conducting tests of biometric card readers and the credential  authentication and validation process to evaluate the reader specification and (2) testing the technology, business processes, and operational  impacts required to deploy TWIC readers on facilities and vessels prior to ;  issuing a final rule. The objectives, as stated, articulate the key goals for the pilot. Identifying clear objectives for an evaluation can help ensure  that the appropriate evaluation data are collected and that performance  can be measured against the objectives.
Performance standards. TSA in conjunction with the Coast Guard  developed standards for determining performance for the TWIC pilot, but  the standards do not fully address important aspects of the pilot  assessment, such as those needed to assess the business and operational  impacts of using TWIC with biometric card readers. For example, the  master plan identifies some operational performance requirements, suc as a minimum reliability threshold, that the card reader is to meet. The  plan also identifies technical requirements readers are to meet, such asmeeting specific biometric standards or, for example, transaction times.  However, the performance standards mostly focus on technology and do  not fully identify standards for the business and operational circumstances that using TWIC with biometric card readers will demand. Business an d  operational circumstances include, for example, the experience a worke will have when attempting to access a secure area of a MTSA-regulated  facility, additional steps a worker may need to take to successfully enter a  facility, or changes to business processes to accommodate the use of  TWIC with readers. Neither the master plan nor subsequent test plans  identify performance standards for assessing business and operational  performance. For example, there is no test for when a user presents a  g  valid but non-functioning TWIC at an access-control point, and assessinthe impact of that scenario on the flow of commerce. TSA officials stated that they had not included this test in the pilot but would consider adding  it and others we identified as part of their pilot test. In addition, DHS  noted that they expect to identify the business and operational impacts  that occur during respective phases of the pilot. While identifying and  collecting information on activities as they occur during a pilot can  enhance address important aspects of the pilot assessment could strengthen DHS’s  efforts in determining to what extent the piloted methods are effective  the amount of data collected, incorporating criteria that fully  .
Clearly articulated evaluation methodology. The methodology for  evaluating the TWIC pilot is not fully defined and documented, does not  account for differences in pilot design, may not be representative of future  plans that individual port facilities have for using TWIC, and does not  provide for testing some of the known requirements under consideration  for inclusion in the card reader rule. Thus, such weaknesses may adve impact the sufficiency and reliability of the information collected from pilot.
The unit of analysis for conducting the pilot, pilot site selection  criteria, and the sampling methodology are not fully defined and  documented. The unit of analysis—or the level at which the analysis is  ies  to be conducted—had not been defined prior to selecting the facilit and vessels to participate in the TWIC pilot. Specifically, while TSA and  Coast Guard intended the unit of analysis to be focused on secure  areas, they did not determine whether analysis of pilot test results  would be conducted at the port level, facility/vessel level, or the acce control point level. As we have previously reported, defining the unit analysis for any evaluation is particularly important because the resu from such an effort will vary depending on this. With regard to the  TWIC pilot, the pilot’s assessment could focus on many different unit of analysis. For example, the pilot could be designed to assess the results at a more aggregate level, such as accessing a secured area in its entirety, such as an entire port, facility or vessel. Or, the pilot could  focus on the use of readers based on a particular function, such as at  tranceways for boarding a cruise liner. When  trucking lanes or at endesigning an evaluation, such as a pilot, it is important to define the unit of analysis and how it may be aggregated at an early stage. This  increases the likelihood that the information collected is representative  of the information needed for evaluation and can be used to project  similar experiences elsewhere. Moreover, as we have previously  reported, confronting data collection analysis issues during the design  stage may lead to a reformulation of the questions to be addressed as  part of an evaluation to ones that can be answered within the time and  resources available.
TSA officials told us that no specific unit of analysis, site selection  criteria, or sampling methodology was developed or documented prior  to selecting the facilities and vessels to participate in the TWIC pilot.  According to TSA officials, they did, however, take the following  factors into account when selecting grant recipients to participate in  the pilot: (1) the TSA Deputy Secretary suggested including the ports  of  Los Angeles and Long Beach because they are large volume operations;  (2) the Port Authority of New York and New Jersey was selected  because of weather conditions and the great mix of traffic (e.g., cargo  containers, bulk commodities, and passenger vessels); and (3 ) the Port  of Brownsville was considered because it was in the Gulf region of the  While these general  United States and it represents a smaller port. factors were used for selecting the grant recipients to participate in the  pilot, the selection factors did not take all evaluation factors into  account, such as ensuring that certain types of facilities with specified risk rankings would be selected at each port to facilitate the  comparison of pilot results between the different locations. According  to TSA officials, they did not identify more specific selection criteria  based on the unit of analysis to be evaluated because they believed the  e  factors that they did consider would produce the breadth of maritim operations needed to conduct the pilot. Further, they stated that they  could meet evaluation needs by subsequently identifying facilities an vessels at the pilot sites by the type of business they represented (i.e container facility, liquid storage facility).
However, the pilot documentation does not identify if and how the  operations of facilities and vessels at one pilot site are to be compared  with those at another site or how the pilot or subsequent evaluation  approach is to compensate for the additional factors. For example,  additional factors that may impact the ability to compare different site may include the size of the operation or business processes in place.  Moreover, according to TSA officials, they now believe that because  TSA and Coast Guard had to rely on volunteer MTSA-regulated  facilities and vessels to participate in the pilot, they were limited in  their ability to ensure the adequacy of the number and type of selected  facilities and vessels for the pilot. The pilot documentation, however,  does not yet identify perceived shortcomings with the selected pilot  participants, methods for compensating for perceived shortcomings, or evaluation methods to be used to ensure data collected at pilot sites  s  will be comparable and will be representative of the experience of  implementing TWIC with biometric card readers across the nation.  Further, the documentation does not identify the unit of analys define how data are to be analyzed, or how the pilot results are to be  compared or contrasted between types of locations, facilities/vessels,or functions. The lack of planning documentation makes it difficult to  judge the basis for pilot selection, related constraints, or the ext ent to  which corrective actions have been subsequently applied to straints. Given that the existing  compensate for the earlier con evaluation plan documentation does not identify the unit of analysis,  define how data are to be analyzed, or how the pilot results are to be  compared or contrasted between types of locations, facilities/ve or functions, there is a risk that the selected pilot sites and test  methods will not result in the information needed to understand the  impacts of TWIC nationwide.
Differences in pilot designs are not accounted for. The pilot test and evaluation documentation does not identify how differences in  individual pilot site designs and resulting variances in the information  collected from each pilot site are to be assessed. This has implications  for both the technology aspect of the pilot as well as the business and  operational aspect. For instance:  While TSA is applying some controls over the technologies tested  individual pilot sites, it has not identified how the pilot is to  compensate for the different technologies tested at each site. For  example, as part of its initial capability evaluation, TSA tested a se number of readers to ensure they met certain performance parameter Furthermore, pilot participants were asked to choose readers that  passed the initial capability evaluation. While TSA controlled the  population of readers pilot participants could select from, it did not  control for alterations made to readers at pilot sites to optimize rea performance or differences in the computers, software, or access  control systems with which pilot sites are integrating TWIC readers. s.
Thus, it will be difficult for TSA and the Coast Guard to extrapolate how the use of TWIC-related technologies will be expected to impact  nsating  the maritime environment as a whole without applying compe strategies to control for variances to some of these variables. For  instance, by not controlling for key variables, such as how a particular site integrates readers with its existing access control system, pilot  results may show that a delay related to the use of biometric ca readers was incurred, but not appropriately identify the root cause  the delay (e.g., the reader itself or the integration approach).  of  Business and operational processes and pilot approaches are not the  same at each pilot site and a methodology for compensating for the  differences has not been developed, thereby complicating the  assessment of the results. For example, officials at the Port of Los  Angeles said they intend to test all access points at the three MTSA- regulated facilities participating in the pilot test. In contrast, the testing  approach at the Port Authority of New York and New Jersey currently  includes testing one function at different facilities—such as testing a  TWIC reader at 2 of 31 truck lanes at one facility and testing a turnsti in a high volume location at a different facility—instead of all access  points at each facility. Further, testing at each port will not necessarily  coincide with the time of year with the highest volume of cargo or  the  environmental conditions for which the pilot sites were selected (e.g.,  New York in the cold winter months, Brownsville, Texas, during the  hottest and most humid months). Without a methodology for  compensating for these differences, the information collected be comparable or captured in a manner that can be aggregated to  assess the impact of TWIC reader deployment on maritime commerce  across the nation. According to DHS officials, they understand that t and other limitations exist with the pilot. However, they have decide to proceed with the pilot in this manner, collecting whatever  information they can instead of all the information that is needed,  because of funding issues. These funding issues include not having the  resources to test for every situation they would like and not having  control over how pilot participants use the dollars available for the  pilot. However, pilot planning documentation does not identify the  resources needed to conduct testing for the additional situations, the  additional situations TSA and DHS would like to test for, or the testin that will not occur because of insufficient resources. Moreover, TSA  e  and FEMA do have some controls in place to ensure participants us some of the grant funds for the pilot. For instance, as part of the grant  process, pilot participants submitted investment justifications to FEMA for approval which were reviewed and approved by FEMA. TSA provided a copy of each reviewed the grantees’ plans. Furthermore, pilot participants must  submit budget and expenditure reports. Given these steps in the grant  was   justification and both TSA and Coast Guard  management process and coordination between FEMA and T SA, DHS  could exert some control over how participants use the dollars  available for the pilot.
Pilot site test designs may not be representative of future plans for  using TWIC. Pilot participants are not necessarily using the  technologies and approaches they intend to use in the future when  TWIC readers are implemented at their sites. In accordance with best  practices, pilots should be performed in an environment that is  characteristic of the environment present in a broadscale deployment.However, officials at two of the seven pilot sites told us that  the  technology and processes expected to be in place during the pilo likely not be the same as will be employed in the post pilot  environment, thereby reducing the reliability of the information  collected at pilot locations. For example, officials we spoke with at  pilot site told us that, during the pilot, the site intends to use a hand  held reader solution, but plans to install fixed readers requiring additional investment in technology infrastructure after the pilot is  complete. They are taking this approach because they want to  participate in the pilot, but do not want to invest heavily in a for the pilot that may not work. As a result of this approach, the  information collected from this pilot participant will not be  representative of the technology, processes, and cost impacts that  implementation of TWIC with biometric card readers will have  location. Moreover, use of the results captured from this pilot site may  hinder the reliability of impact projections made based on this  information. Officials at a third pilot site told us that they are using the cheapest solution possible for the pilot because they do not believe that the use of TWIC will ultimately be applicable to them. They s that they  likely to have to implement the use of TWIC with biometric card  readers.  would, however, select a different approach if they were    The pilot methodology is not analyzing or testing some of the  potential requirements under consideration for inclusion in the  reader rule. On March 27, 2009, the Coast Guard published the  Advanced Notice of Proposed Rulemaking (ANPRM) for the card  reader rule. The ANPRM identifies the requirements under  consideration, as defined by the Coast Guard, for deploying TWIC  readers at MTSA-regulated facilities and vessels that would be  potentially included in the card reader rule on using TWICs with  card  biometric card readers. As such, the ANPRM presents some of  the  technology, business, and operational requirements that are being  considered in developing the card reader rule. Moreover, they  represent potential costs and benefits—or impacts—to be borne by t government, private sector, and population at large as a result of the  regulation being considered. As such, they are representative of the  characteristics that should be included in conducting the TWIC pilot to  help ensure that maritime facilities and vessels for which the rule will  apply can fully comply with the TWIC rule. However, our review of th ANPRM again st the pilot documentation found that the pilot does not  address or test some requirements under consideration for the card  reader rule.
Of the 27 potential requirements contained in the ANPRM that we  assessed, 6 (22 percent) were being tested in the pilot, 10 (37 percent)  were partially being tested, and 11 (41 percent) were not being tested  (see appendix V for more detail). For example, one potential  requirement in the ANPRM is that owners and operators of facilitie the highest risk group may require PINs as an additional level of  security. However, the pilot does not test the use of PINs and the  associated impacts the use of PINs could have on access control  processes, such as increased waiting times for accessing secure areas  or shipping delays. Similarly, another potential requirement being  considered in the ANPRM but not tested for in the pilot includes  requiring that those owners and operators using a separate physical  access control system identify how they are protecting personal  identity information. However, the pilot does not test for the impacts o added security on systems to prevent the disclosure of personal  f  identity information. Such impacts could include, for example, a down of system speed for processing a TWIC holder and costs  associated with ensuring the actual security of the information  maintained in a system. Both of these potential requirements, if  implemented, could have operational, technical, and cost implications  for maritime commerce.
TSA officials told us that they plan to use the results of our an help them identify additional requirements for testing in the pilot.  According to Coast Guard officials, they did not assess each  requirement under consideration in the ANPRM against the TSA  test  documents. Instead, they assessed selected requirements identified i the summary table in the ANPRM. They said that they plan to  supplement the information the pilot provides with data from other  sources. While supplementing the information collected can be  beneficial, designing the pilot to collect the most information possible  about those requirements under consideration  for the card reader rule  could enhance TSA and Coast Guard’s understanding of the viability of  certain requirements and related limitations.
Detailed data analysis plan. TSA has not developed a detailed data  analysis plan to describe how the collected data is to be used to track the program’s performance and evaluate the effectiveness of using TWIC with  biometric card readers. Moreover, the available plans do not identify the  criteria, methodology, unit of analysis, and overall approach to be used in  analyzing the pilot data to ensure that the needed information will result  from the pilot. As we previously reported, a detailed analysis plan is a key  feature of a well-developed, sound evaluation plan as it sets out who will  re  do the analysis and when and how the data is to be analyzed to measu the pilot project’s performance. Because the information from the pilot to be used to identify the impact of using TWICs with biometric card  ing  readers at maritime facilities and inform the card reader rule (includthe related regulatory analysis), a detailed data analysis plan could help ensure that the implementation of the pilot generates performance  information needed to make effective management decisions. Without  such a plan, it will be difficult for TSA and Coast Guard to validate the  results from the pilot and ensure the accuracy and use of the information.
Consequently, the resulting information may not allow others—such as  Congress or external parties affected by the regulation—to independently  assess the results and make conclusions about the impacts—includin g  costs and benefits—of implementing TWIC with biometric card readers .
Because the pilot may not provide all of the information needed for  implementing the card reader rule and supporting regulatory analysis,  Coast Guard officials told us that they would be supplementing the data  collected from the TWIC pilot after the pilot is completed rather th adjusting the pilot approach to collect the information. According to Coas Guard officials, they plan to supplement TWIC pilot data by using  techniques allowable under federal guidance for developing assessmen in support of a federal regulation. We agree that following the fed eral  guidance should help inform the development of the card reader rule.  However, TSA and Coast Guard officials have not identified how  information collected outside of the pilot is to be used as part of the  evaluation methodology. As we have previously reported, defining what   an  data is needed and how the data is to be used and assessed as part of evaluation plan can help to ensure information needs are met and properly considered. TSA and Coast Guard could, for example, augment the  information collected from the pilot by leveraging information from other  ports that are already or are about to begin using TWICs with biometric  card readers. Augmenting the pilot with information from other facilities and vessels that have already implemented TWICs with biometric  card  readers could help TSA and the Coast Guard meet pilot objectives, and  help ensure the pilot effectively informs the card reader rule. By  identifying the additional information to be collected along with its source,  as well as defining compared, TSA and Coast Guard can strengthen their efforts to inform the  c  the approach for how the information will be used and  ard reader rule.
section_title: Conclusions
paragraphs: TSA has made significant progress in enrolling, activating, and issuing  TWICs. As of September 2009, over 1.3 million maritime transportation  workers have been enrolled and over 1.1 million TWICs have been  activated. Consequently, the enrollment and activation phase of the  program for meeting the national compliance date of April 15, 2009, has  reached completion. However, the data acquired from workers during this  phase of the program and in the future needs to be adequately maintain ed  so that the program can continue uninterrupted and the security aspects o the program can be realized. Since the TWIC system has already failed  once—disabling TSA’s ability to reset PINs on TWICs and causing dela in the enrollment of workers and the activation of cards—an approved  information technology contingency plan, disaster recovery plan, and  supporting system(s) for the computers that store TWIC-related data coul help ensure the program’s continuity and effectiveness. While the DHS  Inspector General identified the lack of an approved contingency plan in 2006, no steps have been taken to develop such a plan. TSA officials stated that they are planning to develop a disaster recovery plan in fiscal year  2010 and disaster recovery system by 2012. However, until a contingencyplan for TWIC systems, including a disaster recovery plan and supporting  system(s) as needed are put in place, TWIC systems remain vulnerable.
The potential security benefit of the TWIC program will not be fully  realized until maritime transportation facilities install biometric card readers and integrate them with the facilities’ access control systems. Thepilot test, intended to inform this phase of the program and the regulation on the use of the card readers in the future, has a number of weaknesses  that could negatively affect its rigor and timely completion. Specifically,  weaknesses in the pilot schedule limit its usefulness as a management tool for executing the pilot, monitoring its progress, and determining the pilot’ nts  completion date. Until the pilot schedule is shared with pilot participa and updated to accurately reflect realistic resource and time constraints,  TSA will lack the management information needed  progress towards meeting the planned completion date and pre-emptively  identifying likely slippages in the completion date.
section_title: Recommendations Executive Action
paragraphs: To minimize the effects of any potential losses resulting from TWIC  system failures, and to ensure that adequate processes and capabilities ar in place to minimize the effects of TWIC system interruptions, we  recommend that the Assistant Secretary for the Transportation Security  Administration direct the TWIC program office to take the following  action:    develop an information technology contingency plan for TWIC systems,  including the development and implementation of a disaster recovery  plan and supporting systems, as required, as soon as possible.
To help ensure that the TWIC pilot schedule can be reliably used to guide  e  the pilot and identify the pilot’s completion date, we recommend that th y for the Transportation Security Administration direct  Assistant Secretar the TWIC program office, in concert with pilot participants to take the  following action:  fully incorporate best practices for program scheduling in the pilot  schedule to help ensure that (1) all pilot activities are captured; (2)  sufficient resources are assigned to all activities; (3) the duration of all  activities are established and agreed upon by all stakeholders; (4) a  schedule risk analysis is conducted to determine a level of confidence  in meeting the planned completion date planned activities within scheduled deadlines; and (5) the schedule is  correctly updated on a periodic basis.   and impact of not achieving  To ensure that the information needed to assess the technical, business,  and operational impacts of deploying TWIC biometric card readers at  ent  MTSA-regulated facilities and vessels is acquired prior to the developm of the card reader rule, we recommend that the Assistant Secretary for the  Transportation Security Administration and Commandant of the U.S.  Coast Guard direct their respective TWIC program offices to take the  following two actions:    develop an evaluation plan to guide the remainder of the pilot tha includes performance standards, a clearly articulated evaluation  methodology—including the unit of analysis and criteria—and a data  analysis plan.  identify how they will compensate for areas where the TWIC reader  pilot will not provide the necessary information needed to report to  Congress and implement the card reader rule. The information to  collected and approach for obtaining and evaluating information  obtained through this effort should be documented as part of an  evaluation plan. At a minimum, areas for further review include the  potential requirements identified in the TWIC Reader Advanced Not of Proposed Rulemaking but not addressed by the pilot. Sources of  information to consider include investigating the possibility of using information resulting from the deployment of TWIC readers at non- pilot port facilities to help inform the development of the card reader  rule.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Secretary of Homeland Security  for review and comment. DHS provided written comments on behalf of t department and the Transportation Security Administration, the United  States Coast Guard, and the Federal Emergency Management Agency on  November 5, 2009, which are reprinted in appendix VI. In commenting on  our report, DHS stated that it concurred with three of the four  recommendations and partially concurred with the other one and  identified actions planned or under way to implement them. DHS is taking  steps to address our first recommendation related to information  technology contingency planning for TWIC systems; however, the actions  DHS reported TSA and Coast Guard have taken or plan to take do not fully  address the intent of the remaining three recommendations.
With regard to our first recommendation, DHS concurred with our  recommendation that TSA develop an information technology contingency plan for TWIC systems, including the development and implementation of a disaster recovery plan and supporting systems. DHS reported that TSA  has taken actions to improve contingency planning and disaster recovery  capabilities for TWIC related systems. According to DHS, such actions  include adding TWIC systems enhancements, such as back-up systems  (i.e., redundancy system), and plans for a system Continuity of Operations  Plan (COOP) site as part of its Office of Transportation Threat Assessmen t  and Credentialing’s infrastructure modernization effort. TSA’s actio develop a contingency plan for TWIC systems, including a disaster  recovery plan and supporting system to recover operations in the future.  s, should help enhance TSA’s ability  DHS concurred in part with our second recommendation, that TSA, concert with pilot participants, fully incorporate best practices for  program scheduling in the pilot schedule. In its response, DHS agreed that  a program schedule is a critical management tool for implementation of  the TWIC reader pilot, and notes that its implementation of best practice is tailored to specifically meet the requirements relative to the complex  and unique constraints of the pilot program. For example, according to  DHS, it focuses its outreach and coordination efforts on the completion hile  key tasks when risks to the critical path are identified. However, w DHS has made progress in developing the schedule from the TSA  perspective, it has not developed the schedule in concert with pilot  participants, as we are recommending. As DHS notes, the voluntary natur of the pilot has allowed participants to proceed at their own pace, based  on their own local priorities and procedures, making it difficult to de and maintain accurate activity durations for management purposes.  However, based on our review of the TWIC reader pilot schedule, DHS has  not accounted for each participant’s pace, local priorities and procedures.  Instead, DHS, through TSA, identified the activities it deemed to be key for  nt  completing the pilot without fully understanding what each participa needs to do to accomplish the key tasks and how long it will take to  complete those activities given available resources and local processes. Working individually with its pilot participants to account for program  complexities should help ensure that the overall TWIC pilot schedule is  informed by each participant, and that key elements—such as the critic path—identified in the schedule developed by TSA are more accurate.  Moreover, as noted in our report, the TWIC pilot schedule will not contai the level of information needed for DHS to make effective management  decisions despite its efforts to improve its application of scheduling  practices. Therefore, additional corrective steps by DHS and TSA a needed to help ensure that the program schedule can be used as a  management tool  completion date.  to guide the pilot and accurately identify the pilot’s  DHS also concurred with our third and fourth recommendations, that the  ation  TWIC program offices at TSA and Coast Guard (1) develop an evalu plan to guide the remainder of the pilot that includes performance  standards, a clearly articulated evaluation methodology—including the  unit of analysis and criteria—and a data analysis plan; and (2) identify ho w  the agencies will compensate for areas where the TWIC reader pilot will  not provide the necessary information needed to report to Congress and  implement the card reader rule. We recommended that the information  be collected and approach for obtaining this additional information be  documented as part of the evaluation plan. Developing an evaluation pl for a pilot is a prospective endeavor to help guide the identification of  needed data and data sources and methods for comparing the data and obtaining the information needed. However, it is not clear from DHS’s  comments whether their proposed actions will fully address these two  recommendations. As our report indicates, while TSA developed a test an evaluation master plan for the TWIC pilot, the document did not identi the business and operational data to be collected during the pilot, the  performance standards for assessing the data, or the methodology forevaluating the data. To meet the intent of our recommendations, this  information would need to be included in the evaluation plan prior t proceeding with the pilot to ensure that the needed data points are  planned for and collected during the pilot in order to inform the mandated report to Congress on the results of the pilot. However, DHS’s comment do not indicate that it will take these steps to help inform the report to  Congress or the rulemaking process for the TWIC reader rule. Instead, in  its response, DHS identifies guidance that it plans to use to supplement th data gathered from the pilot. While identifying the guidance is a positive  step, the guidance is not a substitute for a well-developed evaluation pla that defines the information to be collected and approach for obtaining  and analyzing the pilot information. Furthermore, the guidance ca compensate for areas where the TWIC pilot does not provide the  necessary information. The plan would help DHS ensure that the pilot  serves the purpose Congress intended—collecting the data needed to  adequately assess the TWIC program during the pilot.
In its comments to our draft report, DHS, on behalf of TSA, also  commented on the October 21, 2008, power outage at the facility that ho TWIC systems. This outage affected TSA’s ability to reset the PINs (i.e.,  provide users with new PINs) on 410,000 TWIC cards issued prior to th power failure. As part of the regulation that is currently being written,  MTSA-regulated facilities and vessels may require TWIC users to use th PIN to unlock information on a TWIC card, such as the TWIC holder’s  picture, to verify the identity of a TWIC holder. Consequently, TSA will  have to replace the cards for cardholders who forget their PINs instead of  resetting these PINs. In its response, however, TSA questioned whether itwould cost the government and industry up to $26 million to replace the  410,000 TWIC cards potentially affected by the outage. DHS commente d  that in the 11 months since the incident, only 1,246 cards have needed  replacement and TSA officials believe it highly unlikely that all 410,00 affected transportation workers will need their cards to be replaced.
In addition, DHS provided techn into the report as appropriate. ical comments, which we incorporated  As agreed with your offices, unless you publicly announce the contents o this report earlier, we plan no further distribution until 30 days from the  date of this letter. We will then send copies of this report to the Secretary  of Homeland Security, the Assistant Secretary for the Transportation  Security Administration, the Commandant of the United States Coast  Guard, and appropriate congressional committees. In addition, this report  will be available at no charge on the GAO Web site at http://www.gao.gov/.
If you or your staff have any questions about this report, please contac at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last pag of this report. Key contributors to this report are listed in appendix VII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This review examined the Transportation Security Administration’s (TSA)  and Coast Guard’s overall progress in implementing the Transportation  Worker Identification Credential (TWIC) program. We addressed the  following questions: (1) To what extent did TSA, the Coast Guard, and the  maritime industry take steps to meet the TWIC compliance date and  address related challenges? and (2) What management challenges, if any,  do TSA, Coast Guard, and the Department of Homeland Security (DHS)  face in executing the TWIC pilot test for informing Congress and the card  reader rule?
To identify the steps taken by TSA, the Coast Guard, and the maritime  industry to meet the April 15, 2009, TWIC compliance date, and address  related challenges, we reviewed program documentation on the status of  TWIC enrollment and activation as well as implementation efforts from  both TSA and the Coast Guard. Among others, this documentation  includes compliance reports compiled by the Coast Guard from facility- gathered information, TSA’s TWIC communication plan for disseminating  information about the TWIC enrollment process and compliance  deadlines, and program management reviews on TWIC enrollment,  activation, and issuance. We also interviewed U.S. Citizenship and  Immigration Services officials regarding their participation in the TWIC  card production and personalization process. In addition, we visited and  observed the enrollment process with TSA and TSA contractor  representatives at four TWIC enrollment and activation centers. Further,  we reviewed TWIC user population estimates and discussed their data  reliability with TSA and Coast Guard officials as well as efforts taken to  update the population estimates and plan for TWIC enrollment and  activation activities and resources. We analyzed pertinent information  including key statutes such as the Maritime Transportation Security Act of  2002 (MTSA), as amended by the Security and Accountability For Every  (SAFE) Port Act of 2006, and related regulations, policies, and guidance  setting out requirements for the TWIC program. We also    obtained information from maritime industry stakeholders—such as  TWIC Stakeholder Communication Committee—a 15-member advisory  council to TSA, Coast Guard, and their contractor to promote real-time  communications flow between industry, government, and the TWIC  contracting team;  reviewed reports by the National Maritime Security Advisory  Committee—an advisory council to DHS;    met with nine associations whose members are impacted by the  implementation of TWIC, such as the    American Association of Port Authorities—a trade association that  represents more than 160 public port organizations throughout the  Western Hemisphere;    The Independent Liquid Terminals Association—a trade association  representing companies with bulk liquid terminals and above  ground storage tank facilities (“tank farms”) that interconnect with  and provide services to various modes of bulk liquid carriers, such  as oceangoing tank ships, tank barges, tank trucks, tank rail cars,  and pipelines; and    The Association of American Railroads—whose members represent  a 140,000-mile rail network, including the major freight railroads in  the United States, Canada, and Mexico, as well as Amtrak.
We also visited four TWIC enrollment and activation centers, and visited  and/or met with officials of facilities and vessels impacted by TWIC across  the country such as the ports of Los Angeles and Long Beach, California;  Brownsville, Texas; Baltimore, Maryland; and Houston, Texas; as well as  the Port Authority of New York and New Jersey. In addition, we met with  officials representing vessel operations at the Staten Island Ferry in Staten  Island, New York; Magnolia Marine Transports in Vicksburg, Mississippi;  Watermark Cruises in Annapolis, Maryland; and World Cruise Terminal in  San Pedro, California. At each location, we interviewed officials of  facilities and vessels responsible for implementing the use of TWIC. While  information we obtained from these interviews and site visits may not be  generalized across the maritime transportation industry as a whole,  because the facilities, vessels, and enrollment centers we selected are  representative of high and low volume entities in the maritime industry  and the enrollment centers are representative of areas with high  population density, the locations we visited provided us with an overview  of the general progress of the TWIC program, as well as any potential  implementation challenges faced by MTSA-regulated facilities/vessels,  transportation workers, and mariners. Further, we interviewed TWIC  program officials from TSA and the Coast Guard—including the TWIC  Program Director at TSA and the Coast Guard Commander responsible for  the TWIC compliance program—regarding their efforts to implement the  TWIC program. We also interviewed a number of Coast Guard officials at  ports across the country regarding local TWIC implementation and  compliance efforts to better understand the processes and procedures in  place for enforcing compliance with TWIC. Specifically, we interviewed  Coast Guard officials with responsibilities in New York and New Jersey;  Los Angeles and Long Beach, California; Corpus Christi, Texas; and  Baltimore, Maryland. We met with these Coast Guard officials because the  facilities, vessels, and enrollment centers we visited are housed in these  officials’ area(s) of responsibility. To assess the extent to which TSA  planned for the potential failure of information technology systems  supporting the TWIC program in order to minimize the effects of potential  TWIC system failures, we reviewed TWIC program management reviews  and conducted interviews with TWIC program staff. We compared TSA’s  efforts with Office of Management and Budget (OMB) and National  Institute of Standards and Technology (NIST) guidance, government  internal control standards.
To identify and assess the management challenges TSA, the Coast Guard,  and DHS face in executing the TWIC pilot test for informing Congress and  the card reader rule, we reviewed prior GAO reports and testimonies on  the TWIC program issued from December 2004 through September 2008,  and key documents related to the TWIC reader pilot. These documents  included the Broad Agency Announcement-Initial Capability Evaluation,  TWIC Pilot Test and Evaluation Master Plan, the Initial Technical Test  Plan, the Early Operational Assessment Test Plan, the Concept of  Operations Plan, TWIC pilot scenarios, the TSA Pilot Schedule, and the  Advanced Notice of Proposal Rulemaking on TWIC Reader Requirements.  We also collected and analyzed Port Security Grant Program and the  Transit Security Grant Program awards relative to the TWIC pilot  participants to inform our understanding of the TWIC pilot funding  structure and guidance provided to TWIC pilot participants. In addition,  we reviewed relevant legislation, such as the MTSA and amendments to  MTSA made by the SAFE Port Act of 2006 to inform our review of  requirements for TWIC and the TWIC pilot specifically. We also obtained  an in person understanding of the benefits of and barriers to implementing  the pilot by conducting site visits to or interviews with officials at the 7  pilot sites. Specifically, we visited pilot participants at the Ports of Los  Angeles, Long Beach, and Brownsville, and the Port Authority of New  York and New Jersey. We also interviewed and or met with officials at  vessel operations participating in the TWIC pilot, including the Staten  Island Ferry in Staten Island, New York; Magnolia Marine Transports in  Vicksburg, Mississippi; and Watermark Cruises in Annapolis, Maryland. To  assess the viability of the TWIC pilot and better understand stakeholder  contributions within DHS, we met with officials from several components  at DHS. Specifically, we met with officials at DHS’s Office of Screening  Coordination, Science and Technology Directorate, the Coast Guard, the  Federal Emergency Management Agency, and the Transportation Security  Agency. To further enhance our understanding of the TWIC pilot  approach, we also interviewed officials at NIST and the Department of  Defense’s Naval Air Systems Command and Space and Naval Warfare  Systems Command—organizations supporting TSA in the TWIC pilot—to  discuss TWIC pilot testing approaches. We also observed testing of TWIC  readers against environmental conditions at the Naval Warfare laboratory.  In addition, we met with local Coast Guard officials and representatives  from 15 stakeholder organizations, including associations and business  owners from industries impacted by TWIC, such as longshoremen and  truck drivers. While information we obtained from the interviews with  stakeholders may not be generalized across the maritime transportation  industry as a whole, because we selected stakeholders who either  represent national associations or who operate in or access the ports  where the TWIC reader pilot will be conducted, the interviews provided us  with information on the views of individuals and organizations that will be  directly impacted by the program.
In assessing the TWIC pilot approach, we reviewed the information  obtained through these endeavors against practices we identified in  program and project management as well as program evaluation efforts  that are relevant to the TWIC program pilot. These practices were  identified based on a review of (1) guidance issued by OMB; (2) our prior  work on results oriented government, program management and  evaluation, and regulatory analysis; and (3) literature on program  management principles. Based on these recognized standards, practices,  and guidance, we    Assessed the pilot schedule against nine relevant best practices in our  Cost Estimating and Assessment Guide to determine the extent to  which the pilot schedule reflects key estimating practices that are  fundamental to having and maintaining a reliable schedule. In doing so,  we independently assessed the program’s integrated master schedule  and its underlying activities against our nine best practices. We also  interviewed cognizant program officials to discuss their use of best  practices in creating the program’s current schedule and we attended  three walk-throughs to better understand how the schedule was  constructed and maintained. To further assess the reliability of the  schedule, we compared information in the pilot schedule to  information provided by pilot participants and stakeholders.
Reviewed TWIC pilot documentation against identified characteristics  that sound evaluation plans and approaches include. We also assessed  the data to be collected from the TWIC pilot and identified  methodologies for using the data to inform Congress on the impacts of  using TWIC with biometric card readers and further informing the card  reader rule. To help assess the completeness of the TWIC pilot  approach and evaluation methodology, we compared the technology,  business, and operational potential requirements identified in the TWIC  Reader Advanced Notice of Proposed Rulemaking (ANPRM) issued on  March 27, 2009. As part of this assessment we reviewed the program  evaluation approach used by TSA and the Coast Guard for leveraging  pilot efforts and investments to the maximum extent possible for  identifying the cost and other implications on government, the private  sector, and the public at large to be considered when developing the  regulatory analysis.
We conducted this performance audit from July 2008 through November  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Key TWIC Implementation Actions
paragraphs: Table 3 below summarizes key Transportation Worker Identification  Credential (TWIC) program laws and milestones for implementing the  program through April 2009.
section_title: Appendix III: Phased-In Captain of the Port Zone Compliance Schedule (Revised February 19, 2009)
paragraphs: Appendix III: Phased-In Captain of the Port  Zone Compliance Schedule (Revised February  19, 2009)
Table 4 below illustrates the phased-in captain of the port zone  compliance schedule from October 2008 to April 2009.
section_title: Appendix IV: Scheduling Best Practices
paragraphs: Table 5 presents a summary of best practices identified by GAO for  applying a schedule as part of program management.
section_title: Appendix V: Assessment of the TWIC Pilot against the Potential TWIC Requirements under Consideration in the March 27, 2009, TWIC Advanced Notice of Proposed Rulemaking for the Card Reader Rule
paragraphs: The analysis below is a detailed review of key statements made in the  Transportation Worker Identification Credential (TWIC) Reader Advanced  Notice of Proposed Rulemaking (ANPRM) issued by the Coast Guard  compared to the items being tested in the TWIC reader pilot. The ANPRM  contains the potential TWIC reader requirements Coast Guard is  considering as part of a future regulation for MTSA-regulated facilities and  vessels required to use TWIC as an access control mechanism. The Coast  Guard notes that the ANPRM presents preliminary thoughts on potential  requirements for electronic TWIC readers in order to open the public  dialogue on implementing TWIC reader requirements. The requirements  presented in this ANPRM, represent the technology, business processes,  and operational characteristics of TWIC under consideration at the time.  Moreover, they represent potential costs and benefits—or impacts—to be  borne by the government, private sector, and population at large as a  result of the regulation being considered. The TWIC reader pilot, as  defined in the SAFE Port Act of 2006, is to test the business processes,  technology, and operational impacts required to deploy transportation  security card readers at secure areas of the marine transportation system.  Furthermore, The Department of Homeland Security (DHS) is to report on  the following results from the TWIC reader pilot: (1) the findings of the  pilot program with respect to technical and operational impacts of  implementing a transportation security card reader system; (2) any actions  that may be necessary to ensure that all vessels and facilities to which this  section applies are able to comply with such regulations; and (3) an  analysis of the viability of equipment under the extreme weather  conditions of the marine environment. The following defines the  assessment categories used below.  1.  Yes—This assessment category represents that the potential  requirement identified in the ANPRM is being tested for in the TWIC  reader pilot.  2.  Partially—This assessment category represents that the potential  requirement identified in the ANPRM is at least in part being tested for  in the TWIC reader pilot.  3.  No—This assessment category represents that the potential  requirement identified in the ANPRM is not being tested for in the  TWIC reader pilot.
section_title: Appendix VI: Comments from the Department of Homeland Security
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, David Bruno (Assistant Director),  Joseph P. Cruz (analyst-in-charge), Chuck Bausell, Tim Boatwright,  Geoffrey Hamilton, Richard Hung, Lemuel Jackson, Daniel Kaneshiro, Stan  Kostyla, Jason Lee, Linda Miller, Karen Richey, Julie E. Silvers, and Sally  Williamson made key contributions to this report.