Homeland Security: Recommendations to Improve Management of Key Border Security Program Need to Be Implemented
Published: 2006-02-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security (DHS) has established a program--the U.S. Visitor and Immigrant Status Indicator Technology (US-VISIT)--to collect, maintain, and share information, including biometric identifiers, on selected foreign nationals entering and exiting the United States. US-VISIT uses these identifiers (digital fingerscans and photographs) to screen persons against watch lists and to verify that a visitor is the person who was issued a visa or other travel document. Visitors are also to confirm their departure by having their visas or passports scanned and undergoing fingerscanning at selected air and sea ports of entry (POE). GAO has made many recommendations to improve the program, all of which DHS has agreed to implement. GAO was asked to report on DHS's progress in responding to 18 of these recommendations.
section_title: What GAO Found
paragraphs: The current status of DHS's implementation of the 18 recommendations is mixed, but progress in critical areas has been slow. DHS has implemented 2 of the recommendations: it defined program staff positions, roles, and responsibilities, and it hired an independent verification and validation contractor. It has also taken steps to implement the other recommendations, partially completing 11 and beginning to implement another 5. In September 2003, GAO reported that the program had not assessed the costs and benefits of Increment 1 (which provides entry capabilities to air and sea POEs) and recommended that the program determine whether proposed increments will produce mission value commensurate with cost. In the latest cost-benefit analysis, dated June 23, 2005, the program identified potential costs and benefits for three alternatives for an air and sea exit solution. However, the analysis does not meet key Office of Management and Budget criteria; for example, it does not include a complete uncertainty analysis, which helps to provide decision makers with perspective on the potential variability of the cost and benefit estimates should circumstances change. GAO reported in May 2004 and February 2005 that system testing was not based on well-defined test plans and recommended that before testing begins, the program develop and approve test plans meeting certain criteria. However, although the latest test plan did cover many required areas (such as the tests to be performed), it did not adequately trace between test cases and the requirements to be verified by testing. Without complete and traceable test plans, the risk is increased that the deployed system will not perform as intended. In May 2004, GAO reported that the program had not assessed its workforce and facility needs for Increment 2B (which extends entry capabilities to the 50 busiest land POEs) and recommended that it do so. Since then, the program evaluated the processing times to issue and process entry/exit forms at 3 of the 50 busiest POEs and concluded that the results showed that no additional staff and only minor facilities modifications were required. However, the scope of the evaluation was limited. Since then, DHS has deployed and implemented Increment 2B capabilities to these 50 POEs, making the collection of predeployment baseline data for these sites impractical. Nonetheless, other alternatives, such as surveying site officials about the increment's impacts, have yet to be explored. Until they are, the program may not be able to accurately project resource needs or make any needed modifications to achieve its goals of minimizing US-VISIT's impact on POE operations, which was the impetus for GAO's recommendation. DHS attributed the pace of progress to competing demands on time and resources. The longer that US-VISIT takes to implement the recommendations, the greater the risk that the program will not meet its stated goals on time and within budget.

section_title: Letter
paragraphs: The U.S. Visitor and Immigrant Status Indicator Technology (US-VISIT) is a  multibillion-dollar program of the Department of Homeland Security (DHS)  that is intended to record the entry into and exit from the United States of  selected individuals, verify their identity, and confirm their compliance  with the terms of their admission into and stay in the United States. The  goals of the program are to (1) enhance the security of our citizens and  visitors, (2) facilitate legitimate travel and trade, (3) ensure the integrity of  the U.S. immigration system, and (4) protect the privacy of our visitors.
Since fiscal year 2002, DHS has been legislatively directed to submit annual  expenditure plans for the program, and we have been directed to review  these plans and issue reports. These reports have, among other things,  identified risks that face the department in delivering promised program  capabilities and benefits on time and within cost. For example, we  reported that the program office did not have the human capital and  acquisition process discipline needed to effectively manage the program.  Because of the number and severity of program management challenges  that we identified, we concluded that the program was risky.
To address program risks, our reports have included 18 recommendations  in such areas as system acquisition process controls, economic  justification, human capital management, cost estimating, and test  management, all of which DHS has agreed to implement. Because of your  continued interest in ensuring that DHS is taking the necessary actions to  successfully implement US-VISIT, you asked us to determine the progress  being made in implementing these recommendations. To achieve this  objective, we analyzed program plans, reports, and system documentation  relative to the intent of each of our recommendations, and we interviewed  appropriate DHS and program officials. (Further details on our objective,  scope, and methodology are provided in app. I.) Our work was performed  from August 2005 through December 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: US-VISIT is a governmentwide program intended to enhance the security of  U.S. citizens and visitors, facilitate legitimate travel and trade, ensure the  integrity of the U.S. immigration system, and protect the privacy of our  visitors. Its scope includes the pre-entry, entry, status, and exit of hundreds  of millions of foreign national travelers who enter and leave the United  States at over 300 air, sea, and land POEs, and the provision of new  analytical capabilities across the overall process.
To achieve its goals, US-VISIT uses biometric information (digital  fingerscans and photographs) to verify identity. In many cases, the   US-VISIT process begins overseas at U.S. consular offices, which collect  biometric information from applicants for visas and check this information  against a database of known criminals and suspected terrorists. When a  visitor arrives at a POE, the biometric information is used to verify that the  visitor is the person who was issued the visa. In addition, at certain sites,  visitors are required to confirm their departure by undergoing US-VISIT  exit procedures—that is, having their visas or passports scanned and  undergoing fingerscanning. The exit confirmation is added to the visitor’s  travel records to demonstrate compliance with the terms of admission to  the United States. (App. III provides a detailed description of the pre-entry,  entry, status, exit, and analysis processes.)  collecting, maintaining, and sharing information on certain foreign  nationals who enter and exit the United States;  identifying foreign nationals who (1) have overstayed or violated the  terms of their admission; (2) may be eligible to receive, extend, or adjust  their immigration status; or (3) should be apprehended or detained by  law enforcement officials;  detecting fraudulent travel documents, verifying traveler identity, and  determining traveler admissibility through the use of biometrics; and  facilitating information sharing and coordination within the immigration  and border management community.
In July 2003, DHS established a program office with responsibility for  managing the acquisition, deployment, operation, and sustainment of the  US-VISIT system and its associated supporting people (e.g., Customs and  Border Protection (CBP) officers), processes (e.g., entry/exit policies and  procedures), and facilities (e.g., inspection booths and lanes), in  coordination with its stakeholders (CBP and the Department of State).
As of October 2005, about $1.4 billion has been appropriated for the  program, and, according to program officials, about $962 million has been  obligated.
subsections: section_title: Acquisition and Implementation Strategy: A Brief Description
paragraphs: DHS plans to deliver US-VISIT capability in four increments, with  Increments 1 through 3 being interim, or temporary, solutions that fulfill  legislative mandates to deploy an entry/exit system, and Increment 4 being  the implementation of a long-term vision that is to incorporate improved  business processes, new technology, and information sharing to create an  integrated border management system for the future. In Increments 1  through 3, the program is building interfaces among existing (“legacy”)  systems; enhancing the capabilities of these systems; and deploying these  capabilities to air, sea, and land POEs. These increments are to be largely  acquired and implemented through existing system contracts and task  orders.
In May 2004, DHS awarded an indefinite-delivery/indefinite-quantity prime  contract to Accenture and its partners. According to the contract, the prime  contractor will help support the integration and consolidation of processes,  functionality, and data, and it will develop a strategy to build on the  technology and capabilities already available to produce the strategic  solution, while also assisting the program office in leveraging existing  systems and contractors in deploying the interim solutions.
section_title: US-VISIT Is Being Implemented in Four Increments
paragraphs: Increment 1 concentrates on establishing capabilities at air and sea POEs.  It is divided into two parts—1 and 1B.
Increment 1 (air and sea entry) includes the electronic capture and  matching of biographic and biometric information (two digital index  fingerscans and a digital photograph) for selected foreign nationals,  including those from visa waiver countries. Increment 1 was deployed  on January 5, 2004, for individuals requiring a nonimmigrant visa to  enter the United States, through the modification of pre-existing  systems. These modifications accommodated the collection and  maintenance of additional data fields and established interfaces  required to share data among DHS systems in support of entry  processing at 115 airports and 14 seaports.
Increment 1B (air and sea exit) involves the testing of exit devices to  collect biometric exit data for select foreign nationals at 11 airports and  seaports. Three exit alternatives were pilot tested:  Kiosk—A self-service device (which includes a touch-screen  interface, document scanner, finger scanner, digital camera, and  receipt printer) that captures a digital photograph and fingerprint and  prints out an encoded receipt.
Mobile device—A hand-held device that is operated by a workstation  attendant; it includes a document scanner, finger scanner, digital  camera, and receipt printer and is used to capture a digital  photograph and fingerprint.
Validator—A hand-held device that is used to capture a digital  photograph and fingerprint, which are then matched to the  photograph and fingerprint captured via the kiosk and encoded in the  receipt.
Increment 2 focuses primarily on extending US-VISIT to land POEs. It is  divided into three parts—2A, 2B, and 2C.
Increment 2A (air, sea, and land) includes the capability to biometrically  compare and authenticate valid machine-readable visas and other travel  and entry documents issued by State and DHS to foreign nationals at all  POEs. Increment 2A was deployed on October 23, 2005, according to  program officials. It also includes the deployment by October 26, 2006,  of technology to read biometrically enabled passports from visa waiver  countries.
Increment 2B (land entry) redesigns the Increment 1 entry solution and  expands it to the 50 busiest land POEs. The process for issuing Form  I-94 was redesigned to enable the electronic capture of biographic,  biometric (unless the traveler is exempt), and related travel  documentation for arriving travelers. This increment was deployed to  the busiest 50 U.S. land border POEs as of December 29, 2004. Before  Increment 2B, all information on the Form I-94s was handwritten. The  redesigned systems electronically capture the biographic data included  in the travel document. In some cases, the form is completed by CBP  officers, who enter the data electronically and then print the form.
Increment 2C is to provide the capability to automatically, passively, and  remotely record the entry and exit of covered individuals using radio  frequency (RF) technology tags at primary inspection and exit lanes.  An RF tag that includes a unique ID number is to be embedded in each  Form I-94, thus associating a unique number with a record in the  US-VISIT system for the person holding that Form I-94. In August 2005,  the program office deployed the technology to five border crossings  (three POEs) to verify the feasibility of using passive RF technology to  record traveler entries and exits via a unique ID number embedded in  the CBP Form I-94. The results of this demonstration are to be reported  in February 2006.
Increment 3 extended Increment 2B (land entry) capabilities to 104 land  POEs; this increment was essentially completed as of December 19, 2005.
Increment 4 is the strategic US-VISIT program capability, which program  officials stated will likely consist of a further series of incremental releases  or mission capability enhancements that will support business outcomes.  The program reports that it has worked with its prime contractor and  partners to develop this overall vision for the immigration and border  management enterprise.
Increments 1 through 3 include the interfacing and integration of existing  systems and, with Increment 2C, the creation of a new system, the  Automated Identification Management System (AIDMS). The three main  existing systems are as follows:  The Arrival Departure Information System (ADIS) stores   noncitizen traveler arrival and departure data received from air and   arrival data captured by CBP officers at air and sea POEs,   Form I-94 issuance data captured by CBP officers at Increment 2B   departure information captured at US-VISIT biometric departure  pilot (air and sea) locations,   pedestrian arrival information and pedestrian and vehicle departure  information captured at Increment 2C POE locations, and  status update information provided by the Student and Exchange  Visitor Information System (SEVIS) and the Computer Linked  Application Information Management System (CLAIMS 3) (described  below).
ADIS provides record matching, query, and reporting functions.
The passenger processing component of the Treasury Enforcement  Communications System (TECS) includes two systems: Advance  Passenger Information System (APIS), a system that captures arrival  and departure manifest information provided by air and sea carriers,  and the Interagency Border Inspection System, a system that maintains  lookout data and interfaces with other agencies’ databases. CBP officers  use these data as part of the admission process. The results of the  admission decision are recorded in TECS and ADIS.
The Automated Biometric Identification System (IDENT) collects and  stores biometric data on foreign visitors.
US-VISIT also exchanges biographic information with other DHS systems,  including SEVIS and CLAIMS 3. These two systems contain information on  foreign students and foreign nationals who request benefits, such as a  change of status or extension of stay.
Some of the systems previously described, such as IDENT and the new  AIDMS, are managed by the program office, while some systems are  managed by other organizational entities within DHS. For example, TECS  is managed by CBP, SEVIS is managed by Immigration and Customs  Enforcement, CLAIMS 3 is under United States Citizenship and  Immigration Services, and ADIS is jointly managed by CBP and US-VISIT.
US-VISIT also interfaces with other, non-DHS systems for relevant  purposes, including watch list updates and checks to determine whether a  visa applicant has previously applied for a visa or currently has a valid U.S.  visa. In particular, US-VISIT receives biographic and biometric information  from State’s Consular Consolidated Database as part of the visa application  process, and returns fingerscan information and watch list changes.
section_title: Program Management Roles and Responsibilities
paragraphs: The US-VISIT program office structure includes nine component offices.  Each of the program offices includes a director and subordinate  organizational units, as established by the director. The responsibilities for  each office are stated below. Figure 1 shows the program office structure,  including its nine offices.
The roles and responsibilities for each of the nine offices include the  following:  Chief Strategist is responsible for developing and maintaining the  strategic vision, strategic documentation, transition plan, and business  case.
Budget and Financial Management is responsible for establishing the  program’s costs estimates; analysis; and expenditure management  policies, processes, and procedures that are required to implement and  support the program by ensuring proper fiscal planning and execution  of the budget and expenditures.
Mission Operations Management is responsible for developing  business and operational requirements based on strategic direction  provided by the Office of the Chief Strategist.
Outreach Management is responsible for enhancing awareness of  US-VISIT requirements among foreign nationals, key domestic  audiences, and internal stakeholders by coordinating outreach to media,  third parties, key influencers, Members of Congress, and the traveling  public.
Information Technology Management is responsible for developing  technical requirements based on strategic direction provided by the  Office of the Chief Strategist and business requirements developed by  the Office of Mission Operations Management.
Implementation Management is responsible for developing accurate,  measurable schedules and cost estimates for the delivery of mission  systems and capabilities.
Acquisition and Program Management is responsible for establishing  and managing the execution of program acquisition and management  policies, plans, processes, and procedures.
Administration and Training is responsible for developing and  administering a human capital plan that includes recruiting, hiring,  training, and retaining a diverse workforce with the competencies  necessary to accomplish the mission.
Facilities and Engineering Management is responsible for establishing  facilities and environmental policies, procedures, processes, and  guidance required to implement and support the program office.
section_title: Our Prior Work Has Resulted in Several Recommendations
paragraphs: In response to legislative mandate, we have issued four reports on DHS’s  annual expenditure plans for US-VISIT. Our reports have, among other  things, assessed whether the plans satisfied the legislative conditions and  provided observations on the plans and DHS’s program management. As a  result of our assessments, we made 24 recommendations aimed at  improving both plans and program management, all of which DHS has  agreed to implement. Of these 24 recommendations, 18 address risks  stemming from program management.
section_title: The Status of DHS’s Implementation of Our Recommendations Is Mixed
paragraphs: The current status of DHS’s implementation of our 18 recommendations on  program risks is mixed, but progress in critical areas has been slow. For  example, over 2 years have passed, and the program office has yet to  develop a security plan consistent with federal guidance or to economically  justify its investment in system increments. According to the Program  Director, the pace of progress is attributable to competing demands on  time and resources.
DHS agreed to implement all 18 recommendations. Of these 18, DHS has  completely implemented 2, has partially implemented 11, and is in the  process of implementing another 5. Of the 11 that are partially  implemented, 7 are about 2 years old, and 4 are about 10 to 19 months old.  Of the 5 that are in progress, 3 are about 10 months old.
These 18 recommendations are aimed at strengthening the program’s  management effectiveness. The longer that the program takes to  implement the recommendations, the greater the risk that the program will  not meet its goals on time and within budget.
Figure 2 provides an overview of the extent to which each recommendation  has been implemented. The figure is followed by sections providing details  on each recommendation and our assessment of its implementation status.
subsections: section_title: Development and Implementation of a Security Plan and Performance of a Privacy Impact Assessment Are Partially Complete
paragraphs: In June 2003, we reported that the Immigration and Naturalization  Service had not developed a security plan and performed a privacy impact  assessment for the entry exit program (as US-VISIT was then known). A  security plan and privacy impact assessment are important to  understanding system requirements and ensuring that the proper  safeguards are in place to protect system data and resources. System  acquisition best practices and federal guidance advocate understanding  and defining security and privacy requirements both early and continuously  in a system’s life cycle, and effectively planning for their satisfaction.  Accordingly, we recommended that DHS do the following:    Develop and begin implementing a system security plan, and perform a   privacy impact assessment and use the results of the analysis in near-term   and subsequent system acquisition decision making.
subsections: section_title: Security Plan
paragraphs: Since we made the system security plan recommendation about 2½ years  ago, its implementation has been slow. For example, we reported in  September 2003 and again in May 2004 that the program office had not  developed a security plan. In February 2005, we reported that the program  office had developed a security plan, dated September 2004, and that this  plan was generally consistent with federal guidance. That is, the plan  provided an overview of system security requirements, described the  controls in place or planned for meeting those requirements, referred to the  applicable documents that prescribe the roles and responsibilities for  managing the US-VISIT component systems, and addressed security  awareness and training. However, the program office had not conducted a  risk assessment or included in the plan when an assessment would be  completed. According to guidance from the Office of Management and  Budget (OMB), the security plan should describe the methodology that is  used to identify system threats and vulnerabilities and to assess risks, and  it should include the date the risk assessment was completed.
According to program officials, they completed a programwide risk  assessment in December 2005, but have yet to provide a copy of the  assessment to us. Therefore, we cannot confirm that the assessment has  been done, and done properly. The absence of a risk assessment and a  security plan that reflects this assessment is a significant program  weakness. Risk assessments are critical to establishing effective security  controls because they provide the basis for establishing appropriate  policies and selecting cost-effective controls to implement these policies.  Without such an assessment, US-VISIT does not have adequate assurance  that it knows the risks associated with the program and thus whether it has  implemented effective controls to address them.
Notwithstanding these limitations in the security plan, the program office  has begun to implement aspects of its September 2004 security plan. For  example, the Information Systems Security Manager told us that a security  awareness program is established and key personnel have attended  security training.
section_title: Privacy Impact Assessment
paragraphs: Since June 2003, US-VISIT has also developed and periodically updated a  privacy impact assessment. An initial impact assessment was issued in  January 2004, and a revised assessment was issued in September 2004. A  more recent assessment, dated July 2005, reflects changes related to  Increments 1B and 2C. Each of these assessments is generally consistent  with OMB guidance. That is, each of the assessments addressed most  OMB requirements, including the impact that the system will have on  individual privacy, the privacy consequences of collecting the information,  and alternatives considered to collect and handle information. The most  recent impact assessment, for example, states that three alternatives were  considered for Increment 1B—the kiosk, the mobile device, and the  validator (a combination of the two)—and discusses proposals to mitigate  the privacy risks of all three, such as by limiting the duration of data  retention on the exit devices and using encryption.
However, OMB guidance also requires that privacy impact assessments  developed for systems under development address privacy in relevant  system documentation, including statements of need, functional  requirements documents, and cost-benefit analyses. As we reported about  previous privacy impact assessments, privacy is only partially addressed in  system documentation. For example, the Increment 1B cost-benefit  analysis assesses the privacy risk associated with each exit alternative, and  the Increment 2C business requirements state that all solutions are to be  compliant with privacy laws and regulations and adhere to US-VISIT  privacy policy. However, we did not find privacy in the Increment 1B  business requirements or the Increment 2C functional requirements.  Program officials, including the US-VISIT Privacy Officer, acknowledged  that privacy is not included in the system documentation, but stated that  privacy is considered in the development of the documentation and that the  privacy office reviews key system documentation at relevant times during  the system development life cycle. Nevertheless, we did not find evidence  of privacy being addressed in the system documentation, and program  officials acknowledged that it was not included.
Until the program performs a risk assessment and fully implements a  security plan that reflects this assessment, it cannot adequately ensure that  US-VISIT is cost-effectively safeguarding assets and data. Moreover,  without reflecting privacy in system documentation, it cannot adequately  ensure that privacy needs are being fully addressed.
section_title: Development and Implementation of Key Acquisition Controls Are Partially Complete
paragraphs: We reported in September 2003 that the program office had not defined  key acquisition management controls to support the acquisition of  US-VISIT, and therefore its efforts to acquire, deploy, operate, and maintain  system capabilities were at risk of not satisfying system requirements and  of not meeting benefit expectations on time and within budget.
The Capability Maturity Model–Integration® (CMMI) developed by  Carnegie Mellon University’s Software Engineering Institute (SEI)  explicitly defines process management controls that are recognized  hallmarks of successful organizations and that, if implemented effectively,  can greatly increase the chances of successfully acquiring  software-intensive systems. SEI’s CMMI model uses capability levels to  assess process maturity. Because establishing the basic acquisition  process capabilities, according to SEI, can take on average about 19  months, we recognized the importance of starting early to build effective  acquisition management capabilities by recommending that DHS do the  following:  Develop and implement a plan for satisfying key acquisition management   controls, including acquisition planning, solicitation, requirements   management, program management, contract tracking and oversight,   evaluation, and transition to support, and implement the controls in   accordance with SEI guidance.
The program office has recently taken foundational steps to establish key  acquisition management controls. For example, it has developed a process  improvement plan, dated May 16, 2005 (about 20 months after our  recommendation), to define and implement these controls. As part of its  improvement program, the program office is implementing a governance  structure for overseeing improvement activities, consisting of three groups:  a Management Steering Group, an Enterprise Process Group, and Process  Action Teams. Specific roles for each of these groups are described below.
The Management Steering Group is to provide policy and procedural  guidance and to oversee the entire improvement program. The steering  group is chaired by the US-VISIT Director, with the Deputy Director and  the functional office directors serving as core members.
The Enterprise Process Group is to provide planning, management, and  operational guidance in day-to-day process improvement activities. The  group is chaired by the process improvement leader and is composed of  individuals from each functional office.
Process Action Teams are to provide specific process documentation  and to provide implementation support and training services. These  teams are to be active as long as a particular process improvement  initiative is under way. To date, the program office has chartered five  process teams—configuration management, cost analysis, process  development, communications, and policy.
In addition, the program office has recently completed a self-assessment of  its acquisition process maturity, and it plans to use the assessment results  to establish a baseline of its acquisition process maturity for improvement.  According to program officials, the assessment included 13 key process  areas that are generally consistent with the process areas cited in our  recommendation. The program has ranked these 13 process areas  according to their priority, and, for initial implementation, it plans to focus  on the following 6:   Configuration management. Establishing and maintaining the integrity  of the products throughout their life cycle.
Process and product quality assurance. Taking actions to provide  management with objective insight into the quality of products and  processes.
Project monitoring and control. Tracking the project’s progress so that  appropriate corrective actions can be taken when performance deviates  significantly from plans.
Project planning. Establishing and maintaining plans for work  activities.
Requirements management. Managing the requirements and ensuring a  common understanding of the requirements between the customer and  the product developers.
Risk management. Identifying potential problems before they occur so  that they can be mitigated to minimize any adverse impact.
The improvement plan is currently being updated to reflect the results of  the baseline assessment and to include a detailed work breakdown  structure, process prioritization, and resource estimates. According to the  Director, Acquisition and Program Management Office (APMO), the goal is  to conduct a formal SEI appraisal to assess the capability level of some or  all of the six processes by October 2006.
Notwithstanding the recent steps to begin addressing our recommendation,  much work remains to fully implement key acquisition management  controls. Moreover, effectively implementing these controls takes  considerable time. Therefore, it is important that these improvement  efforts stay on track. Until these processes are effectively implemented,  US-VISIT will be at risk of not delivering promised capabilities on time and  within budget.
section_title: Determination and Disclosure of Whether Increments Produce Mission Value Commensurate with Costs and Risks Are Partially Complete
paragraphs: In September 2003, we reported that the program had not assessed the  costs and benefits of Increment 1, which is extremely important because  the decision to invest in any capability should be based on reliable analyses  of return on investment. Further, according to OMB guidance, individual  increments of major systems are to be individually supported by analyses  of benefits, cost, and risk. Without reliable analyses, an organization  cannot adequately know that a proposed investment is a prudent and  justified use of limited resources. Accordingly, we recommended that DHS  do the following:  Determine whether proposed US-VISIT increments will produce mission   value commensurate with cost and risks and disclose to the Congress   planned actions.
As we reported in September 2003 and again in February 2005, the  program office did not justify its planned investment in Increments 1 and  2B, respectively, based on expected return on investment. Since then, the  program has developed a cost-benefit analysis for Increment 1B.
OMB has issued guidance concerning the analysis needed to justify  investments. According to this guidance, such analyses should meet  certain criteria to be considered reasonable. These criteria include, among  other things, comparing alternatives on the basis of net present value and  conducting uncertainty analyses of costs and benefits. DHS has also issued  guidance on such economic analyses that is consistent with that of OMB.
The latest cost-benefit analysis for Increment 1B (dated June 23, 2005)  identifies potential costs and benefits for three exit solutions at air and sea  POEs and provides a general rationale for the viability of the three  alternatives described. This latest analysis meets four of eight OMB  economic analysis criteria. However, it does not, for example, include a  complete uncertainty analysis (i.e., both a sensitivity analysis and a Monte  Carlo simulation) for the three exit alternatives evaluated. That is, the  cost-benefit analysis does include a Monte Carlo simulation, but it does not  include a sensitivity analysis for the three alternatives. An analysis of  uncertainty is important because it provides decision makers with a  perspective on the potential variability of the cost and benefit estimates  should the facts, circumstances, and assumptions change.
Table 1 summarizes our analysis of the extent to which US-VISIT’s June 23,  2005, cost-benefit analysis for Increment 1B satisfies eight OMB criteria.
It is important that the program adhere to relevant guidance in developing  its incremental cost-benefit analyses. If this is not done, the reliability of  the analyses is diminished, and an adequate basis for prudent investment  decision making does not exist. Moreover, if the mission value of a  proposed investment is not commensurate with costs, it is vital that this  information be fully disclosed to DHS and congressional decision makers.  The underlying intent of our recommendation is that this information be  available to inform such decisions.
section_title: Definition of the Operational Context for US-VISIT Is in Progress
paragraphs: In September 2003, we reported that key aspects of the larger homeland  security environment in which US-VISIT would need to operate had not  been defined. For example, we stated that certain policy and standards  decisions had not been made (e.g., whether official travel documents will  be required for all persons who enter and exit the country, including U.S.  and Canadian citizens, and how many fingerprints are to be collected). In  the absence of this operational context, program officials were making  assumptions and decisions that, if they proved inconsistent with  subsequent policy or standards decisions, would require US-VISIT rework.
To minimize the impact of these changes, we recommended that DHS do  the following:  Clarify the operational context in which US-VISIT is to operate.
After about 27 months, defining this operational context remains a work in  progress. According to the Chief Strategist, an immigration and border  management strategic plan was drafted in March 2005 that shows how  US-VISIT is aligned with DHS’s organizational mission and defines an  overall vision for immigration and border management. This official stated  that this vision provides for an immigration and border management  enterprise that unifies multiple internal departmental and other external  stakeholders with common objectives, strategies, processes, and  infrastructures.
Since the plan was drafted, DHS has reported that other relevant initiatives  have been undertaken, such as the Security and Prosperity Partnership of  North America and the Secure Border Initiative. The Security and  Prosperity Partnership is to, among other things, establish a common  approach to securing the countries of North America— the United States,  Canada, and Mexico—by, for example, implementing a border facilitation  strategy to build capacity and improve the legitimate flow of people and  cargo at our shared borders. The Secure Border Initiative is to implement a  comprehensive approach to securing our borders and reducing illegal  immigration. According to the Chief Strategist, while portions of the  strategic plan are being incorporated into these initiatives, these initiatives  and their relationship with US-VISIT are still being defined. We have yet to  receive the US-VISIT strategic plan because, according to program officials,  it had not yet been approved by DHS management.
Until US-VISIT’s operational context is fully defined, DHS is increasing its  risk of defining, establishing, and implementing a program that is  duplicative of other programs and not interoperable with them. This in turn  will require rework to address these areas. While this issue was significant  27 months ago, when we made the recommendation, it is still more  significant now.
section_title: Provision of Program Office Resources Is Partially Complete
paragraphs: We reported in September 2003 that the program had not fully staffed its  program office. Our prior experience with major acquisitions like US-VISIT  shows that to be successful, they need, among other things, to have  adequate resources. Accordingly, we recommended that DHS do the  following:  Ensure that human capital and financial resources are provided to   establish a fully functional and effective program office.
About 2 years later, US-VISIT had filled 102 of its 115 planned government  positions and all of its planned 117 contractor positions. For the remaining  13 government positions, 5 positions had been selected (pending  completion of security clearances), and recruitment action was in process  for filling the remaining 8 vacancies. According to the Office of  Administration and Training Manager, funding is available to complete the  hiring of all 115 government employees.
Notwithstanding this progress, in February 2005, US-VISIT completed a  workforce analysis and requested additional positions based on the results.  According to program officials, a revised analysis was submitted in the  summer of 2005, but the request has not yet been approved. Figure 3 shows  the program office organization structure and functions and how many of  the 115 positions needed have been filled.
Securing necessary resources will be a continuing challenge and an  essential ingredient to the program’s ability to acquire, deploy, operate, and  maintain system capabilities on time and within budget.
section_title: Definition of Program Office Roles and Responsibilities Has Been Completed
paragraphs: We reported in September 2003 that the program had not defined specific  roles and responsibilities for its staff. Our prior experience and leading  practices show that for major acquisitions like US-VISIT to be successful,  program staff need, among other things, to understand what they are to do,  how they relate to each other, and how they fit in their organization.  Accordingly, we recommended that DHS do the following:   Define program office positions, roles, and responsibilities.
The program office has developed charters for its nine component offices  that include roles and responsibilities for each. For example, the  Acquisition and Program Management Office is responsible, among other  things, for establishing acquisition and program management policies;  coordinating development of configuration management plans and project  schedules, including the integrated milestone schedule; and developing  policies and procedures for guidance and oversight of systems  development and implementation activities. The program has also defined  a set of core competencies (knowledge, skills, and abilities) for each  position. For example, it has defined critical competencies for program and  management analysts that include, among others, flexibility, interpersonal  skills, organizational awareness, oral communication, problem solving, and  teamwork.
These efforts to define position, roles, and responsibilities should help in  managing the program effectively.
section_title: Development and Implementation of a Human Capital Strategy Are Partially Complete
paragraphs: As previously stated, we reported in September 2003 that US-VISIT had not  fully staffed its program office or defined roles and responsibilities for its  program staff. We observed that prior research and evaluations of  organizations showed that effective human capital management can help  agencies establish and maintain the workforce they need to accomplish  their missions. Accordingly, we recommended that DHS do the following:  Develop and implement a human capital strategy for the program office   that provides for staffing positions with individuals who have the   appropriate knowledge, skills, and abilities.
In February 2005, we reported that the program office, in conjunction with  the Office of Personnel Management (OPM), developed a draft human  capital plan that employed widely accepted human capital planning tools  and principles. The draft plan included, for example, an action plan that  identified activities, proposed completion dates, and the office (OPM or the  program office) responsible for the action. We also reported that the  program office had completed some of the activities, such as designating a  liaison responsible for ensuring alignment between departmental and  program human capital policies.
Since then, the program office has finalized the human capital plan and  completed more activities. For example, program officials told us that they  have  analyzed the program office’s workforce to determine diversity trends,  retirement and attrition rates, and mission-critical and leadership  competency gaps;  updated the program’s core competency requirements to ensure  alignment between the program’s human capital and business needs;  developed an orientation program for new employees; and  administered competency assessments to incoming employees.
Program officials also told us that they have plans to complete other  activities, such as  developing a staffing forecast to inform succession planning;  analyzing workforce data to maintain strategic focus on preserving the  skills, knowledge, and leadership abilities required for the US-VISIT  program’s success; and  developing organizational leadership competency models for the  program’s senior executive, managerial, and supervisory levels.
In addition, the officials said that several activities in the plan have not  been completed, such as assessing the extent of any current employees’  competency gaps and developing a competency-based listing of training  courses. These officials said that the reason these activities have not been  completed is that they are related to the department’s new human capital  initiative, MAXHR, which is to provide greater flexibility and accountability  in the way employees are paid, developed, evaluated, afforded due process,  and represented by labor organizations. MAXHR is to include the  development of departmentwide competencies. Because of this, the  officials told us that it could potentially impact the program’s ongoing  competency-related activities. As a result, these officials said that they are  coordinating these activities closely with the department as it develops and  implements this new initiative, which is currently being reviewed by the  DHS Deputy Secretary for approval.
Until US-VISIT fully implements a comprehensive human capital strategy, it  will continue to risk not having staff with the right skills and abilities to  successfully execute the program.
section_title: Defining Performance Standards for US-VISIT Increments Is Partially Complete
paragraphs: We reported in September 2003 that the operational performance of initial  system increments was largely dependent on the performance of existing  systems that were to be interfaced to create these increments. For  example, we said that the performance of an increment will be constrained  by the availability and downtime of the existing systems that it includes.  Accordingly, we recommended that DHS do the following:  Define performance standards for each increment that are measurable   and reflect the limitations imposed by relying on existing systems.
In February 2005 (17 months later), we reported that several technical  performance standards for Increments 1 and 2B had been defined, but that  it was not clear that these standards reflected the limitations imposed by  the reliance on existing systems. Since then, for the Increment 2C Proof of  Concept (Phase 1), the program office has defined certain other  performance standards. For example, the functional requirements  document for Increment 2C (Phase 1) defines several technical  performance standards, including reliability, recoverability, and availability.  For each, the document states that the performance standard is largely  dependent on those of Increment 2B. More specifically, the document  states that Phase 1 system availability is largely dependent upon the  individual and collective availability of the current systems. The document  also states that the Increment 2C components shall have an aggregated  availability greater than or equal to 97.5 percent. However, the document  does not contain sufficient information to determine whether these  performance standards actually reflect the limitations imposed by reliance  on existing systems.
To further develop performance standards, the program office has  prepared a Performance Engineering Plan, dated March 31, 2005, that links  US-VISIT performance engineering activities to its System Development  Life Cycle. Further, the plan (1) provides a framework to be used to align  its business, application, and infrastructure performance goals and  measures; (2) describes an approach to translate business goals into  operational measures, and then to quantitative metrics; and (3) identifies  system performance measurement areas (effectiveness, efficiency,  reliability, and availability). According to program officials, they intend to  establish a group to develop action plans for implementing the engineering  plan, but did not have a time frame for doing so.
Without defining performance standards that reflect the limitations of the  existing systems upon which US-VISIT relies, the program lacks the ability  to identify and effectively address performance shortfalls.
section_title: Development and Implementation of a Risk Management Plan Are Partially Complete
paragraphs: In September 2003, we reported that US-VISIT was a risky undertaking  because of several factors inherent to the program, such as its large scope  and complexity, as well as because of various program management  weaknesses. We concluded that these risks, if not effectively managed,  would likely cause program cost, schedule, and performance problems.
Risk management is a continuous, forward-looking process that is intended  either to prevent such problems from occurring or to minimize their impact  if they occur by proactively identifying risks, implementing risk mitigation  strategies, and measuring and disclosing progress in doing so. Because of  the importance of effectively managing program risks, we recommended   that DHS do the following:  Develop and implement a risk management plan and ensure that all high   risks and their status are reported regularly to the executive body.
About 2 years later, the program office has developed and has begun  implementing a risk management plan. The plan, which was approved in  September 2005, includes, among other things, a process for identifying,  analyzing, handling, and monitoring risk. It also defines the governance  structure to be used in overseeing and managing the process. The program  also maintains a risk database, which includes, among other things, a  description of the risk, its priority (e.g., high, medium, or low), and its  mitigation strategy. According to program officials, the database is  currently available to program management and staff.
The program has also begun implementing its risk management plan. For  example, it has established a Risk Review Board, Risk Review Council, and  Risk Owners to govern its risk activities. The roles and responsibilities are  described below.
The Risk Review Board directs all risk governance within the program  and provides the mechanism to escalate/transfer the consideration of  risks to program governing boards and to organizations external to the  program.
The Risk Review Council oversees and manages program-related risks  that are significant, controversial, or cross-project or that may require  escalation to the Risk Review Board.
Risk Owners analyze, handle, and monitor risks.
However, full implementation of the risk management plan has yet to  occur. As part of its CMMI process maturity baseline self-assessment  (previously discussed), the program office found that the risk management  process detailed in its plan was not being consistently applied across the  program. In response, according to program officials, they have developed  risk management training and began conducting training sessions in  November 2005. These officials also stated that the Risk Review Board,  where risks are reviewed with program executives, has been meeting  monthly since September 2005.
With respect to regular risk reports to program executives, the plan  includes thresholds for escalating risks within the risk governance  structure and to DHS governance entities. For example, risks are to be  elevated to the Risk Review Board when the cost of the project exceeds  more than 5 percent of the project baseline cost, the schedule slippage  exceeds more than 5 percent of the baseline schedule, major areas of scope  are affected, or quality reduction requires approval. However, program  officials stated that these thresholds are not currently being applied. They  further stated that although the plan allows for escalation of risks to  officials outside the program office, doing so is at the discretion of the  Program Director; in addition, according to these officials, although high  risks are not routinely escalated outside the program, selected high risks  have been disclosed to the Assistant Secretary for Policy in weekly  program status reports. As of December 5, 2005, the Program Director  proposed submitting monthly reports of high-priority risks and issues  through the Assistant Secretary for Policy to the Deputy Secretary.
Until US-VISIT fully implements its risk management plan and process, it  cannot be assured that all program risks are being identified and managed  in order to effectively mitigate any negative impact on the program’s ability  to deliver promised capabilities on time and within budget.
section_title: Development of Test Plans Is Partially Complete
paragraphs: We reported in May 2004, and again in February 2005, that system testing  was not based on well-defined test plans, and thus the quality of testing  being performed was at risk. The purpose of system testing is to identify  and correct system defects (i.e., unmet system functional, performance,  and interface requirements) and thereby obtain reasonable assurance that  the system performs as specified before it is deployed and operationally  used. To be effective, testing activities should be planned and implemented  in a structured and disciplined fashion. Among other things, this includes  developing effective test plans to guide the testing activities and ensuring  that test plans are developed and approved before test execution.  According to relevant systems development guidance, an effective test plan  (1) specifies the test environment; (2) describes each test to be performed,  including test controls, inputs, and expected outputs; (3) defines the test  procedures to be followed in conducting the tests; and (4) provides  traceability between the test cases and the requirements to be verified by  the testing. Because these criteria were not being met, we recommended  that DHS do the following:  Develop and approve test plans before testing begins that (1) specify the   test environment; (2) describe each test to be performed, including test   controls, inputs, and expected outputs; (3) define the test procedures to   be followed in conducting the tests; and (4) provide traceability between   test cases and the requirements to be verified by the testing.
About 19 months later, the quality of the system test plans, and thus system  testing, is still problematic. To the program’s credit, the test plans for the  Increment 2C Proof of Concept (Phase 1), dated June 28, 2005, satisfied  part of our recommendation. Specifically, the test plan for this increment  was approved on June 30, 2005, and, according to program officials, testing  began on July 5, 2005. Further, the test plan described, for example, the  scope, complexity, and completeness of the test environment, and it  described the tests to be performed, including a high-level description of  controls, inputs, and outputs, and it identified test procedures to be  performed.
However, the test plan did not adequately trace between test cases and the  requirements to be verified by testing. For example, 300 of the 438  functional requirements, or about 70 percent of the requirements that we  analyzed, did not have specific references to test cases.
In addition, we identified traceability inconsistencies, including the  following:  One requirement was mapped to over 50 test cases, but none of the 50  cases referenced the requirement.
One requirement was mapped to a group of test cases in the traceability  matrix, but several of the test cases to which the requirement was  mapped did not reference the requirement, and several test cases  referenced the requirement and were not included in the traceability  matrix.
One requirement was mapped to all but one of the test cases within a  particular group of test cases, but that test case did refer to the  requirement.
Time and resources were identified as the reasons that test plans have not  been complete. Specifically, program officials stated that milestones do not  permit existing testing/quality personnel the time required to adequately  review testing documents. According to these officials, even when the  start of testing activities is delayed because, for example, requirements  definition or product development takes longer than anticipated, testing  milestones are not extended.
Without complete test plans, the program does not have adequate  assurance that the system is being fully tested, and thus unnecessarily  assumes the risk that system defects will not be detected and addressed  before the system is deployed. This means that the system may not perform  as intended when deployed, and defects will not be addressed until late in  the systems development cycle, when they are more difficult and  time-consuming to fix. As we previously reported, this has happened:  postdeployment system interface problems surfaced for Increment 1, and  manual work-arounds had to be implemented after the system was  deployed.
section_title: Assessment of the Impact of Increment 2B on Workforce Levels and Facilities Is Partially Complete
paragraphs: We reported in May 2004 that the program had not assessed its workforce  and facility needs for Increment 2B. Because of this, we questioned the  validity of the program’s workforce and facility assumptions used to  develop its workforce and facility plans, noting that the program lacked a  basis for determining whether its assumptions and thus its plans were  adequate. Accordingly, we recommended that DHS do the following:  Assess the full impact of Increment 2B on land POE workforce levels and   facilities, including performing appropriate modeling exercises.
Seven months later, the program office evaluated Increment 2B operational  performance. The purpose of the evaluation was to determine the  effectiveness of Increment 2B performance at the 50 busiest land POEs. To  assist in the evaluation, the program office established a baseline for  comparing the average Form I-94 or Form I-94W issuance processing  times at 3 of the 50 POEs where processing times were to be evaluated.  The program office then conducted two evaluations of the processing times  at the 3 POEs following Increment 2B deployment. The first was in  December 2004, after Increment 2B was deployed to these sites as a pilot,  and the second was in February 2005, after Increment 2B was deployed to  all 50 POEs. The evaluation results showed that the average processing  times decreased for all 3 sites. Table 2 compares the results of the two  evaluations and the baseline.
According to program officials, these evaluations supported the workforce  and facilities planning assumption that no additional staff were required to  support deployment of Increment 2B, and that minimal modifications to  interior workspace were required to accommodate biometric capture  devices and printers and to install electrical circuits. These officials stated  that modifications to existing officer training and interior space were the  only changes needed.
However, the scope of the evaluation was too limited to satisfy the  evaluation’s stated purpose or our recommendation for assessing the full  impact of Increment 2B. Specifically, program officials stated that the  evaluation focused on the time to process Form I-94s and not on  operational effectiveness, including workforce impacts and traveler  waiting time. Second, the 3 sites were selected, according to program  officials, on the basis of a number of factors, including whether the sites  already had sufficient staff to support the pilot. Selecting sites on the basis  of this factor could affect the results and presupposes that not all POEs  have the staff needed to support Increment 2B. Third, evaluation  conditions were not always held constant. For example, fewer  workstations were used to process travelers in establishing the baseline  processing times at 2 of the POEs—Port Huron (9 versus 14) and Douglas  (4 versus 6)—than were used during the pilot evaluations.
Moreover, CBP officials from 1 POE, which was not an evaluation site, told  us that US-VISIT has actually lengthened processing times. (San Ysidro  processes the highest volume of travelers of all land POEs.) While these  officials did not provide specific data to support this statement, it  nevertheless raises questions about the potential impact of Increment 2B  on the 47 sites that were not evaluated.
It is important that the impact of Increment 2B on workforce and facilities  be fully assessed. Since we made our recommendation, Increment 2B  deployment and operational facts and circumstances have materially  changed, making the implementation of our recommendation using  predeployment baseline data for the other 47 sites impractical.  Nevertheless, other alternatives, such as surveying officials at these sites to  better understand the increment’s impact on workforce levels and  facilities, have yet to be explored. Until they are, the program may not be  able to accurately project resource needs or make required modifications  to achieve its goals of minimizing US-VISIT’s impact on POE processing  times.
section_title: Implementation of Configuration Management Practices Is in Progress
paragraphs: We reported in May 2004 that US-VISIT had not established effective  configuration management practices. Configuration management  establishes and maintains the integrity of system components and items  (e.g., hardware, software, and documentation). A key ingredient is a change  control board to evaluate and approve proposed configuration changes.  Accordingly, we concluded that the program did not have adequate  assurance that approved system changes were actually made, and that  changes made to the component systems (for non–US-VISIT purposes) did  not interfere with US-VISIT functionality. Accordingly, we recommended  that DHS do the following:  Implement effective configuration management practices, including   establishing a US-VISIT change control board to manage and oversee   system changes.
After 19 months, US-VISIT has begun implementing configuration  management practices. To its credit, the program recently issued a  configuration management policy (September 2005) and prepared a draft  configuration management plan (August 2005). The policy contains guiding  principles, direction, and expectations for planning and performing  configuration management, and includes activities, authorities, and  responsibilities. The draft plan describes the configuration management governance structure, including organizational entities and their  responsibilities, the processes and procedures to be applied, and how  controls are to be applied to products. The governance structure includes  the Executive Configuration Control Board and the Configuration  Management Impact Review Team. According to its charter, the  configuration control board is responsible for determining the status of  requested configuration changes and resolving any conflicts related to  those changes for US-VISIT–managed systems (i.e., not for US-VISIT  component systems managed by other DHS organizations). The Impact  Review Team, which reports to the board, is responsible for reviewing  requests for system changes and submitting a recommendation to the  appropriate change review authority (i.e., either the US-VISIT control  board or the control board in the DHS organization that manages the  component system). According to program officials, for  US-VISIT–managed systems, the review authority is the Executive  Configuration Control Board. For other systems, such as TECS (which CBP  manages), the US-VISIT review team may submit a recommendation to the  appropriate control board (in this case, the CBP Control Board).
The APMO director stated that the planned configuration management  program is intended to complement rather than replace the configuration  management programs for the legacy systems. That is, change requests  approved by the US-VISIT Executive Configuration Control Board that  require changes to a legacy system will be coordinated with the board  having responsibility for that system. This means, however, that changes to  component systems (e.g., IDENT, ADIS, and TECS) that are initiated and  approved by another DHS organization, and that could affect US-VISIT  performance, are not subject to US-VISIT configuration management  processes and are not also being examined and approved by the US-VISIT  control board. This lack of US-VISIT control was the impetus for our  recommendation.
Although US-VISIT has recently taken steps to begin addressing our  recommendation, the program still does not adequately control changes to  the component systems upon which US-VISIT performance depends. Until  programwide configuration management practices are implemented, the  program does not have an effective means for ensuring that approved  system changes are actually made and that changes made to the  component systems for non–US-VISIT purposes do not compromise  US-VISIT functionality and performance.
section_title: Efforts to Ensure the Independence of the Verification and Validation Contractor Are Complete
paragraphs: We reported in May 2004 that the program office’s independent verification  and validation (IV&V) contractor was not independent of the products and  processes that it was verifying and validating. The purpose of IV&V is to  provide management with objective insight into the program’s processes  and associated work products. Its use is a recognized best practice for  large and complex system development and acquisition projects like  US-VISIT. To be effective, the verification and validation function is to be  performed by an entity that is independent of the processes and products  that are being reviewed. Accordingly, we recommended that DHS do the  following:  Ensure the independence of the IV&V contractor.
In July 2005, the program office issued a new contract for IV&V services. To  ensure the contactor’s independence, the program office (1) required that  IV&V contract bidders be independent of the development and integration  contractors; (2) reviewed each of the bidder’s affiliations with the prime  contract; (3) included provisions in the contract that prohibit the  contractor from soliciting, proposing, or being awarded work (other than  IV&V services) for the program; (4) required all contractor personnel to  certify that they do not have any conflicts of interest; and (5) ensured that  the contractor’s management plan (Oct. 17, 2005) describes how the  contractor will ensure technical, managerial, and financial independence.
Such steps, if effectively enforced, should adequately ensure that  verification and validation activities are performed in an objective manner  and, thus, should provide valuable assistance to program managers and  decision makers.
section_title: Development of a Plan to Address Open Recommendations Is Partially Complete
paragraphs: We reported in May 2004 that US-VISIT’s overall progress on implementing  our recommendations had been slow, and considerable work remained to  fully address them. As we also noted, given that most of our  recommendations focused on fundamental limitations in US-VISIT’s ability  to manage the program, it was important to implement the  recommendations quickly and completely. Accordingly, we recommended  that DHS do the following:  Develop a plan, including explicit tasks and milestones, for implementing   all of our open recommendations and periodically report to the DHS   Secretary and Under Secretary on progress in implementing this plan;    and report this progress, including reasons for delays, in all future     expenditure plans.
About 19 months after our recommendation, the program assigned  responsibility to specific individuals for preparing a plan, including specific  actions and milestones, to address each recommendation. In addition, it  developed a report that identifies the responsible person for each  recommendation and summarizes progress made in implementing each.  The program office provided this report for the first time to the DHS  Deputy Secretary on October 3, 2005, and plans to forward subsequent  reports every 6 months.
However, the report’s description of progress on 4 recommendations is  inconsistent with our assessment, as discussed below:  First, the report states that the program completed a privacy impact  assessment that is in full compliance with OMB guidance. As previously  discussed, an assessment has been developed, but OMB guidance  requires that these assessments for systems under development (such  as Increment 2C) address privacy in the system’s documentation.  Increment 2C systems documentation does not address privacy and  therefore is not fully compliant with OMB guidance.
Second, the report states that a human capital strategy has been  completed. However, as previously discussed, several of the activities in  the human capital plan have yet to be implemented. For example, the  program has not developed a staffing forecast to inform succession  planning.
Third, the report states that the impact of Increment 2B on land POE  workforce levels and facilities has been fully assessed. However, as we  previously stated, the scope of the evaluations was not sufficient to  satisfy our recommendation. For example, program officials stated that  the evaluation focused on the time to process Form I-94s and not on  operational effectiveness, including workforce impacts and traveler  waiting time. Moreover, officials at the largest land POE told us that the  effect of Increment 2B was the opposite of that reported in the pilot  results.
Fourth, the report states that the program has partially completed  implementing configuration management practices. However, as  previously discussed, the program office has yet to implement practices  or establish a configuration control board with authority over all  changes affecting US-VISIT functionality and performance, including  those made to component systems for non–US-VISIT purposes, which  was the intent of our recommendation.
In addition, the report does not specifically describe progress against 11 of  our other recommendations, so that we could not determine whether the  program’s assessment is consistent with ours (described in this report). For  example, we recommended that the program reassess plans for deploying  an exit capability to ensure that the scope of the exit pilot provides for  adequate evaluation of alternative solutions. The report states that the  program office has completed exit testing and has forwarded the exit  evaluation report to the Deputy Secretary for a decision. However, it does  not state whether the program office had expanded the scope or time  frames of the pilot.
Fully understanding and disclosing progress against our recommendations  are essential to building the capability needed to effectively manage the  program, and to ensuring that key decision makers have the information  needed to make well-informed choices among competing investment  options.
section_title: Establishment of Effective Cost-Estimating Practices Is in Progress
paragraphs: We reported in February 2005 that US-VISIT had not followed effective  practices to develop cost estimates for its system increments, and thus the  reliability of its cost estimates was questionable. Such cost-estimating  practices are embedded in the 13 criteria in SEI’s checklist for determining  the reliability of cost estimates. Of these 13 criteria, we reported in  February 2005 that the program’s cost estimate met 2, partially met 6, and  did not meet 5. Accordingly, we recommended that DHS do the following:  Follow effective practices for estimating the costs of future increments.
The latest US-VISIT–related cost estimate is for Increment 1B. This  estimate is in the June 2005 cost-benefit analysis for Increment 1B and  establishes the costs associated with three exit solutions for air and sea  POEs. As was the case for the estimate described in our February 2005  report, this latest estimate also did not meet all 13 criteria, meeting 3 and  partially meeting another 5. For example, these estimates did not include  a detailed work breakdown structure and omitted important cost elements,  such as system testing. A work breakdown structure serves to organize and  define the work to be performed, so that associated costs can be identified  and estimated. Thus, it provides a reliable basis for ensuring that the  estimates include all relevant costs. In addition, the uncertainties  associated with the Increment 1B cost estimate were not identified. An  uncertainty analysis provides the basis for adjusting these estimates to  reflect unknown facts and circumstances that could affect costs and  identifies the risk associated with the cost estimate. Table 3 summarizes  our analysis of the extent to which US-VISIT’s Increment 1B cost estimates  satisfy SEI’s 13 criteria.
Program officials stated that they recognize the importance of developing  reliable cost estimates and have initiated actions to more reliably estimate  the costs of future increments. For example, as part of its process  improvement program, the program has chartered a cost-analysis process  action team, which is to develop, document, and implement a cost-analysis  policy, process, and plan for the program. Program officials also stated that  they have hired additional contracting staff with cost-estimating  experience.
Strengthening the program’s cost-estimating capability is extremely  important. The absence of reliable cost estimates, among other things,  prevents the development of reliable economic justification for program  decisions and impedes effective performance measurement.
section_title: Reassessment of Plans for Deploying the Exit Capability Is Partially Complete
paragraphs: In February 2005, we reported that US-VISIT had not adequately planned  for evaluating the Increment 1B exit alternative because its exit pilot  evaluation’s scope and timeline were compressed. Accordingly, we  recommended that DHS do the following:  Reassess plans for deploying an exit capability to ensure that the scope    of the exit pilot provides for adequate evaluation of alternative solutions   and better ensures that the exit solution selected is in the best interest of   the program.
Over the last 10 months, the program office has taken actions to expand the  scope and time frames of the pilot. For example, it extended the pilot from  5 to 11 POEs—9 airports and 2 seaports. It also extended the time frame  for data collection and evaluation to April 2005, which is about 7 months  beyond the date for which all exit pilot evaluation tasks were to be  completed. Further, according to program officials, they achieved the  target sample sizes necessary to have a 95 percent confidence level.
Notwithstanding the expanded scope of the pilot, questions remain about  whether the exit alternatives have been evaluated sufficiently to permit  selection of the best exit solution for national deployment. For example,  each of the three exit alternatives was evaluated against three criteria,  including compliance with the US-VISIT exit process (i.e., foreign travelers  providing information as they exit the United States). However, across the  three alternatives, the average compliance with this process was only 24  percent, which raises questions as to the effectiveness of the three  alternatives. The evaluation report cites several reasons for the low  compliance rate, including that compliance during the pilot was voluntary.  The report further concludes that national deployment of the exit solution  will not have the desired compliance rate unless the exit process  incorporates an enforcement mechanism, such as not allowing persons to  reenter the United States if they do not comply with the exit process.  Although an enforcement mechanism might indeed improve compliance,  program officials stated that no formal evaluation has been conducted of  enforcement mechanisms or their effect on compliance. The program  director stated that he agrees that additional evaluation is needed to assess  the impact of implementing potential enforcement mechanisms and plans  to do so.
Until the program office adequately evaluates the exit alternatives and  knows whether the alternative to be selected will be effective, the program  office will not be in a position to select the exit solution that is in the best  interest of the program. This is very important because without an effective  exit capability, the benefits and the mission value of US-VISIT are greatly  diminished.
section_title: Development and Implementation of Capacity Management Processes Are in Progress
paragraphs: We reported in February 2005 that the overall capacity of the system was  not being effectively managed. At that time, US-VISIT, which comprises  several legacy systems, was relying on the capacity management activities  of these systems. It was not focused on the capacity requirements and  performance of the collective systems that make up US-VISIT. This  approach increases the risk that the system may not be properly designed  and configured for efficient performance, and that it has insufficient  processing and storage capacity for current, future, and unpredictable  workload requirements. Accordingly, we recommended that DHS do the  following:  Develop and implement processes for managing the capacity of the    US-VISIT system.
According to program officials, they have initiated efforts to develop a  capacity management process, including a high-level description of the  necessary steps, such as identifying tools needed to implement the process.  However, a plan, including specific tasks and milestones for developing and  implementing capacity management processes, has not yet been  developed.
Until the program office develops a programwide capacity management  program, it increases the risk that US-VISIT may not be able to adequately  support program mission needs.
section_title: Identification of ACE and US-VISIT Relationships and Dependencies Is in Progress
paragraphs: We reported in February 2005 that the program office recognized that  US-VISIT and the Automated Commercial Environment (ACE) have  related missions and operational environments. In addition, US-VISIT and  ACE could potentially develop, deploy, and use common information  technology infrastructures and services. We also reported that managing  this relationship has not been a priority. Accordingly, we recommended  that DHS do the following:  Make understanding the relationships and dependencies between the   US-VISIT and ACE programs a priority matter, and report periodically to   the Under Secretary on progress in doing so.
US-VISIT and ACE managers met in February 2004, to identify potential  areas for collaboration between the two programs and to clarify how the  programs could best support the DHS mission and provide officers with the  information and tools they need. According to program officials, they have  established a US-VISIT/ACE integrated project team to, among other  things, ensure that the two programs are programmatically and technically  aligned. The team has discussed potential areas of focus and agreed to  three areas: RF technology, program control, and data governance.  However, it does not have an approved charter, and it has not developed  explicit plans or milestone dates for identifying the dependencies and  relationships between the two programs. Program officials stated that the  team has met three times and plans to meet on a quarterly basis going  forward.
It is important that the relationships and dependencies between these two  programs be managed effectively. The longer it takes for the programs to  understand and exploit their relationships, the more rework will be needed  at a later date to do so.
section_title: Conclusions
paragraphs: Over the last 3 years, we have made recommendations aimed at correcting  fundamental limitations in US-VISIT’s program management ability and  thereby better ensuring the delivery of mission capability and value on time  and commensurate with costs. While progress on the implementation of  the recommendations is mixed, progress in critical areas has been slow. As  with any program, introducing and institutionalizing the program  management and accountability discipline at which our recommendations  are aimed require investing time and resources while continuing to meet  other program demands. In making such investment choices, it is important  to remember that institutionalizing such program discipline in the near  term will produce long-term payback in a program’s ability to meet these  other demands. Accordingly, the longer that US-VISIT takes to implement  our recommendations, the greater the risk that the program will not meet  its stated goals and commitments.
Our open recommendations are all aimed at strengthening US-VISIT  program management and improving DHS’s ability to make informed   US-VISIT investment decisions. With the exception of one, these  recommendations are still relevant and applicable. Since we made our  recommendation, facts and circumstances surrounding Increment 2B  deployment and operational status have materially changed, making the  collection of Increment 2B predeployment impractical. Nevertheless, the  need remains to better understand the impact of US-VISIT entry  capabilities on all land POEs. Until this understanding exists, the  department will be challenged in its ability to accurately estimate and  provide facilities and staff resource needs.
section_title: Recommendation for Executive Action
paragraphs: To recognize both the need to fully assess the impact of US-VISIT entry  capabilities on staffing levels and facilities at land POEs, as well as the  current operational status of Increment 2B, we are closing our existing  recommendation related to assessing the impact of Increment 2B. We  recommend that the DHS Secretary direct the US-VISIT Program Director  to explore alternative means of obtaining an understanding of the full  impact of US-VISIT at all land POEs, including its impact on workforce  levels and facilities; these alternatives should include surveying the sites  that were not part of the previous assessment.
section_title: Agency Comments and Our Evaluation
paragraphs: In its written comments on a draft of this report, signed by the Director,  Departmental GAO/OIG Liaison Office, and reprinted in appendix II, DHS  stated that it agreed with many areas of the report and that our  recommendations had made US-VISIT a stronger program. Further, the  department stated that while it disagreed with certain areas of the report, it  nevertheless concurred with the need to implement our open  recommendations with all due speed and diligence.
DHS commented specifically on 11 of the 18 recommendations discussed in  the report. The recommendations, the department’s comments, and our  responses follow: 1. Recommendation: Develop and begin implementing a system security  plan, and perform a privacy impact assessment and use the results of  the analysis in near-term and subsequent system acquisition decision  making.
DHS stated that this recommendation has been fully implemented. In  support, it said that it has completed a US-VISIT security plan that is  consistent with National Institute of Standards and Technology (NIST)  guidance, and that it provided the plan to us in September 2004. It also  stated that the security risk assessment aspect of this recommendation was  established in February 2005, 20 months after we made the  recommendation, and thus the age of the recommendation should be  shown as 10 months rather that the 30 months cited in the report.
The department also commented that there is no US-VISIT system, but  rather a US-VISIT program with capabilities delivered by existing  interconnected systems. According to the department, these component  systems have been certified and accredited, consistent with NIST guidance,  and as part of their certification and accreditation, security plans and risk  assessments, as well as risk mitigation strategies, have been developed for  each system. The department stated that it provided us with these  system-level risk assessments, as well as system-specific action plans and  milestones for implementing the mitigation strategies. In addition, the  department noted that it completed a programwide risk assessment in  December 2005 that specifically addresses information security issues that  might not be captured in the system-specific documentation used to certify  and accredit each system. In light of its system-specific certification and  accreditation efforts, existing system-level risk assessments, and the  program-level risk management process (see response 4 for discussion of  the risk management process), DHS commented that it is inaccurate to  state that US-VISIT officials are not in a position to know program risks,  and the recommendation should be closed.
While we agree that we received a copy of the US-VISIT security plan,  dated September 2004, we do not agree that the plan satisfied all relevant  federal guidance and that DHS has fully implemented our recommendation.  In particular, it has not provided us with evidence that a programwide risk  assessment has been done and that a security plan reflective of such an  assessment exists. According to relevant guidance, a security plan should  describe, among other things, the methodology that is to be used to identify  system threats and vulnerabilities and to assess risks, and it should include  the date the risk assessment was completed because the assessment is a  necessary driver of the security controls described in the plan. As we  reported in February 2005 and state in this report, the US-VISIT security  plan did not include this information; further, although DHS stated in its  comments that it completed this risk assessment in December 2005, this  statement is contradicted by a statement elsewhere in its comments that it  is still in the process of doing the assessment. In addition to this  contradiction, DHS’s comments did not include any evidence to  demonstrate that it has developed a complete risk assessment, such as a  copy of the assessment.
With regard to the age of the recommendation, we do not agree with DHS’s  position that we established a new finding regarding the lack of a  programwide risk assessment in our February 2005 report. Rather, as part  of our analysis of actions to implement our prior recommendation to  develop a security plan, which is to include information about the related  security risk assessment, we observed that the plan did not indicate a date  for completing a risk assessment in accordance with federal guidelines.  Therefore, our position that about 30 months had passed from the time of  our initial recommendation (June 2003) is accurate.
With regard to the individual system-level risk assessments, we agree that  we have received them. However, we do not agree that we have received  the action plans and milestones cited in the comments. Regardless, we do  not believe that system-level assessments are a sufficient substitute for a  programwide assessment. Accordingly, our recommendation focused on  the need for an integrated US-VISIT system risk assessment as part of  security planning. While the system-level plans and risk assessments are  relevant and useful, they neither individually nor collectively address the  threats and vulnerabilities imposed as a result of these systems’  integration. By stating in its comments its commitment to having a  programwide risk assessment that identifies and proposes mitigations for  security risks that arise as a result of the interface and integration of the  legacy systems, DHS is agreeing with our position. Moreover, without  evidence that the program has completely assessed its risks, we continue  to find no basis for how program officials would know the full range and  degree of US-VISIT security risks. Our position in this regard has been  reinforced by a recent DHS Inspector General report that identified a  number of US-VISIT security risks.
To further support its position that this recommendation has been fully  implemented, DHS also commented that it has completed numerous  privacy impact assessments and continues to update them to reflect system  changes. In particular, it said that it updated the privacy impact assessment  in December 2005 to reflect all increments and that it considers the  assessment to be part of US-VISIT system documentation. It further  commented that we appear to be unaware of privacy staff activities to  review system documents and perform privacy risk assessments  throughout the system life cycle. Nevertheless, the department  acknowledged that its privacy work was not always noted within US-VISIT  system documentation. Accordingly, DHS stated that it plans to  appropriately reference all privacy requirements and privacy risk  assessments in the program’s system documentation in the future.
We agree that US-VISIT has developed and updated its privacy impact  assessment and would note that our report states this fact. We do not agree,  however, with the comment that we are not aware that the privacy staff  review system documents and perform privacy risk assessments. In fact, it  is because we were aware of these facts that we were careful to ensure that  they were reflected in our report. The point that we are making is that  privacy is not addressed in all relevant systems documentation, which DHS  acknowledged in its comments. With regard to this point of agreement, we  support the department’s stated plans to reference all privacy requirements  and any privacy risk assessments in all relevant system documentation in  the future.  2. Recommendation: Develop and implement a plan for satisfying key  acquisition management controls, including acquisition planning,  solicitation, requirements management, program management,  contract tracking and oversight, evaluation, and transition to support,  and implement the controls in accordance with SEI guidance.
DHS commented that the report should reflect that US-VISIT had initially  adopted Carnegie Mellon University’s Software Engineering Institute (SEI)  Software Acquisition Capability Maturity Model® to guide its  software-related process improvement efforts and that, in December 2004,  it transitioned to SEI’s Capability Maturity Model–Integration (CMMI®). As  a result, it said that the program’s process improvement strategy and plans,  process development, and process appraisals are now aligned to the most  applicable CMMI process areas.
We agree that US-VISIT has transitioned to CMMI. We state in our report  that US-VISIT has done so and that the key process areas it is addressing in  its process improvement strategy and plan are consistent with those cited  in our recommendation. We do not believe that this transition materially  affects our recommendation, however, because even though the names of  the key processes in these two models may in some cases differ, the  processes and respective practices are fundamentally consistent. 3. Recommendation: Clarify the operational context in which US-VISIT is  to operate.
Consistent with our report, DHS commented that the operational context  in which US-VISIT operates is in progress, meaning that it has yet to be  fully established. For example, it said that the mission of DHS, and  therefore the scope of US-VISIT activities to meet the mission, is  continually expanding. Further, it acknowledged that more certainty in the  operational context is desirable. In mitigation of the risks associated with  not having a more stable operational context, DHS made several  statements. For example, it said that the principal role of US-VISIT is to  integrate information and immigration and border management systems  across DHS and the State Department, and to facilitate agencies working  toward a common environment that will eliminate redundancies. It also  said that elements of its draft immigration and border management  strategic plan are being used in current US-VISIT operations. In addition,  the department said that mechanisms to mitigate the risks that we cited  have been developed and are being implemented.
We support DHS’s acknowledgment of the importance of having a  well-defined operational context within which to define and implement  US-VISIT and related border security programs. However, we do not  believe that DHS’s comments provided any evidence showing that  sufficient steps and activities to mitigate the associated risks have been  taken or are planned.  4. Recommendation: Determine whether proposed US-VISIT increments  will produce mission value commensurate with cost and risks and  disclose to the Congress planned actions.
DHS commented that its cost-benefit analysis (CBA) for Increment 1B  conforms to relevant federal guidance, and noted that our expectations as  to the scope and level of detail of analysis that should be included in the  CBA document are inconsistent with its understanding of OMB Circular  A-94 and DHS’s CBA workbook, which were used to guide the development of  the CBA analysis. As an example, the department took exception with our  statement that year-by-year benefit estimates were not reported by noting  that the net present value was based on an estimate of annual benefits and  costs, and that net present value could not be estimated without a  year-by-year benefit analysis.
The department further commented that a comprehensive uncertainty  analysis was conducted because it completed a risk analysis, which is more  comprehensive, rigorous, and appropriate than conducting a sensitivity  analysis. In this regard, it added that the results of the risk analysis  provided an indication of Increment 1B’s worthiness in light of existing  uncertainty, rather than information on a specific CBA variable or another.  The department further noted that it had provided some of these  supporting analyses to us.
DHS also stated that any investment that has a 5-year life cycle and is  considered interim in nature will face considerable challenge in providing  economic benefits commensurate with cost.
We do not agree that the CBA fully conforms to relevant federal guidance.  As our report states, for example, the analysis does not explicitly state the  numerical value of the discount rate used for calculating each alternative’s  net present value, and hence does not conform to OMB guidance. In  addition, the cost estimates used in the analysis were not complete and  reliably derived. In deriving the estimate, for example, the department did  not clearly define the project’s life cycle to ensure that key factors were not  overlooked and that the full cost of the program was included. (See  response 10 below for more information on this point.) Last, while we  agree that a year-by-year benefit analysis is a necessary component of a net  present value determination, OMB nevertheless requires that the  year-by-year benefit estimates be reported in the analysis to promote  independent review of the estimates.
Also, we do not agree that DHS performed a complete uncertainty analysis.  According to OMB and DHS guidance, a complete uncertainty analysis  should include both a risk analysis and a sensitivity analysis. However, the  latter was not done. Thus, our point is not, as DHS comments suggest, that  US-VISIT should have performed a sensitivity analysis instead of a risk  analysis, but rather, that both types of analyses are necessary to completely  examine investment uncertainty.  5. Recommendation: Develop and implement a risk management plan and  ensure that all high risks and their status are reported regularly to the  executive body.
DHS commented that US-VISIT began the development and  implementation of its risk management plan in 2004 immediately after we  made our recommendation. It further commented that, as part of a CMMI  maturity internal appraisal that it completed in July 2005, it found that the  risk management process had not been consistently applied across the  program. To address this, the department cited actions that it has taken to  fully implement risk management, such as approving the risk management  plan in September 2005; defining a risk governance structure; establishing  and maintaining a risk database; and developing risk management training  and providing this training to program personnel and contractors beginning  in November 2005.
We support the recent actions that the program cited as having been taken  to strengthen risk management. However, the actions cited do not  demonstrate that the risk management process is being consistently  applied. Until US-VISIT fully implements its risk management plan and  process, it cannot be assured that all program risks are being identified and  managed in order to effectively mitigate any negative impact on the  program’s ability to deliver promised capabilities on time and within  budget.  6. Recommendation: Develop and approve test plans before testing  begins that (1) specify the test environment; (2) describe each test to be  performed, including test controls, inputs, and expected outputs;  (3) define the test procedures to be followed in conducting the tests;  and (4) provide traceability between test cases and the requirements to  be verified by the testing.
DHS stated that our report does not accurately reflect the status of the  Increment 2C Phase 1 testing. In particular, it said that the issues  associated with the traceability of requirements to test cases were minor  and that the extent of the discrepancies is far less than what our report  presents. It further stated that the discrepancies in our report are based on  old traceability documentation and do not reflect revised documentation  provided to us on November 9, 2005.
We agree that DHS provided us with revised traceability matrixes after we  had shared with them our analysis of the test plans and traceability  matrixes, dated June 28, 2005, and June 27, 2005, respectively. However, the  revised documentation referenced in DHS’s comments was provided in  November 2005, about 4 months after testing began. This means that the  test plans and traceability matrixes available at the time of testing—which  are what we reviewed because they governed the scope and nature of  actual testing performed—did not adequately trace between test cases and  the requirements to be verified. Specifically, 300 of the 438 Increment 2C  requirements, or about 70 percent, did not have specific references to test  cases.   7. Recommendation: Implement effective configuration management  practices, including establishing a US-VISIT change control board to  manage and oversee system changes.
DHS commented that a US-VISIT representative attends all configuration  control board meetings for all applicable legacy component systems, and  that any proposed change request from a legacy component control board  that could affect US-VISIT functionality is brought to the attention of the  US-VISIT Executive Configuration Control Board for consideration.
We do not question these statements. However, we do not believe that they  demonstrate that US-VISIT has adequate control over system changes that  could affect the program. That is, they do not ensure that changes to the  component systems that are initiated and approved by another DHS  organization and that could affect US-VISIT performance are subject to  US-VISIT configuration management and approval processes. US-VISIT  could establish explicit and enforceable control over changes to the legacy  systems through such mechanisms as defined and enforced memorandums  of understanding among the affected DHS organizations. It was the lack of  such control that prompted our recommendation. 8. Recommendation: Assess the full impact of Increment 2B on land POE  workforce levels and facilities, including performing appropriate  modeling exercises.
The department stated that, given the imperative to meet the legislatively  mandated time frames, the scope of Increment 2B was limited to only one  part of POE operations—incorporating the collection of a biometric into  the previously manual Form I-94 issuance process. It also stated that wait  times are affected by various factors, including traffic volume, staffing  levels, and availability of officers. Therefore, DHS focused the Increment  2B evaluation on just the change to this process.
The department further commented that given the events since the  evaluation—namely, Increment 2B full operations—it is not practical to  collect and model baseline data for the 47 sites that were not part of the  initial evaluation.
Regarding the 3 pilot sites included in the assessment, the department  stated that the sites were selected based on criteria developed from input  from US-VISIT, as well as CBP operational constraints. The department  further commented that the 3 sites provided a reasonable mix of travelers  and they did not have other constraints that directly impacted the  collection of performance data specific to Form I-94 issuance. DHS also  stated that the I-94 processing times vary by POE, and therefore they are  not easily generalized from one port to another. Further, the department  commented that the number of workstations and officers available to  operate those workstations to process applicants for a Form I-94 do not  impact the time it takes to issue a Form I-94.
We agree that the scope of the Increment 2B evaluation was limited to the  I-94 issuance process, and that it did not address the increment’s impact on  the POEs’ ability to meet other performance parameters. Our point is that  the limited nature of the evaluation does not satisfy either the intent of our  recommendation or DHS’s own stated purpose for the evaluation, which  was to determine the effectiveness of Increment 2B performance at the 50  busiest land POEs. We also agree that the I-94 processing times vary by  POE and cannot be easily generalized. It is for this reason, among others,  that we questioned whether the 3 sites selected for the assessment were  sufficiently representative to satisfy both our recommendation and the  evaluation’s stated purpose.
In addition, while we also agree that collecting pre-Increment 2B baseline  data is not practical at this time, the fact remains that the operational  impact of Increment 2B on workforce levels and facilities has not been  adequately assessed, as evidenced by officials at 1 large POE telling us that  processing times have increased and DHS’s recognition that each POE is  somewhat different. In light of these new facts and circumstances, we are  closing our existing recommendation and making a new recommendation  to recognize the need for DHS to explore alternative means to assess the  impact of US-VISIT entry capabilities at land POEs. This new  recommendation will be shown as an open recommendation, and the  original recommendation will be closed. 9. Recommendation: Develop a plan, including explicit tasks and  milestones, for implementing all of our open recommendations and  periodically report to the DHS Secretary and Under Secretary on  progress in implementing this plan; and report this progress, including  reasons for delays, in all future expenditure plans.
DHS stated that it is untrue that 19 months had elapsed from the time we  made this recommendation to the time that it assigned responsibilities to  program officials for addressing each of our recommendations. In support,  it commented that it issued its first plan to address our recommendations  on August 18, 2003, and subsequent reports have been issued periodically  that update progress in doing so.
We agree that DHS has assigned responsibilities to specific individuals for  addressing each recommendation. However, we have yet to be provided  any evidence to support its statement that it issued the first report  addressing our recommendations on August 18, 2003. Similarly, we have  not received evidence showing that it has prepared a plan, including  specific actions and milestones, for implementing all of our open  recommendations, which is a focus of this recommendation. We would also  observe that we made this recommendation in May 2004, and at that time  the department stated that it agreed with the recommendation but did not  indicate that it had taken any steps to address it, such as commenting that a  report was issued on August 18, 2003.  10. Recommendation: Follow effective practices for estimating the costs  of future increments.
DHS either tacitly or explicitly agreed with our findings relative to its  satisfaction of 8 of the 13 cost-estimating criteria presented in table 4 (now  table 3) of our draft report. For example, it agreed that it did not clearly  define the life cycle to which the cost estimate applies. It also agreed that it  did not include a work breakdown structure, noting that it used the  available project implementation schedule as a proxy for the activities  related to the deployment of the exit alternatives.
Regarding our five findings concerning its satisfaction of cost-estimating  with which DHS disagreed, the department’s primary area of disagreement  was with the intended purpose of the Increment 1B CBA that used the cost  estimate, which it said in its comments was to inform decision makers  about the relative worthiness of each of the three exit alternatives  considered for deployment. Hence, DHS stated that the purpose of the CBA  was to analyze only the costs associated with deploying an operational  solution, not to analyze the costs and benefits of both developing and  deploying alternative solutions. DHS further stated that the CBA thus  includes only those costs to be incurred in deploying a selected alternative,  and it does not include costs already incurred in developing system  alternatives (i.e., sunk costs). It further commented that DHS guidance  states that sunk costs are not relevant to the current investment analysis  because “only current decisions can affect the future consequences of  investment alternatives.”
DHS also disagreed that the cost estimate in the CBA should have included  nonrecurring development costs, and commented that it did appropriately  size the task described in the cost estimates for each alternative exit  solution, noting that sizing metrics related to software development were  not relevant to deployment of the alternatives because development  activities had already occurred and therefore are sunk costs. The  department added that those sizing metrics that are relevant to the cost  estimate are discussed in the CBA, as are the cost estimating parameters  (i.e., those associated with deployment and not those associated with  development and testing).
In addition, DHS disagreed that DHS’s cost estimate excluded important  cost categories, such as system testing, and stated that the estimate  addresses labor, facilities, operations and maintenance, information  technology, travel, and training costs. Once again, DHS emphasized that  since the focus of the CBA was on operational deployment and not system  design and development, system testing costs were not included because  they were not considered relevant. DHS also reiterated its early point that  the uncertainty analysis that it conducted was comprehensive.
We agree that actual sunk costs should not be included in a CBA cost  estimate. However, we disagree that the cost categories that DHS cited as  not relevant are only costs that are associated with predeployment  activities. Testing, for example, is an activity that is normally performed  before, during, and following deployment, and thus the associated costs  would be relevant to the stated purpose of the Increment 1B CBA.
However, a testing cost category was missing from the CBA cost estimate,  as was a cost category for software maintenance.
Regarding DHS’s statement that it conducted a complete uncertainty  analysis, we reiterate our previous point that a complete uncertainty  analysis should include both a risk analysis and a sensitivity analysis, and  the CBA did not include the latter.  11. Recommendation: Reassess plans for deploying an exit capability to  ensure that the scope of the exit pilot provides for adequate evaluation  of alternative solutions and better ensures that the exit solution  selected is in the best interest of the program.
Concerning the questions we raised about the adequacy of the exit pilots in  light of the 24 percent compliance rate, DHS commented that we failed to  consider the compliance rate of the previous exit pilot program, the  National Security Entry Exit Registration System (NSEERS), which,  according to DHS, had a 75 percent compliance rate. DHS added that  NSEERS achieved this compliance rate with a very limited number of exit  locations, and therefore, any of the three US-VISIT exit alternatives would  have at least a 75 percent compliance rate once national deployment was  completed.
Further, the department commented that Immigration and Customs  Enforcement (ICE) had recently conducted enforcement operations at the  Denver International Airport, and that the compliance rate during these  operations increased from 30 percent to over 90 percent. It then concluded  that the combined results of the exit pilot evaluation, the NSEERS pilot,  and the ICE enforcement activities at the Denver International Airport lead  it to believe that the US-VISIT exit alternatives have been adequately  evaluated.
We do not agree with this conclusion because it is based on unsupported  assumptions. Specifically, DHS did not provide any evidence to support its  claim that that US-VISIT would achieve a comparable compliance rate to  the NSEERS program. Moreover, even if DHS could achieve a 75 percent  compliance rate for US-VISIT exit,that still means that 25 percent of eligible  persons would not be complying with the US-VISIT exit process.
Further, DHS did not provide any information about the recent  enforcement actions conducted by ICE, nor did it provide any evidence  that this is a practical and viable option for the US-VISIT exit solution.
While we agree that enforcement actions may indeed increase the exit  compliance rate, DHS has not yet assessed the impact of such a solution on  the US-VISIT exit process. Further, the US-VISIT program director  acknowledged the need to evaluate the impact of implementing potential  enforcement actions on US-VISIT exit and planned to do so.
We are sending copies of this report to the Chairmen and Ranking Minority  Members of the Senate and House Appropriations Committees, as well as  to the Chairmen and Ranking Minority Members of other Senate and House  committees that have authorization and oversight responsibilities for  homeland security. We are also sending copies to the Secretary of  Homeland Security, Secretary of State, and the Director of OMB. Copies of  this report will also be available at no charge on our Web site at  www.gao.gov.
Should you or your offices have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. Key contributors to this report are  listed in appendix IV.
section_title: Objective, Scope, and Methodology
paragraphs: Our objective was to determine the progress of the Department of  Homeland Security (DHS) in implementing 18 of our recommendations  pertaining to the U.S. Visitor and Immigrant Status Indicator Technology  (US-VISIT) program. To accomplish this objective, we reviewed and  analyzed US-VISIT’s most recent status reports on the implementation of  our open recommendations and related key documents, augmented as  appropriate by interviews with program officials. More specifically, we  analyzed relevant systems acquisition documentation, including the  program’s process improvement plan, risk management plan, and  configuration management plan. We also analyzed the US-VISIT security  plan, privacy impact assessment, cost-benefit analysis, cost estimates, test  plans, human capital plans, and related evaluations and assessments. In  performing our analyses, we compared available documentation and  program officials’ statements with relevant federal guidance and associated  best practices. A more detailed description of our scope and methodology  relative to the cost-benefit analysis, cost estimates, and test plans follows:  Our analysis of the cost-benefit analysis focused on Increment 1B  because this was the latest cost-benefit analysis and cost estimate  prepared. In doing this analysis, we compared the US-VISIT cost-benefit  analysis to eight criteria in Office of Management and Budget (OMB)  guidance.
Our analysis of the cost estimate also focused on Increment 1B for the  same reason previously cited. In doing this analysis, we compared the  estimate to 13 criteria from the Software Engineering Institute that we  have previously reported to be the minimum set of actions needed to  develop a reliable cost estimate. We then determined whether the  criteria were satisfied, partially satisfied, or not satisfied using the  definitions given below.
Our analysis of the test plans focused on Increment 2C because it is the  most recently tested increment. This analysis included determining the  extent to which the test plans for this increment met 4 key criteria that  we have previously reported as essential to effective test plans. In doing  this analysis, we examined Increment 2C systems documentation,  including business and functional requirements and traceability  matrixes. We also independently traced 58 business requirements and  438 functional requirements to the test cases in the test plan. Further,  we independently traced all test cases to the requirements to determine  consistency.
In performing our work, we used the following categories and definitions in  deciding the extent to which each recommendation had been implemented.  Specifically, we considered a recommendation   completely implemented when documentation demonstrated that it had   partially implemented when documentation indicated that actions  were under way to implement it, and   in progress when documentation indicated that action had been  initiated to implement it.
These categories and definitions are consistent with those used in our prior  US-VISIT reports.
In determining the amount of time it has taken to implement actions on our  recommendations, we calculated the time from the date the report was  issued through December 2005.
We conducted our audit work at the US-VISIT program office in Rosslyn,  Virginia, from August 2005 through December 2005, in accordance with  generally accepted government auditing standards.
section_title: Comments from the Department of Homeland Security
section_title: Description of US-VISIT Processes
paragraphs: US-VISIT involves complex processes governing the stages of a traveler’s  visit to the United States (pre-entry, entry, status, and exit) and analysis of  hundreds of millions of foreign national travelers at over 300 air, sea, and  land ports of entry (POE). A simplified depiction of these processes is  shown in figure 4.
subsections: section_title: Pre-entry Process
paragraphs: Pre-entry processing begins with initial petitions for visas, grants of visa  status, or the issuance of travel documentation. When a foreign national  applies for a visa at a U.S. consulate, biographic and biometric data are  collected and shared with border management agencies. The biometric  data are transmitted from the Department of State to DHS, where the prints  are run against the Automated Biometric Identification System (IDENT)  database to verify identity and to run a check against the biometric watch  list. The results of the biometric check are transmitted back to State. A “hit”  response prevents State’s system from printing a visa for the applicant until  the information is reviewed and cleared by a consular officer.
Pre-entry also includes transmission by commercial air and sea carriers of  crew and passenger manifests to appropriate immigration officers before  these carriers arrive in the United States. These manifests are transmitted  through the Advanced Passenger Information System (APIS). The APIS  lists are run against the biographic lookout system to identify those arrivals  for whom biometric data are available. In addition, POEs review the APIS  list in order to identify foreign nationals who need to be scrutinized more  closely.
section_title: Entry Process
paragraphs: When a foreign national arrives at a POE’s primary (air and sea) or  secondary (land) inspection booth, the inspector, using a document reader,  scans the machine-readable travel documents. APIS returns any existing  records on the foreign national to the US-VISIT workstation screen,  including manifest data matches and biographic lookout hits. When a  match is found in the manifest data, the foreign national’s name is  highlighted and outlined on the manifest data portion of the screen.
Biographic information, such as name and date of birth, is displayed on the  bottom half of the computer screen, along with a photograph obtained  from State’s Consular Consolidated Database. The inspector at the booth  scans the foreign national’s fingerprints (left and right index fingers) and  takes a digital photograph. This information is forwarded to the IDENT  database, where it is checked against stored fingerprints in the IDENT  lookout database. If the foreign national’s fingerprints are already in  IDENT, the system performs a match (a comparison of the fingerprint  taken during the primary inspection to the one on file) to confirm that the  person submitting the fingerprints is the person on file. If no prints are  currently in IDENT, the foreign national is enrolled in US-VISIT (i.e.,  biographic and biometric data are entered into IDENT).
During this process, the inspector also questions the foreign national about  the purpose of his or her travel and length of stay. The inspector adds the  class of admission and duration of stay information into the Treasury  Enforcement Communications Systems, and stamps the “admit until” date  on the Form I-94. If the foreign national is ultimately determined to be  inadmissible, the person is detained, lookouts are posted in the databases,  and appropriate actions are taken.
section_title: Status Management Process
paragraphs: The status management process manages the foreign national’s temporary  presence in the United States, including the adjudication of benefits  applications and investigations into possible violations of immigration  regulations.
As part of this process, commercial air and sea carriers transmit departure  manifests electronically for each departing passenger. These manifests are  transmitted through APIS and shared with the Arrival Departure  Information System (ADIS). ADIS matches entry and exit manifest data  (i.e., each record showing a foreign national entering the United States is  matched with a record showing the foreign national exiting the United  States). ADIS also receives status information from the Computer Linked  Application Information Management System and the Student Exchange  Visitor Information System on foreign nationals.
section_title: Exit Process
paragraphs: The exit process includes the carriers’ submission of electronic manifest  data to APIS. This biographic information is transmitted to ADIS, where it  is matched against entry information. At the 11 POEs where the exit  solution is being implemented, the departure is processed by one of three  exit methods. Within each port, one or more of the exit methods may be  used. The three methods are as follows:  Kiosk: At the kiosk, the traveler, guided by a workstation attendant if  needed, scans the machine-readable travel documents, provides  electronic fingerprints, and has a digital photograph taken. A receipt is  printed to provide documentation of compliance with the exit process  and to assist in compliance on the traveler’s next attempted entry to the  country. After the receipt prints, the traveler proceeds to his or her  departure gate. At the conclusion of the transaction, the collected  information is transmitted to IDENT.
Mobile device: At the departure gate, and just before the traveler boards  the departure craft, either a workstation attendant or law enforcement  officer scans the machine-readable travel documents, scans the  traveler’s fingerprints (right and left index fingers), and takes a digital  photograph. A receipt is printed to provide documentation of  compliance with the exit process and to assist in compliance on the  traveler’s next attempted entry to the country. The device wirelessly  transmits the captured data in real time to IDENT via the Transportation  Security Administration’s Data Operations Center.
If the device is being operated by a workstation attendant, he or she  provides a printed receipt to the traveler, and the traveler then boards  the departure craft. If the mobile device is being operated by a law  enforcement officer, the captured biographic and biometric  information is checked in near real time against watch lists. Any  potential match is returned to the device and displayed visually for the  officer. If no match is found, the traveler is allowed to board the  departure craft.
Validator: Using a kiosk, the traveler, guided by a workstation attendant  if needed, scans the machine-readable travel documents, provides  electronic fingerprints, and has a digital photograph taken.
As with the kiosk, a receipt is printed to provide documentation of  compliance with the exit process and to assist in compliance on the  traveler’s next attempted entry to the country. However, this receipt has  biometrics (i.e., the traveler’s fingerprints and photograph) embedded  on the receipt. At the conclusion of the transaction, the collected  information is transmitted to IDENT.
The traveler presents his or her receipt to the attendant or law  enforcement officer at the gate or departure area, who scans the receipt  using a mobile device. The traveler’s identity is verified against the  biometric data embedded on the receipt. Once the traveler’s identity is  verified, he or she is allowed to board the departure craft. The captured  data are not transmitted in real time back to IDENT. Instead, the data  are periodically uploaded through the kiosk to IDENT.
section_title: Analysis Process
paragraphs: An analysis capability is to provide for the continuous screening against  watch lists of individuals enrolled in US-VISIT for appropriate reporting  and action. As more entry and exit information becomes available, it is to  be used for analysis of traffic volume and patterns as well as for risk  assessments. The analysis is also to be used to support resource and  staffing projections across POEs, strategic planning for integrated border  management analysis performed by the intelligence community, and  determination of travel use levels and expedited traveler programs.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following people made key  contributions to this report: Deborah Davis, Assistant Director; Hal  Brumm; Tonia Brown; Joanna Chan; Barbara Collier; Neil Doherty; Jennifer  Echard; James Houtz; Scott Pettis; Karen Richey; and Karl Seifert.