Arizona Border Surveillance Technology: More Information on Plans and Costs Is Needed before Proceeding
Published: 2011-11-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In recent years, nearly half of all annual apprehensions of illegal aliens along the entire Southwest border with Mexico have occurred along the Arizona border. Keeping illegal flows of people and drugs under control remains a top priority for the Department of Homeland Security's (DHS) U.S. Customs and Border Protection (CBP). In 2005, the Secure Border Initiative Network (SBInet) was conceived as a surveillance technology to create a "virtual fence" along the border. After spending nearly $1 billion, DHS deployed SBInet systems along 53 miles of Arizona's border that represent the highest risk for illegal entry. In January 2011, in response to concerns regarding SBInet's performance, cost, and schedule, DHS cancelled future procurements. CBP developed the Arizona Border Surveillance Technology Plan (Plan) for the remainder of the Arizona border. Funding for this Plan for fiscal year 2012 is $242 million. GAO was requested to assess the extent to which CBP (1) has the information needed to support and implement the Plan and (2) estimated life-cycle costs for future investments in accordance with best practices. GAO analyzed Plan documents and cost estimates, compared those estimates with best practices, and interviewed CBP officials.
section_title: What GAO Found
paragraphs: CBP does not have the information needed to fully support and implement its Arizona Border Surveillance Technology Plan in accordance with DHS and Office of Management and Budget (OMB) guidance. In developing the Plan, CBP conducted an analysis of alternatives and outreach to potential vendors. However, CBP has not documented the analysis justifying the specific types, quantities, and deployment locations of border surveillance technologies proposed in the Plan. Best practices for developing and managing costs indicate that a business case analysis should be rigorous enough that independent parties can review it and clearly understand why a particular alternative was chosen to support mission requirements. Without documentation of the analysis, there is no way to verify the process CBP followed, identify how the underlying analyses were used, assess the validity of the decisions made, or justify the funding requested for the Plan. CBP officials also have not yet defined the mission benefits expected from implementing the new Plan. GAO has previously reported that a solid business case providing an understanding of the potential return of large investments can be helpful to decision makers for determining whether continued investment is warranted after deployment. Defining the expected benefit could help improve CBP's ability to assess the effectiveness of the Plan as it is implemented. CBP does not intend to assess and address operational issues regarding the effectiveness and suitability of SBInet, steps that could provide CBP with information to help make decisions regarding alternatives for implementing the Plan. OMB guidance suggests that a post-implementation review occur when a system has been in operation for 6 months or immediately following investment termination. Such a review could help CBP make the most effective use of existing SBInet systems that, in connection with the Plan, could build a comprehensive and integrated approach for surveillance technology along the entire Arizona border. CBP's 10-year life-cycle cost estimate for the Plan of $1.5 billion is based on a rough order of magnitude analysis, and agency officials were unable to determine a level of confidence in their estimate as best practices suggest. Specifically, GAO's review of the estimate concluded that the estimate reflected substantial features of best practices, being both comprehensive and accurate, but it did not sufficiently meet other characteristics of a high-quality cost estimate, such as credibility, because it did not identify a level of confidence or quantify the impact of risks. GAO and OMB guidance emphasize that reliable cost estimates are important for program approval and continued receipt of annual funding. In addition, because CBP was unable to determine a level of confidence in its estimate, it will be difficult for CBP to determine what levels of contingency funding may be needed to cover risks associated with implementing new technologies along the remaining Arizona border. Thus, it will be difficult for CBP to provide reasonable assurance that its cost estimate is reliable and that its budget request for fiscal year 2012 and beyond is realistic and sufficient. A robust cost estimate--one that includes a level of confidence and quantifies the impact of risk--would help ensure that CBP's future technology deployments have sufficient funding levels related to the relative risks.
section_title: What GAO Recommends
paragraphs: GAO recommends that CBP document the analysis justifying the technologies proposed in the Plan, determine its mission benefits, conduct a post-implementation review of SBInet and determine a more robust life-cycle cost estimate for the Plan. DHS concurred with the recommendations.

section_title: Letter
paragraphs: Securing the Arizona portion of the approximately 2,000 miles of  southwest border that the United States shares with Mexico—while  keeping illegal flows of people and drugs under control—is a top priority  for the Department of Homeland Security’s (DHS) U.S. Customs and  Border Protection (CBP). In recent years, nearly half of all annual  apprehensions of illegal aliens along the entire southwest border with  Mexico have occurred along the Arizona border, but that number has  been steadily decreasing. DHS’s Office of Immigration Statistics reported  in June 2011, that the number of apprehensions of people entering the  country illegally in 2010 reflects the fifth consecutive year-to-year  decrease and is now at its lowest level since the early 1970s. As reflected  in table 1, that trend is true for both the southwest border and Arizona in  particular. On the other hand, Arizona remains the highest risk area for  illegal trafficking in marijuana not only because of the upward trend in the  number of pounds of marijuana seized by the Border Patrol but also  because nearly half of all marijuana seizures along the southwest border  are made in Arizona alone, as reflected in table 2, which follows.
CBP began development of the Secure Border Initiative Network (SBInet)  in 2005 as a combination of surveillance technologies that relied primarily  on radar and camera towers to create a “virtual fence” along the  southwest border in order to enhance CBP’s capability to detect, identify,  classify, track, and respond to illegal breaches at and between land ports  of entry. After 5 years and a cost of nearly $1 billion, SBInet systems are  now deployed along the 53 miles of Arizona’s 387-mile border with  Mexico that represent the highest risk area for illegal entry attempts.
In January 2011, in response to internal and external assessments that  identified concerns regarding the performance, cost, and schedule for  implementing the systems, the Secretary of Homeland Security  announced the cancellation of further procurements of SBInet systems.  However, CBP plans to continue to operate the existing SBInet systems  and received $26.4 million in fiscal year 2011 money for operations and  maintenance of the systems. CBP estimates that continued operation and  support of the SBInet systems will cost $10 million in fiscal year 2012 and  that these costs will continue for the foreseeable future.
CBP has taken steps to develop and implement a new Arizona Border  Surveillance Technology Plan (the Plan) for the remainder of the Arizona  border. This Plan is the first step in a multiyear, multibillion dollar effort to  secure the southwest border. The Plan is intended to identify, acquire,  and deploy additional surveillance technology types and quantities, and  suit them to the varying terrain along the Arizona border to enhance  situational awareness of illegal intrusions. In addition to the $185 million  CBP already allocated in fiscal year 2011, CBP has requested $242  million to fund the new Plan for fiscal year 2012 and estimates that the  total costs of acquiring and maintaining all of the proposed new systems  for the Arizona border over their expected 10-year life-cycle will be about  $1.5 billion.
Because of the high cost and challenges faced by CBP’s development of  SBInet and the importance of the revised plan, you asked us to review  CBP’s plans for developing and implementing a new approach for using  surveillance technology along the remainder of the southwest border in  Arizona. As agreed, our objectives were to determine (1) the extent to  which CBP has in accordance with DHS and Office of Management and  Budget (OMB) guidance supported and implemented its Arizona Border  Surveillance Technology Plan, and (2) the extent to which CBP’s  estimated life-cycle costs for the Arizona Border Surveillance Technology  Plan reflect best practices.
To address our first objective, we reviewed key program-planning  documents CBP relied on to support its new approach to identifying,  acquiring, and deploying surveillance technology applicable to specific  types of terrain along the Arizona border and compared them with  requirements in DHS acquisition regulations, including Acquisition  Regulation 102-01, and OMB guidance A-11. We also interviewed CBP  officials responsible for assessing the need for and documenting the cost  and operational effectiveness and suitability of proposed systems to  support its Arizona Border Surveillance Technology Plan (Plan) and for  identifying appropriate metrics to assess progress in border security. We  also assessed documents and evaluations of the SBInet developed and  deployed in Arizona’s Tucson sector from 2005 through 2010 and CBP’s  plans for SBInet’s operation and maintenance over its life-cycle. In doing  so, we reviewed key program documentation that describes the  operational benefits of SBInet and the Army Test and Evaluation  Command’s (ATEC) reports and briefing to CBP on operational test  findings. We interviewed Army leadership involved in the design and  implementation of the operational test and evaluation of test results in  order to determine the reliability of the information we used to support our  finding. We determined that the test results were sufficiently reliable for  the purposes of this report. We also interviewed officials from CBP’s  Office of Technology Innovation and Acquisition (OTIA) on how they  intended to use the operational test findings and recommendations to  inform the continuing operation of existing SBInet technology.
To address our second objective, we reviewed cost and budget  documents CBP relied on to support cost estimates for technology  alternatives contained in the “analysis of alternatives” (AOA) for Arizona  and in the President’s budget request for fiscal year 2012. We also  interviewed OTIA program officials and contractors responsible for  estimating the cost of future investments in surveillance technology,  specifically the life-cycle approach, requirements development and  management, test management, and risk management. We then  compared this information to relevant federal cost-estimating guidance,  derived from leading government and industry practices. To assess the  reliability of the cost data for the rough order of magnitude estimate for  implementation of the Plan, which assumed a 10-year life-cycle for the  acquisition, we relied on data for fiscal year 2010 and beyond to support  the findings in the report. To assess the reliability of the data that we used  to support the findings in this report, we reviewed relevant program  documentation to substantiate evidence obtained through interviews with  knowledgeable agency officials, where available, regarding the integrity of  the data. We determined that these data are sufficiently reliable for the  purposes of this report.
We conducted this performance audit from March 2011 through October  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Further details of our  objectives, scope, and methodology are in appendix I.
subsections: section_title: Background
paragraphs: In November 2005, DHS announced the launch of the Secure Border  Initiative (SBI), a multiyear, multibillion-dollar program aimed at securing  U.S. borders and reducing illegal immigration. CBP is the lead agency  within DHS responsible for the development and deployment of SBI  technology (e.g., cameras, sensors, radars, and tactical communications)  and tactical infrastructure (e.g., pedestrian and vehicle fences, roads, and  lighting). In July 2010, CBP announced the formation of OTIA, which was  created to ensure all of CBP’s technology efforts are properly focused on  the mission and are well integrated, and to strengthen CBP’s expertise  and effectiveness in program management and acquisition. OTIA  assumed the responsibilities of the former SBI program office that  previously managed SBInet.
SBInet was intended to cover the entire southwest border with an  integrated set of fixed sensor towers. These towers were to transmit radar  and camera information into a centralized location that integrated  information to create a Common Operating Picture (COP) at work stations  manned at all times by Border Patrol Agents. SBInet’s initial deployment,  known as Block 1, was deployed to 53 miles of the Arizona border where  it continues to be used by the Border Patrol. Since its inception, SBInet  had continued and repeated technical problems, cost overruns, and  schedule delays, which raised serious questions about SBInet’s ability to  meet Border Patrol’s needs for surveillance technology along the border.  We have issued 26 reports and testimonies identifying operational and  program management weaknesses that contributed to SBInet’s  performance shortfalls, including cost overruns and schedule slippages.  For example, in September 2008 and May 2010, we reported on  deficiencies in CBP’s timely preparation and completion of key acquisition  documents essential to setting operational requirements, identifying and  mitigating risks, and establishing the cost, schedule, and performance of  the project and the technology to be delivered. We also reported that key  acquisition documents, such as a risk management plan, were not  prepared and approved for SBInet prior to the start of the acquisition  process, a lack that precluded a fully informed design for the system that  would meet CBP’s needs within the expected time frame. In May 2010,  we made a number of recommendations to enhance CBP’s acquisition of  SBInet systems. DHS agreed with 10 of our recommendations and  partially agreed with two of them and detailed actions planned to address  each. This included a recommendation to respond to a departmentwide  reassessment of the program.
In January 2010, the Secretary of Homeland Security ordered a  departmentwide reassessment of the SBInet program to consider options  that may more efficiently, effectively, and economically meet the nation’s  border security needs. The assessment focused on two key questions:    whether the SBInet program was viable and could be made to work  effectively and fulfill the intent of the program and    whether SBInet was cost-effective.
After receiving the results of the assessment, in January 2011, the  Secretary of Homeland Security announced that the department had  concluded that SBInet systems were not appropriate for the entire  southwest border and did not meet current standards for viability and  cost-effectiveness. While the department would continue to use those  elements of SBInet that were useful, the Secretary announced that the  department was canceling further deployments of SBInet systems using  the current contract.
In its place, DHS is implementing a new approach for acquiring and  deploying border security technology called Alternative (Southwest)  Border Technology. As the approach’s first step, CBP’s Plan is to deploy  a mix of technologies to complete coverage of the Arizona border  including integrated fixed-tower (IFT) systems, Remote Video  Surveillance Systems (RVSS), Mobile Surveillance Systems (MSS),  hand-held equipment, and unattended ground sensors. CBP plans to  deploy five IFT systems each comprising about 10 radar-and-camera- equipped towers and integrate their signals into a system command  center. According to CBP officials, though similar, the IFT systems’  equipment will be simpler when compared with the equipment for the  tower systems deployed under SBInet. Thus, CBP’s plans include the  currently deployed SBInet system for 53 miles in Arizona along with the  new Plan to acquire and deploy additional towers, mobile surveillance  equipment, unattended ground sensors, and hand-held devices to secure  the rest of the Arizona border. CBP estimates that the total life-cycle cost  of the new Plan will be about $1.5 billion for Arizona. In fiscal year 2011,  CBP allocated $185 million to procure border surveillance technologies  contained in the Plan except for the new IFT systems. The agency has  requested $242 million in fiscal year 2012 appropriations to procure the  first three IFT systems also included in the Plan.
section_title: CBP Does Not Have the Information Needed to Fully Support and Implement Its Plan
paragraphs: CBP does not have the information needed to fully support and  implement its Plan in accordance with DHS and OMB guidance. To  develop this Plan, CBP conducted an analysis of alternatives (AOA) and  outreach to potential vendors, and took other steps to test the viability of  the current system. However, CBP has not    documented the analysis justifying the specific types, quantities, and  deployment locations of border surveillance technologies proposed in  the Plan;    defined the mission benefits or developed performance metrics to  assess its implementation of the Plan; or    developed a plan to assess and address operational issues with the  continuing use of SBInet systems along the highest risk section of the  border that could affect the new Plan’s implementation across the  remainder of Arizona.
For these reasons, CBP’s newly proposed approach is at an increased  risk of not accomplishing its goal in support of Arizona border security.
subsections: section_title: CBP, in Developing a Business Case for Its New Approach, Conducted an Analysis of Alternatives
paragraphs: At the Secretary of Homeland Security’s direction, CBP has adopted a  new approach for developing a technology plan for surveillance at the  border that includes development of a business case to justify the way  forward. CBP officials told us their business case consists of the Arizona  Border Surveillance Technology Plan and a phased independent analysis  of alternatives (AOA).
According to CBP officials, the development of the Arizona Border  Surveillance Technology Plan consisted of a two-step process. First, the  Homeland Security Studies and Analysis Institute (HSI) was enlisted to  conduct a multipart AOA beginning with Arizona. Second, using the  AOA, the Border Patrol conducted an operational assessment of border  surveillance technologies to identify the appropriate mix of technologies  required to gain situational awareness and manage the Arizona border  area. HSI’s AOA considered four technology alternatives: (1) agent- centric hand-held devices, (2) integrated fixed-tower systems, (3) mobile  surveillance equipment, and (4) unmanned aerial vehicles. These  technology alternatives were analyzed in four representative geographic  areas of Arizona. The AOA for Arizona found that integrated fixed-tower  systems, like the other technology alternatives, represent the most  effective choice only in certain circumstances and that there is no one  technology alternative that is appropriate for the entire Arizona border. A  summary of the conclusions reached for each of the four alternatives  examined is presented in table 3.
Unattended ground sensors were not included in the analysis because  they were considered part of the existing baseline of technology and  would co-exist with all of the alternatives in the AOA. In the AOA, HSI  noted that its analysis did not, among other things, identify the optimal  combination of specific equipment and systems, measure the contribution  of situational awareness to achieving control of the border, or quantify the  number of apprehensions that may result from the deployment of any  technology solution.
Upon completion of the AOA, in July 2010, the Secretary of Homeland  Security directed the AOA study team to seek independent validation of  its work. In response, HSI assembled an independent review team  composed of senior subject matter experts with expertise in border  security, operational testing, acquisition, performance measurement, and  the management and execution of AOAs to evaluate the AOA for Arizona.  In the results of the final report in March 2011, the review team concluded  that the AOA for Arizona appeared to have successfully answered the  questions asked and drew appropriate conclusions and insights that  should be useful to DHS and CBP. CBP officials said they planned to  conduct additional analysis of alternatives to incorporate additional  technologies and Border Patrol Sectors.
Following the completion of the AOA, the Border Patrol conducted its  operational assessment, which included a comparison of alternative  border surveillance technologies and an analysis of operational  judgments to consider both effectiveness and cost. According to CBP  officials, they started with the results of the AOA for Arizona, noting that  the AOA considered the technologies in terms of the trade offs between  capability and cost—but did not document the quantities of each  technology needed, the appropriate mix of the technologies, or how a  proposed mix of technologies would be applied to specific border areas.  CBP officials stated that a team of Border Patrol agents familiar with the  Arizona terrain determined the appropriate quantity and mix of  technologies by considering the terrain in each area under consideration  and which mix of technologies appeared to work for that area and terrain.  These officials also stated that they used an iterative process involving  dialogue between trained engineers and Border Patrol agents based on  the team’s understanding of topography and technology, considering the  lowest dollar cost mix of technologies as a starting point to see if  situational awareness provided by the mix sufficiently met the threat. As a  result, according to CBP officials, if the least expensive technology, such  as hand-held portable equipment, met the threat, then that technology  would be chosen. If the threat was not addressed by the hand-held  technologies, then officials said the team considered the next higher cost  technology. The officials added that the IFT systems were the most  expensive.
section_title: CBP Did Not Document How It Derived the Specific Types and Quantities of Technologies Contained in Its Arizona Border Surveillance Technology Plan
paragraphs: CBP has taken a number of steps to develop the Plan; however, program  officials developed and proposed the new Plan without documenting the  analysis justifying the specific types, quantities, and deployment locations  of border surveillance technologies CBP proposed. While the AOA  process itself was well documented, the Border Patrol’s operational  assessment, a key analytical component leading to the Plan, was not  transparent because of the lack of documentation.
The Plan includes quantities of various technologies, prioritized and  planned for implementation on a yearly basis. Specifically, based on the  Plan, CBP allocated $185 million to purchase border surveillance  technologies including mobile and hand-held equipment as well as RVSS  for fiscal year 2011, and has requested $242 million for fiscal year 2012  to acquire and deploy three IFT systems in Arizona, with two others to be  deployed by 2015, depending on funding availability.
Without documentation of the analysis justifying the specific types,  quantities, and deployment locations of border surveillance technologies  proposed in the Plan, an independent party cannot verify the process  followed, identify how the AOA was used, determine whether CBP’s use  of the AOA considered the limitations identified by HSI, assess the validity  of the decisions made, or justify the funding requested. Given that the  number of apprehensions of illegal border crossers is at the lowest level  in 40 years, if threats in the southwest border environment continue to  change and the Plan otherwise requires updating or revision, it will be  difficult for CBP officials to reassess the rationale for and determine what,  if any, changes are needed in the types, quantities, and deployment  locations of border surveillance technologies called for in the Plan.
Internal and management control standards for the federal government  call for agencies to promptly record and clearly document transactions  and significant events to maintain their relevance and value to  management in controlling operations and making decisions and to  ensure that agency objectives are met. The standards also call for  documentation to be readily available for examination. These standards  apply to CBP’s development of the quantities and types of technology and  their suitability to terrain to support the Plan; the expenditure of fiscal year  2011 funds on mobile, RVSS, and hand-held equipment; as well as the  planned acquisition of IFT systems requested in the President’s fiscal  year 2012 budget request.
A senior CBP official responsible for the program’s acquisitions told us  that he believed the AOA and the process used to develop and support  the plan justified acquisition decisions called for in the Arizona Border  Surveillance Technology Plan. According to CBP officials, the agency is  in the process of drafting the acquisition-planning documents required for  the DHS Acquisition Review Board to review and make a decision on  acquiring the IFT systems. These acquisition-planning documents are  required by DHS guidance for planning acquisitions, setting operational  requirements, and establishing acquisition baselines to help ensure  delivery of the required performance at acceptable levels of cost,  schedule, and risk. CBP officials said that they expect the Acquisition  Review Board to meet in November 2011 to consider the IFT acquisition.  The Acquisition Review Board is to consider these documents prior to  approving the program for acquisition and the issuance of a request for  proposal for the new IFT systems.
Nonetheless, in the absence of documentation that describes how CBP  integrated the operational assessments and technology deployment  analyses and used the results of the AOA to develop the types and  quantities of technology and their suitability to the terrain from the various  alternatives, it is unclear whether and how the analyses conducted to  develop the Plan demonstrated the cost and operational effectiveness of  the selected mix of technology versus other less costly solutions, or  whether the analyses determined the most appropriate technology for the  terrain. As a result, CBP cannot demonstrate the validity of the Arizona  Border Surveillance Technology Plan and the acquisition approach and  lacks reasonable assurance that the acquisition-planning documents will  fully support future deployments of border surveillance technology in  Arizona.
section_title: CBP Officials Have Not Yet Defined Expected Mission Benefits nor Quantified Metrics to Assess Progress in Implementing the Plan
paragraphs: Agency officials have not yet defined the mission benefits expected or  quantified metrics to assess the contribution of the selected approaches  in achieving their goal of situational awareness and detection of border  activity using surveillance technology. Without defining the expected  benefit or quantifying metrics, it will be difficult for CBP to assess the  effectiveness of the Plan as it is implemented. Assessing the  effectiveness of the program in Arizona will be essential as CBP works to  develop a more comprehensive plan for the entire southwest border.
Our findings are particularly relevant considering similar deficiencies in  SBInet systems. In May 2010, we reported that in the case of the  deployment of SBInet systems along the first 53 miles of the Arizona  border, CBP did not define or measure the expected mission benefits of  the system. For example, while program officials reported that system  benefits are documented in the SBInet Mission Need Statement dated  October 2006, this document did not include either quantifiable or  qualitative benefits. Rather, it provided general statements such as “the  lack of a program such as SBInet increases the risks of terrorist threats  and other illegal activities.” Moreover, we concluded that DHS had not  demonstrated that its proposed SBInet solution was a cost-effective  course of action, and thus whether the considerable time and money  invested to acquire and deploy it was a prudent use of limited resources.
As a result, we recommended that DHS should reconsider its proposed  SBInet solution. In doing so, it should explore ways to both limit its near- term investment in an initial set of operational capabilities and develop  and share with congressional decision makers reliable projections of the  relative costs and benefits of longer-term alternatives. These longer-term  alternatives would help meet the mission goals and outcomes that SBInet  was intended to advance. DHS should also share with congressional  decision makers the reasons why cost-benefit information was not  available and the uncertainty and risks associated with not having it. DHS  concurred with reconsidering its proposed SBInet solution and the  Secretary cancelled the program in January 2011.
The Secretary of Homeland Security reported in January 2011 that the  new Plan is expected to provide situational awareness for the entire  Arizona border by 2014, but CBP officials have not yet defined the  expected benefits or developed measurable and quantifiable performance  metrics that would show progress toward achieving that goal. The  Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to  ensure that information technology investments, such as IFT systems,  actually produce tangible, observable improvements in mission  performance. We have previously reported that a solid business case  providing an understanding of the potential return of large investments  can be helpful to decision makers for determining whether continued  investment is warranted. Additionally, according to the Government  Performance and Results Act, as amended, activities need to be  established to monitor performance measures and indicators.
The supporting documents CBP officials used to justify its allocation of  fiscal year 2011 funds and its budget requests for fiscal year 2012 did not  include any performance goals related to the expected outcome of the  investment. CBP officials reported that the decision documents that  informed their fiscal year 2012 budget request for $242 million (the AOA,  the Plan, and the Department’s fiscal year 2012-2016 Resource  Allocation Decision) did not contain any measurable and quantifiable  performance metrics by which progress toward achieving performance  goals could be determined. They said that the AOA contained four  measures of effectiveness associated with the alternatives they  assessed; however, these measures do not quantify the mission benefits  associated with implementation of the Plan. Without measurable and  quantifiable performance goals relating to expected outcomes, particularly  for alternatives selected for CBP’s Plan, it will be difficult for decision  makers to assess the costs and benefits provided by acquisition and  deployment of these systems and, more broadly, to measure program  performance and progress in achieving national homeland security goals  for securing the southwest border.
We have previously reported on key attributes of successful performance  measures that should be included in program performance metrics. In  circumstances where complete information is not available to measure  performance outcomes, agencies may need to use intermediate goals  and measures to show progress or contribution to intended results. For  example, Border Patrol may currently lack the capability to detect all  illegal entries of people, drugs, and weapons along the southwest border.  However, they may choose to establish performance measures that can  track progress in terms of using technology to increase the probability of  detection. Once CBP achieves an optimal level in terms of the probability  of detection, or situational awareness, it may then transition to measures  for reducing the flow of illegal activity and interdiction. In September 2011,  CBP officials reported that they are developing new measures to  determine whether and how technology investments impact border  security. They acknowledged that since large investments have been  made in border security, it is critical to assess the impacts these  investments have had on improving border security as well as projecting  the additional impact future investments will have on their ability to  manage the borders. However, CBP officials have not yet determined the  key attributes of these new measures. Measures and key attributes are  generally defined as part of the business case in order to explain how  they contribute to the mission’s benefits. Without a meaningful  understanding and disclosure of the mission benefits of the Plan and  related metrics to assess progress, it will be difficult for CBP to justify and  make informed decisions about its investment as well as measure the  extent to which implementation of the Plan will actually deliver mission  value commensurate with costs, similar to the challenges faced by  SBInet.
section_title: CBP Does Not Have a Plan to Assess and Address Operational Issues for Continuing Use of SBInet Technology for Surveillance
paragraphs: The new Arizona Border Surveillance Technology Plan does not include  the 53 miles covered by previously deployed SBInet systems that have  historically been at the highest risk for illegal crossing. CBP made its  decision to continue using SBInet Block 1 systems in the Tucson sector  before the results of operational testing were available, and CBP does not  have a plan to assess and address operational issues with SBInet  technology in use in this area. Effective use of existing SBInet systems is  essential for a comprehensive and integrated approach for surveillance  technology along the entire Arizona border.
The Secretary of Homeland Security’s January 2011 announcement  stated that in DHS’s assessment, the issue of viability was evaluated  within the context of the SBInet Block 1 deployments in the Tucson and  Ajo Border Patrol Stations’ areas of responsibility—referred to as Tuscon- 1 and Ajo-1. It stated that testing and evaluation of the system was under  way at those sites and that it was too early to quantify the effectiveness of  the technology. However, based on qualitative assessments from the  Border Patrol, which had begun using the systems, SBInet systems  enhanced the Border Patrol’s ability to detect, identify, track, deter, and  respond to threats along some parts of the border. The announcement  further stated that SBInet contributed in part to increasing the likelihood of  the apprehension of illegal entrants.
Since the Secretary’s announcement, CBP has received the U.S. Army’s  Test and Evaluation Command (ATEC) operational test results for the  SBInet system at Tucson-1 that revealed challenges regarding the  effectiveness and suitability of the technology for border surveillance. In  its March 2011 report on operational testing conducted from October  2010 to November 2010, ATEC said that SBInet was “effective with  limitations” because (1) the ability of the system to correctly detect,  identify, and classify items of interest was below initial system acceptance  benchmarks and was (2) further degraded by terrain and weather  conditions, and (3) the radar system generated a high number of  extraneous radar returns or “hits” that overwhelmed operators. ATEC  found that the system was “not operationally suitable” because the  reliability of the system was low.
Specifically, ATEC officials found that the rugged, restrictive terrain and  weather conditions prevalent where SBInet is deployed affected the  performance of the system’s radar, which impacted success in detecting,  identifying, and classifying the items of interest. ATEC officials referred to  this situation as a “terrain/technology mismatch.” ATEC also reported that  the radar’s difficulties with terrain and weather resulted in a high number  of extraneous radar hits’ being generated by the system, hits that  presented a difficult-to-manage workload for operators for which SBInet’s  technical filtering techniques could not compensate fully. Moreover, ATEC  also noted that the system required operators to cull through thousands of  extraneous radar hits (among a total average of 26,000 hits per day). This  generated an unreasonable expectation given the lack of standardized  procedures in how to manage the extraneous radar hits and lack of  training in how to use the system tools to filter them out.
In response to ATEC’s findings, CBP said that problems with using  SBInet to detect, identify, and classify items of interest are less significant  now than when they began using the system since the operators, through  their continued experience with the system, better understand what  causes extraneous radar hits and are better able to deal with them.  Similarly, CBP stated that ATEC’s reliability findings have been mitigated  by the fact that many of the system failures were because of routine  system reboots that are being addressed by enhancements to SBInet  currently in process.
Notwithstanding the findings of the ATEC testers, Border Patrol SBInet  operators and field agents who participated in this testing and completed  questionnaires during and at the end of testing responded favorably about  a number of aspects of the system, including responding that the system  significantly enhanced both agent safety and overall situational  awareness during day-to-day operations for tracking and apprehending  illegal border crossers. Further, in our March 2011 work reviewing the  status of SBInet, all the Border Patrol officials we spoke with told us the  system provided them with capability they did not have previously and  was considerably better than the technology that was available to them  prior to SBInet’s deployment. Nonetheless, based on the factors  mentioned above, ATEC concluded that because of the limitations of the  SBInet radar, the system does not significantly reduce the need for  traditional field agents’ role in the operating environment. ATEC also  concluded that, despite receiving high questionnaire ratings from test  participants who completed questionnaires, actual performance of SBInet  in terms of interdiction was only slightly different than if the system had  not been present in the areas where it is deployed.
According to DHS guidance, project managers are required to conduct a  Post Implementation Review to evaluate the impact of an investment’s  deployment on customers, the mission and program, and technical and/or  mission capabilities. Similarly, OMB’s Capital Programming Guide, a  supplement to OMB Circular A-11, identifies a Post-Implementation  Review as a tool to evaluate an investment’s efficiency and effectiveness  to determine how well an investment achieved the planned functionality  and anticipated benefits. Moreover, as the next step in the evaluation  phase for any major information technology investment, like SBInet, DHS  policy requires that an operational analysis be undertaken to measure the  performance and cost of the asset against the established baseline.  According to the guidance, operational analyses measure how close the  investment is to achieving the project’s expected cost, schedule, and  performance goals. When performance is found deficient, the project  manager must identify and schedule suitable corrective actions.
DHS guidance further states that the Post-Implementation Review should  occur when a system has been in operation at least 6 months or  immediately following investment termination, and the Operational  Analysis should be performed annually for information technology  investments in the steady-state or operations and maintenance phase like  SBInet. Such reviews would be prudent and provide a baseline for CBP to  decide whether to continue the system without adjustment, to modify the  system to improve performance—to the extent that addressing the  operational issues identified by the Army’s operational testing are cost  beneficial—or, if necessary, to consider alternatives to the implemented  system. The reviews could also provide CBP with an opportunity to more  quantitatively determine and document the SBInet system’s ability to  satisfy the agency’s operational requirements, given that CBP plans to  continue to operate the SBInet system along the highest risk 53 miles of  the Arizona border and will be faced with funding operation and  maintenance costs over the remaining 10-year life of the system. (For  example, CBP has requested $10 million for fiscal year 2012 to support  the continuing operation of SBInet systems.)
A Post-Implementation Review and Operational Analysis could also help  inform CBP’s decisions about whether future technology deployments of  similar ground-based radar technologies that are to make up the IFT  systems being used for the next step in its plan to deploy border  surveillance technology in Arizona are necessary where SBInet systems  are currently being used.
CBP program officials initially told us they did not intend to develop an  action plan that addressed ATEC deficiencies and recommendations.  They said that the Secretary of Homeland Security’s decision to cancel  further procurements of SBInet systems was a basis for their decision not  to commit resources to resolve technical, logistical, and operational  issues identified during the Army’s operational testing of the system.  However, in response to our inquiries related to the applicability of this  guidance, CBP told us in August 2011 that the Border Patrol was  considering, but had not yet developed, a plan for reviewing and  addressing the results of the ATEC tests for SBInet.
CBP officials said they had not developed a plan to address SBInet  operational test outcomes or conducted a post-implementation review  because of the Secretary’s cancellation of the program. They said they  were confident that the technology was now available to acquire and  deploy a non-developmental system as part of the new Arizona Border  Surveillance Technology Plan. However, CBP plans to continue using  SBInet for surveillance along the highest risk corridor in Arizona. The  impact of the use of SBInet systems could affect the deployment and use  of other surveillance technologies along the Arizona border. For example,  if SBInet systems are particularly effective, illegal border- crossing traffic  may decrease in the area where the systems are in use. Conversely, if  SBInet is less effective, illegal border crossings may increase in the area  surveilled. Thus, conducting an assessment of SBInet operational test  results and the potential cost- effective resolution of the issues identified  could better position CBP in determining analyses of alternative  technologies for future systems’ deployments in the areas of the Arizona  Border covered by SBInet.
section_title: CBP’s Cost Estimate Reflects Some but Not All Key Cost- Estimating Best Practices
paragraphs: CBP officials have taken steps to develop a cost estimate for the Arizona  Border Surveillance Technology Plan consistent with some best  practices. However, the officials did not determine a level of confidence  around their rough order of magnitude (ROM) estimate, inconsistent with  best practices.
subsections: section_title: CBP’s Cost Estimate Is Substantially Comprehensive and Accurate but Partially Documented and Minimally Credible
paragraphs: Our analysis of CBP’s 10-year life-cycle cost estimate (LCCE) for the  Arizona Border Surveillance Technology Plan (the Plan) found that CBP  did not fully follow best practices for developing a reliable LCCE, which is  at the core of successfully managing a project within cost and affordability  guidelines. CBP’s estimate for the Plan is $1.5 billion. The estimate  includes approximately $750 million in acquisition costs and  approximately $800 million for operations and maintenance costs to  procure and deploy a range of border surveillance technology across  Arizona.
Our guide and OMB guidance emphasize that reliable cost estimates are  important for program approval and continued receipt of annual funding.  DHS policy similarly provides that life-cycle cost estimates are essential  to an effective budget process and form the basis for annual budget  decisions. Reliable LCCEs reflect four characteristics. They are (1) well- documented, (2) comprehensive, (3) accurate, and (4) credible. These  four characteristics encompass 12 best practices for reliable program life- cycle cost estimates. (See app. III that describes the 12 steps of high- quality cost estimates.) The results of our analysis of CBP’s cost estimate  against these four best practice characteristics are summarized in table 4.
CBP’s life-cycle cost estimate for the Plan substantially met best practices  in terms of being both comprehensive and accurate. For example, in  terms of comprehensiveness, the estimate included technical data that  was documented at a sufficient level of detail. This included specific  technology requirements anticipated to provide situational awareness for  each of the focus areas along the Arizona border such as the number of  integrated fixed-t-ower systems, mobile surveillance systems, or other  technologies. However, detailed technical data related to shared IT  infrastructure was missing, and risk information on the technologies,  assumptions, and estimating were not provided. As a result, our analysis  concluded that CBP’s cost estimate substantially, but not fully, reflected  best practices for comprehensiveness. In terms of accuracy, the cost  estimate was continually updated and refined as more information  became known; this helps to provide decision makers with accurate and  current information. Specifically, there were 10 changes documented that  clearly showed what updates were made to the cost estimate. These  changes included new technology quantities, learning-curve adjustments,  and incurred cost adjustments. However, the estimate also relied on  historical data from earlier SBInet deployment, and the accuracy and the  reliability of that data were questionable because some data were still  pending. As a result, CBP’s estimate substantially, but not fully, met  criteria for accuracy.
Moreover, the Plan’s estimate partially met best practices in terms of  being well-documented and minimally met best practices for being  credible. Cost estimates are well-documented when they can be easily  repeated and can be traced to original sources. The documentation  should explicitly identify the primary methods, calculations, assumptions,  and sources of the data used to generate each cost element. However,  according to our review of data provided to us by CBP, while many data  sources were discussed, the actual data used to determine the estimate  were not always shown. Therefore, it is not possible for an unfamiliar  analyst to recreate the estimate with the provided documentation. As a  result of insufficient documentation, the validity and reliability of the CBP’s  life-cycle cost estimate for the Arizona Border Surveillance Technology  Plan cannot be verified. For that reason, we assessed CBP’s cost  estimate as partially meeting criteria for being well-documented.
In terms of credibility, we found that CBP officials did not conduct a  sensitivity analysis, and a cost-risk and uncertainty analysis, to determine  a level of confidence in the $1.5- billion life-cycle cost estimate for  Arizona. Therefore, CBP’s estimate provides an incomplete basis for  management decisions because without a level of confidence, it will be  difficult for decision makers to identify a range of possible costs, higher  and lower, corresponding to the associated risks involved with the  acquisition and deployment of technology across Arizona. A sensitivity  analysis of all cost estimates examines the effects of changing one  assumption or cost driver at a time while holding all other variables  constant. Since uncertainty cannot be avoided, it is necessary to identify  the cost elements that represent the most risk and, if possible, cost  estimators should quantify the risk.
In addition to sensitivity analysis, which looks at the effects of changing  one parameter or cost driver at a time, a cost risk and uncertainty  analysis should be performed to capture the cumulative effect of multiple  variables changing, such as schedules slipping, or proposed solutions’  not meeting user needs, allowing for a known range of potential costs.  Because CBP officials did not perform a cost-risk, and uncertainty  analysis, the estimate for the Plan is likely to be unrealistic because it  does not assess the variability in the cost estimate from such effects as  schedules slipping, missions changing, and proposed solutions’ not  meeting users’ needs. Without this type of analysis for example, it will be  difficult for CBP decision makers to determine a defensible level of  contingency reserves necessary to cover increased costs resulting from  uncertainties associated with the Arizona plan.
Another way to reinforce the credibility of the cost estimate would be for  CBP to commission an independent cost estimate and then reconcile any  differences between the two. This process is considered one of the best  and most reliable estimate validation methods. However, because CBP  officials did not compare their estimate with an independent estimate,  agency decision makers may lack insight regarding the plan’s range of  potential costs because independent cost estimates frequently use  different methods and are less burdened with organizational biases.  Despite these deficiencies, we assessed CBP’s cost estimate as  minimally meeting best practices for credibility rather than not meeting  them because CBP did identify some cost drivers that could be used as a  basis for conducting a sensitivity analysis.
Responding to the results of our cost analysis, CBP officials reported that  their approach was to develop and report a rough order of magnitude  (ROM) cost estimate for the portfolio of technology projects contained in  the Arizona Border Surveillance Technology Plan. Because CBP officials  considered the $1.5-billion estimate an initial ROM estimate, they  reported that it lacked some elements of the technology costs and  complete supporting documentation, and was not subjected to an  independent or corroborating cost-estimating effort.
section_title: Based on a Rough Order of Magnitude Analysis, CBP’s Budget Request for IFT Systems May Not Be Realistic and Is Not Sufficient
paragraphs: CBP officials reported that while they believed the $1.5 billion cost  estimate to complete the Arizona border was reasonable, they cautioned  that they considered it to be a ROM estimate rather than a LCCE.  According to cost-estimating best practices, a ROM cost estimate is  developed when a quick estimate is needed and few details are available.  It is usually based on historical ratio information and typically developed  to support what-if analyses and can be developed for a particular phase  or portion of an estimate to the entire cost estimate, depending on  available data. It is helpful for examining differences in high-level  alternatives to see which are the most feasible. However, according to  cost- estimating best practices, because a ROM is developed from limited  data and in a short time, a ROM analysis should never be considered a  budget-quality cost estimate. However, CBP used the ROM estimate to  support its $242-million budget request for fiscal year 2012 because it  lacked the time needed to develop a more robust estimate.
CBP officials said the request reflected relevant operational information  from authoritative CBP sources as well as comprehensive program  technical descriptions for both the acquisition and sustainment life-cycle  phases. Officials plan to use the fiscal year 2012 appropriations to  purchase IFT systems technology for future deployments in Arizona. The  three initial deployments are planned for the Nogales, Douglas, and Casa  Grande station areas of operation followed by two additional deployments  planned in Sonoita and Wellton station areas. According to OTIA and  Border Patrol officials, depending on the availability of funding, the  deployments of the IFT system component of the Plan are expected to  begin around March 2013 and be completed by the end of 2015 (or  possibly early 2016), with other sector deployments sequentially following  the Arizona sector. CBP estimated that the entire IFT system acquisition  for Arizona would cost about $570 million, including funding for design  and development, equipment procurement, production and deployment,  systems engineering and program management, and a national  operations center.
Nonetheless, there is significant uncertainty regarding the cost of IFT  systems stemming from assumptions made as part of the cost-estimating  process. For example, when developing the ROM estimate, CBP officials  expected that IFT systems would be able to access existing commercial  communication networks in target deployment areas. CBP officials said  that this assumption is no longer valid in all cases and additional  communication relay equipment will likely be necessary. While CBP  officials believe they have adequate risk contingency funds to address  this issue, because they did not undertake a risk and uncertainly analysis  to quantify the impact on the cost estimate of these kinds of risks, it will  be difficult for them to determine whether the contingency funds will be  sufficient to cover this or other risks.
The findings of our analysis are particularly relevant considering that  similar deficiencies were identified with the life-cycle cost estimate for the  SBInet Block 1 deployment. In May 2010, we reported that the life-cycle  cost estimate for the Block 1 deployment was not credible because risk  and uncertainty were not adequately assessed. For example, the risks  associated with software development were not examined, even though  such risks were known to exist. In fact, the only risks considered were  those associated with uncertainty in labor rates and hardware costs, and  instead of being based on historical quantitative analyses, these risks  were expressed by assigning them arbitrary positive or negative  percentages. In addition, the estimate did not specify contingency reserve  amounts to mitigate known risks, and an independent cost estimate was  not used to verify the estimate. Our program assessments have too often  revealed that not integrating cost estimation, system development  oversight, and risk management—three key disciplines, interrelated and  essential to effective acquisition management—has resulted in programs’  costing more than planned and delivering less than promised.
In discussing this issue, CBP officials said they attempted to establish as  much fidelity as possible with the Arizona technology cost estimate and  associated budget requests. However, the officials reported that they  knew that several of their planning and estimating assumptions were  broad and that they lacked some desired details. For those reasons, the  officials continue to call their Arizona technology cost estimates ROM  estimates. CBP officials stated that they used the best information  available to establish budget quality estimates and plan to provide  updated, comprehensive, and thoroughly documented cost estimates in  fall 2011 related to the Plan.
CBP officials said they consider the Arizona Border Surveillance  Technology Plan to be a grouping of multiple projects that will proceed as  independent acquisitions rather than a unified capital asset acquisition.  As such, CBP officials reported that they are preparing LCCE for the  individual acquisition projects in the Plan, initially for the IFT systems and  the Remote Video Surveillance Systems (RVSS) with other projects to  follow. CBP officials reported that OTIA will request baseline approval for  the projects in the Plan later this year from the appropriate department or  CBP acquisition oversight board. They said that this process will further  examine respective cost and schedule estimates, technical performance  and program risks, as well as contracting and related management  concerns. Prior to the major acquisition reviews, CBP officials said that  OTIA is developing detailed program management plans and supporting  documentation for each of the Arizona technology projects within the  portfolio. CBP officials do not expect to release a cost estimate for  technology acquisition and deployment beyond Arizona until February  2012. However, without a complete LCCE that contains all cost  estimating best practices for the Arizona Plan, CBP could experience the  same kind of problems as the ones it encountered in the acquisition of  SBInet.
section_title: Conclusions
paragraphs: CBP has not yet demonstrated the effectiveness and suitability of its new  approach for deploying surveillance technology in Arizona. By taking  steps to document how, where, and why it plans to deploy specific  combinations of technology prior to its acquisition and deployment, CBP  could be better positioned to minimize performance risks associated with  the new approach. Given that apprehensions along the southwest border  are at their lowest levels since the 1970s and, in light of the difficulties  CBP has faced in its efforts to procure and deploy surveillance  technology, documenting the underlying analysis used to justify the  technology types, quantities, and suitability to terrain contained in the  Arizona Border Surveillance Technology Plan could help CBP make its  decisions more transparent. Further, better defining the mission benefits  to be gained from planned procurements and quantifying performance  metrics to assess the effectiveness of technologies selected for Arizona  would help justify program funding and assist CBP in measuring its  progress toward securing the southwest border. Given that CBP plans to  spend $1.5 billion for technologies to enhance surveillance across the  remainder of the Arizona border, conducting a post-implementation  review and operational assessment of the SBInet systems that includes a  review of operational test results, and then weighing costs and benefits of  taking action on the results could give CBP the opportunity to maximize  the effectiveness of the system it has already deployed in the highest risk  area in Arizona. It could also help CBP in making decisions for future  technology deployments along the southwest border and provide a sound  basis for assessing and deploying alternative technologies.
Fully documenting the data used in the cost model could help ensure that  the validity and reliability of the CBP’s life-cycle cost estimate for the  Arizona Border Surveillance Technology Plan can be verified. Because  CBP officials did not conduct a sensitivity analysis and a cost-risk and  uncertainty analysis to determine a level of confidence in the $1.5-billion  life-cycle cost estimate for the Plan, it will be difficult for decision makers  to determine what levels of contingency funding may be needed to cover  risks associated with implementing new technologies along the remaining  Arizona border. Until CBP officials accurately quantify the impacts of the  risks, the budget requests for fiscal year 2012 and beyond may not be  realistic and sufficient to achieve program aims. Because CBP officials do  not expect to release a cost estimate for technology acquisition and  deployment beyond Arizona until February 2012, until that time, the cost  visibility of the total investment required to deploy technology across the  southwest border is unclear. Verification of the new life-cycle cost  estimate with an independent cost estimate and reconciliation of any  differences could further help ensure the credibility of the cost estimate.
section_title: Recommendations
paragraphs: To increase the likelihood of successful implementation of the Arizona  Border Surveillance Technology Plan and maximize the effectiveness of  technology already deployed, we recommend that the Commissioner of  CBP take the following three steps in planning the agency’s new  technology approach:    ensure that the underlying analyses of the Plan are documented in  accordance with DHS guidance and internal controls standards;    determine the mission benefits to be derived from implementation of  the plan and develop and apply key attributes for metrics to assess  program implementation; and  conduct a post-implementation review and operational assessment of  SBInet, including consideration of the ATEC test results, and assess  the costs and benefits of addressing the issues identified to help  ensure the security of the 53 miles already covered by SBInet and  enhance security on the Arizona border.
To increase the reliability of CBP’s Cost Estimate for the Arizona Border  Surveillance Technology Plan, we recommend that the Commissioner of  CBP update its cost estimate for the Plan using best practices, so that the  estimate is comprehensive, accurate, well-documented, and credible.  Specifically, the OTIA program office should (1) fully document data used  in the cost model; (2) conduct a sensitivity analysis and risk and  uncertainty analysis to determine a level of confidence in the estimate so  that contingency funding can be established relative to quantified risk;  and (3) independently verify the new life-cycle cost estimate with an  independent cost estimate and reconcile any differences.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments on a draft of this report from DHS and DOD.   DHS provided written comments which are reprinted in appendix IV.  In  commenting on the draft report, DHS concurred with our  recommendations and identified steps officials planned to take to  implement them, along with estimated dates for their completion.  DHS  also stated that there were several issues raised in the report that could  not be addressed at present.  In an email received on October 14, 2011,  the DOD liaison indicated that DOD had no comments on the report.
Regarding the first recommendation that CBP ensure that the underlying  analyses of the Plan are documented in accordance with DHS guidance  and internal controls standards, DHS concurred.  DHS stated that CBP  plans to work with the DHS Internal Control Program Management Office  to ensure Plan documentation is in accordance with DHS guidance and  internal controls and anticipates completing this action by May 31, 2012.   Such actions should address the intent of the recommendation.
Regarding the second recommendation that CBP determine the mission  benefits to be derived from implementation of the Plan and develop and  apply key attributes for metrics to assess the program’s implementation,  DHS concurred and stated that CBP plans to develop a set of measures  by April 30, 2012, that will assess the effectiveness and mission benefits  of future technology investments. Such action should address the intent  of the recommendation.
With regard to the third recommendation that CBP conduct a post- implementation review and operational assessment of SBlnet, DHS  concurred and stated that CBP's Office of Border Patrol (OBP) is working  with Johns Hopkins University Applied Physics Laboratory on a Block I  after-action review (AAR), which will address the operational test and  evaluation results and offer recommendations on tactics, techniques, and  procedures. DHS also said that OTlA and the Border Patrol will conduct a  post-implementation review and operational assessment required in light  of the OBP AAR, consistent with departmental policy and procedures for  recurring reporting of fielded systems. DHS states that CBP plans to  complete these actions by June 30, 2012. Such actions should address  the intent of the recommendation.
Regarding the three recommendations related to CBP’s life-cycle cost  estimate—that CBP fully document data used in the cost model; conduct  a sensitivity analysis and risk and uncertainty analysis to determine a  level of confidence in the estimate so that contingency funding can be  established relative to quantified risk; and independently verify the new  life-cycle cost estimate with an independent cost estimate and reconcile  any differences—DHS concurred. DHS stated that OTIA is preparing  individual RVSS and IFT project cost estimates consistent with the GAO’s  guidelines and is fully documenting all assumptions, data structures and  sources, methods and calculations, as well as risks and sensitivities for  the two largest elements of the Plan that will enable CBP to refine  contingency funding as needed.  Officials plan to submit the appropriate  project documentation, including the projects' Cost Estimating Baseline  Document and the updated life-cycle cost estimate, to the department for  independent review and verification of the respective projects'  methodology and data sources. The department commented that it plans  to determine the need for an independent cost estimate at a later time but  will complete these actions by April 30, 2012.  While these actions are  positive steps, they do not fully address the recommendation that DHS  implement best practices for cost estimates for the entire Plan.  Instead,  DHS’s response indicates that it plans to implement these best practices  for the two largest projects within the Plan.  To fully understand the  impacts of integrating these separate projects, DHS should update the  life-cycle cost estimate for the entire Plan.
DHS also noted that there were three issues in the draft that it did not  feel, at present, it could address.  First, regarding the need to document  analytical steps taken to develop the Plan, the department stated that  DHS relies on Border Patrol field agents' expert judgment to select the  types and quantities of technologies best suited for their respective  geographic areas of responsibilities. According to DHS, in all cases,  technology selections were verified for consistency with the major findings  of the AoA. In some cases, however, the Border Patrol determined that  operational priorities justified a technology mix that was not necessarily  the lowest cost—for example, the Border Patrol said a higher cost  integrated fixed tower (IFT) solution would be operationally superior to  deploying lower cost mobile systems.  According to DHS, CBP is not  planning further analyses or additional documentation given that they  consider their analyses to be sufficiently documented in the final Plan.
We recognize the value of Border Patrol agents’ expert judgment in  selecting the types and quantities of technologies best suited for their  respective geographic areas of responsibility. Nonetheless, internal  control standards call for documentation to support decision making to be  available for examination. In the Plan, CBP officials documented the  results of their analyses in terms of their planned deployments of  technologies but did not include documentation of the supporting  operational assessment done by the Border Patrol justifying the specific  types, quantities, and deployment locations of border surveillance  technologies, a key analytical component leading to the Plan.  Documentation of the underlying analyses, not just the results, would  enable the analyses supporting the Plan to be independently assessed.  As noted in the report, it is unclear whether and how the analyses  conducted to develop the Plan demonstrated the cost and operational  effectiveness of the selected mix of technology, including whether the  most appropriate technology for the terrain was selected. CBP cannot  demonstrate the validity of the Arizona Border Surveillance Technology  Plan and its acquisition approach in the absence of documentation that  describes how CBP developed the operational assessments and  technology deployment analyses and used the results of the AoA to  develop the types and quantities of technologies and their suitability to the  terrain from the various alternatives.  Further, in the light of the significant  difficulties faced by CBP in its prior efforts to develop and implement the  nearly $1 billion SBInet system to provide unquantified improvements in  border surveillance along 53 miles of the Arizona border after 5 years of  program efforts, we remain concerned that CBP lacks reasonable  assurance that its Plan will fully support its future deployments of border  surveillance technology in Arizona.
The second issue DHS raised regarded the report’s observations about  limitations of SBInet systems currently fielded in Arizona and the need for  CBP to address operational test results. DHS did concur with the  recommendation that CBP conduct a post-implementation review and  operational assessment of SBInet. However, DHS said that, because of  the Border Patrol’s ongoing mitigation efforts and a planned system  enhancement to address these limitations, they are unable to address this  issue at this time. DHS added that they plan to continue to use the  system to maintain enhanced situational awareness while gaining  additional experience with the system until the planned system  enhancement can be implemented in 2012 to address operational  concerns.
The third issue regarded the report’s observations about limitations of the  Plan’s cost estimates and the potential sufficiency of contingency funds to  accommodate unforeseen cost growth. DHS said that CBP program  officials “are mindful” of this concern, were conservative in their budget  requests, and believe this issue has been largely addressed by their prior  efforts to accommodate reasonable cost contingencies. However, DHS  added that, in response to the related recommendation, it is preparing  updated life-cycle cost estimates, consistent with the GAO’s best practice  guidelines, for two projects in the Plan that account for 90 percent of the  estimate. But to fully address this recommendation, DHS will need to  implement best practices for the entire Plan, not just for the two largest  projects, so that the impacts of integrating the separate projects can be  fully understood. DHS and DOD provided technical comments, which we  incorporated as appropriate.
We are sending copies of this report to the Secretary of Homeland  Security, the Commissioner of the U.S. Customs and Border Protection,  and interested congressional committees. In addition, the report will be  available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staff have questions regarding this report, please contact  me at (202) 512-8777 or at StanaR@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine the extent to which (1) U.S. Customs  and Border Protection (CBP) has the information needed to fully support  and implement its Arizona Border Surveillance Technology Plan in  according with Department of Homeland Security (DHS) and Office of  Management and Budget (OMB) guidance, and (2) CBP’s life-cycle cost  estimate for the Arizona Border Surveillance Technology Plan reflects  best practices.
To answer our first objective, we reviewed key program-planning  documents CBP relied on to support its new approach to identifying,  acquiring, and deploying surveillance technology applicable to specific  types of terrain along the Arizona border. We also interviewed CBP  officials responsible for assessing the need for and documenting the cost- and operational effectiveness and suitability of proposed systems to  support its Arizona Border Surveillance Technology Plan and for  identifying appropriate metrics to assess progress in border security.  Specifically, we reviewed the announcement of the Secretary of  Homeland Security and her vision of CBP’s new approach to identifying,  acquiring, and deploying surveillance technology to the Arizona border in  support of Border Patrol’s mission, principle goal, and objective. We also  reviewed CBP’s analysis of alternatives (AOA) for Arizona, the Arizona  Border Surveillance Technology Plan informed by the AOA, the final  report of the independent peer review team on the AOA, CBP’s request  for information on integrated fixed-tower technology, its Industry Day  announcement and answers to industry questions, and CBP’s  comparison of the similarities and differences between integrated fixed  towers and SBInet technology.
In relation to operational test results, we reviewed what the independent  evaluation of SBInet and discussed with officials the extent CBP is using  these findings to inform future investments as well as the continuing  operation of SBInet. We largely focused on the elements of SBInet known  as Block 1, developed and deployed in Arizona’s Tucson sector between  2005 and 2010 and CBP’s plans for its operation and maintenance over  its life-cycle. In doing so, we reviewed program documentation, including  the Army Test and Evaluation Command’s reports and briefing to CBP,  and interviewed the key officials involved in the design and  implementation of the operational test and evaluation of test results in  order to determine the reliability of the information we used to support our  finding. We compared CBP’s program management plans and activities  with requirements in DHS acquisition regulations including Acquisition  Regulation 102-01 and OMB guidance A-11. We also interviewed CBP  officials from its Office of Technology Innovation and Acquisition (OTIA)  on how they intended to use the operational test findings and  recommendations to inform the continuing operation of existing SBInet  technology. Specifically, we reviewed the Army’s operational test plans,  the initial and final test and evaluation reports, and their “Quick Look”  briefing to OTIA officials. We also interviewed CBP and Army officials  about the results of those tests and discussed the soundness of the test  design process, its sampling methodology, and its implementation in  order to determine whether we could rely on test results data. We found  the test results to be sufficiently reliable for the purposes of this report.  We also observed the SBInet systems in operation in the Tucson sector,  and discussed the systems’ performance with Border Patrol Agents in the  Tucson and Ajo station SBInet command centers. We reviewed our body  of work on SBInet since 2005 as a basis for assessing CBP’s proposed  approach for developing and implementing its new Arizona Border  Surveillance Technology Plan.
To answer our second objective, we reviewed cost and budget  documents CBP relied on to support cost estimates for technology  alternatives contained in the AOA for Arizona and in the President’s  budget request for fiscal year 2012. We also interviewed program officials  and contractors responsible for estimating the cost of future investments  in surveillance technology, specifically the life-cycle approach,  requirements development and management, test management, and risk  management. We then compared this information to relevant federal  guidance derived from leading industry practices. To assess the  reliability of the cost data for the rough order-of-magnitude estimate for  implementation of the Plan, which assumed a 10-year life-cycle for the  acquisition, we relied on data for fiscal year 2010 and beyond to support  the findings in the report. We also reviewed relevant program  documentation to substantiate evidence obtained through interviews with  knowledgeable agency officials, where available, regarding the integrity of  the data. We determined that the data used in this report are sufficiently  reliable for the purposes of this report. We compared CBP cost  estimating practices and budget documents to our Cost Estimating and  Assessment Guide, which contains best practices compiled from cost- estimating organizations throughout the federal government and  industry.
We conducted this performance audit from March 2011 through October  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Photographs of Technologies Contained in the Arizona Border Surveillance Technology Plan
section_title: Appendix III: Aspects of High-Quality Cost Estimates
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Chris Keisling, Assistant Director, and Ron Salo, Analyst-in-Charge,  managed this assignment. David Alexander, Seto Bagdoyan, Charles  Bausell, Justin Dunleavy, Mike Harmond, Richard Hung, Karen Richey,  and Sean Seales made important contributions to this report. Frances  Cook provided legal assistance, and Tina Cheng provided graphics  assistance. Katherine Davis contributed to report preparation.
section_title: Related GAO Products
paragraphs: Homeland Security: DHS Could Strengthen Acquisitions and  Development of New Technologies. GAO-11-829T (Washington, D.C.:  July 15, 2011).
Border Security: DHS Progress and Challenges in Securing the U.S.  Southwest and Northern Borders. GAO-11-508T (Washington, D.C.:  March 30, 2011).
Border Security Preliminary Observations on the Status of Key Southwest  Border Technology Programs. GAO-11-448T (Washington, D.C March  15, 2011).
Secure Border Initiative: DHS Needs to Strengthen Management and  Oversight of Its Prime Contractor. GAO-11-6 (Washington, D.C.: October  18, 2010).
U.S. Customs and Border Protection’s Border Security Fencing,  Infrastructure and Technology Fiscal Year 2010 Expenditure Plan,  GAO-10-877R (Washington, D.C.: July 30, 2010).
Department of Homeland Security: Assessments of Selected Complex  Acquisitions, GAO-10-588SP (Washington, D.C.: June 30, 2010).
Secure Border Initiative: DHS Needs to Reconsider Its Proposed  Investment in Key Technology Program. GAO-10-340 (Washington, D.C.:  May, 5, 2010).
Secure Border Initiative: DHS Has Faced Challenges Deploying  Technology and Fencing Along the Southwest Border, GAO-10-651T  (Washington, D.C.: May 4, 2010).
Secure Border Initiative: Testing and Problem Resolution Challenges Put  Delivery of Technology Program at Risk. GAO-10-511T (Washington,  D.C.: March 18, 2010).
Secure Border Initiative: DHS Needs to Address Testing and  Performance Limitations That Place Key Technology Program at Risk.  GAO-10-158 (Washington, D.C.: January 29, 2010).
Secure Border Initiative: Technology Deployment Delays Persist and the  Impact of Border Fencing Has Not Been Assessed. GAO-09-1013T  (Washington, D.C.: September 17, 2009).
Secure Border Initiative: Technology Deployment Delays Persist and the  Impact of Border Fencing Has Not Been Assessed. GAO-09-896  (Washington, D.C.: September 9, 2009).
Customs and Border Protection’s Secure Border Initiative Fiscal Year  2009 Expenditure Plan. GAO-09-274R (Washington, D.C.: April 30,  2009).
Secure Border Initiative Fence Construction Costs. GAO-09-244R  (Washington, D.C.: January 29, 2009).
Northern Border Security: DHS’s Report Could Better Inform Congress by  Identifying Actions, Resources, and Time Frames Needed to Address  Vulnerabilities. GAO-09-93 (Washington, D.C.: November 25, 2008).
Department of Homeland Security: Billions Invested in Major Programs  Lack Appropriate Oversight. GAO-09-29 (Washington, D.C.: November  18, 2008).
Secure Border Initiative: DHS Needs to Address Significant Risks in  Delivering Key Technology Investment. GAO-08-1086 (Washington, D.C.:  September 22, 2008).
Secure Border Initiative: DHS Needs to Address Significant Risks in  Delivering Key Technology Investment. GAO-08-1148T (Washington,  D.C.: September 10, 2008).
Secure Border Initiative: Observations on Deployment Challenges.  GAO-08-1141T (Washington, D.C.: September 10, 2008).
Secure Border Initiative Fiscal Year 2008 Expenditure Plan Shows  Improvement, but Deficiencies Limit Congressional Oversight and DHS  Accountability. GAO-08-739R (Washington, D.C.: June 26, 2008).
Department of Homeland Security: Better Planning and Oversight  Needed to Improve Complex Service Acquisition Outcomes.  GAO-08-765T (Washington, D.C.: May 8, 2008).
Department of Homeland Security: Better Planning and Assessment  Needed to Improve Outcomes for Complex Service Acquisitions.  GAO-08-263 (Washington, D.C.: April 22, 2008).
Secure Border Initiative: Observations on the Importance of Applying  Lessons Learned to Future Projects. GAO-08-508T (Washington, D.C.:  February 27, 2008).
Secure Border Initiative: Observations on Selected Aspects of SBInet  Program Implementation. GAO-08-131T (Washington, D.C.: October 24,  2007).
Secure Border Initiative: SBInet Planning and Management  Improvements Needed to Control Risk. GAO-07-504T (Washington, D.C.:  February 27, 2007).
Secure Border Initiative: SBInet Expenditure Plan Needs to Better  Support Oversight and Accountability. GAO-07-309 (Washington, D.C.:  February 15, 2007).