Arizona Border Surveillance Technology Plan: Additional Actions Needed to Strengthen Management and Assess Effectiveness
Published: 2014-03-03T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: The Department of Homeland Security's (DHS) U.S. Customs and Border Protection's (CBP) schedules and Life-cycle Cost Estimates for the Arizona Border Surveillance Technology Plan (the Plan) reflect some, but not all, best practices. Scheduling best practices are summarized into four characteristics of reliable schedules—comprehensive, well constructed, credible, and controlled (i.e., schedules are periodically updated and progress is monitored). GAO assessed CBP's schedules as of March 2013 for the three highest-cost programs that represent 97 percent of the Plan's estimated cost. GAO found that schedules for two of the programs at least partially met each characteristic (i.e., satisfied about half of the criterion), and the schedule for the other program at least minimally met each characteristic (i.e., satisfied a small portion of the criterion), as shown in the table below. For example, the schedule for one of the Plan's programs partially met the characteristic of being credible in that CBP had performed a schedule risk analysis for the program, but the risk analysis was not based on any connection between risks and specific activities. For another program, the schedule minimally met the characteristic of being controlled in that it did not have valid baseline dates for activities or milestones by which CBP could track progress.
Source: GAO analysis of CBP data.
Note: Not met—CBP provided no evidence that satisfies any of the criterion. Minimally met—CBP provided evidence that satisfies a small portion of the criterion. Partially met—CBP provided evidence that satisfies about half of the criterion. Substantially met—CBP provided evidence that satisfies a large portion of the criterion. Met—CBP provided complete evidence that satisfies the entire criterion.
Further, CBP has not developed an Integrated Master Schedule for the Plan in accordance with best practices. Rather, CBP has used the separate schedules for each program to manage implementation of the Plan, as CBP officials stated that the Plan contains individual acquisition programs rather than integrated programs. However, collectively these programs are intended to provide CBP with a combination of surveillance capabilities to be used along the Arizona border with Mexico, and resources are shared among the programs. According to scheduling best practices, an Integrated Master Schedule is a critical management tool for complex systems that involve a number of different projects, such as the Plan, to allow managers to monitor all work activities, how long activities will take, and how the activities are related to one another. Developing and maintaining an Integrated Master Schedule for the Plan could help provide CBP a comprehensive view of the Plan and help CBP better understand how schedule changes in each individual program could affect implementation of the overall Plan.
Moreover, cost-estimating best practices are summarized into four characteristics—well documented, comprehensive, accurate, and credible. GAO's analysis of CBP's estimate for the Plan and estimates completed at the time of GAO's review for the two highest-cost programs showed that these estimates at least partially met three of these characteristics: well documented, comprehensive, and accurate. In terms of being credible, these estimates had not been verified with independent cost estimates in accordance with best practices. Ensuring that scheduling best practices are applied to the three programs' schedules and verifying Life-cycle Cost Estimates with independent estimates could help better ensure the reliability of the schedules and estimates.
CBP did not fully follow key aspects of DHS's acquisition management guidance for the Plan's three highest-cost programs. For example, CBP plans to conduct limited testing of the highest-cost program—the Integrated Fixed Tower (IFT: towers with cameras and radars)—to determine its mission contributions, but not its effectiveness and suitability for the various environmental conditions, such as weather, in which it will be deployed. This testing, as outlined in CBP's test plan, is not consistent with DHS's guidance, which states that testing should occur to determine effectiveness and suitability in the environmental conditions in which a system will be used. Revising the test plan to more fully test the program in the conditions in which it will be used could help provide CBP with more complete information on how the towers will operate once they are fully deployed.
CBP has identified mission benefits for technologies under the Plan, but has not yet developed performance metrics. CBP has identified such mission benefits as improved situational awareness and agent safety. Further, a DHS database enables CBP to collect data on asset assists, defined as instances in which a technology, such as a camera, or other asset, such as a canine team, contributed to an apprehension or seizure, that in combination with other relevant performance metrics or indicators, could be used to better determine the contributions of CBP's surveillance technologies and inform resource allocation decisions. However, CBP is not capturing complete data on asset assists, as Border Patrol agents are not required to record and track such data. For example, from fiscal year 2010 through June 2013, Border Patrol did not record whether an asset assist contributed to an apprehension event for 69 percent of such events in the Tucson sector. Requiring the reporting and tracking of asset assist data could help CBP determine the extent to which its surveillance technologies are contributing to CBP's border security efforts.
This is a public version of a For Official Use Only—Law Enforcement Sensitive report that GAO issued in February 2014. Information DHS deemed as For Official Use Only—Law Enforcement Sensitive has been redacted.
section_title: What GAO Recommends
paragraphs: GAO recommends that CBP, among other things, apply scheduling best practices, develop an integrated schedule, verify Life-cycle Cost Estimates, revise the IFT test plan, and require tracking of asset assist data. DHS concurred with four of six GAO recommendations. It did not concur with the need for an integrated schedule or a revised IFT test plan. As discussed in this report, GAO continues to believe in the need for a schedule and a revised test plan.

section_title: Letter
paragraphs: In recent years, nearly half of all annual apprehensions of illegal entrants  along the southwest border with Mexico have occurred along the Arizona  border, according to Department of Homeland Security (DHS) data. A top  priority for DHS’s U.S. Customs and Border Protection (CBP) is  preventing, detecting, and apprehending illegal entrants. In November  2005, DHS announced the launch of the Secure Border Initiative (SBI), a  multiyear, multibillion-dollar program aimed at securing U.S. borders and  reducing illegal immigration. CBP intended for the SBI Network (SBInet)  to include technologies such as fixed sensor towers, a common operating  picture, and tactical infrastructure to create a “virtual fence” along the  southwest border to enhance CBP’s capability to detect, identify, classify,  track, and respond to illegal breaches at and between land ports of entry.  At a cost of about $1 billion, in 2010, CBP deployed SBInet systems,  referred to as Block 1 systems, along the 53 miles of Arizona’s 387-mile  border with Mexico that represent one of the highest-risk areas for illegal  entry attempts. However, in January 2011, in response to internal and  external assessments that identified concerns regarding the performance,  cost, and schedule for implementing the systems, the Secretary of  Homeland Security announced the cancellation of further procurements of  SBInet systems.
After the cancellation of SBInet in January 2011, CBP developed the  Arizona Border Surveillance Technology Plan (the Plan), which includes a  mix of radars, sensors, and cameras to help provide security for the  remainder of the Arizona border. Under the Plan, CBP identified seven  programs to be implemented ranging in estimated costs from $3 million to  about $961 million. The three highest-cost programs under the Plan are  the Integrated Fixed Tower (IFT), Remote Video Surveillance System  (RVSS), and Mobile Surveillance Capability (MSC), accounting for 97  percent of the Plan’s estimated cost. In November 2011, we reported on  CBP’s development of, and estimated life-cycle costs for implementing,  the Plan. Specifically, we reported that CBP needed more information for  the Plan and its costs before proceeding with implementation, and we  recommended that CBP (1) ensure the underlying analyses of the Plan  were documented in accordance with DHS guidance and internal control  standards, (2) determine the mission benefits to be derived from the  implementation of the Plan and develop and apply key attributes for  metrics to assess program implementation, (3) conduct a post-  implementation review and operational assessment of SBInet, and (4)  update the cost estimate for the Plan using best practices. DHS  concurred with these recommendations and has actions under way to  address some of them, which we discuss later in this report.
Further, in September 2012, we reported on acquisition management at  DHS. Specifically, we found that DHS acquisition policy reflects many key management practices that could help mitigate risks and increase  chances for successful outcomes; however, most of DHS’s major  acquisition programs continued to cost more than expected, took longer  to deploy than planned, or delivered less capability than promised. These  challenges in DHS’s acquisition management, as well as in the  department’s other management functions, such as financial and human  capital management, have contributed to our designation of DHS’s  management functions as a high-risk area.
You asked us to review the status of DHS’s efforts to develop and  implement the Plan. This report addresses the following questions: To  what extent has CBP (1) developed schedules and Life-cycle Cost  Estimates for the Plan in accordance with best practices, (2) followed key  aspects of DHS’s acquisition management framework in managing the  Plan’s three highest-cost programs, and (3) assessed the performance of  technologies deployed under SBInet and identified mission benefits and  developed performance metrics for surveillance technologies to be  deployed under the Plan?
This report is a public version of the prior sensitive report that we  provided to you. DHS deemed some of the information in the prior report  as For Official Use Only—Law Enforcement Sensitive, which must be  protected from public disclosure. Therefore, this report omits sensitive  information on our analysis of Border Patrol data on apprehension and  seizure events relative to various surveillance technologies. Although the  information in this report is more limited in scope, it addresses the same  questions as the sensitive report. Also, the overall methodology used for  both reports is the same.
To address the first question, we analyzed DHS and CBP documents,  including program schedules and cost estimates, and interviewed DHS  and CBP officials responsible for developing and overseeing schedules  and cost estimates. Specifically, we obtained program schedules as of  March 2013, which were current at the time of our review, for the three  highest-cost programs—IFT, RVSS, and MSC—and compared the  schedules with best practices for developing schedules outlined in an  exposure draft of GAO’s Schedule Assessment Guide. We also  interviewed cognizant officials in CBP’s Office of Technology Innovation  and Acquisition (OTIA) and program offices. By assessing the schedules  against best practices, we identified schedule challenges that CBP was  experiencing in testing, procuring, deploying, and operating technologies  under the Plan, and interviewed CBP officials to determine reasons for  the schedule challenges and steps that CBP had taken or was taking to  address them. For Life-cycle Cost Estimates, we analyzed the Plan’s  June 2013 estimate, the IFT program’s January 2012 estimate, and the  RVSS program’s March 2012 estimate, which were the current estimates  at the time of our review, and compared them against best practices for  cost estimating. We also analyzed DHS and CBP documents and  interviewed officials regarding their efforts to implement our prior  recommendations to update the August 2010 Life-cycle Cost Estimate for  the Plan in accordance with best practices. To assess the reliability of  cost estimate data, we reviewed relevant program documentation, such  as cost estimation spreadsheets, as available, to substantiate evidence  obtained from interviews with knowledgeable agency officials. We found  the data to be sufficiently reliable for the purposes of our report.
To address the second question, we analyzed DHS and CBP documents,  including DHS Acquisition Management Directive 102-01 and its  associated DHS Instruction Manual 102-01-001, program briefing slides,  budget documents, Acquisition Decision Memorandums, schedules, and  program risk sheets. We focused on the IFT, RVSS, and MSC programs  for more in-depth analyses because they are the Plan’s three highest-cost  programs and represent 97 percent of the estimated cost of the Plan.  Specifically, to assess the acquisition strategy for the Plan, we focused on  the IFT, RVSS, and MSC programs and analyzed their respective  acquisition plans and discussed the acquisition approaches with CBP  officials. To assess system requirements and capabilities for the IFT,  RVSS, and MSC programs, we analyzed requirements and capabilities  documents and worked with CBP officials to identify any changes since  the documents were initially approved and whether any requirements or  capabilities had been traded off because of cost, schedule, or other  purposes. To assess the extent to which CBP followed DHS acquisition  guidance, we selected aspects of Acquisition Management Directive 102- 01 that were relevant to where these programs were in the acquisition  process during fiscal year 2013. Specifically, we determined whether  acquisition documents had been approved by the time required, that is,  by the applicable Acquisition Decision Events.
To address the third question, we analyzed performance assessment  documentation and metrics used by CBP to determine the effectiveness  of technologies deployed under SBInet and interviewed CBP officials  responsible for performance measurement activities. Specifically, we  analyzed the results of CBP’s January 2013 post implementation review  of the effectiveness of SBInet technologies in achieving their intended  results. We also analyzed CBP and DHS documents, such as plans to  address the post-implementation review, and interviewed officials to  assess corrective actions taken to improve SBInet performance issues.  We analyzed CBP data on apprehensions, seizures, and asset assists  from fiscal year 2010 through June 2013 to determine the extent to which  the data could be used to measure the contributions of SBInet  technologies in enhancing border security. We selected this time frame  because fiscal year 2010 was the first fiscal year for which data on asset  assists were available following Border Patrol’s deployment of its SBInet  technologies, and the collection of data on Geographic Information  Systems coordinates for apprehensions and seizures was required. To  assess the reliability of these data, we discussed data quality control  procedures with CBP officials. We determined that these data are  sufficiently reliable for the purposes of this report. We compared CBP’s  tracking and recording of data on asset assists for apprehensions and  seizures against criteria outlined in Standards for Internal Control in the  Federal Government. In addition, we visited the Tucson sector in  Arizona to observe Border Patrol agents operating technologies and  discuss agents’ experiences in using these technologies. We selected the  Tucson sector to visit because of the presence of surveillance  technologies, such as SBInet and RVSS towers, in that sector and  because the Tucson sector includes locations for which additional  technology deployments, such as IFTs, are planned. While the  information we obtained from our visit cannot be generalized to all Border  Patrol sectors, it provided us with insights about the use of the deployed  surveillance technologies. Furthermore, we interviewed CBP officials and  analyzed documents to determine the progress CBP and DHS had made  in implementing our prior recommendations to develop mission benefits to  be derived from technologies in the Plan and metrics to measure the  extent to which border security is expected to improve by using these  technologies. For more information on our scope and methodology, see  appendix I.
We conducted this performance audit from September 2012 to March  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Border Patrol and OTIA Roles and Responsibilities
paragraphs: Border Patrol has reported that its primary mission is to prevent terrorists  and weapons of terrorism from entering the United States and also to  detect, interdict, and apprehend those who attempt to illegally enter or  smuggle any person or contraband across the nation’s borders.  Geographic responsibility for the southwest border is divided among nine  Border Patrol sectors, two of which are in Arizona—Tucson and Yuma.  Each sector has a varying number of stations, with agents responsible for  patrolling within defined geographic areas. Border Patrol collects and  analyzes various data on its enforcement efforts and the number and  types of entrants who illegally cross the southwest border between the  land ports of entry. These data include apprehensions and seizures of  drugs and other contraband. The Border Patrol collects and maintains  data on apprehensions and seizures in DHS’s Enforcement Integrated  Database (EID). This database also includes an asset assists field in  which agents can specify whether an asset, such as SBInet surveillance  towers, contributed to apprehensions or seizures.
CBP’s OTIA was created to help ensure CBP’s technology efforts are  properly focused on the mission and are well integrated, and to  strengthen CBP’s expertise and effectiveness in program management  and acquisition. OTIA’s mission is to conduct and facilitate effective  identification, acquisition, and life-cycle support of products and services  while driving innovation to improve CBP’s performance in securing U.S.  borders and facilitating lawful movement of goods and people. OTIA  manages the implementation of the Plan and is acquiring seven  technology programs in the Plan for use by Border Patrol in Arizona. The  goal of the Plan is to achieve situational awareness along the Arizona  border where the Plan’s technologies are deployed. For fiscal year 2013,  OTIA budgeted $297 million in development and deployment funds for the  Plan’s seven technology programs. Table 1 describes the Plan’s  programs, and appendix II provides a photograph of each technology  program.
section_title: DHS’s Acquisition Life- Cycle Framework
paragraphs: The overall policy and structure for acquisition management outlined in  DHS Acquisition Management Directive 102-01 and its associated  Instruction Manual 102-01-001 includes an Acquisition Life-cycle  Framework to plan and execute the department’s acquisition programs.  According to the directive, DHS adopted the Acquisition Life-cycle  Framework to ensure consistent and efficient acquisition management,  support, review, and approval throughout the department. As shown in  figure 1, DHS’s Acquisition Life-cycle Framework includes four acquisition  phases through which DHS develops, deploys, and operates new  capabilities.
During the first three phases, the DHS component pursuing the  acquisition is required to produce key documents to justify, plan, and  execute the acquisition. These phases each culminate in an Acquisition  Decision Event where the Acquisition Review Board—a board of senior  DHS officials—determines whether a proposed acquisition has met the  requirements of the relevant acquisition framework phase and should  proceed. The Acquisition Review Board is chaired by the Acquisition  Decision Authority—the official responsible for ensuring compliance with  Acquisition Management Directive 102-01. DHS classifies acquisitions  into three levels that determine whether the Acquisition Decision Authority  can be a Component Acquisition Executive or should be DHS’s Deputy  Secretary or Under Secretary for Management.program is a Level 2 acquisition, which is overseen by the department,  and the DHS Under Secretary for Management serves as the Acquisition  Decision Authority. The other six programs in the Plan are Level 3  acquisitions, which are overseen by CBP’s Acquisition Review Board, and  the Acquisition Decision Authority is a CBP official who serves as both the  Assistant Commissioner for OTIA and Component Acquisition Executive.
section_title: Status of the Plan’s Seven Programs
paragraphs: As of January 2014, CBP has awarded contracts for four of the Plan’s  seven programs and has initiated or completed deployment of technology  to Arizona for three of the four programs under contract, as shown in  table 2.
section_title: CBP’s Program Schedules and Life- Cycle Cost Estimates Reflect Some but Not All Best Practices
subsections: section_title: CBP’s Program Schedules Have Experienced Delays and the Three Highest- Cost Programs Meet Some but Not All Scheduling Best Practices
subsections: section_title: The Majority of the Plan’s Programs Experienced Some Delays, and Schedules for the IFT, RVSS, and MSC Programs Are Not Fully Reliable
paragraphs: OTIA has developed a schedule for each of the Plan’s seven programs,  and four programs will not meet their originally planned completion dates.  OTIA established schedules for each program, serving as the original  program plans with the required sequence of events, resource  assignments, and dates for deliverables. However, as of March 2013,  five of the Plan’s programs—IFT, RVSS, MSC, APSS, and UGS/IS—have  experienced delays relative to their baseline schedules, as shown in  figure 2.
OTIA officials attributed program delays to various factors, including  higher than expected numbers of proposals from vendors for some of the  programs, system performance problems, and limited resources. In  particular, OTIA officials stated that they initiated acquisitions for a  number of the Plan’s programs around the same time, but OTIA did not  have a sufficient number of acquisition staff with sufficient experience and  skills to review contract proposals or manage the programs, a fact that  contributed to program delays.for both the IFT and RVSS programs, the source selection process to  decide which vendor would be awarded the contract was extended  because of a higher than expected number of proposals received from  vendors and a limited acquisition workforce to review and process the  proposals, including not having a dedicated contracting officer for each of  the programs. In addition, for the MSC program, OTIA officials attributed  delays to problems both vendors who were awarded contracts  experienced with their systems, as previously discussed. OTIA took  various actions in response to these delays, such as extending the  scheduled contract award date for the IFT and RVSS programs and  extending scheduled activities for the MSC program from July 2014 to  September 2015.
For example, OTIA officials stated that  According to best practices, in acquisition programs, agencies may make  modifications to program schedules to reflect changes to programs; CBP  has consistently updated each program’s schedule in response to  program delays. However, we assessed OTIA’s schedules as of March  2013 for the three highest-cost technology programs—IFT, RVSS, and  MSC—and found that these program schedules addressed some, but not  all, best practices for scheduling. The Schedule Assessment Guide  identifies 10 best practices associated with effective scheduling, which  are summarized into four characteristics of a reliable schedule— comprehensive, well constructed, credible, and controlled. summarizes our assessment of the IFT, RVSS, and MSC schedules.  Appendix III provides more detailed information on the description of each  best practice and on the results of our assessment.
GAO-12-120G.
A schedule risk analysis is performed to calculate the amount of contingency time that is needed to  complete the program on time.
According to our overall analysis, OTIA at least partially met the four  characteristics of reliable schedules for the IFT and RVSS schedules and  partially or minimally met the four characteristics for the MSC schedule.  For example:
Comprehensive: OTIA’s schedule for the IFT, RVSS, and MSC  programs partially met best practices in terms of being  comprehensive. For example, our analysis found that all three  program schedules reflected the work that needed to be  accomplished for the schedules, and each schedule had duration  estimates that at least substantially met best practices. The IFT  schedule contained a clear start and a finish milestone, and the RVSS  schedule contained at least a clear start milestone. However, the  schedules for these programs did not meet other best practices in  terms of being comprehensive. For example, the MSC schedule did  not contain fields that map activities to a program work breakdown  structure; and the schedules for the IFT and RVSS programs did not  fully map all schedule activities to each program’s work breakdown  structure in accordance with best practices. Moreover, the IFT and  RVSS schedules did not include the level of detail expected to provide  oversight of ongoing construction work, as activities associated with  the construction work were reflected in the schedules as milestones,  limiting OTIA’s ability to monitor the progress of these efforts.  Specifically, these activities were reflected in the schedules as a  milestone that was a point in time, rather than a range of time, as  called for by best practices. In addition, resources were not assigned  to some activities in all three schedules. According to best practices, a  schedule without resources implies an unlimited number of resources  and their unlimited availability. Best practices note that assigning  resources to activities across programs can help prevent any future  overallocation of resources.
Well constructed: OTIA’s schedule for the IFT program substantially  met the characteristic of being well constructed; the schedules for  RVSS and the MSC programs partially met this characteristic. For  example, our analysis found the IFT program schedule had few  missing or incorrect logic links and the critical path—the chain of  dependent activities with the longest total duration—was found to be a  straightforward, continuous path of activities that depicted the effort  driving the key milestones. Our analysis of the RVSS and MSC  program schedules found that these schedules had no missing or  incorrect logic links. However, we could not verify a reliable critical  path that was continuous from the status date to contract award for  these schedules. In addition, our analysis shows that each of the  three programs’ schedules exhibited unreasonable amounts of total  float—that is, the amount of time by which an activity can slip before  the delay affects the program’s estimated finish date appeared to be  overestimating true schedule flexibility. For example, 25 percent of the  activities in the IFT schedule appeared to be able to slip at least 10  working months before affecting the final milestone of the program.
Credible: OTIA’s schedules for the IFT and RVSS programs partially  met the characteristic of being credible; the MSC program schedule  minimally met this characteristic. For example, our analysis found that  the IFT and RVSS schedules responded when significant delays were  introduced into the planned activities in the schedules; that is, when  we tested the robustness of the schedules by extending activity  durations, forecasted dates recalculated appropriately. However, the  MSC schedule responded to schedule delays in some instances but  not in others, and some forecasted dates did not recalculate to  account for changes we made in the duration of activities when testing  the MSC schedule. Additionally, OTIA performed a risk analysis for  the IFT and RVSS programs; however, the IFT and RVSS analyses  did not include the risks most likely to delay the project or how much  contingency reserve (that is, time held in reserve for potential delays)  was needed for each schedule. For the MSC schedule, OTIA did not  conduct a schedule risk analysis because, according to program  officials, OTIA did not have a tool for conducting schedule risk  assessment at the time the MSC schedule was developed. According  to best practices, without this analysis, the program office may not  sufficiently understand the level of confidence in meeting the  program’s completion date and identify any potential reserves for  contingencies.
Controlled: OTIA’s schedules for the IFT and the RVSS programs  partially met the characteristic of being controlled; the MSC program  schedule minimally met this characteristic. For example, our analyses  determined all three schedules were well maintained, updated  periodically by a trained scheduler, and contained no out-of-sequence  activities. We also found that the IFT and the RVSS schedules  contained no date anomalies, but the MSC schedule did have  anomalies. For example, the MSC schedule contained 13 activities in  the past with no actual start or finish dates. Further, our analysis  showed that none of the schedules had valid baseline dates for  activities or milestones by which management could track current  performance. The IFT baseline schedule was originally approved in  July 2011, and the baseline for the RVSS was approved in September  2012; however, both of these programs have been delayed.  Rebaselining resets the estimated schedule that is used to determine  how the program will be held accountable. Once a program is  rebaselined, OTIA officials stated that the office plans to report on the  performance of the program based on the revised schedule. However,  none of the schedules we assessed contained valid baseline dates  that could be used to track on-time, delayed, or accelerated effort. For  example, a baseline schedule was not established for the MSC  program and both the IFT and RVSS schedules were missing some  baseline dates for activities and milestones. In addition, according to  our analyses, none of the three schedules were supported by a  schedule baseline document, which is a single document that defines  the organization of a schedule, describes the logic of the network,  describes the basic approach to managing resources, and provides a  basis for all parameters used to calculate dates. OTIA officials stated  that the Acquisition Program Baseline for both the IFT and RVSS  serves as the baseline schedule document, which defines the cost,  schedule, and performance baselines; however, the Acquisition  Program Baseline and related guidance present an overview of OTIA  schedule policy rather than assumptions specific to individual program  schedules.
OTIA officials stated that they believe the schedules for the IFT, RVSS,  and MSC programs are generally reliable, but also stated that these  schedules may not fully meet all best practices. OTIA officials stated that  they plan to rebaseline the IFT and RVSS program schedules after  contract award and the MSC program schedule after contract  negotiations. Rebaselining these schedules would help OTIA better  address some of the best practices, such as to help ensure a more full  and consistent allocation of resources, to address gaps in the critical path  to program completion, and to address schedule risk assessments.  However, OTIA’s plans to rebaseline the schedules would not position  OTIA to meet all best practices, which are designed to ensure reliable  schedules. According to best practices, to be considered reliable, a  schedule must substantially or fully meet all four schedule characteristics.  As our analysis indicates, OTIA does not have the information it needs in  the schedules to effectively use them in managing and overseeing the  IFT, RVSS, and MSC programs. While OTIA’s plans to rebaseline the  schedules are positive steps, ensuring that all schedule best practices are  applied to the IFT, RVSS, and MSC schedules when updating them could  help OTIA better ensure the reliability of the three programs’ schedules  and could help better position OTIA to identify and address any potential  further delays in the programs’ commitment dates.
section_title: OTIA Does Not Have an Integrated Master Schedule for the Plan
paragraphs: OTIA has not developed an Integrated Master Schedule for scheduling,  executing, and tracking the work to implement the Plan and its seven  programs. Rather, OTIA has used the separate schedules for each  individual program (or “project”) to manage implementation of the Plan.  The use of an Integrated Master Schedule is a well-established practice  in program and project management and is a necessary tool for  coordination of independently managed projects that have dependencies—including resource dependencies—on one another.According to schedule best practices, an Integrated Master Schedule  shows the effect of delayed or accelerated government activities on  contractor activities, as well as the opposite effect for multiple programs.  In addition, an Integrated Master Schedule that allows managers to  monitor all work activities, how long the activities will take, and how the  activities are related to one another is a critical management tool for  complex systems that involve the incorporation of a number of different  projects, such as the Plan.
OTIA officials stated that an Integrated Master Schedule for the  overarching Plan is not needed because the Plan contains individual  acquisition programs as opposed to a plan consisting of seven integrated  programs. However, collectively, these programs are intended to provide  Border Patrol with a combination of surveillance capabilities to assist in  achieving situational awareness along the Arizona border with Mexico, as  referenced in CBP’s planning documents.  As a document that integrates  the planned work, the resources necessary to accomplish that work, and  the associated budget, an Integrated Master Schedule provides  information and oversight regarding the schedule. According to best  practices, an Integrated Master Schedule also helps agencies monitor  progress against overall completion dates. However, OTIA has not  established a target completion date for an Integrated Master Schedule  for the overall Plan.
Department of Homeland Security, Multi-Year Investment and Management Plan for  Border Security Fencing, Infrastructure, and Technology (BSFIT) for Fiscal Years 2014- 2017 (Washington, D.C.: Apr. 17, 2013). dedicated contracting officers to plan and execute the programs’ source  selection and environmental activities. Developing and maintaining an  Integrated Master Schedule for the Plan could allow OTIA insight into  current or programmed allocation of resources for all programs as  opposed to attempting to resolve any resource constraints for each  program individually. Because OTIA does not have an Integrated Master  Schedule for the Plan, it is not well positioned to understand how  schedule changes in each individual program could affect implementation  of the overall Plan. An Integrated Master Schedule could also help  provide CBP a comprehensive view of the Plan and help CBP to reliably  commit to when the Plan will be fully implemented, as well as help CBP to  better predict whether estimated completion dates are realistic to manage  programs’ performance.
section_title: OTIA Does Not Have Life- Cycle Cost Estimates for the Plan or Its Two Highest-Cost Programs That Fully Meet Best Practices
paragraphs: OTIA has developed a rough order of magnitude estimate for the Plan  and individual Life-cycle Cost Estimates for the IFT and RVSS programs  that meet some but not all best practices for such estimates. Best  practices for cost estimating and Office of Management and Budget  guidance emphasize that reliable cost estimates are important for  program approval and continued receipt of annual funding. DHS policy  similarly provides that Life-cycle Cost Estimates are essential to an  effective budget process and form the basis for annual budget decisions.  Reliable Life-cycle Cost Estimates reflect four characteristics—they are  (1) well documented, (2) comprehensive, (3) accurate, and (4) credible— which encompass 12 best practices. For example, a best practice for a  credible cost estimate is independently verifying a program’s Life-cycle  Cost Estimate with an independent cost estimate and reconciling any  differences.
In August 2010, OTIA developed a rough order of magnitude cost  estimate for the Plan—a high-level estimate without much detail—which  was about $1.54 billion, including approximately $750 million in  acquisition costs and approximately $800 million in operations and  maintenance costs. In June 2013, OTIA revised this cost estimate for the  Plan, estimating the cost at $1.39 billion, including about $480 million in  acquisition costs and about $910 million in operations and maintenance  costs. According to OTIA officials, some of the differences in costs  between the August 2010 and June 2013 estimates are attributable to  using more current information for the June 2013 estimate. Table 4  provides the June 2013 estimated cost and number of units to be  procured and deployed for each of the Plan’s seven programs.
In November 2011, we reported on the results of our analysis of the  Plan’s August 2010 estimate. Specifically, we found that the August  2010 estimate substantially met best practices in terms of being  comprehensive and accurate, and partially met best practices in terms of  being well documented. For example, we reported that, in terms of being  comprehensive, the estimate included documented technical data. In  terms of accuracy, we reported that the cost estimate was continually  updated and refined as more information became known. However, we  also found that the August 2010 estimate minimally met best practices for  being credible. For example, CBP officials had not conducted a sensitivity  analysis and a cost-risk and uncertainty analysis to determine a level of  confidence in the estimate, nor did CBP compare it with an independent  estimate. At that time, OTIA officials stated that CBP’s approach was to  develop and report an initial rough order of magnitude cost estimate for  the programs in the Plan, not necessarily a Life-cycle Cost Estimate that  met all best practices. In our November 2011 report, we recommended  that CBP update its August 2010 cost estimate for the Plan using best  practices, so that the estimate would be comprehensive, accurate, well   CBP concurred with the recommendation.  documented, and credible.
In November 2012, OTIA officials told us that CBP no longer intends to  develop a Life-cycle Cost Estimate for the Plan that meets all best  practices. OTIA officials also stated that they used a risk-based approach  to improve cost-estimating certainty and confidence by focusing on the  Life-cycle Cost Estimates for the IFT and RVSS programs, which  compose 90 percent of the Plan’s estimated cost. According to the  officials, developing a Life-cycle Cost Estimate for the Plan that followed  all best practices at this point in the acquisition cycle would not contribute  much cost management benefit because a number of programs are under  contract and units were being deployed to the field. However, as we  recommended in November 2011, we continue to believe that a Life-cycle  Cost Estimate for the Plan, developed using best practices, is needed to  ensure that the estimate is comprehensive, accurate, well documented,  and credible to help the agency and Congress fully understand the  impacts of the Plan’s various programs. Moreover, CBP’s June 2013  revised cost estimate for the Plan does not address the concerns we  identified in November 2011 with CBP’s original cost estimate. For  example, the IFT and RVSS programs compose 90 percent of the Plan’s  cost in the June 2013 Life-cycle Cost Estimate; however, OTIA has not  independently verified its Life-cycle Cost Estimates for the IFT and RVSS  programs with independent cost estimates and reconciled any differences  with each program’s respective Life-cycle Cost Estimate, consistent with  best practices. Furthermore, the remainder of the June 2013 Life-cycle  Cost Estimate is not fully documented. The costs for programs other than  the IFT and RVSS are provided as a summary program cost without a  detailed description provided. In contrast, the IFT and RVSS Life-cycle  Cost Estimates provided backup documentation, including labor hours  and methodology.
After CBP developed the initial cost estimate for the Plan in August 2010,  CBP developed separate Life-cycle Cost Estimates for the IFT and RVSS  programs in January and March 2012, respectively. The estimates for the  IFT and RVSS programs met some but not all best practices for cost  estimates. Specifically, our analysis shows that, in developing these  estimates, CBP partially documented the data used in the cost model for  the IFT’s Life-cycle Cost Estimate and fully documented the cost model  for the RVSS’s Life-cycle Cost Estimate. CBP also conducted a sensitivity  analysis and risk and uncertainty analysis to determine the level of  confidence in both Life-cycle Cost Estimates so that contingency funding  could be established relative to quantified risk. However, our analysis  showed that CBP did not independently verify its draft Life-cycle Cost  Estimates for the IFT and RVSS programs with independent cost  estimates and reconcile any differences with each program’s respective  Life-cycle Cost Estimate, consistent with best practices.
According to OTIA officials, the IFT program’s Life-cycle Cost Estimate  will be updated after the contract is awarded, the cost model for the  updated Life-cycle Cost Estimate will be fully documented in accordance  with best practices for cost estimating, and DHS’s Office of Program  Accountability and Risk Management is expected to review the updated  IFT Life-cycle Cost Estimate. Also, OTIA officials stated that they expect  to update the RVSS Life-cycle Cost Estimate and receive approval for it in  February 2014. However, OTIA is uncertain as to whether the updated  IFT and RVSS Life-cycle Cost Estimates will be verified with independent  cost estimates and any differences reconciled with the respective updated  Life-cycle Cost Estimates. Specifically, OTIA officials stated that the IFT  contract award will drive changes to the scope, schedule, and cost/budget  baseline for the IFT program; CBP plans to update the Life-cycle Cost  Estimate with programming and cost assumptions; and CBP plans to  provide the updated cost estimate to the department as part of a revised  submission of the Acquisition Program Baseline document. For the RVSS  program, OTIA officials stated that the contract award resulted in changes  that required updates to and reconciliation between the Cost Estimating  Baseline Document and the Life-cycle Cost Estimate for the program’s  scope, schedule, and cost/budget baseline. CBP intends to update the  RVSS program’s Life-cycle Cost Estimate with programming and cost  assumptions during the second quarter of fiscal year 2014 and provide  the updated cost estimate to DHS for review. However, according to OTIA  officials, as of November 2013, the agency had not yet determined  whether to independently verify or validate the IFT and RVSS Life-cycle  Cost Estimates. As CBP no longer intends to develop a Life-cycle Cost  Estimate for the entire Plan, when updating the IFT and RVSS Life-cycle  Cost Estimates, independently verifying the cost estimates and  reconciling any differences, in accordance with cost-estimating best  practices, could help better ensure the reliability of each estimate.
section_title: CBP Followed Some Aspects of DHS Acquisition Guidance, but Did Not Fully Complete Documents for Acquisition Decisions Consistent with the Guidance
subsections: section_title: CBP Followed Some Aspects of DHS Acquisition Guidance to Acquire Commercial-Off- the-Shelf Products
paragraphs: Consistent with DHS acquisition guidance, CBP tailored the DHS  Acquisition Life-cycle Framework for the IFT, RVSS, and MSC programs,  primarily because the agency’s strategy for the three programs includes  acquiring nondevelopmental technologies, preferably commercial-off-the- shelf systems, as opposed to developing technologies. As a result, rather  than entering the DHS acquisition framework at Acquisition Decision  Event 1, when a system includes technology development, the IFT  program entered at combined Acquisition Decision Events 2B/3, and the  RVSS and MSC programs entered at Acquisition Decision Event 2B.
In pursuing its strategy to acquire nondevelopmental systems for the  Plan’s three highest-cost programs, OTIA identified requirements and  capabilities for each program, consistent with DHS acquisition guidance.  Specifically, OTIA identified requirements for the IFT and RVSS programs  that were approved in 2012, and capabilities for the MSC program that  were developed in 2009. As part of the strategy to acquire commercial- off-the-shelf systems, CBP traded off, that is, reduced, some  requirements for the RVSS and expects to trade off some requirements  for the IFT for cost-effectiveness or schedule reasons. For example, with  regard to the RVSS, OTIA traded off two requirements because,  according to OTIA officials, they were not offered with the selected RVSS,  which presented the best value to the government while providing as  many requirements as possible. According to DHS Acquisition  Management Directive 102-01 guidance, as part of the acquisition  process, a program office may make trade-offs among performance, life- cycle cost, schedule, and risk. For example, the guidance states that a  small reduction in performance that does not impair the mission might  result in a large cost reduction.
section_title: DHS and CBP Did Not Consistently Approve Key Acquisition Documents in Accordance with Departmental Guidance
paragraphs: For the Plan’s three highest-cost programs, DHS and CBP did not  consistently approve key acquisition documents before or at the  Acquisition Decision Events, in accordance with DHS’s acquisition  guidance. An important aspect of an Acquisition Decision Event is the  review and approval of key acquisition documents critical to establishing  the need for a program, its operational requirements, an acquisition  baseline, and test and support plans, according to DHS guidance. DHS  Acquisition Management Directive 102-01—and the associated DHS  Instruction Manual 102-01-001 and appendixes—requires program offices  to develop documents demonstrating critical knowledge that would help  leaders make better-informed investment decisions when managing  individual programs. The DHS guidance provides information for  preparing acquisition documents, which require department- or  component-level approval before a program moves to the next acquisition  phase. In a September 2012 report, we found that while DHS had initiated  efforts to validate required acquisition documents in a timely manner at  major milestones, DHS leadership had authorized and continued to invest  in major acquisition programs even though the vast majority of those  programs lacked foundational documents demonstrating the knowledge  needed to help manage risks and measure performance.in September 2012 that this limited DHS’s ability to proactively identify  and address the challenges facing individual programs. We  recommended, among other things, that DHS ensure all major acquisition  programs fully comply with DHS acquisition policy by obtaining  department-level approval for key acquisition documents before  approving their movement through the acquisition life cycle. DHS  concurred and since the time of our September 2012 report has approved   We concluded  additional acquisition documents. However, DHS has not yet  demonstrated progress in obtaining department-level approval for most of  its major acquisition programs’ key acquisition documents.
On the basis of our analysis for IFT, RVSS, and MSC programs under the  Plan, the DHS Acquisition Decision Authority approved the IFT program  and the CBP Acquisition Decision Authority approved the RVSS and MSC  programs to proceed to subsequent phases in the Acquisition Life-cycle  Framework without approving all six required acquisition documents for  each program. We also found that one document for the IFT program, five  documents for the RVSS program, and two documents for the MSC  program were subsequently approved after the programs received  authority to proceed to the next phase. Table 5 provides a comparison of  when key acquisition documents were required to be approved and when  they were approved for the IFT, RVSS, and MSC programs.
We discuss the status of key acquisition documents for the three highest- cost programs below.
IFT program. Our analyses found that the DHS Acquisition Decision  Authority approved four of the six documents required at Acquisition  Decision Event 2B/3—the Acquisition Plan, Acquisition Program Baseline,  Integrated Logistics Support Plan, and Operational Requirements  Document—but did not approve two others—the Life-cycle Cost Estimate  and Test and Evaluation Master Plan. At the time of the Acquisition  Decision Event, CBP had a Life-cycle Cost Estimate for the IFT, but the  cost estimate had not yet been approved by DHS. According to OTIA  officials, the Life-cycle Cost Estimate for the IFT was discussed at the  Acquisition Decision Event 3 meeting and approved by the DHS Under  Secretary for Management and DHS’s Office of Program Accountability  and Risk Management. However, CBP did not provide documentation  showing that the estimate was approved by DHS.
The DHS Director of Operational Test and Evaluation approved the  revised IFT Test and Evaluation Master Plan on November 27, 2013, over   DHS and CBP officials  18 months after it was required to be approved.attributed the delay in approving the Test and Evaluation Master Plan, in  part, to discussions within CBP about the type and level of testing to be  conducted on the IFTs. Specifically, CBP officials stated that a June 2012  version of the draft Test and Evaluation Master Plan did not include  robust operational test and evaluation because of the IFT program’s  strategy to acquire a nondevelopmental system (sometimes referred to as  a commercial-off-the-shelf system). As a result, Border Patrol requested  that rigorous, disciplined testing be included in the Test and Evaluation  Master Plan to obtain familiarization with, and confidence in, the system  and establish baseline performance information.
According to DHS’s acquisition guidance, the Test and Evaluation Master  Plan is important because it describes the strategy for conducting  developmental and operational testing to evaluate a system’s technical  performance, including its operational effectiveness and suitability.  However, the IFT Test and Evaluation Master Plan approved by DHS in  November 2013 does not describe testing to evaluate the operational  effectiveness and suitability of the system. Rather, the Test and  Evaluation Master Plan describes CBP’s plans to conduct a limited user  test of the IFT. According to the Test and Evaluation Master Plan, the  limited user test will be designed to determine the IFT’s mission  contribution. According to OTIA and the Test and Evaluation Master Plan,  this testing is planned to occur during 30 days in environmental conditions  present at one site—the Nogales station. CBP plans to conduct limited  user testing for the IFT under the same process that is typically performed  in any operational test and evaluation, according to the Test and  Evaluation Master Plan.
The November 2013 IFT Test and Evaluation Master Plan notes that,  because the IFT acquisition strategy is to acquire nondevelopmental IFT  systems from the marketplace, a limited user test will provide Border  Patrol with the information it needs to determine the mission contributions  from the IFTs, and thus CBP does not plan to conduct more robust  testing. However, this approach is not consistent with DHS’s acquisition  guidance, which states that even for commercial-off-the-shelf systems,  operational test and evaluation should occur in the environmental  conditions in which a system will be used before a full production decision  for the system is made and the system is subsequently deployed. This  guidance also states that for commercial-off-the-shelf systems,  operational tests should be conducted to ensure that the systems satisfy  user-defined requirements. In addition, DHS guidance states that the  primary purpose of test and evaluation is to provide timely and accurate  information to managers, decision makers, and other stakeholders to  support research, development, and acquisition, in a manner that reduces  programmatic financial, schedule, and performance risk.
We recognize the need to balance the cost and time to conduct testing to  determine the IFT’s operational effectiveness and suitability with the  benefits to be gained from such testing. However, revising the Test and  Evaluation Master Plan to include more robust testing to determine  operational effectiveness and suitability that more fully accounts for the  various environmental conditions under which the IFTs will operate could  better position CBP to evaluate IFT capabilities before moving to full  production for the systems, help provide CBP with information on the  extent to which the towers satisfy the Border Patrol’s user requirements,  and help reduce potential program risks. In particular, although the limited  user test should help provide CBP with information on the IFTs’ mission  contribution and how Border Patrol can use the system in its operations,  the limited user test does not position CBP to obtain information on how  the IFTs may perform under the various environmental conditions the  system could face once deployed. For example, in November 2013, the  DHS Director of Test and Evaluation stated that testing the IFT at only  one location during a clear, warm day without much wind would not  produce representative results for days when it would be, for example,  rainy, windy, freezing, or snowy, or when there was lightning. Likewise,  he said testing in one location, such as Nogales, would not necessarily  produce the same results as testing in Tucson because of the different  terrains for the two locations. Conducting limited user testing in one area  in Arizona—the Nogales station—for 30 days could limit the information  available to CBP on how the IFT may perform in other conditions and  locations along the Arizona border with Mexico. As of November 2013,  CBP intends to deploy IFTs to 50 locations in southern Arizona, which  can include different terrain and differences in climate throughout the  year.
Although the IFT program is not the same as SBInet, according to the  Plan, the IFTs are to be deployed to locations with similar environmental  and terrain conditions as SBInet towers, and IFT and SBInet systems  may have similar types of technologies, such as cameras and radar. CBP  previously encountered testing issues with SBInet. For example, in a  January 2010 report, we found that while DHS’s approach to SBInet  testing appropriately consisted of a series of progressively expansive  developmental and operational events, the test plans and procedures for  some test events were not defined in accordance with guidance. In  January 2010, we concluded that effective testing was integral to  successfully acquiring and deploying a large-scale, complex system, like  SBInet. We further concluded that to do less unnecessarily increased the  risk of problems going undetected until late in the system’s life cycle, such  as when it was being accepted for use. In addition, in a November 2011  report, we found that the U.S. Army Test and Evaluation Command  (ATEC) operationally tested SBInet at Tucson and that testing revealed  challenges regarding the effectiveness and suitability of the technology  for border surveillance. Among other things, this testing found that the  rugged, restrictive terrain and weather conditions prevalent where SBInet  is deployed affected the performance of the system’s radar, which  affected success in detecting, identifying, and classifying items of interest.  Revising the Test and Evaluation Master Plan to more fully test the IFT in  the various environmental conditions in which it will be used to determine  operational effectiveness and suitability before IFTs move to full  production, in accordance with DHS acquisition guidance, could help  provide CBP with more complete information on how the IFTs will operate  under a variety of conditions before beginning full production. It could also  help better position CBP to understand how the IFTs will meet Border  Patrol’s operational requirements for the towers in contributing to Border  Patrol’s border security mission. Without conducting operational testing in  accordance with DHS guidance, the IFT program may be at risk of not  meeting Border Patrol operational needs.
RVSS program. The CBP Acquisition Decision Authority approved the  program at Acquisition Decision Event 2B; however, the official had not  approved any of the six required documents as required by DHS  acquisition guidance at the time of that event. According to OTIA officials,  the Acquisition Decision Authority approved the program for this  Acquisition Decision Event because all of the necessary programmatic  information was sufficiently developed and coordinated to support this  decision. However, the Acquisition Decision Authority did not approve five  of the documents until months after this event, and a sixth document, a  Life-cycle Cost Estimate, was in draft form in November 2013—2 years  after its required approval date. According to OTIA officials, the RVSS  Life-cycle Cost Estimate is expected to be completed and approved in the  second quarter of fiscal year 2014 and provided to DHS for review.
MSC program. The CBP Acquisition Decision Authority approved two of  the required six documents by Acquisition Decision Event 2B—the  Acquisition Plan and Operational Requirements Document. However, the  Integrated Logistics Support Plan was not approved until about 21  months after Acquisition Decision Event 2B. Also, the Acquisition  Program Baseline was not expected to be approved until the second  quarter of fiscal year 2014, more than 3 years after it was required to be  approved for Acquisition Decision Event 2B and at least 16 months after it  was required to be approved for Acquisition Decision Event 3.  Furthermore, a Life-cycle Cost Estimate for the MSC’s operations and  maintenance costs was expected to be completed in late 2013, more than  3 years after it was required to be approved for Acquisition Decision  Event 2B.
section_title: CBP Has Taken Some Steps to Assess Performance and Identify Mission Benefits, but Does Not Capture Complete Data on the Contributions of Its Surveillance Technologies
subsections: section_title: CBP Has Taken Steps to Assess the Performance of SBInet Technologies
paragraphs: Since we last reported on CBP’s efforts to assess the performance of its  SBInet surveillance systems in November 2011, CBP has taken steps to  assess the performance of these technologies. In November 2011, we  found that CBP had not conducted a post-implementation review and  developed a plan to address SBInet operational test outcomes.  Specifically, we found that CBP had not addressed the findings of ATEC’s  March 2011 operational test results for the SBInet system at Tucson,  which revealed challenges regarding the effectiveness and suitability of  the technology for border surveillance and made nine recommendations   At that time, CBP officials stated that  to address performance issues.the agency did not conduct a post-implementation review or develop a  plan to address the ATEC test results because the Secretary of  Homeland Security canceled SBInet in January 2011. In November 2011,  we recommended that CBP, in accordance with DHS guidance, conduct a  post-implementation review and operational assessment of its SBInet  system, and assess costs and benefits of taking action on the results of  ATEC’s operational test.that conducting such a review, and weighing the costs and benefits of  taking action on recommendations resulting from ATEC’s test of the  SBInet system, could inform CBP’s decisions about future deployments of  similar technologies, such as the IFTs.
In making this recommendation, we concluded  In response to our November 2011 recommendation, OTIA tasked the  Johns Hopkins University Applied Physics Laboratory with conducting an  independent post-implementation review of its SBInet Block 1 system. In  January 2013, CBP released the results of the SBInet Block 1 Post  Implementation Review (PIR), an assessment of the performance of its   The PIR  two SBInet surveillance system locations at Tucson and Ajo. concluded that CBP’s SBInet surveillance system has enhanced overall  situational awareness within system viewsheds, improved agent safety,  and been operationally available and effective with costs consistent with  those anticipated for the system. For instance, the PIR concluded that  the system broadened the agents’ situational awareness beyond the  tactical, agent-on-the-ground sphere of awareness, and increased their  ability to monitor incursions.
The PIR also made five recommendations for CBP to improve future  operational assessments of its SBInet surveillance system and to plan for  new acquisition sensor deployments, such as for CBP to conduct a more  detailed assessment of the impacts of Block 1 systems and develop more  on-the-job agent training.as of May 2013, CBP is in the process of documenting and reviewing  each recommendation outlined in the PIR, and intends to document its  plans to address those recommendations that OTIA and the Office of  Border Patrol determine need corrective action. However, these officials  stated that some of the findings and recommendations outlined in the PIR  will not be explicitly addressed or applied to future deployment efforts. For  instance, according to officials, because the technologies planned for  deployment under the Plan are commercial-off-the-shelf products, the PIR  finding about recording the documentation of environmental factors, such  as weather and terrain, that impede the system performance will not  apply to the technologies to be deployed under the Plan, as those  technologies include requirements on documentation of environmental  factors. Border Patrol officials further stated that the contractor and  Border Patrol will have a process to enable them to determine where the  best deployment locations, given the variable terrain, will be for the  technologies to be deployed under the Plan. Moreover, Border Patrol  officials stated that Tucson sector officials have been assigned  responsibility to determine the extent to which corrective actions are  needed to address each recommendation outlined in the PIR because  these sector officials have a better understanding of the environment in  which the SBInet system is operating. According to OTIA officials, the  agency plans to conduct annual operational assessments of its SBInet  system. As additional surveillance technologies are deployed, we will  continue to monitor Border Patrol’s efforts to address issues identified by  the PIR as part of our recommendation follow-up process.
According to OTIA and Border Patrol officials,  In addition, the PIR concluded that as of January 2013, six of the nine  recommendations outlined in ATEC’s operational test have either been  addressed or are in the process of being addressed. The ATEC  recommendations that remain to be addressed include, for example,  addressing software reliability, improving sustainability cost, and reducing  maintenance issues. OTIA officials stated that the agency plans to take  actions to address the remaining three recommendations by, for example,  pursuing alternative technical solutions to extend the life-cycle of the  SBInet system and improving sustainability costs by reducing the  contractor’s responsibility for field maintenance and other functions by  transitioning to government support in 2014.
section_title: CBP Is Not Capturing Complete Data on the Contributions of Its Surveillance Technologies
paragraphs: CBP is not capturing complete asset assist data on the contributions of its  surveillance technologies to apprehensions and seizures, and these data  are not being consistently recorded by Border Patrol agents and across  locations. Although CBP has a field within the EID for maintaining data on  whether technological assets, such as SBInet surveillance towers, and  nontechnological assets, such as canine teams, assisted or contributed to  the apprehension of illegal entrants, and seizure of drugs and other  contraband, according to CBP officials, Border Patrol agents are not  required to record these data. This limits CBP’s ability to collect, track,  and analyze available data on asset assists to help monitor the  contribution of surveillance technologies, including its SBInet system, to  Border Patrol apprehensions and seizures and inform resource allocation  decisions.
Our analysis of EID asset assist data for apprehensions and seizures in  the Tucson and Yuma sectors from fiscal year 2010 through June 2013  shows that information on asset assists was generally not recorded for all  apprehension and seizure events. For instance, for the 166,976  apprehension events reported by the Border Patrol across the Tucson  sector during fiscal year 2010 through June 2013, an asset assist was not  recorded for 115,517 (or about 69 percent) of these apprehension events.  In the Yuma sector, of the 8,237 apprehension events reported by Border  Patrol agents during the specified time period, an asset assist was not  recorded for 7,150 (or about 87 percent) of these apprehension events.  Similarly, data on seizure events reported across the Tucson and Yuma  sectors show that for some seizure events, asset assists were not  reported from fiscal year 2010 through June 2013 (about 32 percent and  about 67 percent, respectively). According to Border Patrol officials, in the  absence of requirements for Border Patrol agents to record data on asset  assists, differences in the reporting of these data at the station level are  likely attributable to the emphasis placed on the recording of these data  by supervisory agents. Appendix IV contains summary statistics on the  extent to which data on asset assists are recorded for apprehensions and  seizures across the Tucson and Yuma sectors from fiscal year 2010  through June 2013.
Since data on asset assists are not required to be reported, it is unclear  whether the data were not reported because an asset was not a  contributing factor in the apprehension or seizure or whether an asset  was a contributing factor but was not recorded by agents. As a result,  CBP is not positioned to determine the contribution of surveillance  technologies in the apprehension of illegal entrants and seizure of drugs  and other contraband during the specified time frame. As shown in  figures 3 and 4, while the recording of asset assists increased from fiscal  year 2010 through June 2013 from about 18 percent to about 47 percent  in the Tucson sector and from about 8 percent to about 21 percent in the  Yuma sector, for more than one-half of the apprehension event records  for the Tucson sector and four-fifths for the Yuma sector, asset assists  were not reported for the first three quarters of fiscal year 2013.
Border Patrol officials did not specify why the agency does not require the  recording and tracking of data on asset assists. However, Border Patrol  officials stated that agents are encouraged to select the appropriate asset  assist code when assets contributed to an apprehension or seizure.  Border Patrol officials also stated that although they do not regularly track  and analyze data on asset assists, including those from surveillance  technologies, these data are tracked and analyzed on an ad hoc basis to  help determine Border Patrol’s resource allocation and operational needs,  and more specifically, what resources are available at the strategic level  to help mitigate the threat of illegal entrants, drugs, and other contraband.  Moreover, an Associate Chief at Border Patrol told us that while data on  asset assists are not systematically recorded and tracked, Border Patrol  recognizes the benefits of assessments of asset assists data, including  those from surveillance technologies, such as the SBInet system, as  these data in combination with other data, such as numbers of  apprehensions and seizures, are used on a limited basis to help the  agency make adjustments to its acquisition plans prior to deploying  resources, thereby enabling the agency to make more informed  deployment decisions. Border Patrol also uses these other data, such as  numbers of apprehensions and seizures, to help inform assessment of its  efforts to secure the border. Border Patrol officials cautioned that while  asset assists data are the only available data directly linking  apprehensions and seizures to the agency’s surveillance technologies,  these data do not enable direct attributions of the SBInet system’s  contribution to border security strategic goals because of several factors,  such as changes in the flows of illegal entrants across sectors or in  economic conditions in the United States and Mexico. Moreover, the  officials said that surveillance technologies such as SBInet and RVSS  towers enable the detection of apprehensions and seizures and  accordingly, it is the agents who identify and track the illegal activity and  ultimately apprehend illegal entrants and seize contraband.
Despite the absence of complete data on the contribution of CBP’s  surveillance technologies to apprehensions and seizures, our analysis of  Border Patrol’s data on the location of apprehensions and seizures  provides some insights into where Border Patrol apprehensions and  seizures occurred in relation to the locations of its two highest-cost  surveillance technologies—SBInet towers and RVSS. For example, our  analysis of apprehensions events data, as determined by Geographic  Information System data entered by Border Patrol agents when recording  apprehensions and seizures, shows that across the Tucson sector from  fiscal year 2010 through June 2013, of the 166,976 apprehension events,  71,397 (or about 43 percent) occurred within the camera and radar range  of SBInet and RVSS towers. As shown in figure 5, the percentage of  apprehension events occurring within the range of both SBInet and RVSS  surveillance technologies has changed little, if at all, over time.  Apprehension events occurring within the radar and camera range of  SBInet towers have remained relatively unchanged, while apprehension  events occurring within the range of RVSS towers increased by about 1  percent during our specified time frame.
Moreover, of those 115,517 apprehension events in the Tucson sector  that do not have data on asset assists, 8,751 (or about 8 percent)  occurred within the camera range, and 9,818 (or about 9 percent)  occurred within the radar range of SBInet towers. Moreover, data on  asset assists were not recorded in 35,147 (or about 30 percent) of  apprehension events within the range of RVSS towers. Table 6 shows the  reporting of asset assists for apprehension and seizure events occurring  across the Tucson sector within the range of SBInet and RVSS towers  during our specified time period.
Border Patrol officials stated that while analyzing data on the  contributions of Border Patrol’s surveillance technologies is a relevant  measure of the agency’s ability to meet its border security goals,  conclusions regarding the contributions and impacts of its surveillance  technologies on Border Patrol’s enforcement efforts cannot be formed  solely on the basis of the proximity of apprehension or seizure events to  the locations of its surveillance technologies. These officials stated that  there are instances in which illegal entrants were detected by some  combination of cameras or radar closer to the border; however, to gain a  better tactical advantage, Border Patrol agents made the apprehensions  farther from the border. As we reported in December 2012, Border Patrol  officials stated that apprehensions occur in areas farther from the border  because several factors preclude greater border presence, including  terrain that is inaccessible or creates a tactical disadvantage, the distance  from Border Patrol stations to the border, and access to ranches and  lands that are federally protected and environmentally sensitive.
Standards for Internal Control in the Federal Government calls for  agencies to ensure that ongoing monitoring occurs during the course of  normal operations to help evaluate program effectiveness. standards also state that agencies should promptly and accurately record  transactions to maintain their relevance and value for management  decision making and that this information should be readily available for  use by agency management and others so that they can carry out their  duties with the goal of achieving all of their objectives, including making  operating decisions and allocating resources. These standards further  state that to be effective, agencies need to clearly document all  transactions in a timely manner to ensure that they are making  appropriately informed decisions. Moreover, the standards call for clear  documentation of and procedures that are readily available for  examination. In addition, these standards call for comparisons and  assessments relating different sets of data to one another so that  analyses of the relationships can be made and appropriate actions taken.  Because DHS’s EID database already includes the asset assists data  field and these data are used by Border Patrol on a limited basis to make  decisions about resources, requiring agents to record and track asset  assists data could help ensure that these data are complete and, if  analyzed, could help better inform CBP’s resource allocation decisions.  Moreover, we acknowledge that conclusions regarding the contributions  of surveillance technologies based on location and proximity data alone  may not be sufficient to examine the contribution of CBP’s surveillance  technologies in achieving their strategic goals. However, analyzing data  on apprehensions, seizures, and asset assists in combination with other  relevant performance metrics or indicators as appropriate could provide  more robust analysis of the contributions of surveillance technologies,  and accordingly could better position CBP to be able determine the extent  to which its technology investments have contributed to the agency’s  border security efforts.
GAO/AIMD-00-21.3.1.
section_title: CBP Has Identified Mission Benefits for Surveillance Technologies under the Plan but Has Not Yet Developed Performance Metrics
paragraphs: In response to our November 2011 recommendation regarding the  identification of mission benefits and development of key attributes for  performance metrics for the surveillance technologies to be deployed as  part of the Plan, CBP has identified mission benefits expected from the  implementation of the surveillance technologies to be acquired or  deployed as part of the Plan, but has not fully developed key attributes for   In November 2011, we  performance metrics for these technologies.reported that agency officials had not yet defined the mission benefits  expected or quantified metrics to assess the contribution of the selected  approaches in achieving their goal of situational awareness and detection  of border activity using surveillance technology. We recommended that  CBP determine the mission benefits to be derived from implementation of  the Plan and develop and apply key attributes for metrics to assess  program implementation. CBP concurred with our recommendation.
In April 2013, CBP issued its Multi-Year Investment and Management  Plan for Border Security Fencing, Infrastructure, and Technology for  Fiscal Years 2014-2017, which identifies specific mission benefits to be  achieved by the deployment of each of the seven technologies under the  Plan.technologies will provide the mission benefits of improved situational  awareness and agent safety. Furthermore, CBP officials stated that each  of the seven technologies deployed or planned for deployment as part of  the Plan will help enhance the ability of Border Patrol agents to detect,  identify, deter, and respond to threats along the border. A summary of the  mission benefits of each surveillance technology deployed or planned for  deployment under the Plan is presented in appendix V.
According to CBP officials, the majority of these surveillance  While CBP has defined mission benefits for the technology programs  under the Plan, the agency has not yet developed key attributes for  performance metrics for all surveillance technologies to be deployed as  part of the Plan. The Clinger-Cohen Act of 1996 and Office of  Management and Budget (OMB) guidance emphasize the need to ensure  that information technology investments, such as IFT systems, produce  tangible, observable improvements in mission performance. In our April  2013 update on the progress made by the agencies to address our  findings on duplication and cost savings across the federal government,  CBP officials stated that operations of its two SBInet surveillance systems  identified examples of key attributes for metrics that can be useful in  assessing the Plan’s implementation for technologies. For example,  according to CBP officials, to help measure whether illegal activity has  decreased, examples of key attributes include decreases in the amount of  arrests, complaints by ranchers and other citizens, and destruction of  public and private lands and property. While the development of key  attributes for metrics for the two SBInet surveillance systems is a positive  step, as of April 2013, CBP has not yet identified attributes for metrics for  all technologies to be acquired and deployed as part of the Plan.
In addition to these efforts, CBP officials stated that in response to our  prior recommendations regarding the establishment of a performance  goal or goals and associated performance metrics that define how border  security is to be measured, Border Patrol, as of December 2013, was in  the process of developing and implementing performance goals and  measures to assess Border Patrol’s efforts to secure the border.  However, CBP officials stated that none of the current measures directly  address the operational impact of technology. The officials further stated  that the Tucson sector has submitted an issue paper that identifies  potential data that can attribute a certain level of effectiveness to its  SBInet system, but it is still under review by CBP. While these are  positive steps, to fully address the intent of our recommendation, CBP  would need to develop and apply key attributes for performance metrics  for each of the technologies to be deployed under the Plan to assess its  progress in implementing the Plan and determine when mission benefits  have been fully realized.
section_title: Conclusions
paragraphs: CBP has established schedules for the Plan and the IFT, RVSS, and  MSC programs that meet some but not all best practices for scheduling,  hindering CBP’s ability to reliably commit to when it will deliver all of the  Plan’s technologies to Arizona. Ensuring that all schedule best practices  are applied to the IFT, RVSS, and MSC schedules when updating them  could help OTIA better ensure the schedules’ reliability and could help  better position OTIA to identify and address any potential further delays in  the program’s milestone commitment dates. Further, developing and  maintaining an Integrated Master Schedule for the Plan, in accordance  with best practices, could allow insight into current or programmed  allocation of resources for the Plan and help CBP to reliably commit to  when the Plan will be fully implemented. Also, CBP has developed Life- cycle Cost Estimates for the IFT and RVSS programs. Although OTIA  officials stated that DHS’s Office of Program Accountability and Risk  Management conducted an assessment of the IFT Life-cycle Cost  Estimate, an assessment is not equivalent to verifying the estimate with  an independent cost estimate. When updating the Life-cycle Cost  Estimates for the IFT and RVSS programs, verifying the estimates with  independent cost estimates and reconciling any differences, consistent  with best practices, could help to better ensure the credibility of CBP’s  cost estimates for these programs.
DHS and CBP have approved some key acquisition documents as  directed by DHS and CBP Acquisition Review Boards, but work remains  to approve all key acquisition documents in accordance with DHS  acquisition guidance. Specifically, revising the Test and Evaluation  Master Plan to more fully test the IFTs in the various environmental  conditions in which they will be used to determine operational  effectiveness and suitability, in accordance with DHS acquisition  guidance, could help provide CBP with more complete information on  how the IFTs will operate under a variety of conditions before beginning  full production.
Requiring the collection of data on the extent to which technology assets  assisted in apprehensions and seizures could better position Border  Patrol to assess the contribution of surveillance technologies to its  enforcement efforts and its goals of achieving and maintaining operational  control and situational awareness along the southwest border.  Conducting analysis of such data, once collected, in combination with  other relevant performance metrics or indicators as appropriate, could  help better position CBP to be able to determine the extent to which its  technology investments have contributed to border security efforts.
section_title: Recommendations for Executive Action
paragraphs: To improve the acquisition management of the Plan and the reliability of  its cost estimates and schedules, assess the effectiveness of deployed  technologies, and better inform CBP’s deployment decisions, we  recommend that the Commissioner of CBP take the following six actions:
When updating the schedules for the IFT, RVSS, and MSC programs,  ensure that scheduling best practices, as outlined in our schedule  assessment guide, are applied to the three programs’ schedules.
Develop and maintain an Integrated Master Schedule for the Plan that  is consistent with scheduling best practices.
When updating Life-cycle Cost Estimates for the IFT and RVSS  programs, verify the Life-cycle Cost Estimates with independent cost  estimates and reconcile any differences.
Revise the IFT Test and Evaluation Master Plan to more fully test the  IFT program, before beginning full production, in the various  environmental conditions in which IFTs will be used to determine  operational effectiveness and suitability, in accordance with DHS  acquisition guidance.
Require data on asset assists to be recorded and tracked within the  Enforcement Integrated Database, which contains data on  apprehensions and seizures.
Once data on asset assists are required to be recorded and tracked,  analyze available data on apprehensions and seizures and technological  assists, in combination with other relevant performance metrics or  indicators, as appropriate, to determine the contribution of surveillance  technologies to CBP’s border security efforts.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for review and comment. DHS  provided written comments, which are summarized below and reproduced  in full in appendix VI, and technical comments, which we incorporated as  appropriate. DHS concurred with four of the recommendations in the  report. DHS did not concur with the other two recommendations in the  report.
With regard to the first recommendation, that CBP ensure that scheduling  best practices are applied to the IFT, RVSS, and MSC programs’  schedules when they are updated, DHS concurred and stated that OTIA  plans to ensure that scheduling best practices are applied as far as  practical when updating the three programs’ schedules. DHS plans to  update the programs’ schedules by July 2015.
With regard to the second recommendation, that CBP develop and  maintain an Integrated Master Schedule for the Plan, DHS did not concur  with this recommendation. DHS stated that maintaining an Integrated  Master Schedule for the Plan undermines the DHS-approved  implementation strategy for the individual programs making up the Plan  and that a key element of the Plan has been the disaggregation of  technology procurements. According to DHS, the implementation of this  recommendation would essentially create a large, aggregated program,  similar to SBInet, and effectively create an aggregate “system of  systems.” DHS stated that CBP believes its strategy of disaggregation  has been effective and has reduced overall risk and cost. DHS also  stated that each program within the Plan has its own schedule and that  forcing linkages among the Plan’s programs into a single Integrated  Master Schedule contradicts lessons learned and the approved  implementation strategy for the Plan.
We continue to believe that developing and maintaining an Integrated  Master Schedule for the Plan, consistent with best practices for  scheduling, is needed. As noted in the report, the use of an Integrated  Master Schedule is a well-established practice in program and project  management and is a necessary tool to coordinate independently  managed projects that have dependencies—including resource  dependencies—on one another. The programs under the Plan are  intended to provide Border Patrol with a combination of surveillance  capabilities to assist in achieving situational awareness along the Arizona  border with Mexico; and while the programs themselves may be  independent of one another, the Plan’s resources are being shared  among the programs. Furthermore, this recommendation is not intended  to imply that DHS needs to re-aggregate the Plan’s seven programs into  a "system of systems" or change its procurement strategy in any form.  Rather, the intent of our recommendation is for DHS to insert the  individual schedules for each of the Plan’s programs into a single  electronic Integrated Master Schedule file in order to identify any resource  allocation issues among the programs’ schedules. Developing and  maintaining an Integrated Master Schedule for the Plan could allow OTIA  insight into current or programmed allocation of resources for all  programs as opposed to attempting to resolve any resource constraints  for each program individually.
In addition to helping identify resource constraints, an Integrated Master  Schedule can be a useful tool for consolidating multiple projects or  program files into a single master file, even if those projects or programs  have no direct links among activities. For example, aggregating individual  files into a master schedule is useful for reporting purposes, particularly if  the projects or programs are under the purview of a single management  organization or a single customer. In this case, the master schedule  would allow for a concise view of all projects or programs for which the  stakeholder is responsible or has an interest. A master schedule of this  nature is often referred to as a consolidated schedule, although the terms  “consolidated schedule” and “Integrated Master Schedule” are often  synonymous. We continue to believe that developing and maintaining an  Integrated Master Schedule for the Plan could help provide CBP a  comprehensive view of the Plan and help CBP to reliably commit to when  the Plan will be fully implemented and better predict whether estimated  completion dates are realistic to manage programs’ performance, as  noted in the report.
With regard to the third recommendation, that CBP verify the Life-cycle  Cost Estimates for the IFT and RVSS programs with independent cost  estimates and reconcile any differences, DHS concurred, although its  planned actions will not fully address the intent of the recommendation  unless assumptions underlying the cost estimates change.  DHS stated  that while OTIA did not obtain a traditional independent cost estimate for  the programs, the Life-cycle Cost Estimates were meant to be  conservative in managing program risk and that the estimated life-cycle  costs to date are less than originally projected. DHS further stated that at  this point it does not believe that there is a benefit in expending funds to  obtain independent cost estimates and that if the costs realized to date  continue to hold, there may be no requirement or value added in  conducting full-blown updates with independent cost estimates. DHS  noted, though, that if this assumption changes, OTIA will complete  updates and consider preparing independent cost estimates, as  appropriate. We recognize the need to balance the cost and time to verify  the Life-cycle Cost Estimates with the benefits to be gained from  verification with independent cost estimates. However, as noted in this  report, independently verifying the cost estimates is consistent with best  practices and could help provide CBP with more insights into program  costs. An independent cost estimate provides an independent view of  expected program costs that tests the program office’s estimate for  reasonableness. Independent cost estimates frequently use different  methods and are less burdened with organizational bias than a program  office’s estimate, helping to provide decision makers with insight into a  program’s potential costs. Thus, we continue to believe that  independently verifying the Life-cycle Cost Estimates for the IFT and  RVSS programs and reconciling any differences, consistent with best  practices, could help CBP better ensure the reliability of the estimates.
With regard to the fourth recommendation, that CBP revise the IFT Test  and Evaluation Master Plan to more fully test the IFT program in the  various environmental conditions in which IFTs will be used to determine  operational effectiveness and suitability, DHS did not concur with the  recommendation. Specifically, DHS stated that the Test and Evaluation  Master Plan includes tailored testing and user assessments that will  provide much, if not all, of the insight contemplated by the intent of the  recommendation. According to DHS, the approved non-developmental  item acquisition strategy for the IFT program was based on market  surveys and observations during field use by other customers and the  incorporation of system demonstrations conducted during source  selection. DHS also stated that there is no requirement for expansive,  formal operational test and evaluation and to re-write the Test and  Evaluation Master Plan to incorporate operational testing undermines and  removes the benefits of the non-developmental item strategy. Moreover,  DHS stated that the user test currently outlined in the Test and Evaluation  Master Plan will provide the operational user the information needed to  validate system requirements and operational characteristics. DHS also  noted that Acquisition Decision Event 3 has been approved for IFT  production, and after the initial IFT system undergoes testing in  accordance with the Test and Evaluation Master Plan, the Office of  Border Patrol will make the determination regarding operational readiness  prior to deploying additional systems.
We continue to believe that DHS should revise the Test and Evaluation  Master Plan to more fully test the IFT program, before beginning full  production, in the various environmental conditions in which the IFT will  be used to determine operational effectiveness and suitability. DHS’s  acquisition guidance states that the Test and Evaluation Master Plan is  important because it describes the strategy for conducting developmental  and operational testing to evaluate a system’s technical performance,  including its operational effectiveness and suitability. The guidance states  that, even for commercial-off-the-shelf systems, such as the IFT program,  operational test and evaluation should occur in the environmental  conditions in which a system will be used before a full production decision  for the system is made and the system is subsequently deployed. In  addition, DHS guidance states that the primary purpose of test and  evaluation is to provide timely and accurate information to managers,  decision makers, and other stakeholders to support research,  development, and acquisition in a manner that reduces programmatic  financial, schedule, and performance risks.
The current Test and Evaluation Master Plan describes CBP’s plans to  conduct a limited user test of the IFT, which will be designed to determine  the IFT’s mission contribution. However, determining mission contribution  is not equivalent to determining operational effectiveness and suitability,  which specifically identifies how effective and reliable a system is in  meeting its operational requirements in its intended environment. DHS  plans to conduct limited user testing during a 30-day period in  environmental conditions present at one site—the Nogales station.  However, as of November 2013, CBP intended to deploy IFTs to 50  locations in southern Arizona, which can include different terrain and  differences in climate throughout the year. As we noted in the report,  conducting limited user testing in one area in Arizona for 30 days could  limit the information available to CBP on how the IFTs may perform in  other conditions and locations along the Arizona border. Therefore, CBPs  approach to use limited user testing will not specifically identify how  effective and reliable a system is in meeting its operational requirements  in its intended environment. Moreover, while DHS has approved the IFT  program for production at Acquisition Decision Event 3, testing for the  IFTs has not yet begun. As noted in the report, revising the Test and  Evaluation Master Plan to include more robust testing to determine  operational effectiveness and suitability could better position CBP to  evaluate IFT capabilities before moving to full production for the system,  help provide CBP with information on the extent to which the towers  satisfy the Border Patrol’s user requirements, and help reduce potential  program risks.
Furthermore, although the IFT program is not the same as SBInet,  according to the Plan, the IFTs are to be deployed to locations with  similar environmental and terrain conditions as SBInet towers, and IFT  and SBInet systems may have similar types of technologies, such as  cameras and radar. As noted in the report, we previously identified testing  issues CBP encountered with SBInet, such as DHS’s test plans and  procedures for some SBInet test events not being defined in accordance  with guidance and that operational tests of SBInet at Tucson revealed  challenges regarding the effectiveness and suitability of the technology  for border surveillance. Thus, we continue to believe that revising the Test  and Evaluation Master Plan to more fully test the IFT in the various  environmental conditions in which it will be used to determine operational  effectiveness and suitability, before beginning full production, could help  provide CBP with more complete information on how the IFTs will operate  under a variety of conditions. Without conducting operational testing in  accordance with DHS guidance, the IFT program may be at increased  risk of not meeting Border Patrol operational needs.
With regard to the fifth recommendation, that CBP require data on asset  assists to be recorded and tracked within the Enforcement Integrated  Database, DHS concurred and stated that Border Patrol is changing its  data collection process to allow for improved reporting on asset assists  for apprehensions and seizures and intends to make it mandatory to  record whether an asset assisted in an apprehension or seizure. DHS  plans to change its process by December 31, 2014.
With regard to the sixth recommendation, that CBP analyze available  data on apprehensions and seizures and technology assists to determine  the contribution of surveillance technologies to its border security efforts,  DHS concurred and stated that Border Patrol intends to create a plan of  action with milestones to explore and develop a process to answer how  different classes of technology, within a certain environment, contribute to  Border Patrol’s mission. DHS stated that Border Patrol plans to develop  an initial set of quantitative and qualitative technology-related measures  by September 30, 2014, as an interim milestone; gather baseline data for  the measures in fiscal year 2015 and begin to use these data to evaluate  the contributions of specific technology assets by the end of that fiscal  year; and by the end of fiscal year 2016, use measures associated with  technology to assist in determining levels of situational awareness in  different areas of the border. These planned actions, if implemented  effectively, should address the intent of the recommendations.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of  Homeland Security, appropriate congressional committees, and other  interested parties. In addition, the report will be available at no charge on  the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-8777 or gamblerr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix VII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine the extent to which U.S. Customs and  Border Protection (CBP) has (1) developed schedules and Life-cycle Cost  Estimates for the Arizona Border Surveillance Technologies Plan (the  Plan) in accordance with best practices; (2) followed key aspects of the  Department of Homeland Security’s (DHS) acquisition management  framework in managing the Plan’s three highest cost programs; and (3)  assessed the performance of technologies deployed under the Secure  Border Initiative Network (SBInet), identified mission benefits, and  developed performance metrics for surveillance technologies to be  deployed under the Plan.
To determine the extent to which CBP has followed best practices in  developing schedules and Life-cycle Cost Estimates for the Plan’s three  highest-cost programs—the Integrated Fixed Tower (IFT), Remote Video  Surveillance System (RVSS), and Mobile Surveillance Capability  (MSC)—we obtained CBP’s Office of Technology Innovation and  Acquisition’s (OTIA) program schedules as of March 2013, which were  current at the time of our review, for the these programs and compared  them against best practices for developing schedules. Specifically, we  assessed the extent to which the schedules for these three programs met  each of the 10 best practices identified in the schedule assessment  guide. We characterized whether the schedules met each of the 10 best  practices based on the following scale:
Not met—the program provided no evidence that satisfies any of the  criterion.
Minimally met—the program provided evidence that satisfies a small  portion of the criterion.
Partially met—the program provided evidence that satisfies about half  of the criterion.
Substantially met—the program provided evidence that satisfies a  large portion of the criterion.
Met—the program provided complete evidence that satisfies the entire  criterion.
In conducting our analysis, we focused, for example, on whether the  schedules reflect best practices for a reliable schedule, such as whether  the schedules define the work necessary to accomplish a program’s  objectives. More details on our assessment and methodology are  presented in appendix III. By assessing the schedules against best  practices, we also identified schedule challenges that CBP was  experiencing in testing, procuring, deploying, and operating technologies  in the Plan and interviewed CBP officials to determine the reasons for the  schedule challenges and steps that CBP had taken or was taking to  address them. In addition, we obtained and analyzed the August 2010  and June 2013 Life-cycle Cost Estimates for the Plan. We also analyzed  the IFT and RVSS January 2012 and March 2012 Life-cycle Cost  Estimates, respectively, which were current at the time of our review, and  compared them against best practices for cost estimating. We analyzed  DHS and CBP documents and interviewed officials regarding their efforts  to implement our November 2011 recommendations to update the Life-  To  cycle Cost Estimate for the Plan in accordance with best practices.assess the reliability of cost estimate data that we used, we reviewed  relevant program documentation, such as cost estimation spreadsheets,  as available, to substantiate evidence obtained from interviews with  knowledgeable agency officials. We found the data to be sufficiently  reliable for the purposes of our report.
To determine the extent to which CBP followed key aspects of DHS’s  acquisition management framework in managing the Plan’s three highest-  cost programs, we analyzed DHS and CBP documents, including DHS  Acquisition Management Directive 102-01 and its associated DHS  Instruction Manual 102-01-001, program briefing slides, budget  documents, Acquisition Decision Memorandums, schedules, and program   We focused on the IFT, RVSS, and MSC programs for more  risk sheets.in-depth analyses because they are the Plan’s three highest-cost  programs and represent 97 percent of the estimated cost of the Plan.  Specifically, to assess the acquisition strategy for the Plan, we focused on  the IFT, RVSS, and MSC programs and analyzed their acquisition plans  and discussed the approaches with CBP officials. To assess system  requirements and capabilities for the IFT, RVSS, and MSC programs, we  obtained and analyzed requirements and capabilities documents and  worked with CBP officials to identify any changes to requirements and  capabilities since they were initially approved and whether any  requirements or capabilities had been traded off for cost, schedule, or  other purposes. To assess the extent to which CBP followed DHS  acquisition guidance, we selected aspects of Acquisition Management  Directive 102-01 that were relevant to where these programs were in the  acquisition process during fiscal year 2013. Specifically, we determined  whether acquisition documents had been approved by the time of the  applicable Acquisition Decision Events as required by DHS acquisition  guidance.
To determine the extent to which CBP has assessed the performance of  technologies deployed under SBInet and developed performance metrics  to assess the performance of surveillance technologies planned for  deployment under the Plan, we analyzed performance assessment  documentation and interviewed CBP officials responsible for performance  measurement activities regarding the establishment of performance  metrics used by CBP to determine the effectiveness and the contributions  of its surveillance technologies toward the agency’s stated border security  goals. With respect to CBP’s assessment of the performance of  technologies deployed under SBInet, we analyzed the results of the  January 2013 post implementation review, which was conducted by the  Johns Hopkins University Applied Physics Laboratory to determine the  effectiveness of SBInet technologies in achieving their intended results.  We reviewed our November 2011 report to determine the extent to which  CBP’s post implementation review aligned with DHS guidance and the  Office of Management and Budget’s (OMB) Capital Programming Guide,  a supplement to OMB Circular A-11, which identifies a post  implementation review as a tool to evaluate an investment’s efficiency  and effectiveness. We also analyzed CBP and DHS documents, such as  CBP’s July 2013 SBInet Block 1 After Action Report, and interviewed  officials to assess corrective actions taken to improve SBInet  performance issues. Specifically, we analyzed CBP documentation and  interviewed agency officials within OTIA and the Office of Border Patrol to  determine the progress the agency has made in addressing findings and  recommendations outlined in CBP’s post implementation review and prior  performance assessments, including the Army Test and Evaluation  Command’s assessment of its SBInet technologies. On the basis of  interviews with agency officials regarding the methodology and  implementation of the review, we found the review to be sufficiently  reliable for our report.
In addition, we analyzed CBP data on apprehensions of illegal entrants  and seizures of drugs and other contraband for the Tucson and Yuma  sectors maintained in the Enforcement Integrated Database (EID), a  DHS-shared common database repository for several DHS law  enforcement and homeland security applications, as well as policy,  planning, and budget documents provided by Border Patrol to determine  whether such data could be used to determine the contributions of the  SBInet technologies to apprehensions and seizures. We analyzed  apprehension and seizure data for the Tucson and Yuma sectors within  Arizona, because these are the Border Patrol sectors contained within  Arizona and covered by the Plan. For the purposes of this report, we  analyzed apprehension and seizure events recorded in the EID for fiscal  years 2010 through June of fiscal year 2013. An apprehension or  seizure event is defined as an occasion on which Border Patrol agents  apprehend an illegal entrant or seize drugs and other contraband. Each  reported apprehension or seizure event is assigned a unique identifier in  the EID, and Border Patrol agents assign an additional identifier to each  individual illegal entrant or type of seized item associated with the event.  As a result, a single apprehension event may involve the apprehension of  multiple illegal entrants, and a single seizure event may result in the  seizure of multiple items. Appendix IV contains the results of our analysis  of all recorded apprehensions and seizures occurring across the Tucson  and Yuma sectors during the specified time frame. For our analysis, we  also obtained data on asset assists recorded in the EID for  apprehensions and seizures. According to Border Patrol officials, the  asset assist data field was added to the EID in May of 2009. Agents may  select from a drop-down menu to identify whether a technological or  nontechnological asset assisted in the apprehension or seizure. Multiple  assets can be selected for a single event, if relevant. For the purposes of  this report, technological assets identified within the Border Patrol’s asset  assists data field drop-down menu are those assets for which Border  Patrol continues to make significant funding investments and are included  as part of the Plan, and include Cameras, Mobile Surveillance Systems,  Scope Trucks, and Unattended Ground Sensors. According to Border  Patrol headquarters officials, agents identifying “Cameras” are most likely  attributing the asset assist to either SBInet towers or Remote Video  Surveillance Systems.
In addition, for our analysis, we obtained Geographic Information  Systems data for apprehensions, seizures, and Border Patrol’s two  highest-cost surveillance systems—SBInet and RVSS towers—to show  the latitude and longitude coordinates of apprehensions and seizures in  relation to the location of SBInet towers and RVSS towers. We used  Geographic Information Systems data to determine the percentage of  apprehensions and seizures that occurred within the proximity of the  radar and camera range of SBInet and camera range of RVSS towers,  and the extent to which asset assists were reported for apprehensions  and seizures occurring within the proximity of the surveillance systems.  For the purposes of this report, the ranges of the SBInet and RVSS  towers are “buffer ranges” that, according to Border Patrol headquarters  officials, do not account for obstructions due to terrain, land features, and  vegetation. To perform these analyses, we compared Border Patrol data  on the longitude and latitude of apprehensions, seizures, SBInet towers,  and RVSS towers with agency mapping data, which allowed us to  determine the extent to which apprehensions and seizures occurred  within the proximity of SBInet and RVSS towers. We interviewed Border  Patrol headquarters officials regarding data collection and analysis  procedures, and performance assessment activities.
We analyzed apprehensions and seizures data from fiscal year 2010  through June 2013 because fiscal year 2010 was the first fiscal year for  which data on asset assists were available following Border Patrol’s  deployment of its SBInet technologies, and the collection of data on  Geographic Information Systems coordinates for apprehensions and  seizures was required. To assess the reliability of apprehensions and  seizures data, including the asset assist and Geographic Information  Systems data, we interviewed Border Patrol headquarters officials who  oversee the maintenance and analyses of the data about agency  guidance and processes for collecting and reporting the data. We  determined that the apprehensions, seizures, asset assists, and  Geographic Information Systems data were sufficiently reliable for the  purposes of this report. However, as we reported in December 2012,  because of potential inconsistencies in how the data are collected, these  data cannot be compared across sectors but can be compared within a  sector over time.were sufficiently reliable for the purposes of this report, but found  limitations with the consistency in which these data are recorded for all  apprehensions and seizures, a fact that we discuss in the report.  Although we determined that the latitude and longitude coordinates for  some apprehensions and seizures were invalid—e.g., they were identified  as occurring outside U.S. national boundaries—the numbers were not  significant, and we determined that the Geographic Information Systems  data were sufficiently reliable for the purposes of this report. Location  data that were determined to be invalid were not included in our analysis.  We compared CBP’s reporting requirements and use of asset assists  data against criteria in Standards for Internal Control in the Federal  Government, which, among other things, call for ensuring effectiveness   We determined that the recorded data on asset assists  and efficiency of management operations, including the use of the entity’s  resources.
In addition, we visited Border Patrol’s Tucson sector in Arizona to observe  Border Patrol agents operating SBInet technologies and other selected  technologies, such as RVSS towers, and discussed agents’ experiences  in using these technologies. While visiting the Tucson sector, we  interviewed officials regarding the deployment and contributions of  surveillance technologies within the sector. We visited the Tucson sector  because of the presence of surveillance technologies, such as SBInet  and RVSS towers, in that sector and because, under the Plan, the  Tucson sector has locations for which additional technology deployments,  such as IFTs, are planned. While the information we obtained from our  visit cannot be generalized to all Border Patrol sectors, it did provide us  with insights about the use of the deployed surveillance technologies.
Finally, we analyzed documents, including CBP’s Multi-Year Investment  and Management Plan for Border Security Fencing, Infrastructure, and  Technology for Fiscal Years 2014-2017, and interviewed CBP officials  responsible for overseeing the progress CBP and DHS have made in  implementing our November 2011 recommendations to identify the  mission benefits to be derived from technologies in the Plan and metrics  to measure the extent to which border security is expected to improve by  using these technologies.
We conducted this performance audit from September 2012 through  March 2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Photographs of Technologies in the Arizona Border Surveillance Technology Plan
section_title: Appendix III: Our Schedule Assessment Results for the Integrated Fixed Towers, Remote Video Surveillance System, and Mobile Surveillance Capability Programs
paragraphs: Best practices for cost estimating and scheduling identify 10 practices  associated with effective scheduling. These are (1) capturing all  activities, (2) sequencing all activities, (3) assigning resources to all  activities, (4) establishing the duration of all activities, (5) verifying that the  schedule is traceable horizontally and vertically, (6) confirming that the  critical path is valid, (7) ensuring reasonable total float, (8) conducting a  schedule risk analysis, (9) updating the schedule with actual progress and  logic, and (10) maintaining a baseline schedule. These practices are  summarized into four characteristics of a reliable schedule— comprehensive, well constructed, credible, and controlled.
We assessed the extent to which the March 2013 schedules for CBP’s  three highest-cost technology programs under the Arizona Border  Surveillance Technology Plan—IFT, RVSS, and MSC—met each of the  10 best practices. the 10 best practices as follows:   We characterized whether the schedules met each of
Not met—the program provided no evidence that satisfies any of the  criterion.
Minimally met—the program provided evidence that satisfies a small  portion of the criterion.
Partially met—the program provided evidence that satisfies about half  of the criterion.
Substantially met—the program provided evidence that satisfies a  large portion of the criterion.
We determined the overall assessment rating by assigning each individual rating a  number: Not met = 1, minimally met = 2, partially met = 3, substantially met = 4, and met =  5. Then, we took the average of the individual assessment ratings to determine the overall  rating for each of the four characteristics. The resulting average becomes the Overall  Assessment as follows: Not met = 1.0 to 1.4, minimally met = 1.5 to 2.4, partially met = 2.5  to 3.4, substantially met = 3.5 to 4.4, and met = 4.5 to 5.0. We developed this rating scale  in consultation with cost-estimating experts who helped develop the Cost Estimating and  Assessment Guide.
Met—the program provided complete evidence that satisfies the entire  criterion.
Table 7 provides the results of our analysis of the IFT, RVSS, and MSC  schedules as of March 2013.
section_title: Appendix IV: Summary Statistics on the Reporting of Asset Assists Data for Apprehensions and Seizures across the Tucson and Yuma Sectors from Fiscal Year 2010 through June 2013
subsections: section_title: Numbers of Apprehension and Seizure Events
paragraphs: DHS’s Enforcement Integrated Database includes a field that enables  Border Patrol agents to identify whether a technological or  nontechnological asset assisted in the apprehension of illegal entrants or  the seizure of drugs or other contraband. This appendix provides  summary statistics on the reporting of asset assists by Border Patrol  agents in the apprehension of illegal entrants and seizure of drugs and  other contraband across the Tucson and Yuma sectors from fiscal year  2010 through June 2013.
For the purposes of this report, “unreported asset assists” could include instances for  which an asset was not a contributing factor in apprehensions and seizures. sectors, conclusions about the differences in reported asset assists  across sectors and differences within sectors over time cannot be made.
As shown in tables 8 and 9, the 166,976 apprehension events that  occurred in the Tucson sector from fiscal year 2010 through June 2013  resulted in the apprehension of 549,357 illegal entrants, and 20,322  seizure events that occurred in the Tucson sector over the same period  resulted in the seizure of 21,973 items. and 6,828 seizure events occurring in the Yuma sector during the same  time period resulted in the apprehension of 17,580 illegal entrants and  7,892 seized items. The two tables also show that the percentages of  reported asset assists for apprehension events and seizure events and  the resulting apprehensions and seizures differed across the Tucson and  Yuma sectors and in both sectors and over time.
section_title: Reported Asset Assists by Type of Asset for Apprehensions and Seizures
paragraphs: Figure 13 shows that the percentages of apprehension events and  apprehensions in the Tucson sector for which asset assists were not  reported decreased from fiscal year 2010 through June 2013. Figure 14  shows that in the Yuma sector, the percentages of apprehension events  and apprehensions for which asset assists were not reported were higher  than in the Tucson sector, but similarly decreased over that time period.  For the first three quarters of fiscal year 2013, asset assist information  was not reported for more than one-half of the apprehension events in the  Tucson sector and nearly four-fifths of the apprehension events in the  Yuma sector.
Figures 13 and 14 also show that the percentages of apprehension  events and apprehensions for which technological asset assists and other  asset assists were reported increased during that period in both sectors.  Because it is difficult to determine whether unreported asset assists do  not involve asset assists, or do involve asset assists that were not  recorded, it is difficult to determine whether the increases in the  percentages of apprehension events and apprehensions involving  technology asset assets and other assists involve real increases, or  increases resulting from fewer asset assists going unreported. The higher  percentage of apprehension events and apprehensions involving  technology asset assists in the Tucson sector relative to the Yuma sector  may also be partly due to differences in the two sectors in unreported  asset assists, and also to differences in the number of technology assets  in the two sectors.
Figure 15 shows that for seizure events and seizures in the Tucson sector  from fiscal year 2010 through June 2013, as was the case for  apprehension events and apprehensions, the percentages for which  asset assists were unreported declined, while the percentages for which  technology asset assists and other asset assists were reported increased.  Unreported asset assists were lower for seizures than for apprehensions  in the Tucson sector, and the changes with respect to the percentages of  seizures involving unreported asset assists, technology asset assists, and  other asset assists in the Tucson sector were not as pronounced as the  changes with respect to apprehensions.
Figure 16 shows that in the Yuma sector over the same period, the  percentages for which asset assists were unreported declined, and the  percentage for which other (nontechnology) assets were reported  increased. In the Yuma sector, the percentage of technology asset  assists was small (less than 1 percent) in each of the fiscal years and  there was no discernible trend in the percentage of technology asset  assists. As with apprehensions, it is difficult to determine how much of the  increase in technology asset assists in the Tucson sector and other asset  assists in both sectors involves the increased use of technology and other  assets, or changes in reporting of asset assists.
section_title: Appendix V: Mission Benefits Identified by U.S. Customs and Border Protection of Its Surveillance Technologies
paragraphs: Table 10 summarizes the mission benefits to be derived from each of the  technologies to be deployed as part of the Arizona Border Surveillance  Technology Plan as outlined in CBP’s Multi-Year Investment and  Management Plan for Border Security Fencing, Infrastructure, and  Technology for Fiscal Years 2014-2017. According to CBP officials, each  of the seven technologies deployed or planned for deployment as part of  the Plan will increase situational awareness and enhance the ability of  Border Patrol agents to detect, identify, deter, and respond to threats  along the border.
section_title: Appendix VI: Comments from the Department of Homeland Security
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Jeanette Espinola (Assistant  Director), David Alexander, Charles Bausell, Frances Cook, Katherine  Davis, Joseph E. Dewechter, Jennifer Echard, Shannon Grabich, Yvette  Gutierrez, Eric Hauswirth, Richard Hung, Jason Lee, Grant Mallie, Linda  Miller, John Mingus, Anna Maria Ortiz, Karen Richey, Doug Sloane, Karl  Seifert, Nate Tranquilli, Katherine Trimble, Jim Ungvarsky, and Michelle  Woods made key contributions to this report.