Arizona Border Surveillance Technology Plan: Additional Actions Needed to Strengthen Management and Assess Effectiveness
Released: 2014-03-12T00:00:00

Letter

CBP’s Program Schedules and Life- cycle Cost Estimates Reflect Some but Not All Best Practices

In our March 2014 report, we assessed OTIA’s schedules as of March 2013 for the IFT, RVSS, and MSC programs and found that these program schedules addressed some, but not all, best practices for scheduling. The Schedule Assessment Guide identifies 10 best practices associated with effective scheduling, which are summarized into four characteristics of a reliable schedule—comprehensive, well constructed, credible, and controlled. According to our overall analysis, OTIA at least partially met the four characteristics of reliable schedules for the IFT and RVSS schedules (i.e., satisfied about half of the criterion), and partially or minimally met the four characteristics for the MSC schedule, as shown in table 1. For example, we reported that the schedule for the IFT program partially met the characteristic of being credible in that CBP had performed a schedule risk analysis for the program, but the risk analysis was not based on any connection between risks and specific activities.

We recommended that CBP ensure that scheduling best practices are applied to the IFT, RVSS, and MSC schedules. DHS concurred with the recommendation and stated that OTIA plans to ensure that scheduling best practices are applied as far as practical when updating the three programs’ schedules.

Further, in March 2014 we reported that CBP has not developed an Integrated Master Schedule for the Plan in accordance with best practices. Rather, OTIA has used the separate schedules for each individual program (or “project”) to manage implementation of the Plan. OTIA officials stated that an Integrated Master Schedule for the overarching Plan is not needed because the Plan contains individual acquisition programs as opposed to a plan consisting of seven integrated programs. However, collectively these programs are intended to provide CBP with a combination of surveillance capabilities to be used along the Arizona border with Mexico. Moreover, while the programs themselves may be independent of one another, the Plan’s resources are being shared among the programs.

OTIA officials stated that when schedules were developed for the Plan’s programs, they assumed that personnel would be dedicated to work on individual programs and not be shared between programs. However, as OTIA has initiated and continued work on the Plan’s programs, it has shared resources such as personnel among the programs, contributing, in part, to delays experienced by the programs. According to schedule best practices, an Integrated Master Schedule that allows managers to monitor all work activities, how long the activities will take, and how the activities are related to one another is a critical management tool for complex systems that involve the incorporation of a number of different projects, such as the Plan. Thus, we recommended that CBP develop an Integrated Master Schedule for the Plan.

DHS did not concur with this recommendation. In particular, DHS stated that maintaining an Integrated Master Schedule for the Plan undermines the DHS-approved implementation strategy for the individual programs making up the Plan and that a key element of the Plan has been the disaggregation of technology procurements. However, we continue to believe that developing an Integrated Master Schedule for the Plan is needed. As we reported in March 2014, this recommendation is not intended to imply that DHS needs to re-aggregate the Plan’s seven programs into a “system of systems” or change its procurement strategy in any form. The intent of the recommendation is for DHS to insert the individual schedules for each of the Plan’s programs into a single electronic Integrated Master Schedule file in order to identify any resource allocation issues among the programs’ schedules. Developing and maintaining an Integrated Master Schedule for the Plan could allow OTIA insight into current or programmed allocation of resources for all programs as opposed to attempting to resolve any resource constraints for each program individually.

In addition in March 2014, we reported that OTIA’s rough order of magnitude estimate for the Plan and individual Life-cycle Cost Estimates for the IFT and RVSS programs met some but not all best practices for such estimates. Cost-estimating best practices are summarized into four characteristics—well documented, comprehensive, accurate, and credible.completed at the time of our review for the IFT and RVSS programs showed that these estimates at least partially met three of these characteristics—well documented, comprehensive, and accurate. In terms of being credible, these estimates had not been verified with independent cost estimates in accordance with best practices. We recommended that CBP verify the Life-cycle Cost Estimates for the IFT Our analysis of CBP’s estimate for the Plan and estimates and RVSS programs with independent cost estimates and reconcile any differences.

DHS said it concurred with this recommendation, although we reported that DHS’s planned actions will not fully address the intent of the recommendation unless assumptions underlying the cost estimates change. In particular, DHS stated that at this point it does not believe that there would be a benefit in expending funds to obtain independent cost estimates and that if the costs realized to date continue to hold, there may be no requirement or value added in conducting full-blown updates with independent cost estimates. DHS noted, though, that if this assumption changes, OTIA will complete updates and consider preparing independent cost estimates, as appropriate. We recognize the need to balance the cost and time to verify the Life-cycle Cost Estimates with the benefits to be gained from verification with independent cost estimates. However, we continue to believe that independently verifying the Life- cycle Cost Estimates for the IFT and RVSS programs and reconciling any differences, consistent with best practices, could help CBP better ensure the reliability of the estimates.

CBP Did Not Fully Complete Documents for Acquisition Decisions Consistent with the Guidance

In March 2014, we reported for the Plan’s three highest-cost programs— IFT, RVSS, and MSC—DHS and CBP did not consistently approve key acquisition documents before or at the Acquisition Decision Events, in accordance with DHS’s acquisition guidance. An important aspect of an Acquisition Decision Event is the review and approval of key acquisition documents critical to establishing the need for a program, its operational requirements, an acquisition baseline, and test and support plans, according to DHS guidance. On the basis of our analysis for IFT, RVSS, and MSC programs under the Plan, we reported that the DHS Acquisition Decision Authority approved the IFT program and the CBP Acquisition Decision Authority approved the RVSS and MSC programs to proceed to subsequent phases in the Acquisition Life-cycle Framework without approving all six required acquisition documents for each program. Furthermore, we reported that one document for the IFT program, five documents for the RVSS program, and two documents for the MSC program were subsequently approved after the programs received authority to proceed to the next phase. DHS plans to complete and approve those documents for the IFT, RVSS, and MSC programs that have not yet been completed and approved.

With regard to one of the required documents—the Test and Evaluation Master Plan—we reported in March 2014 that this document for the IFT program, which was approved by DHS in November 2013, does not describe testing to evaluate the operational effectiveness and suitability of the system. Rather, the Test and Evaluation Master Plan describes CBP’s plans to conduct a limited user test of the IFT. According to the Test and Evaluation Master Plan, the limited user test will be designed to determine the IFT’s mission contribution. According to OTIA and the Test and Evaluation Master Plan, this testing is planned to occur during 30 days in environmental conditions present at one site—the Nogales station. CBP plans to conduct limited user testing for the IFT under the same process that is typically performed in any operational test and evaluation, according to the Test and Evaluation Master Plan. The November 2013 IFT Test and Evaluation Master Plan notes that, because the IFT acquisition strategy is to acquire non-developmental IFT systems from the marketplace (sometimes referred to as a commercial off-the- shelf system), a limited user test will provide Border Patrol with the information it needs to determine the mission contributions from the IFTs, and thus CBP does not plan to conduct more robust testing. However, this approach is not consistent with DHS’s acquisition guidance, which states that even for commercial off-the-shelf systems, operational test and evaluation should occur in the environmental conditions in which a system will be used before a full production decision for the system is made and the system is subsequently deployed.

As we reported, we recognize the need to balance the cost and time to conduct testing to determine the IFT’s operational effectiveness and suitability with the benefits to be gained from such testing. Although the limited user test should help provide CBP with information on the IFTs’ mission contribution and how Border Patrol can use the system in its operations, the limited user test does not position CBP to obtain information on how the IFTs may perform under the various environmental conditions the system could face once deployed. Conducting limited user testing in one area in Arizona—the Nogales station—for 30 days could limit the information available to CBP on how the IFT may perform in other conditions and locations along the Arizona border with Mexico. As of November 2013, CBP intended to deploy IFTs to 50 locations in southern Arizona, which can include differences in terrain and climate throughout the year.

We recommended that CBP revise the IFT Test and Evaluation Master Plan to more fully test the IFT program, before beginning full production, in the various environmental conditions in which IFTs will be used to determine operational effectiveness and suitability. DHS did not concur with this recommendation and stated that the Test and Evaluation Master Plan includes tailored testing and user assessments that will provide much, if not all, of the insight contemplated by the intent of the recommendation. However, as we reported in March 2014, we continue to believe that revising the Test and Evaluation Master Plan to include more robust testing to determine operational effectiveness and suitability could better position CBP to evaluate IFT capabilities before moving to full production for the system, help provide CBP with information on the extent to which the towers satisfy Border Patrol’s user requirements, and help reduce potential program risks.

CBP Has Identified Mission Benefits, but Does Not Capture Complete Data on the Contributions of Its Surveillance Technologies

We reported in March 2014 that CBP has identified the mission benefits of its surveillance technologies, but does not capture complete data on the contributions of these technologies, which in combination with other relevant performance metrics or indicators, could be used to better determine the contributions of CBP’s surveillance technologies and inform resource allocation decisions. CBP has identified mission benefits of surveillance technologies to be deployed under the Plan, such as improved situational awareness and agent safety.

While CBP has defined these mission benefits, the agency has not developed key attributes for performance metrics for all surveillance technologies to be deployed as part of the Plan, as we recommended in November 2011. In our April 2013 update on the progress made by the agencies to address our findings on duplication and cost savings across the federal government, CBP officials stated that operations of its two SBInet surveillance systems identified examples of key attributes for metrics that can be useful in assessing the Plan’s implementation for technologies. For example, according to CBP officials, to help measure whether illegal activity has decreased, examples of key attributes include decreases in the amount of arrests, complaints by ranchers and other citizens, and destruction of public and private lands and property. While the development of key attributes for metrics for the two SBInet surveillance systems is a positive step, CBP has not identified attributes for metrics for all technologies to be acquired and deployed as part of the Plan. Thus, to fully address the intent of our recommendation, CBP would need to develop and apply key attributes for performance metrics for each of the technologies to be deployed under the Plan to assess its progress in implementing the Plan and determine when mission benefits have been fully realized.

Furthermore, we reported in March 2014 that CBP is not capturing complete asset assist data on the contributions of its surveillance technologies to apprehensions and seizures, and these data are not being consistently recorded by Border Patrol agents and across locations. Although CBP has a field within its Enforcement Integrated Database (EID) for maintaining data on whether technological assets, such as SBInet surveillance towers, and non-technological assets, such as canine teams, assisted or contributed to the apprehension of illegal entrants, and seizure of drugs and other contraband, according to CBP officials, Border Patrol agents are not required to record these data. This limits CBP’s ability to collect, track, and analyze available data on asset assists to help monitor the contribution of surveillance technologies, including its SBInet system, to Border Patrol apprehensions and seizures and inform resource allocation decisions.

In our March 2014 report, we defined an “apprehension or seizure event” as the occasion on which Border Patrol agents make an apprehension of an illegal entrant or a seizure of drugs or other contraband. The event is recorded in the EID and a date and unique identifying number are assigned. An event can involve the apprehension of one or multiple illegal entrants or types of items, and each individual illegal entrant apprehended or type of item seized in the event is associated with the assigned unique identifying number. Our analysis of apprehension events included instances in which an event had at least one deportable individual. apprehension events. Since data on asset assists are not required to be reported, it is unclear whether the data were not reported because an asset was not a contributing factor in the apprehension or seizure or whether an asset was a contributing factor but was not recorded by agents.

As a result, CBP is not positioned to determine the contribution of surveillance technologies in the apprehension of illegal entrants and seizure of drugs and other contraband during the specified time frame. We reported that an Associate Chief at Border Patrol told us that while data on asset assists are not systematically recorded and tracked, Border Patrol recognizes the benefits of assessments of asset assists data, including those from surveillance technologies, such as the SBInet system. The Associate Chief further noted that these data in combination with other data, such as numbers of apprehensions and seizures, are used on a limited basis to help the agency make adjustments to its acquisition plans prior to deploying resources, thereby enabling the agency to make more informed deployment decisions.

We recommended that CBP require data on asset assists to be recorded and tracked within EID and that once these data are required to recorded and tracked, analyze available data on apprehensions and technological assists, in combination with other relevant performance metrics or indicators, as appropriate, to determine the contribution of surveillance technologies to CBP’s border security efforts. CBP concurred with our recommendations and stated that Border Patrol is changing its data collection process to allow for improved reporting on asset assists for apprehensions and seizures and intends to make it mandatory to record whether an asset assisted in an apprehension or seizure. DHS plans to change its process by December 31, 2014.

Chairwoman Miller, Ranking Member Jackson Lee, and members of the subcommittee, this concludes my prepared statement. I would be pleased to answer any questions that you may have.

GAO Contact and Staff Acknowledgments

For questions about this statement, please contact Rebecca Gambler at (202) 512-6912 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony and our related report are Jeanette Espinola, Assistant Director, and Michelle Woods, Analyst-in-Charge. Additional contributors include David Alexander, Frances Cook, Joseph E. Dewechter, Jennifer Echard, Yvette Gutierrez, Richard Hung, Jason Lee, Grant Mallie, Karen Richey, Doug Sloane, Nate Tranquilli, Katherine Trimble, and Jim Ungvarsky.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.