Secure Border Initiative: DHS Needs to Reconsider Its Proposed Investment in Key Technology Program
Published: 2010-05-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The technology component of the Department of Homeland Security's (DHS) Secure Border Initiative (SBI), referred to as SBInet, is to put observing systems along our nation's borders and provide Border Patrol command centers with the imagery and related tools and information needed in deciding whether to deploy agents. SBInet is being acquired and deployed in incremental blocks of capability, with the first block to cost about $1.3 billion. Because of the program's importance, size, and challenges, GAO was asked to, among other things, determine the extent to which DHS has (1) defined the scope of its proposed SBInet solution, (2) developed a reliable schedule for this solution, (3) demonstrated the cost-effectiveness of this solution, and (4) acquired the solution using key management processes. To do this, GAO compared key program documentation to relevant guidance and industry practices.
section_title: What GAO Found
paragraphs: DHS has defined the scope of the first incremental block of SBInet capabilities; however, these capabilities have continued to shrink from what the department previously committed to deliver. For example, the geographical "footprint" of the initially deployed capability has been reduced from three border sectors spanning about 655 miles to two sectors spanning about 387 miles. Further, the stringency of the performance capabilities has been relaxed, to the point that, for example, system performance will be deemed acceptable if it identifies less than 50 percent of items of interest that cross the border. The result is a system that is unlikely to live up to expectations. DHS has not developed a reliable integrated master schedule for delivering the first block of SBInet. Specifically, the schedule does not sufficiently comply with seven of nine key practices that relevant guidance states are important to having a reliable schedule. For example, the schedule does not adequately capture all necessary activities, assign resources to them, and reflect schedule risks. As a result, it is unclear when the first block will be completed, and continued delays are likely. DHS has also not demonstrated the cost-effectiveness of this first system block. In particular, it has not reliably estimated the costs of this block over its entire life cycle. To do so requires DHS to ensure that the estimate meets key practices that relevant guidance states are important to having an estimate that is comprehensive, well-documented, accurate, and credible. However, DHS's cost estimate for the initial block does not sufficiently possess any of these characteristics. Further, DHS has yet to identify expected benefits from the initial block, whether quantitative or qualitative, and analyze them relative to costs. As a result, it does not know whether its planned investment will produce mission value commensurate with costs. DHS has also not acquired the initial SBInet block in accordance with key life cycle management processes. While processes associated with, among other things, requirements development and management and risk management, have been adequately defined, they have not been adequately implemented. For example, key risks have not been captured in the risk management repository and thus have not been proactively mitigated. As a result, DHS is at increased risk of delivering a system that does not perform as intended. SBInet's decreasing scope, uncertain timing, unclear value proposition, and limited life cycle management discipline and rigor are due to a range of factors, including limitations in both defined requirements and the capabilities of commercially available system components, as well as the need to address competing program priorities, such as meeting aggressive system deployment milestones. As a result, it remains unclear whether the department's pursuit of SBInet is a cost effective course of action, and if it is, that it will produce expected results on time and within budget.

section_title: Letter
paragraphs: Securing the 6,000 miles of international borders that the contiguous  United States shares with Canada and Mexico is a challenge and a mission  imperative to the Department of Homeland Security (DHS). Although  hundreds of thousands of illegal aliens are prevented from entering the  country each year, many more are not detected. To enhance border  security and reduce illegal immigration, DHS launched its multiyear,  multibillion dollar Secure Border Initiative (SBI) program in November  2005. Through SBI, DHS intends to enhance surveillance technologies,  raise staffing levels, increase domestic enforcement of immigration laws,  and improve the physical infrastructure along the nation’s borders.
Within SBI, Secure Border Initiative Network (SBInet) is a multibillion  dollar program that includes the acquisition, development, integration,  deployment, and operation and maintenance of surveillance technologies  to create a “virtual fence” along the border, as well as command, control,  communications, and intelligence (C3I) technologies to create a picture of  the border in command centers and vehicles. Managed by DHS’s Customs  and Border Protection (CBP), SBInet is intended to strengthen the ability  of CBP to detect, identify, classify, track, and respond to illegal breaches  at and between land ports of entry.
In September 2008, we reported that SBInet was at risk because of a  number of acquisition management weaknesses, and we made  recommendations to address them that DHS largely agreed with and  committed to addressing. Because of the importance, high cost, and  challenges facing SBInet, you subsequently asked us to continue to review  DHS’s management of SBInet. As agreed, our objectives were to determine  the extent to which DHS has (1) defined the scope of its proposed system  solution, (2) developed a reliable schedule for delivering this solution,   (3) demonstrated the cost-effectiveness of this solution, (4) acquired this  solution in accordance with key life cycle management processes, and   (5) addressed our recent recommendations.
To accomplish our objectives, we largely focused on the first increment of  SBInet known as Block 1. In doing so, we reviewed key program  documentation, including guidance, plans, schedules, cost estimates, and  artifacts related to system life cycle events, requirements, risks, and  testing. We also analyzed a random probability sample of requirements  and their related verification methods. In addition, we interviewed  program officials about SBInet cost and schedule estimates, program  commitments, the development and implementation of the SBInet system  life cycle approach, requirements development and management, test  management, and risk management. We then compared this information to  relevant federal guidance, leading industry practices, and the  recommendations in our September 2008 report on SBInet to identify any  deviations and interviewed program officials as to the reasons for any  deviations. To assess the reliability of the data that we relied on to support  the findings in the report, we reviewed relevant program documentation to  substantiate evidence obtained through interviews with knowledgeable  agency officials, where available. We determined that the data used in this  report are sufficiently reliable. We have also made appropriate attribution  indicating the sources of the data used.
We conducted this performance audit from December 2008 to May 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Further details of our objectives, scope, and  methodology are in appendix I.
subsections: section_title: Background
paragraphs: SBInet includes the acquisition, development, integration, deployment,  and operations and maintenance of a mix of surveillance technologies,  such as cameras, radars, sensors, and C3I technologies. The initial focus of  SBInet has been on addressing the requirements of CBP’s Office of Border  Patrol, which is responsible for securing the borders between the land  ports of entry. Longer term, SBInet is to address requirements of CBP’s  Office of Field Operations, which controls vehicle and pedestrian traffic at  the ports of entry, and its Office of Air and Marine Operations, which  operates helicopters, fixed-wing aircraft, and marine vessels used in  securing the borders. (See fig. 1 for the potential long-term SBInet concept  of operations.)
Surveillance technologies are to include a variety of sensor systems.  Specifically, unattended ground sensors are to be used to detect heat and  vibrations associated with foot traffic and metal associated with vehicles.  Radar mounted on fixed and mobile towers is to detect movement, and  cameras on fixed and mobile towers are to be used by operators to  identify and classify items of interest detected and tracked by ground  sensors and radar. Aerial assets are also to be used to provide video and  infrared imaging to enhance tracking targets. These technologies are  generally to be acquired through the purchase of commercial off-the-shelf  (COTS) products.
C3I technologies (software and hardware) are to produce a common  operating picture (COP)—a uniform presentation of activities within  specific areas along the border. Together, the sensors, radar, and cameras  are to gather information along the border and transmit this informatio COP terminals located in command centers and agents’ vehicles, which in  turn are to assemble it to provide CBP agents with border situational  awareness. Among other things, COP hardware and software are to allow  agents to (1) view data from radar and sensors that detect and track  movement in the border areas, (2) control cameras to help identify and  classify illegal entries, (3) correlate entries wit h the positions of nearby  agents, and (4) enhance tactical decision making regarding the appropriate  r esponse to apprehend an entry, if necessary.
subsections: section_title: Overview of SBInet Management Structure, Acquisition Approach, and Status
paragraphs: To increase border security and decrease illegal immigration, DHS  launched SBI more than 4 years ago after canceling its America’s Shield  Initiative program. Since fiscal year 2006, DHS has received about $4.4  billion in appropriations for SBI, including about $2.5 billion for physical  fencing and related infrastructure, about $1.5 billion for virtual fencing  (surveillance systems) and related technical infrastructure (towers), and  about $300 million for program management. The SBI Program Ex Office, which is organizationally within CBP, is responsible for managing  key acquisition functions associated with SBInet, including prime  contractor tracking and oversight. It is organized into four components:  SBInet System Program Office (referred to as the SPO in this report),  Systems Engineering, Business Management, and Operational Integration As of Dece mber 31, 2009, the SBI Program Executive Office was staffed  with 188 people—87 government employees, 78 contractor staff, and 13  detailees. .
In September 2006, CBP awarded a 3-year prime contract to the Boeing Company, with three additional 1-year options for designing, producing,  testing, deploying, and sustaining SBI. In 2009, CBP exercised the first  option year. Under this contract, CBP has issued 10 task orders tha to SBInet, covering for example, COP design and development, system  deployment, and system maintenance and logistics support. As of  December 2009, 4 of the 10 task orders had been completed and 6 were  ongoing. (See table 1 for a summary of the SBInet task orders.)
One of the completed task orders is for an effort known as Project 28,  which is a prototype system that covers 28 miles of the border in CBP’s  Tucson Sector in Arizona, and has been operating since February 2008.  However, its completion took 8 months longer than planned because of  problems in integrating system components (e.g., cameras and radars)  with the COP software. As we have reported, these problems were  attributable to, among other things, limitations in requirements  development and contractor oversight.
Through the task orders, CBP’s strategy is to deliver SBInet capabilities  incrementally. To accomplish this, the SPO has adopted an evolutionary  system life cycle management approach in which system capabilities are  to be delivered to designated locations in a series of discrete subsets of  system functional and performance capabilities that are referred to as  blocks. The first block, which has been designated as Block 1, includes the  purchase of commercially available surveillance systems, development of  customized COP systems and software, and use of existing CBP  communications and network capabilities. Such an incremental approach  is a recognized best practice for acquiring large-scale, complex systems  because it allows users access to new capabilities and tools sooner, and  thus permits both their early operational use and evaluation. Subsequent  increments of SBInet capabilities are to be delivered based on feedback  and unmet requirements, as well as the availability of new technologies.
In general, the SBInet life cycle management approach consists of four  primary work flow activities: (1) Planning Activity, (2) System Block  Activity, (3) Project Laydown Activity, and (4) Sustainment Activity.  During the Planning Activity, the most critical user needs are to be  identified and balanced against what is affordable and technologically  available. The outcome of this process is to be a set of capability  requirements that are to be acquired, developed, and deployed as a  specific block. This set of capabilities, once agreed to by all stakeholders,  is then passed to the System Block Activity, during which the baseline  system solution to be fielded is designed and built. Also as part of this  activity, the verification steps are to be conducted on the individual system  components and the integrated system solution to ensure that they meet  defined requirements. The Project Laydown Activity is performed to  configure the block solution to a specific geographic area’s unique  operational characteristics. This activity involves assessing the unique  threats, terrain, and environmental concerns associated with a particular  area, incorporating these needs into the system configuration to be  deployed to that area, obtaining any needed environmental permits, and  constructing the infrastructure and installing the configured system. It also  involves test and evaluation activities, including system acceptance  testing, to verify that the installed block system was built as designed. The  final activity, Sustainment, is focused on the operations and maintenance  of the deployed block solution and supporting the user community.
Associated with each of these activities are various milestone or gate  reviews. For example, a key review for the System Block Activity is the  Critical Design Review (CDR). At this review, the block design and  requirements are baselined and formally controlled to approve and track  any changes. Among other things, this review is to verify that the block  solution will meet the stated requirements within the program’s cost and  schedule commitments. An important review conducted during the Project  Laydown Activity is the Deployment Design Review. At this review,  information such as the status of environmental reviews and land  acquisitions for a specific geographic area is assessed, and the location- specific system configuration is determined. The Deployment Readiness  Review is another key event during this activity. During this review,  readiness to begin site preparation and construction is assessed.
In addition to the four above described workflow activities are various key  life cycle management processes, such as requirements development and  management, risk management, and test management.
Requirements development and management, among other things,  involves defining and aligning a hierarchy of five types of SBInet  requirements. These five types begin with high-level operational  requirements and are followed by increasingly more detailed lower-level  requirements, to include system, component, C3I/COP software, and  design requirements. To help it manage the requirements, the SPO relies  on Boeing’s use of a database known as the Dynamic Object-Oriented  Requirements System (DOORS). The various types of SBInet requirements  are described in table 2.
Risk management entails taking proactive steps to identify and mitigate  potential problems before they become actual problems. The SPO has  defined a “risk” to be an uncertain event or condition that, if it occurs, will  have a negative effect on at least one program objective, such as schedule,  cost, scope, or technical performance. The SPO has defined an “issue” as a  risk that has been realized (i.e., a negative event or condition that  currently exists or has a 100 percent future certainty of occurring).  According to SBInet’s risk management process, anyone involved in the  program can identify a risk. Identified risks are submitted to the Risk  Management Team, which includes both the SPO Risk Manager and  Boeing Risk Manager, for preliminary review. If approved for further  consideration, the risk is entered into the Boeing-owned risk database,  which is accessible by SPO and Boeing officials. These risks are  subsequently reviewed by the Joint Risk Review Board, which is  composed of approximately 20 SPO and Boeing officials. If a risk is  approved, it is to be assigned an owner who will be responsible for  managing its mitigation.
Test management involves planning, conducting, documenting, and  reporting on a series of test events that first focus on the performance of  individual system components, then on the performance of integrated  system components, followed by system-level tests that focus on whether  the system (or major system increments) are acceptable and operationally  suitable. For SBInet, the program’s formal test events fall into two major  phases: developmental test and evaluation (DT&E) and operational test  and evaluation (OT&E). DT&E is to verify and validate the systems  engineering process and provide confidence that the system design  solution satisfies the desired capabilities. It consists of four test events— integration testing, component qualification testing, system qualification  testing, and system acceptance testing. OT&E is to ensure that the system  is effective and suitable in its operational environment with respect to key  considerations, including reliability, availability, compatibility, and  maintainability. SBInet defines three operational testing events—User  Assessment, Operational Test, and Follow-on Operational Test and  Evaluation. (See table 3 for each test event’s purpose, responsible parties,  and location.)
As of December 2009, the program was in the Project Laydown Activity.  Specifically, the SBInet CDR was completed in October 2008, and the  Block 1 design has been configured and is being tested and readied for  deployment to the Tucson Border Patrol Station (TUS-1), and then to the  Ajo Border Patrol Station (AJO-1), both of which are located in the CBP’s  Tucson Sector of the southwest border. More specifically, the Deployment  Design Review covering both TUS-1 and AJO-1 was completed in June  2007, the TUS-1 Deployment Readiness Review was completed in April  2009, and the AJO-1 Deployment Readiness Review was completed in  December 2009. Together, these two deployments are to cover 53 miles of  the 1,989-mile-long southern border (see fig. 2). Once a deployed  configuration has been accepted and is operational, the program will be in  the Sustainment Activity. As of November 2009, program documentation  showed that TUS-1 and AJO-1 were to be accepted in January and July  2010, respectively. However, the SBI Executive Director told us in  December 2009 that these and other SBInet scheduled milestones are  currently being re-evaluated. As of February 2010, TUS-1 and AJO-1 were  proposed to be accepted in September 2010 and November 2010,  respectively. However, this proposed schedule has yet to be approved by  CBP.
section_title: GAO Has Previously Reported on Numerous SBInet Management Weaknesses and Risks
paragraphs: Since 2007, we have identified a range of management weaknesses and  risks facing SBInet and we have made a number of recommendations to  address them that DHS has largely agreed with and, to varying degrees,  taken actions to address. For example, in February 2007, we reported that  DHS had not fully defined activities, milestones, and costs for  implementing the program; demonstrated how program activities would  further the strategic goals and objectives of SBI; and reported on the costs  incurred, activities, and progress made by the program in obtaining  operational control of the border. Further, we reported that the  program’s schedule contained a high level of concurrency among related  tasks and activities, which introduced considerable risk. Accordingly, we  recommended that DHS define explicit and measurable commitments  relative to, among other things, program capabilities, schedules, and costs,  and re-examine the level of concurrency in the schedule and adjust the  acquisition strategy appropriately. We are currently reviewing DHS’s  Fiscal Year 2010 SBI Expenditure Plan to, among other things, determine  the status of DHS’s actions to address these recommendations.
In October 2007, we testified that DHS had fallen behind in implementing  Project 28 due to software integration problems, although program  officials stated at that time that Boeing was making progress in correcting  the problems. Shortly thereafter, we testified that while DHS had  accepted Project 28, it did not fully meet expectations. To benefit from  this experience, program officials stated that they identified a number of  lessons learned, including the need to increase input from Border Patrol  agents and other users in SBInet design and development.
In September 2008, we reported that important aspects of SBInet were  ambiguous and in a continued state of flux, making it unclear and  uncertain what technological capabilities were to be delivered when. We  concluded that the absence of clarity and stability in key aspects of SBInet  impaired the ability of Congress to oversee the program and hold DHS  accountable for results, and hampered DHS’s ability to measure program  performance. As a result, we recommended that the SPO establish and  baseline the specific program commitments, including the specific system  functional and performance capabilities that are to be deployed, and when  they were to be deployed.
Also, we reported that the SPO had not effectively performed key  requirements definition and management practices. For example, it had  not ensured that different levels of requirements were properly aligned, as  evidenced by our analysis of a random probability sample of component  requirements showing that a large percentage of them could not be traced  to higher-level system and operational requirements. Also, some of  SBInet’s operational requirements, which are the basis for all lower-level  requirements, were found by an independent DHS review to be  unaffordable and unverifiable, thus casting doubt on the quality of lower- level requirements that were derived from them. As a result of these  limitations, we concluded that the risk of SBInet not meeting mission  needs and performing as intended was increased, as were the chances of  the program needing expensive and time-consuming system rework. We  recommended that the SPO implement key requirements development and  management practices to include (1) baselining requirements before  system design and development efforts begin; (2) analyzing requirements  prior to being baselined to ensure that that they are complete, achievable,  and verifiable; and (3) tracing requirements to higher-level requirements,  lower-level requirements, and test cases.
We also reported that SBInet testing was not being effectively managed.  For example, the SPO had not tested the individual system components to  be deployed to the initial deployment locations, even though the  contractor had initiated integration testing of these components with other  system components and subsystems. Further, while a test management  strategy was drafted, it had not been finalized and approved, and it did not  contain, among other things, a clear definition of testing roles and  responsibilities; a high-level master schedule of SBInet test activities; or  sufficient detail to effectively guide project-specific test planning, such as  milestones and metrics for specific project testing. We concluded that  without a structured and disciplined approach to testing, the risk that  SBInet would not satisfy user needs and operational requirements, thus  requiring system rework, was increased. We recommended that the SPO  (1) develop and document test practices prior to the start of testing; (2)  conduct appropriate component-level testing prior to integrating system  components; and (3) approve a test management strategy that, at a  minimum, includes a relevant testing schedule, establishes accountability  for testing activities by clearly defining testing roles and responsibilities,  and includes sufficient detail to allow for testing and oversight activities to  be clearly understood and communicated to test stakeholders.
In light of these weaknesses and risks, we further recommended that   (1) the risks associated with planned SBInet acquisition, development,  testing, and deployment activities be immediately assessed and (2) the  results, including proposed alternative courses of action for mitigating the  risks, be provided to the CBP Commissioner and DHS’s senior leadership,  as well as to the department’s congressional authorization and  appropriations committees. DHS agreed with all but one of the  recommendations in our September 2008 report. The status of DHS’s  efforts to implement these recommendations is summarized later in this  report and discussed in detail in appendix III.
In September 2009, we reported that SBInet had continued to experience  delays. For example, deployment to the entire southwest border had  slipped from early 2009 to 2016, and final acceptance of TUS-1 and AJO-1  had slipped from November 2009 and March 2010 to December 2009 and  June 2010, respectively. We did not make additional SBInet  recommendations at that time.
Most recently, we reported in January 2010 that SBInet testing was not  being effectively managed. Specifically, while DHS’s approach to testing  appropriately consisted of a series of progressively expansive  developmental and operational test events, the test plans, cases, and  procedures for the most recent test events were not defined in accordance  with important elements of relevant guidance. For example, none of the  plans adequately described testing risks and only two of the plans included  quality assurance procedures for making changes to test plans during their  execution. Further, a relatively small percentage of test cases for these  events described the test inputs and the test environment (e.g., facilities  and personnel to be used), both of which are essential to effective testing.
In addition, a large percentage of the test cases for these events were  changed extemporaneously during execution. While some of the changes  were minor, others were more significant, such as re-writing entire  procedures and changing the mapping of requirements to test cases.  Moreover, these changes to procedures were not made in accordance with  documented quality assurance processes, but rather were based on an  undocumented understanding that program officials said they established  with the contractor. Compounding the number and significance of changes  were questions raised by the SPO and a support contractor about the  appropriateness of some changes. For example, the SPO wrote to the  prime contractor that changes made to system qualification test cases and  procedures appeared to be designed to pass the test instead of being  designed to qualify the system.
Further, we reported that from March 2008 through July 2009, that about  1,300 SBInet defects had been found, with the number of new defects  identified during this time generally increasing faster than the number  being fixed—a trend that is not indicative of a system that is maturing and  ready for deployment. While the full magnitude of these unresolved  defects was unclear because the majority were not assigned a priority for  resolution, some of the defects that had been found were significant.  Although DHS reported that these defects had been resolved, they had  nevertheless caused program delays, and related problems had surfaced  that continued to impact the program’s schedule. Further, an early user  assessment of SBInet had raised significant concerns about the  performance of key system components and the system’s operational  suitability.
In light of these weaknesses, we recommended that DHS (1) revise the  program’s overall test plan to include (a) explicit criteria for assessing the  quality of test documentation, including test plans and test cases, and (b) a  process for analyzing, prioritizing, and resolving program defects; (2)  ensure that test schedules, plans, cases, and procedures are adequately  reviewed and approved consistent with the revised test plan; (3) ensure  that sufficient time is provided for reviewing and approving test  documents prior to beginning a given test event; and (4) triage the full  inventory of unresolved system problems, including identified user  concerns, and periodically report on their status to CBP and DHS  leadership. DHS fully agreed with the last three recommendations and  partially agreed with the first.
section_title: Block 1 Capabilities, Geographic Coverage, and Performance Standards Continue to Decrease
paragraphs: For Block 1, functional and performance capabilities and the number of  geographic locations to which they are to be deployed have continued to  decrease. We reported in September 2008 that the capabilities and  deployment locations of SBInet were decreasing. Since that time, the  number of component-level requirements to be deployed to TUS-1 and  AJO-1 has decreased by about 32 percent. In addition, the number of  sectors that the system is to be deployed to has been reduced from three  to two, and the stringency of the system performance measures that the  deployed system is to meet has been reduced. According to program  officials, the decreases are due to poorly defined requirements and  limitations in the capabilities of commercially available system  components. The result will be a deployed and operational system that,  like Project 28, does not live up to user expectations and provides less  mission support than was envisioned.
subsections: section_title: Functional Capabilities Have Been Reduced
paragraphs: Since our September 2008 report, the number of requirements that Block 1  is to meet has dropped considerably. Specifically, in September 2008, DHS  directed the SPO to identify the operational requirements to be allocated  to Block 1. In response, 106 operational requirements were established,  such as providing border surveillance, facilitating decision support and  situational awareness, enabling communications, providing operational  status and readiness metrics, and enabling system audits. Of the 106  requirements, 69 were to be included in the initial technology deployments  planned for TUS-1 and AJO-1. The remaining 37 were to be addressed in  future blocks.
To implement the 69 operational requirements, the SPO developed a  system-level requirement specification and 12 component-level  requirements specifications. More specifically, as part of CDR, which  concluded in October 2008, the 69 operational requirements for TUS-1 and  AJO-1 were associated with 97 system-level requirements. Also during  CDR, the 97 system-level requirements were associated with 1,286  component-level requirements.
However, between October 2008 and September 2009, the number of  component-level requirements was reduced from 1,286 to 880, or by about  32 percent. First, 281 requirements related to the specifications for three  components—communications, network operations, and network  security—were eliminated, leaving 1,005 baselined requirements.
Examples of the 281 requirements that were eliminated include the  following:  the failure in a single piece of hardware or software would not affect  mission critical functions which include detection and resolution of border  incursions;  the failure of a Network Operations Center/Security Operations Center  (NOC/SOC) workstation would not prevent the system from operating;  and the failure of one network power supply would be compensated for by  additional backup power supplies.
In addition, another 125 component-level requirements were granted  “waivers” or “deviations,” further reducing the number of Block 1  requirements to be deployed to TUS-1 and AJO-1 to 880 (as of September  2009). For example, the unattended ground sensors were required to  differentiate between human, vehicle, and animal targets. However,  because the sensors that are to be deployed to TUS-1 and AJO-1 are only  able to identify potential vehicles and are not able to differentiate between  humans and animals, this requirement was deviated. Similarly, the radar  was required to classify targets as humans or vehicles. However, the radar  also cannot differentiate between classes of targets (e.g., humans and  vehicles). As a result, the requirement in the radar specification was also  deviated.
Figure 3 summarizes the roughly 32 percent drop in requirements that has  occurred over the last 15 months.
According to program officials, component requirements were eliminated  because they were either poorly written or duplicative of other  requirements, or because the capabilities of commercially available  products were limited. In addition, they attributed a significant number of  eliminated requirements to a decision to not use a Boeing designed and  developed network and instead to use an existing DHS network. To the  SPO’s credit, this decision was made to align SBInet with DHS technical  standards and to increase the use of COTS products.
Compounding this reduction in Block 1 requirements is the likelihood that  further requirements deviations and waivers will be granted based on the  results of an early user assessment of the system. According to the July  2009 assessment report, certain SBInet components did not meet  requirements. For example:    The daytime cameras were judged to be operationally ineffective over 5  kilometers for identifying humans, while the requirement is that the  cameras be usable to 10 kilometers.
The laser range finder was determined to have an effective range of less  than 2 kilometers, while the requirement is for the effective range to be 10  kilometers.
Program officials told us that many of the limitations found during the user  assessment were previously known, and corrective actions were already  under way or planned for future technology upgrades to address them.  However, the officials also stated they plan to issue a waiver or deviation  for the camera and the laser range finder to address the two problems  discussed above. In addition, they stated that a previously known  limitation of the range of the radar will also need to be addressed through  a deviation. In this case, the radar is required to have a range of 20  kilometers, but testing shows a maximum range of 10 kilometers.
section_title: Geographic Coverage Has Been Reduced
paragraphs: Beyond the requirement reductions, the geographic locations to receive  the initial SBInet capabilities have also been reduced. As of September  2008, the initial Block 1 deployment was to span three border patrol  sectors: Tucson, Yuma, and El Paso—a total of 655 miles. According to  program officials, deployment to these three areas was the expressed  priority of the Border Patrol due to the high threat levels in these areas.  However, the Acquisition Program Baseline, which was drafted in  December 2008, states that initial deployment will be to just the Tucson  and Yuma Sectors, which will cover only 387 miles.
According to program officials, deployment to the 268 miles of the El Paso  Sector was dropped from the initial deployment in anticipation that the  sector will instead receive the capabilities slated for the next SBInet  increment (i.e., build). However, plans for the next increment have not  been developed. According to the SBI Executive Director in December  2009, the SPO is re-evaluating where and when future deployments of  SBInet will occur, and a date for when the revised deployment plans will  be available has not been set.
section_title: Performance Capabilities Have Decreased
paragraphs: System performance measures define how well a system is to perform  certain functions, and thus are important in ensuring that the system  meets mission and user needs. According to program documentation,  failure to meet a key performance parameter can limit the value of the  system and render it unsuccessful. In November 2008, the SPO re- evaluated its existing SBInet key performance parameters and determined  that SBInet must meet three such parameters: (1) the probability of  detecting items of interest between the border and the control boundary;  (2) the probability of correctly identifying items of interest as human,  conveyance, or others; and (3) the operational availability of the system.  According to program officials, subject matter experts and CBP staff  concluded that these three were critical to determining whether the  system successfully meets mission and user needs.
Associated with each parameter is a threshold for acceptable  performance. In November 2008, the SPO re-evaluated the thresholds for  its three key performance parameters, and it significantly relaxed each of  the thresholds:    The threshold for detecting items of interest dropped from 95 percent to  70 percent.
The threshold for identifying items of interest declined from 95 percent to  70 percent.
The threshold for operational availability decreased from 95 to 85 percent.
These threshold reductions significantly lower what constitutes  acceptable system performance. For example, the system will meet its  detection and identification performance requirements if it identifies   70 percent of the 70 percent of items that it detects, thus producing a   49 percent probability of identifying items of interest that cross the border.  Furthermore, the reduction in operational availability means that the time  that the system can be unavailable for use has gone from 18.25 days per  year to 54.75 days per year—or from approximately 2.5 weeks to about 7  weeks per year, excluding downtime for planned maintenance.
The SBI Executive Director attributed the performance reductions to  program officials’ limited understanding of needed operational capabilities  at the time the parameters and thresholds were set. The director further  stated that once Block 1 has been deployed and Border Patrol personnel  gain experience operating it, decisions will be made as to what additional  changes to make to the key performance parameters and associated  thresholds. Until then, system performance relative to identifying items of  interest and operational availability will remain as described above, which  program officials agreed fall short of expectations.
section_title: A Reliable Schedule for Completing Block 1 Has Not Been Developed
paragraphs: The success of a large-scale system acquisition program like SBInet  depends in part on having a reliable schedule of when the program’s set of  work activities and milestone events will occur, how long they will take,  and how they are related to one another. Among other things, a reliable  schedule provides a road map for systematic execution of a program and  the means by which to gauge progress, identify and address potential  problems, and promote accountability. Our research has identified nine  best practices associated with developing and maintaining a reliable  schedule. These are (1) capturing all activities, (2) sequencing all  activities, (3) assigning resources to all activities, (4) establishing the  duration of all activities, (5) integrating activities horizontally and  vertically, (6) establishing the critical path for all activities, (7) identifying  reasonable float between activities, (8) conducting a schedule risk  analysis, and (9) updating the schedule using logic and durations. To be  considered reliable, a schedule should meet all nine practices.
The August 2009 SBInet integrated master schedule, which was the most  current version available for our review, is not reliable because it  substantially complies with only two of the nine key schedule estimating  practices and it does not comply with, or only partially or minimally  complies with, the remaining seven practices (see table 4 for a summary  and app. IV for the detailed results of our analysis of the extent to which  the schedule meets each of the nine practices).
Examples of practices that were either substantially, partially, minimally,  or not met are provided below. Without having a reliable schedule, it is  unlikely that actual program execution will track to plans, thus increasing  the risk of cost, schedule, and performance shortfalls.
Capturing all activities: The schedule does not capture all activities as  defined in the program’s work breakdown structure or integrated master  plan. First, 57 percent of the activities listed in the work breakdown  structure (71 of 125) and 67 percent of the activities listed in the integrated  master plan (46 of 69) were not in the integrated master schedule. For  example, the schedule is missing efforts associated with systems  engineering, sensor towers, logistics, system test and evaluation,  operations support, and program management. Second, the schedule does  not include key activities to be performed by the government. For  example, while the schedule shows the final activity in the government  process for obtaining an environmental permit in order to construct  towers, it does not include the related government activities needed to  obtain the permit.
Sequencing all activities: The schedule identifies virtually all of the  predecessor and successor activities. Specifically, only 9 of 1,512 activities  (less than 1 percent) were missing predecessor links. Further, only 21 of  1,512 activities (about 1 percent) had improper predecessor and successor  links. While the number of unlinked activities is very small, not linking a  given activity can cause problems because changes to the durations of  these activities will not accurately change the dates for related activities.  More importantly, 403 of 1,512 activities (about 27 percent) are  constrained by “start no earlier than” dates, which is significant because it  means that these activities are not allowed to start earlier, even if their  respective predecessor activities have been completed.
Establishing the critical path for all activities: The schedule does not  reflect a valid critical path for several reasons. First, and as noted above,  it is missing government and contractor activities, and is thus not  complete. Second, as mentioned above, the schedule is missing some  predecessor links, and improperly establishes other predecessor and  successor links. Problems with the critical path were recognized by the  Defense Contract Management Agency as early as November 2008, when  it reported that the contractor could not develop a true critical path that  incorporates all program elements.
Conducting a schedule risk analysis: An analysis of the schedule’s  vulnerability to slippages in the completion of tasks has not been  performed. Further, program officials described the schedule as not  sufficiently stable to benefit from a risk analysis.
Reasons that these practices were not fully met vary and include the  program’s use of Boeing to develop and maintain the integrated master  schedule, even though Boeing’s processes and tools do not allow it to  include in the schedule work that it does not have under contract to  perform, as well as the constantly changing nature of the work to be  performed. Without a reliable schedule that includes all activities  necessary to complete Block 1, the SPO cannot accurately determine the  amount of time required to complete Block 1, and it does not have an  adequate basis for guiding the program’s execution and measuring  progress, thus reducing the likelihood of meeting the program’s  completion dates.
Collectively, the weaknesses in meeting the nine key practices for the  program’s integrated master schedule increase the risk of schedule  slippages and related cost overruns and make meaningful measurement  and oversight of program status and progress, as well as accountability for  results, difficult to achieve. In the case of Block 1, this risk has continued  to be realized. For example, the dates presented at the December 2008 to  November 2009 monthly program review meetings for government  acceptance of Block 1 at TUS-1 and AJO-1 showed a pattern of delays,  with TUS-1 and AJO-1 acceptance slipping by 4 months and 7 months,  respectively. (See fig. 4.) Moreover, these slipped dates have not been met,  and the SBI Executive Director told us in December 2009 that when Block  1 will be accepted and operational continues to change and remains  uncertain. As of February 2010, TUS-1 and AJO-1 were proposed to be  accepted in September 2010 and November 2010, respectively; however,  this proposed schedule has yet to be approved by CBP.
section_title: Cost-Effectiveness of Block 1 Has Not Been Demonstrated
paragraphs: As we have previously reported, the decision to invest in any system or  major system increment should be based on reliable estimates of costs and  meaningful forecasts of quantifiable and qualitative benefits over the  system’s useful life. For Block 1, DHS does not have a complete and  current life cycle cost estimate. Moreover, it has not projected the mission  benefits expected to accrue from Block 1 over the same life cycle.  According to program officials, it is premature to project such benefits  given the uncertainties surrounding the role that Block 1 will ultimately  play in overall border control operations. Without a meaningful  understanding of SBInet costs and benefits, DHS lacks an adequate basis  for knowing whether the initial system solution on which it plans to spend  at least $1.3 billion is cost-effective. Moreover, DHS and congressional  decision makers continue to lack a basis for deciding what investment in  SBInet beyond this initial capability is economically prudent.
subsections: section_title: Life Cycle Costs Have Not Been Reliably Estimated
paragraphs: A reliable cost estimate is critical to successfully delivering large-scale  information technology (IT) systems, like SBInet, as well as major system  increments, like Block 1. Such an estimate provides the basis for informed  investment decision making, realistic budget formulation, meaningful  progress measurement, and accountability for results. According to the  Office of Management and Budget (OMB), federal agencies must  maintain current and well-documented estimates of program costs, and  these estimates must encompass the program’s full life cycle. Among o things, OMB states that a reliable life cycle cost estimate is critical to the  capital planning and investment control process. Without such an  estimate, agencies are at increased risk of making poorly informed  investment decisions and securing insufficient resources to effectively  execute defined program plans and schedules, and thus experiencing  program cost, schedule, and performance shortfalls.
Our research has identified a number of practices that form the basis of  effective program cost estimating. These practices are aligned with four  characteristics of a reliable cost estimate. To be reliable, a cost estimate  should possess all four characteristics, each of which is summarized  below. (See app. V for the key practices associated with each  characteristic, including a description of each practice and our analysis of  the extent to which the SBInet cost estimate meets each practice.)
Comprehensive: The cost estimate should include all government and  contractor costs over the program’s full life cycle, from program inception  through design, development, deployment, and operation and maintenance  to retirement. It should also provide sufficient detail to ensure that cost  elements are neither omitted nor double counted, and it should document  all cost-influencing ground rules and assumptions.
Well-documented: The cost estimate should capture in writing things such  as the source and significance of the data used, the calculations performed  and their results, and the rationale for choosing a particular estimating  method or reference. Moreover, this information should be captured in  such a way that the data used to derive the estimate can be traced back to,  and verified against, their sources. Finally, the cost estimate should be  reviewed and accepted by management to demonstrate confidence in the  estimating process and the estimate.
Accurate: The cost estimate should not be overly conservative or  optimistic, and should be, among other things, based on an assessment of  most likely costs, adjusted properly for inflation, and validated against an  independent cost estimate. In addition, the estimate should be updated  regularly to reflect material changes in the program and actual cost  experience on the program. Further, steps should be taken to minimize  mathematical mistakes and their significance and to ground the estimate  in documented assumptions and a historical record of actual cost and  schedule experiences on comparable programs.
Credible: The cost estimate should discuss any limitations in the analysis  due to uncertainty or biases surrounding the data and assumptions. Major  assumptions should be varied and other outcomes computed to determine  how sensitive the estimate is to changes in the assumptions. Risk and  uncertainty inherent in the estimate should be assessed and disclosed.  Further, the estimate should be properly verified by, for example,  comparing the results with one or more independent cost estimates.
The SPO’s Block 1 life cycle cost estimate includes the costs to complete  those portions of Block 1 that are to be deployed to the Tucson and Yuma  Sectors, which together cover about 387 miles of the southwest border (53  miles associated with both TUS-1 and AJO-1, which are in the Tucson  Sector, as well as an additional 209 miles in the Tucson Sector and 125  miles in the Yuma Sector). More specifically, this estimate, which is dated  December 2008, shows the minimum cost to acquire and deploy Block 1 to  the Tucson and Yuma Sectors to be $758 million, with another $544 million  to operate and maintain this initial capability, for a total of about $1.3  billion.
However, this Block 1 cost estimate is not reliable because it does not  sufficiently possess any of the above four characteristics. Specifically:    The estimate is not comprehensive because it does not include all relevant  costs, such as support contractor costs and costs associated with system  and software design, development, and testing activities that were  incurred prior to December 2008. Moreover, it includes only 1 year of  operations and maintenance costs rather than these costs over the  expected life of the system. Further, the estimate does not document and  assess the risks associated with all ground rules and assumptions, such as  known budget constraints, staff and schedule variations, and technology  maturity.
The estimate is not well-documented because, among other things, the  sources and significance of key data have not been captured and the  quality of key data, such as historical costs and actual cost reports, is  limited. For example, instead of identifying and relying on historical costs  from similar programs, the estimate was based, in part, on engineering  judgment. Further, the calculations performed and their results, while  largely documented, did not document contingency reserves and the  associated confidence level for the risk-adjusted cost estimate. Also, as  noted above, assumptions integral to the estimate, such as those for  budget constraints, and staff and schedule variances, were not  documented.
The estimate is not accurate because it was not, for example, validated  against an independent cost estimate. Further, it has not been updated to  reflect material program changes since the estimate was developed. For  example, the estimate does not reflect development and testing activities  that were added since the estimate was approved to correct problems  discovered during testing. Further, the estimate has not been updated with  actual cost data available from the contractor.
The estimate is not credible because its inherent risk and uncertainty were  not adequately assessed, and thus the estimate does not address  limitations associated with the assumptions used to create it. For example,  the risks associated with software development were not examined, even  though such risks were known to exist. In fact, the only risks considered  were those associated with uncertainty in labor rates and hardware costs,  and instead of being based on historical quantitative analyses, these risks  were expressed by assigning them arbitrary positive or negative  percentages. In addition, and for the reasons mentioned above, the  estimate did not specify contingency reserve amounts to mitigate known  risks, and an independent cost estimate was not used to verify the  estimate.
Program officials attributed these limitations in the cost estimate’s  comprehensiveness, documentation, accuracy, and credibility to a range of  factors, including competing program office priorities and the  department’s limited cost estimating capabilities. For example, program  officials stated that the DHS Cost Analysis Division did not prepare an  independent estimate because it did not have, among other things, the  people and tools needed to do so. In this regard, this division reports that  as of July 2009, DHS only had eight cost estimators (six in headquarters  and two in program offices) for departmentwide needs.
Because the estimate does not adequately display these four  characteristics, it does not provide a reliable picture of Block 1’s life cycle  costs. As a result, DHS does not have complete information on which to  base informed investment decision making, understand system  affordability, and develop justifiable budget requests. Moreover, the Block  1 cost estimate does not provide a meaningful standard against which to  measure cost performance, is likely to show large cost overruns, and does  not provide a good basis for informing future cost estimates.
section_title: Expected Mission Benefits Have Yet to Be Adequately Defined
paragraphs: The Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to  ensure that IT investments actually produce tangible, observable  improvements in mission performance. As we have previously reported,to accomplish this, benefits that are expected to accrue from investments  need to be forecast and their actual accrual needs to be measured.
In the case of Block 1, however, expected mission benefits have not been  defined and measured. For example, while program officials told us that  system benefits are documented in the SBInet Mission Need Statement  dated October 2006, this document does not include either quantifiable or  qualitative benefits. Rather, it provides general statements such as “the  lack of a program such as SBInet increases the risks of terrorist threats  and other illegal activities.”
Congress recognized the importance of having a meaningful understanding  of SBInet’s value proposition when it required DHS in 2008 to provide in  its Border Security, Fencing, Infrastructure, and Technology Fiscal Year  2009 Expenditure Plan a description of how the department’s planned  expenditure of funds would be linked to expected SBI mission benefits  and outcomes. However, we reported that the plan DHS submitted only  described links among planned activities, expenditures, and outputs. It did  not link these to outcomes associated with improving operational control  of the border. More recently, we reported that while SBI technology and  physical infrastructure, along with increases in Border Patrol personnel,  are intended to allow DHS to gain effective control of U.S. borders, CBP’s  measures of effective control are limited. Thus, we recommended that  CBP conduct a cost-effectiveness evaluation of the SBI tactical  infrastructure’s impact on effective control of the border, and DHS agreed  with this recommendation. Further, program officials noted that  uncertainty about SBInet’s role in and contribution to effective control of  the border makes it difficult to forecast SBInet benefits. Rather, they said  that operational experience with Block 1 is first needed in order to  estimate such benefits.
While we recognize the value of operationally evaluating an early,  prototypical version of a system in order to better understand, among  other things, its mission impact, and thus to better inform investment  decisions, we question the basis for spending in excess of a billion dollars  to gain this operational experience. Without a meaningful understanding  and disclosure of SBInet benefits, to include the extent to which expected  mission benefits are known and unknown, DHS did not have the necessary  basis for justifying and making informed decisions about its sizeable  investment in Block 1, as well as for measuring the extent to which the  deployed Block 1 will actually deliver mission value commensurate with  costs.
section_title: Block 1 Has Not Been Managed in Accordance with Key Life Cycle Management Processes
paragraphs: Successful management of large IT programs, like SBInet, depends in large  part on having clearly defined and consistently applied life cycle  management processes. Our evaluations and research show that applying  system life cycle management rigor and discipline increases the likelihood  of delivering expected capabilities on time and within budget. In other  words, the quality of a system is greatly influenced by the quality of the  processes used to manage it.
To the SPO’s credit, it has defined key life cycle management processes  that are largely consistent with relevant guidance and associated best  practices. However, it has not effectively implemented these processes.  Specifically, it has not consistently followed its systems engineering plan,  requirements development and management plan, and risk management  approach. Reasons cited by program officials for not implementing these  processes include the decision by program officials to rely on contract  task order requirements that were developed prior to the systems  engineering plan, and competing SPO priorities, including meeting an  aggressive deployment schedule. Until the SPO consistently implements  these processes, it will remain challenged in its ability to successfully  deliver SBInet.
subsections: section_title: Key System Life Cycle Management Activities Have Not Been Consistently Performed
paragraphs: Each of the steps in a life cycle management approach serves an important  purpose and has inherent dependencies with one or more other steps. In  addition, the steps used in the approach should be clearly defined and  repeatable. Thus, if a life cycle management step is omitted or not  performed effectively, later steps can be affected, potentially resulting in  costly and time-consuming rework. For example, a system can be  effectively tested to determine whether it meets requirements only if these  requirements have been completely and correctly defined. To the extent  that interdependent life cycle management steps or activities are not  effectively performed, or are performed concurrently, a program will be at  increased risk of cost, schedule, and performance shortfalls.
The SPO’s Systems Engineering Plan documents its life cycle management  approach for SBInet definition, development, testing, deployment, and  sustainment. As noted earlier, we reported in September 2008 on a number  of weaknesses in the SBInet life cycle management approach and made  recommendations to improve it.In response, the SPO revised its Systems  Engineering Plan in November 2008, and to its credit, the revised plan is  largely consistent with DHS and other relevant guidance. For example, it  defines a number of key life cycle milestone or “gate” reviews that are  important in managing the program, such as initial planning reviews,  requirements reviews, system design reviews, and test reviews. In  addition, the revised plan requires most of the key artifacts and program  documents that DHS guidance identified as important to each gate review,  such as a concept of operations, an operational requirements document, a  deployment plan, a risk management plan, a life cycle cost estimate,  requirements documentation, and test plans. To illustrate, the plan  identifies CDR as the important milestone event where a design baseline is  to be established, requirements traceability is to be demonstrated, and  verification and testing plans are to be in place.
However, the Systems Engineering Plan does not address the content of  the key artifacts that it requires. For example, it does not provide a sample  document or content template for the concept of operations, the  operational requirements document, or the deployment plan. As a result,  the likelihood of the developers and reviewers of these artifacts sharing  and applying a consistent and repeatable understanding of their content is  minimized, thus increasing the risk that they will require costly and time- consuming rework. As we recently reported, the absence of content  guidance or criteria for assessing the quality of the prime contractor’s test- related deliverables was a primary reason that limitations were found in  test plans.
Beyond the content of the Systems Engineering Plan, the SPO has not  consistently implemented key system life cycle management activities for  Block 1 that are identified by the plan. For example, the following artifacts  were not reviewed or considered during the CDR that concluded in  October 2008:    Security Test Plan, which describes the process for assessing the  robustness of the system’s security capabilities (e.g., physical facilities,  hardware, software, and communications) in light of their vulnerabilities.
Quality Plan, which documents the process for verifying that the  contractor deliverables satisfy contractual requirements and meet or  exceed quality standards.
Test Plan, which describes the overall process for the test and evaluation,  including the development of detailed test event plans, test procedure  instructions, data collection methods, and evaluation reports.
Block Training Plan, which outlines the objectives, strategy, and  curriculum for training that are specific to each block, including the  activities needed to support the development of training materials,  coordination of training schedules, and reservation of personnel and  facilities.
Block Maintenance Plan, which lays out the policies and concepts to be  used to maintain the operational availability of hardware and software.
To the SPO’s credit, it reviewed and considered all but one of the key  artifacts for the TUS-1 Deployment Readiness Review that concluded in  April 2009. The omitted artifact was the Site Specific Training Plan, which  outlines the objectives, strategy, and curriculum for training that are  specific to each geographic site, including the activities needed to support  the development of training materials, coordination of training schedules,  and reservation of personnel and facilities. According to program officials,  even though the Systems Engineering Plan cites the training plan as  integral to the Deployment Readiness Review, this training plan is to be  reviewed as part of a later milestone review.
Program officials stated that a reason that the artifacts were omitted is  that they have yet to begin implementing the Systems Engineering Plan.  Instead, they have, for example, enforced the CDR requirements in the  System Task Order that Boeing was contractually required to follow. To  address this, they added that the SPO intends to bring the task orders into  alignment with the Systems Engineering Plan, but they did not specify  when this would occur. As a result, key milestone reviews and decisions  have not always benefited from life cycle management documentation that  the SPO has determined to be relevant and important to these milestone  events. More specifically, the Systems Engineering Plan states that the  gate reviews are intended to identify and address problems early and thus  minimize future costs and avoid subsequent operational issues. By not  fully informing these gate reviews and associated decisions with key life  cycle management documentation, the risk of Block 1 design and  deployment problems is increased, as is the likelihood of expensive and  time-consuming system rework.
section_title: Key Block 1 Requirements Have Not Been Adequately Developed and Managed
paragraphs: Well-defined and managed requirements are essential to successfully  acquiring large-scale systems, like SBInet. According to relevant  guidance,effective requirements development and management include  establishing a baseline set of requirements that are complete,  unambiguous, and testable. It also includes ensuring that system-level  requirements are traceable backwards to higher-level operational  requirements and forward to design requirements and the methods used to  verify that they are met. Among other things, this guidance states that such  traceability should be used to verify that higher-level requirements have  been met by first verifying that the corresponding lower-level  requirements have been satisfied.
However, not all Block 1 component requirements were sufficiently  defined at the time that they were baselined, and operational requirements  continue to be unclear and unverifiable. In addition, while requirements  are now largely traceable backwards to operational requirements and  forward to design requirements and verification methods, this traceability  has not been used until recently to verify that higher-level requirements  have been satisfied. Program officials attributed these limitations to  competing SPO priorities, including aggressive schedule demands. Without  ensuring that requirements are adequately defined and managed, the risks  of Block 1 not performing as intended, not meeting user needs, and  costing more and taking longer than necessary to complete are increased.
subsections: section_title: Not All Requirements Were Adequately Baselined
paragraphs: The SBInet Requirements Development and Management Plan states that a  baseline set of requirements should be established by the time of the CDR  and that these requirements should be complete, unambiguous, and  testable. Further, the program’s Systems Engineering Plan states that the  CDR is intended to establish the final allocated requirements baseline and  ensure that system development, integration, and testing can begin.
To the SPO’s credit, it established a baseline set of requirements for the  TUS-1 and AJO-1 system deployments at CDR. However, the baseline  requirements associated with the NOC/SOC were not adequately defined  at this time, as evidenced by the fact that they were significantly changed 2  months later. Specifically, about 33 percent of the component-level  requirements and 43 percent of the design specifications for NOC/SOC  were eliminated from the Block 1 design after CDR. Program officials  attributed these changes to the NOC/SOC requirements to (1)  requirements that were duplicative of another specification, and thus were  redundant; (2) requirements that were poorly written, and thus did not  accurately describe needs; and (3) requirements that related to the  security of a system that SBInet would not interface with, and thus were  unnecessary.
According to program officials, the NOC/SOC was a late addition to the  program, and at the time of CDR, the component’s requirements were  known to need additional work. Further, they stated that while the  requirements were not adequately baselined at the time of CDR, the  interface requirements were understood well enough to begin system  development.
Without properly baselined requirements, system testing challenges are  likely to occur, and the risk of system performance shortfalls, and thus  cost and schedule problems, are increased. In this regard, we recently  reported that NOC/SOC testing was hampered by incorrect mapping of  requirements to test cases, failure to test all of the requirements, and  significant changes to test cases made during the testing events. This  occurred in part because ambiguities in requirements caused testers to  rewrite test steps during execution based on interpretations of what they  thought the requirements meant, and they required the SPO to conduct  multiple events to test NOC/SOC requirements.
section_title: Block 1 Operational Requirements Remain Poorly Defined
paragraphs: According to the SBInet Requirements Development and Management  Plan, requirements should be achievable, verifiable, unambiguous, and  complete. To ensure this, the plan contains a checklist that is to be used in  verifying that each requirement possesses these characteristics.
However, not all of the SBInet operational requirements that pertain to  Block 1 possess these characteristics. Specifically, a November 2007 DHS  assessmentdetermined that 19 operational requirements, which form the  basis for the lower-level requirements used to design and build the system,  were not complete, achievable, verifiable, or affordable. Further, our  analysis of the 12 Block 1 requirements that are included in these 19  operational requirements shows that they have not been changed to  respond to the DHS findings. According to the assessment, 6 of the 12  were unaffordable and unverifiable, and the other 6 were incomplete.  Examples of these requirements and DHS’s assessment follow:   A requirement that the system should provide for complete coverage of  the border was determined to be unverifiable and unaffordable because  defining what complete coverage meant was too difficult and ensuring  complete coverage, given the varied and difficult terrain along the border,  was cost prohibitive.
A requirement that the system should be able to detect and identify  multiple simultaneous events with different individuals or groups was  determined to be incomplete because the requirement did not specify the  number of events to be included, the scope of the area to be covered, and  the system components to be involved.
As we have previously reported,these limitations in the operational  requirements affect the quality of system, component, and software  requirements. This is significant because, as of September 2009, these 12  operational requirements were associated with 16 system-level  requirements, which were associated with 152 component-level  requirements, or approximately 15 percent of the total number of  component-level requirements. According to program officials, these  requirements were not updated because the SPO planned to resolve the  problems through the testing process. However, we recently reported that  requirements limitations actually contributed to testing challenges.  Specifically, we reported that about 71 percent of combined system  qualification and component qualification test cases had to be rewritten  extemporaneously during test execution. According to program officials,  this was partly due to ambiguities in requirements, which led to differing  opinions among the program and contractor staff about what was required  to effectively demonstrate that the requirements were met.
Further, program officials stated that a number of requirements have been  granted deviations or waivers because they were poorly written. For  example:    A requirement for camera equipment to “conform to the capabilities and  limitations of the users to operate and maintain it in its operational  environment and not exceed user capabilities” was determined to be  subjective and unquantifiable and thus was waived.
A requirement for the tower design to accommodate the future integration  of components “without causing impact on cost, schedule, and/or  technical performance” was determined to have no specific criteria to  objectively demonstrate closure decision and thus was also waived.
As a result of these deviations and waivers, the system capabilities that are  to be delivered as part of Block 1 will be less than originally envisioned.
section_title: Requirements Traceability Has Improved, but Until Recently Has Not Been Used To Determine If Operational Requirements Were Met
paragraphs: Consistent with relevant guidance,the SBInet Requirements  Development and Management Plan provides for maintaining bidirectional  traceability from high-level operational requirements through detailed low- level requirements to test plans. More specifically, it states that  operational requirements should trace to system requirements, which in  turn should trace to component requirements that trace to design  requirements, which further trace to v erification methods.
Since September 2008, the SPO has worked with Boeing to manually  review each requirement and develop a bidirectional traceability matrix.  Further, it has used this matrix to update the DOORS requirements  database. Our analysis of the traceability of a random sample of Block 1  component-level requirements in the DOORS database shows that they are  largely traceable backwards to operational requirements and forward to  design requirements and verification methods. For example, we estimate  that only 5 percent (with a 95 percent confidence interval between 1 and  14 percent) of a random sample of component requirements cannot be  traced to the system requirements and then to the operational  requirements. In addition, we estimate that 0 percent (with a 95 percent  confidence interval between 0 and 5 percent) of the component  requirements in the same sample do not trace to a verification method.  (See table 5 for the results of our analysis along with the associated  confidence intervals.)
By establishing this traceability, the SPO is better positioned to know the  extent to which the acquired and deployed system can meet operational  requirements.
However, the SPO has not used its requirements traceability in closing  higher-level component requirements. According to relevant guidance,all  lower-level requirements (i.e., children) should be closed in order to  sufficiently demonstrate that the higher-level requirements (i.e., parents)  have been met. Consistent with this guidance, the SBInet Requirements  Development and Management Plan states that ensuring the traceability of  requirements from children to their parents is an integral part of ensuring  that testing is properly planned and conducted. However, 4 of 8 higher- level component requirements (parents) in the above cited random sample  of system-level requirements were closed regardless of whether their  corresponding lower-level design requirements (children) had been  closed. According to program officials, this is because their standard  practice in closing parent requirements, until recently, was to sometimes  close them before their children were closed. Further, they said that this  was consistent with their verification criteria for closing higher-level  requirements, which did not require closure of the corresponding lower- level requirements. They also said that the reason parent verification  criteria did not always reflect children verification criteria was that  traceability was still being established when the verification criteria were  developed and thus parent-child relationships were not always available to  inform the closure criteria. Furthermore, they stated that schedule  demands did not permit them to ensure that the verification criteria for  requirements were aligned with the traceability information.
After we shared our findings on parent requirement closure with the SPO,  officials stated that they had changed their approach and will no longer  close parent requirements without ensuring that all of the children  requirements have first been closed. However, they did not commit to  reviewing previously closed parents to determine that all of the children  were closed. Without fully ensuring traceability among requirements  verification methods, the risks of delivering a system solution that does  not fully meet user needs or perform as intended, and thus requires  additional time and resources to deliver, are increased.
section_title: Key Risks Have Not Been Effectively Managed and Disclosed
paragraphs: Risk management is a continuous, forward-looking process that effectively  anticipates and mitigates risks that may have a critical impact on a  program’s success. In 2008, the SPO documented a risk management  approach that largely complies with relevant guidance. However, it has  not effectively implemented this approach for all risks. Moreover,  available documentation does not demonstrate that significant risks were  disclosed to DHS and congressional decision makers in a timely fashion,  as we previously recommended and, while risk disclosure to DHS  leadership has recently improved, not all risks have been formally  captured and thus shared. As a result, the program will likely continue to  experience actual cost, schedule, and performance shortfalls, and key  decision makers will continue to be less than fully informed.
subsections: section_title: Risk Management Approach Has Been Adequately Defined
paragraphs: According to relevant guidance, effective risk management includes  defining a process that, among other things, proactively identifies and  analyzes risks on the basis of likelihood of occurrence and impact, assigns  ownership, provides for mitigation, and monitors status. To the SPO’s  credit, it has developed an approach for risk management that is largely  consistent with this guidance. For example, the approach provides for    continuously identifying risks throughout the program’s life cycle before  they develop into actual problems, including suggested methods for doing  so, such as conducting brainstorming sessions and interviewing subject  matter experts;   analyzing identified risks to determine their likelihood of occurring and    assigning responsibility for risks;    developing a risk mitigation plan, to include a set of discrete, measurable  actions or events which, if successfully accomplished, can avoid or reduce  the likelihood of occurrence or severity of impact of the risk; and   executing and regularly monitoring risk mitigation plans to ensure that  they are implemented and to allow for corrective actions if the desired  results are not being achieved.
In February 2007, we reported that the program’s risk management  approach was in the process of being established. Specifically, we noted  that at that time the SPO had drafted a risk management plan, established  a governance structure, developed a risk management database, and  identified 30 risks. In April 2009, we reported that the DHS Chief  Information Officer had certified that this approach provided for the  regular identification, evaluation, mitigation, and monitoring of risks  throughout the system life cycle, and that it provided for communicating  high-risk conditions to DHS investment decision makers.
section_title: Risk Management Approach Has Not Been Fully Implemented, although Improvements Are Under Way
paragraphs: The SPO has not adhered to key aspects of its defined process for  managing program risks. In particular, the program’s risk management  repository, which is the tool used for capturing and tracking risks and  their mitigation, has not included key risks that have been identified by  stakeholders. For example, our analysis of reports from the repository  showing all open and closed risks from April 2006 to September 2009  shows that the following program risks that have been identified by us and  others were not captured in the repository:    program cost and schedule risks briefed by the SPO to senior SBInet  officials in January 2009, such as unplanned and unauthorized work  impacting the credibility of the program cost data, and program costs and  schedule plans lacking traceability;   program schedule and cost estimate risks identified by the Defense  Contract Management Agency prior to March 2009, such as contractor- provided documentation not permitting adequate assessment of critical  path accuracy, and cost projections not including all applicable elements  and thus lacking credibility; and the risk of the SPO’s heavy reliance on contractors, reported by the DHS  Office of Inspector General in June 2009.
In addition, the SBI Executive Director told us that the program faces a  number of other risks, all but one of which were also not in the repository.  These include the lack of well-defined acquisition management processes,  staff with the appropriate acquisition expertise, and agreement on key  system performance parameters. According to program officials, some of  these risks are not in the repository because Boeing is responsible for  operating and maintaining the repository, and the specifics surrounding  the risks and their mitigation are considered acquisition sensitive, meaning  that they should not be shared with Boeing. In this regard, the officials  acknowledged that the SPO needs a risk database independent of the  contractor to manage these acquisition-sensitive risks.
Further, the Risk Manager identified other limitations that have hindered  the SPO’s risk management efforts, along with recent actions intended to  address them. For example:    Risk review meetings were only being held once a month, which was  resulting in lost opportunities to mitigate risks that were to be realized as  actual problems within 30 days. As a result, the frequency of these  meetings has been increased to twice a month.
Risk information provided to senior SBI managers at monthly Joint  Program Management Review Meetingswas not sufficiently detailed, and  thus has been expanded.
Changes were being made to the risk management repository by  contractor staff without sufficient justification and without the approval of  the Joint Risk Review Board. For example, program officials cited an  instance in which a risk’s severity was changed from medium to high and  no board member knew the reason for the change. As a result, the number  of contractor staff authorized to modify data in the repository was  reduced.
The repository did not include all requisite information for all identified  risks. For example, some risks were missing the rationale for the  likelihood of occurrence and the potential impact. As a result, the Joint  Risk Review Board has adopted a policy of not accepting risks that are  missing requisite information.
According to the Risk Manager, competing program priorities have  resulted in insufficient resources devoted to risk management activities,  which has contributed to the state of the SPO’s risk management efforts.  However, he added that the SPO is taking steps to improve risk  management by revising risk management guidance, implementing a CBP- approved database tool for managing government-only risks, and  increasing risk management training and oversight.
Until the program’s risk management is strengthened and effectively  implemented, the program will continue to be challenged in its ability to  forestall cost, schedule, and performance problems.
section_title: Risks Have Not Been Fully Disclosed, but Improvement Has Recently Occurred
paragraphs: As noted earlier, we recommended in September 2008 that the SPO assess  SBInet risks and that the results of these assessments, along with  alternative courses of action to address them, be provided to DHS  leadership and congressional committees. According to program  officials, shortly after receiving our draft report they briefed the DHS  Acquisition Review Board on, among other things, SBInet risks.  However, the briefing slides used for this meeting do not identify  individual risks. Instead, the briefing contains one slide that only identifies  “contributing factors” to changes in the program’s schedule, including a reallocation SBInet funding to SBI physical infrastructure, concurrencies and delays that have occurred in testing, and the need for environmental  studies. The slides do not identify risks and alternative courses of action to address or mitigate them.
In addition, program officials told us that they briefed congressional  committees during the fall of 2008 on the program’s status, which they said  included disclosure of program risks. However, they did not have any  documentation of these briefings to show which committees were briefed,  when the briefings occurred, who was present, and what was discussed  and disclosed. Further, House Committee on Homeland Security staff  stated that while program officials briefed them following our September  2008 report, specific program risks were not disclosed. As a result, it does  not appear that either DHS or congressional stakeholders received timely  information on risks facing the program at a crucial juncture in its life  cycle.
To the SPO’s credit, it has recently improved its disclosure of risks facing  the program. In particular, the SBI Executive Director briefed the DHS  Chief Information Officer in November 2009 on specific program risks.  However, this briefing states that the risks presented were the Block 1  risks as captured in the contractor’s risk repository and that additional  risks have not yet been formalized (see above discussion about repository  limitations). Until all key risks are formally managed and regularly  disclosed to department and congressional stakeholders, informed SBInet  investment decision making will be constrained.
section_title: DHS Has Yet to Implement GAO’s Recent SBInet Recommendations
paragraphs: As noted earlier, we reported on a number of SBInet program management  weaknesses in September 2008, and we concluded that these weaknesses  introduced considerable risk that the program would not meet  expectations and would require time-consuming and expensive rework.  In summary, these problems included a lack of clarity and certainty  surrounding what technological capabilities would be delivered when, and  a lack of rigor and discipline around requirements definition and  management and test management. To address these problems and  thereby reduce the program’s exposure to cost, schedule, and  performance risks, we made eight recommendations. DHS concurred with  seven of the recommendations and disagreed with one aspect of the  remaining one.
In summary, the department has not implemented two of the  recommendations and has partially implemented the remaining six. See  table 6 for a summary and appendix III for a detailed discussion of the  status of each recommendation.
section_title: Conclusions
paragraphs: DHS has yet to demonstrate that its proposed SBInet solution is a cost- effective course of action, and thus whether the considerable time and  money being invested to acquire and deploy it is a wise and prudent use of  limited resources. Given that the magnitude of the initial investment in  SBInet spans more than 3 years of effort and totals hundreds of millions of  dollars, coupled with the fact that the scope of the initial system’s  capabilities and areas of deployment have continued to shrink, the  program is fraught with risk and uncertainty. As a result, the time is now  for DHS to thoughtfully reconsider its proposed SBInet solution, and in  doing so, to explore ways to both limit its near-term investment in an  initial set of operational capabilities and develop and share with  congressional decision makers reliable projections of the relative costs  and benefits of longer-term alternatives for meeting the mission goals and  outcomes that SBInet is intended to advance, or reasons why such  information is not available and the uncertainty and risks associated with  not having it.
Compounding the risks and uncertainty surrounding whether the  department is pursuing the right course of action are a number of system  life cycle management concerns, including limitations in the integrated  master schedule; shortcomings in the documentation available to inform  key milestone decisions; and weaknesses in how requirements have been  developed and managed, risks have been managed, and tests have been  conducted. Collectively, these concerns mean that the program is not  employing the kind of acquisition management rigor and discipline needed  to reasonably ensure that proposed system capabilities and benefits will  be delivered on time and on budget.
Because of SBInet’s decreased scope, uncertain timing, unclear costs  relative to benefits, and limited life cycle management discipline and rigor,  in combination with its size and mission importance, the program  represents a risky undertaking. To minimize the program’s exposure to  risk, it is imperative for DHS to move swiftly to first ensure that SBInet, as  proposed, is the right course of action for meeting its stated border  security and immigration management goals and outcomes, and once this  is established, for it to move with equal diligence to ensure that it is being  managed the right way. To this end, our prior recommendations to DHS  relative to SBInet provide for strengthening a number of life cycle  management processes, including requirements development and  management and test management. Accordingly, we are not making  additional recommendations that focus on these processes at this time.
section_title: Recommendations for Executive Action
paragraphs: To address the considerable risks and uncertainties facing DHS on its  SBInet program, we are making 12 recommendations. Specifically, we  recommend that the Secretary of Homeland Security direct the  Commissioner of U.S. Customs and Border Protection to limit future  investment in the program to only work that meets one or both of the  following two conditions: (1) is already under contract and supports  deployment, acceptance, and operational evaluation of only those Block 1  capabilities (functions and performance levels) that are currently targeted  for TUS-1 and AJO-1; or (2) provides the analytical basis for informing a  departmental decision as to what, if any, expanded investment in SBInet,  both in terms of capabilities (functions and performance) and deployment  locations, represents a prudent, responsible, and affordable use of  resources for achieving the department’s border security and immigration  management mission.
With respect to the first condition, we further recommend that the  Secretary of Homeland Security direct the Commissioner of U.S. Customs  and Border Protection to have the SBI Executive Director make it a  program priority to ensure that  the integrated master schedule for delivering Block 1 capabilities to TUS-1  and AJO-1 is revised to address the key schedule estimating practices  discussed in this report; the currently defined Block 1 requirements, including key performance  parameters, are independently validated as complete, verifiable, and  affordable and any limitations found in the requirements are addressed; the Systems Engineering Plan is revised to include or reference  documentation templates for key artifacts required at milestone gate  reviews;   all parent requirements that have been closed are supported by evidence  of the closure of all corresponding and associated child requirements; and   all significant risks facing the program are captured, mitigated, tracked,  and periodically reported to DHS and congressional decision makers.
Also with respect to the first condition, we reiterate our prior  recommendations, as stated in our September 2008 report,relative to  establishing program commitments, implementing the Systems  Engineering Plan, defining and managing requirements, and testing.
With respect to the second condition, we further recommend that the  Secretary of Homeland Security direct the Commissioner of U.S. Customs  and Border Protection to have the SBI Executive Director make it a  program priority to ensure that    a life cycle cost estimate for any incremental block of SBInet capabilities  that is to include capabilities and cover locations beyond those associated  with the TUS-1 and AJO-1 deployments is developed in a manner that  reflects the four characteristics of a reliable estimate discussed in this  report;   a forecast of the qualitative and quantitative benefits to be derived from  any such incremental block of SBInet over its useful life, or reasons why  such forecasts are not currently possible, are developed and documented; the estimated life cycle costs and benefits and associated net present value  of any such incremental block of SBInet capabilities, or reasons why such  an economic analysis cannot be performed, are prepared and documented;  and the results of these analyses, or the documented reasons why such  analyses cannot be provided, are provided to the Commissioner of U.S.  Customs and Border Protection and the DHS Acquisition Review Board.
Also with respect to this second condition, we recommend that the  Secretary of Homeland Security direct the Deputy Secretary of Homeland  Security, as the Chair of the DHS Acquisition Review Board, to (1) decide,  in consultation with the board and Commissioner of U.S. Customs and  Border Protection, what, if any, expanded investment in SBInet, both in  terms of capabilities (functions and performance) and deployment  locations, represents a prudent, responsible, and affordable use of  resources for achieving the department’s border security and immigration  management mission; and (2) report the decision, and the basis for it, to  the department’s authorization and appropriations committees.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Director,  Departmental GAO/Office of Inspector General Liaison, and reprinted in  appendix II, DHS stated that it agreed with ten of our recommendations  and partially agreed with the remaining two. In this regard, it described  ongoing and planned actions to address each, and it provided milestones  for completing these actions. In addition, DHS provided technical  comments, which we have incorporated in the report as appropriate.
In agreeing with our first recommendation, however, DHS commented that  the words “one of” were omitted before the two conditions contained in  the recommendation. However, this interpretation is not correct. Rather,  the intent of our recommendation is to limit future investment on the  program to either of the conditions, meaning “one or both of.”  Notwithstanding DHS’s interpretation, we believe that actions that it  described to address this recommendation, which include freezing funding  beyond the initial deployments to TUS-1 and AJO-1 until it completes a  comprehensive reassessment of the program that includes an analysis of  the cost and mission effectiveness of alternative technologies, is  consistent with the intent of the recommendation. Nevertheless, we have  slightly modified the recommendation to avoid any further confusion.
Regarding its partial agreement with our recommendation for revising the  integrated master schedule in accordance with a range of best practices  embodied in our cost and schedule estimating guide, DHS acknowledged  the merits of employing these practices and stated that it is committed to  adopting and deploying them. However, it added that the current contract  structure limits its ability to fully implement all the practices prior to  completing the TUS-1 and AJO-1 deployments. We understand that  program facts and circumstances create practical limitations associated  with some of the practices, and believe that DHS’s planned actions are  consistent with the intent of our recommendation.
Regarding its partial agreement with our recommendation that reiterated a  number of the recommendations that we made in a prior report, DHS  stated that, while these prior recommendations reflect program  management best practices and it continues to make incremental  improvements to address each, the scope of the program had narrowed  since these recommendations were made. As a result, DHS stated that  these prior recommendations were not fully applicable until and unless a  decision was made to move the program forward and conduct future  deployments beyond TUS-1 and AJO-1. We acknowledge that the facts and  circumstances surrounding the program have recently changed and that  these changes impact the nature and timing of actions appropriate for  implementing them. Moreover, we believe that DHS’s planned actions are  consistent with the intent of our recommendation.
DHS also commented that it believed that it had implemented two of our  recommendations and that these recommendations should be closed.  Because closure of our recommendations requires evidentiary validation  of described actions, and because many of the actions that DHS described  were planned rather than completed, we are not closing any of our  recommendations at this time. As part of our recurring review of the status  of all of our open recommendations, we will determine if and when the  recommendations have been satisfied and thus can be closed.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees and other parties. We will also send copies to  the Secretary of Homeland Security, the Commissioner of the U.S.  Customs and Border Protection, and the Director of the Office of  Management and Budget. In addition, this report will be available at no  cost on the GAO Web site at http://www.gao.gov.
Should you or your offices have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine the extent to which the Department of  Homeland Security (DHS) has (1) defined the scope of its proposed Secure  Border Initiative Network (SBInet) solution, (2) developed a reliable  schedule for delivering this solution, (3) demonstrated the cost- effectiveness of this solution, (4) acquired this solution in accordance with  key life cycle management processes, and (5) addressed our recent SBInet  recommendations. To accomplish our objectives, we largely focused on  the first increment of SBInet, known as Block 1.
To determine the extent to which DHS has defined the scope of its  proposed system solution, we reviewed key program documentation  related to the Block 1 functional and performance requirements and  deployment locations, such as the SBInet Acquisition Program Baseline  and related acquisition decision memorandums, the Operational  Requirements Document, the Operational Requirements Document  Elements Applicable to Block 1 System, the Requirements Traceability  Matrix, the Requirements Verification Matrix, and the SBInet Block 1 User  Assessment. In addition, we compared Block 1 requirements that were  baselined in October 2008 as part of the Critical Design Review (CDR) to  the Block 1 requirements as defined as of September 2009 to identify what,  if any, changes had occurred, and we interviewed program officials as to  the reasons for any changes. We also compared the locations, including  the miles of border associated with these locations, that were to receive  Block 1 as of September 2008 to the locations specified in the program’s  March 2009 Acquisition Program Baseline to identify any changes, and we  interviewed program officials as to the reasons for any changes. Further,  we compared the key performance parameters listed in the Operational  Requirements Document, dated March 2007, to the key performance  parameters in the program’s Acquisition Program Baseline dated March  2009.
To determine the extent to which DHS has developed a reliable schedule  for its proposed system solution, we analyzed the SBInet integrated master  schedule as of June 2009 against the nine key schedule estimating  practices in our Cost Estimating and Assessment Guide. In doing so, we  used commercially available software tools to determine whether it, for  example, included all critical activities, a logical sequence of activities, and  reasonable activity durations. Further, we observed a demonstration of the  schedule in June 2009 provided by contractor officials responsible for  maintaining the schedule and program officials responsible for overseeing  the contractor. In July 2009, we observed a demonstration of the program  office’s efforts to reconcile the version of the integrated master schedule  that is exported for the government’s use with the version of the schedule  that the prime contractor uses to manage the program. During this  demonstration, we discussed some of our concerns regarding the  integrated master schedule with program officials and we inquired about  deviations from some of the key practices. Subsequently, the program  office provided us with a revised version of the integrated master schedule  as of August 2009, which we analyzed. In doing so, we repeated the above  described steps. Further, we characterized the extent to which the revised  schedule met each of the practices as either Not Met, Minimally Met,  Partially Met, Substantially Met, or Met. In addition, we analyzed changes  in the scheduled Block 1 deployment dates presented at each of the  monthly program reviews for the 1-year period beginning in December  2008 and ending in November 2009.
To determine the extent to which DHS has demonstrated the cost- effectiveness of the proposed solution, we evaluated the reliability of the  Block 1 life cycle cost estimate and the definition of expected system  benefits, both of which are addressed below.
Cost estimate: We first observed a demonstration of the cost model used  to develop the estimate, which was provided by the contractor officials  who are responsible for maintaining it and the program officials who are  responsible for overseeing the contractor. We then analyzed the derivation  of the cost estimate relative to 12 key practices associated with four  characteristics of a reliable estimate. As defined in our Cost Estimating  and Assessment Guide, these four characteristics are comprehensive,  well-documented, accurate, and credible, and the practices address, for  example, the methodologies, assumptions, and source data used. We also  interviewed program officials responsible for the cost estimate about the  estimate’s derivation. We then characterized the extent to which each of  the four characteristics was met as either Not Met, Minimally Met,  Partially Met, Substantially Met, or Met. To do so, we scored each of the  12 individual key practices associated with the four characteristics on a  scale of 1-5 (Not Met = 1, Minimally Met = 2, Partially Met = 3,  Substantially Met = 4, and Met = 5), and then averaged the individual  practice scores associated with a given characteristic to determine the  score for that characteristic.
Benefits: We interviewed program officials to identify any forecasts of  qualitative and quantitative benefits that the system was to produce. In this  regard, we were directed to the SBInet Mission Need Statement dated  October 2006, which we analyzed. In addition, we reviewed our prior  reports on the Secure Border Initiative (SBI), including a report on the SBI  expenditure plan, which is a plan that DHS has been required by statute to  submit to the House and Senate Appropriations Committees to, among  other things, identify expected system benefits. We also interviewed  program officials to determine the extent to which the system’s life cycle  costs and expected benefits had been analyzed together to economically  justify DHS’s proposed investment in SBInet.
To determine the extent to which DHS has acquired its proposed system  solution in accordance with key life cycle management processes, we  focused on three key processes: the system engineering approach,  requirements development and management, and risk management, each  of which is addressed below.
Systems engineering approach: We compared the program’s defined  system engineering approach, as defined in the SBInet Systems Program  Office’s (SPO) Systems Engineering Plan, to DHS and other relevant  guidance. To determine the extent to which the defined systems  engineering approach had been implemented, we focused on two major  “gates” (i.e., life cycle milestone reviews)—the CDR and the Deployment  Readiness Review. For each of these reviews, we compared the package of  documentation prepared for and used during these reviews to the  program’s defined system engineering approach as specified in the  Systems Engineering Plan to determine what, if any, deviations existed.  We also interviewed program officials as to the reason for any deviations.
Requirements development and management: We compared relevant  requirements management documentation, such as the Requirements  Development and Management Plan, the Requirements Management Plan,  the Configuration and Data Management Plan, the Operational  Requirements Document, the system-level requirements specification,  and the component-level requirements specifications, to relevant  requirements development and management guidance to identify an variances, focusing on the extent to which requirements were properly  baselined, adequately defined, and fully traced. With respect to  requirements baselining, we compared the component and system  requirements as of September 2008, which were approved during the CDR that concluded in October 2008, to the component and system  requirements as of November 2008, and identified the number and  percentage of requirements changes. We also interviewed program  officials as to the reasons for any changes. For requirements definition, weassessed the extent to which operational requirements that were identified  as poorly defined in November 2007 had been clarified in the Operatio Requirements Document, Elements Applicable to Block 1 System, dat November 2008. In doing so, we focused on those operational  requirements that are associated with Block 1. We also traced these Blo 1 operational requirements to the lower-level system requirements (i.e.,  system and component requirements) to determine how many of the  lower-level requirements were associated with any unchanged operationa requirements. For requirements traceability, we randomly selected a  sample of 60 requirements from 1,008 component requirements in the  program’s requirements management tool, known as the Dynamic Object- Oriented Requirements System (DOORS), as of July 2009. Before doing so we reviewed the quality of the access controls for the database, and we  interviewed program and contractor officials and received a DOOR tutorial to understand their respective roles in requirements management  and development and the use of DOORS. Once satisfied as to the reliability of the data in DOORS, we then traced each of the 60 requirements  S  backwards to the system requirements and then to the operational  requirements and forward to design requirements and verification  methods. Because we followed a probability procedure based on ra selection, we are 95 percent confident that each of the confidence  intervals in this report will include the true values in the study population.  We used statistical methods appropriate for audit compliance testing to  estimate 95 percent confiden r equirements in our sample. ce intervals for the traceability of    Risk management: We reviewed relevant documentation, such as t SBInet Risk/Issue/Opportunity Management Plan, the SBInet SPO  Risk/Issue/Opportunity Management Process, and the SBInet Risk  Management Policy, as well as extracts from the SBInet risk management  database and minutes of meetings and agendas from the Risk Management  Team and the Joint Risk Review Board. In doing so, we compared the risk  management process defined in these documents to relevant guidance todetermine the extent to which the program has defined an effective risk  management approach. Further, we observed a demonstration of the r database, and we compared SBInet risks identified by us and others, including the SBI Executive Director, to the risks in the database to  determine the extent to which all key risks were being actively managed.  Further, we discussed actions recently taken and planned to improve risk  management with the person responsible for SBInet risk management. W also reviewed briefings and related material provided to DHS leadership  during oversight reviews of SBInet and interviewed program officials  to  ascertain the extent to which program risks were disclosed at these  reviews and at meetings with congressional committees. In this regard, we also asked cognizant staff with the House Homeland Security Commi about the extent to which pr o ogram risks were disclosed by program  fficials in status briefings.
To determine the extent to which DHS has addressed our prior SBInet  recommendations, we focused on the eight recommendations that we made in our September 2008 report. For each recommendation, we leveraged the work described above, augmenting it as necessary to  determine any plans or actions peculiar to a given recommendat example, to determine the status of efforts to address our prior  recommendation related to SBInet testing, we reviewed key testing  ion. For  documentation, such as the Test and Evaluation Master Plan; SBInet  component and system qualification test plans, test procedures, and test  reports; program management reviews; program office briefings; and D Acquisition Review p Board decision memoranda. We also interviewed  rogram officials.
To support our work across the above objectives, we also interviewed  officials from the Department of Defense’s Defense Contract Managemen Agency, which provides contractor oversight services, to understand i reviews of the contractor’s integrated master schedule, requirements  development and management activities, risk management practices, and  testing activities. We also reviewed Defense Contract Management Agenc reports pertaining to  documentation, such as monthly status reports and  the integrated master schedule and cost reporting.
To assess the reliability of the data that we relied on to support the  findings in the report, we reviewed relevant program documentation to  substantiate evidence obtained through interviews with knowledgeable  agency officials, where available. We determined that the data used in this  also made appropriate attribution  report are sufficiently reliable. We have  indicating the sources of the data used.
We performed our work at the Customs and Border Protection  headquarters and contractor facilities in the Washington, D.C.,  metropolitan area and at a contractor facility and a Defense Contract  Management Agency office in Huntsville, Alabama. We conducted this  performance audit from December 2008 to May 2010 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropria evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.  (CBP)
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: Status of Key GAO Recommendations
paragraphs: In September 2008, we reported on a number of SBInet program  management weaknesses and associated risks related to establishing  program commitments, developing an integrated master schedule, defining  and implementing a life cycle management approach, developing and  managing requirements, and testing. To address these weaknesses and  risks, we made a number of recommendations. Table 7 provides details on  DHS efforts to address each recommendation.
section_title: Appendix IV: Detailed Results of GAO Assessment of SBInet Program Schedule
paragraphs: Our research has identified a range of best practices associated with  effective schedule estimating.  These are (1) capturing all activities, (2)  sequencing all activities, (3) assigning resources to all activities, (4)  establishing the duration of all activities, (5) integrating activities  horizontally and vertically, (6) establishing the critical path for all  activities, (7) identifying reasonable float time between activities, (8)  conducting a schedule risk analysis, and (9) updating the schedule using  logic and durations. We assessed the extent to which the SBInet integrated  master schedule, dated August 2009, met each of the nine practices as  either Not Met (the program provided no evidence that satisfies any  portion of the criterion), Minimally Met (the program provided evidence  that satisfies less than one-half of the criterion), Partially Met (the program  provided evidence that satisfies about one-half of the criterion),  Substantially Met (the program provided evidence that satisfies more than  one-half of the criterion), and Met (the program provided evidence that  satisfies the entire criterion). Table 8 shows the detailed results of our  analysis.
section_title: Appendix V: Detailed Results of GAO Assessment of SBInet Cost Estimate
paragraphs: Our research has identified 12 practices that are integral to effective  program life cycle cost estimating. These 12 practices in turn relate to  four characteristics of a high-quality and reliable cost estimate:    Comprehensive: The cost estimate should include all government and  contractor costs over the program’s full life cycle, from program inception  through design, development, deployment, and operation and maintenance  to retirement. It should also provide sufficient detail to ensure that cost  elements are neither omitted nor double-counted, and it should document  all cost-influencing ground rules and assumptions.
Well-documented: The cost estimate should capture in writing things such  as the source and significance of the data used, the calculations performed  and their results, and the rationale for choosing a particular estimating  method or reference. Moreover, this information should be captured in  such a way that the data used to derive the estimate can be traced back to,  and verified against, their sources. Finally, the cost estimate should be  reviewed and accepted by management to demonstrate confidence in the  estimating process and the estimate.
Accurate: The cost estimate should not be overly conservative or  optimistic, and should be, among other things, based on an assessment of  most likely costs, adjusted properly for inflation, and validated against an  independent cost estimate. In addition, the estimate should be updated  regularly to reflect material changes in the program and actual cost  experience with the program. Further, steps should be taken to minimize  mathematical mistakes and their significance and to ground the estimate  in documented assumptions and a historical record of actual cost and  schedule experiences with other comparable programs.
Credible: The cost estimate should discuss any limitations in the analysis  due to uncertainty or biases surrounding data or assumptions. Major  assumptions should be varied and other outcomes computed to determine  how sensitive the estimate is to changes in the assumptions. Risk and  uncertainty inherent in the estimate should be assessed and disclosed.  Further, the estimate should be properly verified by, for example,  comparing the results with an independent cost estimate.
Our analysis of the $1.3 billion SBInet life cycle cost estimate relative to  each of the 12 best practices, as well as to each of the four characteristics,  is summarized in table 9. A detailed analysis relative to the 12 practices is  in table 10.
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Deborah Davis (Assistant  Director), David Alexander, Rebecca Alvarez, Carl Barden, Tisha  Derricotte, Neil Doherty, Nancy Glover, Dan Gordon, Cheryl Dottermusch,  Thomas J. Johnson, Kaelin P. Kuhn, Jason T. Lee, Lee McCracken, Jamelyn  Payan, Karen Richey, Karl W.D. Seifert, Matt Snyder, Sushmita Srikanth,  Jennifer Stavros-Turner, Stacey L. Steele, and Karen Talley made key  contributions to this report.