Secure Border Initiative: DHS Needs to Reconsider Its Proposed Investment in Key Technology Program
Released: 2010-06-17T00:00:00

Letter

Background

SBInet includes the acquisition, development, integration, deployment, and operations and maintenance of a mix of surveillance technologies, such as cameras, radars, sensors, and C3I technologies. The initial focus of SBInet has been on addressing the requirements of CBP’s Office of Border Patrol, which is responsible for securing the borders between the land ports of entry. Longer term, SBInet is to address requirements of CBP’s Office of Field Operations, which controls vehicle and pedestrian traffic at the ports of entry, and its Office of Air and Marine Operations, which operates helicopters, fixed-wing aircraft, and marine vessels used in securing the borders. (See fig. 1 for the potential long-term SBInet concept of operations.)

Surveillance technologies are to include a variety of sensor systems. Specifically, unattended ground sensors are to be used to detect heat and vibrations associated with foot traffic and metal associated with vehicles. Radar mounted on fixed and mobile towers is to detect movement, and cameras on fixed and mobile towers are to be used by operators to identify and classify items of interest detected and tracked by ground sensors and radar. Aerial assets are also to be used to provide video and infrared imaging to enhance tracking targets. These technologies are generally to be acquired through the purchase of commercial off-the-shelf (COTS) products.

C3I technologies (software and hardware) are to produce a common operating picture (COP)—a uniform presentation of activities within specific areas along the border. Together, the sensors, radar, and cameras are to gather information along the border and transmit this informatio COP terminals located in command centers and agents’ vehicles, which in turn are to assemble it to provide CBP agents with border situational awareness. Among other things, COP hardware and software are to allow agents to (1) view data from radar and sensors that detect and track movement in the border areas, (2) control cameras to help identify and classify illegal entries, (3) correlate entries wit h the positions of nearby agents, and (4) enhance tactical decision making regarding the appropriate r esponse to apprehend an entry, if necessary.

Overview of SBInet Management Structure, Acquisition Approach, and Status

To increase border security and decrease illegal immigration, DHS launched SBI more than 4 years ago after canceling its America’s Shield Initiative program. Since fiscal year 2006, DHS has received about $4.4 billion in appropriations for SBI, including about $2.5 billion for physical fencing and related infrastructure, about $1.5 billion for virtual fencing (surveillance systems) and related technical infrastructure (towers), and about $300 million for program management. The SBI Program Ex Office, which is organizationally within CBP, is responsible for managing key acquisition functions associated with SBInet, including prime contractor tracking and oversight. It is organized into four components: SBInet System Program Office (referred to as the SPO in this report), Systems Engineering, Business Management, and Operational Integration As of Dece mber 31, 2009, the SBI Program Executive Office was staffed with 188 people—87 government employees, 78 contractor staff, and 13 detailees. .

In September 2006, CBP awarded a 3-year prime contract to the Boeing Company, with three additional 1-year options for designing, producing, testing, deploying, and sustaining SBI. In 2009, CBP exercised the first option year. Under this contract, CBP has issued 10 task orders tha to SBInet, covering for example, COP design and development, system deployment, and system maintenance and logistics support. As of December 2009, 4 of the 10 task orders had been completed and 6 were ongoing. (See table 1 for a summary of the SBInet task orders.)

One of the completed task orders is for an effort known as Project 28, which is a prototype system that covers 28 miles of the border in CBP’s Tucson Sector in Arizona, and has been operating since February 2008. However, its completion took 8 months longer than planned because of problems in integrating system components (e.g., cameras and radars) with the COP software. As we have reported, these problems were attributable to, among other things, limitations in requirements development and contractor oversight.

Through the task orders, CBP’s strategy is to deliver SBInet capabilities incrementally. To accomplish this, the SPO has adopted an evolutionary system life cycle management approach in which system capabilities are to be delivered to designated locations in a series of discrete subsets of system functional and performance capabilities that are referred to as blocks. The first block, which has been designated as Block 1, includes the purchase of commercially available surveillance systems, development of customized COP systems and software, and use of existing CBP communications and network capabilities. Such an incremental approach is a recognized best practice for acquiring large-scale, complex systems because it allows users access to new capabilities and tools sooner, and thus permits both their early operational use and evaluation. Subsequent increments of SBInet capabilities are to be delivered based on feedback and unmet requirements, as well as the availability of new technologies.

In general, the SBInet life cycle management approach consists of four primary work flow activities: (1) Planning Activity, (2) System Block Activity, (3) Project Laydown Activity, and (4) Sustainment Activity. During the Planning Activity, the most critical user needs are to be identified and balanced against what is affordable and technologically available. The outcome of this process is to be a set of capability requirements that are to be acquired, developed, and deployed as a specific block. This set of capabilities, once agreed to by all stakeholders, is then passed to the System Block Activity, during which the baseline system solution to be fielded is designed and built. Also as part of this activity, the verification steps are to be conducted on the individual system components and the integrated system solution to ensure that they meet defined requirements. The Project Laydown Activity is performed to configure the block solution to a specific geographic area’s unique operational characteristics. This activity involves assessing the unique threats, terrain, and environmental concerns associated with a particular area, incorporating these needs into the system configuration to be deployed to that area, obtaining any needed environmental permits, and constructing the infrastructure and installing the configured system. It also involves test and evaluation activities, including system acceptance testing, to verify that the installed block system was built as designed. The final activity, Sustainment, is focused on the operations and maintenance of the deployed block solution and supporting the user community.

Associated with each of these activities are various milestone or gate reviews. For example, a key review for the System Block Activity is the Critical Design Review (CDR). At this review, the block design and requirements are baselined and formally controlled to approve and track any changes. Among other things, this review is to verify that the block solution will meet the stated requirements within the program’s cost and schedule commitments. An important review conducted during the Project Laydown Activity is the Deployment Design Review. At this review, information such as the status of environmental reviews and land acquisitions for a specific geographic area is assessed, and the location- specific system configuration is determined. The Deployment Readiness Review is another key event during this activity. During this review, readiness to begin site preparation and construction is assessed.

In addition to the four above described workflow activities are various key life cycle management processes, such as requirements development and management, risk management, and test management.

Requirements development and management, among other things, involves defining and aligning a hierarchy of five types of SBInet requirements. These five types begin with high-level operational requirements and are followed by increasingly more detailed lower-level requirements, to include system, component, C3I/COP software, and design requirements. To help it manage the requirements, the SPO relies on Boeing’s use of a database known as the Dynamic Object-Oriented Requirements System (DOORS). The various types of SBInet requirements are described in table 2.

Risk management entails taking proactive steps to identify and mitigate potential problems before they become actual problems. The SPO has defined a “risk” to be an uncertain event or condition that, if it occurs, will have a negative effect on at least one program objective, such as schedule, cost, scope, or technical performance. The SPO has defined an “issue” as a risk that has been realized (i.e., a negative event or condition that currently exists or has a 100 percent future certainty of occurring). According to SBInet’s risk management process, anyone involved in the program can identify a risk. Identified risks are submitted to the Risk Management Team, which includes both the SPO Risk Manager and Boeing Risk Manager, for preliminary review. If approved for further consideration, the risk is entered into the Boeing-owned risk database, which is accessible by SPO and Boeing officials. These risks are subsequently reviewed by the Joint Risk Review Board, which is composed of approximately 20 SPO and Boeing officials. If a risk is approved, it is to be assigned an owner who will be responsible for managing its mitigation.

Test management involves planning, conducting, documenting, and reporting on a series of test events that first focus on the performance of individual system components, then on the performance of integrated system components, followed by system-level tests that focus on whether the system (or major system increments) are acceptable and operationally suitable. For SBInet, the program’s formal test events fall into two major phases: developmental test and evaluation (DT&E) and operational test and evaluation (OT&E). DT&E is to verify and validate the systems engineering process and provide confidence that the system design solution satisfies the desired capabilities. It consists of four test events— integration testing, component qualification testing, system qualification testing, and system acceptance testing. OT&E is to ensure that the system is effective and suitable in its operational environment with respect to key considerations, including reliability, availability, compatibility, and maintainability. SBInet defines three operational testing events—User Assessment, Operational Test, and Follow-on Operational Test and Evaluation. (See table 3 for each test event’s purpose, responsible parties, and location.)

As of December 2009, the program was in the Project Laydown Activity. Specifically, the SBInet CDR was completed in October 2008, and the Block 1 design has been configured and is being tested and readied for deployment to the Tucson Border Patrol Station (TUS-1), and then to the Ajo Border Patrol Station (AJO-1), both of which are located in the CBP’s Tucson Sector of the southwest border. More specifically, the Deployment Design Review covering both TUS-1 and AJO-1 was completed in June 2007, the TUS-1 Deployment Readiness Review was completed in April 2009, and the AJO-1 Deployment Readiness Review was completed in December 2009. Together, these two deployments are to cover 53 miles of the 1,989-mile-long southern border (see fig. 2). Once a deployed configuration has been accepted and is operational, the program will be in the Sustainment Activity. As of November 2009, program documentation showed that TUS-1 and AJO-1 were to be accepted in January and July 2010, respectively. However, the SBI Executive Director told us in December 2009 that these and other SBInet scheduled milestones are currently being re-evaluated. As of February 2010, TUS-1 and AJO-1 were proposed to be accepted in September 2010 and November 2010, respectively. However, this proposed schedule has yet to be approved by CBP.

GAO Has Previously Reported on Numerous SBInet Management Weaknesses and Risks

Since 2007, we have identified a range of management weaknesses and risks facing SBInet and we have made a number of recommendations to address them that DHS has largely agreed with and, to varying degrees, taken actions to address. For example, in February 2007, we reported that DHS had not fully defined activities, milestones, and costs for implementing the program; demonstrated how program activities would further the strategic goals and objectives of SBI; and reported on the costs incurred, activities, and progress made by the program in obtaining operational control of the border. Further, we reported that the program’s schedule contained a high level of concurrency among related tasks and activities, which introduced considerable risk. Accordingly, we recommended that DHS define explicit and measurable commitments relative to, among other things, program capabilities, schedules, and costs, and re-examine the level of concurrency in the schedule and adjust the acquisition strategy appropriately. We are currently reviewing DHS’s Fiscal Year 2010 SBI Expenditure Plan to, among other things, determine the status of DHS’s actions to address these recommendations.

In October 2007, we testified that DHS had fallen behind in implementing Project 28 due to software integration problems, although program officials stated at that time that Boeing was making progress in correcting the problems. Shortly thereafter, we testified that while DHS had accepted Project 28, it did not fully meet expectations. To benefit from this experience, program officials stated that they identified a number of lessons learned, including the need to increase input from Border Patrol agents and other users in SBInet design and development.

In September 2008, we reported that important aspects of SBInet were ambiguous and in a continued state of flux, making it unclear and uncertain what technological capabilities were to be delivered when. We concluded that the absence of clarity and stability in key aspects of SBInet impaired the ability of Congress to oversee the program and hold DHS accountable for results, and hampered DHS’s ability to measure program performance. As a result, we recommended that the SPO establish and baseline the specific program commitments, including the specific system functional and performance capabilities that are to be deployed, and when they were to be deployed.

Also, we reported that the SPO had not effectively performed key requirements definition and management practices. For example, it had not ensured that different levels of requirements were properly aligned, as evidenced by our analysis of a random probability sample of component requirements showing that a large percentage of them could not be traced to higher-level system and operational requirements. Also, some of SBInet’s operational requirements, which are the basis for all lower-level requirements, were found by an independent DHS review to be unaffordable and unverifiable, thus casting doubt on the quality of lower- level requirements that were derived from them. As a result of these limitations, we concluded that the risk of SBInet not meeting mission needs and performing as intended was increased, as were the chances of the program needing expensive and time-consuming system rework. We recommended that the SPO implement key requirements development and management practices to include (1) baselining requirements before system design and development efforts begin; (2) analyzing requirements prior to being baselined to ensure that that they are complete, achievable, and verifiable; and (3) tracing requirements to higher-level requirements, lower-level requirements, and test cases.

We also reported that SBInet testing was not being effectively managed. For example, the SPO had not tested the individual system components to be deployed to the initial deployment locations, even though the contractor had initiated integration testing of these components with other system components and subsystems. Further, while a test management strategy was drafted, it had not been finalized and approved, and it did not contain, among other things, a clear definition of testing roles and responsibilities; a high-level master schedule of SBInet test activities; or sufficient detail to effectively guide project-specific test planning, such as milestones and metrics for specific project testing. We concluded that without a structured and disciplined approach to testing, the risk that SBInet would not satisfy user needs and operational requirements, thus requiring system rework, was increased. We recommended that the SPO (1) develop and document test practices prior to the start of testing; (2) conduct appropriate component-level testing prior to integrating system components; and (3) approve a test management strategy that, at a minimum, includes a relevant testing schedule, establishes accountability for testing activities by clearly defining testing roles and responsibilities, and includes sufficient detail to allow for testing and oversight activities to be clearly understood and communicated to test stakeholders.

In light of these weaknesses and risks, we further recommended that (1) the risks associated with planned SBInet acquisition, development, testing, and deployment activities be immediately assessed and (2) the results, including proposed alternative courses of action for mitigating the risks, be provided to the CBP Commissioner and DHS’s senior leadership, as well as to the department’s congressional authorization and appropriations committees. DHS agreed with all but one of the recommendations in our September 2008 report. The status of DHS’s efforts to implement these recommendations is summarized later in this report and discussed in detail in appendix III.

In September 2009, we reported that SBInet had continued to experience delays. For example, deployment to the entire southwest border had slipped from early 2009 to 2016, and final acceptance of TUS-1 and AJO-1 had slipped from November 2009 and March 2010 to December 2009 and June 2010, respectively. We did not make additional SBInet recommendations at that time.

Most recently, we reported in January 2010 that SBInet testing was not being effectively managed. Specifically, while DHS’s approach to testing appropriately consisted of a series of progressively expansive developmental and operational test events, the test plans, cases, and procedures for the most recent test events were not defined in accordance with important elements of relevant guidance. For example, none of the plans adequately described testing risks and only two of the plans included quality assurance procedures for making changes to test plans during their execution. Further, a relatively small percentage of test cases for these events described the test inputs and the test environment (e.g., facilities and personnel to be used), both of which are essential to effective testing.

In addition, a large percentage of the test cases for these events were changed extemporaneously during execution. While some of the changes were minor, others were more significant, such as re-writing entire procedures and changing the mapping of requirements to test cases. Moreover, these changes to procedures were not made in accordance with documented quality assurance processes, but rather were based on an undocumented understanding that program officials said they established with the contractor. Compounding the number and significance of changes were questions raised by the SPO and a support contractor about the appropriateness of some changes. For example, the SPO wrote to the prime contractor that changes made to system qualification test cases and procedures appeared to be designed to pass the test instead of being designed to qualify the system.

Further, we reported that from March 2008 through July 2009, that about 1,300 SBInet defects had been found, with the number of new defects identified during this time generally increasing faster than the number being fixed—a trend that is not indicative of a system that is maturing and ready for deployment. While the full magnitude of these unresolved defects was unclear because the majority were not assigned a priority for resolution, some of the defects that had been found were significant. Although DHS reported that these defects had been resolved, they had nevertheless caused program delays, and related problems had surfaced that continued to impact the program’s schedule. Further, an early user assessment of SBInet had raised significant concerns about the performance of key system components and the system’s operational suitability.

In light of these weaknesses, we recommended that DHS (1) revise the program’s overall test plan to include (a) explicit criteria for assessing the quality of test documentation, including test plans and test cases, and (b) a process for analyzing, prioritizing, and resolving program defects; (2) ensure that test schedules, plans, cases, and procedures are adequately reviewed and approved consistent with the revised test plan; (3) ensure that sufficient time is provided for reviewing and approving test documents prior to beginning a given test event; and (4) triage the full inventory of unresolved system problems, including identified user concerns, and periodically report on their status to CBP and DHS leadership. DHS fully agreed with the last three recommendations and partially agreed with the first.

Block 1 Capabilities, Geographic Coverage, and Performance Standards Continue to Decrease

For Block 1, functional and performance capabilities and the number of geographic locations to which they are to be deployed have continued to decrease. We reported in September 2008 that the capabilities and deployment locations of SBInet were decreasing. Since that time, the number of component-level requirements to be deployed to TUS-1 and AJO-1 has decreased by about 32 percent. In addition, the number of sectors that the system is to be deployed to has been reduced from three to two, and the stringency of the system performance measures that the deployed system is to meet has been reduced. According to program officials, the decreases are due to poorly defined requirements and limitations in the capabilities of commercially available system components. The result will be a deployed and operational system that, like Project 28, does not live up to user expectations and provides less mission support than was envisioned.

Functional Capabilities Have Been Reduced

Since our September 2008 report, the number of requirements that Block 1 is to meet has dropped considerably. Specifically, in September 2008, DHS directed the SPO to identify the operational requirements to be allocated to Block 1. In response, 106 operational requirements were established, such as providing border surveillance, facilitating decision support and situational awareness, enabling communications, providing operational status and readiness metrics, and enabling system audits. Of the 106 requirements, 69 were to be included in the initial technology deployments planned for TUS-1 and AJO-1. The remaining 37 were to be addressed in future blocks.

To implement the 69 operational requirements, the SPO developed a system-level requirement specification and 12 component-level requirements specifications. More specifically, as part of CDR, which concluded in October 2008, the 69 operational requirements for TUS-1 and AJO-1 were associated with 97 system-level requirements. Also during CDR, the 97 system-level requirements were associated with 1,286 component-level requirements.

However, between October 2008 and September 2009, the number of component-level requirements was reduced from 1,286 to 880, or by about 32 percent. First, 281 requirements related to the specifications for three components—communications, network operations, and network security—were eliminated, leaving 1,005 baselined requirements.

Examples of the 281 requirements that were eliminated include the following: the failure in a single piece of hardware or software would not affect mission critical functions which include detection and resolution of border incursions; the failure of a Network Operations Center/Security Operations Center (NOC/SOC) workstation would not prevent the system from operating; and the failure of one network power supply would be compensated for by additional backup power supplies.

In addition, another 125 component-level requirements were granted “waivers” or “deviations,” further reducing the number of Block 1 requirements to be deployed to TUS-1 and AJO-1 to 880 (as of September 2009). For example, the unattended ground sensors were required to differentiate between human, vehicle, and animal targets. However, because the sensors that are to be deployed to TUS-1 and AJO-1 are only able to identify potential vehicles and are not able to differentiate between humans and animals, this requirement was deviated. Similarly, the radar was required to classify targets as humans or vehicles. However, the radar also cannot differentiate between classes of targets (e.g., humans and vehicles). As a result, the requirement in the radar specification was also deviated.

Figure 3 summarizes the roughly 32 percent drop in requirements that has occurred over the last 15 months.

According to program officials, component requirements were eliminated because they were either poorly written or duplicative of other requirements, or because the capabilities of commercially available products were limited. In addition, they attributed a significant number of eliminated requirements to a decision to not use a Boeing designed and developed network and instead to use an existing DHS network. To the SPO’s credit, this decision was made to align SBInet with DHS technical standards and to increase the use of COTS products.

Compounding this reduction in Block 1 requirements is the likelihood that further requirements deviations and waivers will be granted based on the results of an early user assessment of the system. According to the July 2009 assessment report, certain SBInet components did not meet requirements. For example: The daytime cameras were judged to be operationally ineffective over 5 kilometers for identifying humans, while the requirement is that the cameras be usable to 10 kilometers.

The laser range finder was determined to have an effective range of less than 2 kilometers, while the requirement is for the effective range to be 10 kilometers.

Program officials told us that many of the limitations found during the user assessment were previously known, and corrective actions were already under way or planned for future technology upgrades to address them. However, the officials also stated they plan to issue a waiver or deviation for the camera and the laser range finder to address the two problems discussed above. In addition, they stated that a previously known limitation of the range of the radar will also need to be addressed through a deviation. In this case, the radar is required to have a range of 20 kilometers, but testing shows a maximum range of 10 kilometers.

Geographic Coverage Has Been Reduced

Beyond the requirement reductions, the geographic locations to receive the initial SBInet capabilities have also been reduced. As of September 2008, the initial Block 1 deployment was to span three border patrol sectors: Tucson, Yuma, and El Paso—a total of 655 miles. According to program officials, deployment to these three areas was the expressed priority of the Border Patrol due to the high threat levels in these areas. However, the Acquisition Program Baseline, which was drafted in December 2008, states that initial deployment will be to just the Tucson and Yuma Sectors, which will cover only 387 miles.

According to program officials, deployment to the 268 miles of the El Paso Sector was dropped from the initial deployment in anticipation that the sector will instead receive the capabilities slated for the next SBInet increment (i.e., build). However, plans for the next increment have not been developed. According to the SBI Executive Director in December 2009, the SPO is re-evaluating where and when future deployments of SBInet will occur, and a date for when the revised deployment plans will be available has not been set.

Performance Capabilities Have Decreased

System performance measures define how well a system is to perform certain functions, and thus are important in ensuring that the system meets mission and user needs. According to program documentation, failure to meet a key performance parameter can limit the value of the system and render it unsuccessful. In November 2008, the SPO re- evaluated its existing SBInet key performance parameters and determined that SBInet must meet three such parameters: (1) the probability of detecting items of interest between the border and the control boundary; (2) the probability of correctly identifying items of interest as human, conveyance, or others; and (3) the operational availability of the system. According to program officials, subject matter experts and CBP staff concluded that these three were critical to determining whether the system successfully meets mission and user needs.

Associated with each parameter is a threshold for acceptable performance. In November 2008, the SPO re-evaluated the thresholds for its three key performance parameters, and it significantly relaxed each of the thresholds: The threshold for detecting items of interest dropped from 95 percent to 70 percent.

The threshold for identifying items of interest declined from 95 percent to 70 percent.

The threshold for operational availability decreased from 95 to 85 percent.

These threshold reductions significantly lower what constitutes acceptable system performance. For example, the system will meet its detection and identification performance requirements if it identifies 70 percent of the 70 percent of items that it detects, thus producing a 49 percent probability of identifying items of interest that cross the border. Furthermore, the reduction in operational availability means that the time that the system can be unavailable for use has gone from 18.25 days per year to 54.75 days per year—or from approximately 2.5 weeks to about 7 weeks per year, excluding downtime for planned maintenance.

The SBI Executive Director attributed the performance reductions to program officials’ limited understanding of needed operational capabilities at the time the parameters and thresholds were set. The director further stated that once Block 1 has been deployed and Border Patrol personnel gain experience operating it, decisions will be made as to what additional changes to make to the key performance parameters and associated thresholds. Until then, system performance relative to identifying items of interest and operational availability will remain as described above, which program officials agreed fall short of expectations.

A Reliable Schedule for Completing Block 1 Has Not Been Developed

The success of a large-scale system acquisition program like SBInet depends in part on having a reliable schedule of when the program’s set of work activities and milestone events will occur, how long they will take, and how they are related to one another. Among other things, a reliable schedule provides a road map for systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability. Our research has identified nine best practices associated with developing and maintaining a reliable schedule. These are (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying reasonable float between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations. To be considered reliable, a schedule should meet all nine practices.

The August 2009 SBInet integrated master schedule, which was the most current version available for our review, is not reliable because it substantially complies with only two of the nine key schedule estimating practices and it does not comply with, or only partially or minimally complies with, the remaining seven practices (see table 4 for a summary and app. IV for the detailed results of our analysis of the extent to which the schedule meets each of the nine practices).

Examples of practices that were either substantially, partially, minimally, or not met are provided below. Without having a reliable schedule, it is unlikely that actual program execution will track to plans, thus increasing the risk of cost, schedule, and performance shortfalls.

Capturing all activities: The schedule does not capture all activities as defined in the program’s work breakdown structure or integrated master plan. First, 57 percent of the activities listed in the work breakdown structure (71 of 125) and 67 percent of the activities listed in the integrated master plan (46 of 69) were not in the integrated master schedule. For example, the schedule is missing efforts associated with systems engineering, sensor towers, logistics, system test and evaluation, operations support, and program management. Second, the schedule does not include key activities to be performed by the government. For example, while the schedule shows the final activity in the government process for obtaining an environmental permit in order to construct towers, it does not include the related government activities needed to obtain the permit.

Sequencing all activities: The schedule identifies virtually all of the predecessor and successor activities. Specifically, only 9 of 1,512 activities (less than 1 percent) were missing predecessor links. Further, only 21 of 1,512 activities (about 1 percent) had improper predecessor and successor links. While the number of unlinked activities is very small, not linking a given activity can cause problems because changes to the durations of these activities will not accurately change the dates for related activities. More importantly, 403 of 1,512 activities (about 27 percent) are constrained by “start no earlier than” dates, which is significant because it means that these activities are not allowed to start earlier, even if their respective predecessor activities have been completed.

Establishing the critical path for all activities: The schedule does not reflect a valid critical path for several reasons. First, and as noted above, it is missing government and contractor activities, and is thus not complete. Second, as mentioned above, the schedule is missing some predecessor links, and improperly establishes other predecessor and successor links. Problems with the critical path were recognized by the Defense Contract Management Agency as early as November 2008, when it reported that the contractor could not develop a true critical path that incorporates all program elements.

Conducting a schedule risk analysis: An analysis of the schedule’s vulnerability to slippages in the completion of tasks has not been performed. Further, program officials described the schedule as not sufficiently stable to benefit from a risk analysis.

Reasons that these practices were not fully met vary and include the program’s use of Boeing to develop and maintain the integrated master schedule, even though Boeing’s processes and tools do not allow it to include in the schedule work that it does not have under contract to perform, as well as the constantly changing nature of the work to be performed. Without a reliable schedule that includes all activities necessary to complete Block 1, the SPO cannot accurately determine the amount of time required to complete Block 1, and it does not have an adequate basis for guiding the program’s execution and measuring progress, thus reducing the likelihood of meeting the program’s completion dates.

Collectively, the weaknesses in meeting the nine key practices for the program’s integrated master schedule increase the risk of schedule slippages and related cost overruns and make meaningful measurement and oversight of program status and progress, as well as accountability for results, difficult to achieve. In the case of Block 1, this risk has continued to be realized. For example, the dates presented at the December 2008 to November 2009 monthly program review meetings for government acceptance of Block 1 at TUS-1 and AJO-1 showed a pattern of delays, with TUS-1 and AJO-1 acceptance slipping by 4 months and 7 months, respectively. (See fig. 4.) Moreover, these slipped dates have not been met, and the SBI Executive Director told us in December 2009 that when Block 1 will be accepted and operational continues to change and remains uncertain. As of February 2010, TUS-1 and AJO-1 were proposed to be accepted in September 2010 and November 2010, respectively; however, this proposed schedule has yet to be approved by CBP.

Cost-Effectiveness of Block 1 Has Not Been Demonstrated

As we have previously reported, the decision to invest in any system or major system increment should be based on reliable estimates of costs and meaningful forecasts of quantifiable and qualitative benefits over the system’s useful life. For Block 1, DHS does not have a complete and current life cycle cost estimate. Moreover, it has not projected the mission benefits expected to accrue from Block 1 over the same life cycle. According to program officials, it is premature to project such benefits given the uncertainties surrounding the role that Block 1 will ultimately play in overall border control operations. Without a meaningful understanding of SBInet costs and benefits, DHS lacks an adequate basis for knowing whether the initial system solution on which it plans to spend at least $1.3 billion is cost-effective. Moreover, DHS and congressional decision makers continue to lack a basis for deciding what investment in SBInet beyond this initial capability is economically prudent.

Life Cycle Costs Have Not Been Reliably Estimated

A reliable cost estimate is critical to successfully delivering large-scale information technology (IT) systems, like SBInet, as well as major system increments, like Block 1. Such an estimate provides the basis for informed investment decision making, realistic budget formulation, meaningful progress measurement, and accountability for results. According to the Office of Management and Budget (OMB), federal agencies must maintain current and well-documented estimates of program costs, and these estimates must encompass the program’s full life cycle. Among o things, OMB states that a reliable life cycle cost estimate is critical to the capital planning and investment control process. Without such an estimate, agencies are at increased risk of making poorly informed investment decisions and securing insufficient resources to effectively execute defined program plans and schedules, and thus experiencing program cost, schedule, and performance shortfalls.

Our research has identified a number of practices that form the basis of effective program cost estimating. These practices are aligned with four characteristics of a reliable cost estimate. To be reliable, a cost estimate should possess all four characteristics, each of which is summarized below. (See app. V for the key practices associated with each characteristic, including a description of each practice and our analysis of the extent to which the SBInet cost estimate meets each practice.)

Comprehensive: The cost estimate should include all government and contractor costs over the program’s full life cycle, from program inception through design, development, deployment, and operation and maintenance to retirement. It should also provide sufficient detail to ensure that cost elements are neither omitted nor double counted, and it should document all cost-influencing ground rules and assumptions.

Well-documented: The cost estimate should capture in writing things such as the source and significance of the data used, the calculations performed and their results, and the rationale for choosing a particular estimating method or reference. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources. Finally, the cost estimate should be reviewed and accepted by management to demonstrate confidence in the estimating process and the estimate.

Accurate: The cost estimate should not be overly conservative or optimistic, and should be, among other things, based on an assessment of most likely costs, adjusted properly for inflation, and validated against an independent cost estimate. In addition, the estimate should be updated regularly to reflect material changes in the program and actual cost experience on the program. Further, steps should be taken to minimize mathematical mistakes and their significance and to ground the estimate in documented assumptions and a historical record of actual cost and schedule experiences on comparable programs.

Credible: The cost estimate should discuss any limitations in the analysis due to uncertainty or biases surrounding the data and assumptions. Major assumptions should be varied and other outcomes computed to determine how sensitive the estimate is to changes in the assumptions. Risk and uncertainty inherent in the estimate should be assessed and disclosed. Further, the estimate should be properly verified by, for example, comparing the results with one or more independent cost estimates.

The SPO’s Block 1 life cycle cost estimate includes the costs to complete those portions of Block 1 that are to be deployed to the Tucson and Yuma Sectors, which together cover about 387 miles of the southwest border (53 miles associated with both TUS-1 and AJO-1, which are in the Tucson Sector, as well as an additional 209 miles in the Tucson Sector and 125 miles in the Yuma Sector). More specifically, this estimate, which is dated December 2008, shows the minimum cost to acquire and deploy Block 1 to the Tucson and Yuma Sectors to be $758 million, with another $544 million to operate and maintain this initial capability, for a total of about $1.3 billion.

However, this Block 1 cost estimate is not reliable because it does not sufficiently possess any of the above four characteristics. Specifically: The estimate is not comprehensive because it does not include all relevant costs, such as support contractor costs and costs associated with system and software design, development, and testing activities that were incurred prior to December 2008. Moreover, it includes only 1 year of operations and maintenance costs rather than these costs over the expected life of the system. Further, the estimate does not document and assess the risks associated with all ground rules and assumptions, such as known budget constraints, staff and schedule variations, and technology maturity.

The estimate is not well-documented because, among other things, the sources and significance of key data have not been captured and the quality of key data, such as historical costs and actual cost reports, is limited. For example, instead of identifying and relying on historical costs from similar programs, the estimate was based, in part, on engineering judgment. Further, the calculations performed and their results, while largely documented, did not document contingency reserves and the associated confidence level for the risk-adjusted cost estimate. Also, as noted above, assumptions integral to the estimate, such as those for budget constraints, and staff and schedule variances, were not documented.

The estimate is not accurate because it was not, for example, validated against an independent cost estimate. Further, it has not been updated to reflect material program changes since the estimate was developed. For example, the estimate does not reflect development and testing activities that were added since the estimate was approved to correct problems discovered during testing. Further, the estimate has not been updated with actual cost data available from the contractor.

The estimate is not credible because its inherent risk and uncertainty were not adequately assessed, and thus the estimate does not address limitations associated with the assumptions used to create it. For example, the risks associated with software development were not examined, even though such risks were known to exist. In fact, the only risks considered were those associated with uncertainty in labor rates and hardware costs, and instead of being based on historical quantitative analyses, these risks were expressed by assigning them arbitrary positive or negative percentages. In addition, and for the reasons mentioned above, the estimate did not specify contingency reserve amounts to mitigate known risks, and an independent cost estimate was not used to verify the estimate.

Program officials attributed these limitations in the cost estimate’s comprehensiveness, documentation, accuracy, and credibility to a range of factors, including competing program office priorities and the department’s limited cost estimating capabilities. For example, program officials stated that the DHS Cost Analysis Division did not prepare an independent estimate because it did not have, among other things, the people and tools needed to do so. In this regard, this division reports that as of July 2009, DHS only had eight cost estimators (six in headquarters and two in program offices) for departmentwide needs.

Because the estimate does not adequately display these four characteristics, it does not provide a reliable picture of Block 1’s life cycle costs. As a result, DHS does not have complete information on which to base informed investment decision making, understand system affordability, and develop justifiable budget requests. Moreover, the Block 1 cost estimate does not provide a meaningful standard against which to measure cost performance, is likely to show large cost overruns, and does not provide a good basis for informing future cost estimates.

Expected Mission Benefits Have Yet to Be Adequately Defined

The Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to ensure that IT investments actually produce tangible, observable improvements in mission performance. As we have previously reported,to accomplish this, benefits that are expected to accrue from investments need to be forecast and their actual accrual needs to be measured.

In the case of Block 1, however, expected mission benefits have not been defined and measured. For example, while program officials told us that system benefits are documented in the SBInet Mission Need Statement dated October 2006, this document does not include either quantifiable or qualitative benefits. Rather, it provides general statements such as “the lack of a program such as SBInet increases the risks of terrorist threats and other illegal activities.”

Congress recognized the importance of having a meaningful understanding of SBInet’s value proposition when it required DHS in 2008 to provide in its Border Security, Fencing, Infrastructure, and Technology Fiscal Year 2009 Expenditure Plan a description of how the department’s planned expenditure of funds would be linked to expected SBI mission benefits and outcomes. However, we reported that the plan DHS submitted only described links among planned activities, expenditures, and outputs. It did not link these to outcomes associated with improving operational control of the border. More recently, we reported that while SBI technology and physical infrastructure, along with increases in Border Patrol personnel, are intended to allow DHS to gain effective control of U.S. borders, CBP’s measures of effective control are limited. Thus, we recommended that CBP conduct a cost-effectiveness evaluation of the SBI tactical infrastructure’s impact on effective control of the border, and DHS agreed with this recommendation. Further, program officials noted that uncertainty about SBInet’s role in and contribution to effective control of the border makes it difficult to forecast SBInet benefits. Rather, they said that operational experience with Block 1 is first needed in order to estimate such benefits.

While we recognize the value of operationally evaluating an early, prototypical version of a system in order to better understand, among other things, its mission impact, and thus to better inform investment decisions, we question the basis for spending in excess of a billion dollars to gain this operational experience. Without a meaningful understanding and disclosure of SBInet benefits, to include the extent to which expected mission benefits are known and unknown, DHS did not have the necessary basis for justifying and making informed decisions about its sizeable investment in Block 1, as well as for measuring the extent to which the deployed Block 1 will actually deliver mission value commensurate with costs.

Block 1 Has Not Been Managed in Accordance with Key Life Cycle Management Processes

Successful management of large IT programs, like SBInet, depends in large part on having clearly defined and consistently applied life cycle management processes. Our evaluations and research show that applying system life cycle management rigor and discipline increases the likelihood of delivering expected capabilities on time and within budget. In other words, the quality of a system is greatly influenced by the quality of the processes used to manage it.

To the SPO’s credit, it has defined key life cycle management processes that are largely consistent with relevant guidance and associated best practices. However, it has not effectively implemented these processes. Specifically, it has not consistently followed its systems engineering plan, requirements development and management plan, and risk management approach. Reasons cited by program officials for not implementing these processes include the decision by program officials to rely on contract task order requirements that were developed prior to the systems engineering plan, and competing SPO priorities, including meeting an aggressive deployment schedule. Until the SPO consistently implements these processes, it will remain challenged in its ability to successfully deliver SBInet.

Key System Life Cycle Management Activities Have Not Been Consistently Performed

Each of the steps in a life cycle management approach serves an important purpose and has inherent dependencies with one or more other steps. In addition, the steps used in the approach should be clearly defined and repeatable. Thus, if a life cycle management step is omitted or not performed effectively, later steps can be affected, potentially resulting in costly and time-consuming rework. For example, a system can be effectively tested to determine whether it meets requirements only if these requirements have been completely and correctly defined. To the extent that interdependent life cycle management steps or activities are not effectively performed, or are performed concurrently, a program will be at increased risk of cost, schedule, and performance shortfalls.

The SPO’s Systems Engineering Plan documents its life cycle management approach for SBInet definition, development, testing, deployment, and sustainment. As noted earlier, we reported in September 2008 on a number of weaknesses in the SBInet life cycle management approach and made recommendations to improve it.In response, the SPO revised its Systems Engineering Plan in November 2008, and to its credit, the revised plan is largely consistent with DHS and other relevant guidance. For example, it defines a number of key life cycle milestone or “gate” reviews that are important in managing the program, such as initial planning reviews, requirements reviews, system design reviews, and test reviews. In addition, the revised plan requires most of the key artifacts and program documents that DHS guidance identified as important to each gate review, such as a concept of operations, an operational requirements document, a deployment plan, a risk management plan, a life cycle cost estimate, requirements documentation, and test plans. To illustrate, the plan identifies CDR as the important milestone event where a design baseline is to be established, requirements traceability is to be demonstrated, and verification and testing plans are to be in place.

However, the Systems Engineering Plan does not address the content of the key artifacts that it requires. For example, it does not provide a sample document or content template for the concept of operations, the operational requirements document, or the deployment plan. As a result, the likelihood of the developers and reviewers of these artifacts sharing and applying a consistent and repeatable understanding of their content is minimized, thus increasing the risk that they will require costly and time- consuming rework. As we recently reported, the absence of content guidance or criteria for assessing the quality of the prime contractor’s test- related deliverables was a primary reason that limitations were found in test plans.

Beyond the content of the Systems Engineering Plan, the SPO has not consistently implemented key system life cycle management activities for Block 1 that are identified by the plan. For example, the following artifacts were not reviewed or considered during the CDR that concluded in October 2008: Security Test Plan, which describes the process for assessing the robustness of the system’s security capabilities (e.g., physical facilities, hardware, software, and communications) in light of their vulnerabilities.

Quality Plan, which documents the process for verifying that the contractor deliverables satisfy contractual requirements and meet or exceed quality standards.

Test Plan, which describes the overall process for the test and evaluation, including the development of detailed test event plans, test procedure instructions, data collection methods, and evaluation reports.

Block Training Plan, which outlines the objectives, strategy, and curriculum for training that are specific to each block, including the activities needed to support the development of training materials, coordination of training schedules, and reservation of personnel and facilities.

Block Maintenance Plan, which lays out the policies and concepts to be used to maintain the operational availability of hardware and software.

To the SPO’s credit, it reviewed and considered all but one of the key artifacts for the TUS-1 Deployment Readiness Review that concluded in April 2009. The omitted artifact was the Site Specific Training Plan, which outlines the objectives, strategy, and curriculum for training that are specific to each geographic site, including the activities needed to support the development of training materials, coordination of training schedules, and reservation of personnel and facilities. According to program officials, even though the Systems Engineering Plan cites the training plan as integral to the Deployment Readiness Review, this training plan is to be reviewed as part of a later milestone review.

Program officials stated that a reason that the artifacts were omitted is that they have yet to begin implementing the Systems Engineering Plan. Instead, they have, for example, enforced the CDR requirements in the System Task Order that Boeing was contractually required to follow. To address this, they added that the SPO intends to bring the task orders into alignment with the Systems Engineering Plan, but they did not specify when this would occur. As a result, key milestone reviews and decisions have not always benefited from life cycle management documentation that the SPO has determined to be relevant and important to these milestone events. More specifically, the Systems Engineering Plan states that the gate reviews are intended to identify and address problems early and thus minimize future costs and avoid subsequent operational issues. By not fully informing these gate reviews and associated decisions with key life cycle management documentation, the risk of Block 1 design and deployment problems is increased, as is the likelihood of expensive and time-consuming system rework.

Key Block 1 Requirements Have Not Been Adequately Developed and Managed

Well-defined and managed requirements are essential to successfully acquiring large-scale systems, like SBInet. According to relevant guidance,effective requirements development and management include establishing a baseline set of requirements that are complete, unambiguous, and testable. It also includes ensuring that system-level requirements are traceable backwards to higher-level operational requirements and forward to design requirements and the methods used to verify that they are met. Among other things, this guidance states that such traceability should be used to verify that higher-level requirements have been met by first verifying that the corresponding lower-level requirements have been satisfied.

However, not all Block 1 component requirements were sufficiently defined at the time that they were baselined, and operational requirements continue to be unclear and unverifiable. In addition, while requirements are now largely traceable backwards to operational requirements and forward to design requirements and verification methods, this traceability has not been used until recently to verify that higher-level requirements have been satisfied. Program officials attributed these limitations to competing SPO priorities, including aggressive schedule demands. Without ensuring that requirements are adequately defined and managed, the risks of Block 1 not performing as intended, not meeting user needs, and costing more and taking longer than necessary to complete are increased.

Not All Requirements Were Adequately Baselined

The SBInet Requirements Development and Management Plan states that a baseline set of requirements should be established by the time of the CDR and that these requirements should be complete, unambiguous, and testable. Further, the program’s Systems Engineering Plan states that the CDR is intended to establish the final allocated requirements baseline and ensure that system development, integration, and testing can begin.

To the SPO’s credit, it established a baseline set of requirements for the TUS-1 and AJO-1 system deployments at CDR. However, the baseline requirements associated with the NOC/SOC were not adequately defined at this time, as evidenced by the fact that they were significantly changed 2 months later. Specifically, about 33 percent of the component-level requirements and 43 percent of the design specifications for NOC/SOC were eliminated from the Block 1 design after CDR. Program officials attributed these changes to the NOC/SOC requirements to (1) requirements that were duplicative of another specification, and thus were redundant; (2) requirements that were poorly written, and thus did not accurately describe needs; and (3) requirements that related to the security of a system that SBInet would not interface with, and thus were unnecessary.

According to program officials, the NOC/SOC was a late addition to the program, and at the time of CDR, the component’s requirements were known to need additional work. Further, they stated that while the requirements were not adequately baselined at the time of CDR, the interface requirements were understood well enough to begin system development.

Without properly baselined requirements, system testing challenges are likely to occur, and the risk of system performance shortfalls, and thus cost and schedule problems, are increased. In this regard, we recently reported that NOC/SOC testing was hampered by incorrect mapping of requirements to test cases, failure to test all of the requirements, and significant changes to test cases made during the testing events. This occurred in part because ambiguities in requirements caused testers to rewrite test steps during execution based on interpretations of what they thought the requirements meant, and they required the SPO to conduct multiple events to test NOC/SOC requirements.

Block 1 Operational Requirements Remain Poorly Defined

According to the SBInet Requirements Development and Management Plan, requirements should be achievable, verifiable, unambiguous, and complete. To ensure this, the plan contains a checklist that is to be used in verifying that each requirement possesses these characteristics.

However, not all of the SBInet operational requirements that pertain to Block 1 possess these characteristics. Specifically, a November 2007 DHS assessmentdetermined that 19 operational requirements, which form the basis for the lower-level requirements used to design and build the system, were not complete, achievable, verifiable, or affordable. Further, our analysis of the 12 Block 1 requirements that are included in these 19 operational requirements shows that they have not been changed to respond to the DHS findings. According to the assessment, 6 of the 12 were unaffordable and unverifiable, and the other 6 were incomplete. Examples of these requirements and DHS’s assessment follow: A requirement that the system should provide for complete coverage of the border was determined to be unverifiable and unaffordable because defining what complete coverage meant was too difficult and ensuring complete coverage, given the varied and difficult terrain along the border, was cost prohibitive.

A requirement that the system should be able to detect and identify multiple simultaneous events with different individuals or groups was determined to be incomplete because the requirement did not specify the number of events to be included, the scope of the area to be covered, and the system components to be involved.

As we have previously reported,these limitations in the operational requirements affect the quality of system, component, and software requirements. This is significant because, as of September 2009, these 12 operational requirements were associated with 16 system-level requirements, which were associated with 152 component-level requirements, or approximately 15 percent of the total number of component-level requirements. According to program officials, these requirements were not updated because the SPO planned to resolve the problems through the testing process. However, we recently reported that requirements limitations actually contributed to testing challenges. Specifically, we reported that about 71 percent of combined system qualification and component qualification test cases had to be rewritten extemporaneously during test execution. According to program officials, this was partly due to ambiguities in requirements, which led to differing opinions among the program and contractor staff about what was required to effectively demonstrate that the requirements were met.

Further, program officials stated that a number of requirements have been granted deviations or waivers because they were poorly written. For example: A requirement for camera equipment to “conform to the capabilities and limitations of the users to operate and maintain it in its operational environment and not exceed user capabilities” was determined to be subjective and unquantifiable and thus was waived.

A requirement for the tower design to accommodate the future integration of components “without causing impact on cost, schedule, and/or technical performance” was determined to have no specific criteria to objectively demonstrate closure decision and thus was also waived.

As a result of these deviations and waivers, the system capabilities that are to be delivered as part of Block 1 will be less than originally envisioned.

Requirements Traceability Has Improved, but Until Recently Has Not Been Used To Determine If Operational Requirements Were Met

Consistent with relevant guidance,the SBInet Requirements Development and Management Plan provides for maintaining bidirectional traceability from high-level operational requirements through detailed low- level requirements to test plans. More specifically, it states that operational requirements should trace to system requirements, which in turn should trace to component requirements that trace to design requirements, which further trace to v erification methods.

Since September 2008, the SPO has worked with Boeing to manually review each requirement and develop a bidirectional traceability matrix. Further, it has used this matrix to update the DOORS requirements database. Our analysis of the traceability of a random sample of Block 1 component-level requirements in the DOORS database shows that they are largely traceable backwards to operational requirements and forward to design requirements and verification methods. For example, we estimate that only 5 percent (with a 95 percent confidence interval between 1 and 14 percent) of a random sample of component requirements cannot be traced to the system requirements and then to the operational requirements. In addition, we estimate that 0 percent (with a 95 percent confidence interval between 0 and 5 percent) of the component requirements in the same sample do not trace to a verification method. (See table 5 for the results of our analysis along with the associated confidence intervals.)

By establishing this traceability, the SPO is better positioned to know the extent to which the acquired and deployed system can meet operational requirements.

However, the SPO has not used its requirements traceability in closing higher-level component requirements. According to relevant guidance,all lower-level requirements (i.e., children) should be closed in order to sufficiently demonstrate that the higher-level requirements (i.e., parents) have been met. Consistent with this guidance, the SBInet Requirements Development and Management Plan states that ensuring the traceability of requirements from children to their parents is an integral part of ensuring that testing is properly planned and conducted. However, 4 of 8 higher- level component requirements (parents) in the above cited random sample of system-level requirements were closed regardless of whether their corresponding lower-level design requirements (children) had been closed. According to program officials, this is because their standard practice in closing parent requirements, until recently, was to sometimes close them before their children were closed. Further, they said that this was consistent with their verification criteria for closing higher-level requirements, which did not require closure of the corresponding lower- level requirements. They also said that the reason parent verification criteria did not always reflect children verification criteria was that traceability was still being established when the verification criteria were developed and thus parent-child relationships were not always available to inform the closure criteria. Furthermore, they stated that schedule demands did not permit them to ensure that the verification criteria for requirements were aligned with the traceability information.

After we shared our findings on parent requirement closure with the SPO, officials stated that they had changed their approach and will no longer close parent requirements without ensuring that all of the children requirements have first been closed. However, they did not commit to reviewing previously closed parents to determine that all of the children were closed. Without fully ensuring traceability among requirements verification methods, the risks of delivering a system solution that does not fully meet user needs or perform as intended, and thus requires additional time and resources to deliver, are increased.

Key Risks Have Not Been Effectively Managed and Disclosed

Risk management is a continuous, forward-looking process that effectively anticipates and mitigates risks that may have a critical impact on a program’s success. In 2008, the SPO documented a risk management approach that largely complies with relevant guidance. However, it has not effectively implemented this approach for all risks. Moreover, available documentation does not demonstrate that significant risks were disclosed to DHS and congressional decision makers in a timely fashion, as we previously recommended and, while risk disclosure to DHS leadership has recently improved, not all risks have been formally captured and thus shared. As a result, the program will likely continue to experience actual cost, schedule, and performance shortfalls, and key decision makers will continue to be less than fully informed.

Risk Management Approach Has Been Adequately Defined

According to relevant guidance, effective risk management includes defining a process that, among other things, proactively identifies and analyzes risks on the basis of likelihood of occurrence and impact, assigns ownership, provides for mitigation, and monitors status. To the SPO’s credit, it has developed an approach for risk management that is largely consistent with this guidance. For example, the approach provides for continuously identifying risks throughout the program’s life cycle before they develop into actual problems, including suggested methods for doing so, such as conducting brainstorming sessions and interviewing subject matter experts; analyzing identified risks to determine their likelihood of occurring and assigning responsibility for risks; developing a risk mitigation plan, to include a set of discrete, measurable actions or events which, if successfully accomplished, can avoid or reduce the likelihood of occurrence or severity of impact of the risk; and executing and regularly monitoring risk mitigation plans to ensure that they are implemented and to allow for corrective actions if the desired results are not being achieved.

In February 2007, we reported that the program’s risk management approach was in the process of being established. Specifically, we noted that at that time the SPO had drafted a risk management plan, established a governance structure, developed a risk management database, and identified 30 risks. In April 2009, we reported that the DHS Chief Information Officer had certified that this approach provided for the regular identification, evaluation, mitigation, and monitoring of risks throughout the system life cycle, and that it provided for communicating high-risk conditions to DHS investment decision makers.

Risk Management Approach Has Not Been Fully Implemented, although Improvements Are Under Way

The SPO has not adhered to key aspects of its defined process for managing program risks. In particular, the program’s risk management repository, which is the tool used for capturing and tracking risks and their mitigation, has not included key risks that have been identified by stakeholders. For example, our analysis of reports from the repository showing all open and closed risks from April 2006 to September 2009 shows that the following program risks that have been identified by us and others were not captured in the repository: program cost and schedule risks briefed by the SPO to senior SBInet officials in January 2009, such as unplanned and unauthorized work impacting the credibility of the program cost data, and program costs and schedule plans lacking traceability; program schedule and cost estimate risks identified by the Defense Contract Management Agency prior to March 2009, such as contractor- provided documentation not permitting adequate assessment of critical path accuracy, and cost projections not including all applicable elements and thus lacking credibility; and the risk of the SPO’s heavy reliance on contractors, reported by the DHS Office of Inspector General in June 2009.

In addition, the SBI Executive Director told us that the program faces a number of other risks, all but one of which were also not in the repository. These include the lack of well-defined acquisition management processes, staff with the appropriate acquisition expertise, and agreement on key system performance parameters. According to program officials, some of these risks are not in the repository because Boeing is responsible for operating and maintaining the repository, and the specifics surrounding the risks and their mitigation are considered acquisition sensitive, meaning that they should not be shared with Boeing. In this regard, the officials acknowledged that the SPO needs a risk database independent of the contractor to manage these acquisition-sensitive risks.

Further, the Risk Manager identified other limitations that have hindered the SPO’s risk management efforts, along with recent actions intended to address them. For example: Risk review meetings were only being held once a month, which was resulting in lost opportunities to mitigate risks that were to be realized as actual problems within 30 days. As a result, the frequency of these meetings has been increased to twice a month.

Risk information provided to senior SBI managers at monthly Joint Program Management Review Meetingswas not sufficiently detailed, and thus has been expanded.

Changes were being made to the risk management repository by contractor staff without sufficient justification and without the approval of the Joint Risk Review Board. For example, program officials cited an instance in which a risk’s severity was changed from medium to high and no board member knew the reason for the change. As a result, the number of contractor staff authorized to modify data in the repository was reduced.

The repository did not include all requisite information for all identified risks. For example, some risks were missing the rationale for the likelihood of occurrence and the potential impact. As a result, the Joint Risk Review Board has adopted a policy of not accepting risks that are missing requisite information.

According to the Risk Manager, competing program priorities have resulted in insufficient resources devoted to risk management activities, which has contributed to the state of the SPO’s risk management efforts. However, he added that the SPO is taking steps to improve risk management by revising risk management guidance, implementing a CBP- approved database tool for managing government-only risks, and increasing risk management training and oversight.

Until the program’s risk management is strengthened and effectively implemented, the program will continue to be challenged in its ability to forestall cost, schedule, and performance problems.

Risks Have Not Been Fully Disclosed, but Improvement Has Recently Occurred

As noted earlier, we recommended in September 2008 that the SPO assess SBInet risks and that the results of these assessments, along with alternative courses of action to address them, be provided to DHS leadership and congressional committees. According to program officials, shortly after receiving our draft report they briefed the DHS Acquisition Review Board on, among other things, SBInet risks. However, the briefing slides used for this meeting do not identify individual risks. Instead, the briefing contains one slide that only identifies “contributing factors” to changes in the program’s schedule, including a reallocation SBInet funding to SBI physical infrastructure, concurrencies and delays that have occurred in testing, and the need for environmental studies. The slides do not identify risks and alternative courses of action to address or mitigate them.

In addition, program officials told us that they briefed congressional committees during the fall of 2008 on the program’s status, which they said included disclosure of program risks. However, they did not have any documentation of these briefings to show which committees were briefed, when the briefings occurred, who was present, and what was discussed and disclosed. Further, House Committee on Homeland Security staff stated that while program officials briefed them following our September 2008 report, specific program risks were not disclosed. As a result, it does not appear that either DHS or congressional stakeholders received timely information on risks facing the program at a crucial juncture in its life cycle.

To the SPO’s credit, it has recently improved its disclosure of risks facing the program. In particular, the SBI Executive Director briefed the DHS Chief Information Officer in November 2009 on specific program risks. However, this briefing states that the risks presented were the Block 1 risks as captured in the contractor’s risk repository and that additional risks have not yet been formalized (see above discussion about repository limitations). Until all key risks are formally managed and regularly disclosed to department and congressional stakeholders, informed SBInet investment decision making will be constrained.

DHS Has Yet to Implement GAO’s Recent SBInet Recommendations

As noted earlier, we reported on a number of SBInet program management weaknesses in September 2008, and we concluded that these weaknesses introduced considerable risk that the program would not meet expectations and would require time-consuming and expensive rework. In summary, these problems included a lack of clarity and certainty surrounding what technological capabilities would be delivered when, and a lack of rigor and discipline around requirements definition and management and test management. To address these problems and thereby reduce the program’s exposure to cost, schedule, and performance risks, we made eight recommendations. DHS concurred with seven of the recommendations and disagreed with one aspect of the remaining one.

In summary, the department has not implemented two of the recommendations and has partially implemented the remaining six. See table 6 for a summary and appendix III for a detailed discussion of the status of each recommendation.

Conclusions

DHS has yet to demonstrate that its proposed SBInet solution is a cost- effective course of action, and thus whether the considerable time and money being invested to acquire and deploy it is a wise and prudent use of limited resources. Given that the magnitude of the initial investment in SBInet spans more than 3 years of effort and totals hundreds of millions of dollars, coupled with the fact that the scope of the initial system’s capabilities and areas of deployment have continued to shrink, the program is fraught with risk and uncertainty. As a result, the time is now for DHS to thoughtfully reconsider its proposed SBInet solution, and in doing so, to explore ways to both limit its near-term investment in an initial set of operational capabilities and develop and share with congressional decision makers reliable projections of the relative costs and benefits of longer-term alternatives for meeting the mission goals and outcomes that SBInet is intended to advance, or reasons why such information is not available and the uncertainty and risks associated with not having it.

Compounding the risks and uncertainty surrounding whether the department is pursuing the right course of action are a number of system life cycle management concerns, including limitations in the integrated master schedule; shortcomings in the documentation available to inform key milestone decisions; and weaknesses in how requirements have been developed and managed, risks have been managed, and tests have been conducted. Collectively, these concerns mean that the program is not employing the kind of acquisition management rigor and discipline needed to reasonably ensure that proposed system capabilities and benefits will be delivered on time and on budget.

Because of SBInet’s decreased scope, uncertain timing, unclear costs relative to benefits, and limited life cycle management discipline and rigor, in combination with its size and mission importance, the program represents a risky undertaking. To minimize the program’s exposure to risk, it is imperative for DHS to move swiftly to first ensure that SBInet, as proposed, is the right course of action for meeting its stated border security and immigration management goals and outcomes, and once this is established, for it to move with equal diligence to ensure that it is being managed the right way. To this end, our prior recommendations to DHS relative to SBInet provide for strengthening a number of life cycle management processes, including requirements development and management and test management. Accordingly, we are not making additional recommendations that focus on these processes at this time.

Recommendations for Executive Action

To address the considerable risks and uncertainties facing DHS on its SBInet program, we are making 12 recommendations. Specifically, we recommend that the Secretary of Homeland Security direct the Commissioner of U.S. Customs and Border Protection to limit future investment in the program to only work that meets one or both of the following two conditions: (1) is already under contract and supports deployment, acceptance, and operational evaluation of only those Block 1 capabilities (functions and performance levels) that are currently targeted for TUS-1 and AJO-1; or (2) provides the analytical basis for informing a departmental decision as to what, if any, expanded investment in SBInet, both in terms of capabilities (functions and performance) and deployment locations, represents a prudent, responsible, and affordable use of resources for achieving the department’s border security and immigration management mission.

With respect to the first condition, we further recommend that the Secretary of Homeland Security direct the Commissioner of U.S. Customs and Border Protection to have the SBI Executive Director make it a program priority to ensure that the integrated master schedule for delivering Block 1 capabilities to TUS-1 and AJO-1 is revised to address the key schedule estimating practices discussed in this report; the currently defined Block 1 requirements, including key performance parameters, are independently validated as complete, verifiable, and affordable and any limitations found in the requirements are addressed; the Systems Engineering Plan is revised to include or reference documentation templates for key artifacts required at milestone gate reviews; all parent requirements that have been closed are supported by evidence of the closure of all corresponding and associated child requirements; and all significant risks facing the program are captured, mitigated, tracked, and periodically reported to DHS and congressional decision makers.

Also with respect to the first condition, we reiterate our prior recommendations, as stated in our September 2008 report,relative to establishing program commitments, implementing the Systems Engineering Plan, defining and managing requirements, and testing.

With respect to the second condition, we further recommend that the Secretary of Homeland Security direct the Commissioner of U.S. Customs and Border Protection to have the SBI Executive Director make it a program priority to ensure that a life cycle cost estimate for any incremental block of SBInet capabilities that is to include capabilities and cover locations beyond those associated with the TUS-1 and AJO-1 deployments is developed in a manner that reflects the four characteristics of a reliable estimate discussed in this report; a forecast of the qualitative and quantitative benefits to be derived from any such incremental block of SBInet over its useful life, or reasons why such forecasts are not currently possible, are developed and documented; the estimated life cycle costs and benefits and associated net present value of any such incremental block of SBInet capabilities, or reasons why such an economic analysis cannot be performed, are prepared and documented; and the results of these analyses, or the documented reasons why such analyses cannot be provided, are provided to the Commissioner of U.S. Customs and Border Protection and the DHS Acquisition Review Board.

Also with respect to this second condition, we recommend that the Secretary of Homeland Security direct the Deputy Secretary of Homeland Security, as the Chair of the DHS Acquisition Review Board, to (1) decide, in consultation with the board and Commissioner of U.S. Customs and Border Protection, what, if any, expanded investment in SBInet, both in terms of capabilities (functions and performance) and deployment locations, represents a prudent, responsible, and affordable use of resources for achieving the department’s border security and immigration management mission; and (2) report the decision, and the basis for it, to the department’s authorization and appropriations committees.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Director, Departmental GAO/Office of Inspector General Liaison, and reprinted in appendix II, DHS stated that it agreed with ten of our recommendations and partially agreed with the remaining two. In this regard, it described ongoing and planned actions to address each, and it provided milestones for completing these actions. In addition, DHS provided technical comments, which we have incorporated in the report as appropriate.

In agreeing with our first recommendation, however, DHS commented that the words “one of” were omitted before the two conditions contained in the recommendation. However, this interpretation is not correct. Rather, the intent of our recommendation is to limit future investment on the program to either of the conditions, meaning “one or both of.” Notwithstanding DHS’s interpretation, we believe that actions that it described to address this recommendation, which include freezing funding beyond the initial deployments to TUS-1 and AJO-1 until it completes a comprehensive reassessment of the program that includes an analysis of the cost and mission effectiveness of alternative technologies, is consistent with the intent of the recommendation. Nevertheless, we have slightly modified the recommendation to avoid any further confusion.

Regarding its partial agreement with our recommendation for revising the integrated master schedule in accordance with a range of best practices embodied in our cost and schedule estimating guide, DHS acknowledged the merits of employing these practices and stated that it is committed to adopting and deploying them. However, it added that the current contract structure limits its ability to fully implement all the practices prior to completing the TUS-1 and AJO-1 deployments. We understand that program facts and circumstances create practical limitations associated with some of the practices, and believe that DHS’s planned actions are consistent with the intent of our recommendation.

Regarding its partial agreement with our recommendation that reiterated a number of the recommendations that we made in a prior report, DHS stated that, while these prior recommendations reflect program management best practices and it continues to make incremental improvements to address each, the scope of the program had narrowed since these recommendations were made. As a result, DHS stated that these prior recommendations were not fully applicable until and unless a decision was made to move the program forward and conduct future deployments beyond TUS-1 and AJO-1. We acknowledge that the facts and circumstances surrounding the program have recently changed and that these changes impact the nature and timing of actions appropriate for implementing them. Moreover, we believe that DHS’s planned actions are consistent with the intent of our recommendation.

DHS also commented that it believed that it had implemented two of our recommendations and that these recommendations should be closed. Because closure of our recommendations requires evidentiary validation of described actions, and because many of the actions that DHS described were planned rather than completed, we are not closing any of our recommendations at this time. As part of our recurring review of the status of all of our open recommendations, we will determine if and when the recommendations have been satisfied and thus can be closed.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees and other parties. We will also send copies to the Secretary of Homeland Security, the Commissioner of the U.S. Customs and Border Protection, and the Director of the Office of Management and Budget. In addition, this report will be available at no cost on the GAO Web site at http://www.gao.gov.

Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine the extent to which the Department of Homeland Security (DHS) has (1) defined the scope of its proposed Secure Border Initiative Network (SBInet) solution, (2) developed a reliable schedule for delivering this solution, (3) demonstrated the cost- effectiveness of this solution, (4) acquired this solution in accordance with key life cycle management processes, and (5) addressed our recent SBInet recommendations. To accomplish our objectives, we largely focused on the first increment of SBInet, known as Block 1.

To determine the extent to which DHS has defined the scope of its proposed system solution, we reviewed key program documentation related to the Block 1 functional and performance requirements and deployment locations, such as the SBInet Acquisition Program Baseline and related acquisition decision memorandums, the Operational Requirements Document, the Operational Requirements Document Elements Applicable to Block 1 System, the Requirements Traceability Matrix, the Requirements Verification Matrix, and the SBInet Block 1 User Assessment. In addition, we compared Block 1 requirements that were baselined in October 2008 as part of the Critical Design Review (CDR) to the Block 1 requirements as defined as of September 2009 to identify what, if any, changes had occurred, and we interviewed program officials as to the reasons for any changes. We also compared the locations, including the miles of border associated with these locations, that were to receive Block 1 as of September 2008 to the locations specified in the program’s March 2009 Acquisition Program Baseline to identify any changes, and we interviewed program officials as to the reasons for any changes. Further, we compared the key performance parameters listed in the Operational Requirements Document, dated March 2007, to the key performance parameters in the program’s Acquisition Program Baseline dated March 2009.

To determine the extent to which DHS has developed a reliable schedule for its proposed system solution, we analyzed the SBInet integrated master schedule as of June 2009 against the nine key schedule estimating practices in our Cost Estimating and Assessment Guide. In doing so, we used commercially available software tools to determine whether it, for example, included all critical activities, a logical sequence of activities, and reasonable activity durations. Further, we observed a demonstration of the schedule in June 2009 provided by contractor officials responsible for maintaining the schedule and program officials responsible for overseeing the contractor. In July 2009, we observed a demonstration of the program office’s efforts to reconcile the version of the integrated master schedule that is exported for the government’s use with the version of the schedule that the prime contractor uses to manage the program. During this demonstration, we discussed some of our concerns regarding the integrated master schedule with program officials and we inquired about deviations from some of the key practices. Subsequently, the program office provided us with a revised version of the integrated master schedule as of August 2009, which we analyzed. In doing so, we repeated the above described steps. Further, we characterized the extent to which the revised schedule met each of the practices as either Not Met, Minimally Met, Partially Met, Substantially Met, or Met. In addition, we analyzed changes in the scheduled Block 1 deployment dates presented at each of the monthly program reviews for the 1-year period beginning in December 2008 and ending in November 2009.

To determine the extent to which DHS has demonstrated the cost- effectiveness of the proposed solution, we evaluated the reliability of the Block 1 life cycle cost estimate and the definition of expected system benefits, both of which are addressed below.

Cost estimate: We first observed a demonstration of the cost model used to develop the estimate, which was provided by the contractor officials who are responsible for maintaining it and the program officials who are responsible for overseeing the contractor. We then analyzed the derivation of the cost estimate relative to 12 key practices associated with four characteristics of a reliable estimate. As defined in our Cost Estimating and Assessment Guide, these four characteristics are comprehensive, well-documented, accurate, and credible, and the practices address, for example, the methodologies, assumptions, and source data used. We also interviewed program officials responsible for the cost estimate about the estimate’s derivation. We then characterized the extent to which each of the four characteristics was met as either Not Met, Minimally Met, Partially Met, Substantially Met, or Met. To do so, we scored each of the 12 individual key practices associated with the four characteristics on a scale of 1-5 (Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5), and then averaged the individual practice scores associated with a given characteristic to determine the score for that characteristic.

Benefits: We interviewed program officials to identify any forecasts of qualitative and quantitative benefits that the system was to produce. In this regard, we were directed to the SBInet Mission Need Statement dated October 2006, which we analyzed. In addition, we reviewed our prior reports on the Secure Border Initiative (SBI), including a report on the SBI expenditure plan, which is a plan that DHS has been required by statute to submit to the House and Senate Appropriations Committees to, among other things, identify expected system benefits. We also interviewed program officials to determine the extent to which the system’s life cycle costs and expected benefits had been analyzed together to economically justify DHS’s proposed investment in SBInet.

To determine the extent to which DHS has acquired its proposed system solution in accordance with key life cycle management processes, we focused on three key processes: the system engineering approach, requirements development and management, and risk management, each of which is addressed below.

Systems engineering approach: We compared the program’s defined system engineering approach, as defined in the SBInet Systems Program Office’s (SPO) Systems Engineering Plan, to DHS and other relevant guidance. To determine the extent to which the defined systems engineering approach had been implemented, we focused on two major “gates” (i.e., life cycle milestone reviews)—the CDR and the Deployment Readiness Review. For each of these reviews, we compared the package of documentation prepared for and used during these reviews to the program’s defined system engineering approach as specified in the Systems Engineering Plan to determine what, if any, deviations existed. We also interviewed program officials as to the reason for any deviations.

Requirements development and management: We compared relevant requirements management documentation, such as the Requirements Development and Management Plan, the Requirements Management Plan, the Configuration and Data Management Plan, the Operational Requirements Document, the system-level requirements specification, and the component-level requirements specifications, to relevant requirements development and management guidance to identify an variances, focusing on the extent to which requirements were properly baselined, adequately defined, and fully traced. With respect to requirements baselining, we compared the component and system requirements as of September 2008, which were approved during the CDR that concluded in October 2008, to the component and system requirements as of November 2008, and identified the number and percentage of requirements changes. We also interviewed program officials as to the reasons for any changes. For requirements definition, weassessed the extent to which operational requirements that were identified as poorly defined in November 2007 had been clarified in the Operatio Requirements Document, Elements Applicable to Block 1 System, dat November 2008. In doing so, we focused on those operational requirements that are associated with Block 1. We also traced these Blo 1 operational requirements to the lower-level system requirements (i.e., system and component requirements) to determine how many of the lower-level requirements were associated with any unchanged operationa requirements. For requirements traceability, we randomly selected a sample of 60 requirements from 1,008 component requirements in the program’s requirements management tool, known as the Dynamic Object- Oriented Requirements System (DOORS), as of July 2009. Before doing so we reviewed the quality of the access controls for the database, and we interviewed program and contractor officials and received a DOOR tutorial to understand their respective roles in requirements management and development and the use of DOORS. Once satisfied as to the reliability of the data in DOORS, we then traced each of the 60 requirements S backwards to the system requirements and then to the operational requirements and forward to design requirements and verification methods. Because we followed a probability procedure based on ra selection, we are 95 percent confident that each of the confidence intervals in this report will include the true values in the study population. We used statistical methods appropriate for audit compliance testing to estimate 95 percent confiden r equirements in our sample. ce intervals for the traceability of Risk management: We reviewed relevant documentation, such as t SBInet Risk/Issue/Opportunity Management Plan, the SBInet SPO Risk/Issue/Opportunity Management Process, and the SBInet Risk Management Policy, as well as extracts from the SBInet risk management database and minutes of meetings and agendas from the Risk Management Team and the Joint Risk Review Board. In doing so, we compared the risk management process defined in these documents to relevant guidance todetermine the extent to which the program has defined an effective risk management approach. Further, we observed a demonstration of the r database, and we compared SBInet risks identified by us and others, including the SBI Executive Director, to the risks in the database to determine the extent to which all key risks were being actively managed. Further, we discussed actions recently taken and planned to improve risk management with the person responsible for SBInet risk management. W also reviewed briefings and related material provided to DHS leadership during oversight reviews of SBInet and interviewed program officials to ascertain the extent to which program risks were disclosed at these reviews and at meetings with congressional committees. In this regard, we also asked cognizant staff with the House Homeland Security Commi about the extent to which pr o ogram risks were disclosed by program fficials in status briefings.

To determine the extent to which DHS has addressed our prior SBInet recommendations, we focused on the eight recommendations that we made in our September 2008 report. For each recommendation, we leveraged the work described above, augmenting it as necessary to determine any plans or actions peculiar to a given recommendat example, to determine the status of efforts to address our prior recommendation related to SBInet testing, we reviewed key testing ion. For documentation, such as the Test and Evaluation Master Plan; SBInet component and system qualification test plans, test procedures, and test reports; program management reviews; program office briefings; and D Acquisition Review p Board decision memoranda. We also interviewed rogram officials.

To support our work across the above objectives, we also interviewed officials from the Department of Defense’s Defense Contract Managemen Agency, which provides contractor oversight services, to understand i reviews of the contractor’s integrated master schedule, requirements development and management activities, risk management practices, and testing activities. We also reviewed Defense Contract Management Agenc reports pertaining to documentation, such as monthly status reports and the integrated master schedule and cost reporting.

To assess the reliability of the data that we relied on to support the findings in the report, we reviewed relevant program documentation to substantiate evidence obtained through interviews with knowledgeable agency officials, where available. We determined that the data used in this also made appropriate attribution report are sufficiently reliable. We have indicating the sources of the data used.

We performed our work at the Customs and Border Protection headquarters and contractor facilities in the Washington, D.C., metropolitan area and at a contractor facility and a Defense Contract Management Agency office in Huntsville, Alabama. We conducted this performance audit from December 2008 to May 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropria evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. (CBP)

Appendix II: Comments from the Department of Homeland Security

Appendix III: Status of Key GAO Recommendations

In September 2008, we reported on a number of SBInet program management weaknesses and associated risks related to establishing program commitments, developing an integrated master schedule, defining and implementing a life cycle management approach, developing and managing requirements, and testing. To address these weaknesses and risks, we made a number of recommendations. Table 7 provides details on DHS efforts to address each recommendation.

Appendix IV: Detailed Results of GAO Assessment of SBInet Program Schedule

Our research has identified a range of best practices associated with effective schedule estimating. These are (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying reasonable float time between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations. We assessed the extent to which the SBInet integrated master schedule, dated August 2009, met each of the nine practices as either Not Met (the program provided no evidence that satisfies any portion of the criterion), Minimally Met (the program provided evidence that satisfies less than one-half of the criterion), Partially Met (the program provided evidence that satisfies about one-half of the criterion), Substantially Met (the program provided evidence that satisfies more than one-half of the criterion), and Met (the program provided evidence that satisfies the entire criterion). Table 8 shows the detailed results of our analysis.

Appendix V: Detailed Results of GAO Assessment of SBInet Cost Estimate

Our research has identified 12 practices that are integral to effective program life cycle cost estimating. These 12 practices in turn relate to four characteristics of a high-quality and reliable cost estimate: Comprehensive: The cost estimate should include all government and contractor costs over the program’s full life cycle, from program inception through design, development, deployment, and operation and maintenance to retirement. It should also provide sufficient detail to ensure that cost elements are neither omitted nor double-counted, and it should document all cost-influencing ground rules and assumptions.

Well-documented: The cost estimate should capture in writing things such as the source and significance of the data used, the calculations performed and their results, and the rationale for choosing a particular estimating method or reference. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources. Finally, the cost estimate should be reviewed and accepted by management to demonstrate confidence in the estimating process and the estimate.

Accurate: The cost estimate should not be overly conservative or optimistic, and should be, among other things, based on an assessment of most likely costs, adjusted properly for inflation, and validated against an independent cost estimate. In addition, the estimate should be updated regularly to reflect material changes in the program and actual cost experience with the program. Further, steps should be taken to minimize mathematical mistakes and their significance and to ground the estimate in documented assumptions and a historical record of actual cost and schedule experiences with other comparable programs.

Credible: The cost estimate should discuss any limitations in the analysis due to uncertainty or biases surrounding data or assumptions. Major assumptions should be varied and other outcomes computed to determine how sensitive the estimate is to changes in the assumptions. Risk and uncertainty inherent in the estimate should be assessed and disclosed. Further, the estimate should be properly verified by, for example, comparing the results with an independent cost estimate.

Our analysis of the $1.3 billion SBInet life cycle cost estimate relative to each of the 12 best practices, as well as to each of the four characteristics, is summarized in table 9. A detailed analysis relative to the 12 practices is in table 10.

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Deborah Davis (Assistant Director), David Alexander, Rebecca Alvarez, Carl Barden, Tisha Derricotte, Neil Doherty, Nancy Glover, Dan Gordon, Cheryl Dottermusch, Thomas J. Johnson, Kaelin P. Kuhn, Jason T. Lee, Lee McCracken, Jamelyn Payan, Karen Richey, Karl W.D. Seifert, Matt Snyder, Sushmita Srikanth, Jennifer Stavros-Turner, Stacey L. Steele, and Karen Talley made key contributions to this report.