Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering Key Technology Investment
Released: 2008-09-22T00:00:00

Letter

Background

CBP’s SBI program is to leverage technology, tactical infrastructure, and people to allow CBP agents to gain control of the nation’s borders. Within SBI, SBInet is the program for acquiring, developing, integrating, and deploying an appropriate mix of (1) surveillance technologies, such as cameras, radars, and sensors, and (2) command, control, communications, and intelligence (C3I) technologies.

The initial focus of SBInet has been on addressing the requirements of CBP’s Office of Border Patrol, which is responsible for securing the borders between the established ports of entry. The longer-term SBInet systems solution also is to address requirements of CBP’s two other major components—the Office of Field Operations, which controls vehicle and pedestrian traffic at the ports of entry, and the Office of Air and Marine Operations, which operates helicopters, fixed-wing aircraft, and marine vessels used in securing the borders. Figure 1 provides a high-level, operational concept of the long-term SBInet systems solution.

Surveillance technologies are to include a variety of sensor systems that improve CBP’s ability to detect, identify, classify, and track items of interest along the borders. Unattended ground sensors are to be used to detect heat and vibrations associated with foot traffic and metal associated with vehicles. Radars mounted on fixed and mobile towers are to detect movement, and cameras on fixed and mobile towers are to be used to identify, classify, and track items of interest detected by the ground sensors and the radars. Aerial assets are also to be used to provide video and infrared imaging to enhance tracking of targets.

The C3I technologies are to include software and hardware to produce a Common Operating Picture (COP)—a uniform presentation of activities within specific areas along the border. The sensors, radars, and cameras are to gather information along the border, and the system is to transmit this information to the COP terminals located in command centers and agent vehicles and assemble this information to provide CBP agents with border situational awareness. More specifically, the COP technology is to allow agents to (1) view data from radars and sensors that detect and track movement in the border areas, (2) control cameras to help identify and classify illegal entries, (3) correlate entries with the positions of nearby agents, and (4) enhance tactical decision making regarding the appropriate response to apprehend an entry, if necessary.

Initially, COP information is to be distributed to terminals in command centers. We observed that these terminals look like a standard computer workstation with multiple screens. From this workstation, an operator is to be able to view an area of interest in several different ways. For example, the operator is to see different types of maps, satellite images, and camera footage on the multiple screens. The operator is also to be able to move the cameras to track images on the screen. According to program officials, eventually, when the radars detect potential items of interest, the system is to automatically move the cameras so the operator does not always need to initiate the search in the area.

We observed that COP data are also available on laptop computers, known as mobile data terminals, mounted in select agent vehicles in the field. These terminals are to enable field agents to see information similar to that seen by command center operators. Eventually, the COP technology is to be capable of providing distributed surveillance and tactical decision- support information to other DHS agencies and stakeholders external to DHS, such as local law enforcement. Figure 2 shows examples of COP technology in a command station and an agent vehicle.

The first SBInet capabilities were deployed under a pilot or prototype effort known as “Project 28.” Project 28 is currently operating along 28 miles of the southwest border in the Tucson Sector of Arizona. Project 28 was accepted by the government for deployment 8 months behind schedule (in February 2008); this delay occurred because the contractor- delivered system did not perform as intended. As we have previously reported, reasons for Project 28 performance shortfalls and delays include the following: System requirements were not adequately defined, and users were not involved in developing the requirements.

System integration testing was not adequately performed.

Contractor oversight was limited.

Project scope and complexity were underestimated.

To manage SBInet, DHS established a program office within CBP. The program office is led by a program manager and deputy program managers for program operations and mission operations. The program manager is responsible for the execution of the program, including developing, producing, deploying, and sustaining the system to meet the users’ needs. Among other things, this includes developing and analyzing requirements and system alternatives, managing system design and development, evaluating the system’s operational effectiveness, and managing program risk.

SBInet Program Office Description of Its Life Cycle Management Approach

A system life cycle management approach typically consists of a series of phases, milestone reviews, and related processes to guide the acquisition, development, deployment, and operation and maintenance of a system. Among other things, the phases, reviews, and processes cover such important life cycle activities as requirements development and management, design, software development, and testing. Based on available program documentation, augmented by program official briefings and statements, key aspects of the SBInet system life cycle management approach are described below.

Approach Is Intended to Deliver Incremental Capabilities through a Series of Blocks

In general, SBInet surveillance systems are to be acquired through the purchase of commercially available products, while the COP systems involve development of new, customized systems and software. Together, both categories are to form a deployable increment of the SBInet capabilities, which the program office refers to as a “block.” Each block is to include a release or version of the COP.

SBInet documentation shows that the program office is acquiring the blocks incrementally using a “spiral” approach, under which an initial system capability is to be delivered based on a defined subset of the system’s total requirements. This approach is intended to allow CBP agents access to new technological tools sooner rather than later for both operational use and feedback on needed enhancements or changes. Subsequent spirals or iterations of system capability are to be delivered based on feedback and unmet requirements, as well as the availability of new technologies. Figure 3 illustrates conceptually how the different capabilities are to come together to form a block and how future blocks are to introduce more capabilities.

The approach used to design and develop SBInet system capabilities for each block includes such key activities as requirements development, system design, system acquisition and development, and testing. The approach, as explained by program officials and depicted in part in various documents, also includes various reviews, or decision points, to help ensure that these activities are being done properly and that the system meets user needs and requirements. These reviews are to be used in developing both the overall SBInet Block 1 capability and the COP software. Table 1 provides a high-level description of the major reviews that are to be performed in designing and developing the system prior to deployment to the field and in the order that they occur.

Before a set of capabilities (i.e., block) is deployed to a specific area or sector of the border, activities such as site selection, surveys, and environmental impact assessments are conducted to determine the area’s unique environmental requirements. The border area that receives a given block, or set of system capabilities, is referred to as a “project.” Each project is to have a given block configured to its unique environmental requirements, referred to as a project “laydown.”

The deployment approach is to include such key activities as requirements development, system design, project laydown, integration, testing, and installation. The deployment approach is also to entail various reviews, or decision points, to help ensure that these activities are being done properly and that the system meets user needs and requirements. Table 2 provides a high-level description of the major reviews that are to be part of project laydown in the order that they occur.

Among the key processes provided for in the SBInet system life cycle management approach are processes for developing and managing requirements and for managing testing activities. With respect to requirements development and management, SBInet requirements are to consist of a hierarchy of six types of requirements, with the high-level operational requirements at the top. These high-level requirements are to be decomposed into lower-level, more detailed system, component, design, software, and project requirements. Having a decomposed hierarchy of requirements is a characteristic of complex information technology (IT) projects. The various types of SBInet requirements are described in table 3. Figure 4 shows how each of these requirements relate to or are derived from the other requirements.

With respect to test management, SBInet testing consists of a sequence of tests that are intended to verify first that individual system parts meet specified requirements, and then verify that these combined parts perform as intended as an integrated and operational system. Such an incremental approach to testing is a characteristic of complex IT system acquisition and development efforts. Through such an approach, the source of defects can be isolated more easily and sooner, before they are more difficult and expensive to address. Table 4 summarizes these tests.

Limited Definition of SBInet Deployments, Capabilities, Schedule, and Life Cycle Management Process Increases Program’s Exposure to Risk

Important aspects of SBInet remain ambiguous and in a continued state of flux, making it unclear and uncertain what technology capabilities will be delivered, when and where they will be delivered, and how they will be delivered. For example, the scope and timing of planned SBInet deployments and capabilities have continued to change since the program began and, even now, remain unclear. Further, the approach that is being used to define, develop, acquire, test, and deploy SBInet is similarly unclear and has continued to change. According to SBInet officials, schedule changes are due largely to an immature system design, and the lack of a stable development approach is due to insufficient staff and turnover. The absence of clarity and stability in these key aspects of SBInet introduces considerable program risks, hampers DHS’s ability to measure program progress, and impairs the ability of the Congress to oversee the program and hold DHS accountable for program results.

Scope and Timing of Planned Deployments and Capabilities Are Not Clear and Stable

One key aspect of successfully managing large IT programs, like SBInet, is establishing program commitments, including what capabilities are to be deployed and when and where they are to be deployed. Only when such commitments are clearly established can program progress be measured and can responsible parties be held accountable.

The scope and timing of planned SBInet deployments and capabilities that are to be delivered have not been clearly established, but rather have continued to change since the program began. Specifically, as of December 2006, the SBInet System Program Office planned to deploy an “initial” set of capabilities along the entire southwest border by late 2008 and planned to deploy a “full” set of operational capabilities along the southern and northern borders (a total of about 6,000 miles) by late 2009. As of March 2007, the program office had modified its plans, deciding instead to deploy the initial set of capabilities along the southwest border by the end of fiscal year 2007 (almost a year earlier than originally planned) and delayed the deployment of the final set of capabilities for the southern and northern borders until 2011.

In March 2008, the program office again modified its deployment plans, this time significantly reducing the area to which SBInet capabilities are to be deployed. At this time, DHS planned to complete deployments to three out of nine sectors along the southwest border—specifically, to Tucson Sector by 2009, Yuma Sector by 2010, and El Paso Sector by 2011. According to program officials, other than the dates for the Tucson, Yuma, and El Paso Sectors, no other deployment dates have been established for the remainder of the southwest or northern borders. (Figure 5 shows the changes in the planned deployment areas.)

The figure also shows the two sites within the Tucson Sector, Tucson 1 and Ajo 1, at which an initial Block 1 capability is to be deployed. Together, these two deployments cover 53 miles of the 1,989-mile-long southern border. According to the March 2008 SBI expenditure plan and agency documentation as of June 2008, these two sites were to have been operational by the end of 2008. However, as of late July 2008, program officials reported that the deployment schedule for these two sites has again been modified, and they will not be operational until “sometime” in 2009. According to program officials, the slippage in the deployment schedule is due to the need to complete environmental impact assessment documentation for these locations. The slippages in the dates for the first two Tucson deployments, according to a program official, will, in turn, delay subsequent Tucson deployments, although revised dates for these subsequent deployments have not been set.

Just as the scope and timing of planned deployments have not been clear and have changed over time, the specific capabilities that are to be deployed have been unclear. For example, in April 2008, program officials stated that they would not know which of the SBInet requirements would be met by Block 1 until the Critical Design Review, which at that time was scheduled for June 2008. At that time, program officials stated that the capabilities to be delivered would be driven by the functionality of the COP. In June, the review was held, but according to available documentation, the government did not consider the design put forth by the contractor to be mature. As a result, the system design was not accepted in June as planned. Among the design limitations found was a lack of evidence that the system requirements were used as the basis for the Tucson 1 and Ajo 1 design, lack of linkage between the performance of surveillance components and the system requirements, and incomplete definition of system interfaces. As of late July 2008, these issues were unresolved, and thus the design still had not been accepted. In addition, in late July 2008, agency officials stated that the capabilities to be delivered will be driven by the functionality of the surveillance components, not the COP.

In addition, the design does not provide key capabilities that are in requirements documents and were anticipated to be part of the Block 1 deployments to Tucson 1 and Ajo 1. For example, the first deployments of Block 1 will not have the mobile data terminals in border patrol vehicles, even though (1) such terminals are part of Project 28 capabilities and (2) workshops were held with the users in February 2008 and June 2008 to specifically define the requirements for these terminals for inclusion in Block 1. According to program officials, these terminals will not be part of Block 1 because the wireless communications infrastructure needed to support these terminals will not be available in time for the Tucson 1 deployment. Rather, they expect the wireless infrastructure to be ready “sometime” in 2009 and said that they will include the mobile data terminals in Block 1 deployments when the infrastructure is ready. Without the mobile data terminals, agents will not be able to obtain key information from their vehicles, such as maps of activity in a specific area, incident reports, and the location of other agents in the area. Instead, the agents will have to use radios to communicate with the sector headquarters to obtain this information.

In addition, program officials told us that a number of other requirements cannot be included in the Block 1 version of the COP, referred to as version 0.5, due to cost and schedule issues. However, we have yet to receive a list of these requirements. According to program officials, they hope to upgrade the COP in 2009 to include these requirements.

Without clearly establishing program commitments, such as capabilities to be deployed and when and where they are to be deployed, program progress cannot be measured and responsible parties cannot be held accountable.

Program Schedule Is Unsettled

Another key aspect of successfully managing large programs like SBInet is having a schedule that defines the sequence and timing of key activities and events and is realistic, achievable, and minimizes program risks. However, the program office does not yet have an approved integrated master schedule to guide the execution of SBInet, and according to program officials, such a schedule has not been in place since late 2007. In the absence of an approved integrated master schedule, program officials stated in mid-August 2008 that they have managed the program largely using task-order-specific baselined schedules, and have been working to create a more integrated approach. A program official also stated that they have recently developed an integrated master schedule but that this schedule is already out of date and undergoing revision. For example, the deployment of the SBInet system to the Tucson Sector will not be completed in 2009 as planned.

To understand where the program is relative to established commitments, we analyzed schedule-related information obtained from other available program documents, briefings, and interviews. In short, our analysis shows a schedule in which key activities and events are subject to constant change, as depicted in the following two figures. Figure 6 shows the changes to the schedule of planned and held reviews and anticipated deployment dates, and figure 7 shows the changes to the schedule of testing activities.

In May 2008, program officials stated that the schedule changes were due largely to the fact that the contractor had not yet provided a satisfactory system-level design. They also noted that the contractor’s workforce has experienced considerable turnover, including three different program managers and three different lead system engineers. They also stated that the System Program Office has experienced attrition, including turnover in the SBInet Program Manager position. Without stability and certainty in the program’s schedule, program cost and schedule risks increase, and meaningful measurement and oversight of program status and progress cannot occur, in turn, limiting accountability for results.

SBInet Life Cycle Management Approach Has Not Been Clearly Defined and Has Continued to Change

System quality and performance are in large part governed by the processes followed in developing and acquiring the system. To the extent that a system’s life cycle management approach and related development and acquisition processes are well-defined, the chances of delivering promised system capabilities and benefits on time and within budget are increased. To be well-defined, the approach and processes should be fully documented, so that they can be understood and properly implemented by those responsible for doing so.

The life cycle management approach and processes being used by the SBInet System Program Office to manage the definition, design, development, testing, and deployment of system capabilities has not been fully and clearly documented. Rather, what is defined in various program documents is limited and not fully consistent across these documents. Moreover, in discussions with agency officials to clarify our understanding of these processes, new terms and processes have been routinely introduced, indicating that the processes are continuing to evolve. Agency officials acknowledge that they are still learning about and improving their processes. Without a clearly documented and universally understood life cycle management approach and supporting processes, the program is at increased risk of not meeting expectations.

Key program documentation that is to be used to guide acquisition and development activities, including testing and deployment activities, is incomplete, even though SBInet acquisition and development are already under way. For example, officials have stated that they are using the draft Systems Engineering Plan, dated February 2008, to guide the design, development, and deployment of system capabilities, and the draft Test and Evaluation Master Plan, dated May 2008, to guide the testing process—but both of these documents are lacking sufficient information to clearly guide system activities, as the following examples explain: The Systems Engineering Plan includes a diagram of the engineering process; however, the steps of the process and the gate reviews are not defined or described in the text of the document. For example, this document does not contain sufficient information to understand what occurs at key reviews, such as the Preliminary Design Review, the Critical Design Review, and the Test Readiness Review.

The Test and Evaluation Master Plan describes in more detail some of the reviews that are not described in the Systems Engineering Plan, but the reviews included are not consistent between the documents. For example, the Test and Evaluation Master Plan includes a System Development and Demonstration Review that is not listed in the Systems Engineering Plan. In addition, it is not clear from the Test and Evaluation Master Plan how the reviews fit into the overall engineering process.

Statements by program officials responsible for system development and testing activities, as well as briefing materials and diagrams that these officials provided, did not add sufficient clarity to describe a well-defined life cycle management approach. Moreover, these descriptions were not always consistent with what was contained in the documentation, as the following examples demonstrate: Component testing is not described in the Test and Evaluation Master Plan in a manner consistent with how officials described this testing. Specifically, while the plan states that components will be tested against the corresponding component requirements to ensure all component performance can be verified, program officials stated that not all components will undergo component testing. Instead, they said that testing is not required if component vendors submit a certificate of compliance for certain specifications.

Functional qualification testing was described to us by program officials in July 2008 as a type of testing to be performed during software development activities. However, this type of testing is not defined in available program documentation, and it was not included in any versions of the documentation associated with the life cycle management approach and related engineering processes.

Certain reviews specified in documentation of the life cycle management process are not sufficiently defined. For example, the Systems Engineering Plan shows a Production Readiness Review as part of the system-level process and an Operational Readiness Review as part of the project-level process. However, program officials stated that these reviews are not completely relevant to SBInet because they are targeted for informational systems rather than tactical support systems, such as SBInet. According to the officials, they are in the process of determining how to apply the reviews to SBInet. For example, in July 2008, officials reported that they may move the Production Readiness Review from the system-level set of activities, as shown in the Systems Engineering Plan, to the project-level set of activities. Program officials also stated that they are working to better define these reviews in the Systems Engineering Plan.

Program officials told us that the SBInet life cycle management approach and related engineering processes are understood by both government and contractor staff through the combination of the draft Systems Engineering Plan and government-contractor interactions during design meetings. Nevertheless, they acknowledged that the approach and processes are not well documented, citing a lack of sufficient staff to both document the processes and oversee the system’s design, development, testing, and deployment. They also told us that they are adding new people to the project with different acquisition backgrounds, and that they are still learning about, evolving, and improving the approach and processes. According to these officials, a revised and updated Systems Engineering Plan should be finalized by September 2008.

The lack of definition and stability in the approach and related processes being used to define, design, develop, acquire, test, and deploy SBInet introduce considerable risk that both the program officials and contractor staff will not understand what needs to be done when, and thus that the program will not consistently employ disciplined and rigorous methods. Without the use of such methods, the risk of delivering a system that does not meet operational needs and does not perform as intend is increased. Moreover, without a well-defined approach and processes, it is difficult to gauge progress and thus promote performance and accountability for results.

Limitations of SBInet Requirements Development and Management Efforts Increase Program Risk

Well-defined and managed requirements are a cornerstone of effective system development and acquisition. According to recognized guidance, documenting and implementing a disciplined process for developing and managing requirements can help reduce the risks of developing a system that does not meet user needs, cannot be adequately tested, and does not perform or function as intended. Such a process includes, among other things, eliciting user needs and involving users in the development process; ensuring that requirements are complete, feasible, verifiable, and approved by all stakeholders; documenting and approving the requirements to establish a baseline for subsequent development and change control; and ensuring that requirements are traceable both back to operational requirements and forward to detailed system requirements and test cases.

To the program office’s credit, it recently developed guidance for developing and managing requirements that is consistent with recognized leading practices. For example, the program’s guidance states that a requirements baseline should be established and that requirements are to be traceable both back to higher-level requirements and forward to verification methods. However, this guidance was not finalized until February 2008 and thus was not used in performing a number of key requirements-related activities.

In the absence of well-defined guidance, the program’s efforts to implement leading practices for developing and managing requirements have been mixed. For example, while the program has elicited user needs as part of its efforts to develop high-level operational requirements, it has not baselined all requirements. Further, it has not ensured that the operational requirements were, for example, verifiable, and it has not made certain that all of the different levels of requirements are aligned to one another. As a result, the risk of SBInet not meeting mission needs and performing as intended is increased, as are the chances of expensive and time-consuming system rework.

Requirements Development and Management Guidance Is Consistent with Leading Practices but Was Developed After Many Requirements Activities Were Performed

The SBInet program office has developed guidance for developing and managing requirements that is generally consistent with recognized leading practices. According to these practices, effectively developing and managing requirements includes, among other things, eliciting users’ needs early in the development process and involving them throughout the process; ensuring that requirements are complete, feasible, verifiable, and approved by all stakeholders; documenting and approving the requirements to establish a baseline for subsequent development and change control; and ensuring that requirements are traceable both back to operational requirements and forward to detailed system requirements and test cases.

In February 2008, the program office approved its SBInet Requirements Development and Management Plan. According to the plan, its purpose is to describe a comprehensive approach to developing and managing requirements for the SBInet program. Our analysis of this plan shows that it is consistent with leading practices. For example, the plan states that users should provide input to the requirements definition process through early and ongoing participation in integrated product teams, user conferences, and other requirements-gathering and verifying activities; requirements developers are to ensure that requirements are complete, unambiguous, achievable, verifiable, and not redundant; a requirements baseline should be created to provide a common understanding of the system to be built and to prevent deviations to the requirements from entering during design, development, or testing; and bidirectional traceability both back to higher-level requirements and forward to detailed test methods should be established and maintained and that the requirements management team is responsible for maintaining the requirements management database and holding the contractor responsible for any traceability issues.

Moreover, the plan defines procedures and steps for accomplishing each of these goals. For example, the procedure for requirements development outlines the purpose of the procedure, who is involved, what documentation is necessary to begin the procedure, and what outputs are expected at the end. The procedure then describes 16 steps necessary to achieve the requirements development activities. A separate procedure is provided for requirements verification and validation.

However, the plan was not approved until after key SBInet requirements documents were written and baselined. Specifically, user operational requirements were approved and baselined in March 2007; system requirements were first baselined in March 2007; and several component requirements were baselined in June, August, and October 2007. As a result, the plan was not used to guide these requirements development and management efforts.

Program Office Has Taken Steps to Involve Users in Developing High-Level Requirements

As noted above, one of the leading practices associated with effective requirements development and management is engaging system users early and continuously. In doing so, the chances of defining, designing, and delivering a system that meets their needs and performs as intended are increased.

In developing the operational requirements, the System Program Office involved SBInet users in a manner that is consistent with leading practices and that reflects lessons learned from Project 28. Specifically, it conducted requirements-gathering workshops from October 2006 through April 2007 to ascertain the needs of Border Patrol agents. In addition, it established work groups in September 2007 to inform the next revision of the operational requirements by soliciting input from both the Office of Air and Marine Operations and the Office of Field Operations. Further, to develop the COP technology for SBInet, the program office is following a software development methodology that allows end users to be directly involved in software development activities and, thereby, permits software solutions to be tailored to users’ needs.

Through such efforts to identify and elicit user needs in developing high- level requirements, the chances of developing a system that will meet user needs are increased.

Not All Levels of Requirements Have Been Adequately Baselined

The creation of a requirements baseline is important for providing a stable basis for system design, development, and testing. Such a baseline establishes a set of requirements that have been formally reviewed and agreed on and thus serve as the basis for further development or delivery. Until requirements are baselined, they remain unclear and subject to considerable and uncontrolled change, which in turn makes system design, development, testing, and deployment efforts equally uncertain. According to SBInet program officials, the SBInet Requirements Development and Management Plan, and leading practices, requirements should be baselined before key system design activities begin, since the requirements are intended to inform, guide, and constrain the system’s design.

For SBInet, while many of the requirements have been baselined, two types have not yet been baselined. According to the System Program Office, the operational requirements and the system requirements were approved and baselined in March 2007. In addition, various system component requirements were baselined in June, August, and October of 2007. However, the program had not baselined its COP software requirements as of July 2008, although according to program officials, the COP has been designed and is under development. Further, it has yet to baseline its project-level requirements, which define the requirements for the system configuration to be deployed to a specific geographical area, such as Tucson 1.

With respect to the COP, requirements for the software and hardware had not been baselined as of the end of July 2008, despite the fact that a combined Preliminary Design Review and Critical Design Review for the COP was held in February 2008 and a Critical Design Review for the system as whole was held in June 2008. According to agency officials and the SBInet Requirements Development and Management Plan, requirements should be baselined before the Critical Design Review. Regardless, program officials state that the contractor has developed several “builds” (i.e., versions) of the COP, which are currently being tested. According to program officials, the requirements were not complete because certain interface requirements had not yet been completely identified and defined. Without baselined requirements, the basis of the system design and the degree to which it satisfies requirements are unclear. Moreover, the risk of the design not aligning to requirements is increased. According to the results of the Critical Design Review, this risk was realized. Specifically, the System Program Office notified the contractor that there was no evidence linking the performance of surveillance components to the system requirements, that the review could not be completed until the interface requirements had been finalized, and that a mature design had not been presented at the review.

With respect to project-level (i.e., geographic area) deployment requirements, baselined requirements do not yet exist. Specifically, requirements for the Tucson Sector, which includes Tucson 1 and Ajo 1, have yet to be baselined. According to the SBInet Requirements Development and Management Plan, requirements should be baselined before the Project Requirements Review, and a new requirements baseline should be created following the subsequent Deployment Design Review. However, project-level requirements were not baselined at a Project Requirements Review held for the Tucson Sector Project in March 2007 or at a Deployment Design Review in June 2007. Officials stated that this is because the plan was not approved until February 2008 and thus was not in effect. However, since the plan became effective, Deployment Design Reviews were held for Tucson 1 and Ajo 1 in April 2008 and May 2008, respectively, but the project-level requirements were not baselined.

Despite the absence of baselined requirements, the System Program Office has proceeded with development, integration, and testing activities for the Block 1 capabilities to be delivered to Tucson 1 and Ajo l. As a result, it faces an increased risk of deploying systems that do not align well with requirements and thus may require subsequent rework. The lack of project requirements has already had an effect on testing activities. Specifically, the draft system integration test plan notes that, without project requirements, testing will have to be guided by a combination of other documents, including engineering development requirements, related component requirements, and architectural design documents.

Baselined Operational Requirements Used to Inform Lower-Level Requirements Are Limited

As stated above, one of the leading practices for developing and managing requirements—which is reflected in the program office’s own plan—is that requirements should be sufficiently analyzed to ensure that the requirements are, among other things, complete, unambiguous, and verifiable. However, an independent review of SBInet operational requirements reported numerous problems. Specifically, a review of the SBInet program commissioned by the Office of the Deputy Secretary of Homeland Security found that several requirements were unaffordable and unverifiable. Examples of these requirements include the following: Allow for complete coverage of the specified area or zone to be surveilled.

Maximize intended deterrence and minimize countermeasure effectiveness.

Function with high reliability under reasonably foreseeable circumstances.

Reliably provide the appropriate power and bandwidth at the least cost that will support the demand.

In April 2008, a program official stated that the operational requirements document is currently being rewritten to address concerns raised by this review. For example, we were told that certain system performance requirements are being revised in response to a finding that the requirements are insufficient to ensure delivery of a properly functioning system. Program officials stated that they expect to finalize the revised operational requirements document in October 2008.

However, given the number and types of problems associated with the operational requirements—which are the program’s most basic customer requirements and form the basis for all lower-level requirements—it is unclear how the system, component, and software requirements can be viewed as verifiable, testable, or affordable. Until these problems are addressed, the risk of building and deploying a system that does not meet mission needs and customer expectations is increased, which in turn increases the chances of expensive and time-consuming system rework.

SBInet Requirements Have Not Been Sufficiently Aligned

As noted above, one of the leading practices associated with developing and managing requirements is maintaining bidirectional traceability from high-level operational requirements through detailed low-level requirements to test cases. The SBInet Requirements Development and Management Plan recognizes the importance of traceability, stating that a traceability relationship should exist among the various levels of requirements. For example, it states that operational requirements should trace to the system requirements, which in turn should trace to component requirements. Further, it states that component requirements should trace to design requirements and to a verification method. In addition, the SBInet System Program Office established detailed guidance for populating and maintaining the requirements database for maintaining linkages among the various levels of requirements and test verification methods.

To provide for requirements traceability, the prime contractor established such a requirements management database. However, the reliability of the requirements in this database is questionable, and the SBInet System Program Office has not effectively overseen the contractor’s management of requirements through this database. Specifically, we attempted to trace requirements in the version of this database that the program office received in March 2008 and were unable to trace large percentages of component requirements to either higher-level or lower-level requirements. For example, an estimated 76 percent (with a 95 percent degree of confidence of being between 64 and 86 percent) of the component requirements that we randomly sampled could not be traced to the system requirements and then to the operational requirements. In addition, an estimated 20 percent (with a 95 percent degree of confidence of being between 11 and 33 percent) of the component requirements in our sample failed to trace to a verification method. See table 5 for the failure rates for each of our tracing analyses, along with the related confidence intervals.

While program officials could not explain the reason for this lack of traceability in most cases, they did attribute the 100-percent failure in tracing component requirements to the design requirements to the absence of any design requirements in the program office’s copy of the database.

A contributing factor to the program office’s inability to explain why requirements were not traceable is its limited oversight of the contractor’s efforts to manage requirements through this database. According to program officials, the contractor created the SBInet requirements management database in December 2006, but the program office did not receive a copy of the database until March 2008, despite requests for it beginning in fall 2007. In early May 2008, the Chief Engineer told us the contractor had been reluctant to provide the database because it viewed the database’s maturity level as low. Moreover, the program office’s direct access to the database had not been established because of security issues, according to this official.

Following our efforts to trace requirements, the program office obtained direct access to the contractor’s database and initiated efforts with the contractor to resolve the traceability gaps. However, program officials told us that they are still not certain that the database currently contains all of the system requirements or even the reduced Block 1 system requirements. As a result, they did not rely on it during the recent Critical Design Review to verify requirements traceability. Instead, they said that manual tracking methods were used.

Without ensuring that requirements are fully traceable, the program office does not have a sufficient basis for knowing that the scope of the contractor’s design, development, and testing efforts will produce a system solution that meets operational needs and performs as intended.

As a result, the risk of expensive and time-consuming system rework is increased.

Limitations in Key SBInet Testing and Test Management Activities Increase Program Risk

To be effectively managed, testing should be planned and conducted in a structured and disciplined fashion. This includes, among other things, having an overarching test plan or strategy as a basis for managing system testing, developing well-defined and approved plans for executing testing activities, and testing individual system components to ensure that they satisfy defined requirements prior to integrating them into the overall system.

The SBInet System Program Office is not effectively managing its testing activities. Specifically, it has not tested individual system components prior to integrating these components with other components and the COP software. In addition, although the program’s draft Test and Evaluation Master Plan is currently being used as the program’s overall strategy to manage SBInet testing, the plan is incomplete and unclear with respect to several test management functions. As a result, the chances of SBInet testing being effectively performed are reduced, which in turn increases the risk that the delivered and deployed system will not meet operational needs and not perform as intended.

System Integration Testing Has Not Been Adequately Planned or Executed

To be effectively managed, relevant federal guidance states that testing should, among other things, be governed by a well-defined and approved plan, and it should be executed in accordance with this plan. Further, integration testing should be preceded by tests of system components (whether acquired or developed) that are to be integrated to form the overall system. Once the components are tested to ensure that they satisfy defined requirements, the integrated system can be tested to verify that it performs as required. For SBInet, this has not occurred. As a result, the risk of the system not meeting operational needs and not performing as intended is increased, which in turn is likely to introduce the need for expensive and time-consuming system rework.

The SBInet Systems Program Office reports that it began Block 1 system integration testing in June 2008. However, it still does not have an approved system integration test plan. Specifically, the system’s Critical Design Review, which was held in June 2008, found numerous problems with the contractor’s plan for system integration testing, and as a result, the program office did not accept the plan. Examples of problems were that the test plan refers to other test plans that the contractor had yet to deliver and the plan identified system components that were not expected to be part of Block 1. As a result, the program office decided that the system integration plan needed to be revised to document and describe the full set of actual testing activities that were to occur, including identifying where and to what level the different phases of integration tests would occur.

Notwithstanding these problems and the absence of an approved plan, the contractor began integration testing in June 2008. According to program officials, this was necessary to meet the tight time frames in the schedule. However, without an accepted system integration test plan in place, testing cannot be effectively managed. For example, the adequacy of the test scope cannot be assured, and the progress in completing test activities cannot be measured. As a result, there is an increased risk that the delivered system will not meet operational needs and will not perform as intended and that expensive and time-consuming system rework will be required.

Moreover, the SBInet draft Test and Evaluation Master Plan describes system integration testing as first testing individual components to verify that the smallest defined module of a system works as intended (i.e., meets functional and performance requirements). This allows defects with individual components to be identified and corrected before they are integrated with other system components. Once the components are tested, their respective hardware and software interfaces are to be tested before subsystems are tested in combination with the COP software. Such an incremental approach to testing permits system defects to be found and addressed before system components are integrated and component problems become more expensive and time-consuming to correct.

However, the SBInet System Program Office has not performed individual component testing as part of integration testing. As of July 2008, agency officials reported that component-level tests had not been completed and were not scheduled to occur. Instead, officials stated that Block 1 components were evaluated based on what they described as “informal tests” (i.e., contractor observations of cameras and radar suites in operation at a National Guard facility in the Tucson Sector) and stated that the contractors’ self-certification that the components meet functional and performance requirements was acceptable. However, this approach is not consistent with the Test and Evaluation Master Plan. Moreover, program officials acknowledged that this approach did not verify if the individual components in fact met requirements. Nevertheless, they said that they have recently modified their definition of component testing to allow the contractor’s certification to be used.

In our view, relying solely on contractor certification is not a sufficient substitute for component testing—as defined in the Test and Evaluation Master Plan—because it increases the risk that components, and thus the entire system, will not perform as intended. This risk is already starting to be realized. Specifically, the results of the Block 1 Critical Design Review performed in early in June 2008 show that design documents did not link components’ performance to the system requirements.

Key Test Management Activities Have Not Been Adequately Defined and Addressed

To ensure that system testing is effectively performed, federal guidance provides for having an overarching test plan or strategy to use as a basis for managing system testing. Among other things, this test management plan should define the schedule of high-level test activities in sufficient detail to allow for more detailed test planning and execution to occur and to ensure that test progress can be tracked and results can be reported and addressed. The plan should also define the roles and responsibilities of the various groups responsible for different levels of testing and include a description of how the test organization manages and oversees these groups in their activities.

The SBInet Test and Evaluation Master Plan, which documents the program’s test strategy and is being used to manage system testing, has yet to be approved by the SBInet Acting Program Manager. As of July 2008, program officials told us that they did not expect the draft plan to be approved until August 2008, even though testing activities began in June 2008.

Moreover, the draft Test and Evaluation Master Plan is not complete. For example, it does not contain an accurate and up-to-date test schedule with milestones and completion dates for all levels of test activities. Rather, the schedule information included in the plan has been overtaken by events, to the point that program officials stated that many of the dates on the schedules have changed or are not accurate. Moreover, they described attempting to have an accurate schedule of testing activities and events as “futile” because the program’s schedule is constantly changing. As a another example, the draft Test and Evaluation Master Plan does not identify any metrics for measuring testing progress for any type of testing to be performed. According to federal guidance, an accurate schedule is necessary to inform planning for and sequencing of each type of testing to be performed, including ensuring that test resources are available when needed and that predecessor test events occur before successor events begin. This guidance also states that without performance metrics, it is difficult to understand where test activities stand and what they show in a manner that can inform program decision making.

As another example, the draft Test and Evaluation Master Plan does not clearly define the roles and responsibilities of various entities that are involved in system testing. Specifically, the plan identifies seven entities, but it only provides vague descriptions of their respective roles and responsibilities that are not meaningful enough to effectively guide their efforts. For example, the plan identifies two entities that are to be involved in operational testing: the DHS Science and Technology Test and Evaluation Office and the U.S. Army Test and Evaluation Command. According to the plan, the DHS office is to function as the operational test authority and will be responsible for initial planning of “dedicated initial” and “follow-on” operational testing and evaluation, and the Army group is to conduct operational testing. With no further clarification, it is not clear what is expected of each of these entities, including how they are to interact. Table 6 lists each of the identified entities and provides their respective roles and responsibilities copied from the draft plan.

Besides being vague, the descriptions of roles and responsibilities are also incomplete and not consistent with other program documents. For example, according to the draft plan, the Test and Evaluation Division of the SBInet System Program Office is responsible only for developing test and evaluation reports. However, according to the draft SBInet Systems Engineering Plan, this entity is to act as a subject matter expert for the oversight or conduct of various testing activities.

Beyond this lack of clearly defined roles and responsibilities, there are other problems with the groups assigned testing roles. First, some of the entities identified in the draft plan are not yet operational and thus are unavailable to participate and perform their assigned roles and responsibilities. According to program officials, the independent verification and validation agent has not been selected, the Integrated Project Teams have not been chartered, and DHS is still in the process of establishing the DHS Science and Technology Test and Evaluation Office. Second, although CBP has an interagency agreement with the U.S. Army Test and Evaluation Command for operational testing, no such agreement exists with the SBInet program specifically for Block 1 testing.

Finally, neither the draft Test and Evaluation Master Plan nor the draft Systems Engineering Plan clearly defines the program office’s role and responsibilities for managing and overseeing each of the other six test entities and their respective test activities. The SBInet System Program Office is responsible and accountable for ensuring that the system is successfully deployed and operates as intended. Given the criticality of testing in ensuring a successful program, this means that the program office must ensure that each of these entities executes its assigned roles effectively. However, the draft Test and Evaluation Master Plan does not recognize this role and its associated responsibilities. Further, while the draft Systems Engineering Plan states that the program office is responsible for engaging external test agents, it provides no further description of the program office’s roles and responsibilities.

Without clearly defined roles and responsibilities for all entities involved in SBInet testing, the risk of test activities not being effectively and efficiently performed increases. As a result, the chances are increased that the deployed system will not meet operational requirements and perform as intended. Ultimately, this could lead to expensive and time-consuming system rework.

Conclusions

A fundamental aspect of successfully implementing a large program like SBInet is establishing program commitments, including what capabilities will be delivered and when and where they will be delivered. Only through establishing such commitments and by adequately defining the approach and processes to be used in delivering these commitments, can DHS effectively position itself for measuring progress, ensuring accountability for results, and delivering a system solution with its promised capabilities and benefits on time and within budget constraints. For SBInet, this has not occurred to the extent that it needs to for the program to have a meaningful chance of succeeding. In particular, commitments to the timing and scope of system capabilities remain unclear and continue to change, with the program committing to far fewer capabilities than originally envisioned. Further, how the SBInet system solution is to be delivered has been equally unclear and inadequately defined. Moreover, while the program office has defined key practices for developing and managing requirements, these practices were developed after several key requirements activities were performed. In addition, efforts performed to date to test whether the system meets requirements and functions as intended have been limited.

Collectively, these limitations are significant in that they increase the risk that the delivered system solution will not meet user needs and operational requirements and will not perform as intended. These consequences, in turn, increase the chances that the system will require expensive and time-consuming rework. In light of these circumstances and risks surrounding SBInet, it is important for the program office to reassess its approach to and plans for the program—including its associated exposure to cost, schedule, and performance risks—and to disclose these risks and alternative courses of action for addressing them to DHS and congressional decision makers. It is also important for the program to correct the weaknesses discussed in this report surrounding the program’s unclear and constantly changing commitments and its life cycle management approach and processes, including the processes and efforts performed to date relating to requirements development and management and testing.

While doing so will not guarantee a successful program, it will minimize the program’s exposure to risk and thus decrease the likelihood that it will fall short of expectations. For SBInet, living up to expectations is important because the program is a large, complex, and integral component of DHS’s border security and immigration control strategy.

Recommendations for Executive Action

To improve DHS’s efforts to acquire and implement SBInet we are making eight recommendations.

To permit meaningful measurement and oversight of and accountability for the program, we recommend that the Secretary of Homeland Security direct the CBP Commissioner to ensure that (1) the risks associated with planned SBInet acquisition, development, testing, and deployment activities are immediately assessed and (2) the results, including proposed alternative courses of action for mitigating the risks, are provided to the Commissioner and DHS’s senior leadership, as well as to the department’s congressional authorization and appropriation committees.

We further recommend that the Secretary of Homeland Security direct the CBP Commissioner to have the Acting SBInet Program Manager take the following additional actions: Establish and baseline the specific program commitments, including the specific system functional and performance capabilities that are to be deployed to the Tucson, Yuma, and El Paso Sectors, and establish when these capabilities are to be deployed and are to be operational.

Finalize and approve an integrated master schedule that reflects the timing and sequencing of the work needed to achieve these commitments.

Revise and approve versions of the SBInet life cycle management approach, including the draft Systems Engineering Plan and draft Test and Evaluation Management Plan, and in doing so, ensure that these revised and approved versions are consistent with one another, reflect program officials’ recently described changes to the engineering and testing approaches, and reflect relevant federal guidance and associated leading practices.

Ensure that the revised and approved life cycle management approach is fully implemented.

Implement key requirements development and management practices to include (1) baselining requirements before system design and development efforts begin; (2) analyzing requirements prior to being baselined to ensure that they are complete, achievable, and verifiable; and (3) tracing requirements to higher-level requirements, lower-level requirements, and test cases.

Implement key test management practices to include (1) developing and documenting test plans prior to the start of testing; (2) conducting appropriate component level testing prior to integrating system components; and (3) approving a test management strategy that, at a minimum, includes a relevant testing schedule, establishes accountability for testing activities by clearly defining testing roles and responsibilities, and includes sufficient detail to allow for testing and oversight activities to be clearly understood and communicated to test stakeholders.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Director, Departmental GAO/Office of Inspector General Liaison and reprinted in appendix II, the department stated that it agrees with seven of our eight recommendations, and partially disagrees with one aspect of the remaining recommendation. The department also stated that our report is factually sound and that it is working to address our recommendations and resolve the management and operational challenges identified in the report as expeditiously as possible. In this regard, it described actions recently completed, underway, and planned that it said addresses our recommendations. It also provided technical comments that we have incorporated in the report, as appropriate.

Regarding our recommendation to implement key test management practices, including conducting appropriate component-level testing prior to integrating system components, DHS commented that its current test strategy provides for the appropriate degree of technical confidence for commercially available products, as evidenced by either certificates of conformance from the original equipment manufacturer, test documentation from independent government laboratories, or the prime contractor’s component/integration level testing. We support DHS’s current test strategy, as it is consistent with our recommendation. Specifically, it expands on the department’s prior strategy for component testing, which was limited to manufacturer self-certification of component conformance and informal observations of system components, by adding the use of independent government laboratories to test the components. We would emphasize, however, that regardless of the method used, it is important that confidence be gained in components prior to integrating them, which our recommendation recognizes. As our report states, component-level testing was not performed for Block 1 components prior to initiating integration testing. Federal guidance and the SBInet program office’s own Test and Evaluation Master Plan recognize the need to first test individual components to verify that the system modules work as intended (i.e., meet functional and performance requirements) before conducting integration testing. By adopting such a hierarchical approach to testing, the source of any system defects can be discovered and isolated sooner rather than later, thus helping to avoid the potential for expensive and time-consuming system rework.

We are sending copies of this report to the Chairmen and Ranking Members of the Senate and House Appropriations Committees and other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We will also send copies to the Secretary of Homeland Security, the Commissioner of U.S. Customs and Border Protection, and the Director of the Office of Management and Budget. In addition, this report will be available at no cost on the GAO Web site at http://www.gao.gov.

Should your offices have any questions on matters discussed in this report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine whether the Department of Homeland Security (DHS) (1) has defined the scope and timing of planned SBInet capabilities and how these capabilities will be developed and deployed, (2) is effectively defining and managing SBInet requirements, and (3) is effectively managing SBInet testing.

To determine the extent to which DHS has defined the scope and timing of planned SBInet capabilities and how these capabilities will be developed and deployed, we reviewed program documentation, such as the draft Systems Engineering Plan, the Systems Engineering Management Plan, the Operational Requirements Document, the Mission Engineering Process, the draft Test and Evaluation Master Plan, and the 2008 SBI Expenditure Plan to understand the SBInet engineering process and the scope and timing of planned deployments. We also interviewed SBInet officials and contractors to gain clarity beyond what was included in the program documentation and to obtain schedule information in the absence of an integrated master schedule for the program.

To determine if DHS is effectively defining and managing SBInet requirements, we reviewed relevant documentation, such as the Requirements Development and Management Plan, the Requirements Management Plan, the Configuration and Data Management Plan, the Operational Requirements Document, System of Systems A-Level Specification, B-2 Specifications, and Vendor Item Control Drawings, and compared them to industry best practices to determine the extent to which the program has effectively managed the systems requirements and maintained traceability backwards to high-level operational requirements and system requirements, and forward to system design and verification methods.

To assess reliability of the requirements data, we reviewed quality and access controls of the requirements database. We then randomly selected 59 requirements from a sample of 1,666 component requirements and traced them backwards to the system requirements and then to the operational requirements and forward to design requirements and verification methods. Because we followed a probability procedure based on random selection, we are 95 percent confident that each of the confidence intervals in this report will include the true values in the study population. We used statistical methods appropriate for audit compliance testing to estimate 95 percent confidence intervals for the traceability of requirements in our sample. In addition, we interviewed program and contractor officials involved in requirements management to understand their roles and responsibilities. We also visited a contractor development facility in Huntsville, Alabama, to understand the contractor’s role in requirements management and development and the use of its requirements management tool, known as the Dynamic Object-Oriented Requirements System (DOORS). In addition, we attended a demonstration of SBInet Rapid Application Development/Joint Application Design to understand how the users are involved in developing requirements.

To determine if DHS is effectively managing SBInet testing, we reviewed relevant documentation, such as the SBInet Test and Evaluation Master Plan, the Systems Integration Test Plan, the Quality Assurance Surveillance Plan, the Requirements Verification Plan, the Characterization Test Plan, and the Prime Mission Product Design, and compared them to relevant federal guidance to determine the extent to which the program has effectively managed its testing activities. We also interviewed SBInet officials to gain clarity beyond what was included in the program documentation and to obtain schedule information in the absence of a formal testing schedule. In addition, we visited a contractor facility in Huntsville, Alabama, to better understand the contractor’s role in testing activities and to observe the test lab and how testing is performed.

In addition, we visited the Tucson Sector Border Patrol Headquarters in Tucson, Arizona, to see the technology that was deployed as a prototype to understand the scope of the technology, how the Border Patrol agents use the technology, and future plans.

To assess data reliability, we reviewed related program documentation to substantiate data provided in interviews with knowledgeable agency officials, where available. For the information contained in the DHS independent study on SBInet, we interviewed the individuals responsible for conducting the review to understand their methodology, and determined that the information derived from this study was sufficiently reliable for the purposes of this report. We have made appropriate attribution indicating the data’s sources.

We performed our work at the U.S. Customs and Border Protection headquarters and contractor facilities in the Washington, D.C., metropolitan area; the Tucson Sector Border Patrol headquarters in Tucson, Arizona; and a contractor facility in Huntsville, Alabama. We conducted this performance audit from August 2007 to September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Deborah Davis (Assistant Director), Carl Barden, Neil Doherty, Lee McCracken, Jamelyn Payan, Karl Seifert, Sushmita Srikanth, Karen Talley, and Merry Woo made key contributions to this report.