Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering Key Technology Investment
Published: 2008-09-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security's (DHS) Secure Border Initiative (SBI) is a multiyear, multibillion-dollar program to secure the nation's borders through, among other things, new technology, increased staffing, and new fencing and barriers. The technology component of SBI, which is known as SBInet, involves the acquisition, development, integration, and deployment of surveillance systems and command, control, communications, and intelligence technologies. GAO was asked to determine whether DHS (1) has defined the scope and timing of SBInet capabilities and how these capabilities will be developed and deployed, (2) is effectively defining and managing SBInet requirements, and (3) is effectively managing SBInet testing. To do so, GAO reviewed key program documentation and interviewed program officials, analyzed a random sample of requirements, and observed operations of a pilot project.
section_title: What GAO Found
paragraphs: Important aspects of SBInet remain ambiguous and in a continued state of flux, making it unclear and uncertain what technology capabilities will be delivered, when and where they will be delivered, and how they will be delivered. For example, the scope and timing of planned SBInet deployments and capabilities have continued to change since the program began and, even now, are unclear. Further, the program office does not have an approved integrated master schedule to guide the execution of the program, and GAO's assimilation of available information indicates that the schedule has continued to change. This schedule-related risk is exacerbated by the continuous change in and the absence of a clear definition of the approach that is being used to define, develop, acquire, test, and deploy SBInet. The absence of clarity and stability in these key aspects of SBInet impairs the ability of the Congress to oversee the program and hold DHS accountable for program results, and it hampers DHS's ability to measure program progress. SBInet requirements have not been effectively defined and managed. While the program office recently issued guidance that defines key practices associated with effectively developing and managing requirements, such as eliciting user needs and ensuring that different levels of requirements and associated verification methods are properly aligned with one another, the guidance was developed after several key activities had been completed. In the absence of this guidance, the program has not effectively performed key requirements definition and management practices. For example, it has not ensured that different levels of requirements are properly aligned, as evidenced by GAO's analysis of a random probability sample of component requirements showing that a large percentage of them could not be traced to higher-level system and operational requirements. Also, some of SBInet's operational requirements, which are the basis for all lower-level requirements, were found by an independent DHS review to be unaffordable and unverifiable, thus casting doubt on the quality of lower-level requirements that are derived from them. As a result, the risk of SBInet not meeting mission needs and performing as intended is increased, as are the chances of expensive and time-consuming system rework. SBInet testing has not been effectively managed. For example, the program office has not tested the individual system components to be deployed to the initial deployment locations, even though the contractor initiated integration testing of these components with other system components and subsystems in June 2008. Further, while a test management strategy was drafted in May 2008, it has not been finalized and approved, and it does not contain, among other things, a clear definition of testing roles and responsibilities; a high-level master schedule of SBInet test activities; or sufficient detail to effectively guide project-specific test planning, such as milestones and metrics for specific project testing. Without a structured and disciplined approach to testing, the risk that SBInet will not satisfy user needs and operational requirements, thus requiring system rework, is increased.

section_title: Letter
paragraphs: Securing the nation’s borders from illegal entry of aliens and contraband  continues to be a major challenge because much of the 6,000 miles of  international borders with Canada and Mexico remains vulnerable to  unlawful activities. Although the Department of Homeland Security (DHS)  apprehends hundreds of thousands of people entering the country illegally  each year, many more unauthorized entrants go undetected.
As we have reported, previous attempts to acquire and deploy surveillance  technologies along the nation’s borders to assist in detecting and  responding to illegal entries have not been successful. Specifically, the  former Immigration and Naturalization Service’s Integrated Surveillance  Intelligence System, begun in the late 1990s, was difficult and expensive to  maintain; it provided limited command, control, and situational awareness  capability; and its component systems were not integrated. In response,  DHS established the America’s Shield Initiative in 2004, but this program  was halted in 2005 because of the program’s limited scope and the  department’s shift in strategy for achieving border security and interior  enforcement goals. In November 2005, DHS launched the Secure Border  Initiative (SBI), a multiyear, multibillion-dollar program to secure the  nation’s borders through enhanced use of surveillance technologies,  increased staffing levels, improved infrastructure, and increased domestic  enforcement of immigration laws. One component of SBI, known as  SBInet, is focused on the acquisition and deployment of surveillance and  communications technologies. This program is managed by the SBInet  System Program Office within U.S. Customs and Border Protection (CBP).
Because of the size and complexity of SBInet, and the problems  experienced by its predecessors, you asked us to determine whether DHS  (1) has defined the scope and timing of planned SBInet capabilities and  how these capabilities will be developed and deployed, (2) is effectively  defining and managing SBInet requirements, and (3) is effectively  managing SBInet testing.
To accomplish our objectives, we reviewed key program documentation,  including guidance, plans, and requirements and testing documentation. In  cases where such documentation was not available, we interviewed  program officials about the development of capabilities and the  management of requirements and testing. We then compared this  information to relevant federal system acquisition guidance. We also  analyzed a random probability sample of system requirements and  observed operations of the initial SBInet project.
We conducted this performance audit from August 2007 to September 2008  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Further details of our  objectives, scope, and methodology are included in appendix I.
subsections: section_title: Background
paragraphs: CBP’s SBI program is to leverage technology, tactical infrastructure, and  people to allow CBP agents to gain control of the nation’s borders. Within  SBI, SBInet is the program for acquiring, developing, integrating, and  deploying an appropriate mix of (1) surveillance technologies, such as  cameras, radars, and sensors, and (2) command, control, communications,  and intelligence (C3I) technologies.
The initial focus of SBInet has been on addressing the requirements of  CBP’s Office of Border Patrol, which is responsible for securing the  borders between the established ports of entry. The longer-term SBInet  systems solution also is to address requirements of CBP’s two other major  components—the Office of Field Operations, which controls vehicle and  pedestrian traffic at the ports of entry, and the Office of Air and Marine  Operations, which operates helicopters, fixed-wing aircraft, and marine  vessels used in securing the borders. Figure 1 provides a high-level,  operational concept of the long-term SBInet systems solution.
Surveillance technologies are to include a variety of sensor systems that  improve CBP’s ability to detect, identify, classify, and track items of  interest along the borders. Unattended ground sensors are to be used to  detect heat and vibrations associated with foot traffic and metal  associated with vehicles. Radars mounted on fixed and mobile towers are  to detect movement, and cameras on fixed and mobile towers are to be  used to identify, classify, and track items of interest detected by the  ground sensors and the radars. Aerial assets are also to be used to provide  video and infrared imaging to enhance tracking of targets.
The C3I technologies are to include software and hardware to produce a  Common Operating Picture (COP)—a uniform presentation of activities  within specific areas along the border. The sensors, radars, and cameras  are to gather information along the border, and the system is to transmit  this information to the COP terminals located in command centers and  agent vehicles and assemble this information to provide CBP agents with  border situational awareness. More specifically, the COP technology is to  allow agents to (1) view data from radars and sensors that detect and  track movement in the border areas, (2) control cameras to help identify  and classify illegal entries, (3) correlate entries with the positions of  nearby agents, and (4) enhance tactical decision making regarding the  appropriate response to apprehend an entry, if necessary.
Initially, COP information is to be distributed to terminals in command  centers. We observed that these terminals look like a standard computer  workstation with multiple screens. From this workstation, an operator is  to be able to view an area of interest in several different ways. For  example, the operator is to see different types of maps, satellite images,  and camera footage on the multiple screens. The operator is also to be  able to move the cameras to track images on the screen. According to  program officials, eventually, when the radars detect potential items of  interest, the system is to automatically move the cameras so the operator  does not always need to initiate the search in the area.
We observed that COP data are also available on laptop computers, known  as mobile data terminals, mounted in select agent vehicles in the field.  These terminals are to enable field agents to see information similar to  that seen by command center operators. Eventually, the COP technology is  to be capable of providing distributed surveillance and tactical decision- support information to other DHS agencies and stakeholders external to  DHS, such as local law enforcement. Figure 2 shows examples of COP  technology in a command station and an agent vehicle.
The first SBInet capabilities were deployed under a pilot or prototype  effort known as “Project 28.” Project 28 is currently operating along 28  miles of the southwest border in the Tucson Sector of Arizona. Project 28  was accepted by the government for deployment 8 months behind  schedule (in February 2008); this delay occurred because the contractor- delivered system did not perform as intended. As we have previously  reported, reasons for Project 28 performance shortfalls and delays include  the following:    System requirements were not adequately defined, and users were not  involved in developing the requirements.
System integration testing was not adequately performed.
Contractor oversight was limited.
Project scope and complexity were underestimated.
To manage SBInet, DHS established a program office within CBP. The  program office is led by a program manager and deputy program managers  for program operations and mission operations. The program manager is  responsible for the execution of the program, including developing,  producing, deploying, and sustaining the system to meet the users’ needs.  Among other things, this includes developing and analyzing requirements  and system alternatives, managing system design and development,  evaluating the system’s operational effectiveness, and managing program  risk.
subsections: section_title: SBInet Program Office Description of Its Life Cycle Management Approach
paragraphs: A system life cycle management approach typically consists of a series of  phases, milestone reviews, and related processes to guide the acquisition,  development, deployment, and operation and maintenance of a system.  Among other things, the phases, reviews, and processes cover such  important life cycle activities as requirements development and  management, design, software development, and testing. Based on  available program documentation, augmented by program official  briefings and statements, key aspects of the SBInet system life cycle  management approach are described below.
subsections: section_title: Approach Is Intended to Deliver Incremental Capabilities through a Series of Blocks
paragraphs: In general, SBInet surveillance systems are to be acquired through the  purchase of commercially available products, while the COP systems  involve development of new, customized systems and software. Together,  both categories are to form a deployable increment of the SBInet  capabilities, which the program office refers to as a “block.” Each block is  to include a release or version of the COP.
SBInet documentation shows that the program office is acquiring the  blocks incrementally using a “spiral” approach, under which an initial  system capability is to be delivered based on a defined subset of the  system’s total requirements. This approach is intended to allow CBP  agents access to new technological tools sooner rather than later for both  operational use and feedback on needed enhancements or changes.  Subsequent spirals or iterations of system capability are to be delivered  based on feedback and unmet requirements, as well as the availability of  new technologies. Figure 3 illustrates conceptually how the different  capabilities are to come together to form a block and how future blocks  are to introduce more capabilities.
The approach used to design and develop SBInet system capabilities for  each block includes such key activities as requirements development,  system design, system acquisition and development, and testing. The  approach, as explained by program officials and depicted in part in various  documents, also includes various reviews, or decision points, to help  ensure that these activities are being done properly and that the system  meets user needs and requirements. These reviews are to be used in  developing both the overall SBInet Block 1 capability and the COP  software. Table 1 provides a high-level description of the major reviews  that are to be performed in designing and developing the system prior to  deployment to the field and in the order that they occur.
Before a set of capabilities (i.e., block) is deployed to a specific area or  sector of the border, activities such as site selection, surveys, and  environmental impact assessments are conducted to determine the area’s  unique environmental requirements. The border area that receives a given  block, or set of system capabilities, is referred to as a “project.” Each  project is to have a given block configured to its unique environmental  requirements, referred to as a project “laydown.”
The deployment approach is to include such key activities as requirements  development, system design, project laydown, integration, testing, and  installation. The deployment approach is also to entail various reviews, or  decision points, to help ensure that these activities are being done  properly and that the system meets user needs and requirements. Table 2  provides a high-level description of the major reviews that are to be part of  project laydown in the order that they occur.
Among the key processes provided for in the SBInet system life cycle  management approach are processes for developing and managing  requirements and for managing testing activities. With respect to  requirements development and management, SBInet requirements are to  consist of a hierarchy of six types of requirements, with the high-level  operational requirements at the top. These high-level requirements are to  be decomposed into lower-level, more detailed system, component,  design, software, and project requirements. Having a decomposed  hierarchy of requirements is a characteristic of complex information  technology (IT) projects. The various types of SBInet requirements are  described in table 3. Figure 4 shows how each of these requirements relate  to or are derived from the other requirements.
With respect to test management, SBInet testing consists of a sequence of  tests that are intended to verify first that individual system parts meet  specified requirements, and then verify that these combined parts perform  as intended as an integrated and operational system. Such an incremental  approach to testing is a characteristic of complex IT system acquisition  and development efforts. Through such an approach, the source of defects  can be isolated more easily and sooner, before they are more difficult and  expensive to address. Table 4 summarizes these tests.
section_title: Limited Definition of SBInet Deployments, Capabilities, Schedule, and Life Cycle Management Process Increases Program’s Exposure to Risk
paragraphs: Important aspects of SBInet remain ambiguous and in a continued state of  flux, making it unclear and uncertain what technology capabilities will be  delivered, when and where they will be delivered, and how they will be  delivered. For example, the scope and timing of planned SBInet  deployments and capabilities have continued to change since the program  began and, even now, remain unclear. Further, the approach that is being  used to define, develop, acquire, test, and deploy SBInet is similarly  unclear and has continued to change. According to SBInet officials,  schedule changes are due largely to an immature system design, and the  lack of a stable development approach is due to insufficient staff and  turnover. The absence of clarity and stability in these key aspects of  SBInet introduces considerable program risks, hampers DHS’s ability to  measure program progress, and impairs the ability of the Congress to  oversee the program and hold DHS accountable for program results.
subsections: section_title: Scope and Timing of Planned Deployments and Capabilities Are Not Clear and Stable
paragraphs: One key aspect of successfully managing large IT programs, like SBInet, is  establishing program commitments, including what capabilities are to be  deployed and when and where they are to be deployed. Only when such  commitments are clearly established can program progress be measured  and can responsible parties be held accountable.
The scope and timing of planned SBInet deployments and capabilities that  are to be delivered have not been clearly established, but rather have  continued to change since the program began. Specifically, as of  December 2006, the SBInet System Program Office planned to deploy an  “initial” set of capabilities along the entire southwest border by late 2008  and planned to deploy a “full” set of operational capabilities along the  southern and northern borders (a total of about 6,000 miles) by late 2009.  As of March 2007, the program office had modified its plans, deciding  instead to deploy the initial set of capabilities along the southwest border  by the end of fiscal year 2007 (almost a year earlier than originally  planned) and delayed the deployment of the final set of capabilities for the  southern and northern borders until 2011.
In March 2008, the program office again modified its deployment plans,  this time significantly reducing the area to which SBInet capabilities are to  be deployed. At this time, DHS planned to complete deployments to three  out of nine sectors along the southwest border—specifically, to Tucson  Sector by 2009, Yuma Sector by 2010, and El Paso Sector by 2011.  According to program officials, other than the dates for the Tucson, Yuma,  and El Paso Sectors, no other deployment dates have been established for  the remainder of the southwest or northern borders. (Figure 5 shows the  changes in the planned deployment areas.)
The figure also shows the two sites within the Tucson Sector, Tucson 1  and Ajo 1, at which an initial Block 1 capability is to be deployed.  Together, these two deployments cover 53 miles of the 1,989-mile-long  southern border. According to the March 2008 SBI expenditure plan and  agency documentation as of June 2008, these two sites were to have been  operational by the end of 2008. However, as of late July 2008, program  officials reported that the deployment schedule for these two sites has  again been modified, and they will not be operational until “sometime” in  2009. According to program officials, the slippage in the deployment  schedule is due to the need to complete environmental impact assessment  documentation for these locations. The slippages in the dates for the first  two Tucson deployments, according to a program official, will, in turn,  delay subsequent Tucson deployments, although revised dates for these  subsequent deployments have not been set.
Just as the scope and timing of planned deployments have not been clear  and have changed over time, the specific capabilities that are to be  deployed have been unclear. For example, in April 2008, program officials  stated that they would not know which of the SBInet requirements would  be met by Block 1 until the Critical Design Review, which at that time was  scheduled for June 2008. At that time, program officials stated that the  capabilities to be delivered would be driven by the functionality of the  COP. In June, the review was held, but according to available  documentation, the government did not consider the design put forth by  the contractor to be mature. As a result, the system design was not  accepted in June as planned. Among the design limitations found was a  lack of evidence that the system requirements were used as the basis for  the Tucson 1 and Ajo 1 design, lack of linkage between the performance of  surveillance components and the system requirements, and incomplete  definition of system interfaces. As of late July 2008, these issues were  unresolved, and thus the design still had not been accepted. In addition, in  late July 2008, agency officials stated that the capabilities to be delivered  will be driven by the functionality of the surveillance components, not the  COP.
In addition, the design does not provide key capabilities that are in  requirements documents and were anticipated to be part of the Block 1  deployments to Tucson 1 and Ajo 1. For example, the first deployments of  Block 1 will not have the mobile data terminals in border patrol vehicles,  even though (1) such terminals are part of Project 28 capabilities and (2)  workshops were held with the users in February 2008 and June 2008 to  specifically define the requirements for these terminals for inclusion in  Block 1. According to program officials, these terminals will not be part of  Block 1 because the wireless communications infrastructure needed to  support these terminals will not be available in time for the Tucson 1  deployment. Rather, they expect the wireless infrastructure to be ready  “sometime” in 2009 and said that they will include the mobile data  terminals in Block 1 deployments when the infrastructure is ready.  Without the mobile data terminals, agents will not be able to obtain key  information from their vehicles, such as maps of activity in a specific area,  incident reports, and the location of other agents in the area. Instead, the  agents will have to use radios to communicate with the sector  headquarters to obtain this information.
In addition, program officials told us that a number of other requirements  cannot be included in the Block 1 version of the COP, referred to as  version 0.5, due to cost and schedule issues. However, we have yet to  receive a list of these requirements. According to program officials, they  hope to upgrade the COP in 2009 to include these requirements.
Without clearly establishing program commitments, such as capabilities to  be deployed and when and where they are to be deployed, program  progress cannot be measured and responsible parties cannot be held  accountable.
section_title: Program Schedule Is Unsettled
paragraphs: Another key aspect of successfully managing large programs like SBInet is  having a schedule that defines the sequence and timing of key activities  and events and is realistic, achievable, and minimizes program risks.  However, the program office does not yet have an approved integrated  master schedule to guide the execution of SBInet, and according to  program officials, such a schedule has not been in place since late 2007. In  the absence of an approved integrated master schedule, program officials  stated in mid-August 2008 that they have managed the program largely  using task-order-specific baselined schedules, and have been working to  create a more integrated approach. A program official also stated that they  have recently developed an integrated master schedule but that this  schedule is already out of date and undergoing revision. For example, the  deployment of the SBInet system to the Tucson Sector will not be  completed in 2009 as planned.
To understand where the program is relative to established commitments,  we analyzed schedule-related information obtained from other available  program documents, briefings, and interviews. In short, our analysis  shows a schedule in which key activities and events are subject to  constant change, as depicted in the following two figures. Figure 6 shows  the changes to the schedule of planned and held reviews and anticipated  deployment dates, and figure 7 shows the changes to the schedule of  testing activities.
In May 2008, program officials stated that the schedule changes were due  largely to the fact that the contractor had not yet provided a satisfactory  system-level design. They also noted that the contractor’s workforce has  experienced considerable turnover, including three different program  managers and three different lead system engineers. They also stated that  the System Program Office has experienced attrition, including turnover in  the SBInet Program Manager position. Without stability and certainty in  the program’s schedule, program cost and schedule risks increase, and  meaningful measurement and oversight of program status and progress  cannot occur, in turn, limiting accountability for results.
section_title: SBInet Life Cycle Management Approach Has Not Been Clearly Defined and Has Continued to Change
paragraphs: System quality and performance are in large part governed by the  processes followed in developing and acquiring the system. To the extent  that a system’s life cycle management approach and related development  and acquisition processes are well-defined, the chances of delivering  promised system capabilities and benefits on time and within budget are  increased. To be well-defined, the approach and processes should be fully  documented, so that they can be understood and properly implemented by  those responsible for doing so.
The life cycle management approach and processes being used by the  SBInet System Program Office to manage the definition, design,  development, testing, and deployment of system capabilities has not been  fully and clearly documented. Rather, what is defined in various program  documents is limited and not fully consistent across these documents.  Moreover, in discussions with agency officials to clarify our understanding  of these processes, new terms and processes have been routinely  introduced, indicating that the processes are continuing to evolve. Agency  officials acknowledge that they are still learning about and improving their  processes. Without a clearly documented and universally understood life  cycle management approach and supporting processes, the program is at  increased risk of not meeting expectations.
Key program documentation that is to be used to guide acquisition and  development activities, including testing and deployment activities, is  incomplete, even though SBInet acquisition and development are already  under way. For example, officials have stated that they are using the draft  Systems Engineering Plan, dated February 2008, to guide the design,  development, and deployment of system capabilities, and the draft Test  and Evaluation Master Plan, dated May 2008, to guide the testing  process—but both of these documents are lacking sufficient information  to clearly guide system activities, as the following examples explain:    The Systems Engineering Plan includes a diagram of the engineering  process; however, the steps of the process and the gate reviews are not  defined or described in the text of the document. For example, this  document does not contain sufficient information to understand what  occurs at key reviews, such as the Preliminary Design Review, the Critical  Design Review, and the Test Readiness Review.
The Test and Evaluation Master Plan describes in more detail some of the  reviews that are not described in the Systems Engineering Plan, but the  reviews included are not consistent between the documents. For example,  the Test and Evaluation Master Plan includes a System Development and  Demonstration Review that is not listed in the Systems Engineering Plan.  In addition, it is not clear from the Test and Evaluation Master Plan how  the reviews fit into the overall engineering process.
Statements by program officials responsible for system development and  testing activities, as well as briefing materials and diagrams that these  officials provided, did not add sufficient clarity to describe a well-defined  life cycle management approach. Moreover, these descriptions were not  always consistent with what was contained in the documentation, as the  following examples demonstrate:    Component testing is not described in the Test and Evaluation Master Plan  in a manner consistent with how officials described this testing.  Specifically, while the plan states that components will be tested against  the corresponding component requirements to ensure all component  performance can be verified, program officials stated that not all  components will undergo component testing. Instead, they said that  testing is not required if component vendors submit a certificate of  compliance for certain specifications.
Functional qualification testing was described to us by program officials in  July 2008 as a type of testing to be performed during software  development activities. However, this type of testing is not defined in  available program documentation, and it was not included in any versions  of the documentation associated with the life cycle management approach  and related engineering processes.
Certain reviews specified in documentation of the life cycle management  process are not sufficiently defined. For example, the Systems Engineering  Plan shows a Production Readiness Review as part of the system-level  process and an Operational Readiness Review as part of the project-level  process. However, program officials stated that these reviews are not  completely relevant to SBInet because they are targeted for informational  systems rather than tactical support systems, such as SBInet. According to  the officials, they are in the process of determining how to apply the  reviews to SBInet. For example, in July 2008, officials reported that they  may move the Production Readiness Review from the system-level set of  activities, as shown in the Systems Engineering Plan, to the project-level  set of activities. Program officials also stated that they are working to  better define these reviews in the Systems Engineering Plan.
Program officials told us that the SBInet life cycle management approach  and related engineering processes are understood by both government and  contractor staff through the combination of the draft Systems Engineering  Plan and government-contractor interactions during design meetings.  Nevertheless, they acknowledged that the approach and processes are not  well documented, citing a lack of sufficient staff to both document the  processes and oversee the system’s design, development, testing, and  deployment. They also told us that they are adding new people to the  project with different acquisition backgrounds, and that they are still  learning about, evolving, and improving the approach and processes.  According to these officials, a revised and updated Systems Engineering  Plan should be finalized by September 2008.
The lack of definition and stability in the approach and related processes  being used to define, design, develop, acquire, test, and deploy SBInet  introduce considerable risk that both the program officials and contractor  staff will not understand what needs to be done when, and thus that the  program will not consistently employ disciplined and rigorous methods.  Without the use of such methods, the risk of delivering a system that does  not meet operational needs and does not perform as intend is increased.  Moreover, without a well-defined approach and processes, it is difficult to  gauge progress and thus promote performance and accountability for  results.
section_title: Limitations of SBInet Requirements Development and Management Efforts Increase Program Risk
paragraphs: Well-defined and managed requirements are a cornerstone of effective  system development and acquisition. According to recognized guidance,  documenting and implementing a disciplined process for developing and  managing requirements can help reduce the risks of developing a system  that does not meet user needs, cannot be adequately tested, and does not  perform or function as intended. Such a process includes, among other  things, eliciting user needs and involving users in the development  process; ensuring that requirements are complete, feasible, verifiable, and  approved by all stakeholders; documenting and approving the  requirements to establish a baseline for subsequent development and  change control; and ensuring that requirements are traceable both back to  operational requirements and forward to detailed system requirements and  test cases.
To the program office’s credit, it recently developed guidance for  developing and managing requirements that is consistent with recognized  leading practices. For example, the program’s guidance states that a  requirements baseline should be established and that requirements are to  be traceable both back to higher-level requirements and forward to  verification methods. However, this guidance was not finalized until  February 2008 and thus was not used in performing a number of key  requirements-related activities.
In the absence of well-defined guidance, the program’s efforts to  implement leading practices for developing and managing requirements  have been mixed. For example, while the program has elicited user needs  as part of its efforts to develop high-level operational requirements, it has  not baselined all requirements. Further, it has not ensured that the  operational requirements were, for example, verifiable, and it has not  made certain that all of the different levels of requirements are aligned to  one another. As a result, the risk of SBInet not meeting mission needs and  performing as intended is increased, as are the chances of expensive and  time-consuming system rework.
subsections: section_title: Requirements Development and Management Guidance Is Consistent with Leading Practices but Was Developed After Many Requirements Activities Were Performed
paragraphs: The SBInet program office has developed guidance for developing and  managing requirements that is generally consistent with recognized  leading practices. According to these practices, effectively developing and  managing requirements includes, among other things, eliciting users’ needs  early in the development process and involving them throughout the  process; ensuring that requirements are complete, feasible, verifiable, and  approved by all stakeholders; documenting and approving the  requirements to establish a baseline for subsequent development and  change control; and ensuring that requirements are traceable both back to  operational requirements and forward to detailed system requirements and  test cases.
In February 2008, the program office approved its SBInet Requirements  Development and Management Plan. According to the plan, its purpose is  to describe a comprehensive approach to developing and managing  requirements for the SBInet program. Our analysis of this plan shows that  it is consistent with leading practices. For example, the plan states that    users should provide input to the requirements definition process through  early and ongoing participation in integrated product teams, user  conferences, and other requirements-gathering and verifying activities; requirements developers are to ensure that requirements are complete,  unambiguous, achievable, verifiable, and not redundant;    a requirements baseline should be created to provide a common  understanding of the system to be built and to prevent deviations to the  requirements from entering during design, development, or testing; and    bidirectional traceability both back to higher-level requirements and  forward to detailed test methods should be established and maintained  and that the requirements management team is responsible for  maintaining the requirements management database and holding the  contractor responsible for any traceability issues.
Moreover, the plan defines procedures and steps for accomplishing each  of these goals. For example, the procedure for requirements development  outlines the purpose of the procedure, who is involved, what  documentation is necessary to begin the procedure, and what outputs are  expected at the end. The procedure then describes 16 steps necessary to  achieve the requirements development activities. A separate procedure is  provided for requirements verification and validation.
However, the plan was not approved until after key SBInet requirements  documents were written and baselined. Specifically, user operational  requirements were approved and baselined in March 2007; system  requirements were first baselined in March 2007; and several component  requirements were baselined in June, August, and October 2007. As a  result, the plan was not used to guide these requirements development and  management efforts.
section_title: Program Office Has Taken Steps to Involve Users in Developing High-Level Requirements
paragraphs: As noted above, one of the leading practices associated with effective  requirements development and management is engaging system users  early and continuously. In doing so, the chances of defining, designing, and  delivering a system that meets their needs and performs as intended are  increased.
In developing the operational requirements, the System Program Office  involved SBInet users in a manner that is consistent with leading practices  and that reflects lessons learned from Project 28. Specifically, it conducted  requirements-gathering workshops from October 2006 through April 2007  to ascertain the needs of Border Patrol agents. In addition, it established  work groups in September 2007 to inform the next revision of the  operational requirements by soliciting input from both the Office of Air  and Marine Operations and the Office of Field Operations. Further, to  develop the COP technology for SBInet, the program office is following a  software development methodology that allows end users to be directly  involved in software development activities and, thereby, permits software  solutions to be tailored to users’ needs.
Through such efforts to identify and elicit user needs in developing high- level requirements, the chances of developing a system that will meet user  needs are increased.
section_title: Not All Levels of Requirements Have Been Adequately Baselined
paragraphs: The creation of a requirements baseline is important for providing a stable  basis for system design, development, and testing. Such a baseline  establishes a set of requirements that have been formally reviewed and  agreed on and thus serve as the basis for further development or delivery.  Until requirements are baselined, they remain unclear and subject to  considerable and uncontrolled change, which in turn makes system  design, development, testing, and deployment efforts equally uncertain.  According to SBInet program officials, the SBInet Requirements  Development and Management Plan, and leading practices, requirements  should be baselined before key system design activities begin, since the  requirements are intended to inform, guide, and constrain the system’s  design.
For SBInet, while many of the requirements have been baselined, two  types have not yet been baselined. According to the System Program  Office, the operational requirements and the system requirements were  approved and baselined in March 2007. In addition, various system  component requirements were baselined in June, August, and October of  2007. However, the program had not baselined its COP software  requirements as of July 2008, although according to program officials, the  COP has been designed and is under development. Further, it has yet to  baseline its project-level requirements, which define the requirements for  the system configuration to be deployed to a specific geographical area,  such as Tucson 1.
With respect to the COP, requirements for the software and hardware had  not been baselined as of the end of July 2008, despite the fact that a  combined Preliminary Design Review and Critical Design Review for the  COP was held in February 2008 and a Critical Design Review for the  system as whole was held in June 2008. According to agency officials and  the SBInet Requirements Development and Management Plan,  requirements should be baselined before the Critical Design Review.  Regardless, program officials state that the contractor has developed  several “builds” (i.e., versions) of the COP, which are currently being  tested. According to program officials, the requirements were not  complete because certain interface requirements had not yet been  completely identified and defined. Without baselined requirements, the  basis of the system design and the degree to which it satisfies  requirements are unclear. Moreover, the risk of the design not aligning to  requirements is increased. According to the results of the Critical Design  Review, this risk was realized. Specifically, the System Program Office  notified the contractor that there was no evidence linking the performance  of surveillance components to the system requirements, that the review  could not be completed until the interface requirements had been  finalized, and that a mature design had not been presented at the review.
With respect to project-level (i.e., geographic area) deployment  requirements, baselined requirements do not yet exist. Specifically,  requirements for the Tucson Sector, which includes Tucson 1 and Ajo 1,  have yet to be baselined. According to the SBInet Requirements  Development and Management Plan, requirements should be baselined  before the Project Requirements Review, and a new requirements baseline  should be created following the subsequent Deployment Design Review.  However, project-level requirements were not baselined at a Project  Requirements Review held for the Tucson Sector Project in March 2007 or  at a Deployment Design Review in June 2007. Officials stated that this is  because the plan was not approved until February 2008 and thus was not  in effect. However, since the plan became effective, Deployment Design  Reviews were held for Tucson 1 and Ajo 1 in April 2008 and May 2008,  respectively, but the project-level requirements were not baselined.
Despite the absence of baselined requirements, the System Program Office  has proceeded with development, integration, and testing activities for the  Block 1 capabilities to be delivered to Tucson 1 and Ajo l. As a result, it  faces an increased risk of deploying systems that do not align well with  requirements and thus may require subsequent rework. The lack of project  requirements has already had an effect on testing activities. Specifically,  the draft system integration test plan notes that, without project  requirements, testing will have to be guided by a combination of other  documents, including engineering development requirements, related  component requirements, and architectural design documents.
section_title: Baselined Operational Requirements Used to Inform Lower-Level Requirements Are Limited
paragraphs: As stated above, one of the leading practices for developing and managing  requirements—which is reflected in the program office’s own plan—is that  requirements should be sufficiently analyzed to ensure that the  requirements are, among other things, complete, unambiguous, and  verifiable. However, an independent review of SBInet operational  requirements reported numerous problems. Specifically, a review of the  SBInet program commissioned by the Office of the Deputy Secretary of  Homeland Security found that several requirements were unaffordable and  unverifiable. Examples of these requirements include the following:    Allow for complete coverage of the specified area or zone to be surveilled.
Maximize intended deterrence and minimize countermeasure  effectiveness.
Function with high reliability under reasonably foreseeable circumstances.
Reliably provide the appropriate power and bandwidth at the least cost  that will support the demand.
In April 2008, a program official stated that the operational requirements  document is currently being rewritten to address concerns raised by this  review. For example, we were told that certain system performance  requirements are being revised in response to a finding that the  requirements are insufficient to ensure delivery of a properly functioning  system. Program officials stated that they expect to finalize the revised  operational requirements document in October 2008.
However, given the number and types of problems associated with the  operational requirements—which are the program’s most basic customer  requirements and form the basis for all lower-level requirements—it is  unclear how the system, component, and software requirements can be  viewed as verifiable, testable, or affordable. Until these problems are  addressed, the risk of building and deploying a system that does not meet  mission needs and customer expectations is increased, which in turn  increases the chances of expensive and time-consuming system rework.
section_title: SBInet Requirements Have Not Been Sufficiently Aligned
paragraphs: As noted above, one of the leading practices associated with developing  and managing requirements is maintaining bidirectional traceability from  high-level operational requirements through detailed low-level  requirements to test cases. The SBInet Requirements Development and  Management Plan recognizes the importance of traceability, stating that a  traceability relationship should exist among the various levels of  requirements. For example, it states that operational requirements should  trace to the system requirements, which in turn should trace to component  requirements. Further, it states that component requirements should trace  to design requirements and to a verification method. In addition, the  SBInet System Program Office established detailed guidance for  populating and maintaining the requirements database for maintaining  linkages among the various levels of requirements and test verification  methods.
To provide for requirements traceability, the prime contractor established  such a requirements management database. However, the reliability of the  requirements in this database is questionable, and the SBInet System  Program Office has not effectively overseen the contractor’s management  of requirements through this database. Specifically, we attempted to trace  requirements in the version of this database that the program office  received in March 2008 and were unable to trace large percentages of  component requirements to either higher-level or lower-level  requirements. For example, an estimated 76 percent (with a 95 percent  degree of confidence of being between 64 and 86 percent) of the  component requirements that we randomly sampled could not be traced to  the system requirements and then to the operational requirements. In  addition, an estimated 20 percent (with a 95 percent degree of confidence  of being between 11 and 33 percent) of the component requirements in our  sample failed to trace to a verification method. See table 5 for the failure  rates for each of our tracing analyses, along with the related confidence  intervals.
While program officials could not explain the reason for this lack of  traceability in most cases, they did attribute the 100-percent failure in  tracing component requirements to the design requirements to the  absence of any design requirements in the program office’s copy of the  database.
A contributing factor to the program office’s inability to explain why  requirements were not traceable is its limited oversight of the contractor’s  efforts to manage requirements through this database. According to  program officials, the contractor created the SBInet requirements  management database in December 2006, but the program office did not  receive a copy of the database until March 2008, despite requests for it  beginning in fall 2007. In early May 2008, the Chief Engineer told us the  contractor had been reluctant to provide the database because it viewed  the database’s maturity level as low. Moreover, the program office’s direct  access to the database had not been established because of security  issues, according to this official.
Following our efforts to trace requirements, the program office obtained  direct access to the contractor’s database and initiated efforts with the  contractor to resolve the traceability gaps. However, program officials told  us that they are still not certain that the database currently contains all of  the system requirements or even the reduced Block 1 system  requirements. As a result, they did not rely on it during the recent Critical  Design Review to verify requirements traceability. Instead, they said that  manual tracking methods were used.
Without ensuring that requirements are fully traceable, the program office  does not have a sufficient basis for knowing that the scope of the  contractor’s design, development, and testing efforts will produce a  system solution that meets operational needs and performs as intended.
As a result, the risk of expensive and time-consuming system rework is  increased.
section_title: Limitations in Key SBInet Testing and Test Management Activities Increase Program Risk
paragraphs: To be effectively managed, testing should be planned and conducted in a  structured and disciplined fashion. This includes, among other things,  having an overarching test plan or strategy as a basis for managing system  testing, developing well-defined and approved plans for executing testing  activities, and testing individual system components to ensure that they  satisfy defined requirements prior to integrating them into the overall  system.
The SBInet System Program Office is not effectively managing its testing  activities. Specifically, it has not tested individual system components  prior to integrating these components with other components and the COP  software. In addition, although the program’s draft Test and Evaluation  Master Plan is currently being used as the program’s overall strategy to  manage SBInet testing, the plan is incomplete and unclear with respect to  several test management functions. As a result, the chances of SBInet  testing being effectively performed are reduced, which in turn increases  the risk that the delivered and deployed system will not meet operational  needs and not perform as intended.
subsections: section_title: System Integration Testing Has Not Been Adequately Planned or Executed
paragraphs: To be effectively managed, relevant federal guidance states that testing  should, among other things, be governed by a well-defined and approved  plan, and it should be executed in accordance with this plan. Further,  integration testing should be preceded by tests of system components  (whether acquired or developed) that are to be integrated to form the  overall system. Once the components are tested to ensure that they satisfy  defined requirements, the integrated system can be tested to verify that it  performs as required. For SBInet, this has not occurred. As a result, the  risk of the system not meeting operational needs and not performing as  intended is increased, which in turn is likely to introduce the need for  expensive and time-consuming system rework.
The SBInet Systems Program Office reports that it began Block 1 system  integration testing in June 2008. However, it still does not have an  approved system integration test plan. Specifically, the system’s Critical  Design Review, which was held in June 2008, found numerous problems  with the contractor’s plan for system integration testing, and as a result,  the program office did not accept the plan. Examples of problems were  that the test plan refers to other test plans that the contractor had yet to  deliver and the plan identified system components that were not expected  to be part of Block 1. As a result, the program office decided that the  system integration plan needed to be revised to document and describe  the full set of actual testing activities that were to occur, including  identifying where and to what level the different phases of integration  tests would occur.
Notwithstanding these problems and the absence of an approved plan, the  contractor began integration testing in June 2008. According to program  officials, this was necessary to meet the tight time frames in the schedule.  However, without an accepted system integration test plan in place,  testing cannot be effectively managed. For example, the adequacy of the  test scope cannot be assured, and the progress in completing test activities  cannot be measured. As a result, there is an increased risk that the  delivered system will not meet operational needs and will not perform as  intended and that expensive and time-consuming system rework will be  required.
Moreover, the SBInet draft Test and Evaluation Master Plan describes  system integration testing as first testing individual components to verify  that the smallest defined module of a system works as intended (i.e.,  meets functional and performance requirements). This allows defects with  individual components to be identified and corrected before they are  integrated with other system components. Once the components are  tested, their respective hardware and software interfaces are to be tested  before subsystems are tested in combination with the COP software. Such  an incremental approach to testing permits system defects to be found and  addressed before system components are integrated and component  problems become more expensive and time-consuming to correct.
However, the SBInet System Program Office has not performed individual  component testing as part of integration testing. As of July 2008, agency  officials reported that component-level tests had not been completed and  were not scheduled to occur. Instead, officials stated that Block 1  components were evaluated based on what they described as “informal  tests” (i.e., contractor observations of cameras and radar suites in  operation at a National Guard facility in the Tucson Sector) and stated that  the contractors’ self-certification that the components meet functional and  performance requirements was acceptable. However, this approach is not  consistent with the Test and Evaluation Master Plan. Moreover, program  officials acknowledged that this approach did not verify if the individual  components in fact met requirements. Nevertheless, they said that they  have recently modified their definition of component testing to allow the  contractor’s certification to be used.
In our view, relying solely on contractor certification is not a sufficient  substitute for component testing—as defined in the Test and Evaluation  Master Plan—because it increases the risk that components, and thus the  entire system, will not perform as intended. This risk is already starting to  be realized. Specifically, the results of the Block 1 Critical Design Review  performed in early in June 2008 show that design documents did not link  components’ performance to the system requirements.
section_title: Key Test Management Activities Have Not Been Adequately Defined and Addressed
paragraphs: To ensure that system testing is effectively performed, federal guidance  provides for having an overarching test plan or strategy to use as a basis  for managing system testing. Among other things, this test management  plan should define the schedule of high-level test activities in sufficient  detail to allow for more detailed test planning and execution to occur and  to ensure that test progress can be tracked and results can be reported and  addressed. The plan should also define the roles and responsibilities of the  various groups responsible for different levels of testing and include a  description of how the test organization manages and oversees these  groups in their activities.
The SBInet Test and Evaluation Master Plan, which documents the  program’s test strategy and is being used to manage system testing, has yet  to be approved by the SBInet Acting Program Manager. As of July 2008,  program officials told us that they did not expect the draft plan to be  approved until August 2008, even though testing activities began in June  2008.
Moreover, the draft Test and Evaluation Master Plan is not complete. For  example, it does not contain an accurate and up-to-date test schedule with  milestones and completion dates for all levels of test activities. Rather, the  schedule information included in the plan has been overtaken by events,  to the point that program officials stated that many of the dates on the  schedules have changed or are not accurate. Moreover, they described  attempting to have an accurate schedule of testing activities and events as  “futile” because the program’s schedule is constantly changing. As a  another example, the draft Test and Evaluation Master Plan does not  identify any metrics for measuring testing progress for any type of testing  to be performed. According to federal guidance, an accurate schedule is  necessary to inform planning for and sequencing of each type of testing to  be performed, including ensuring that test resources are available when  needed and that predecessor test events occur before successor events  begin. This guidance also states that without performance metrics, it is  difficult to understand where test activities stand and what they show in a  manner that can inform program decision making.
As another example, the draft Test and Evaluation Master Plan does not  clearly define the roles and responsibilities of various entities that are  involved in system testing. Specifically, the plan identifies seven entities,  but it only provides vague descriptions of their respective roles and  responsibilities that are not meaningful enough to effectively guide their  efforts. For example, the plan identifies two entities that are to be involved  in operational testing: the DHS Science and Technology Test and  Evaluation Office and the U.S. Army Test and Evaluation Command.  According to the plan, the DHS office is to function as the operational test  authority and will be responsible for initial planning of “dedicated initial”  and “follow-on” operational testing and evaluation, and the Army group is  to conduct operational testing. With no further clarification, it is not clear  what is expected of each of these entities, including how they are to  interact. Table 6 lists each of the identified entities and provides their  respective roles and responsibilities copied from the draft plan.
Besides being vague, the descriptions of roles and responsibilities are also  incomplete and not consistent with other program documents. For  example, according to the draft plan, the Test and Evaluation Division of  the SBInet System Program Office is responsible only for developing test  and evaluation reports. However, according to the draft SBInet Systems  Engineering Plan, this entity is to act as a subject matter expert for the  oversight or conduct of various testing activities.
Beyond this lack of clearly defined roles and responsibilities, there are  other problems with the groups assigned testing roles. First, some of the  entities identified in the draft plan are not yet operational and thus are  unavailable to participate and perform their assigned roles and  responsibilities. According to program officials, the independent  verification and validation agent has not been selected, the Integrated  Project Teams have not been chartered, and DHS is still in the process of  establishing the DHS Science and Technology Test and Evaluation Office.  Second, although CBP has an interagency agreement with the U.S. Army  Test and Evaluation Command for operational testing, no such agreement  exists with the SBInet program specifically for Block 1 testing.
Finally, neither the draft Test and Evaluation Master Plan nor the draft  Systems Engineering Plan clearly defines the program office’s role and  responsibilities for managing and overseeing each of the other six test  entities and their respective test activities. The SBInet System Program  Office is responsible and accountable for ensuring that the system is  successfully deployed and operates as intended. Given the criticality of  testing in ensuring a successful program, this means that the program  office must ensure that each of these entities executes its assigned roles  effectively. However, the draft Test and Evaluation Master Plan does not  recognize this role and its associated responsibilities. Further, while the  draft Systems Engineering Plan states that the program office is  responsible for engaging external test agents, it provides no further  description of the program office’s roles and responsibilities.
Without clearly defined roles and responsibilities for all entities involved  in SBInet testing, the risk of test activities not being effectively and  efficiently performed increases. As a result, the chances are increased that  the deployed system will not meet operational requirements and perform  as intended. Ultimately, this could lead to expensive and time-consuming  system rework.
section_title: Conclusions
paragraphs: A fundamental aspect of successfully implementing a large program like  SBInet is establishing program commitments, including what capabilities  will be delivered and when and where they will be delivered. Only through  establishing such commitments and by adequately defining the approach  and processes to be used in delivering these commitments, can DHS  effectively position itself for measuring progress, ensuring accountability  for results, and delivering a system solution with its promised capabilities  and benefits on time and within budget constraints. For SBInet, this has  not occurred to the extent that it needs to for the program to have a  meaningful chance of succeeding. In particular, commitments to the  timing and scope of system capabilities remain unclear and continue to  change, with the program committing to far fewer capabilities than  originally envisioned. Further, how the SBInet system solution is to be  delivered has been equally unclear and inadequately defined. Moreover,  while the program office has defined key practices for developing and  managing requirements, these practices were developed after several key  requirements activities were performed. In addition, efforts performed to  date to test whether the system meets requirements and functions as  intended have been limited.
Collectively, these limitations are significant in that they increase the risk  that the delivered system solution will not meet user needs and  operational requirements and will not perform as intended. These  consequences, in turn, increase the chances that the system will require  expensive and time-consuming rework. In light of these circumstances and  risks surrounding SBInet, it is important for the program office to reassess  its approach to and plans for the program—including its associated  exposure to cost, schedule, and performance risks—and to disclose these  risks and alternative courses of action for addressing them to DHS and  congressional decision makers. It is also important for the program to  correct the weaknesses discussed in this report surrounding the program’s  unclear and constantly changing commitments and its life cycle  management approach and processes, including the processes and efforts  performed to date relating to requirements development and management  and testing.
While doing so will not guarantee a successful program, it will minimize  the program’s exposure to risk and thus decrease the likelihood that it will  fall short of expectations. For SBInet, living up to expectations is  important because the program is a large, complex, and integral  component of DHS’s border security and immigration control strategy.
section_title: Recommendations for Executive Action
paragraphs: To improve DHS’s efforts to acquire and implement SBInet we are making  eight recommendations.
To permit meaningful measurement and oversight of and accountability  for the program, we recommend that the Secretary of Homeland Security  direct the CBP Commissioner to ensure that (1) the risks associated with  planned SBInet acquisition, development, testing, and deployment  activities are immediately assessed and (2) the results, including proposed  alternative courses of action for mitigating the risks, are provided to the  Commissioner and DHS’s senior leadership, as well as to the department’s  congressional authorization and appropriation committees.
We further recommend that the Secretary of Homeland Security direct the  CBP Commissioner to have the Acting SBInet Program Manager take the  following additional actions:    Establish and baseline the specific program commitments, including the  specific system functional and performance capabilities that are to be  deployed to the Tucson, Yuma, and El Paso Sectors, and establish when  these capabilities are to be deployed and are to be operational.
Finalize and approve an integrated master schedule that reflects the timing  and sequencing of the work needed to achieve these commitments.
Revise and approve versions of the SBInet life cycle management  approach, including the draft Systems Engineering Plan and draft Test and  Evaluation Management Plan, and in doing so, ensure that these revised  and approved versions are consistent with one another, reflect program  officials’ recently described changes to the engineering and testing  approaches, and reflect relevant federal guidance and associated leading  practices.
Ensure that the revised and approved life cycle management approach is  fully implemented.
Implement key requirements development and management practices to  include (1) baselining requirements before system design and  development efforts begin; (2) analyzing requirements prior to being  baselined to ensure that they are complete, achievable, and verifiable; and  (3) tracing requirements to higher-level requirements, lower-level  requirements, and test cases.
Implement key test management practices to include (1) developing and  documenting test plans prior to the start of testing; (2) conducting  appropriate component level testing prior to integrating system  components; and (3) approving a test management strategy that, at a  minimum, includes a relevant testing schedule, establishes accountability  for testing activities by clearly defining testing roles and responsibilities,  and includes sufficient detail to allow for testing and oversight activities to  be clearly understood and communicated to test stakeholders.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Director,  Departmental GAO/Office of Inspector General Liaison and reprinted in  appendix II, the department stated that it agrees with seven of our eight  recommendations, and partially disagrees with one aspect of the  remaining recommendation. The department also stated that our report is  factually sound and that it is working to address our recommendations  and resolve the management and operational challenges identified in the  report as expeditiously as possible. In this regard, it described actions  recently completed, underway, and planned that it said addresses our  recommendations. It also provided technical comments that we have  incorporated in the report, as appropriate.
Regarding our recommendation to implement key test management  practices, including conducting appropriate component-level testing prior  to integrating system components, DHS commented that its current test  strategy provides for the appropriate degree of technical confidence for  commercially available products, as evidenced by either certificates of  conformance from the original equipment manufacturer, test  documentation from independent government laboratories, or the prime  contractor’s component/integration level testing. We support DHS’s  current test strategy, as it is consistent with our recommendation.  Specifically, it expands on the department’s prior strategy for component  testing, which was limited to manufacturer self-certification of component  conformance and informal observations of system components, by adding  the use of independent government laboratories to test the components.  We would emphasize, however, that regardless of the method used, it is  important that confidence be gained in components prior to integrating  them, which our recommendation recognizes. As our report states,  component-level testing was not performed for Block 1 components prior  to initiating integration testing. Federal guidance and the SBInet program  office’s own Test and Evaluation Master Plan recognize the need to first  test individual components to verify that the system modules work as  intended (i.e., meet functional and performance requirements) before  conducting integration testing. By adopting such a hierarchical approach  to testing, the source of any system defects can be discovered and isolated  sooner rather than later, thus helping to avoid the potential for expensive  and time-consuming system rework.
We are sending copies of this report to the Chairmen and Ranking  Members of the Senate and House Appropriations Committees and other  Senate and House committees and subcommittees that have authorization  and oversight responsibilities for homeland security. We will also send  copies to the Secretary of Homeland Security, the Commissioner of U.S.  Customs and Border Protection, and the Director of the Office of  Management and Budget. In addition, this report will be available at no  cost on the GAO Web site at http://www.gao.gov.
Should your offices have any questions on matters discussed in this report,  please contact me at (202) 512-3439 or at hiter@gao.gov. Contact points for  our Office of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors to this report are listed in  appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine whether the Department of Homeland  Security (DHS) (1) has defined the scope and timing of planned SBInet  capabilities and how these capabilities will be developed and deployed, (2)  is effectively defining and managing SBInet requirements, and (3) is  effectively managing SBInet testing.
To determine the extent to which DHS has defined the scope and timing of  planned SBInet capabilities and how these capabilities will be developed  and deployed, we reviewed program documentation, such as the draft  Systems Engineering Plan, the Systems Engineering Management Plan, the  Operational Requirements Document, the Mission Engineering Process,  the draft Test and Evaluation Master Plan, and the 2008 SBI Expenditure  Plan to understand the SBInet engineering process and the scope and  timing of planned deployments. We also interviewed SBInet officials and  contractors to gain clarity beyond what was included in the program  documentation and to obtain schedule information in the absence of an  integrated master schedule for the program.
To determine if DHS is effectively defining and managing SBInet  requirements, we reviewed relevant documentation, such as the  Requirements Development and Management Plan, the Requirements  Management Plan, the Configuration and Data Management Plan, the  Operational Requirements Document, System of Systems A-Level  Specification, B-2 Specifications, and Vendor Item Control Drawings, and  compared them to industry best practices to determine the extent to  which the program has effectively managed the systems requirements and  maintained traceability backwards to high-level operational requirements  and system requirements, and forward to system design and verification  methods.
To assess reliability of the requirements data, we reviewed quality and  access controls of the requirements database. We then randomly selected  59 requirements from a sample of 1,666 component requirements and  traced them backwards to the system requirements and then to the  operational requirements and forward to design requirements and  verification methods. Because we followed a probability procedure based  on random selection, we are 95 percent confident that each of the  confidence intervals in this report will include the true values in the study  population. We used statistical methods appropriate for audit compliance  testing to estimate 95 percent confidence intervals for the traceability of  requirements in our sample. In addition, we interviewed program and  contractor officials involved in requirements management to understand  their roles and responsibilities. We also visited a contractor development  facility in Huntsville, Alabama, to understand the contractor’s role in  requirements management and development and the use of its  requirements management tool, known as the Dynamic Object-Oriented  Requirements System (DOORS). In addition, we attended a demonstration  of SBInet Rapid Application Development/Joint Application Design to  understand how the users are involved in developing requirements.
To determine if DHS is effectively managing SBInet testing, we reviewed  relevant documentation, such as the SBInet Test and Evaluation Master  Plan, the Systems Integration Test Plan, the Quality Assurance  Surveillance Plan, the Requirements Verification Plan, the  Characterization Test Plan, and the Prime Mission Product Design, and  compared them to relevant federal guidance to determine the extent to  which the program has effectively managed its testing activities. We also  interviewed SBInet officials to gain clarity beyond what was included in  the program documentation and to obtain schedule information in the  absence of a formal testing schedule. In addition, we visited a contractor  facility in Huntsville, Alabama, to better understand the contractor’s role  in testing activities and to observe the test lab and how testing is  performed.
In addition, we visited the Tucson Sector Border Patrol Headquarters in  Tucson, Arizona, to see the technology that was deployed as a prototype  to understand the scope of the technology, how the Border Patrol agents  use the technology, and future plans.
To assess data reliability, we reviewed related program documentation to  substantiate data provided in interviews with knowledgeable agency  officials, where available. For the information contained in the DHS  independent study on SBInet, we interviewed the individuals responsible  for conducting the review to understand their methodology, and  determined that the information derived from this study was sufficiently  reliable for the purposes of this report. We have made appropriate  attribution indicating the data’s sources.
We performed our work at the U.S. Customs and Border Protection  headquarters and contractor facilities in the Washington, D.C.,  metropolitan area; the Tucson Sector Border Patrol headquarters in  Tucson, Arizona; and a contractor facility in Huntsville, Alabama. We  conducted this performance audit from August 2007 to September 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Deborah Davis (Assistant  Director), Carl Barden, Neil Doherty, Lee McCracken, Jamelyn Payan, Karl  Seifert, Sushmita Srikanth, Karen Talley, and Merry Woo made key  contributions to this report.