Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of Its Prime Contractor
Published: 2010-10-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security's (DHS) Secure Border Initiative Network (SBInet) is to place surveillance systems along our nation's borders and provide Border Patrol command centers with the imagery and related tools and information needed to detect breaches and make agent deployment decisions. To deliver SBInet, DHS has relied heavily on its prime contractor. Because of the importance of effective contractor management and oversight to SBInet, GAO was asked to determine the extent to which DHS (1) defined and implemented effective controls for managing and overseeing the prime contractor and (2) effectively monitored the contractor's progress in meeting cost and schedule expectations. To do this, GAO analyzed key program documentation against relevant guidance and best practices, and interviewed program officials.
section_title: What GAO Found
paragraphs: DHS has largely defined but has not adequately implemented the full range of controls that is reflected in relevant guidance and related best practices and is needed to effectively manage and oversee its SBInet prime contractor. To the department's credit, it has defined a number of key policies and procedures for verifying and accepting contract deliverables and conducting technical reviews, such as the criteria that need to be met before commencing and concluding a critical design review. Moreover, it has implemented some of these defined practices, such as those associated with training key contractor management and oversight officials. However, DHS has not effectively implemented other controls. For example, it has not adequately documented its review of contract deliverables and communicated to the prime contractor the basis for rejecting certain deliverables. Further, it has not ensured that key documentation satisfied relevant criteria before concluding key technical reviews. These weaknesses can be attributed in part to limitations in the defined verification and acceptance deliverable process, a program office decision to exclude certain deliverables from the process, and insufficient time to review technical review documentation. All told, DHS has not effectively managed and overseen its SBInet prime contractor, thus resulting in costly rework and contributing to SBInet's well-chronicled history of not delivering promised capabilities and benefits on time and within budget. DHS has not effectively monitored the SBInet prime contractor's progress in meeting cost and schedule expectations. While DHS has used earned value management (EVM), which is a proven management approach for understanding program status and identifying early warning signs of impending schedule delays and cost overruns, it has not ensured that its contractor has effectively implemented EVM. In particular, DHS has not ensured that validated performance baselines (the estimated value of planned work against which performance is measured) were timely, complete, and accurate. For example, the contractor was allowed to perform work on task orders for several months without a validated baseline in place. Further, not all work to be performed was included in the baselines that were eventually established, and the schedules for completing this work did not substantially comply with six of the eight key practices that relevant guidance states are important to having a reliable schedule. Also, DHS regularly received incomplete and anomalous EVM data from the prime contractor, which it had to rely on to measure progress and project the time and cost to complete the program. As a result, DHS has not been able to gain meaningful and proactive insight into potential cost and schedule performance shortfalls, and thus take corrective actions to avoid shortfalls in the future. Program officials attributed these weaknesses in part to the instability in the scope of the work to be performed, and the contractor's use of estimated, rather than actual, costs for subcontractor work and the subsequent adjustments that are made when actual costs are received. This inability has contributed to the program's failure to live up to expectations and to it costing more and taking longer than was necessary.
section_title: What GAO Recommends
paragraphs: GAO is making recommendations to DHS aimed at revising and implementing policies and procedures related to contractor deliverables and technical reviews, and improving EVM baselines and data. DHS agreed with GAO's recommendations and described actions to address them, but took exception with selected findings and conclusions regarding EVM implementation. GAO stands by these findings and conclusions for reasons discussed in the report.

section_title: Letter
paragraphs: To enhance border security and reduce illegal immigration, the  Department of Homeland Security (DHS) launched its multiyear,  multibillion dollar Secure Border Initiative (SBI) program in November  2005. Through SBI, DHS intends to enhance surveillance technologies,  increase staffing levels, enforce immigration laws, and improve the  physical infrastructure along our borders. Within SBI, the Secure Border  Initiative Network (SBInet) is a multibillion-dollar program that involves  the acquisition, development, integration, deployment, and operation and  maintenance of surveillance technologies to create a “virtual fence” along  the border as well as command, control, communications, and intelligence  (C3I) technologies to create a picture of the border in command centers  and vehicles.
DHS’s strategy is to deliver SBInet capabilities incrementally. In doing so,  the department has relied heavily on its prime contractor—the Boeing  Company. Because of the importance of effective contractor management  and oversight to the successful deployment and operation of SBInet  capabilities, you asked us to determine to what extent DHS (1) has defined  and implemented effective controls for managing and overseeing the  SBInet prime contractor and (2) is effectively monitoring the prime  contractor’s progress in meeting cost and schedule expectations.
To accomplish our objectives, we focused on DHS’s management and  oversight of four key task orders associated with designing, developing,  and deploying the first increment of SBInet, known as Block 1, to its initial  deployment sites in Arizona—the Tucson Border Patrol Station (TUS-1)  and the Ajo Border Patrol Station (AJO-1). In doing so, we reviewed DHS  and program guidance, policies, and plans for managing and overseeing  the contractor; task orders and associated modifications; contract  deliverable review comments; contractor schedules; the work breakdown  structure governing all task orders; integrated baseline review documents  for each task order; and task order contract cost performance reports. We  also analyzed a nonprobability, random sample of 28 contract deliverables;  a nonprobability, random sample of 8 technical reviews conducted with  the contractor; and a nonprobability sample of 11 action items identified  during program management reviews. In addition, we interviewed  program officials about contractor management and oversight activities,  such as verifying and accepting contract deliverables and conducting  technical and management reviews. We also interviewed program officials  about the prime contractor’s cost and schedule performance. We then  compared this information to relevant guidance and leading industry  practices on contractor management and oversight, schedule estimating,  and earned value management (EVM), such as the Software Engineering  Institute’s (SEI) Capability Maturity Model Integration® (CMMI®) for  Acquisition, GAO’s Cost Estimating and Assessment Guide, and the  American National Standards Institute (ANSI) standard.
We conducted this performance audit from June 2009 to October 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. Further details of our objectives, scope, and  methodology are in appendix I.
subsections: section_title: Background
paragraphs: Managed by U.S. Customs and Border Protection (CBP), SBInet is to  strengthen DHS’s ability to detect, identify, classify, track, and respond to  illegal breaches at and between ports of entry. It includes the acquisition,  development, integration, deployment, and operations and maintenance of  a mix of surveillance technologies, such as cameras, radars, sensors, and  C3I technologies. Surveillance technologies include unattended ground  sensors that can detect heat and vibrations associated with foot traffic and  metal associated with vehicles, radar mounted on fixed towers that can  detect movement, and cameras mounted on fixed towers that can be used  to identify and classify items of interest detected and tracked by ground  sensors and radar. These technologies are generally commercial off-the- shelf products. C3I technologies include customized software  development to produce a common operating picture (COP)—a uniform  presentation of activities within specific areas along the border—as well  as the use of CBP network capabilities. Together, the surveillance  technologies are to gather information along the border and transmit this  information to COP terminals located in command centers, which are to  assemble it to provide CBP agents with border situational awareness, to,  among other things, enhance agents’ tactical decisionmaking regarding  potential apprehensions.
Since fiscal year 2006, DHS has received about $4.4 billion in  appropriations for SBI, including about $2.5 billion for physical fencing  and related infrastructure, about $1.5 billion for virtual fencing (e.g.,  surveillance systems) and related infrastructure (e.g., towers), and about  $300 million for program management. As of May 2010, DHS had  obligated about $1.02 billion for SBInet.
subsections: section_title: Program and Acquisition Organizational Structure
paragraphs: The SBI Program Management Office, which is organizationally within  CBP, is responsible for managing key acquisition functions associated  with SBInet, including prime contractor management and oversight. It is  organized into four components: the SBInet System Program Office (SPO),  Operational Integration Division, Business Operations Division, and  Systems Engineering Division. The SPO is responsible for such key  contractor oversight activities as verifying and accepting contract  deliverables and conducting contractor management and technical  reviews. In addition, the Business Operations Division has primary  responsibility for monitoring the prime contractor’s cost and schedule  performance. As of May 2010, the SBI Program Management Office was  staffed with 194 people—105 government employees, 78 contractor staff,  and 11 detailees. (See figure 1 for a partial SBI program organization   organization  chart.)  chart.)
In addition, CBP has engaged the Defense Contract Management Agency  to, among other things, perform surveillance of the prime contractor’s  EVM, systems engineering, hardware and software quality assurance, and  risk management.
Further, the SBI Contracting Division, which is organizationally within the  CBP Office of Administration Procurement Directorate’s Enterprise  Contracting Office, is responsible for performing contract administration  activities, such as maintaining the contract file and notifying the  contractor in writing of whether deliverables are accepted or rejected. The  SBI Contracting Division is organized into three branches: SBI  Contracting, SBInet System Contracting, and SBInet Deployment  Contracting. As of May 2010, the SBI Contracting Division was staffed with  22 people—18 government employees and 4 contractor staff. (See figure 2  for a partial SBI Contracting Division organization chart.)
section_title: Overview of SBInet Acquisition Strategy and Program Status
paragraphs: In September 2006, CBP awarded an Indefinite Delivery/Indefinite  Quantity (IDIQ) prime contract to the Boeing Company. The contract was  for 3 base years with three additional 1-year options for designing,  producing, testing, deploying, and sustaining SBI. In September 2009, CBP  exercised the first option year. Under the prime contract, CBP has issued  11 task orders that relate to SBInet, covering for example, COP design and  development, system deployment, and system maintenance and logistics  support. As of May 2010, 5 of the 11 task orders were complete and 6 were  ongoing. (See table 1 for a summary of the SBInet task orders.)
Through the task orders, CBP’s strategy is to deliver SBInet capabilities  incrementally. To accomplish this, the program office adopted an  evolutionary system life cycle management approach in which system  capabilities are to be delivered to designated locations in a series of  discrete subsets of system functional and performance capabilities that  are referred to as blocks. The first block (Block 1) includes the purchase  of commercially available surveillance systems, development of  customized COP systems and software, and use of existing CBP  communications and network capabilities. According to program officials,  as of July 2010, the Block 1 design was deployed to TUS-1 and was being  deployed to AJO-1, both of which are located in CBP’s Tucson Sector of  the southwest border. Together, these two deployments cover 53 miles of  the 1,989-mile-long southern border. Also, according to program officials,  as of July 2010, TUS-1 and AJO-1 were to be accepted in September 2010  and December 2010, respectively.
In January 2010, the Secretary of Homeland Security ordered an  assessment of the SBI program. In addition, on March 16, 2010, the  Secretary froze fiscal year 2010 funding for any work on SBInet beyond  TUS-1 and AJO-1 until the assessment is completed. Also at that time, the  Secretary redirected $35 million that had been allocated to SBInet Block 1  deployment to other tested and commercially-available technologies, such  as truck-mounted cameras and radar, called Mobile Surveillance Systems,  to meet near-term needs. According to the SBI Executive Director, the  department’s assessment would consist of a comprehensive and science- based assessment to determine if there are alternatives to SBInet that may  more efficiently, effectively, and economically meet U.S. border security  needs. Further, the Executive Director stated that if the assessment  suggests that the SBInet capabilities are worth the cost, DHS will extend  its deployment to sites beyond TUS-1 and AJO-1. However, if the  assessment suggests that alternative technology options represent the best  balance of capability and cost-effectiveness, DHS will redirect resources to  these other options. According to program officials, the initial phase of the  assessment, which addresses the Arizona border, was completed in July  2010, and the results are currently being reviewed by senior DHS  management. Officials were unable to provide a date for completion of the  review.
section_title: Use of Acquisition Management Disciplines Is Important to Programs Like SBInet
paragraphs: Our research and evaluations of information technology programs have  shown that delivering promised system capabilities and benefits on time  and within budget largely depends on the extent to which key acquisition  management disciplines are employed. These disciplines include, among  others, requirements management, risk management, and test  management. As is discussed in the following section, we have previously  reported on the extent to which these disciplines have been employed on  SBInet.
Contractor management and oversight, which is the focus of this report, is  another acquisition management discipline. Among other things, this  discipline helps to ensure that the contractor performs the requirements of  the contract and the government receives the service or product intended.  DHS acquisition policies and guidance, along with other relevant guidance,  recognize the importance of these management and oversight disciplines.  As we have previously reported, not employing them increases the risk  that system acquisitions will not perform as intended and will require  expensive and time-consuming rework.
section_title: GAO Has Previously Reported on Numerous SBInet Acquisition Management Weaknesses and Risks
paragraphs: Since 2007, we have identified a range of management weaknesses and  risks facing SBInet, and we have made a number of recommendations to  address them that DHS has largely agreed with, and to varying degrees,  taken actions to address. For example, in February 2007, we reported that  DHS had not fully defined activities, milestones, and costs for  implementing the program or demonstrated how program activities would  further the strategic goals and objectives of SBI. Further, we reported that  the program’s schedule contained a high level of concurrency among  related tasks and activities, which introduced considerable risk.  Accordingly, we recommended that DHS define explicit and measurable  commitments relative to, among other things, program capabilities,  schedules, and costs, and re-examine the level of concurrency in the  schedule and adjust the acquisition strategy appropriately.
In September 2008, we reported that a number of program management  weaknesses put SBInet at risk of not performing as intended and taking  longer and costing more to deliver than was necessary. Specifically, we  reported the following:  Important aspects of SBInet were ambiguous and in a continued state of  flux, making it unclear and uncertain what technology capabilities were to  be delivered when. For example, the scope and timing of planned SBInet  deployments and capabilities had continued to change since the program  began and remained unclear. Further, the SPO did not have an approved  integrated master schedule to guide the execution of the program, and our  assimilation of available information indicated that key milestones  continued to slip. This schedule-related risk was exacerbated by the  continuous change in and the absence of a clear definition of the approach  used to define, develop, acquire, test, and deploy SBInet. Accordingly, we  concluded that the absence of clarity and stability in these key aspects of  SBInet impaired the ability of Congress to oversee the program and hold  DHS accountable for results, and it hampered DHS’s ability to measure the  program’s progress.
SBInet requirements had not been effectively defined and managed. While  the SPO had issued guidance that defined key practices associated with  effectively developing and managing requirements, the guidance was  developed after several key activities had been completed. In the absence  of this guidance, the SPO had not effectively performed key requirements  development and management practices, such as ensuring that different  levels of requirements were properly aligned. As a result, we concluded  that the risk of SBInet not meeting mission needs and performing as  intended was increased, as were the chances of the system needing  expensive and time-consuming rework.
SBInet testing was not being effectively managed. For example, the  program office had not tested the individual system components to be  deployed to initial locations, even though the contractor had initiated  integration testing of these components. Further, its test management  strategy did not contain, among other things, a clear definition of testing  roles and responsibilities or sufficient detail to effectively guide planning  for specific test events.
To address these issues, we recommended that DHS assess and disclose  the risks associated with its planned SBInet development, testing, and  deployment activities and that it address the system deployment,  requirements management, and testing weaknesses that we had identified.  DHS largely agreed to implement our recommendations.
In September 2009, we reported that SBInet had continued to experience  delays. For example, deployment to the entire southwest border had  slipped from early 2009 to 2016, and final acceptance of TUS-1 and AJO-1  had slipped from November 2009 and March 2010 to December 2009 and  June 2010, respectively. We did not make additional recommendations at  that time.
In January 2010, we reported that SBInet testing was not being effectively  managed. Specifically, we reported the following:    Test plans, cases, and procedures for the most recent test events were not  defined in accordance with important elements of relevant guidance.  Further, a large percentage of the test cases for these events were  changed extemporaneously during execution. While some of the changes  were minor, others were more significant, such as rewriting entire  procedures and changing the mapping of requirements to test cases.  Moreover, these changes to procedures were not made in accordance with  documented quality assurance processes, but rather were based on an  undocumented understanding that program officials said they established  with the contractor. Compounding the number and significance of changes  were questions raised by the SPO and a support contractor about the  appropriateness of some changes, noting that some changes made to  system qualification test cases and procedures appeared to be designed to  pass the test instead of being designed to qualify the system.
About 1,300 SBInet defects had been found from March 2008 through July  2009, with the number of new defects identified during this time generally  increasing faster than the number being fixed—a trend that is not  indicative of a system that is maturing and ready for deployment. While  the full magnitude of these unresolved defects was unclear because the  majority were not assigned a priority for resolution, some of the defects  that had been found were significant. Although DHS reported that these  defects had been resolved, they had nevertheless caused program delays,  and related problems had surfaced that continued to impact the program’s  schedule.
In light of these weaknesses, we recommended that DHS (1) revise the  program’s overall test plan; (2) ensure that test schedules, plans, cases,  and procedures are adequately reviewed and approved consistent with the  revised test plan; (3) ensure that sufficient time is provided for reviewing  and approving test documents prior to beginning a given test event; and  (4) triage the full inventory of unresolved system problems, including  identified user concerns, and periodically report on their status to CBP  and DHS leadership. DHS fully agreed with the last three  recommendations and partially agreed with the first.
In May 2010, we raised further concerns about the program and called for  DHS to reconsider its planned investment in SBInet. Specifically, we  reported the following:    DHS had defined the scope of the first incremental block of SBInet  capabilities that it intended to deploy and make operational; however,  these capabilities and the number of geographic locations to which they  were to be deployed continued to shrink. For example, the geographic  “footprint” of the initially deployed capability has been reduced from three  border sectors spanning about 655 miles to two sectors spanning about  387 miles.
DHS had not developed a reliable integrated master schedule for  delivering the first block of SBInet. Specifically, the schedule did not  sufficiently comply with seven of nine key practices that relevant guidance  states are important to having a reliable schedule. For example, the  schedule did not adequately capture all necessary activities, assign  resources to them, and reflect schedule risks. As a result, it was unclear  when the first block was to be completed, and continued delays were  likely.
DHS had not demonstrated the cost-effectiveness of Block 1. In particular,  it had not reliably estimated the costs of this block over its entire life  cycle. Further, DHS had yet to identify expected quantifiable and  qualitative benefits from this block and analyze them relative to costs.  Without a meaningful understanding of SBInet costs and benefits, DHS  lacks an adequate basis for knowing whether the initial system solution is  cost effective.
DHS had not acquired the initial SBInet block in accordance with key life  cycle management processes. While processes associated with, among  other things, requirements development and management and risk  management, were adequately defined they were not adequately  implemented. For example, key risks had not been captured in the risk  management repository and thus had not been proactively mitigated. As a  result, DHS is at increased risk of delivering a system that does not  perform as intended.
We concluded that it remains unclear whether the department’s pursuit of  SBInet is a cost-effective course of action, and if it is, that it will produce  expected results on time and within budget. Accordingly, we  recommended that DHS (1) limit near-term investment in the first  incremental block of SBInet, (2) economically justify any longer-term  investment in SBInet, and (3) improve key program management  disciplines. DHS largely agreed with our recommendations, and noted  ongoing and planned actions to address each of them.
In June 2010, we reported several technical, cost, and schedule challenges  facing SBInet, such as problems with radar functionality, and noted that  the program office was staffed substantially below planned levels for  government positions. We did not make any additional recommendations  at that time.
section_title: DHS Has Largely Defined but Has Not Effectively Implemented the Full Range of Needed Contractor Management and Oversight Controls
paragraphs: Federal regulations and relevant guidance recognize effective contractor  management and oversight as a key acquisition management discipline. In  addition, they describe a number of practices associated with it, including  training persons who are responsible for contractor management and  oversight, verifying and deciding whether to accept contract deliverables,  conducting technical reviews with the contractor to ensure that products  and services will satisfy user needs, and holding management reviews with  the contractor to monitor contractor performance.
To DHS’s credit, it has largely defined key practices aimed at employing  each of these contractor management and oversight controls. Moreover, it  has implemented some of them, such as training key contractor  management and oversight officials and holding management reviews with  the prime contractor. However, it has not effectively implemented others,  such as documenting contract deliverable reviews and using entry and exit  criteria when conducting technical reviews. Reasons for these weaknesses  include limitations in the defined verification and acceptance deliverable  process and a SPO decision to exclude some deliverables from the  process, and insufficient time to review technical review documentation.  Without employing the full range of practices needed to effectively manage  and oversee the prime contractor, DHS is limited in its ability to know  whether the contractor is meeting performance and product expectations.  Moreover, it increases the chances that SBInet will not function as  intended and will take more time and resources than necessary to deliver,  which we have previously reported is the case for Block 1.
subsections: section_title: SBInet Contractor Management and Oversight Officials Have Been Properly Trained
paragraphs: Training supports the successful performance of relevant activities and  tasks by helping to ensure that the responsible people have the necessary  skills and expertise to perform the tasks. According to relevant guidance,  organizations should define training expectations and should ensure that  these expectations are met for individuals responsible for contractor  management and oversight.
DHS’s acquisition management directives define training requirements for,  among others, program managers, contracting officers, and contracting  officer’s technical representatives (COTR). Specifically:    Program managers must be certified at a level commensurate with their  acquisition responsibilities. For a Level 1 information technology  program, like SBInet, the designated program manager must be certified  at a program management Level 3.
Contracting officers must be certified at the appropriate level to support  their respective warrant authority.
COTRs must be trained and certified within 60 days of appointment to a  contract or task order. The minimum mandatory requirements are the  completion of 40 hours of COTR training and a 1-hour procurement ethics  training course.
For SBInet, CBP has ensured that people in each of these key positions  have been trained in accordance with DHS requirements. Specifically, the  two program managers associated with the development and deployment  of SBInet Block 1—the SBI Executive Director and the SPO Executive  Director—were issued training certifications that qualify each as an  Acquisition Professional. Further, each contracting officer responsible for  executing actions on the Arizona Deployment Task Order (ADTO), the  Design Task Order (DTO), the C3I/COP task order, and the System Task  Order (STO) between June 2008 and February 2010 was certified  commensurate with his or her respective warrant authority. In addition,  for the same time period, each COTR assigned to each of the four task  orders received DHS-issued certificates indicating that each had met the  minimum training requirements before being assigned to a task order.
According to CBP officials, DHS leadership has made contractor  management and oversight training a high priority, which helped to ensure  that key officials were trained. By doing so, CBP has established one of the  key controls to help ensure that prime contractor products and services  meet expectations.
section_title: DHS Has Not Effectively Verified and Accepted All Contract Deliverables
paragraphs: Effectively implementing a well-defined process for verifying and  accepting prime contractor deliverables is vital to SBInet’s success. DHS  has a defined process for verifying and accepting contract deliverables,  but this process does not ensure that deliverable reviews are sufficiently  documented. Further, while the SPO has followed key aspects of this  process, it has not effectively documented its review of certain  deliverables and has not effectively communicated to the prime contractor  the basis for rejecting all deliverables. Reasons for not doing so include  limitations in the defined verification and acceptance process and a SPO  decision to exclude some deliverables from the process. Without  documenting all its reviews and effectively communicating the review  results to the contractor, the SPO has increased the chances of accepting  deliverables that do not meet requirements and having the contractor  repeat work to correct deliverable problems.
subsections: section_title: CBP Has a Defined Process for Verifying and Accepting SBInet Contract Deliverables
paragraphs: The purpose of contractor deliverable verification and acceptance is to  ensure that contractor-provided products and services meet specified  requirements and otherwise satisfy the terms of the contract. According to  relevant guidance, organizations should have written policies and  procedures for verifying and accepting deliverables that, among other  things, (1) assign roles and responsibilities for performing verification and  acceptance tasks and (2) provide for conducting and documenting  deliverable reviews and for effectively communicating to the contractor  deliverable acceptance and rejection decisions.
To its credit, CBP has defined policies and procedures for verifying and  accepting SBInet deliverables. Specifically, it issued its Deliverable Review  and Approval Process in July 2007, which specifies how the SPO is to  receive, review, and respond to all contract deliverables. Among other  things, this guide assigns verification and acceptance roles and  responsibilities to key program management positions. For example, it  assigns the project manager responsibility for overseeing the review  process and determining the acceptability of the deliverables, designates  reviewers responsibilities for examining the deliverable, and assigns the  contracting officer responsibility for communicating the decision to  accept or reject the deliverable to the contractor. In addition, it provides  for conducting deliverable reviews, which according to program officials,  involves comparing the deliverable to the requirements enumerated in the  applicable task order statement of work (typically within the Data Item  Description). The process further specifies that the decision to accept or  reject the deliverable is to be communicated in writing to the contractor.
However, the process does not state that the results of all reviews are to  be documented. Instead, its states that a deliverable review comment form  is to be prepared only when deficiencies or problems exist. If the  deliverable is acceptable, the form does not need to be prepared. Program  officials could not explain why review documentation was not required for  acceptable deliverables. As a result, and as discussed in the next section of  this report, the SPO cannot demonstrate its basis for accepting a number  of deliverables, which in turn has increased the risk of, and actually  resulted in, deliverables being accepted that do not meet requirements.
section_title: CBP Has Not Fully Followed its Defined Process for Verifying and Accepting SBInet Deliverables
paragraphs: The SPO followed its process for verifying and accepting SBInet-related  deliverables about 62 percent of the time, based on the 29 deliverables that  we reviewed. Specifically, the process was fully followed for 18 of the  deliverables: (1) 6 that were accepted without documented review  comments, (2) 5 that were accepted with documented review comments,  and (3) 7 that were rejected with documented review comments. In  addition, the acceptance or rejection of all 18 deliverables was  communicated in writing to the contractor. For example, the ADTO  Security Plan Addendum was delivered to the SPO in August 2008. The  SPO reviewed the plan and documented its review comments, which  included a determination that the plan did not address all required items  specified in the task order’s Data Item Description. The CBP contracting  officer subsequently notified the contractor in writing that the plan was  rejected for this and other reasons. In February 2009, the contractor  resubmitted the plan, the SPO reviewed the plan and documented its  comments, and the contracting officer again notified the contractor in  writing that the plan was rejected. The contractor resubmitted the plan in  May 2009, and program officials documented their review of the  deliverable. The contracting officer subsequently communicated the  deliverable’s acceptance in writing to the contractor. (Figure 3 summarizes  the number of contract deliverables that did and did not follow the defined  process.)
The remaining 11 deliverables, however, did not fully adhere to the defined  process. Of these, five were accepted without any documented review  comments and without communicating the acceptance in writing to the  contractor. The following are examples of these five and reasons for not  adhering to the process.
Three of the five deliverables related to the C3I/COP task order did not  require government approval. However, the Deliverable Review and  Approval Process document does not provide for such treatment of these  deliverables, and thus this practice is not in accordance with the process.  Program officials told us that they have since recognized that this was a  poor practice, and they have modified the task order to now require  approval of all C3I/COP task order deliverables.
One of the five deliverables relates to the STO and is for the C2  Component Qualification Test package, which includes, among other  things, the test plan and test cases and procedures for conducting the C2  qualification test event. In this case, the SPO accepted the test plan  because, according to program officials, several days had passed since the  deliverable was received, and they had not received any comments from  the reviewers. They said that they therefore accepted the deliverable on  the basis of not receiving any review comments to the contrary, but did  not notify the contractor in writing of the acceptance.
The 11 deliverables also include 3 that were rejected without documented  review comments and without the rejection being communicated to the  contractor in writing. The following are examples of these three and  reasons for not adhering to the process are discussed below.
One of the three deliverables relates to the C3I/COP task order and is for  the Network Operations Center/Security Operations Center (NOC/SOC)  Test Plan/Procedures/Description. According to program officials, the  contractor did not submit the plan on time, thus requiring them to review  it during the readiness review. Based on this review, the plan was  rejected, which was communicated verbally to the contractor during the  review. Despite rejecting the plan, the program office began testing the  NOC/SOC component on the day of the review, without a revised plan  being submitted, reviewed, and approved. According to program officials,  this occurred in part because of insufficient time and resources to review  contractor test-related deliverables.
Another one of the three deliverables also relates to the C3I/COP task  order, and is for the Release 0.5 Software Test Plan/Procedures/  Description. According to program officials, the contractor submitted the  plan late. The program office rejected the plan and provided oral  comments during a teleconference prior to the review. Nevertheless, the  test event again occurred without a revised plan being submitted,  reviewed, and accepted. According to program officials, this was also due  to insufficient time and resources to review the test plan. In this case, the  plan was approved in late April 2009, which was 5 months after the test  event was conducted.
The 11 deliverables also include 3 for which a decision to accept or reject  the deliverable was not made. See the following examples:    One of the three relates to the C3I/COP task order, and is for the NOC/SOC  Interface Control Document. For this deliverable, review comments were  not documented and no written communication with the contractor  occurred. The deliverable was subsequently submitted three times and  ultimately accepted. However, program officials could not explain  whether the initial deliverable was accepted or rejected, or why the  deliverable was submitted multiple times.
Another one of these relates to STO, and is for the Dynamic Object- Oriented Requirements System. For this deliverable, review comments  were not documented, but CBP communicated in writing to the contractor  that it was withholding comment on this submission of the deliverable and  was to provide a consolidated set of comments on the subsequent  submission. Subsequently, the contractor resubmitted the deliverable, and  because it was accepted, the review was not documented. The contracting  officer communicated the deliverable’s acceptance in writing to the  contractor.
By not effectively verifying and accepting contractor deliverables, the SPO  cannot ensure that the deliverables will satisfy stated requirements, thus  increasing the risk of costly and time-consuming rework. For example, we  recently reported that contractor-delivered test plans were poorly defined  and resulted in problems during testing. In particular, NOC/SOC testing  was hampered by requirements incorrectly mapped to test cases, did not  provide for testing all requirements, and required significant  extemporaneous changes to test cases during the test events. As a result  of the testing problems, the SPO had to conduct multiple test events.
section_title: Key Contract Technical Reviews Have Not Been Adequately Conducted
paragraphs: Technical reviews are performed throughout the project life cycle to  confirm that products and services being produced by the contractor  provide the desired capability and ultimately satisfy user needs. To its  credit, DHS has defined a process for conducting technical reviews, but it  has not effectively implemented it. In particular, the SPO did not ensure  that all key documentation was reviewed and relevant criteria were  satisfied before concluding key technical reviews. Program officials  attributed these limitations to the program’s aggressive schedule, which  resulted in insufficient time to review relevant documentation. Concluding  technical reviews without adequate justification has resulted in schedule  delays and costly rework.
subsections: section_title: CBP Has a Defined Process for Conducting Technical Reviews
paragraphs: According to relevant guidance, organizations should have written policies  and procedures for conducting technical reviews that, among other things,  (1) assign roles and responsibilities for performing the specific technical  review tasks and (2) establish entry and exit criteria to determine the  readiness of the technical solution to proceed to the technical review and  to demonstrate and confirm completion of required accomplishments.
To its credit, DHS has policies and guidance for conducting technical  reviews. Specifically, DHS’s Systems Engineering Life Cycle outlines the  key reviews to be performed as well as how these reviews are aligned with  the department’s governance process. In addition, DHS guidance defines  expectations for technical review exit criteria, stating that compliance  with exit criteria is based upon the satisfaction of the content of the  criteria, and not upon only the delivery of specified documents.
To augment DHS policy and guidance, the SBInet Systems Engineering  Plan (SEP), dated November 2008, identifies and describes the technical  reviews to be conducted. They include, for example:    Requirements Review, which is to ensure that requirements have been  completely and properly identified and are understood by the SPO and the  contractor. Documentation associated with this review is to include,  among other things, a requirements traceability matrix (i.e., a tool for  demonstrating that component-level requirements are traceable to higher- level system level requirements).
Critical Design Review (CDR), which is to (1) demonstrate that the  designs are complete and baselined and (2) ensure that the solution is  ready for fabrication, coding, assembly, and integration. Documentation  for this review is to include, among other things, (1) baselined  requirements, (2) interface descriptions, and (3) identified risks and  mitigation plans.
Test Readiness Review, which is to assess the readiness of the system solution to begin f include, among other things, (1) test plans that include test cases and  procedures and (2) a traceability matrix that maps each requirement to tested to a corresponding test case.  ormal testing. Documentation for this review is to  In addition, the SBInet SEP describes high-level roles and responsibilities  for performing these reviews, and establishes entry and exit criteria for  each. For example, it states that the SPO program manager and the chief  engineer are responsible for leading the reviews. Further, the SEP defines  entry and exit criteria for the CDR. For example:  Entry. System-level requirements should be traceable to component-level  requirements; system internal and external interfaces should be defined.
Exit. Design baseline should be established and balanced across cost,  schedule, performance, and risk considerations over the investment’s lifecycle; system risks should be identified and mitigation plans should be in  place.
section_title: Contractor Technical Reviews Have Not Been Performed Adequately
paragraphs: The SPO did not follow the defined process for conducting technical  reviews defined in the respective task orders to guide each review. However, the  task orders do not define entry and exit criteria. Rather, they list a set  of  documents that the contractor is to provide and the SPO is to review. For  example, for the Block 1 CDR, the relevant task order requires that the  contractor deliver, among other documents, (1) baselined component and system requirements, (2) interface descriptions (i.e., descriptions of the  data to be exchanged and the protocols used to exchange the data), and. Instead, program officials told us that they used the requirements  (3) all identified risks and mitigation plans for those risks. However, the  task orders do not associate these documents with either entry or exit  criteria, and they do not specify characteristics or qualities that the  documents are to satisfy.
Without explicit entry and exit criteria, the basis for beginning and e the technical reviews is unclear, thus increasing the risk that a program  will be allowed to proceed and begin the next phase of development  before it is ready to do so. In fact, this risk was realized for SBInet.   Technical reviews were concluded without adequate justification, which ultimately resulted in problems that required additional time and  resources to fix. For example:  nding  NOC/SOC Requirements Review. At this review, the contractor did not  deliver a requirements traceability matrix, as required by the relevant task  order, until almost a month after the review was completed. Nonetheless,  program officials stated that they concluded the review in June 2008,  without knowing whether the applicable higher-level system requirements  were fully satisfied.
Block 1 CDR. For this review, the contractor delivered (1) the baselined  component and syste (3) all identified risks and mitigation plans for those risks. m requirements, (2) the interface descriptions, and  However, these deliverables did not demonstrate that all component-leve requirements were baselined and interface descriptions we As we previously reported, baselined requirements associated with the  NOC/SOC were not adequately defined at the time of the CDR, as  evidenced by the fact that they were significantly changed 2 months later.Program officials stated that while they knew that requirements were not  adequately baselined at this review, they believed that the interface  requirements were understood well enough to begin system development.  However, this was also not the case. Specifically, 39 of 90 NOC/SOC  interface requirements were removed from the baseline, and 2 new  interface requirements were added after CDR.  l  re understood.
Further, all relevant risks were not identified, and not all identified r had mitigation plans. Specifically, 7 of 31 identi mitigation plans, including risks associated with poorly established  requirements traceability and inadequately defined requirements for  integration suppliers. Moreover, the risks identified were as of May 200 prior to the beginning of CDR, and did not include four risks identifi ed  between June and October 2008, when CDR was concluded. For exam a risk associated with the instability of the C3I/COP software was not  addressed during CDR.  fied risks did not have  Without properly baselined requirements (including interfaces) and  proactive mitigation of known risks, system performance shortfalls are  likely. To illustrate, we previously reported that ambiguities in  requirements actually forced testers to rewrite test steps during exec based on interpretations of what they thought the requirements meant,  ution  and they required the SPO to incur the time and expense of conducting  multiple events to test NOC/SOC requirements.
NOC/SOC Component Qualification Test Rea SPO did not ensure that a well-defined test plan was in place, to include,  among other things, test cases and procedures and a traceability matrix  that maps each requirement to be tested to a corresponding test case.  Specifically, the contractor delivered the test plan on the day of the  review, rather than 10 days prior to the review, as required by the relev task order. Nevertheless, the SPO concluded the review based on its  review of the plan during the test readiness review. In this regard, we  previously reported problems with the NOC/SOC test plan, noting tha plan mapped 28 out of 100 requirements to incorrect test cases. Progra officials attributed the test plan limitations to, among other things,  insufficient time and resources to review the deliverables.
The SBInet independent verification and validation (IV&V)  also identified weaknesses within technical reviews. Specifically, the IV&V  contractor reported that the SPO was not provided with documentation,  including the test plan, early enough for the NOC/SOC test readiness  review to allow sufficient time for review. Moreover, in December 2009,  the program identified technical oversight of technical review milestones  as a major risk to the cost, schedule, and performance goals of the  program. According to program officials, they are developing a technical  review manual that is to supplement the SEP and provide detailed  guidance for conducting technical reviews. In commenting on a draft of  this report, DHS stated that it plans to complete and implement its  technical review guide by December 2010.
section_title: Contractor Management Reviews Were Conducted as Planned, but Were Limited by Unreliable Contractor Performance Data
paragraphs: Management reviews help to ensure that the contractor’s interpretation  and implementation of the requirements are consistent with those of the  program office. According to relevant guidance, organizations should have  written policies and procedures for conducting management reviews tha t,  among other things, (1) involve relevant stakeholders; (2) assign roles andresponsibilities for performing management review tasks; (3)  communicate project status information, including cost and schedule  information, and risks; and (4) identify, document, and track action items  to closure.
CBP policy also recognizes the importance of these reviews by requiring  the conduct of management reviews. Management Plan identifies the types of management reviews that are to  be conducted with the contractor. For SBInet, the primary management  The  review is known as the Joint Program Management Review (JPMR).  plan also identifies, for example, the stakeholders that are to participate in the reviews, including the program manager, project managers, program  control staff, and the risk management team; and it specifies the topics  that are to be discussed at the reviews, such as project status, cost and  schedule performance, and risks.
CBP, System Life Cycle Handbook, Version 1.2 (Sept. 30, 2008). risk management process and tool, including reviewing lessons learnedfrom other programs. The results of the review were presented during a  February 2010 briefing, and the action item was closed.
Effectively conducting management reviews has helped  program leadership with an understanding of the contractor’s progress  and the program’s exposure to risks so that appropriate corrective a can be taken and the chances of delivering a system solution that meets mission needs within budget are enhanced. However, as discussed in the  next section, the EVM performance data presented at these management reviews were not reliable, thus rendering those reviews, at best, limited in the extent to which they disclosed the true status of the program.
section_title: DHS Has Not Effectively Mo the Contractor’s Progress in Meeti Cost and Schedule Expectations
paragraphs: Measuring and reporting progress against cost and schedule expectations  ( i.e., baselines) is a vital element of effective contractor management and  oversight. As noted earlier, EVM provides a proven means for measuring  progress against cost and schedule commitments and thereby identifying  potential cost overruns and schedule delays early, when the impact can be minimized.
However, DH S has not ensured that its prime contractor’s EVM system,  which was certified as meeting relevant standards, has been effectively  implemented on SBInet. In particular, it has not ensured that performanc measurement baselines were validated in a timely manner, that establish baselines were complete and realistic, and that contractor-provided cost  and schedule data were reliable. Reasons cited by program officials for  these weaknesses include the instability in the scope of the work to be  performed, an unexpected temporary stop in Block 1 design and  deployment work when SBInet funding was redirected, and the  contractor’s use of estimated, rather than actual, costs for subcon work, which are subsequently adjusted when actual costs are rec Without effectively implementing EVM, DHS has not been positioned to  identify potential cost and schedule problems early, and thus has not bee able to take timely actions to correct problems and avoid program  schedule delays and cost increases.  eived.
subsections: section_title: Contractor EVM System Has Been Certified
paragraphs: In August 2005, the Office of Management and Budget issued guidance  that, among other things, directs agencies to ensure that EVM systems are  compliant with the American National Standards Institute (ANSI)  standard. The ANSI standard consists of 32 guidelines associated with a  sound EVM system that are intended to ensure that data are reliable and  can be used for informed decision-making.
The program office relies on the prime contractor’s EVM system to  provide cost and schedule performance data. This system was certified in  April 2005 by DCMA as being compliant with the ANSI standard. DCMA  certified the contractor’s EVM system again in February 2009.
Notwithstanding these certifications, DCMA identified a number of issues  with the contractor’s implementation of its EVM system. In particular, in  January 2010, DCMA reported that the SBInet prime contractor’s  implementation of EVM was not consistent with all of the 32 ANSI  guidelines. Specifically, DCMA identified concerns with the quality of  scheduling and reporting, and the identification of significant differences  between planned and actual cost and schedule performance, as well as  reasons for those differences.
section_title: Program Office Has Not Established Timely Performance Baselines
paragraphs: According to relevant guidance, the performance measurement baseline,  which is the foundation of an EVM system and the estimated cumulative  value of planned work, serves as the value against which performance is  measured for the life of the program or task order. As such, it should be  established as early as possible after contract or task order award, or  whenever a major contract modification or baseline change occurs. DHS  guidance further states that a baseline should be validated within 90 days  of the contract or task order award.
However, the program office validated a performance measurement  baseline within 90 days for only two of the six baselines that we reviewed  (see figure 4). For the other four, the length of time to establish a validated  baseline ranged from 5 to 10 months. For example, the program office  issued the ADTO in June 2008, and it did not establish a validated baseline  until 10 months later in April 2009. Similarly, in February 2009, the program  office modified the scope of the STO and extended the period of  performance, but it did not validate the revised baseline to include the  additional scope and time until 7 months later in September 2009. Figure 4  summarizes the periods of time during which earned value was, and was  s  not, measured against a validated baseline.  not, measured against a validated baseline.
According to program officials, the delays in validating performance  baselines were due to instability in the work to be performed, and the  need to temporarily stop Block 1 design and deployment work between  September 2008 and January 2009 because of DHS’s decision to redirect  funds from SBInet to the physical infrastructure. Without validated  baselines, DHS was not positioned to identify potential cost and schedule  problems early and to take timely corrective actions to mitigate those  problems.
section_title: DHS Has Not Established Reliable Performance Measurement Baselines
paragraphs: An integrated baseline review (IBR) is used to validate the performance  measurement baseline. This review is intended to verify that the baseline  is realistic and ensure that the contractor and the government mutually  understand scope, schedule, and risks for a given task order before a  substantial amount of work is performed. According to relevant guidance,  establishing a complete and realistic performance measurement baseline  includes (1) assigning responsibility for managing, tracking, and reporting  earned value data for work performed; (2) estimating needed resources  (i.e., budgets and staff), including management reserve, for performing  assigned tasks; (3) defining a product-oriented description of all work to  be performed; (4) scheduling all work in a time-phased sequence that  reflects the duration of the program’s activities; and (5) establishing  objective performance measures for each task.
In validating the performance measurement baselines for the four task  orders that we reviewed, the program office implemented two of the above  elements, but it did not implement the other three. Specifically, for each of  the six baselines associated with the task orders, the program office (1)  assigned responsibility for managing, tracking, and reporting earned value  data associated with each work breakdown structure element and (2)  estimated a time-phased budget, including the anticipated staff needed, for  each work breakdown structure element, and established a management  reserve. However, as discussed in the following section, the program  office did not (1) define a product-oriented description of all work to be  performed, (2) reliably estimate schedule baselines, and (3) adequately  measure earned value performance.
Program officials attribute these limitations in establishing comprehensive  baselines to instability in the nature of the work to be performed and the  prime contractor’s method for determining subcontractor performance.  Nevertheless, without complete and realistic baselines, the SPO has been  hampered in its ability to conduct meaningful measurement and oversight  of the prime contractor’s status and progress, as well as holding the  contractor accountable for results. More importantly, the lack of  meaningful measurement and oversight has contributed to program cost  overruns and schedule delays.
subsections: section_title: Work Breakdown Structure Was Not Product Oriented and Did Not Include All Work
paragraphs: According to relevant guidance, a work breakdown structure deconstructs  a program’s end product into successively smaller levels until the work is  subdivided to a level suitable for management control. Further, a work  breakdown structure should be product oriented and include all work to  be performed.
The work breakdown structure that was used to define each of the task  order baselines was not product oriented. Instead, it was defined in terms  of functions that span multiple system products, such as systems  engineering, system test and evaluation, and program management.
Additionally, the work breakdown structure did not reflect all work to be  performed. Specifically, for four of the six performance measurement  baselines, the work breakdown structure did not include all work  described in the corresponding task order’s statement of work. For  example, the work breakdown structure used to define the May 2008 STO  baseline did not include the work associated with identifying and selecting  components that meet system requirements and program security.  Similarly, DCMA reported in June 2008 that the work breakdown structure  included in this baseline did not account for all work identified in the  system task order.
section_title: Estimated Baseline Schedules Were Not Reliable
paragraphs: A reliable schedule provides a road map for systematic execution of a  program and the means by which to gauge progress, identify and address  potential problems, and promote accountability. Our research has  identified nine best practices associated with developing and maintaining  a reliable schedule: (1) capturing all activities, (2) sequencing all activities,  (3) assigning resources to all activities, (4) establishing the duration of all  activities, (5) integrating activities horizontally and vertically, (6)  establishing the critical path for all activities, (7) identifying reasonable  “float” between activities, (8) conducting a schedule risk analysis, and (9)  updating the schedule using logic and durations.
The six task order baselines were not reliable because they substantially  complied with only two of the eight key schedule estimating practices, and  they did not comply with, or only partially or minimally complied with, the  remaining six practices. (See figure 5 for a summary of the extent to  which each of the baseline schedules met each of the eight practices.)
Capturing all activities. The six schedules did not capture all activities  defined in the task order baseline. Specifically, five of the six schedules  did not reflect the work to be performed across the four task orders (i.e.,  integrated master schedule). Further, as previously mentioned, four of six  work breakdown structures were missing elements defined in the  respective task order statements of work. Moreover, two of the six  schedules did not reflect all work that was defined in the work breakdown  structure. For example, the December 2009 DTO schedule omitted efforts  associated with design work for TUS-1 and AJO-1.
Sequencing all activities. The six schedules substantially met this  practice. Each of the schedules identified almost all of the predecessor  and successor activities. However, each contained improper predecessor  and successor relationships. For example, the May 2008 STO baseline  included 52 of 538 activities (about 10 percent) with improper predecessor  and successor relationships. Additionally, many activities in four of the  schedules were constrained by “start no earlier than” dates. For example,  as previously reported, the September 2009 baseline schedule contained  403 of 1,512 activities (about 27 percent) with “start no earlier than”  constraints, which means that these activities are not allowed to start  earlier than their assigned dates, even if their respective predecessor  activities have been completed.
Assigning resources to all activities. Two of the six schedules partially  met this practice. Specifically, two schedules included resources;  however, those resources were allocated to less than 15 percent of the  activities identified in each schedule. Moreover, the remaining four  schedules did not include estimated resources. Instead, resources for all  six schedules were maintained separately as part of the contractor’s  earned value system and only available to DHS upon request.
Establishing the duration of all activities. Each of the six baseline  schedules substantially met this practice. Specifically, each schedule  established the duration of key activities and included baseline start and  end dates for most of the activities. Further, reasonable durations were  established for the majority of the activities in the schedules, meaning that  the durations established were less than 44 days. Nevertheless, each of the  schedules included activities that were not of short duration, that is, more  than 44 days. For example, the April 2009 ADTO baseline included 29 of  1,009 activities with durations ranging from 45 days to 352 days.
Integrating activities horizontally and vertically. Each of the  schedules partially met this practice. As mentioned previously, the six  schedules did not capture all activities defined in the task order baseline.  Further, four of six work breakdown structures were missing elements  defined in respective task order statements of work. Additionally, five of  six schedules did not reflect the work performed across the four task  orders (i.e., integrated master schedule), and each had improper  predecessor and successor relationships.
Establishing the critical path for all activities. Each of the six  schedules partially met this practice. Specifically, four of six work  breakdown structures were missing elements defined in the respective  task order statements of work. Additionally, four of the six schedules were  missing predecessor and successor activities, and each of the schedules  included improper predecessor and successor relationships. Further, five  of the six schedules did not reflect the work to be performed across the  four task orders (i.e., integrated master schedule). Unless all activities are  included and properly linked, it is not possible to generate a true critical  path.
Identifying reasonable float between activities. Each of the  schedules identified float; however, the amount of float was excessive. For  example, the February 2008 C3I/COP task order baseline included 259 of  294 activities (about 88 percent) with float greater than 100 days and 189  of the 259 (about 73 percent) with float in excess of 200 days.
Conducting a schedule risk analysis. DHS did not conduct a risk  analysis of any of the schedules.
section_title: Earned Value Performance Was Not Adequately Measured
paragraphs: According to the ANSI standard for EVM systems, only work for which  measurement is impractical may be classified as “level-of-effort.” Our  research shows that if more than 15 percent of a program’s budget is  measured using level-of-effort, then that amount should be scrutinized  because it does not allow schedule performance to be measured (i.e.,  performance equals planned work).
However, the six baselines had between 34 and 85 percent of the baseline  dollar value categorized as level-of-effort, including four with more than   50 percent (see table 2). Moreover, for five of the six baselines, program  documentation showed that the program office did not identify any action  items during the respective IBRs related to the high use of level-of-effort.
According to program officials, the STO, which categorized between 70  and 85 percent of the baseline dollar value as level-of-effort, includes many  program management activities (e.g., cost, schedule, and subcontractor  management). Nevertheless, they recognized that the level-of-effort for  this task order was high, and in November 2009, they directed the  contractor to minimize the use of level-of-effort for STO. According to  program officials, the high level-of-effort was due, in part, to the prime  contractor’s use of this measurement for subcontractor work. In  November 2009, DCMA stated that the SPO’s use of level-of-effort  activities was high, noting that this could be masking true contractor  performance.
section_title: Unreliable EVM Data Limit DHS’s Ability to Measure Contractor Performance
paragraphs: If performed properly, EVM can provide an objective means for measuring  program status and forecasting potential program cost overruns and  schedule slippages so that timely action can be taken to minimize their  impact. To do so, however, the underlying EVM data must be reliable,  meaning that they are complete and accurate and all data anomalies are  explained.
In the case of SBInet, the EVM data provided by the prime contractor for  the 21-month period ending in February 2010 have not been reliable, as  evidenced by numerous and unexplained anomalies in monthly EVM  reports. Reasons for the anomalies include the contractor’s use of  estimated, rather than actual, costs for subcontractor work, which are  subsequently adjusted when actual costs are received. Without reliable  performance data, the true status of the SBInet program is unclear, thus  limiting the SPO’s ability to identify potential cost and schedule shortfalls.
subsections: section_title: EVM Data Can Provide Objective Insight into Program Cost and Schedule Performance
paragraphs: EVM is a proven program measurement approach that, if implemented  appropriately, can create a meaningful and coherent understanding of a  program’s true health and status. As a result, the use of EVM can alert  decision makers to potential program problems sooner than is possible by  using actual versus planned expenditure alone, and thereby reduce the  chance and magnitude of program cost overruns and schedule slippages.
Simply stated, EVM measures the value of completed work in a given  period (i.e., earned value) against (1) the actual cost of work completed  for that period (i.e., actual cost) and (2) the value of the work that is  expected to be completed for that period (i.e., planned value). Differences  in these values are referred to as cost and schedule variances,  respectively.
Cost variances compare the value of the work completed with the actual  cost of the work performed. For example, if a contractor completed $5  million worth of work and the work actually cost $6.7 million, there would  be a negative $1.7 million cost variance.
Schedule variances are also measured in dollars, but they compare the  value of the work completed with the value of the work that was expected  to be completed. For example, if a contractor completed $5 million worth  of work at the end of the month but was expected to complete $10 million  worth of work, there would be a negative $5 million schedule variance.
Positive variances indicate that activities are costing less or are being  completed ahead of schedule. Negative variances indicate activities are  costing more or are falling behind schedule. To determine both cost and  schedule variances, all three values are necessary.
section_title: SBInet EVM Data Have Not Been Reliable
paragraphs: According to relevant guidance, EVM data should be valid and free from  unexplained anomalies (e.g., missing or negative values) because they can  limit program management’s ability to identify potential cost and schedule  shortfalls. Therefore, anomalies should be minimized for each of the  three values—earned value, planned value, and actual cost. Moreover, all  anomalies should be identified, and the reason for each should be fully  explained in the monthly EVM reports. To do less limits the completeness  and accuracy of these values, and thus makes the resulting variance  determinations unreliable. While an industry standard for what constitutes  an acceptable volume of anomalies does not exist, EVM experts in the  public and private sectors that we interviewed stated that the occurrence  of EVM data anomalies should be rare. Of these experts, some agreed that  an anomaly should occur in no more than 5 percent of the work  breakdown structure elements for a given contract or task order, while  some of these advocated an occurrence percentage of no more than 1-2  percent.
However, the EVM data that the prime contractor delivered to the SPO  from June 2008 through February 2010 (21 months) contained numerous,  unexplained anomalies. Specifically, the monthly EVM reports for all four  task orders that we reviewed showed one or more anomalies (e.g., missing  or negative values for earned value, planned value, and actual cost) in  each of the months that had a validated performance measurement  baseline. More specifically, the average number of work breakdown  structure elements across the four task orders that had data anomalies  during this 21-month period ranged from 11 percent to 41 percent. For the  C3I/COP task order in particular, the monthly percentage of work  breakdown structure elements with anomalies ranged between 25 and 67  percent over the 21 months. (See figure 6 for the percentage of work  breakdown structure elements with anomalies by month for each of the  four task orders.)
The October 2009 STO monthly EVM report illustrates how the anomalies  can distort the contractor’s performance. According to this report, about  $13,000 worth of work was planned to be completed on integration and  management, approximately $13,000 worth of work was performed, and  actual costs were about negative $550,000. Thus, the report erroneously  suggests that the contractor performed $13,000 worth of work, and  actually saved about $550,000 in doing so. Similarly, the September 2009  ADTO monthly report showed that about $200,000 worth of work was  planned to be completed on tower sites and infrastructure, and $25,000  worth of work was performed, but that no costs were incurred, suggesting  that the work was performed for free.
Exacerbating the large percentage of monthly data anomalies across the  four task orders is the fact that in most cases the reasons for the  anomalies were not explained in the monthly EVM variance analysis  reports. Specifically, about 79 percent of all anomalies across all four task  orders during the 21-month period were not explained. In particular, 82 of  119 (or about 69 percent) of all data anomalies for the STO task order  were not explained in the monthly reports, and none of the anomalies  were explained for DTO. (See figure 7 for the total number of data  anomalies and the number that were explained in the monthly reports  across the four task orders.)
Program officials acknowledged problems with the EVM data and stated  that they meet with the prime contractor each month to discuss the EVM  reports, including the reliability of the data. According to program  officials, limitations in the EVM data are due, in part, to the contractor’s  use of estimated, rather than actual, costs for subcontractor work, which  are subsequently adjusted when actual costs are received. Officials further  stated that they have been working with the contractor to reduce the  volume of unexplained anomalies, and they believe that the reliability of  the data has improved since February 2010. However, program officials  did not provide any documentation to support this statement. Without  reliable EVM data, the program office is unable to identify actual cost and  schedule shortfalls, which along with the other contractor tracking and  oversight weaknesses discussed in this report, has limited its ability to  effectively minimize program cost increases and schedule delays.
section_title: Conclusions
paragraphs: Effective management and oversight of a program’s prime contractor is  essential to successfully acquiring and deploying a system like SBInet.  Integral to accomplishing this is defining and implementing a range of  contractor management and oversight controls (e.g., processes and  practices) that reflect relevant federal guidance and best practices. To do  less increases the chances that contractor-delivered products and services  will not satisfy stated requirements and will not meet customer  expectations. The result is incurring the additional time and expense to  redo or rework contractor deliverables, accepting products and services  that do not perform as intended and do not meet mission needs, or both.
Overall, DHS has not done an effective job of managing and overseeing its  prime contractor, including monitoring the contractor’s performance. DHS  has largely defined key management and oversight processes and  practices that it should have followed, and it implemented a number of  these processes and practices. However, several key management and  oversight controls were not adequately defined, and essential controls  were not implemented. Most significantly, DHS did not adequately  document deliverable reviews and communicate the basis for rejecting  certain deliverables in writing to the contractor, which contributed to  deliverables that did not live up to expectations and necessitated rework  and caused later problems. Further, technical reviews were not grounded  in explicit criteria for determining when reviews should begin and  conclude, which also contributed to contract deliverables requiring costly  and time-consuming rework. In addition, the cost and schedule baselines  for measuring the contractor’s performance were frequently validated too  late and without sufficient accuracy and completeness to provide a  meaningful basis for understanding performance, which precluded DHS  from taking timely action to correct unfavorable results and trends.  Compounding these serious baseline limitations was contractor-provided  data about actual performance that were replete with unexplained  anomalies, thus rendering the data unfit for effective contractor  management and oversight. Notwithstanding of a number of contractor  management and oversight definition and implementation efforts that DHS  executed well, such as defining key processes and practices and training  key staff, these above-cited weaknesses collectively mean that DHS’s  management and oversight of its prime contractor has been a major  contributor to the SBInet program’s well-chronicled history of not  delivering promised system capabilities on time and on budget.
These limitations can be attributed to a number of factors, including gaps in  how certain processes and practices were defined, as well as not enforcing  other processes and practices that were defined and applicable and not  taking sufficient time to review deliverables that were submitted late. The  limitations can be further attributed to the fact that SBInet has from its  outset lacked clear definition and stability, and thus experienced continuous  change in scope and direction—an issue that we have previously reported  and made recommendations to address. Collectively, these factors have  helped to create a contractor management and oversight environment,  which, when combined with the many other acquisition management  weaknesses that we have previously reported about and made  recommendations to address, have produced a program that to date has not  been successful, and if not corrected, can become worse.
section_title: Recommendations for Executive Action
paragraphs: To improve DHS management and oversight of the SBInet prime  contractor, we recommend that the Secretary of Homeland Security direct  the Commissioner of the U.S. Customs and Border Protection to have the  SBI Executive Director, in collaboration with the SBInet Program  Director, take the following four actions:    Revise and implement, as applicable, contractor deliverable review  processes and practices to ensure that (1) contractor deliverables are  thoroughly reviewed and are not constrained by late contractor  deliverables and imposed milestones, (2) the reviews are sufficiently  documented, and (3) the acceptance or the rejection of each contractor  deliverable is communicated in writing to the contractor, to include  explicit explanations of the basis for any rejections.
Ensure that applicable entry and exit criteria for each technical review are  used and satisfied before initiating and concluding, respectively, a given  review.
Establish and validate timely, complete, and accurate performance  measurement baselines for each new task order or major modification of  an existing task order, as appropriate, to include, but not be limited to,  ensuring that (1) the work breakdown structure includes all work to be  performed, (2) baseline schedules reflect the key schedule estimating  practices discussed in this report, and (3) level-of-effort performance  measurement in excess of 15 percent is scrutinized, justified, and  minimized.
Ensure that all anomalies in contractor-delivered monthly earned value  management reports are identified and explained, and report periodically  to DHS acquisition leadership on relevant trends in the number of  unexplained anomalies.
Because we have already made recommendations in prior reports to  address the other management and oversight weaknesses discussed in this  report, such as those related to requirements management, risk  management, and Systems Engineering Plan implementation, we are not  making any additional recommendations at this time.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Director,  Departmental GAO/OIG Liaison Office and reprinted in appendix II, DHS  agreed with our four recommendations and described actions under way  or planned, which we summarize below, to address them.
With respect to our recommendation to revise and implement the  contractor deliverable review process, DHS stated that it is updating the  process to require written documentation of each review and the  communication to the contractor of review results.
With respect to our recommendation to ensure that entry and exit criteria  are used to initiate and conclude each technical review, DHS stated that it  has established an SBI Systems Engineering Directorate to focus on  technical oversight of the acquisition process, adding that the Directorate  is developing a technical review guide that describes in detail the review  process and the relevant entry and exit criteria for each technical review.
With respect to our recommendation to establish and validate timely,  complete, and accurate performance measurement baselines, DHS stated  that it is mindful of the need to establish and maintain current  performance baselines, and to plan and implement baseline updates as  completely and promptly as practicable, which it indicated is done through  IBRs. DHS also noted that while scheduling practices remain a challenge,  it continues to make improvements to its process, including implementing  scheduling tools and templates.
With respect to our recommendation to identify and explain all anomalies  in monthly EVM reports, and to report periodically relevant trends to DHS  acquisition leadership, DHS acknowledged the need to correctly document  anomalies in the monthly EVM reports, and stated that it is working with  DCMA to improve contractor quality control issues and the content of the  monthly EVM reports. It also stated that it is augmenting the reports with  routine conversations between contractor and project management staff.  The department also committed to advising the appropriate acquisition  leaders through established reporting and oversight opportunities as  issues arise with contractor performance or reporting.
Notwithstanding its agreement with our recommendations, the department  also commented that it took exception to selected findings and  conclusions regarding the program’s implementation of EVM. A summary  of DHS’s comments and our responses are provided below.
The department stated that it took exception to our finding that it did not  ensure performance measurement baselines were validated in a timely  manner, and said that it was not accurate to conclude that the lack of  validated baselines precluded the program office from identifying cost and  schedule problems and taking corrective action. In support of these  positions, the department made the following three points, which our  response addresses.
First, the department stated that the SBInet program office delayed formal  IBRs until it had finalized negotiated modifications to the task orders, and  in doing so, was able to complete an IBR within 90 days of each major task  order modification. We do not question whether the program office held  IBRs within 90 days of final negotiation of major task order modifications.  Our point is that DHS did not validate task order performance  measurement baselines (i.e., hold IBRs) within 90 days of task order  award, which is what DHS guidance states should occur. As our report  states, the program office only met this 90-day threshold for two of the six  baselines that we reviewed. Further, the length of time to validate the  performance baselines for the four task orders far exceeded 90 days (5 to  10 months), during which time DHS reports show that significant work  was performed and millions of dollars were expended. In fact, the DHS  reports show that most of the planned work for some of these task orders  had already been performed by the time the IBR was held and the baseline  was validated. As we state in our report, and DHS acknowledged in its  comments, the purpose of an IBR is to verify that the performance  baseline is realistic and that the scope, schedule, and risks are mutually  understood by the contractor and the government before a substantial  amount of work is performed.
Second, DHS commented that the program office maintained what it  referred to as “interim” performance measurement baselines during the  period of major program scope, schedule, and budget changes. We  acknowledge that in some cases the program office had these “interim”  baselines. However, these baselines are the contractor-provided baselines,  meaning that the program office and the contractor had not mutually  agreed to the scope, schedule, and risks associated with the work to be  performed. Moreover, for two of the task orders, the program office did  not have an “interim” baseline, even though the contractor performed  significant work under these task orders.
Third, the department stated that program leadership reviewed the  contractor’s technical and financial performance information relative to  performance measurement baselines and implemented management  actions as needed. We do not question whether program leadership  reviewed contractor-provided performance information or whether  actions to address problems may have been taken. However, our report  does conclude, as is discussed later in this section, that the combination of  the EVM weaknesses that our report cites, to include unreliable  performance baselines and contractor-provided performance data, did not  allow the program office to identify performance problems early and to  take timely actions to avoid the well-documented schedule delays and cost  increases that the program has experienced.
The department expressed two concerns with how it said our report  characterized and quantified EVM anomalies.
First, the department stated that our report failed to distinguish between  factual errors and legitimate monthly accounting adjustments. We agree  that our report does not distinguish between the two types of anomalies,  and would add that this was intentional because making the distinction  was not relevant to our finding. Specifically, our finding is that the reasons  for the numerous anomalies were not explained in the monthly EVM  variance analysis reports, therefore making the true status of the program  unclear.
Second, DHS stated that we incorrectly concluded that both errors and  adjustments are problematic, distort cost performance, and limit  management insight. In response, we did not conclude that all errors and  adjustments have these impacts, but rather that the lack of explanation  associated with such a large volume of anomalies made the true status of  the program unclear, thus limiting the program office’s ability to identify  actual cost and schedule shortfalls, which is certainly problematic.  Further, our report cites examples of cost performance data that provide a  distorted picture of actual performance vis-à-vis expectations.  Accordingly, the correct characterization of the report’s conclusion  concerning the reliability of EVM data is that the lack of explanation of the  numerous anomalies in monthly reports is problematic, provides a  distorted picture of cost performance, and limits management insight. To  this very point, DHS acknowledged in its comments the importance of  explaining the reason for anomalies in the monthly variance reports,  regardless of whether they are due to factual errors or accounting  adjustments.
The department stated that it took exception to our conclusion that the  program office’s lack of validated baselines in particular, and EVM  shortcomings in general, contributed to cost and schedule growth and  made it unable to identify cost and schedule problems early and take  corrective actions to avoid them. In response, we did not conclude that the  lack of validated baselines alone had either of these impacts. However, we  did conclude that the collection of EVM weaknesses discussed in our  report, to include untimely validated baselines, incomplete and unreliable  baselines, and unreliable performance data, together precluded the  program office from identifying problems early and taking corrective  action needed to avoid the program’s well-chronicled history of schedule  delays and cost increases. In support of this conclusion, we state in the  report, for example, that the performance measurement baselines that we  reviewed understated the cost and time necessary to complete the work  because they did not capture all work in the task orders’ statements of  work and because they were not grounded in a range of scheduling best  practices. Given that cost and schedule growth is a function of the  baseline against which actual cost and schedule performance is measured,  it follows logically that an understated baseline would produce actual cost  overruns and schedule delays. In addition, we would note that beyond  these EVM shortcomings, our report also recognizes other contract  tracking and oversight, test management, and requirements management  weaknesses that have collectively contributed to the program’s cost,  schedule, and performance shortfalls.
In addition to the above points, DHS provided technical comments, which  we have incorporated in the report as appropriate.
We are sending copies of this report to the Chairmen and Ranking  Members of the Senate and House Appropriations Committees and other  Senate and House committees and subcommittees that have authorization  and oversight responsibilities for homeland security. We will also send  copies to the Secretary of Homeland Security, the Commissioner of U.S.  Customs and Border Protection, and the Director of the Office of  Management and Budget. In addition, the report will be available at no  cost on the GAO Web site at http://www.gao.gov.
Should you or your offices have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine the extent to which the Department of  Homeland Security (DHS) (1) has defined and implemented effective  controls for managing and overseeing the Secure Border Initiative  Network (SBInet) prime contractor and (2) is effectively monitoring the  prime contractor’s progress in meeting cost and schedule expectations. To  accomplish our objectives, we focused on four key task orders—the  Arizona Deployment Task Order, the Design Task Order, the System Task  Order, and the Command, Control, Communications, and Intelligence/  Common Operating Picture Task Order—that are integral to the design,  development, and deployment of the first increment of SBInet, known as  Block 1. We also focused on the SBInet System Program Office’s (SPO)  contracting and oversight activities that occurred from June 2008 through  February 2010.
To determine the extent to which DHS has defined and implemented  effective controls for managing and overseeing the SBInet prime  contractor, we focused on training, verifying and accepting contract  deliverables, conducting technical reviews, and conducting management  reviews. We chose these four because each is important to tracking and  overseeing contractor performance for the four task orders.
Training. We compared relevant DHS contractor management and  oversight training requirements to the program managers’, contracting  officers’, and contracting officer’s technical representatives’ training  certifications.
Verifying and accepting contract deliverables. We compared U.S.
Customs and Border Protection’s (CBP) process for verifying and  accepting contract deliverables to leading industry practices to identify  any variances. We then assessed a nonprobability, random sample of 28  contract deliverables that the SPO identified as being delivered between  June 1, 2008, and August 31, 2009. We also judgmentally selected one  additional review that was conducted between September 1, 2009, and  February 28, 2010. For each of the 29 deliverables, we reviewed relevant  documentation, such as the contract deliverables, review comment forms,  and documented communications with the prime contractor indicating  acceptance or rejection of the deliverable, and compared them to the CBP  process and leading industry practices to determine what, if any,  deviations existed.
Technical reviews. We compared relevant DHS and CBP guidance and  entry and exit criteria in the task order Data Item Descriptions to leading  industry practices to identify any variances. We assessed a nonprobability,  random sample of technical reviews. Specifically, we assessed a technical  review from each of the eight unique combinations of task orders and  review types held between June 1, 2008, and August 31, 2009. We also  judgmentally selected one additional review that was conducted between  September 1, 2009, and February 28, 2010. For each of the nine reviews,  we compared the package of documentation prepared for and used during  these reviews to the criteria defined in the relevant task orders to  determine the extent to which the reviews satisfied the criteria.
Management reviews. We compared relevant CBP guidance to leading  industry practices to identify any variances. We then compared relevant  documentation prepared for and used during monthly joint program  management reviews to determine the extent to which the reviews  addressed cost, schedule, and program risks. We also assessed a  nonprobability sample of 11 action items that were identified during the  reviews held from October 2008 through October 2009, and assessed  relevant documentation to determine the extent to which they were  tracked to closure.
To determine the extent to which DHS is effectively monitoring the prime  contractor’s progress in meeting cost and schedule expectations, we  focused on the program’s implementation of earned value management  (EVM) because it was the tool used to monitor the contractor’s cost and  schedule performance. Specifically, we analyzed the six performance  measurement baselines, and the associated integrated baseline review  documentation, such as briefings, the work breakdown structure (WBS)  governing all task orders, task order statements of work, schedules,  monthly contract performance reports, control account plans, and  responsibility assignment matrixes. In doing so, we compared this  documentation to EVM and scheduling best practices as identified in our  Cost Estimating and Assessment Guide. Specifically, for each of the six  baselines:    We reviewed control account plans and responsibility assignment  matrixes to determine the period of performance and scope of work for  each baseline, compared the work described in the respective task order  statements of work to the work described in the responsibility assignment  matrix, and reviewed the control account plans to determine the extent to  which the level-of-effort measurement method was to measure contractor  performance.
We analyzed the schedule presented at each baseline review against eight  key schedule estimating practices in our Cost Estimating and Assessment  Guide. In doing so, we used commercially available software tools to  determine whether each schedule, for example, included all critical  activities, a logical sequence of activities, and reasonable activity  durations. Further, we characterized the extent to which the schedule met  each of the practices as either not met, minimally met, partially met,  substantially met, or met.
We analyzed the contract performance reports for each of the four task  orders for each month that there was a validated baseline. Specifically, we  identified instances of the following: (1) negative planned value, earned  value, or actual cost; (2) planned value and earned value without actual  cost; (3) earned value and actual cost without planned value; (4) actual  cost without planned value or earned value; (5) earned value without  planned value and actual cost; (6) inconsistencies between the estimated  cost at completion and the planned cost at completion; (7) actual cost  exceeding estimated cost at completion; and (8) planned or earned values  exceeding planned cost at completion. To determine the number of  anomalies, we identified each WBS element that had one or more of the  above anomalies. Then, we identified the number of WBS elements at the  beginning and the end of the baseline period of performance, and  calculated the average number of WBS elements. We used this to  determine the percentage of WBS elements with anomalies for each task  order and for each month for which there was a validated baseline.
To support our work across this objective, we interviewed officials from  the Department of Defense’s Defense Contract Management Agency  (DCMA), which provides contractor oversight services to the SPO,  including oversight of EVM implementation, and prime contractor  officials. We also reviewed DCMA monthly status reports and corrective  action reports.
For both objectives, we interviewed program officials to obtain  clarification on the practices, and to determine the reasons for any  deviations.
To assess the reliability of the data that we used to support the findings in  this report, we reviewed relevant program documentation to substantiate  evidence obtained through interviews with knowledgeable agency  officials, where available. We determined that the data used in this report  are sufficiently reliable. We have also made appropriate attribution  indicating the sources of the data.
We performed our work at the CBP headquarters and prime contractor  facilities in the Washington, D.C., metropolitan area and with DCMA  officials from Huntsville, Alabama. We conducted this performance audit  from June 2009 to October 2010 in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Deborah A. Davis (Assistant  Director), Tisha Derricotte, Neil Doherty, Kaelin Kuhn, Lee McCracken,  Jamelyn Payan, Karen Richey, Matt Snyder, Sushmita Srikanth, Stacey  Steele, and Matthew Strain made key contributions to this report.