Secure Border Initiative: DHS Needs to Strengthen Management and Oversight of Its Prime Contractor
Released: 2010-10-18T00:00:00

Letter

Background

Managed by U.S. Customs and Border Protection (CBP), SBInet is to strengthen DHS’s ability to detect, identify, classify, track, and respond to illegal breaches at and between ports of entry. It includes the acquisition, development, integration, deployment, and operations and maintenance of a mix of surveillance technologies, such as cameras, radars, sensors, and C3I technologies. Surveillance technologies include unattended ground sensors that can detect heat and vibrations associated with foot traffic and metal associated with vehicles, radar mounted on fixed towers that can detect movement, and cameras mounted on fixed towers that can be used to identify and classify items of interest detected and tracked by ground sensors and radar. These technologies are generally commercial off-the- shelf products. C3I technologies include customized software development to produce a common operating picture (COP)—a uniform presentation of activities within specific areas along the border—as well as the use of CBP network capabilities. Together, the surveillance technologies are to gather information along the border and transmit this information to COP terminals located in command centers, which are to assemble it to provide CBP agents with border situational awareness, to, among other things, enhance agents’ tactical decisionmaking regarding potential apprehensions.

Since fiscal year 2006, DHS has received about $4.4 billion in appropriations for SBI, including about $2.5 billion for physical fencing and related infrastructure, about $1.5 billion for virtual fencing (e.g., surveillance systems) and related infrastructure (e.g., towers), and about $300 million for program management. As of May 2010, DHS had obligated about $1.02 billion for SBInet.

Program and Acquisition Organizational Structure

The SBI Program Management Office, which is organizationally within CBP, is responsible for managing key acquisition functions associated with SBInet, including prime contractor management and oversight. It is organized into four components: the SBInet System Program Office (SPO), Operational Integration Division, Business Operations Division, and Systems Engineering Division. The SPO is responsible for such key contractor oversight activities as verifying and accepting contract deliverables and conducting contractor management and technical reviews. In addition, the Business Operations Division has primary responsibility for monitoring the prime contractor’s cost and schedule performance. As of May 2010, the SBI Program Management Office was staffed with 194 people—105 government employees, 78 contractor staff, and 11 detailees. (See figure 1 for a partial SBI program organization organization chart.) chart.)

In addition, CBP has engaged the Defense Contract Management Agency to, among other things, perform surveillance of the prime contractor’s EVM, systems engineering, hardware and software quality assurance, and risk management.

Further, the SBI Contracting Division, which is organizationally within the CBP Office of Administration Procurement Directorate’s Enterprise Contracting Office, is responsible for performing contract administration activities, such as maintaining the contract file and notifying the contractor in writing of whether deliverables are accepted or rejected. The SBI Contracting Division is organized into three branches: SBI Contracting, SBInet System Contracting, and SBInet Deployment Contracting. As of May 2010, the SBI Contracting Division was staffed with 22 people—18 government employees and 4 contractor staff. (See figure 2 for a partial SBI Contracting Division organization chart.)

Overview of SBInet Acquisition Strategy and Program Status

In September 2006, CBP awarded an Indefinite Delivery/Indefinite Quantity (IDIQ) prime contract to the Boeing Company. The contract was for 3 base years with three additional 1-year options for designing, producing, testing, deploying, and sustaining SBI. In September 2009, CBP exercised the first option year. Under the prime contract, CBP has issued 11 task orders that relate to SBInet, covering for example, COP design and development, system deployment, and system maintenance and logistics support. As of May 2010, 5 of the 11 task orders were complete and 6 were ongoing. (See table 1 for a summary of the SBInet task orders.)

Through the task orders, CBP’s strategy is to deliver SBInet capabilities incrementally. To accomplish this, the program office adopted an evolutionary system life cycle management approach in which system capabilities are to be delivered to designated locations in a series of discrete subsets of system functional and performance capabilities that are referred to as blocks. The first block (Block 1) includes the purchase of commercially available surveillance systems, development of customized COP systems and software, and use of existing CBP communications and network capabilities. According to program officials, as of July 2010, the Block 1 design was deployed to TUS-1 and was being deployed to AJO-1, both of which are located in CBP’s Tucson Sector of the southwest border. Together, these two deployments cover 53 miles of the 1,989-mile-long southern border. Also, according to program officials, as of July 2010, TUS-1 and AJO-1 were to be accepted in September 2010 and December 2010, respectively.

In January 2010, the Secretary of Homeland Security ordered an assessment of the SBI program. In addition, on March 16, 2010, the Secretary froze fiscal year 2010 funding for any work on SBInet beyond TUS-1 and AJO-1 until the assessment is completed. Also at that time, the Secretary redirected $35 million that had been allocated to SBInet Block 1 deployment to other tested and commercially-available technologies, such as truck-mounted cameras and radar, called Mobile Surveillance Systems, to meet near-term needs. According to the SBI Executive Director, the department’s assessment would consist of a comprehensive and science- based assessment to determine if there are alternatives to SBInet that may more efficiently, effectively, and economically meet U.S. border security needs. Further, the Executive Director stated that if the assessment suggests that the SBInet capabilities are worth the cost, DHS will extend its deployment to sites beyond TUS-1 and AJO-1. However, if the assessment suggests that alternative technology options represent the best balance of capability and cost-effectiveness, DHS will redirect resources to these other options. According to program officials, the initial phase of the assessment, which addresses the Arizona border, was completed in July 2010, and the results are currently being reviewed by senior DHS management. Officials were unable to provide a date for completion of the review.

Use of Acquisition Management Disciplines Is Important to Programs Like SBInet

Our research and evaluations of information technology programs have shown that delivering promised system capabilities and benefits on time and within budget largely depends on the extent to which key acquisition management disciplines are employed. These disciplines include, among others, requirements management, risk management, and test management. As is discussed in the following section, we have previously reported on the extent to which these disciplines have been employed on SBInet.

Contractor management and oversight, which is the focus of this report, is another acquisition management discipline. Among other things, this discipline helps to ensure that the contractor performs the requirements of the contract and the government receives the service or product intended. DHS acquisition policies and guidance, along with other relevant guidance, recognize the importance of these management and oversight disciplines. As we have previously reported, not employing them increases the risk that system acquisitions will not perform as intended and will require expensive and time-consuming rework.

GAO Has Previously Reported on Numerous SBInet Acquisition Management Weaknesses and Risks

Since 2007, we have identified a range of management weaknesses and risks facing SBInet, and we have made a number of recommendations to address them that DHS has largely agreed with, and to varying degrees, taken actions to address. For example, in February 2007, we reported that DHS had not fully defined activities, milestones, and costs for implementing the program or demonstrated how program activities would further the strategic goals and objectives of SBI. Further, we reported that the program’s schedule contained a high level of concurrency among related tasks and activities, which introduced considerable risk. Accordingly, we recommended that DHS define explicit and measurable commitments relative to, among other things, program capabilities, schedules, and costs, and re-examine the level of concurrency in the schedule and adjust the acquisition strategy appropriately.

In September 2008, we reported that a number of program management weaknesses put SBInet at risk of not performing as intended and taking longer and costing more to deliver than was necessary. Specifically, we reported the following: Important aspects of SBInet were ambiguous and in a continued state of flux, making it unclear and uncertain what technology capabilities were to be delivered when. For example, the scope and timing of planned SBInet deployments and capabilities had continued to change since the program began and remained unclear. Further, the SPO did not have an approved integrated master schedule to guide the execution of the program, and our assimilation of available information indicated that key milestones continued to slip. This schedule-related risk was exacerbated by the continuous change in and the absence of a clear definition of the approach used to define, develop, acquire, test, and deploy SBInet. Accordingly, we concluded that the absence of clarity and stability in these key aspects of SBInet impaired the ability of Congress to oversee the program and hold DHS accountable for results, and it hampered DHS’s ability to measure the program’s progress.

SBInet requirements had not been effectively defined and managed. While the SPO had issued guidance that defined key practices associated with effectively developing and managing requirements, the guidance was developed after several key activities had been completed. In the absence of this guidance, the SPO had not effectively performed key requirements development and management practices, such as ensuring that different levels of requirements were properly aligned. As a result, we concluded that the risk of SBInet not meeting mission needs and performing as intended was increased, as were the chances of the system needing expensive and time-consuming rework.

SBInet testing was not being effectively managed. For example, the program office had not tested the individual system components to be deployed to initial locations, even though the contractor had initiated integration testing of these components. Further, its test management strategy did not contain, among other things, a clear definition of testing roles and responsibilities or sufficient detail to effectively guide planning for specific test events.

To address these issues, we recommended that DHS assess and disclose the risks associated with its planned SBInet development, testing, and deployment activities and that it address the system deployment, requirements management, and testing weaknesses that we had identified. DHS largely agreed to implement our recommendations.

In September 2009, we reported that SBInet had continued to experience delays. For example, deployment to the entire southwest border had slipped from early 2009 to 2016, and final acceptance of TUS-1 and AJO-1 had slipped from November 2009 and March 2010 to December 2009 and June 2010, respectively. We did not make additional recommendations at that time.

In January 2010, we reported that SBInet testing was not being effectively managed. Specifically, we reported the following: Test plans, cases, and procedures for the most recent test events were not defined in accordance with important elements of relevant guidance. Further, a large percentage of the test cases for these events were changed extemporaneously during execution. While some of the changes were minor, others were more significant, such as rewriting entire procedures and changing the mapping of requirements to test cases. Moreover, these changes to procedures were not made in accordance with documented quality assurance processes, but rather were based on an undocumented understanding that program officials said they established with the contractor. Compounding the number and significance of changes were questions raised by the SPO and a support contractor about the appropriateness of some changes, noting that some changes made to system qualification test cases and procedures appeared to be designed to pass the test instead of being designed to qualify the system.

About 1,300 SBInet defects had been found from March 2008 through July 2009, with the number of new defects identified during this time generally increasing faster than the number being fixed—a trend that is not indicative of a system that is maturing and ready for deployment. While the full magnitude of these unresolved defects was unclear because the majority were not assigned a priority for resolution, some of the defects that had been found were significant. Although DHS reported that these defects had been resolved, they had nevertheless caused program delays, and related problems had surfaced that continued to impact the program’s schedule.

In light of these weaknesses, we recommended that DHS (1) revise the program’s overall test plan; (2) ensure that test schedules, plans, cases, and procedures are adequately reviewed and approved consistent with the revised test plan; (3) ensure that sufficient time is provided for reviewing and approving test documents prior to beginning a given test event; and (4) triage the full inventory of unresolved system problems, including identified user concerns, and periodically report on their status to CBP and DHS leadership. DHS fully agreed with the last three recommendations and partially agreed with the first.

In May 2010, we raised further concerns about the program and called for DHS to reconsider its planned investment in SBInet. Specifically, we reported the following: DHS had defined the scope of the first incremental block of SBInet capabilities that it intended to deploy and make operational; however, these capabilities and the number of geographic locations to which they were to be deployed continued to shrink. For example, the geographic “footprint” of the initially deployed capability has been reduced from three border sectors spanning about 655 miles to two sectors spanning about 387 miles.

DHS had not developed a reliable integrated master schedule for delivering the first block of SBInet. Specifically, the schedule did not sufficiently comply with seven of nine key practices that relevant guidance states are important to having a reliable schedule. For example, the schedule did not adequately capture all necessary activities, assign resources to them, and reflect schedule risks. As a result, it was unclear when the first block was to be completed, and continued delays were likely.

DHS had not demonstrated the cost-effectiveness of Block 1. In particular, it had not reliably estimated the costs of this block over its entire life cycle. Further, DHS had yet to identify expected quantifiable and qualitative benefits from this block and analyze them relative to costs. Without a meaningful understanding of SBInet costs and benefits, DHS lacks an adequate basis for knowing whether the initial system solution is cost effective.

DHS had not acquired the initial SBInet block in accordance with key life cycle management processes. While processes associated with, among other things, requirements development and management and risk management, were adequately defined they were not adequately implemented. For example, key risks had not been captured in the risk management repository and thus had not been proactively mitigated. As a result, DHS is at increased risk of delivering a system that does not perform as intended.

We concluded that it remains unclear whether the department’s pursuit of SBInet is a cost-effective course of action, and if it is, that it will produce expected results on time and within budget. Accordingly, we recommended that DHS (1) limit near-term investment in the first incremental block of SBInet, (2) economically justify any longer-term investment in SBInet, and (3) improve key program management disciplines. DHS largely agreed with our recommendations, and noted ongoing and planned actions to address each of them.

In June 2010, we reported several technical, cost, and schedule challenges facing SBInet, such as problems with radar functionality, and noted that the program office was staffed substantially below planned levels for government positions. We did not make any additional recommendations at that time.

DHS Has Largely Defined but Has Not Effectively Implemented the Full Range of Needed Contractor Management and Oversight Controls

Federal regulations and relevant guidance recognize effective contractor management and oversight as a key acquisition management discipline. In addition, they describe a number of practices associated with it, including training persons who are responsible for contractor management and oversight, verifying and deciding whether to accept contract deliverables, conducting technical reviews with the contractor to ensure that products and services will satisfy user needs, and holding management reviews with the contractor to monitor contractor performance.

To DHS’s credit, it has largely defined key practices aimed at employing each of these contractor management and oversight controls. Moreover, it has implemented some of them, such as training key contractor management and oversight officials and holding management reviews with the prime contractor. However, it has not effectively implemented others, such as documenting contract deliverable reviews and using entry and exit criteria when conducting technical reviews. Reasons for these weaknesses include limitations in the defined verification and acceptance deliverable process and a SPO decision to exclude some deliverables from the process, and insufficient time to review technical review documentation. Without employing the full range of practices needed to effectively manage and oversee the prime contractor, DHS is limited in its ability to know whether the contractor is meeting performance and product expectations. Moreover, it increases the chances that SBInet will not function as intended and will take more time and resources than necessary to deliver, which we have previously reported is the case for Block 1.

SBInet Contractor Management and Oversight Officials Have Been Properly Trained

Training supports the successful performance of relevant activities and tasks by helping to ensure that the responsible people have the necessary skills and expertise to perform the tasks. According to relevant guidance, organizations should define training expectations and should ensure that these expectations are met for individuals responsible for contractor management and oversight.

DHS’s acquisition management directives define training requirements for, among others, program managers, contracting officers, and contracting officer’s technical representatives (COTR). Specifically: Program managers must be certified at a level commensurate with their acquisition responsibilities. For a Level 1 information technology program, like SBInet, the designated program manager must be certified at a program management Level 3.

Contracting officers must be certified at the appropriate level to support their respective warrant authority.

COTRs must be trained and certified within 60 days of appointment to a contract or task order. The minimum mandatory requirements are the completion of 40 hours of COTR training and a 1-hour procurement ethics training course.

For SBInet, CBP has ensured that people in each of these key positions have been trained in accordance with DHS requirements. Specifically, the two program managers associated with the development and deployment of SBInet Block 1—the SBI Executive Director and the SPO Executive Director—were issued training certifications that qualify each as an Acquisition Professional. Further, each contracting officer responsible for executing actions on the Arizona Deployment Task Order (ADTO), the Design Task Order (DTO), the C3I/COP task order, and the System Task Order (STO) between June 2008 and February 2010 was certified commensurate with his or her respective warrant authority. In addition, for the same time period, each COTR assigned to each of the four task orders received DHS-issued certificates indicating that each had met the minimum training requirements before being assigned to a task order.

According to CBP officials, DHS leadership has made contractor management and oversight training a high priority, which helped to ensure that key officials were trained. By doing so, CBP has established one of the key controls to help ensure that prime contractor products and services meet expectations.

DHS Has Not Effectively Verified and Accepted All Contract Deliverables

Effectively implementing a well-defined process for verifying and accepting prime contractor deliverables is vital to SBInet’s success. DHS has a defined process for verifying and accepting contract deliverables, but this process does not ensure that deliverable reviews are sufficiently documented. Further, while the SPO has followed key aspects of this process, it has not effectively documented its review of certain deliverables and has not effectively communicated to the prime contractor the basis for rejecting all deliverables. Reasons for not doing so include limitations in the defined verification and acceptance process and a SPO decision to exclude some deliverables from the process. Without documenting all its reviews and effectively communicating the review results to the contractor, the SPO has increased the chances of accepting deliverables that do not meet requirements and having the contractor repeat work to correct deliverable problems.

CBP Has a Defined Process for Verifying and Accepting SBInet Contract Deliverables

The purpose of contractor deliverable verification and acceptance is to ensure that contractor-provided products and services meet specified requirements and otherwise satisfy the terms of the contract. According to relevant guidance, organizations should have written policies and procedures for verifying and accepting deliverables that, among other things, (1) assign roles and responsibilities for performing verification and acceptance tasks and (2) provide for conducting and documenting deliverable reviews and for effectively communicating to the contractor deliverable acceptance and rejection decisions.

To its credit, CBP has defined policies and procedures for verifying and accepting SBInet deliverables. Specifically, it issued its Deliverable Review and Approval Process in July 2007, which specifies how the SPO is to receive, review, and respond to all contract deliverables. Among other things, this guide assigns verification and acceptance roles and responsibilities to key program management positions. For example, it assigns the project manager responsibility for overseeing the review process and determining the acceptability of the deliverables, designates reviewers responsibilities for examining the deliverable, and assigns the contracting officer responsibility for communicating the decision to accept or reject the deliverable to the contractor. In addition, it provides for conducting deliverable reviews, which according to program officials, involves comparing the deliverable to the requirements enumerated in the applicable task order statement of work (typically within the Data Item Description). The process further specifies that the decision to accept or reject the deliverable is to be communicated in writing to the contractor.

However, the process does not state that the results of all reviews are to be documented. Instead, its states that a deliverable review comment form is to be prepared only when deficiencies or problems exist. If the deliverable is acceptable, the form does not need to be prepared. Program officials could not explain why review documentation was not required for acceptable deliverables. As a result, and as discussed in the next section of this report, the SPO cannot demonstrate its basis for accepting a number of deliverables, which in turn has increased the risk of, and actually resulted in, deliverables being accepted that do not meet requirements.

CBP Has Not Fully Followed its Defined Process for Verifying and Accepting SBInet Deliverables

The SPO followed its process for verifying and accepting SBInet-related deliverables about 62 percent of the time, based on the 29 deliverables that we reviewed. Specifically, the process was fully followed for 18 of the deliverables: (1) 6 that were accepted without documented review comments, (2) 5 that were accepted with documented review comments, and (3) 7 that were rejected with documented review comments. In addition, the acceptance or rejection of all 18 deliverables was communicated in writing to the contractor. For example, the ADTO Security Plan Addendum was delivered to the SPO in August 2008. The SPO reviewed the plan and documented its review comments, which included a determination that the plan did not address all required items specified in the task order’s Data Item Description. The CBP contracting officer subsequently notified the contractor in writing that the plan was rejected for this and other reasons. In February 2009, the contractor resubmitted the plan, the SPO reviewed the plan and documented its comments, and the contracting officer again notified the contractor in writing that the plan was rejected. The contractor resubmitted the plan in May 2009, and program officials documented their review of the deliverable. The contracting officer subsequently communicated the deliverable’s acceptance in writing to the contractor. (Figure 3 summarizes the number of contract deliverables that did and did not follow the defined process.)

The remaining 11 deliverables, however, did not fully adhere to the defined process. Of these, five were accepted without any documented review comments and without communicating the acceptance in writing to the contractor. The following are examples of these five and reasons for not adhering to the process.

Three of the five deliverables related to the C3I/COP task order did not require government approval. However, the Deliverable Review and Approval Process document does not provide for such treatment of these deliverables, and thus this practice is not in accordance with the process. Program officials told us that they have since recognized that this was a poor practice, and they have modified the task order to now require approval of all C3I/COP task order deliverables.

One of the five deliverables relates to the STO and is for the C2 Component Qualification Test package, which includes, among other things, the test plan and test cases and procedures for conducting the C2 qualification test event. In this case, the SPO accepted the test plan because, according to program officials, several days had passed since the deliverable was received, and they had not received any comments from the reviewers. They said that they therefore accepted the deliverable on the basis of not receiving any review comments to the contrary, but did not notify the contractor in writing of the acceptance.

The 11 deliverables also include 3 that were rejected without documented review comments and without the rejection being communicated to the contractor in writing. The following are examples of these three and reasons for not adhering to the process are discussed below.

One of the three deliverables relates to the C3I/COP task order and is for the Network Operations Center/Security Operations Center (NOC/SOC) Test Plan/Procedures/Description. According to program officials, the contractor did not submit the plan on time, thus requiring them to review it during the readiness review. Based on this review, the plan was rejected, which was communicated verbally to the contractor during the review. Despite rejecting the plan, the program office began testing the NOC/SOC component on the day of the review, without a revised plan being submitted, reviewed, and approved. According to program officials, this occurred in part because of insufficient time and resources to review contractor test-related deliverables.

Another one of the three deliverables also relates to the C3I/COP task order, and is for the Release 0.5 Software Test Plan/Procedures/ Description. According to program officials, the contractor submitted the plan late. The program office rejected the plan and provided oral comments during a teleconference prior to the review. Nevertheless, the test event again occurred without a revised plan being submitted, reviewed, and accepted. According to program officials, this was also due to insufficient time and resources to review the test plan. In this case, the plan was approved in late April 2009, which was 5 months after the test event was conducted.

The 11 deliverables also include 3 for which a decision to accept or reject the deliverable was not made. See the following examples: One of the three relates to the C3I/COP task order, and is for the NOC/SOC Interface Control Document. For this deliverable, review comments were not documented and no written communication with the contractor occurred. The deliverable was subsequently submitted three times and ultimately accepted. However, program officials could not explain whether the initial deliverable was accepted or rejected, or why the deliverable was submitted multiple times.

Another one of these relates to STO, and is for the Dynamic Object- Oriented Requirements System. For this deliverable, review comments were not documented, but CBP communicated in writing to the contractor that it was withholding comment on this submission of the deliverable and was to provide a consolidated set of comments on the subsequent submission. Subsequently, the contractor resubmitted the deliverable, and because it was accepted, the review was not documented. The contracting officer communicated the deliverable’s acceptance in writing to the contractor.

By not effectively verifying and accepting contractor deliverables, the SPO cannot ensure that the deliverables will satisfy stated requirements, thus increasing the risk of costly and time-consuming rework. For example, we recently reported that contractor-delivered test plans were poorly defined and resulted in problems during testing. In particular, NOC/SOC testing was hampered by requirements incorrectly mapped to test cases, did not provide for testing all requirements, and required significant extemporaneous changes to test cases during the test events. As a result of the testing problems, the SPO had to conduct multiple test events.

Key Contract Technical Reviews Have Not Been Adequately Conducted

Technical reviews are performed throughout the project life cycle to confirm that products and services being produced by the contractor provide the desired capability and ultimately satisfy user needs. To its credit, DHS has defined a process for conducting technical reviews, but it has not effectively implemented it. In particular, the SPO did not ensure that all key documentation was reviewed and relevant criteria were satisfied before concluding key technical reviews. Program officials attributed these limitations to the program’s aggressive schedule, which resulted in insufficient time to review relevant documentation. Concluding technical reviews without adequate justification has resulted in schedule delays and costly rework.

CBP Has a Defined Process for Conducting Technical Reviews

According to relevant guidance, organizations should have written policies and procedures for conducting technical reviews that, among other things, (1) assign roles and responsibilities for performing the specific technical review tasks and (2) establish entry and exit criteria to determine the readiness of the technical solution to proceed to the technical review and to demonstrate and confirm completion of required accomplishments.

To its credit, DHS has policies and guidance for conducting technical reviews. Specifically, DHS’s Systems Engineering Life Cycle outlines the key reviews to be performed as well as how these reviews are aligned with the department’s governance process. In addition, DHS guidance defines expectations for technical review exit criteria, stating that compliance with exit criteria is based upon the satisfaction of the content of the criteria, and not upon only the delivery of specified documents.

To augment DHS policy and guidance, the SBInet Systems Engineering Plan (SEP), dated November 2008, identifies and describes the technical reviews to be conducted. They include, for example: Requirements Review, which is to ensure that requirements have been completely and properly identified and are understood by the SPO and the contractor. Documentation associated with this review is to include, among other things, a requirements traceability matrix (i.e., a tool for demonstrating that component-level requirements are traceable to higher- level system level requirements).

Critical Design Review (CDR), which is to (1) demonstrate that the designs are complete and baselined and (2) ensure that the solution is ready for fabrication, coding, assembly, and integration. Documentation for this review is to include, among other things, (1) baselined requirements, (2) interface descriptions, and (3) identified risks and mitigation plans.

Test Readiness Review, which is to assess the readiness of the system solution to begin f include, among other things, (1) test plans that include test cases and procedures and (2) a traceability matrix that maps each requirement to tested to a corresponding test case. ormal testing. Documentation for this review is to In addition, the SBInet SEP describes high-level roles and responsibilities for performing these reviews, and establishes entry and exit criteria for each. For example, it states that the SPO program manager and the chief engineer are responsible for leading the reviews. Further, the SEP defines entry and exit criteria for the CDR. For example: Entry. System-level requirements should be traceable to component-level requirements; system internal and external interfaces should be defined.

Exit. Design baseline should be established and balanced across cost, schedule, performance, and risk considerations over the investment’s lifecycle; system risks should be identified and mitigation plans should be in place.

Contractor Technical Reviews Have Not Been Performed Adequately

The SPO did not follow the defined process for conducting technical reviews defined in the respective task orders to guide each review. However, the task orders do not define entry and exit criteria. Rather, they list a set of documents that the contractor is to provide and the SPO is to review. For example, for the Block 1 CDR, the relevant task order requires that the contractor deliver, among other documents, (1) baselined component and system requirements, (2) interface descriptions (i.e., descriptions of the data to be exchanged and the protocols used to exchange the data), and. Instead, program officials told us that they used the requirements (3) all identified risks and mitigation plans for those risks. However, the task orders do not associate these documents with either entry or exit criteria, and they do not specify characteristics or qualities that the documents are to satisfy.

Without explicit entry and exit criteria, the basis for beginning and e the technical reviews is unclear, thus increasing the risk that a program will be allowed to proceed and begin the next phase of development before it is ready to do so. In fact, this risk was realized for SBInet. Technical reviews were concluded without adequate justification, which ultimately resulted in problems that required additional time and resources to fix. For example: nding NOC/SOC Requirements Review. At this review, the contractor did not deliver a requirements traceability matrix, as required by the relevant task order, until almost a month after the review was completed. Nonetheless, program officials stated that they concluded the review in June 2008, without knowing whether the applicable higher-level system requirements were fully satisfied.

Block 1 CDR. For this review, the contractor delivered (1) the baselined component and syste (3) all identified risks and mitigation plans for those risks. m requirements, (2) the interface descriptions, and However, these deliverables did not demonstrate that all component-leve requirements were baselined and interface descriptions we As we previously reported, baselined requirements associated with the NOC/SOC were not adequately defined at the time of the CDR, as evidenced by the fact that they were significantly changed 2 months later.Program officials stated that while they knew that requirements were not adequately baselined at this review, they believed that the interface requirements were understood well enough to begin system development. However, this was also not the case. Specifically, 39 of 90 NOC/SOC interface requirements were removed from the baseline, and 2 new interface requirements were added after CDR. l re understood.

Further, all relevant risks were not identified, and not all identified r had mitigation plans. Specifically, 7 of 31 identi mitigation plans, including risks associated with poorly established requirements traceability and inadequately defined requirements for integration suppliers. Moreover, the risks identified were as of May 200 prior to the beginning of CDR, and did not include four risks identifi ed between June and October 2008, when CDR was concluded. For exam a risk associated with the instability of the C3I/COP software was not addressed during CDR. fied risks did not have Without properly baselined requirements (including interfaces) and proactive mitigation of known risks, system performance shortfalls are likely. To illustrate, we previously reported that ambiguities in requirements actually forced testers to rewrite test steps during exec based on interpretations of what they thought the requirements meant, ution and they required the SPO to incur the time and expense of conducting multiple events to test NOC/SOC requirements.

NOC/SOC Component Qualification Test Rea SPO did not ensure that a well-defined test plan was in place, to include, among other things, test cases and procedures and a traceability matrix that maps each requirement to be tested to a corresponding test case. Specifically, the contractor delivered the test plan on the day of the review, rather than 10 days prior to the review, as required by the relev task order. Nevertheless, the SPO concluded the review based on its review of the plan during the test readiness review. In this regard, we previously reported problems with the NOC/SOC test plan, noting tha plan mapped 28 out of 100 requirements to incorrect test cases. Progra officials attributed the test plan limitations to, among other things, insufficient time and resources to review the deliverables.

The SBInet independent verification and validation (IV&V) also identified weaknesses within technical reviews. Specifically, the IV&V contractor reported that the SPO was not provided with documentation, including the test plan, early enough for the NOC/SOC test readiness review to allow sufficient time for review. Moreover, in December 2009, the program identified technical oversight of technical review milestones as a major risk to the cost, schedule, and performance goals of the program. According to program officials, they are developing a technical review manual that is to supplement the SEP and provide detailed guidance for conducting technical reviews. In commenting on a draft of this report, DHS stated that it plans to complete and implement its technical review guide by December 2010.

Contractor Management Reviews Were Conducted as Planned, but Were Limited by Unreliable Contractor Performance Data

Management reviews help to ensure that the contractor’s interpretation and implementation of the requirements are consistent with those of the program office. According to relevant guidance, organizations should have written policies and procedures for conducting management reviews tha t, among other things, (1) involve relevant stakeholders; (2) assign roles andresponsibilities for performing management review tasks; (3) communicate project status information, including cost and schedule information, and risks; and (4) identify, document, and track action items to closure.

CBP policy also recognizes the importance of these reviews by requiring the conduct of management reviews. Management Plan identifies the types of management reviews that are to be conducted with the contractor. For SBInet, the primary management The review is known as the Joint Program Management Review (JPMR). plan also identifies, for example, the stakeholders that are to participate in the reviews, including the program manager, project managers, program control staff, and the risk management team; and it specifies the topics that are to be discussed at the reviews, such as project status, cost and schedule performance, and risks.

CBP, System Life Cycle Handbook, Version 1.2 (Sept. 30, 2008). risk management process and tool, including reviewing lessons learnedfrom other programs. The results of the review were presented during a February 2010 briefing, and the action item was closed.

Effectively conducting management reviews has helped program leadership with an understanding of the contractor’s progress and the program’s exposure to risks so that appropriate corrective a can be taken and the chances of delivering a system solution that meets mission needs within budget are enhanced. However, as discussed in the next section, the EVM performance data presented at these management reviews were not reliable, thus rendering those reviews, at best, limited in the extent to which they disclosed the true status of the program.

DHS Has Not Effectively Mo the Contractor’s Progress in Meeti Cost and Schedule Expectations

Measuring and reporting progress against cost and schedule expectations ( i.e., baselines) is a vital element of effective contractor management and oversight. As noted earlier, EVM provides a proven means for measuring progress against cost and schedule commitments and thereby identifying potential cost overruns and schedule delays early, when the impact can be minimized.

However, DH S has not ensured that its prime contractor’s EVM system, which was certified as meeting relevant standards, has been effectively implemented on SBInet. In particular, it has not ensured that performanc measurement baselines were validated in a timely manner, that establish baselines were complete and realistic, and that contractor-provided cost and schedule data were reliable. Reasons cited by program officials for these weaknesses include the instability in the scope of the work to be performed, an unexpected temporary stop in Block 1 design and deployment work when SBInet funding was redirected, and the contractor’s use of estimated, rather than actual, costs for subcon work, which are subsequently adjusted when actual costs are rec Without effectively implementing EVM, DHS has not been positioned to identify potential cost and schedule problems early, and thus has not bee able to take timely actions to correct problems and avoid program schedule delays and cost increases. eived.

Contractor EVM System Has Been Certified

In August 2005, the Office of Management and Budget issued guidance that, among other things, directs agencies to ensure that EVM systems are compliant with the American National Standards Institute (ANSI) standard. The ANSI standard consists of 32 guidelines associated with a sound EVM system that are intended to ensure that data are reliable and can be used for informed decision-making.

The program office relies on the prime contractor’s EVM system to provide cost and schedule performance data. This system was certified in April 2005 by DCMA as being compliant with the ANSI standard. DCMA certified the contractor’s EVM system again in February 2009.

Notwithstanding these certifications, DCMA identified a number of issues with the contractor’s implementation of its EVM system. In particular, in January 2010, DCMA reported that the SBInet prime contractor’s implementation of EVM was not consistent with all of the 32 ANSI guidelines. Specifically, DCMA identified concerns with the quality of scheduling and reporting, and the identification of significant differences between planned and actual cost and schedule performance, as well as reasons for those differences.

Program Office Has Not Established Timely Performance Baselines

According to relevant guidance, the performance measurement baseline, which is the foundation of an EVM system and the estimated cumulative value of planned work, serves as the value against which performance is measured for the life of the program or task order. As such, it should be established as early as possible after contract or task order award, or whenever a major contract modification or baseline change occurs. DHS guidance further states that a baseline should be validated within 90 days of the contract or task order award.

However, the program office validated a performance measurement baseline within 90 days for only two of the six baselines that we reviewed (see figure 4). For the other four, the length of time to establish a validated baseline ranged from 5 to 10 months. For example, the program office issued the ADTO in June 2008, and it did not establish a validated baseline until 10 months later in April 2009. Similarly, in February 2009, the program office modified the scope of the STO and extended the period of performance, but it did not validate the revised baseline to include the additional scope and time until 7 months later in September 2009. Figure 4 summarizes the periods of time during which earned value was, and was s not, measured against a validated baseline. not, measured against a validated baseline.

According to program officials, the delays in validating performance baselines were due to instability in the work to be performed, and the need to temporarily stop Block 1 design and deployment work between September 2008 and January 2009 because of DHS’s decision to redirect funds from SBInet to the physical infrastructure. Without validated baselines, DHS was not positioned to identify potential cost and schedule problems early and to take timely corrective actions to mitigate those problems.

DHS Has Not Established Reliable Performance Measurement Baselines

An integrated baseline review (IBR) is used to validate the performance measurement baseline. This review is intended to verify that the baseline is realistic and ensure that the contractor and the government mutually understand scope, schedule, and risks for a given task order before a substantial amount of work is performed. According to relevant guidance, establishing a complete and realistic performance measurement baseline includes (1) assigning responsibility for managing, tracking, and reporting earned value data for work performed; (2) estimating needed resources (i.e., budgets and staff), including management reserve, for performing assigned tasks; (3) defining a product-oriented description of all work to be performed; (4) scheduling all work in a time-phased sequence that reflects the duration of the program’s activities; and (5) establishing objective performance measures for each task.

In validating the performance measurement baselines for the four task orders that we reviewed, the program office implemented two of the above elements, but it did not implement the other three. Specifically, for each of the six baselines associated with the task orders, the program office (1) assigned responsibility for managing, tracking, and reporting earned value data associated with each work breakdown structure element and (2) estimated a time-phased budget, including the anticipated staff needed, for each work breakdown structure element, and established a management reserve. However, as discussed in the following section, the program office did not (1) define a product-oriented description of all work to be performed, (2) reliably estimate schedule baselines, and (3) adequately measure earned value performance.

Program officials attribute these limitations in establishing comprehensive baselines to instability in the nature of the work to be performed and the prime contractor’s method for determining subcontractor performance. Nevertheless, without complete and realistic baselines, the SPO has been hampered in its ability to conduct meaningful measurement and oversight of the prime contractor’s status and progress, as well as holding the contractor accountable for results. More importantly, the lack of meaningful measurement and oversight has contributed to program cost overruns and schedule delays.

Work Breakdown Structure Was Not Product Oriented and Did Not Include All Work

According to relevant guidance, a work breakdown structure deconstructs a program’s end product into successively smaller levels until the work is subdivided to a level suitable for management control. Further, a work breakdown structure should be product oriented and include all work to be performed.

The work breakdown structure that was used to define each of the task order baselines was not product oriented. Instead, it was defined in terms of functions that span multiple system products, such as systems engineering, system test and evaluation, and program management.

Additionally, the work breakdown structure did not reflect all work to be performed. Specifically, for four of the six performance measurement baselines, the work breakdown structure did not include all work described in the corresponding task order’s statement of work. For example, the work breakdown structure used to define the May 2008 STO baseline did not include the work associated with identifying and selecting components that meet system requirements and program security. Similarly, DCMA reported in June 2008 that the work breakdown structure included in this baseline did not account for all work identified in the system task order.

Estimated Baseline Schedules Were Not Reliable

A reliable schedule provides a road map for systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability. Our research has identified nine best practices associated with developing and maintaining a reliable schedule: (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying reasonable “float” between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations.

The six task order baselines were not reliable because they substantially complied with only two of the eight key schedule estimating practices, and they did not comply with, or only partially or minimally complied with, the remaining six practices. (See figure 5 for a summary of the extent to which each of the baseline schedules met each of the eight practices.)

Capturing all activities. The six schedules did not capture all activities defined in the task order baseline. Specifically, five of the six schedules did not reflect the work to be performed across the four task orders (i.e., integrated master schedule). Further, as previously mentioned, four of six work breakdown structures were missing elements defined in the respective task order statements of work. Moreover, two of the six schedules did not reflect all work that was defined in the work breakdown structure. For example, the December 2009 DTO schedule omitted efforts associated with design work for TUS-1 and AJO-1.

Sequencing all activities. The six schedules substantially met this practice. Each of the schedules identified almost all of the predecessor and successor activities. However, each contained improper predecessor and successor relationships. For example, the May 2008 STO baseline included 52 of 538 activities (about 10 percent) with improper predecessor and successor relationships. Additionally, many activities in four of the schedules were constrained by “start no earlier than” dates. For example, as previously reported, the September 2009 baseline schedule contained 403 of 1,512 activities (about 27 percent) with “start no earlier than” constraints, which means that these activities are not allowed to start earlier than their assigned dates, even if their respective predecessor activities have been completed.

Assigning resources to all activities. Two of the six schedules partially met this practice. Specifically, two schedules included resources; however, those resources were allocated to less than 15 percent of the activities identified in each schedule. Moreover, the remaining four schedules did not include estimated resources. Instead, resources for all six schedules were maintained separately as part of the contractor’s earned value system and only available to DHS upon request.

Establishing the duration of all activities. Each of the six baseline schedules substantially met this practice. Specifically, each schedule established the duration of key activities and included baseline start and end dates for most of the activities. Further, reasonable durations were established for the majority of the activities in the schedules, meaning that the durations established were less than 44 days. Nevertheless, each of the schedules included activities that were not of short duration, that is, more than 44 days. For example, the April 2009 ADTO baseline included 29 of 1,009 activities with durations ranging from 45 days to 352 days.

Integrating activities horizontally and vertically. Each of the schedules partially met this practice. As mentioned previously, the six schedules did not capture all activities defined in the task order baseline. Further, four of six work breakdown structures were missing elements defined in respective task order statements of work. Additionally, five of six schedules did not reflect the work performed across the four task orders (i.e., integrated master schedule), and each had improper predecessor and successor relationships.

Establishing the critical path for all activities. Each of the six schedules partially met this practice. Specifically, four of six work breakdown structures were missing elements defined in the respective task order statements of work. Additionally, four of the six schedules were missing predecessor and successor activities, and each of the schedules included improper predecessor and successor relationships. Further, five of the six schedules did not reflect the work to be performed across the four task orders (i.e., integrated master schedule). Unless all activities are included and properly linked, it is not possible to generate a true critical path.

Identifying reasonable float between activities. Each of the schedules identified float; however, the amount of float was excessive. For example, the February 2008 C3I/COP task order baseline included 259 of 294 activities (about 88 percent) with float greater than 100 days and 189 of the 259 (about 73 percent) with float in excess of 200 days.

Conducting a schedule risk analysis. DHS did not conduct a risk analysis of any of the schedules.

Earned Value Performance Was Not Adequately Measured

According to the ANSI standard for EVM systems, only work for which measurement is impractical may be classified as “level-of-effort.” Our research shows that if more than 15 percent of a program’s budget is measured using level-of-effort, then that amount should be scrutinized because it does not allow schedule performance to be measured (i.e., performance equals planned work).

However, the six baselines had between 34 and 85 percent of the baseline dollar value categorized as level-of-effort, including four with more than 50 percent (see table 2). Moreover, for five of the six baselines, program documentation showed that the program office did not identify any action items during the respective IBRs related to the high use of level-of-effort.

According to program officials, the STO, which categorized between 70 and 85 percent of the baseline dollar value as level-of-effort, includes many program management activities (e.g., cost, schedule, and subcontractor management). Nevertheless, they recognized that the level-of-effort for this task order was high, and in November 2009, they directed the contractor to minimize the use of level-of-effort for STO. According to program officials, the high level-of-effort was due, in part, to the prime contractor’s use of this measurement for subcontractor work. In November 2009, DCMA stated that the SPO’s use of level-of-effort activities was high, noting that this could be masking true contractor performance.

Unreliable EVM Data Limit DHS’s Ability to Measure Contractor Performance

If performed properly, EVM can provide an objective means for measuring program status and forecasting potential program cost overruns and schedule slippages so that timely action can be taken to minimize their impact. To do so, however, the underlying EVM data must be reliable, meaning that they are complete and accurate and all data anomalies are explained.

In the case of SBInet, the EVM data provided by the prime contractor for the 21-month period ending in February 2010 have not been reliable, as evidenced by numerous and unexplained anomalies in monthly EVM reports. Reasons for the anomalies include the contractor’s use of estimated, rather than actual, costs for subcontractor work, which are subsequently adjusted when actual costs are received. Without reliable performance data, the true status of the SBInet program is unclear, thus limiting the SPO’s ability to identify potential cost and schedule shortfalls.

EVM Data Can Provide Objective Insight into Program Cost and Schedule Performance

EVM is a proven program measurement approach that, if implemented appropriately, can create a meaningful and coherent understanding of a program’s true health and status. As a result, the use of EVM can alert decision makers to potential program problems sooner than is possible by using actual versus planned expenditure alone, and thereby reduce the chance and magnitude of program cost overruns and schedule slippages.

Simply stated, EVM measures the value of completed work in a given period (i.e., earned value) against (1) the actual cost of work completed for that period (i.e., actual cost) and (2) the value of the work that is expected to be completed for that period (i.e., planned value). Differences in these values are referred to as cost and schedule variances, respectively.

Cost variances compare the value of the work completed with the actual cost of the work performed. For example, if a contractor completed $5 million worth of work and the work actually cost $6.7 million, there would be a negative $1.7 million cost variance.

Schedule variances are also measured in dollars, but they compare the value of the work completed with the value of the work that was expected to be completed. For example, if a contractor completed $5 million worth of work at the end of the month but was expected to complete $10 million worth of work, there would be a negative $5 million schedule variance.

Positive variances indicate that activities are costing less or are being completed ahead of schedule. Negative variances indicate activities are costing more or are falling behind schedule. To determine both cost and schedule variances, all three values are necessary.

SBInet EVM Data Have Not Been Reliable

According to relevant guidance, EVM data should be valid and free from unexplained anomalies (e.g., missing or negative values) because they can limit program management’s ability to identify potential cost and schedule shortfalls. Therefore, anomalies should be minimized for each of the three values—earned value, planned value, and actual cost. Moreover, all anomalies should be identified, and the reason for each should be fully explained in the monthly EVM reports. To do less limits the completeness and accuracy of these values, and thus makes the resulting variance determinations unreliable. While an industry standard for what constitutes an acceptable volume of anomalies does not exist, EVM experts in the public and private sectors that we interviewed stated that the occurrence of EVM data anomalies should be rare. Of these experts, some agreed that an anomaly should occur in no more than 5 percent of the work breakdown structure elements for a given contract or task order, while some of these advocated an occurrence percentage of no more than 1-2 percent.

However, the EVM data that the prime contractor delivered to the SPO from June 2008 through February 2010 (21 months) contained numerous, unexplained anomalies. Specifically, the monthly EVM reports for all four task orders that we reviewed showed one or more anomalies (e.g., missing or negative values for earned value, planned value, and actual cost) in each of the months that had a validated performance measurement baseline. More specifically, the average number of work breakdown structure elements across the four task orders that had data anomalies during this 21-month period ranged from 11 percent to 41 percent. For the C3I/COP task order in particular, the monthly percentage of work breakdown structure elements with anomalies ranged between 25 and 67 percent over the 21 months. (See figure 6 for the percentage of work breakdown structure elements with anomalies by month for each of the four task orders.)

The October 2009 STO monthly EVM report illustrates how the anomalies can distort the contractor’s performance. According to this report, about $13,000 worth of work was planned to be completed on integration and management, approximately $13,000 worth of work was performed, and actual costs were about negative $550,000. Thus, the report erroneously suggests that the contractor performed $13,000 worth of work, and actually saved about $550,000 in doing so. Similarly, the September 2009 ADTO monthly report showed that about $200,000 worth of work was planned to be completed on tower sites and infrastructure, and $25,000 worth of work was performed, but that no costs were incurred, suggesting that the work was performed for free.

Exacerbating the large percentage of monthly data anomalies across the four task orders is the fact that in most cases the reasons for the anomalies were not explained in the monthly EVM variance analysis reports. Specifically, about 79 percent of all anomalies across all four task orders during the 21-month period were not explained. In particular, 82 of 119 (or about 69 percent) of all data anomalies for the STO task order were not explained in the monthly reports, and none of the anomalies were explained for DTO. (See figure 7 for the total number of data anomalies and the number that were explained in the monthly reports across the four task orders.)

Program officials acknowledged problems with the EVM data and stated that they meet with the prime contractor each month to discuss the EVM reports, including the reliability of the data. According to program officials, limitations in the EVM data are due, in part, to the contractor’s use of estimated, rather than actual, costs for subcontractor work, which are subsequently adjusted when actual costs are received. Officials further stated that they have been working with the contractor to reduce the volume of unexplained anomalies, and they believe that the reliability of the data has improved since February 2010. However, program officials did not provide any documentation to support this statement. Without reliable EVM data, the program office is unable to identify actual cost and schedule shortfalls, which along with the other contractor tracking and oversight weaknesses discussed in this report, has limited its ability to effectively minimize program cost increases and schedule delays.

Conclusions

Effective management and oversight of a program’s prime contractor is essential to successfully acquiring and deploying a system like SBInet. Integral to accomplishing this is defining and implementing a range of contractor management and oversight controls (e.g., processes and practices) that reflect relevant federal guidance and best practices. To do less increases the chances that contractor-delivered products and services will not satisfy stated requirements and will not meet customer expectations. The result is incurring the additional time and expense to redo or rework contractor deliverables, accepting products and services that do not perform as intended and do not meet mission needs, or both.

Overall, DHS has not done an effective job of managing and overseeing its prime contractor, including monitoring the contractor’s performance. DHS has largely defined key management and oversight processes and practices that it should have followed, and it implemented a number of these processes and practices. However, several key management and oversight controls were not adequately defined, and essential controls were not implemented. Most significantly, DHS did not adequately document deliverable reviews and communicate the basis for rejecting certain deliverables in writing to the contractor, which contributed to deliverables that did not live up to expectations and necessitated rework and caused later problems. Further, technical reviews were not grounded in explicit criteria for determining when reviews should begin and conclude, which also contributed to contract deliverables requiring costly and time-consuming rework. In addition, the cost and schedule baselines for measuring the contractor’s performance were frequently validated too late and without sufficient accuracy and completeness to provide a meaningful basis for understanding performance, which precluded DHS from taking timely action to correct unfavorable results and trends. Compounding these serious baseline limitations was contractor-provided data about actual performance that were replete with unexplained anomalies, thus rendering the data unfit for effective contractor management and oversight. Notwithstanding of a number of contractor management and oversight definition and implementation efforts that DHS executed well, such as defining key processes and practices and training key staff, these above-cited weaknesses collectively mean that DHS’s management and oversight of its prime contractor has been a major contributor to the SBInet program’s well-chronicled history of not delivering promised system capabilities on time and on budget.

These limitations can be attributed to a number of factors, including gaps in how certain processes and practices were defined, as well as not enforcing other processes and practices that were defined and applicable and not taking sufficient time to review deliverables that were submitted late. The limitations can be further attributed to the fact that SBInet has from its outset lacked clear definition and stability, and thus experienced continuous change in scope and direction—an issue that we have previously reported and made recommendations to address. Collectively, these factors have helped to create a contractor management and oversight environment, which, when combined with the many other acquisition management weaknesses that we have previously reported about and made recommendations to address, have produced a program that to date has not been successful, and if not corrected, can become worse.

Recommendations for Executive Action

To improve DHS management and oversight of the SBInet prime contractor, we recommend that the Secretary of Homeland Security direct the Commissioner of the U.S. Customs and Border Protection to have the SBI Executive Director, in collaboration with the SBInet Program Director, take the following four actions: Revise and implement, as applicable, contractor deliverable review processes and practices to ensure that (1) contractor deliverables are thoroughly reviewed and are not constrained by late contractor deliverables and imposed milestones, (2) the reviews are sufficiently documented, and (3) the acceptance or the rejection of each contractor deliverable is communicated in writing to the contractor, to include explicit explanations of the basis for any rejections.

Ensure that applicable entry and exit criteria for each technical review are used and satisfied before initiating and concluding, respectively, a given review.

Establish and validate timely, complete, and accurate performance measurement baselines for each new task order or major modification of an existing task order, as appropriate, to include, but not be limited to, ensuring that (1) the work breakdown structure includes all work to be performed, (2) baseline schedules reflect the key schedule estimating practices discussed in this report, and (3) level-of-effort performance measurement in excess of 15 percent is scrutinized, justified, and minimized.

Ensure that all anomalies in contractor-delivered monthly earned value management reports are identified and explained, and report periodically to DHS acquisition leadership on relevant trends in the number of unexplained anomalies.

Because we have already made recommendations in prior reports to address the other management and oversight weaknesses discussed in this report, such as those related to requirements management, risk management, and Systems Engineering Plan implementation, we are not making any additional recommendations at this time.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Director, Departmental GAO/OIG Liaison Office and reprinted in appendix II, DHS agreed with our four recommendations and described actions under way or planned, which we summarize below, to address them.

With respect to our recommendation to revise and implement the contractor deliverable review process, DHS stated that it is updating the process to require written documentation of each review and the communication to the contractor of review results.

With respect to our recommendation to ensure that entry and exit criteria are used to initiate and conclude each technical review, DHS stated that it has established an SBI Systems Engineering Directorate to focus on technical oversight of the acquisition process, adding that the Directorate is developing a technical review guide that describes in detail the review process and the relevant entry and exit criteria for each technical review.

With respect to our recommendation to establish and validate timely, complete, and accurate performance measurement baselines, DHS stated that it is mindful of the need to establish and maintain current performance baselines, and to plan and implement baseline updates as completely and promptly as practicable, which it indicated is done through IBRs. DHS also noted that while scheduling practices remain a challenge, it continues to make improvements to its process, including implementing scheduling tools and templates.

With respect to our recommendation to identify and explain all anomalies in monthly EVM reports, and to report periodically relevant trends to DHS acquisition leadership, DHS acknowledged the need to correctly document anomalies in the monthly EVM reports, and stated that it is working with DCMA to improve contractor quality control issues and the content of the monthly EVM reports. It also stated that it is augmenting the reports with routine conversations between contractor and project management staff. The department also committed to advising the appropriate acquisition leaders through established reporting and oversight opportunities as issues arise with contractor performance or reporting.

Notwithstanding its agreement with our recommendations, the department also commented that it took exception to selected findings and conclusions regarding the program’s implementation of EVM. A summary of DHS’s comments and our responses are provided below.

The department stated that it took exception to our finding that it did not ensure performance measurement baselines were validated in a timely manner, and said that it was not accurate to conclude that the lack of validated baselines precluded the program office from identifying cost and schedule problems and taking corrective action. In support of these positions, the department made the following three points, which our response addresses.

First, the department stated that the SBInet program office delayed formal IBRs until it had finalized negotiated modifications to the task orders, and in doing so, was able to complete an IBR within 90 days of each major task order modification. We do not question whether the program office held IBRs within 90 days of final negotiation of major task order modifications. Our point is that DHS did not validate task order performance measurement baselines (i.e., hold IBRs) within 90 days of task order award, which is what DHS guidance states should occur. As our report states, the program office only met this 90-day threshold for two of the six baselines that we reviewed. Further, the length of time to validate the performance baselines for the four task orders far exceeded 90 days (5 to 10 months), during which time DHS reports show that significant work was performed and millions of dollars were expended. In fact, the DHS reports show that most of the planned work for some of these task orders had already been performed by the time the IBR was held and the baseline was validated. As we state in our report, and DHS acknowledged in its comments, the purpose of an IBR is to verify that the performance baseline is realistic and that the scope, schedule, and risks are mutually understood by the contractor and the government before a substantial amount of work is performed.

Second, DHS commented that the program office maintained what it referred to as “interim” performance measurement baselines during the period of major program scope, schedule, and budget changes. We acknowledge that in some cases the program office had these “interim” baselines. However, these baselines are the contractor-provided baselines, meaning that the program office and the contractor had not mutually agreed to the scope, schedule, and risks associated with the work to be performed. Moreover, for two of the task orders, the program office did not have an “interim” baseline, even though the contractor performed significant work under these task orders.

Third, the department stated that program leadership reviewed the contractor’s technical and financial performance information relative to performance measurement baselines and implemented management actions as needed. We do not question whether program leadership reviewed contractor-provided performance information or whether actions to address problems may have been taken. However, our report does conclude, as is discussed later in this section, that the combination of the EVM weaknesses that our report cites, to include unreliable performance baselines and contractor-provided performance data, did not allow the program office to identify performance problems early and to take timely actions to avoid the well-documented schedule delays and cost increases that the program has experienced.

The department expressed two concerns with how it said our report characterized and quantified EVM anomalies.

First, the department stated that our report failed to distinguish between factual errors and legitimate monthly accounting adjustments. We agree that our report does not distinguish between the two types of anomalies, and would add that this was intentional because making the distinction was not relevant to our finding. Specifically, our finding is that the reasons for the numerous anomalies were not explained in the monthly EVM variance analysis reports, therefore making the true status of the program unclear.

Second, DHS stated that we incorrectly concluded that both errors and adjustments are problematic, distort cost performance, and limit management insight. In response, we did not conclude that all errors and adjustments have these impacts, but rather that the lack of explanation associated with such a large volume of anomalies made the true status of the program unclear, thus limiting the program office’s ability to identify actual cost and schedule shortfalls, which is certainly problematic. Further, our report cites examples of cost performance data that provide a distorted picture of actual performance vis-à-vis expectations. Accordingly, the correct characterization of the report’s conclusion concerning the reliability of EVM data is that the lack of explanation of the numerous anomalies in monthly reports is problematic, provides a distorted picture of cost performance, and limits management insight. To this very point, DHS acknowledged in its comments the importance of explaining the reason for anomalies in the monthly variance reports, regardless of whether they are due to factual errors or accounting adjustments.

The department stated that it took exception to our conclusion that the program office’s lack of validated baselines in particular, and EVM shortcomings in general, contributed to cost and schedule growth and made it unable to identify cost and schedule problems early and take corrective actions to avoid them. In response, we did not conclude that the lack of validated baselines alone had either of these impacts. However, we did conclude that the collection of EVM weaknesses discussed in our report, to include untimely validated baselines, incomplete and unreliable baselines, and unreliable performance data, together precluded the program office from identifying problems early and taking corrective action needed to avoid the program’s well-chronicled history of schedule delays and cost increases. In support of this conclusion, we state in the report, for example, that the performance measurement baselines that we reviewed understated the cost and time necessary to complete the work because they did not capture all work in the task orders’ statements of work and because they were not grounded in a range of scheduling best practices. Given that cost and schedule growth is a function of the baseline against which actual cost and schedule performance is measured, it follows logically that an understated baseline would produce actual cost overruns and schedule delays. In addition, we would note that beyond these EVM shortcomings, our report also recognizes other contract tracking and oversight, test management, and requirements management weaknesses that have collectively contributed to the program’s cost, schedule, and performance shortfalls.

In addition to the above points, DHS provided technical comments, which we have incorporated in the report as appropriate.

We are sending copies of this report to the Chairmen and Ranking Members of the Senate and House Appropriations Committees and other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We will also send copies to the Secretary of Homeland Security, the Commissioner of U.S. Customs and Border Protection, and the Director of the Office of Management and Budget. In addition, the report will be available at no cost on the GAO Web site at http://www.gao.gov.

Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine the extent to which the Department of Homeland Security (DHS) (1) has defined and implemented effective controls for managing and overseeing the Secure Border Initiative Network (SBInet) prime contractor and (2) is effectively monitoring the prime contractor’s progress in meeting cost and schedule expectations. To accomplish our objectives, we focused on four key task orders—the Arizona Deployment Task Order, the Design Task Order, the System Task Order, and the Command, Control, Communications, and Intelligence/ Common Operating Picture Task Order—that are integral to the design, development, and deployment of the first increment of SBInet, known as Block 1. We also focused on the SBInet System Program Office’s (SPO) contracting and oversight activities that occurred from June 2008 through February 2010.

To determine the extent to which DHS has defined and implemented effective controls for managing and overseeing the SBInet prime contractor, we focused on training, verifying and accepting contract deliverables, conducting technical reviews, and conducting management reviews. We chose these four because each is important to tracking and overseeing contractor performance for the four task orders.

Training. We compared relevant DHS contractor management and oversight training requirements to the program managers’, contracting officers’, and contracting officer’s technical representatives’ training certifications.

Verifying and accepting contract deliverables. We compared U.S.

Customs and Border Protection’s (CBP) process for verifying and accepting contract deliverables to leading industry practices to identify any variances. We then assessed a nonprobability, random sample of 28 contract deliverables that the SPO identified as being delivered between June 1, 2008, and August 31, 2009. We also judgmentally selected one additional review that was conducted between September 1, 2009, and February 28, 2010. For each of the 29 deliverables, we reviewed relevant documentation, such as the contract deliverables, review comment forms, and documented communications with the prime contractor indicating acceptance or rejection of the deliverable, and compared them to the CBP process and leading industry practices to determine what, if any, deviations existed.

Technical reviews. We compared relevant DHS and CBP guidance and entry and exit criteria in the task order Data Item Descriptions to leading industry practices to identify any variances. We assessed a nonprobability, random sample of technical reviews. Specifically, we assessed a technical review from each of the eight unique combinations of task orders and review types held between June 1, 2008, and August 31, 2009. We also judgmentally selected one additional review that was conducted between September 1, 2009, and February 28, 2010. For each of the nine reviews, we compared the package of documentation prepared for and used during these reviews to the criteria defined in the relevant task orders to determine the extent to which the reviews satisfied the criteria.

Management reviews. We compared relevant CBP guidance to leading industry practices to identify any variances. We then compared relevant documentation prepared for and used during monthly joint program management reviews to determine the extent to which the reviews addressed cost, schedule, and program risks. We also assessed a nonprobability sample of 11 action items that were identified during the reviews held from October 2008 through October 2009, and assessed relevant documentation to determine the extent to which they were tracked to closure.

To determine the extent to which DHS is effectively monitoring the prime contractor’s progress in meeting cost and schedule expectations, we focused on the program’s implementation of earned value management (EVM) because it was the tool used to monitor the contractor’s cost and schedule performance. Specifically, we analyzed the six performance measurement baselines, and the associated integrated baseline review documentation, such as briefings, the work breakdown structure (WBS) governing all task orders, task order statements of work, schedules, monthly contract performance reports, control account plans, and responsibility assignment matrixes. In doing so, we compared this documentation to EVM and scheduling best practices as identified in our Cost Estimating and Assessment Guide. Specifically, for each of the six baselines: We reviewed control account plans and responsibility assignment matrixes to determine the period of performance and scope of work for each baseline, compared the work described in the respective task order statements of work to the work described in the responsibility assignment matrix, and reviewed the control account plans to determine the extent to which the level-of-effort measurement method was to measure contractor performance.

We analyzed the schedule presented at each baseline review against eight key schedule estimating practices in our Cost Estimating and Assessment Guide. In doing so, we used commercially available software tools to determine whether each schedule, for example, included all critical activities, a logical sequence of activities, and reasonable activity durations. Further, we characterized the extent to which the schedule met each of the practices as either not met, minimally met, partially met, substantially met, or met.

We analyzed the contract performance reports for each of the four task orders for each month that there was a validated baseline. Specifically, we identified instances of the following: (1) negative planned value, earned value, or actual cost; (2) planned value and earned value without actual cost; (3) earned value and actual cost without planned value; (4) actual cost without planned value or earned value; (5) earned value without planned value and actual cost; (6) inconsistencies between the estimated cost at completion and the planned cost at completion; (7) actual cost exceeding estimated cost at completion; and (8) planned or earned values exceeding planned cost at completion. To determine the number of anomalies, we identified each WBS element that had one or more of the above anomalies. Then, we identified the number of WBS elements at the beginning and the end of the baseline period of performance, and calculated the average number of WBS elements. We used this to determine the percentage of WBS elements with anomalies for each task order and for each month for which there was a validated baseline.

To support our work across this objective, we interviewed officials from the Department of Defense’s Defense Contract Management Agency (DCMA), which provides contractor oversight services to the SPO, including oversight of EVM implementation, and prime contractor officials. We also reviewed DCMA monthly status reports and corrective action reports.

For both objectives, we interviewed program officials to obtain clarification on the practices, and to determine the reasons for any deviations.

To assess the reliability of the data that we used to support the findings in this report, we reviewed relevant program documentation to substantiate evidence obtained through interviews with knowledgeable agency officials, where available. We determined that the data used in this report are sufficiently reliable. We have also made appropriate attribution indicating the sources of the data.

We performed our work at the CBP headquarters and prime contractor facilities in the Washington, D.C., metropolitan area and with DCMA officials from Huntsville, Alabama. We conducted this performance audit from June 2009 to October 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Deborah A. Davis (Assistant Director), Tisha Derricotte, Neil Doherty, Kaelin Kuhn, Lee McCracken, Jamelyn Payan, Karen Richey, Matt Snyder, Sushmita Srikanth, Stacey Steele, and Matthew Strain made key contributions to this report.