Department of Homeland Security: Better Planning and Oversight Needed to Improve Complex Service Acquisition Outcomes
Released: 2008-05-08T00:00:00

Letter

Background

Over the last decade, the use of federal service contracting has increased and now accounts for over 60 percent of federal procurement dollars spent annually. A performance-based approach to federal service contracting was introduced during the 1990s, representing a shift from specifying the way in which contractors should perform work to specifying acquisition outcomes. Regardless of the contracting method, focusing on outcomes and collaboration among multiple stakeholders in the contracting process has been acknowledged as sound contract management. In 2000, federal procurement law established a performance- based approach as the preferred acquisition method for services. The Federal Acquisition Regulation requires all performance-based service acquisitions to include a performance work statement that describes outcome-oriented requirements in terms of results required rather than the methods of performance of the work; measurable performance standards describing how to measure contractor performance in terms of quality, timeliness, and quantity; and the method of assessing contract performance against performance standards, commonly accomplished through the use of a quality assurance surveillance plan.

A 1998 Office of Federal Procurement Policy (OFPP) study on performance-based contracts—based largely on contracts for basic services, such as janitorial or maintenance services—showed that a number of anticipated benefits had been achieved, including reduced acquisition costs, increased competition for contracts, and improved contractor performance. However, implementing a performance-based approach is often more difficult for complex acquisitions, such as information technology, than it is for basic services, because agencies begin with requirements that are less stable, making it difficult to establish measurable outcomes. Such complex acquisitions may need to have requirements and performance standards continually refined throughout the life-cycle of the acquisition for a contractor to deliver a valuable service over an extended period of time. OFPP also has noted in policy that certain types of services, such as research and development, may not lend themselves to outcome-oriented requirements.

To encourage agencies to apply a performance-based approach to service acquisitions, the Office of Management and Budget (OMB) established governmentwide performance targets, which increased to 50 percent of eligible service contract dollars for the current fiscal year. In January 2007, the congressionally mandated Acquisition Advisory Panel reported that performance-based acquisition has not been fully implemented in the federal government, despite OMB encouragement, and recommended that OMB adjust the governmentwide target to reflect individual agency assessments and plans. In May 2007, OMB’s OFPP issued a memo providing that agencies, at a minimum, were expected to meet targets established and report on them in their management plans. In response, DHS’s CPO established a performance-based target of 25 percent for fiscal year 2007, increasing to 40 percent by fiscal year 2010, that was included in DHS’s Performance-Based Management Plan. The Acquisition Advisory Panel also recommended that OFPP issue more explicit implementation guidance and create an “Opportunity Assessment” tool to help agencies identify when they should consider using this acquisition method.

Requirements and Standards Definition Influence Program Outcomes

Our work has found that performance-based acquisitions must be appropriately planned and structured to minimize the risk of the government receiving services that are over cost estimates, delivered late, and of unacceptable quality. Specifically, we have emphasized the importance of clearly defined requirements to achieving desired results and measurable performance standards to ensuring control and accountability. Prior GAO and DHS Inspector General reviews of complex DHS investments using a performance-based approach point to a number of shortcomings. For example, in June 2007, we reported that a performance-based contract for a DHS financial management system, eMerge2, lacked clear and complete requirements, which led to schedule delays and unacceptable contractor performance. Ultimately, the program was terminated after a $52 million investment. In March 2007, we similarly reported that the Coast Guard’s performance-based contract for replacing or modernizing its fleet of vessels and aircraft, Deepwater, had requirements that were set at unrealistic levels and were frequently changed. This resulted in cost escalation, schedule delays, and reduced contractor accountability. The DHS Inspector General has also indicated numerous opportunities for DHS to make better use of sound practices, such as well-defined requirements.

Consistent with our prior work, definition of requirements and performance standards influenced outcomes for the eight complex investments we reviewed. In using a performance-based approach, sound contracting practices dictate that required contract outcomes or requirements be well-defined, providing clear descriptions of results to be achieved. While all eight contracts for these investments had outcome- oriented requirements, the requirements were not always well-defined.

Further, contracts for half of the investments did not have a complete set of measurable performance standards. Appendix I provides a summary of our analysis of the requirements, performance standards, and outcomes for the eight performance-based contracts for major investments we reviewed.

Complex investments with contracts that did not have well-defined requirements or complete measurable performance standards at the time of contract award or start of work experienced either cost overruns, schedule delays, or did not otherwise meet performance expectations. For example, contracts for systems development for two CBP major investments lacked both well-defined requirements and measurable performance standards prior to the start of work and both experienced poor outcomes. The first, for DHS’s Automated Commercial Environment (ACE) Task Order 23 project—a trade software modernization effort—was originally estimated to cost $52.7 million over a period of approximately 17 months. However, the program lacked stable requirements at contract award and, therefore, could not establish measurable performance standards and valid cost or schedule baselines for assessing contractor performance. Software requirements were added after contract award, contributing to a project cost increase of approximately $21.1 million, or 40 percent, over the original estimate. Because some portions of the work were delayed to better define requirements, the project is not expected to be completed until June 2009—about 26 months later than planned.

The second, Project 28 for systems development for CBP’s Secure Border Initiative (SBInet)—a project to help secure a section of the United States- Mexico border using a surveillance system—did not meet expected outcomes due to a lack of both well-defined requirements and measurable performance standards. CBP awarded the Project 28 contract planned as SBInet’s proof of concept and the first increment of the fielded SBInet system before the overall SBInet operational requirements and system specifications were finalized. More than 3 months after Project 28 was awarded, DHS’s Inspector General reported that CBP had not properly defined SBInet’s operational requirements and needed to do so quickly to avoid rework of the contractor’s systems engineering. We found that several performance standards were not clearly defined to isolate the contractor’s performance from that of CBP employees, making it difficult to determine whether any problems were due to the contractor’s system design, CBP employees, or both. As a result, it was not clear how CBP intended to measure compliance with the Project 28 standard for probability of detecting persons attempting to illegally cross the border. Although it did not fully meet user needs and its design will not be used as a basis for future SBInet development, DHS fully accepted the project after an 8-month delay. In addition, DHS officials have stated that much of the Project 28 system will be replaced by new equipment and software.

Conversely, we found that contracts with well-defined requirements linked to measurable performance standards delivered results within budget and provided quality service. For example, contracted security services at the San Francisco International Airport for TSA’s Screening Partnership Program had well-defined requirements, and all measurable performance standards corresponded to contract requirements—an improvement from our prior reviews of the program. The requirements for gate, checkpoint, and baggage screening services clearly stated that the contractor should use technology and staff to prevent prohibited items from entering sterile areas of the airport and should work to minimize customer complaints while addressing in a timely manner any complaints received. The performance standards assessed how often screeners could successfully detect test images of prohibited items in checked baggage; the percentage of audited records and inspected equipment, property, and materials that were well-kept, operational, and recorded on maintenance logs; and whether all new hires received the required training before assuming their screening responsibilities. In terms of expected outcomes, the contractor achieved a 2.2 percent cost underrun during the first 5 months of the contract and exceeded most requirements.

Unreliable Data and Lack of Management Review Constrain Oversight

In managing its service acquisitions, including those that are performance- based, DHS has faced oversight challenges, including a lack of reliable data and systematic management reviews. DHS contracting and program representatives told us that they use a performance-based approach to the maximum extent practicable. However, DHS does not have reliable data— either from the Federal Procurement Data System-Next Generation (FPDS-NG), the governmentwide database for procurement spending, or at a departmentwide level—to systematically monitor or evaluate or report on service acquisitions, including those that are performance-based. Reliable data are essential to overseeing and assessing the implementation of contracting approaches, acquisition outcomes, and making informed management decisions. Moreover, the Chief Procurement Officer (CPO), who has responsibility for departmentwide procurement oversight, has begun some initial review of performance-based service acquisitions, but has not conducted systematic management assessments of this acquisition method.

Our analysis of information provided by contracting representatives at the Coast Guard, CBP, Immigration and Customs Enforcement (ICE), and TSA showed that about 51 percent of the 138 contracts we identified in FPDS- NG as performance-based had none of the required performance-based elements: a performance work statement, measurable performance standards, and a method of assessing contractor performance against performance standards. Only 42 of the 138 contracts, or 30 percent, had all of the elements, and about 18 percent had some but not all of the required performance-based acquisition elements (see table 1).

Lacking reliable FPDS-NG data, reports on the use of performance-based contracts for eligible service obligations are likely inaccurate. Data reported on the use of performance-based contracts by service types— ranging from basic, such as janitorial and landscaping, to complex, such as information technology or systems development—requested by OFPP in July 2006—are also likely misleading. The Acquisition Advisory Panel and DHS’s CPO also have raised concerns regarding the accuracy of the performance-based designation in FPDS-NG. The Acquisition Advisory Panel’s 2007 report noted from its review at 10 federal agencies that 42 percent of the performance-based contracts the panel reviewed had been incorrectly coded.

Inaccurate federal procurement data is a long-standing governmentwide concern. Our prior work and the work of the General Services Administration’s Inspector General have noted issues with the accuracy and completeness of FPDS and FPDS-NG data. OMB has stressed the importance of submitting timely and accurate procurement data to FPDS- NG and issued memos on this topic in August 2004 and March 2007. Accurate FPDS-NG data could facilitate the CPO’s departmentwide oversight of service acquisitions, including those that are performance- based.

At a departmentwide level, CPO representatives responsible for procurement oversight indicated that they have not conducted systematic assessments including costs, benefits, and other outcomes of a performance-based approach. To improve the implementation of performance-based acquisitions, CPO representatives established a work group in May 2006 to leverage knowledge among DHS components. They also noted that they are working with OFPP to develop a best practices guide on measurable performance standards and to gather good examples of performance-based contracts. In addition, the CPO has implemented a departmentwide acquisition oversight program, which was designed with the flexibility to address specific procurement issues, such as performance-based service acquisitions, and is based on a series of component-level reviews. Some initial review of performance-based acquisitions has begun under this program, but management assessment or evaluation of the outcomes of this acquisition method has not been conducted.

Conclusion and Recommendations

Consistent with federal procurement policy, DHS has emphasized a performance-based approach to improve service acquisition outcomes. However, in keeping with our prior findings, DHS’s designation of a service acquisition as performance-based was not as relevant as the underlying contract conditions. Sound acquisition practices, such as clearly defining requirements and establishing complementary measurable performance standards, are hallmarks of successful service acquisitions. In the cases we reviewed as well as in prior findings where these key elements were lacking, DHS did not always achieve successful acquisition outcomes. The report we are releasing today recommends that the Secretary of Homeland Security take several actions to increase DHS’s ability to achieve improved outcomes for its service acquisitions, including those that are performance-based. These actions include routinely assessing requirements for complex investments to ensure that they are well-defined and developing consistently measurable standards linked to those requirements; systematically evaluating outcomes of major investments and relevant contracting methods; and improving the quality of FPDS-NG data to facilitate identifying and assessing the use of various contracting methods. DHS generally concurred with our recommendations, noting some departmental initiatives under way to improve acquisition management. However, the department’s response did not address how the CPO’s process and organizational changes at the departmental level will impact component-level management and assessment of complex acquisitions to improve outcomes. Improving acquisition management has been an ongoing challenge since the department was established and requires sustained management attention.

Mr. Chairman, this concludes my prepared statement. I would be pleased to respond to any questions that you or other members of the committee may have at this time.

Contacts and Acknowledgments

For further information about this statement, please contact John P. Hutton at (202) 512-4841 or huttonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Staff making key contributions to this statement were Amelia Shachoy, Assistant Director; Jeffrey Hartnett; Sean Seales; Karen Sloan; and Don Springman.

Appendix I: Key Characteristics of Eight Performance-Based Service Contracts

Research, analysis, and financial and information management

Contractor submitted all required documentation on time; met project management quality standards; and maintained electronic archiving and restoration standards.

Trade systems software development (task order 23)

Costs increased by 40 percent ($21.1 million). More than a year behind schedule; unplanned software redesign.

Costs increased by 53 percent ($24 million). Maintenance wait times were longer than planned.

DHS rejected initial acceptance of Project 28. The project was delayed 8 months with final acceptance in February 2008. DHS noted that the contractor met the requirements, but the project did not fully meet DHS’s needs and the technology will not be replicated in future SBInet development.

Contractor exceeded the performance standard for machine downtime with a score 1 hour less than required and operated at cost through the second quarter of fiscal year 2007.

Contractor exceeded most performance standards; for example, threat detection performance and false alarm rates exceeded the quality standards. Contractor had cost underrun of 2.2 percent ($677,000).

Initial contractor planning reports were inadequate; system experienced operational downtime; surveillance reports identified poor contractor performance. Contractor generally met time frames and delivered within budget.

Outcomes not available at the time of our review.

Legend: contract met or mostly met the criteria; contract partially met the criteria; contract did not meet the criteria.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.