Department of Homeland Security: Better Planning and Oversight Needed to Improve Complex Service Acquisition Outcomes
Published: 2008-05-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security (DHS) has relied on service acquisitions to meet its expansive mission. In fiscal year 2006, DHS spent $12.7 billion to procure services. To improve service acquisition outcomes, federal procurement policy establishes a preference for a performance-based approach, which focuses on developing measurable outcomes rather than prescribing how contractors should perform services. This testimony focuses on how contract outcomes are influenced by how well DHS components have defined and developed contract requirements and performance standards, as well as the need for improved assessment and oversight to ensure better acquisition outcomes. GAO's statement is based on its report being released today, which reviewed judgmentally selected contracts for eight major investments at three DHS components--the Coast Guard, Customs and Border Protection (CBP), and the Transportation Security Administration (TSA)-- totaling $1.53 billion in fiscal years 2005 and 2006; prior GAO and DHS Inspector General reviews; management documents and plans; and related data, including 138 additional contracts, primarily for basic services from the Coast Guard, CBP, TSA, and Immigration and Customs Enforcement.
section_title: What GAO Found
paragraphs: Over the past several years, GAO has found that appropriate planning, structuring, and monitoring of agency service acquisitions, including those that are performance-based, can help minimize the risk of cost overruns, delayed delivery, and unacceptably quality. Several prior GAO and DHS Inspector General reviews of major DHS investments using a performance-based approach point to such shortcomings. While all of the contracts GAO reviewed at the Coast Guard, CBP, and TSA had outcome-oriented requirements, contracts for four of the eight investments did not have well-defined requirements, or a complete set of measurable performance standards, or both at the time of contract award or start of work. These service contracts experienced cost overruns, schedule delays, or did not otherwise meet performance expectations. In contrast, contracts for the other four investments had well-defined requirements linked to measurable performance standards and met the standards for contracts that had begun work. In managing its service acquisitions, including those that are performance-based, DHS has faced oversight challenges that have limited its visibility over service acquisitions and its ability to make informed acquisition management decisions. Notably, the department lacks reliable data on performance-based service acquisitions. About half of the 138 contracts identified by DHS as performance-based had none of the elements DHS requires for such contracts: a performance work statement, measurable performance standards, or a quality assurance surveillance plan. Such inaccurate data limit DHS's ability to perform management assessments of these acquisitions. In addition, the Chief Procurement Officer, who is responsible for departmentwide procurement oversight, has not conducted management assessments of performance-based service acquisitions. To help DHS improve outcomes for its service acquisitions, including those that are performance-based, GAO recommended that DHS routinely assess requirements for complex investments to ensure that they are well-defined, and develop consistently measurable performance standards linked to those requirements. GAO also recommended that DHS systematically evaluate the outcomes of major investments and relevant contracting methods and improve the quality of data to facilitate identifying and assessing the use of various contracting methods. DHS generally concurred with GAO's recommendations, noting some departmental initiatives to improve acquisition management.

section_title: Letter
paragraphs: Thank you for inviting me here today to discuss the Department of  Homeland Security’s (DHS) planning and assessment of its complex  service acquisitions. To meet its expansive homeland security mission,  DHS spends billions of dollars on service acquisitions for critical trade,  transportation, and border security investments. In fact, more than 80  percent of DHS’s total procurement dollars are spent on services. Prior  GAO work has found that appropriate planning, structuring, and  monitoring of acquisitions is critical to ensuring that the services provided  meet the government’s needs. To help improve service acquisition  outcomes, federal procurement policy calls for agencies to use a  performance-based approach to the maximum extent practicable. This  approach includes a performance work statement that describes outcome- oriented requirements, measurable performance standards, and quality  assurance surveillance. If properly implemented, these characteristics can  help ensure that contracted services meet cost, schedule, and performance  requirements. Other factors, such as pressure to get programs up and  running, additional external requirements, and technological challenges  also impact the ability to achieve good acquisition outcomes.
While a performance-based approach has been widely accepted, we have  found that agencies face certain challenges in implementing the  approach—especially for complex or major investments. My testimony  today will focus on the particular challenges DHS has confronted.  Specifically, I will discuss how contract outcomes are influenced by how  well DHS components have defined and developed contract requirements  and performance standards. I will also discuss the need for improved  assessment and oversight to ensure better outcomes.
My statement is based on our report that is being released today. This  report focused on contracts for major investments with complex service  acquisitions at the Coast Guard, Customs and Border Protection (CBP),  and the Transportation Security Administration (TSA)—three of the DHS  components reporting among the highest obligations for performance- based service acquisitions in fiscal years 2005 and 2006. We conducted this  performance audit in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Over the last decade, the use of federal service contracting has increased  and now accounts for over 60 percent of federal procurement dollars  spent annually. A performance-based approach to federal service  contracting was introduced during the 1990s, representing a shift from  specifying the way in which contractors should perform work to  specifying acquisition outcomes. Regardless of the contracting method,  focusing on outcomes and collaboration among multiple stakeholders in  the contracting process has been acknowledged as sound contract  management. In 2000, federal procurement law established a performance- based approach as the preferred acquisition method for services. The  Federal Acquisition Regulation requires all performance-based service  acquisitions to include    a performance work statement that describes outcome-oriented  requirements in terms of results required rather than the methods of  performance of the work;    measurable performance standards describing how to measure contractor  performance in terms of quality, timeliness, and quantity; and  the method of assessing contract performance against performance  standards, commonly accomplished through the use of a quality assurance  surveillance plan.
A 1998 Office of Federal Procurement Policy (OFPP) study on  performance-based contracts—based largely on contracts for basic  services, such as janitorial or maintenance services—showed that a  number of anticipated benefits had been achieved, including reduced  acquisition costs, increased competition for contracts, and improved  contractor performance. However, implementing a performance-based  approach is often more difficult for complex acquisitions, such as  information technology, than it is for basic services, because agencies  begin with requirements that are less stable, making it difficult to establish  measurable outcomes. Such complex acquisitions may need to have  requirements and performance standards continually refined throughout  the life-cycle of the acquisition for a contractor to deliver a valuable  service over an extended period of time. OFPP also has noted in policy  that certain types of services, such as research and development, may not  lend themselves to outcome-oriented requirements.
To encourage agencies to apply a performance-based approach to service  acquisitions, the Office of Management and Budget (OMB) established  governmentwide performance targets, which increased to 50 percent of  eligible service contract dollars for the current fiscal year. In January 2007,  the congressionally mandated Acquisition Advisory Panel reported that  performance-based acquisition has not been fully implemented in the  federal government, despite OMB encouragement, and recommended that  OMB adjust the governmentwide target to reflect individual agency  assessments and plans. In May 2007, OMB’s OFPP issued a memo  providing that agencies, at a minimum, were expected to meet targets  established and report on them in their management plans. In response,  DHS’s CPO established a performance-based target of 25 percent for fiscal  year 2007, increasing to 40 percent by fiscal year 2010, that was included in  DHS’s Performance-Based Management Plan. The Acquisition Advisory  Panel also recommended that OFPP issue more explicit implementation  guidance and create an “Opportunity Assessment” tool to help agencies  identify when they should consider using this acquisition method.
section_title: Requirements and Standards Definition Influence Program Outcomes
paragraphs: Our work has found that performance-based acquisitions must be  appropriately planned and structured to minimize the risk of the  government receiving services that are over cost estimates, delivered late,  and of unacceptable quality. Specifically, we have emphasized the  importance of clearly defined requirements to achieving desired results  and measurable performance standards to ensuring control and  accountability. Prior GAO and DHS Inspector General reviews of complex  DHS investments using a performance-based approach point to a number  of shortcomings. For example, in June 2007, we reported that a  performance-based contract for a DHS financial management system,  eMerge2, lacked clear and complete requirements, which led to schedule  delays and unacceptable contractor performance. Ultimately, the program  was terminated after a $52 million investment. In March 2007, we similarly  reported that the Coast Guard’s performance-based contract for replacing  or modernizing its fleet of vessels and aircraft, Deepwater, had  requirements that were set at unrealistic levels and were frequently  changed. This resulted in cost escalation, schedule delays, and reduced  contractor accountability. The DHS Inspector General has also indicated  numerous opportunities for DHS to make better use of sound practices,  such as well-defined requirements.
Consistent with our prior work, definition of requirements and  performance standards influenced outcomes for the eight complex  investments we reviewed. In using a performance-based approach, sound  contracting practices dictate that required contract outcomes or  requirements be well-defined, providing clear descriptions of results to be  achieved. While all eight contracts for these investments had outcome- oriented requirements, the requirements were not always well-defined.
Further, contracts for half of the investments did not have a complete set  of measurable performance standards. Appendix I provides a summary of  our analysis of the requirements, performance standards, and outcomes  for the eight performance-based contracts for major investments we  reviewed.
Complex investments with contracts that did not have well-defined  requirements or complete measurable performance standards at the time  of contract award or start of work experienced either cost overruns,  schedule delays, or did not otherwise meet performance expectations. For  example, contracts for systems development for two CBP major  investments lacked both well-defined requirements and measurable  performance standards prior to the start of work and both experienced  poor outcomes. The first, for DHS’s Automated Commercial Environment  (ACE) Task Order 23 project—a trade software modernization effort—was  originally estimated to cost $52.7 million over a period of approximately 17  months. However, the program lacked stable requirements at contract  award and, therefore, could not establish measurable performance  standards and valid cost or schedule baselines for assessing contractor  performance. Software requirements were added after contract award,  contributing to a project cost increase of approximately $21.1 million, or  40 percent, over the original estimate. Because some portions of the work  were delayed to better define requirements, the project is not expected to  be completed until June 2009—about 26 months later than planned.
The second, Project 28 for systems development for CBP’s Secure Border  Initiative (SBInet)—a project to help secure a section of the United States- Mexico border using a surveillance system—did not meet expected  outcomes due to a lack of both well-defined requirements and measurable  performance standards. CBP awarded the Project 28 contract planned as  SBInet’s proof of concept and the first increment of the fielded SBInet  system before the overall SBInet operational requirements and system  specifications were finalized. More than 3 months after Project 28 was  awarded, DHS’s Inspector General reported that CBP had not properly  defined SBInet’s operational requirements and needed to do so quickly to  avoid rework of the contractor’s systems engineering. We found that  several performance standards were not clearly defined to isolate the  contractor’s performance from that of CBP employees, making it difficult  to determine whether any problems were due to the contractor’s system  design, CBP employees, or both. As a result, it was not clear how CBP  intended to measure compliance with the Project 28 standard for  probability of detecting persons attempting to illegally cross the border.  Although it did not fully meet user needs and its design will not be used as  a basis for future SBInet development, DHS fully accepted the project after  an 8-month delay. In addition, DHS officials have stated that much of the  Project 28 system will be replaced by new equipment and software.
Conversely, we found that contracts with well-defined requirements linked  to measurable performance standards delivered results within budget and  provided quality service. For example, contracted security services at the  San Francisco International Airport for TSA’s Screening Partnership  Program had well-defined requirements, and all measurable performance  standards corresponded to contract requirements—an improvement from  our prior reviews of the program. The requirements for gate, checkpoint,  and baggage screening services clearly stated that the contractor should  use technology and staff to prevent prohibited items from entering sterile  areas of the airport and should work to minimize customer complaints  while addressing in a timely manner any complaints received. The  performance standards assessed how often screeners could successfully  detect test images of prohibited items in checked baggage; the percentage  of audited records and inspected equipment, property, and materials that  were well-kept, operational, and recorded on maintenance logs; and  whether all new hires received the required training before assuming their  screening responsibilities. In terms of expected outcomes, the contractor  achieved a 2.2 percent cost underrun during the first 5 months of the  contract and exceeded most requirements.
section_title: Unreliable Data and Lack of Management Review Constrain Oversight
paragraphs: In managing its service acquisitions, including those that are performance- based, DHS has faced oversight challenges, including a lack of reliable  data and systematic management reviews. DHS contracting and program  representatives told us that they use a performance-based approach to the  maximum extent practicable. However, DHS does not have reliable data— either from the Federal Procurement Data System-Next Generation  (FPDS-NG), the governmentwide database for procurement spending, or  at a departmentwide level—to systematically monitor or evaluate or report  on service acquisitions, including those that are performance-based.  Reliable data are essential to overseeing and assessing the implementation  of contracting approaches, acquisition outcomes, and making informed  management decisions. Moreover, the Chief Procurement Officer (CPO),  who has responsibility for departmentwide procurement oversight, has  begun some initial review of performance-based service acquisitions, but  has not conducted systematic management assessments of this acquisition  method.
Our analysis of information provided by contracting representatives at the  Coast Guard, CBP, Immigration and Customs Enforcement (ICE), and TSA  showed that about 51 percent of the 138 contracts we identified in FPDS- NG as performance-based had none of the required performance-based  elements: a performance work statement, measurable performance  standards, and a method of assessing contractor performance against  performance standards. Only 42 of the 138 contracts, or 30 percent, had all  of the elements, and about 18 percent had some but not all of the required  performance-based acquisition elements (see table 1).
Lacking reliable FPDS-NG data, reports on the use of performance-based  contracts for eligible service obligations are likely inaccurate. Data  reported on the use of performance-based contracts by service types— ranging from basic, such as janitorial and landscaping, to complex, such as  information technology or systems development—requested by OFPP in  July 2006—are also likely misleading. The Acquisition Advisory Panel and  DHS’s CPO also have raised concerns regarding the accuracy of the  performance-based designation in FPDS-NG. The Acquisition Advisory  Panel’s 2007 report noted from its review at 10 federal agencies that 42  percent of the performance-based contracts the panel reviewed had been  incorrectly coded.
Inaccurate federal procurement data is a long-standing governmentwide  concern. Our prior work and the work of the General Services  Administration’s Inspector General have noted issues with the accuracy  and completeness of FPDS and FPDS-NG data. OMB has stressed the  importance of submitting timely and accurate procurement data to FPDS- NG and issued memos on this topic in August 2004 and March 2007.  Accurate FPDS-NG data could facilitate the CPO’s departmentwide  oversight of service acquisitions, including those that are performance- based.
At a departmentwide level, CPO representatives responsible for  procurement oversight indicated that they have not conducted systematic  assessments including costs, benefits, and other outcomes of a  performance-based approach. To improve the implementation of  performance-based acquisitions, CPO representatives established a work  group in May 2006 to leverage knowledge among DHS components. They  also noted that they are working with OFPP to develop a best practices  guide on measurable performance standards and to gather good examples  of performance-based contracts. In addition, the CPO has implemented a  departmentwide acquisition oversight program, which was designed with  the flexibility to address specific procurement issues, such as  performance-based service acquisitions, and is based on a series of  component-level reviews. Some initial review of performance-based  acquisitions has begun under this program, but management assessment  or evaluation of the outcomes of this acquisition method has not been  conducted.
section_title: Conclusion and Recommendations
paragraphs: Consistent with federal procurement policy, DHS has emphasized a  performance-based approach to improve service acquisition outcomes.  However, in keeping with our prior findings, DHS’s designation of a  service acquisition as performance-based was not as relevant as the  underlying contract conditions. Sound acquisition practices, such as  clearly defining requirements and establishing complementary measurable  performance standards, are hallmarks of successful service acquisitions.  In the cases we reviewed as well as in prior findings where these key  elements were lacking, DHS did not always achieve successful acquisition  outcomes. The report we are releasing today recommends that the  Secretary of Homeland Security take several actions to increase DHS’s  ability to achieve improved outcomes for its service acquisitions, including  those that are performance-based. These actions include routinely  assessing requirements for complex investments to ensure that they are  well-defined and developing consistently measurable standards linked to  those requirements; systematically evaluating outcomes of major  investments and relevant contracting methods; and improving the quality  of FPDS-NG data to facilitate identifying and assessing the use of various  contracting methods. DHS generally concurred with our  recommendations, noting some departmental initiatives under way to  improve acquisition management. However, the department’s response did  not address how the CPO’s process and organizational changes at the  departmental level will impact component-level management and  assessment of complex acquisitions to improve outcomes. Improving  acquisition management has been an ongoing challenge since the  department was established and requires sustained management attention.
Mr. Chairman, this concludes my prepared statement. I would be pleased  to respond to any questions that you or other members of the committee  may have at this time.
section_title: Contacts and Acknowledgments
paragraphs: For further information about this statement, please contact John P.  Hutton at (202) 512-4841 or huttonj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this statement. Staff making key contributions to this  statement were Amelia Shachoy, Assistant Director; Jeffrey Hartnett; Sean  Seales; Karen Sloan; and Don Springman.
section_title: Appendix I: Key Characteristics of Eight Performance-Based Service Contracts
subsections: section_title: Research, analysis, and financial and information management
paragraphs: Contractor submitted all required documentation on  time; met project management quality standards;  and maintained electronic archiving and restoration  standards.
Trade systems  software  development (task  order 23)
Costs increased by 40 percent ($21.1 million).  More than a year behind schedule; unplanned  software redesign.
Costs increased by 53 percent ($24 million).  Maintenance wait times were longer than planned.
DHS rejected initial acceptance of Project 28. The  project was delayed 8 months with final acceptance  in February 2008. DHS noted that the contractor  met the requirements, but the project did not fully  meet DHS’s needs and the technology will not be  replicated in future SBInet development.
Contractor exceeded the performance standard for  machine downtime with a score 1 hour less than  required and operated at cost through the second  quarter of fiscal year 2007.
Contractor exceeded most performance standards;  for example, threat detection performance and  false alarm rates exceeded the quality standards.  Contractor had cost underrun of 2.2 percent  ($677,000).
Initial contractor planning reports were inadequate;  system experienced operational downtime;  surveillance reports identified poor contractor  performance. Contractor generally met time frames  and delivered within budget.
Outcomes not available at the time of our review.
Legend:  contract met or mostly met the criteria;  contract partially met the criteria;  contract did  not meet the criteria.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.