Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results
Released: 2002-09-13T00:00:00

Letter

Background

DOE’s missions include developing, maintaining, and securing the nation’s nuclear weapons capability; cleaning up the environmental legacy resulting from over 50 years of producing nuclear weapons; and conducting basic energy and science research and development. The department carries out these diverse missions at over 50 major installations in 35 states. With a DOE workforce of about 16,000 employees and over 100,000 contractor staff, the department relies on its contractors to manage and operate its facilities and accomplish its missions. DOE manages these functions through its program offices at DOE headquarters and its field offices. The three largest program offices— Environmental Management, Defense Programs, and Science—accounted for over 70 percent of DOE’s budget for fiscal year 2001.

DOE’s reliance on contractors to carry out its missions and the department’s history of both inadequate management and oversight and failure to hold its contractors accountable for results led us to designate DOE contract management as a high-risk area vulnerable to fraud, waste, abuse, and mismanagement. In response to these and other criticisms, DOE began evaluating its contracting practices and, in February 1994, issued a report—Making Contracting Work Better and Cost Less—that contained 48 recommendations. The recommendations included three key areas: selecting alternatives to traditional contracting arrangements used for management and operation of its sites, increasing competition to improve performance, and developing and using performance-based contracting tools. To facilitate and oversee the implementation of the contract reform recommendations, in June 1994, DOE established the Contract Reform Project Office, which became the Office of Contract Reform and Privatization in 1997. This office, which monitored and assessed the progress of DOE’s contract reform initiative, was disbanded in late 2001 as part of the department’s reorganization of its support offices. DOE’s Office of Management, Budget, and Evaluation/Chief Financial Officer is now responsible for oversight of DOE’s contract reform efforts.

DOE Has Made Progress in Implementing Contract Reforms, but Challenges Remain

Since 1996, the department has made progress in implementing three key contract reform initiatives—developing alternative contracting approaches, increasing competition, and converting to performance-based contracts, although DOE continues to address challenges in implementing these initiatives. Concerning alternative contracting approaches, DOE encouraged the use of different types of contracts aimed at improving contractor performance and results. However, DOE did not use a systematic approach to determine the best contract type for a given situation and experienced problems with implementation. To become more systematic in making this contract selection decision, DOE has been developing a formal strategy to evaluate contract and financing alternatives and the risks associated with various approaches. In the second reform area—increasing competition—DOE changed its contracting rules to set competition as the standard approach to awarding contracts. Under these rules, the percentage of major site contracts awarded competitively (competed) increased to 56 percent as of 2001, up from 38 percent as of 1996. All but one of the 11 contracts that had not been competed were for managing research and development centers exempted by statute from mandatory competition. The department evaluates these contracts to determine whether they should be extended or competed. DOE has thus far decided on non-competitive extensions for these contracts, including some for contractors that have experienced performance problems. DOE opted to address these performance problems with specific contract provisions, but it remains to be seen whether this approach will succeed. Finally, all of DOE’s major site contracts are now performance-based, incorporating results-oriented statements of work and the performance objectives and measures used to evaluate contractor performance. To further emphasize the importance of the performance-based approach, DOE has increased the proportion of contractor fees tied to achieving the performance objectives to 70 percent in fiscal year 2001 from 34 percent in fiscal year 1996. However, development of good performance measures has continued to be a challenge, and DOE acknowledges that it must make further progress in this area.

DOE Implemented Alternative Contract Approaches, but Did Not Establish an Acquisition Strategy

One of the major focuses of DOE’s contract reform initiative has been developing alternatives to the traditional contracts used for the management and operation of its major sites and facilities. Under these “management and operating” contracts, one primary contractor performed almost all of the work at a site, the contractor had broadly defined statements of work, and DOE reimbursed the contractor for virtually all costs. As a result, work under these contracts focused more on annual work plans and budgets rather than on specific schedule and cost targets for accomplishing work.

In implementing alternatives to its traditional contracting arrangements, DOE’s intent was to use the best contracting alternative given the required work and the objectives and risks associated with that work. DOE implemented four main actions as alternatives to these management and operating contracts, but has experienced problems with implementation, in part due to difficulties in determining the most appropriate approach for a given situation, as follows: Reducing the number of large, cost-reimbursement contracts that cover virtually all of the activities at a DOE site. DOE has modified a total of 20 site contracts since 1994, so that no single contractor manages and operates those sites. Some of these management and operating contracts were divided into smaller service contracts, such as for guard services. Other management and operating contracts were changed to integration contracts (commonly called management and integration contracts). According to DOE officials, integration contracts were used to better reflect the changing mission of the site and to better tailor the contract scope to the program requirements. Under a management and integration contract, one contractor is responsible for integrating the work of a variety of subcontractors that carry out most of the actual work at the sites. The integrating contractor is responsible for selecting “best-in-class” subcontractors for specific work activities, overseeing the work done by the subcontractors, and ensuring that activities at the site are effectively coordinated. DOE has used this integration contract approach at sites such as Oak Ridge in Tennessee for environmental restoration work. However, DOE’s Office of Inspector General reported in March 2001 that the integrating contractor at Oak Ridge has subcontracted out a third less work than originally proposed, resulting in less cost savings to the government.

Implementing a more disciplined approach to “make-or-buy” decisions by site contractors. DOE revised its regulations in 1997 to require that its major site contractors develop make-or-buy plans instead of having most of the work at a site performed by the primary contractor. Under these plans, the primary contractor must identify work functions that could be performed at less cost or more effectively through subcontracts. Although all of its major contractors have approved make-or-buy plans, DOE acknowledges that it does not routinely gather information on how much work is done by subcontractors, making it difficult to determine the extent to which this approach was implemented. In addition, DOE’s Office of Inspector General reported in February 2000 that three of the four contractors that it reviewed had either not included all functions in their make-or-buy plans or had not done the required cost-benefit analysis on work functions that could have been subcontracted.

Implementing an alternative contracting and financing approach called privatization. DOE started its “privatization initiative” in 1995 as a way to reduce the cost and speed the cleanup of its contaminated sites. This initiative was primarily an alternative contracting and financing strategy to foster open competition for fixed-price contracts; to require the contractor to design, finance, build, own, and operate the facilities necessary to meet waste treatment requirements; and to pay the contractor for units of successfully treated waste. DOE’s experiences with this approach showed that privatization could achieve cost savings on projects with a well-defined scope of work and few uncertainties, such as laundry facilities for contaminated uniforms and other items at the Hanford site. However, on complex cleanup projects such as the effort at Idaho Falls to clean up Pit 9, privatization had little success in achieving cost savings, keeping the project moving forward on schedule, or getting improved contractor performance.

Establishing “closure contracts” that tie performance incentives to contract completion, not to annual activities. DOE has used closure contracts at several sites that are scheduled for cleanup and closure, including the Rocky Flats site in Colorado and the Fernald site in Ohio. These contracts emphasize completing all work at a site or a portion of a site by a target date and at a target cost. Most of the fee or profit to be earned by the contractor depends upon meeting the schedule and cost targets. If the contractor can complete all work on time or sooner and below the target cost, then the contractor can earn additional fee. For example, under the Rocky Flats closure contract, the amount of incentive fee that the contractor can earn ranges from $130 million to $460 million, depending on cost and schedule performance against the targets. Since the target closure date for this contract is December 2006, it remains to be seen whether this approach will be effective in completing the work on time and at lower costs to the government.

These problems reflected the lack of a systematic approach to deciding which contract type was best for a given situation. For example, we reported in May of 1998 that DOE’s use of fixed-price contracting was appropriate when projects were well-defined, when uncertainties could be allocated between DOE and the contractor, and when either adequate cost information or multiple competing bidders were available to determine a fair and reasonable price for the work. However, when these conditions did not exist, cost overruns and schedule delays could occur on these fixed-price contracts.

DOE has begun to develop a more systematic approach to determining the best contract type for a given situation. For example, in October 2000, DOE issued new policy and guidance for the acquisition of capital assets such as waste treatment facilities. The guidance includes developing an acquisition plan that considers the financial, technical, and performance risks associated with a new project. This policy is consistent with DOE’s overall goal of tailoring the contract type to the work to be performed and the business and technical risks associated with that work. In addition, to strengthen oversight of major acquisitions, in November 2001 DOE issued additional guidance that requires approval of acquisition plans for projects of $5 million and above at the assistant secretary level or higher. Despite these initial steps, DOE is still developing and implementing its formal acquisition strategy, and it is too soon to tell whether this new strategy will help DOE make better decisions about how to acquire capital assets.

Use of Competitively Awarded Contracts Increased, and Allowable Exceptions Continue

DOE has increased the proportion of major site contracts awarded competitively, but still extends a number of these site contracts non- competitively, as allowed by procurement law, including contracts for some sites that have experienced contractor performance problems. DOE competed 56 percent of its major site contracts that were up for award or renewal from 1997 through 2001, a significant increase over the 38 percent it had competed from 1991 through 1996 (see table 1).

During the 1997 through 2001 period, DOE selected new contractors for 10 of the 14 competitively awarded contracts, compared to 9 new contractors for the 11 competitive awards from 1991 through 1996. (Appendix I contains a listing of DOE’s major site contracts in 2001 and the extent to which they have been competed). The growth in competition at major DOE sites is largely a result of new regulations the department issued under contract reform. The new rules generally require competition for major site contracts and allow a contract period consisting of an initial term of up to 5 years with options to extend the contract provided that the total contract period does not exceed 10 years.

Many of the contracts that DOE did not compete have been for its federally funded research and development centers for which DOE may extend contracts non-competitively under the Competition in Contracting Act of 1984. By 2001, all but one of the 11 contracts extended without competition fell under this exemption for research and development centers. The exception was the major site contract for the management of DOE’s West Valley Demonstration Project in New York. DOE extended the contract in 1998 and recently announced plans for another extension. According to DOE procurement officials, this recent extension was because of the limited amount of cleanup work remaining at the site and the lack of interest by other contractors to compete for the work.

As part of its overall effort to increase competition for site contracts, DOE also reassessed which sites it should continue to designate as federally funded research and development centers. As a result of the reassessment, DOE has removed six of 22 sites from the federally funded research and development center designation. The department subsequently competed the contracts for two of these, the Knolls and Bettis Atomic Power Laboratories in New York and Pennsylvania. The department restructured the other four contracts and no longer regards them as major site contracts. In six other instances, although DOE has thus far decided the sites should remain designated as federally funded research and development centers, the department has competed the contracts even though federal law and regulations allow DOE to extend the contracts non-competitively. These six competed contracts included those for the Oak Ridge National Laboratory in Tennessee and the Idaho National Engineering and Environmental Laboratory.

In addition to its reassessment effort, in 1996 the department issued guidance that it must follow to support any recommendation for a non- competitive extension of any major site contract. Among other things, the guidance called for DOE to provide a certification that competition is not in the best interest of the department, a description of the incumbent contractor’s past performance, an outline of the principal issues and/or significant changes to be negotiated in the contract extension, and in the case of a federally funded research and development center, a showing of the continued need for the research and development center.

Based on such documentation, the agency head can authorize a contract extension of up to 5 years. Table 2 lists the ten federally funded research and development centers for which DOE has awarded contracts non- competitively since this guidance was issued.

DOE’s decision not to compete some of the federally funded research and development center contracts has not been without controversy. For example, in 2001, DOE extended the management and operating contracts with the University of California for the Los Alamos and Lawrence Livermore National Laboratories. The University of California has operated these sites for 50 years or more and is the only contractor ever to have operated them. In recent years, we and other organizations have documented significant problems with laboratory operations and management at these two laboratories—particularly in the areas of safeguards, security, and project management. Congressional committees and others have called for DOE to compete these contracts. Even with these problems and concerns, however, DOE chose not to compete these contracts. This decision was made at the highest levels in the department and was based on national security considerations. Rather than compete these contracts, DOE intends to address these performance problems using contract mechanisms.

In the 2001 contract extension, DOE required the university to focus on strengthening management performance in five areas, including initiatives for safety and project management. For the first 2 years of the 5-year contract period, the University of California must meet specific requirements before it can earn any of the $17 million in incentive fees available under the contract. DOE is to assess the university’s performance on these specific requirements on a pass/fail basis. After the first 2 years of the contract, performance in these 5 areas will be assessed as part of the regular performance measures in the contract. The department’s first (2001) annual assessment found that the contractor was meeting the required milestones for all of the improvement initiatives. However, many of the milestones in the first year involved evaluating existing systems or developing action plans. For other objectives that focus on results, such as demonstrating improved performance in nuclear facility operations, the final outcomes will not be known for several years. Therefore, it remains to be seen whether DOE will be successful in improving the University of California’s performance using these contracting tools. If the University of California does not make significant improvements in its performance, DOE may need to reconsider its decision not to compete the contracts.

Contracts Are Performance-Based, but DOE Has Difficulty Developing Effective Performance Objectives and Measures

DOE has reported that all of its major site contracts incorporate performance-based techniques to define requirements and measure results. Before DOE initiated its contract reforms, major site contracts generally had broad statements of work that focused more on annual budgets and work plans rather than specific results to be achieved. Feesunder these contracts usually consisted of a base fee that was guaranteed (fixed) plus an award fee that was paid if the contractor met general performance expectations. In the mid-1990s, DOE began restructuring its major site contracts to use results-oriented statements of work and, for most of the major site contracts, to incorporate performance incentive fees that were designed to reward the contractor if it met or exceeded specific performance expectations in priority areas. These fees may be tied to either subjective or objective performance measures, but DOE regulations suggest the use of specific and quantifiable measures whenever possible. In 1999, DOE issued additional regulations that limited the use of base fee and established a clear preference for contracts where all of the fee was based on a contractor’s performance.

Since DOE changed its policy in favor of using incentive fees, there has been a substantial shift in the type of fees available on DOE contracts. As shown in figure 1, between fiscal years 1996 and 2001, DOE decreased the total aggregate amount of base and award fee available to its contractors and substantially increased the amount of fee that is based on performance incentives. For individual contracts, the percentage of each fee type varied widely. For example, in fiscal year 2001, the Sandia National Laboratories contract had 100 percent base fee, and the Oak Ridge National Laboratory contract had 100 percent performance incentive fee.

In addition to shifting most of the fee available to incentive fee, in 1999, DOE also established a new contract clause making payment of fee conditional on meeting certain safety requirements and other minimum requirements in the contract. According to language in this clause, in order to receive all of the earned fee, the contractor must meet, among other requirements, minimum environment, safety, and health requirements and avoid any “catastrophic” events such as a fatality or serious workplace- related injury. Since 1999, DOE has withheld over $5 million in fees from six contractors under this conditional payment of fee clause. The largest fee withheld—$2 million—was from CH2M Hill Hanford Group, Inc., for “failures to meet the contractually imposed minimum environment, safety, and health performance requirements” as defined by the contractor’s integrated safety management system.

Although these changes reflect a marked shift in DOE’s approach, the lack of good performance measures blunted their effect. Since 1997, numerous studies and reports—both internal and external to the department— criticized DOE’s performance-based contracts for ineffective performance measures. Examples include the following: DOE’s Office of Inspector General has issued 11 reports since 1997 that found multiple problems with DOE’s performance measures. In 2001, the Inspector General reported, after reviewing the Office of River Protection Tank Farm Management, Oak Ridge Y-12 Plant, and Kansas City Plant contracts, that DOE was not focusing on high priority outcomes, was loosening performance requirements over time without adequate justification, and was failing to match appropriately challenging contract requirements with fee amounts. The department disagreed with this report, stating that it was not appropriate to evaluate the overall success of performance-based contracts by looking at individual performance measures.

In 1999, reporting on a self-assessment of its performance-based contracting practices, DOE concluded that while significant improvements had been made in the management of performance- based contracts, several issues had arisen. These issues included difficulties with measuring the results of basic science activities, establishing performance measures that were consistent with project baselines, determining the appropriate use of incentive fees for non- profit contractors, and balancing incentives that both challenge the contractor and continue to reward performance that has been sustained at an excellent level.

In its 1999 review of project management at DOE, the National Research Council found that DOE did not always take advantage of the performance-based incentive approach and did not have standard methods for measuring project performance. The council’s 2001 follow- up assessment stressed the importance of using methods such as performance-based contracting to focus contractors on achieving desired results. The council added that success would be determined by how well these methods are followed and recommended that DOE strengthen its performance-based contracting guidance and practices.

In response to these and other criticisms of its performance-based incentives, DOE has taken several actions that include issuing criteria for a performance incentive development process at the field office level and focusing on developing performance incentives more directly linked to a site’s strategic objectives. For example, DOE officials said that multi-year incentives in the Hanford contract and multi-site incentives that tie together activities at four production sites—Kansas City in Missouri, Savannah River in South Carolina, Pantex in Texas, and Y-12 at Oak Ridge, Tennessee—strive to establish the strategic focus that was absent from performance incentives in earlier contracts. DOE officials pointed out that, with these new incentives, greater progress was being made. For example, the Hanford site had reached its cleanup goals for fiscal year 2001. However, it remains to be seen if contractors will meet milestones throughout the contracts’ full length and, if they do not, if DOE will require contractors to forfeit the provisional fee payments as allowed under the contracts.

Contract Reforms May Not Have Improved Contractor Performance

Although DOE has made strides in implementing its contract reform initiatives and has reviewed the performance measures in many of its contracts, the department has developed little objective information to demonstrate whether the reforms have resulted in improved contractor performance. In the early years of contract reform, DOE measured progress in terms of developing and issuing new contracting policies and guidance. As new policies were established, the department also focused on assessing its progress in implementing these policies in key areas of competition and performance-based contracting. More recently, DOE has reviewed many of its site contracts to determine, among other things, whether the performance incentives are working properly. While these steps are useful, this information does not help DOE determine outcomes—whether, for example, competing more contracts resulted in more favorable contract terms for the government or better performance from its contractors. DOE program managers and procurement officials at DOE headquarters and several sites believe that contract reforms have resulted in improved contractor performance, and they cite a number of examples where they believe contractor performance has improved. However, there are also numerous examples of contractors who performed poorly. Furthermore, DOE’s February 2002 review of its Environmental Management program observed that significant progress in cleanup and risk reduction had not been achieved despite the performance-based contracting approach. Since DOE does not have measures to determine whether the contract reform initiatives had resulted in improved performance, we examined the extent of cost overruns and schedule delays on a number of DOE’s major projects as a partial indicator of success. For these projects, cost and schedule data showed no improvement when compared to similar data in 1996. While this performance information provides only a limited view of department- wide contractor performance, it does raise questions regarding the overall effectiveness of the reform initiatives.

DOE’s Measurement Focused on Implementation, Not Outcomes

At the outset of contract reform, DOE established specific action steps and related time frames for changing its contracting practices. For example, DOE set a goal of developing guidance by August 1994 for increasing competition in awarding contracts. Subsequently, DOE proposed new regulations concerning contract reforms in the areas of competition, performance-based contracting and fee policies. As the department’s contract reform activities shifted from issuing guidance to restructuring actual contracts, officials began to monitor the extent to which its contracting organizations adopted DOE’s contracting policy changes in key reform areas. Because the contract cycle for the large site contracts was so long—typically contracts were renewed about every 5 years—DOE encouraged early incorporation of contract reform principles as each contract came up for renewal.

Over the 8 years since the contract reform initiative was introduced, DOE has primarily gauged its progress by monitoring implementation of the reforms and reviewing individual contracts rather than by developing objective measures to determine whether the reforms have resulted in improved contractor performance. In addition to tracking the number of contracts that incorporated the new requirements to use competition and performance-based features, the department reviewed the implementation of performance-based contracting for many of its major contracts. Some examples of DOE’s monitoring activities include: DOE’s annual performance reports required under the Government Performance and Results Act contained measures for both competing major site contracts and converting them to performance-based contracts. In 1999, DOE reported that it exceeded the goal of awarding at least 50 percent of the major site contracts using competitive procedures. In the reports for the years 1999, 2000, and 2001, DOE met its performance goals to convert all major site contracts awarded in each year to performance-based contracts.

DOE’s Office of Procurement and Assistance Management monitored the contracts awarded at major sites. For the years 1997 through 2000, the office reported that DOE met its annual goal of awarding contracts that were performance-based at all of the major sites.

DOE maintains a Web site that provides information on the status of its procurement goals. These goals include increasing the use of competition in awarding contracts and of performance-based concepts in those contracts. DOE’s Web site reports that as of 2001, 26 of its major site and facility contracts were competed and that 100 percent of these major contracts are performance-based.

In 1997, the department’s self-assessment of contract reform determined that progress had been made in implementing contract reforms across the complex. However, the report noted difficulties in identifying and quantifying contract reform data and recommended on- going analysis of key reform areas such as the effectiveness of fixed- price contracting.

In both 1997 and 1999, the department reported on its use of performance-based incentives in major site contracts. The department documented considerable progress in developing guidance and in incorporating performance-based incentives but also found that early incorporation of performance-based concepts had resulted in some poorly structured incentives. For example, performance incentives were sometimes overly focused on process milestones rather than outcomes. The 1997 report recommended issuing guidance on how to restructure performance objectives, but not on how to assess the effectiveness of the restructured incentives. The 1999 report concluded that the quality of contractor performance incentives had improved and that the performance incentives were incorporated into contracts in a more timely manner. The report further stated that the best measure of the effectiveness of the incentives was improvement in contractor performance. The report discussed specific contracts but did not present overall data on contractor performance.

Procurement and program officials in headquarters continue to be actively involved in developing and reviewing performance measures in major site contracts. DOE officials said this oversight is improving the quality of performance incentives and providing valuable information on lessons learned. They acknowledged, however, that DOE has not developed objective information on the outcomes associated with the reforms. Such results-oriented information is important to determine the extent to which the contract changes have resulted in improved contractor performance.

Anecdotal Evidence Provides No Overall Measure of Improved Performance

Although objective performance information focusing on results is not available, DOE program managers and procurement officials at both DOE headquarters and field operations offices believe that contract reforms have made a difference. In support of this view, DOE officials generally provide examples that they believe demonstrate improved contractor performance. For example, officials at DOE’s Albuquerque operations office pointed out that after competing the contract for the Pantex site, the new contractor met required production levels that were not achieved by the previous contractor. These officials also mentioned that the poor performance by the previous contractor was one of the deciding factors in competing the contract for the Pantex site.

In addition to the examples of improved performance provided by DOE officials, DOE’s 1999 review of its performance-based contracting practices reported that “anecdotal evidence supports that the proper use of well-structured performance-based incentives is leading to improvements in performance at some DOE sites.” One of the examples cited in this internal review was improved performance at the Rocky Flats site under a performance-based contract established in 1995. Under the previous contract with a broad statement of work, the contractor was primarily safeguarding and maintaining facilities at the site, and no buildings had been decontaminated, demolished, and removed. When DOE competed the contract in 1995 and selected a new contractor, DOE also incorporated performance measures into the contract. Consistent with these measures, the new contractor decontaminated, demolished, and removed six buildings during fiscal year 1996 and 12 during fiscal year 1998.

Other examples demonstrate, however, that the instances DOE cites are not necessarily representative of the overall performance of DOE’s contractors. Examples of poor performance by DOE’s contractors include the following: DOE has experienced major cost overruns and schedule delays on the National Ignition Facility at Lawrence Livermore National Laboratory in California. This facility, the size of a football stadium, is designed to produce intense pressures and temperatures to simulate in a laboratory the thermonuclear conditions created in nuclear explosions. DOE considers the facility to be an essential component of the program to ensure the safety and reliability of the nuclear weapons stockpile in the absence of nuclear testing. Although DOE had incorporated performance-based measures and incentives into the overall contract with the University of California, which operates the laboratory and manages the construction project, performance problems still occurred. We reported in August 2000 that the estimated cost of this facility had increased from $2.1 billion to $3.3 billion and that the scheduled completion date had been extended by 6 years to 2008. We attributed these major cost and schedule changes to inadequate management by the contractor and DOE oversight failures. We also found that the performance-based contract placed little emphasis on the National Ignition Facility project even though it dominated the laboratory’s budget and mission. DOE withheld $2 million of the fiscal year 1999 performance fee in recognition of the “significant mission disruption” caused by problems with this project. DOE officials said that the department has since modified the performance-based contract to increase the emphasis on this project and has taken additional steps to improve both contractor management and DOE oversight.

DOE has had problems with cost and schedule performance on its contract for the Mound site in Ohio. In August 1997, DOE awarded a cost-plus-award fee performance-based contract for the accelerated cleanup of the Mound site. This contract called for cleaning up the site and transferring facilities to the local community by no later than September 2005 at a total estimated cost of $427 million. In May 2001, DOE’s Office of Inspector General reported that the department and the contractor had committed to that schedule without knowing whether the date was achievable and that the cost and schedule had been established with limited knowledge of the soil and building contamination. The report added that completion of this work was estimated for December 2009 at a cost of over $1 billion.

DOE is becoming aware of the problems with relying heavily on anecdotal information when trying to assess outcomes. Officials in one of DOE’s largest program offices—Environmental Management, representing almost a third of the department’s overall budget—recently reported fundamental problems with their program, and with the department’s ability to manage for results. In a February 2002 review, the office stated that although the Environmental Management program had spent over $60 billion since 1989, little progress had been made toward cleaning up radioactive and hazardous wastes resulting from over 50 years of producing nuclear weapons, or toward reducing risks to the public and the environment.During fiscal years 2000 and 2001, however, most of the contractors at Environmental Management sites had earned more than 90 percent of their available performance incentive fee, indicating that the contractors were successfully achieving the performance goals established in their contracts. The Assistant Secretary for Environmental Management reported that if such “successes” can take place without significant progress in cleanup and risk reduction, the program has been using the wrong set of indicators to measure success. She added that Environmental Management program indicators “measured process, not progress, opinions, not results.” Among the conclusions in the report was that the Environmental Management program needed to significantly improve its management of performance-based contracts, focus on accomplishing measurable results, and align contractors’ performance fees with end points rather than intermediate milestones.

Status of Major DOE Projects Does Not Indicate Improved Performance

Based on our review of the performance of selected projects, it does not appear that DOE’s contractors have significantly improved their performance since 1996. Because we could not determine whether DOE’s contract reform initiatives had resulted in improved performance using the department’s measures, we reviewed DOE’s ongoing projects to assess whether they were experiencing cost overruns or schedule delays. We compared current ongoing DOE projects with estimated total costs exceeding $200 million with similar information we developed in 1996 on projects with estimated total costs exceeding $100 million.

In both 1996 and 2001, over half of the projects we reviewed had both schedule delays and cost increases. Furthermore, as shown in table 3, the proportion of projects experiencing cost increases of more than double the initial cost estimates or schedule delays of 5 years or more increased during the 6-year period. For example, the initial cost estimate in 1998 for the spent nuclear fuels dry storage project at Idaho Falls, Idaho, was $123.8 million with a completion date of 2001. Currently, the cost estimate for this project is $273 million with a completion date of 2006. Appendix II contains additional information on DOE’s ongoing major projects as of December 2001.

The projects we reviewed—with estimated costs ranging from $270 million to $8.4 billion—may not be representative of all DOE projects. Although this comparison provides only a limited measure of contractor performance, it does raise questions about the overall impact of DOE’s contract reform initiative on improving contractor performance.

DOE’s Approach to Contract Reform Was Not Consistent with Best Practices

The problems with DOE’s ability to track the results of contract reform reflect a broader need to develop an approach to managing its initiatives that is more consistent with best practices. As part of our review, we looked at best practices for managing improvement initiatives. We found that high-performing organizations use a systematic results-oriented management approach that includes defining goals for the initiative and gauging progress towards those goals. They also use information on results to continuously adjust the implementation of the initiative and sustain improvements. DOE’s approach to contract reform did not incorporate these best practices, and its emphasis on measuring progress in terms of implementation indicated a focus primarily on contract reform itself as a goal rather than improved performance. Furthermore, DOE faces the same fundamental challenge—lack of a results-oriented approach—in several other management improvement initiatives that, if successful, could enhance its contract reform efforts.

DOE Has Not Followed Best Practices in Implementing Its Contract Reform Initiative

DOE’s approach to implementing its contract reform initiatives has not followed best management practices. In our review of authoritative literature we found that leading organizations were able to sustain such management improvement initiatives by using a systematic, results- oriented approach that incorporated a rigorous measurement of progress. Such an approach typically included the following steps: (1) define clear goals for the initiative, (2) develop an implementation strategy that sets milestones and establishes responsibility, (3) establish results-oriented outcome measures to gauge progress toward the goals, and (4) use results- oriented data to evaluate the effectiveness of the initiative and make additional changes where warranted. While DOE followed an implementation strategy for its contract reform initiatives, it implemented those initiatives largely without clearly defining goals, gauging progress toward those goals with results-oriented measures, or using results- oriented data to evaluate the effectiveness of its reforms.

Although DOE had set general, overarching goals for its contract reform efforts, the department did not further define those goals. As stated in the 1994 report of the Contract Reform Team, the overall goal of contract reform was to make the department’s contracting process “…work better and cost less.” The secretary’s preface to the report presented the fundamental problem: “DOE is not adequately in control of its contractors. As a result, the contractors are not sufficiently accountable to the department, and we are not in a position to ensure prudent expenditure of taxpayer dollars in pursuit of our principle missions.” However, DOE did not further align those broad goals in relation to the specific contract reform efforts. For example, the department did not frame its contract reform initiatives to increase competition in terms of improved contractor accountability, better performance, or reduced costs. While increasing the number of competitively awarded contracts is a positive development, it does not by itself indicate that the department’s contracting processes work better or cost less.

DOE was effective at establishing an implementation strategy that set milestones and assigned responsibility for carrying it out. For example, DOE’s February 1994 report by its contract reform team contained 48 specific reform actions, each containing a required action, establishing a deadline, and assigning a specific DOE office with responsibility for developing the reform action. These reform actions, for the most part, involved developing policies, procedures, guidance, and plans to implement reforms such as competitive procurements and performance incentives. Our 1996 assessment of DOE progress toward implementing those goals found that DOE had completed 47 of 48 reform actions. Since that time, DOE has continued to set milestones and assign responsibility for its reform initiatives. For example, following an internal review in 1997, the department developed another series of actions to improve its implementation of reform initiatives pertaining to performance-based incentives. Those actions also had milestones for completion and assigned responsibility for carrying them out.

DOE did not establish results-oriented outcome measures for its contract reform initiatives. Instead, as discussed earlier, DOE generally focused on measuring the progress of implementing its reform initiatives and reviewing individual contracts, but did not develop ways to gauge progress towards its overarching reform goals of making contracting work better and cost less. A shortcoming of goals defined so generally is the lack of objective ways in which to measure progress in meeting those goals. Translating the general goal of “working better” into a more specific objective, such as having contractors complete a greater number of their projects on time and within budget, would have helped the department to identify ways it could measure results and, therefore, gauge progress towards the goals of contract reform.

Finally, DOE does not have the results-oriented data to evaluate the effectiveness of its contract reform initiatives. Because the department did not develop clear goals and results-oriented measures, it does not have the results-oriented data necessary to systematically review progress, take corrective action, and reinforce success. Although DOE has received feedback on its reform efforts from internal reviews such as self- assessment reports and external reports by the DOE Inspector General, GAO, and others, these outside reviews are not a substitute for a systematic feedback process.

Despite not following best practices for reform initiatives, DOE has taken steps to strengthen the management and oversight of its activities. For example, DOE has recently taken steps to integrate contract, project, and financial management functions under a single office—the Office of Management, Budget, and Evaluation/Chief Financial Officer. DOE officials believe that this action will improve the coordination, oversight, and control of these important activities.

Other Management Improvement Initiatives That Could Enhance Contract Reform Have Similar Problems

Although DOE’s contract reform initiative has focused on increasing competition and holding contractors more accountable for results, DOE recognizes that contract reform by itself is not enough to ensure that improved contractor performance actually occurs. DOE has begun several other initiatives that, if successfully implemented, could enhance its contract reform efforts. These initiatives include efforts to strengthen its management of projects, develop and use information systems for oversight and control, and improve the training and expertise of the DOE staff overseeing contractor activities. We conducted only a limited review of these initiatives and did not fully assess DOE’s implementation against all four steps in a “best practices” approach. Nevertheless, we identified instances where, as with the contract reform initiative, DOE’s management of the initiative fell short of best management practices in one or more areas. Table 4 below outlines these initiatives, how they could enhance the contract reform efforts, and the potential management weakness that could limit their effectiveness.

Although none of these initiatives have been fully implemented, their effectiveness may be limited by the same lack of a results-oriented approach to managing the initiative and sustaining improvement as does the department’s contract reform efforts.

Conclusions

Poor performance by DOE contractors and inadequate DOE management and oversight of those contractors led us to conclude in 1990 that DOE’s contracting practices were at high risk for fraud, waste, abuse, and mismanagement. Subsequently, DOE began its contract reform initiative to improve the performance and accountability of its contractors. Although DOE has undertaken a number of reforms over the years and has monitored its progress in implementing those reforms, it has no good measure of the results of the reforms. Aside from individual examples of good or poor performance on specific projects, DOE cannot tell, for example, if the contract reforms have resulted in better performance by its contractors or more favorable contract terms for the government. Limited evidence we developed suggests that contractors managing DOE’s major projects are performing no better in 2001 than on similar projects in 1996.

DOE faces a fundamental challenge to ensuring the effectiveness of its contract reform initiative—developing an approach to managing the initiative that is more consistent with the best practices of high-performing organizations. DOE’s practices in managing its contract reform initiative, as well as its other initiatives such as project management, that could also help to improve contractor performance, fall short of the best practices followed by high-performing organizations. Unless DOE strengthens the way in which it manages initiatives such as contract reform, DOE may not be able to fully realize the benefits of these initiatives and ensure that its programs are adequately protected from fraud, waste, abuse, and mismanagement.

Recommendation for Executive Action

To improve the effectiveness of DOE’s contract reform initiative, as well as other management improvement initiatives, we recommend that the department develop an approach to implementing its initiatives that incorporates best practices including the key elements of (1) clearly defined goals, (2) an implementation strategy that sets milestones and establishes responsibility, (3) results-oriented outcome measures, and (4) a mechanism that uses results-oriented data to evaluate the effectiveness of the department’s initiatives and to take corrective actions as needed.

Agency Comments

We provided a draft of this report to the Department of Energy for its review and comment. DOE’s Director, Office of Management, Budget, and Evaluation/Chief Financial Officer responded that DOE had three main concerns about our report but agreed with our recommendation that DOE develop an approach to its management improvement initiatives, such as contract reform, that is more consistent with the practices of high- performing organizations. DOE’s first concern was that the report characterizes contract reform as DOE’s fundamental management challenge but the report also discusses program and project management issues. DOE believes this creates the misperception that the procurement system can be used to address the myriad of issues facing the department. We believe that our report fairly and accurately describes the context of contract management in DOE. Our report identifies contract management as a major management challenge for DOE, and one that we have reported on for over 10 years. The report does not suggest that contract management is DOE’s primary or most fundamental management challenge. In fact, we have issued other reports such as our December 2001 report on DOE’s major mission, structure, and accountability problems that discuss more fundamental management issues. However, within the context of those more fundamental management challenges, DOE can and should strive to effectively manage its contracts. Our report does not imply that effective contract management will solve the other problems facing the department. In fact, the report discusses initiatives other than contract reform that are under way at DOE, including the project management initiative, because those initiatives could also have an impact on the results of the contract reform initiative.

DOE’s second concern was that our report concluded that its contract reform initiative was not managed in a systematic manner. DOE said that its 1994 contract reform initiative was managed systematically and included top management oversight, a matrixed implementation team, clearly defined goals and objectives, an implementation strategy, and identified outcomes. DOE also said it used internal assessments of the effectiveness of specific reform initiatives. Our analysis involved comparing DOE’s approach to contract reform with the best practices for managing improvement initiatives followed by high-performing organizations. That comparison showed that DOE’s approach to contract reform, and to several other management improvement initiatives, was not consistent with those best practices, particularly in the areas of defining measurable goals, establishing results-oriented outcome measures, or developing results-oriented data with which to measure the effectiveness of the initiatives. We revised our report to clarify this point. DOE also questioned how we could criticize its approach to contract reform when we had recommended in earlier reports that it pursue contract reform. Our report does not question the need for contract reform in DOE or the components of DOE’s reform initiative, such as increasing competition and the use of performance-based contracts. Rather, our report assesses what progress DOE has made in implementing the initiatives, whether the initiatives have resulted in improved contractor performance, and any challenges DOE faces in ensuring that its contract reform initiatives are effective.

DOE’s third concern was that the report identifies a limited number of projects to support a conclusion that DOE’s contract management system is in trouble. DOE believes the problems are more likely due to program and project management issues and the risks generally associated with unique, technically complex projects and DOE’s funding and political environment. We believe that our report fairly characterizes DOE’s contract management system. Our report clearly states that DOE has developed little objective information to demonstrate whether its contract reforms have improved contractor performance. We pointed out that anecdotal examples can be used to illustrate both improved contractor performance and continued poor contractor performance. And we identify other evidence to suggest that contractor performance may not have improved. We also acknowledged that other factors, such as DOE’s approach to managing projects, could also affect the outcome of DOE’s contract reform efforts.

Regarding our recommendation that DOE develop an approach to implementing its management improvement initiatives that includes the key elements found in the best practices of high-performing organizations, DOE agreed with the recommendation and said that it would incorporate our observations and recommendation into its future improvement efforts.

DOE also provided technical corrections, which we incorporated as appropriate. DOE’s written comments on our draft report are included in appendix III.

We conducted our review from October 2001 through August 2002, in accordance with generally accepted government auditing standards. Appendix IV provides details on our scope and methodology.

This report contains a recommendation to you. As you know, 31 U.S.C. 720 requires the head of a federal agency to submit a written statement of the actions taken on our recommendations to the Senate Committee on Governmental Affairs and to the House Committee on Government Reform not later than 60 days from the date of this letter and to the House and Senate Committees on Appropriations with the agency’s first request for appropriations made more than 60 days after the date of this letter.

Appendix I: DOE’s Decisions to either Compete or Extend Non-Competitively Its Major Site Contracts

Contractor in 2001 Iowa State University University of Chicago Bechtel Bettis Inc. Brookhaven Science Associates Universities Research Association Fluor Fernald, Inc.

Bechtel Hanford Inc.

Fluor Hanford Inc. Bechtel BWXT Idaho, LLC Southeastern Universities Research Association Honeywell Federal Manufacturing and Technologies KAPL, Inc.

BWXT of Ohio Midwest Research Institute Bechtel Nevada Corp. Bechtel Jacobs Company, LLC BWXT Pantex, LLC Princeton University Sandia Corporation Westinghouse Savannah River Co.

Appendix II: Cost and Schedule Performance on DOE’s Major Projects as of December 2001

The following table shows the original and current cost estimates and completion dates for ongoing DOE projects with estimated costs greater than $200 million. The table does not include 10 additional DOE projects with estimated costs greater than $200 million because the projects were suspended or only recently started as of December 2001.

Appendix III: Comments from the Department of Energy

Appendix IV: Scope and Methodology

To assess the progress that DOE has made since 1996 in implementing contract reform initiatives in the key areas of developing alternative contracting approaches, increasing competition, and using performance- based contracts, we reviewed DOE’s three self-assessment reports on contract reform efforts and GAO and DOE Office of Inspector General reports on DOE contract and project management since 1996. We also interviewed officials from DOE’s Offices of Contract Management and Procurement and Assistance Policy, and procurement officials with the National Nuclear Security Administration. The National Nuclear Security Administration, a semi-autonomous agency within DOE, has its own procurement organization. However, since both entities follow the same policies, regulations, and guidance, we have not made a distinction in this report between contracts and projects of the two organizations. To assess the extent to which DOE had incorporated the key contract reforms into its major facility contracts, we obtained information on 33 contracts that DOE’s headquarters procurement office identified as site or facility management contracts. We reviewed the contract award history of these major facility contracts, to determine which contracts had been competed as of 1996 and as of 2001. To qualify as a competitively awarded contract, DOE must have issued a request for proposals and a public announcement inviting proposals. We also obtained data on annual budgets and fees available and earned for these same contractors for fiscal years 1996 through 2001. We did not attempt to validate this information provided by DOE. In addition, we reviewed documentation for major facility contracts obtained from DOE’s Albuquerque Operations Office, Richland Operations Office, and the Office of River Protection.

To determine the extent to which these initiatives have resulted in improved contractor performance, we interviewed DOE officials from the Office of Contract Management and the three largest program offices— Environmental Management, Defense Programs, and Science. In addition, we interviewed procurement and program office officials at DOE’s Albuquerque Operations Office, Richland Operations Office, and the Office of River Protection. We reviewed documents they provided, including the procurement organization’s balanced scorecard. In addition, we reviewed DOE’s February 2002 review of the Environmental Management program, and numerous GAO and Inspector General reports. Because DOE did not have objective results-oriented measures of contractor performance, as a potential indicator of that performance, we developed information as of December 2001 on the cost and schedule performance of DOE’s ongoing projects and compared that information with similar information we developed in 1996 on DOE major system acquisitions. In 1996, DOE categorized a “major system acquisition” as a project with a total project cost greater than $100 million. When we began our review in January 2002, we learned that DOE had since raised the threshold of “major project” to $400 million. Since our compilation of DOE reported data revealed only 19 ongoing projects that meet the current $400 million threshold (nine of which had recently started or were on hold), we expanded our scope to projects with total project costs greater than $200 million, in order to compare results on a similar number of projects. Those projects were under the management and oversight of DOE’s site contractors or under privatization projects under DOE’s oversight. There may be other projects with total project costs greater than $200 million, but they were not identified by DOE during our review. Because DOE does not maintain centralized data on its projects, we obtained information from project management offices within DOE and its National Nuclear Security Administration. We did not verify the data obtained from DOE, but we did examine the reasonableness of these data based on information in prior GAO reports and audits. For consistency, we used, when available, preliminary budget estimates submitted to the Congress as the basis for original cost estimates and completion dates, comparing those to current cost estimates and completions dates as of December 2001. For this report, we used, wherever possible, the projects’ “total project cost,” which includes construction and operating funds. Where these costs are not available, we used the “total estimated cost,” which includes construction costs. We have footnoted the latter. (See appendix II.)

To identify the challenges, if any, that DOE faces in ensuring the effectiveness of its contract reform initiatives, we reviewed the reports of the National Research Council on improving DOE project management. In addition, we reviewed reports and other documentation from the National Academy of Public Administration, the Project Management Institute, and prior GAO work to develop best practices criteria for managing improvement initiatives. We compared DOE’s implementation of its contract reform initiative to these best practices criteria to determine areas of concern. To identify the other management improvement initiatives that could impact contract reform, we reviewed the reports of the National Research Council, GAO and Inspector General; the President’s Management Agenda for fiscal year 2002; and DOE’s 5-year workforce restructuring plans. We also interviewed DOE officials in the Office of Engineering and Construction Management and the Office of Program Analysis and Evaluation.

We conducted our review from October 2001 through August 2002 in accordance with generally accepted government auditing standards.

Appendix V: GAO Staff and Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, Carole Blackwell, Robert Crystal, Doreen Feldman, Molly Laster, Patricia Rennie, Carol Shulman, Stan Stenersen, and Arvin Wu made key contributions to this report.