GAO-01-302

Report to the Administrator,

Environmental Protection Agency

United States General Accounting Office

GAO

March 2001 SUPERFUND

EPA’s Contract CostEstimating Initiatives

Show Promise and

Should Be Monitored
Page i GAO-01-302 EPA Contract Cost Estimating

Letter 1

Results in Brief 2

Background 3

EPA Has Launched Initiatives to Address Cost-Estimating

Weaknesses GAO and the Corps Identified 4

EPA Could Better Assess the Ultimate Effectiveness of Its

Initiatives 7

Conclusions 8

Recommendations for Executive Action 9

Agency Comments 9

Contents
Page 1 GAO-01-302 EPA Contract Cost Estimating

March 2, 2001

The Honorable Christine Todd Whitman

Administrator,

Environmental Protection Agency

Dear Ms. Whitman:

The Environmental Protection Agency (EPA), which manages the cleanup

of the nation’s most hazardous abandoned sites through the Superfund

program,1

 relies heavily on contractors to conduct its cleanup activities.

Currently, EPA spends about 50 percent of its approximately $1.5 billion

annual Superfund budget on contractors. With so much at stake, it is

critical that the government gets the best contract price for this cleanup

work.

EPA can best achieve this goal by developing an independent estimate of

what the work should cost and using this estimate to help negotiate the

price it will award the contractor for the work. However, in the early

1990s, we found that EPA was not generating independent estimates but

was generally adopting the contractors’ cost estimate as the price awarded

for the work.2

 Since then, we have reviewed EPA’s actions to address this

problem about every 2 years. In 1999, we reported that EPA had taken

steps to generate estimates for all of the contracted work we reviewed and

to use these estimates when negotiating the award price of each contract,

although the quality of some estimates remained a concern.3

 The regional

work assignment managers we contacted, who are generally responsible

for preparing the estimates, consistently said that additional training and

access to cost data on similar contracted work would help them better

develop independent estimates. We recommended that EPA take further

action to address these needs.

 1

The Superfund program was created under the Comprehensive Environmental Response,

Compensation, and Liability Act of 1980.

2

Superfund: EPA Has Not Corrected Long-Standing Contract Management Problems

(GAO/RCED-92-45, Oct. 24, 1991).

3

Superfund: Progress Made by EPA and Other Federal Agencies to Resolve Program

Management Issues (GAO/RCED-99-111, Apr. 29, 1999). We reviewed the 35 highest-dollarvalue work assignments in three EPA regions.

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-01-302 EPA Contract Cost Estimating

In response to our recommendations, in 1999, EPA hired the U.S. Army

Corps of Engineers—an agency with considerable cost-estimating and

contract management expertise—to conduct a comprehensive review of

EPA’s cost-estimating processes.4

 The Corps made a number of

recommendations to EPA in its review, including providing better training

and estimating tools, such as access to data on the actual costs of cleanup

work already contracted. The Corps also recommended that EPA

reinforce its commitment to controlling costs, consolidate its various cost

guidance documents, and better document the basis for any significant

variances in the original estimates, awarded prices, and actual costs

incurred.

In light of the Corps’ and our findings and recommendations, we initiated

work to determine (1) what initiatives EPA has taken in response to the

two reviews and (2) what means EPA will use to determine whether its

initiatives have improved the quality and usefulness of its estimates. To

respond to these objectives, we reviewed the Corps’ and our previous

documented findings and recommendations to EPA; discussed current

EPA initiatives to improve cost estimates with Superfund program

managers and EPA’s Office of Acquisition Management, which is

responsible for the agency’s contracting practices; reviewed

documentation on EPA’s initiatives; and discussed the initiatives with the

Corps’ officials who had reviewed EPA’s estimating processes and had

interviewed contracting officials from three EPA regions with large

cleanup workloads. We did not review any individual estimates because

these had been incorporated into the Corps’ comprehensive review. We

conducted our review from August 2000 through February 2001 in

accordance with generally accepted government auditing standards.

EPA has designed, but not fully implemented, two initiatives—a new

training course and new guidance—to address the Corps’ and our

concerns about its cost-estimating processes. EPA’s new training course

on cost estimating has been well received by the approximately 50

regional work assignment managers (or 10 percent) who have taken the

course in the 2 years since its inception. However, the agency has not

resolved how to make the training available to the remaining staff. EPA

also plans to release new guidance by the end of 2001 on how managers

 4

EPA/USACE Assessment of Cost Procedures and Methodologies: Comprehensive IGCE

Review (U.S. Army Corps of Engineers, Walla Walla District, Dec. 1999).

Results in Brief
Page 3 GAO-01-302 EPA Contract Cost Estimating

can prepare more detailed and complete estimates. The guidance will

include a Web-based tool with data on the actual, current costs of cleanup

activities. However, to make the data useful, the agency needs to ensure

that all data remain current. We are recommending that EPA resolve the

issues surrounding its training implementation and keeping cost data

updated and current.

Similarly, EPA could enhance the way it assesses its initiatives’ impact on

estimate quality. The agency plans to continue its current approach—

relying primarily on the on-site reviews of estimates it conducts in each

region once every 3 years—to evaluate an estimate’s quality and

usefulness. According to the Superfund managers in charge of contracting,

limited resources preclude more frequent on-site reviews. The managers

also believe that the current review process is successful because it

indicates improvements in estimates. While we recognize the resource

limitations of the Superfund program and the progress EPA has made to

date on improving estimates, we believe EPA could take relatively simple

actions to augment the regional reviews. For example, some regions

currently collect data on the estimated costs and awarded prices for their

contracted work. Superfund managers could analyze these data to look for

systemic problems across the agency that need to be addressed or

successful best practices that should be emulated. EPA’s and Corps’

contracting officials concurred that more frequent monitoring of estimates

would help EPA determine the initiatives’ effectiveness. Accordingly, we

are recommending that the agency routinely analyze data on its initiatives

across regions to facilitate any necessary adjustments and improvements.

Because of their experience with the technical aspects of cleanups, EPA

regional work assignment managers prepare detailed cost estimates for all

new work on Superfund projects before the agency contracts for that

work. These estimates should specify the scope of the work assignment

(such as conducting a cleanup feasibility study), the required hours, and

the expected cost of each major activity for that assignment. Regional

contract officers then use these estimates to negotiate with contractors

the best price that EPA can award for each work assignment, documenting

any significant differences between the estimate and the awarded price.

The agency also uses this process to negotiate any changes in the scope of

a work assignment as the cleanup proceeds that may lead to discrepancies

between the original awarded price and the final price of the contracted

work. The work assignment manager uses the estimate and any revisions

to develop the work plan and to manage the contractor’s work.

Background
Page 4 GAO-01-302 EPA Contract Cost Estimating

When we last examined EPA’s cost estimates in April 1999, we reported

that 11 (or about 31 percent) of the 35 estimates we reviewed fell

substantially below the awarded price—an indication that the estimates

may be of poor quality, according to the agency’s Financial Manager’s

Financial Integrity Act Report. In addition, in 29 percent of the cases, the

awarded price matched the contractor’s estimate—an indication that EPA

may not be using the estimates to negotiate the best price for the

government. In response to our report, the agency said it was awaiting the

results of the Corps’ review and would then take further action to correct

any cost-estimating problems.

EPA is in the process of implementing two initiatives designed to address

the weaknesses with its cost-estimating processes identified by the Corps

and us. The initiatives are a new training course and new guidance,

including a Web-based tool that provides work assignment managers with

data on the actual, current costs of contracted work to help them develop

more complete and accurate estimates. EPA has yet to determine how it

will make the training available to all managers when needed or how it will

ensure that the actual cost data in the tool remain up-to-date.

In our April 1999 report, we noted that work assignment managers, who

prepare the estimates, cited two barriers that kept them from developing

better estimates. First, 15 of the 34 work assignment managers we

interviewed (about 44 percent) cited their inexperience with estimate

preparation. Although work assignment managers are chiefly responsible

for managing the technical aspects of cleanup work, they had relatively

little cost-estimating experience, because they tended to prepare only one

or two estimates each year. They told us that more and better training

were needed to compensate for this lack of experience. Second, all of the

work assignment managers cited the need to access data on the actual

costs of previously contracted work. They believed that better data on

actual costs could serve as a baseline when determining what new

contracted work should cost.

Similarly, the Corps recommended in its December 1999 report that EPA

(1) improve the training it provides work assignment managers, and (2)

develop tools that contain better information on actual costs of

assignments to help the work assignment managers generate better

estimates. It also recommended that EPA (1) consolidate its various costestimating guidance documents, (2) encourage work assignment managers

to make more of a commitment to using their estimates to control contract

costs, and (3) better document the assumptions work assignment

EPA Has Launched

Initiatives to Address

Cost-Estimating

Weaknesses GAO and

the Corps Identified
Page 5 GAO-01-302 EPA Contract Cost Estimating

managers used to develop each estimate and any significant differences in

the estimate, the awarded price, and actual costs incurred. Actual costs

may differ if changes in the scope of the work occur as the work proceeds.

To address the Corps’ and our concerns, EPA designed a new training

course for regional work assignment managers and is currently in the

process of consolidating and revising its guidance on preparing estimates.

The agency offered its new half-day course targeted at preparing estimates

as an optional seminar during the last two annual conferences for work

assignment managers. According to EPA officials, about 50 (or 10 percent)

of the agency’s 500 work assignment managers attended one of these two

sessions. The course offered instruction on which cost components to

include in an estimate and addressed pricing the components and the

resources available to help staff with this pricing activity. These resources

included an introduction to a database on historical costs for contracted

work compiled by the New York regional office and tables detailing the

typical amount of labor hours needed to complete various contracted

cleanup activities. In addition, the course showed managers how to

adequately document the assumptions and data used to generate EPA’s

estimate. The work assignment managers who attended the course found

it very helpful and suggested improvements—such as additional

instruction on documenting estimates—that could make it even more

useful, according to the course evaluations they completed.

Superfund managers in charge of contracting practices have not yet

decided how to provide the training to all work assignment managers who

need it when they need it. In response to our questions, the managers said

that they planned to continue offering the course at the annual conference.

However, it is unlikely that a sufficient number of work assignment

managers will obtain the training they need in a timely and useful manner,

since the agency has only reached about 10 percent of the work

assignment managers in the past 2 years by relying on this method. To

provide more opportunities for work assignment managers to attend,

Superfund managers also said they are considering offering more than one

session during each conference and training key staff in each region, who

could, in turn, train the work assignment managers in that region.

However, Superfund managers do not have such a plan in place at this

point.

According to the Atlanta region’s cost estimator, who was instrumental in

developing the training course, it would be more effective for EPA to offer

future training in the regions rather than through the annual conferences

New Training Course
Page 6 GAO-01-302 EPA Contract Cost Estimating

because more useful and detailed cost information can be presented in

that forum. State project managers also attend the annual conferences,

and unlike EPA project managers, some of them are required to disclose to

contractors all of the information they used to develop their estimates.

EPA does not disclose how it prices estimates because that could diminish

the independence of the estimates and result in higher costs to the

government. By presenting the training in the regions exclusively to EPA

project managers, EPA can include the cost information without fearing

that it will be disclosed to contractors.

As its second initiative, the Superfund managers formed a workgroup—

composed of headquarters and regional contracting staff—to revise the

agency’s guidance to regional staff on how to prepare independent cost

estimates. According to the managers, the new guidance will consolidate

eight separate guidance documents currently in use and will emphasize

how good estimates are critical to controlling contract costs. The guidance

will also take users through the steps of developing a thorough estimate,

providing them with checklists detailing the scope of a work assignment,

giving users instructions for adequately documenting the steps, and

identifying reliable sources of cost data. EPA has contracted with the

Corps to use its expertise and databases to develop the data on the actual

costs of similarly contracted work and other information, such as data on

the labor hours needed for cleanup activities, to be included in the

guidance. EPA also plans to ask cleanup estimators from other agencies,

such as the U.S. Navy, to review these data before they are made available

to work assignment managers. The Superfund managers intend to publish

the guidance in paper form and make it available as an interactive tool via

EPA’s Web site by the end of 2001.

Data on the actual costs and the other information, such as the labor hours

needed, will only be useful if they are updated regularly and kept current.

To date, the Superfund managers have told us that their primary focus is to

develop and implement the revised guidance to ensure that the initial

version contains accurate data. They have not yet determined how they

will routinely keep the data current, including the data originally obtained

from the Corps, although they have discussed obtaining the Corps’

assistance on these issues. EPA plans to include a mechanism in the Webbased tool that allows work assignment managers to provide feedback on

the tool, which they hope will help them keep the data current. However,

because this feedback is voluntary, it does not provide a comprehensive

method for updating the actual cost data.

New Guidance
Page 7 GAO-01-302 EPA Contract Cost Estimating

EPA plans to continue its current method of evaluating the quality and

usefulness of estimates by relying primarily on its on-site reviews of

estimates in each region once every 3 years. Superfund managers believe

the reviews have been effective; furthermore, they do not believe they

have the resources needed to visit the regions more frequently. EPA could

supplement its regional reviews fairly easily by routinely analyzing

information on estimate quality that is now becoming available—such as

regional data comparing estimates to awarded prices—to look for

systemic problems that need addressing. According to contracting officials

from EPA and the Corps, additional monitoring would help EPA better

ascertain the initiatives’ effectiveness in the regions.

Currently, Superfund managers and officials from EPA’s Office of

Acquisition Management—the office responsible for the agency’s overall

contracting practices—review all Superfund contracting activities in the

regions by visiting each of the 10 regions approximately once every 3 years

to conduct reviews. Superfund managers assured us that, in response to

the concerns we raised in our 1999 report regarding estimate quality, the

reviews conducted in the past 2 years have included an examination of the

quality of a region’s estimates. The three most recent reviews, beginning

with the review of the Seattle region in June 1999, contain assessments of

estimates and recommendations for improvement.

In addition, Superfund managers told us that they use the periodic

meetings and monthly teleconferences of the Senior Regional Management

Acquisition Council as a forum to exchange information on issues or

problems affecting estimates. The Council includes representatives from

EPA’s Office of Acquisition Management and the Superfund program in

headquarters, and regional work assignment managers. Furthermore, the

managers told us that the discussion of estimates is a regular topic on the

agenda for the Council’s semiannual conferences.

Superfund managers plan to continue to rely on the regional reviews,

supplemented by the Council’s and other discussions, to determine

whether the quality and usefulness of contract estimates have improved as

a result of the new guidance and training. They pointed out that recent

regional reviews have found that estimate quality has generally improved

in the regions. The managers believe that examining estimates in three to

four regions each year, supplemented by the periodic Council discussions,

provides them with a good indication of any problems that may be

occurring with estimates nationwide. Furthermore, they cited a lack of

resources as a barrier limiting more frequent regional reviews.

EPA Could Better

Assess the Ultimate

Effectiveness of Its

Initiatives
Page 8 GAO-01-302 EPA Contract Cost Estimating

According to the director of the division in the Office of Acquisition

Management that oversees Superfund contracting activities, if the

resources for more frequent monitoring were available, EPA could better

determine how effective the initiatives are in the regions. Similarly,

according to the Corps’ manager who oversaw the review of EPA’s

estimating practices, EPA should review regional implementation of the

new guidance within 6 months to 1 year of its taking effect to ensure that

regional estimates are improving. Monitoring would be particularly

important in the early implementation phases. However, these officials

recognize that resource constraints are likely to rule out more frequent

regional reviews.

Superfund managers have at least two possible ways to augment the

regional reviews and better monitor the agency’s initiatives. First, in their

recent regional reviews, managers have recommended that the regions

begin tracking data that compare the estimates, the awarded prices, and

the final prices paid for the work as a way to gauge estimate quality in that

region. The Atlanta region has already begun tracking its comparison data,

according to the cost estimator in that region, who found the analysis very

helpful for making improvements. Superfund managers have not yet

required the remaining regions to collect the data or to forward it to

headquarters, where the managers could analyze the information across

regions. According to a Corps contracting official, the Corps performs

such an analysis to determine whether its estimates are adequate

nationwide. He believed that a similar analysis could help Superfund

managers quickly identify and correct any systemic problems or

incorporate any successful best practices.

Second, Superfund managers plan to include a comment section in the

new guidance, once it has been posted to the Superfund Web site, where

regional work assignment managers can voluntarily provide feedback on

the guidance as they begin to use it. The managers have not planned how

they will analyze and use this information but said they would develop a

plan before launching the new guidance by the end of 2001.

EPA’s and GAO’s reviews have shown that the agency has made significant

progress over the past decade in addressing the weakness of its costestimating processes. EPA regional work assignment managers are

currently developing independent estimates, which contracting officers

are using to negotiate the prices for cleanup work. The agency’s current

initiatives should help the agency successfully address the Corps’ and our

remaining concerns by providing the managers with the training and tools

Conclusions
Page 9 GAO-01-302 EPA Contract Cost Estimating

they need to develop better estimates. By incorporating some relatively

simple additional steps to more fully implement and better scrutinize the

effectiveness of the initiatives, the agency can better ensure that its efforts

improve cost estimates agencywide.

In order to more fully implement and better evaluate the improvements

currently being made to EPA’s cost-estimating processes, we recommend

that the Administrator, EPA, direct the Assistant Administrator, Office of

Solid Waste and Emergency Response, to complete plans on how to

• most effectively deliver improved, timely, and effective training to work

assignment managers nationwide,

• work with the Corps to keep the data on actual costs of contracted work

in its Web-based tool updated and current, and

• consolidate and routinely analyze regional data (which compare estimates,

awarded prices, and final prices paid to contractors) and the feedback

from work assignment managers on the Web-based tool to determine

whether systemic estimating problems exist that EPA needs to address or

whether best-estimating practices are available that it could adopt.

We provided a draft of this report to EPA for review and comment. We

subsequently met with designated Superfund managers, who agreed with

our recommendations and promised to take actions to begin their

implementation. Specifically, with respect to our first recommendation,

they agreed that regional training seminars could be tailored and timed to

meet the specific needs of the work assignment managers in the region.

Therefore, they are likely to develop region-specific training as the primary

training vehicle. In that regard, the Atlanta region’s cost estimator is

conducting a training session in February 2001 for the work assignment

managers in that region. Project officers from other regions are also

attending the training in Atlanta, and the Superfund managers said they

may consider whether it would be best for those project officers to

customize the training, and in turn, train the work assignment managers in

their own regions.

Responding to our second recommendation, Superfund managers agreed

that the Corps can most efficiently and effectively update cost data in the

Web-based tool. However, EPA has yet to work out the details of the

Corps’ future involvement in that effort.

Recommendations for

Executive Action

Agency Comments
Page 10 GAO-01-302 EPA Contract Cost Estimating

Finally, Superfund managers agreed that uniform regional data would help

them judge the quality of estimates agencywide, identify any potential

problems, and address them in a timely manner. Therefore, they agreed to

formalize the requirements for reporting regional data to track cost

estimates, awarded prices, and the actual prices of contracted cleanup

work.

We are sending copies of this report to appropriate congressional

committees and interested Members of Congress. We are also sending

copies to the Honorable Mitchell E. Daniels, Jr., Director, Office of

Management and Budget. In addition, we will make copies available to

others on request.

If you or your staff have any questions about this report, please contact me

at (202) 512-3841. Key contributors to this report were Eileen Larence,

Karla Springer, Elizabeth Erdmann, Jonathan S. McMurray, and Roger

Bothun.

Sincerely yours,

David G. Wood

Director, Natural Resources

 and Environment

(160548)
The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs