Superfund: EPA's Contract Cost-Estimating Initiatives Show Promise and Should Be Monitored
Released: 2001-03-02T00:00:00

Letter

Background

Because of their experience with the technical aspects of cleanups, EPA regional work assignment managers prepare detailed cost estimates for all new work on Superfund projects before the agency contracts for that work. These estimates should specify the scope of the work assignment (such as conducting a cleanup feasibility study), the required hours, and the expected cost of each major activity for that assignment. Regional contract officers then use these estimates to negotiate with contractors the best price that EPA can award for each work assignment, documenting any significant differences between the estimate and the awarded price. The agency also uses this process to negotiate any changes in the scope of a work assignment as the cleanup proceeds that may lead to discrepancies between the original awarded price and the final price of the contracted work. The work assignment manager uses the estimate and any revisions to develop the work plan and to manage the contractor’s work.

When we last examined EPA’s cost estimates in April 1999, we reported that 11 (or about 31 percent) of the 35 estimates we reviewed fell substantially below the awarded price—an indication that the estimates may be of poor quality, according to the agency’s Financial Manager’s Financial Integrity Act Report. In addition, in 29 percent of the cases, the awarded price matched the contractor’s estimate—an indication that EPA may not be using the estimates to negotiate the best price for the government. In response to our report, the agency said it was awaiting the results of the Corps’ review and would then take further action to correct any cost-estimating problems.

EPA Has Launched Initiatives to Address Cost-Estimating Weaknesses GAO and the Corps Identified

EPA is in the process of implementing two initiatives designed to address the weaknesses with its cost-estimating processes identified by the Corps and us. The initiatives are a new training course and new guidance, including a Web-based tool that provides work assignment managers with data on the actual, current costs of contracted work to help them develop more complete and accurate estimates. EPA has yet to determine how it will make the training available to all managers when needed or how it will ensure that the actual cost data in the tool remain up-to-date.

In our April 1999 report, we noted that work assignment managers, who prepare the estimates, cited two barriers that kept them from developing better estimates. First, 15 of the 34 work assignment managers we interviewed (about 44 percent) cited their inexperience with estimate preparation. Although work assignment managers are chiefly responsible for managing the technical aspects of cleanup work, they had relatively little cost-estimating experience, because they tended to prepare only one or two estimates each year. They told us that more and better training were needed to compensate for this lack of experience. Second, all of the work assignment managers cited the need to access data on the actual costs of previously contracted work. They believed that better data on actual costs could serve as a baseline when determining what new contracted work should cost.

Similarly, the Corps recommended in its December 1999 report that EPA (1) improve the training it provides work assignment managers, and (2) develop tools that contain better information on actual costs of assignments to help the work assignment managers generate better estimates. It also recommended that EPA (1) consolidate its various cost- estimating guidance documents, (2) encourage work assignment managers to make more of a commitment to using their estimates to control contract costs, and (3) better document the assumptions work assignment managers used to develop each estimate and any significant differences in the estimate, the awarded price, and actual costs incurred. Actual costs may differ if changes in the scope of the work occur as the work proceeds.

New Training Course

To address the Corps’ and our concerns, EPA designed a new training course for regional work assignment managers and is currently in the process of consolidating and revising its guidance on preparing estimates. The agency offered its new half-day course targeted at preparing estimates as an optional seminar during the last two annual conferences for work assignment managers. According to EPA officials, about 50 (or 10 percent) of the agency’s 500 work assignment managers attended one of these two sessions. The course offered instruction on which cost components to include in an estimate and addressed pricing the components and the resources available to help staff with this pricing activity. These resources included an introduction to a database on historical costs for contracted work compiled by the New York regional office and tables detailing the typical amount of labor hours needed to complete various contracted cleanup activities. In addition, the course showed managers how to adequately document the assumptions and data used to generate EPA’s estimate. The work assignment managers who attended the course found it very helpful and suggested improvements—such as additional instruction on documenting estimates—that could make it even more useful, according to the course evaluations they completed.

Superfund managers in charge of contracting practices have not yet decided how to provide the training to all work assignment managers who need it when they need it. In response to our questions, the managers said that they planned to continue offering the course at the annual conference. However, it is unlikely that a sufficient number of work assignment managers will obtain the training they need in a timely and useful manner, since the agency has only reached about 10 percent of the work assignment managers in the past 2 years by relying on this method. To provide more opportunities for work assignment managers to attend, Superfund managers also said they are considering offering more than one session during each conference and training key staff in each region, who could, in turn, train the work assignment managers in that region. However, Superfund managers do not have such a plan in place at this point.

According to the Atlanta region’s cost estimator, who was instrumental in developing the training course, it would be more effective for EPA to offer future training in the regions rather than through the annual conferences because more useful and detailed cost information can be presented in that forum. State project managers also attend the annual conferences, and unlike EPA project managers, some of them are required to disclose to contractors all of the information they used to develop their estimates. EPA does not disclose how it prices estimates because that could diminish the independence of the estimates and result in higher costs to the government. By presenting the training in the regions exclusively to EPA project managers, EPA can include the cost information without fearing that it will be disclosed to contractors.

New Guidance

As its second initiative, the Superfund managers formed a workgroup— composed of headquarters and regional contracting staff—to revise the agency’s guidance to regional staff on how to prepare independent cost estimates. According to the managers, the new guidance will consolidate eight separate guidance documents currently in use and will emphasize how good estimates are critical to controlling contract costs. The guidance will also take users through the steps of developing a thorough estimate, providing them with checklists detailing the scope of a work assignment, giving users instructions for adequately documenting the steps, and identifying reliable sources of cost data. EPA has contracted with the Corps to use its expertise and databases to develop the data on the actual costs of similarly contracted work and other information, such as data on the labor hours needed for cleanup activities, to be included in the guidance. EPA also plans to ask cleanup estimators from other agencies, such as the U.S. Navy, to review these data before they are made available to work assignment managers. The Superfund managers intend to publish the guidance in paper form and make it available as an interactive tool via EPA’s Web site by the end of 2001.

Data on the actual costs and the other information, such as the labor hours needed, will only be useful if they are updated regularly and kept current. To date, the Superfund managers have told us that their primary focus is to develop and implement the revised guidance to ensure that the initial version contains accurate data. They have not yet determined how they will routinely keep the data current, including the data originally obtained from the Corps, although they have discussed obtaining the Corps’ assistance on these issues. EPA plans to include a mechanism in the Web- based tool that allows work assignment managers to provide feedback on the tool, which they hope will help them keep the data current. However, because this feedback is voluntary, it does not provide a comprehensive method for updating the actual cost data.

EPA Could Better Assess the Ultimate Effectiveness of Its Initiatives

EPA plans to continue its current method of evaluating the quality and usefulness of estimates by relying primarily on its on-site reviews of estimates in each region once every 3 years. Superfund managers believe the reviews have been effective; furthermore, they do not believe they have the resources needed to visit the regions more frequently. EPA could supplement its regional reviews fairly easily by routinely analyzing information on estimate quality that is now becoming available—such as regional data comparing estimates to awarded prices—to look for systemic problems that need addressing. According to contracting officials from EPA and the Corps, additional monitoring would help EPA better ascertain the initiatives’ effectiveness in the regions.

Currently, Superfund managers and officials from EPA’s Office of Acquisition Management—the office responsible for the agency’s overall contracting practices—review all Superfund contracting activities in the regions by visiting each of the 10 regions approximately once every 3 years to conduct reviews. Superfund managers assured us that, in response to the concerns we raised in our 1999 report regarding estimate quality, the reviews conducted in the past 2 years have included an examination of the quality of a region’s estimates. The three most recent reviews, beginning with the review of the Seattle region in June 1999, contain assessments of estimates and recommendations for improvement.

In addition, Superfund managers told us that they use the periodic meetings and monthly teleconferences of the Senior Regional Management Acquisition Council as a forum to exchange information on issues or problems affecting estimates. The Council includes representatives from EPA’s Office of Acquisition Management and the Superfund program in headquarters, and regional work assignment managers. Furthermore, the managers told us that the discussion of estimates is a regular topic on the agenda for the Council’s semiannual conferences.

Superfund managers plan to continue to rely on the regional reviews, supplemented by the Council’s and other discussions, to determine whether the quality and usefulness of contract estimates have improved as a result of the new guidance and training. They pointed out that recent regional reviews have found that estimate quality has generally improved in the regions. The managers believe that examining estimates in three to four regions each year, supplemented by the periodic Council discussions, provides them with a good indication of any problems that may be occurring with estimates nationwide. Furthermore, they cited a lack of resources as a barrier limiting more frequent regional reviews.

According to the director of the division in the Office of Acquisition Management that oversees Superfund contracting activities, if the resources for more frequent monitoring were available, EPA could better determine how effective the initiatives are in the regions. Similarly, according to the Corps’ manager who oversaw the review of EPA’s estimating practices, EPA should review regional implementation of the new guidance within 6 months to 1 year of its taking effect to ensure that regional estimates are improving. Monitoring would be particularly important in the early implementation phases. However, these officials recognize that resource constraints are likely to rule out more frequent regional reviews.

Superfund managers have at least two possible ways to augment the regional reviews and better monitor the agency’s initiatives. First, in their recent regional reviews, managers have recommended that the regions begin tracking data that compare the estimates, the awarded prices, and the final prices paid for the work as a way to gauge estimate quality in that region. The Atlanta region has already begun tracking its comparison data, according to the cost estimator in that region, who found the analysis very helpful for making improvements. Superfund managers have not yet required the remaining regions to collect the data or to forward it to headquarters, where the managers could analyze the information across regions. According to a Corps contracting official, the Corps performs such an analysis to determine whether its estimates are adequate nationwide. He believed that a similar analysis could help Superfund managers quickly identify and correct any systemic problems or incorporate any successful best practices.

Second, Superfund managers plan to include a comment section in the new guidance, once it has been posted to the Superfund Web site, where regional work assignment managers can voluntarily provide feedback on the guidance as they begin to use it. The managers have not planned how they will analyze and use this information but said they would develop a plan before launching the new guidance by the end of 2001.

Conclusions

EPA’s and GAO’s reviews have shown that the agency has made significant progress over the past decade in addressing the weakness of its cost- estimating processes. EPA regional work assignment managers are currently developing independent estimates, which contracting officers are using to negotiate the prices for cleanup work. The agency’s current initiatives should help the agency successfully address the Corps’ and our remaining concerns by providing the managers with the training and tools they need to develop better estimates. By incorporating some relatively simple additional steps to more fully implement and better scrutinize the effectiveness of the initiatives, the agency can better ensure that its efforts improve cost estimates agencywide.

Recommendations for Executive Action

In order to more fully implement and better evaluate the improvements currently being made to EPA’s cost-estimating processes, we recommend that the Administrator, EPA, direct the Assistant Administrator, Office of Solid Waste and Emergency Response, to complete plans on how to most effectively deliver improved, timely, and effective training to work work with the Corps to keep the data on actual costs of contracted work in its Web-based tool updated and current, and consolidate and routinely analyze regional data (which compare estimates, awarded prices, and final prices paid to contractors) and the feedback from work assignment managers on the Web-based tool to determine whether systemic estimating problems exist that EPA needs to address or whether best-estimating practices are available that it could adopt.

Agency Comments

We provided a draft of this report to EPA for review and comment. We subsequently met with designated Superfund managers, who agreed with our recommendations and promised to take actions to begin their implementation. Specifically, with respect to our first recommendation, they agreed that regional training seminars could be tailored and timed to meet the specific needs of the work assignment managers in the region. Therefore, they are likely to develop region-specific training as the primary training vehicle. In that regard, the Atlanta region’s cost estimator is conducting a training session in February 2001 for the work assignment managers in that region. Project officers from other regions are also attending the training in Atlanta, and the Superfund managers said they may consider whether it would be best for those project officers to customize the training, and in turn, train the work assignment managers in their own regions.

Responding to our second recommendation, Superfund managers agreed that the Corps can most efficiently and effectively update cost data in the Web-based tool. However, EPA has yet to work out the details of the Corps’ future involvement in that effort.

Finally, Superfund managers agreed that uniform regional data would help them judge the quality of estimates agencywide, identify any potential problems, and address them in a timely manner. Therefore, they agreed to formalize the requirements for reporting regional data to track cost estimates, awarded prices, and the actual prices of contracted cleanup work.

We are sending copies of this report to appropriate congressional committees and interested Members of Congress. We are also sending copies to the Honorable Mitchell E. Daniels, Jr., Director, Office of Management and Budget. In addition, we will make copies available to others on request.

If you or your staff have any questions about this report, please contact me at (202) 512-3841. Key contributors to this report were Eileen Larence, Karla Springer, Elizabeth Erdmann, Jonathan S. McMurray, and Roger Bothun.