Department of Energy: Further Actions Are Needed to Strengthen Contract Management for Major Projects
Published: 2005-03-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Energy (DOE) pays its contractors billions of dollars each year to implement its major projects--those costing more than $400 million each. Many major projects have experienced substantial cost and schedule overruns, largely because of contract management problems. GAO was asked to assess, for major departmental projects, (1) DOE's use of performance incentives to effectively control costs and maintain schedules, (2) the reliability of the data DOE uses to monitor and assess contractor performance, and (3) the reliability of the Project Assessment and Reporting System (PARS) data that senior managers use for project oversight.
section_title: What GAO Found
paragraphs: DOE could use performance incentives more effectively for controlling costs and schedules if it developed performance incentive guidance and assigned responsibility for reviewing a contract's project management provisions prior to award. DOE has awarded contracts for 15 of 33 major projects that use a schedule or other performance incentive without an associated cost incentive or constraint; thus a contractor could receive full fees by meeting all schedule baselines while substantially overrunning costs. DOE has relied on unvalidated contractor data to monitor contractors' progress in executing major projects and to award fees for performance. In particular, DOE's self-assessment of contract administration in 2002 found that field personnel overly relied on contractors' accounting systems and contractor-collected data in assessing performance, without significant validation of those data. No subsequent self-assessment has been conducted to determine if this problem continues. Furthermore, DOE has not required that its contracting officers receive the training needed to assess the adequacy of contractors' project management systems that generate data used to monitor progress. A lthough development of PARS is a positive step, the reliability of the project performance data that PARS provides to senior DOE managers is limited by problems with accuracy, completeness, and timeliness. Regarding accuracy, DOE has not assessed the reliability of contractors' project management systems that feed data into PARS for 31 of 33 major projects, even though DOE believes that some systems are deficient. Regarding completeness, GAO identified 3 major projects that are not in PARS. As to timeliness, cost and schedule data for 6 major projects in the June 2004 PARS report were significantly out of date because DOE has not required contractors to submit timely performance data. These contract management problems limit DOE's ability to effectively manage its major projects and avoid further cost and schedule slippages.

section_title: Letter
paragraphs: The Department of Energy (DOE) pays its contractors billions of dollars  each year to implement projects costing more than $400 million each,  which the department designates as major projects. These major projects  include environmental cleanup at its current and former nuclear weapons  production facilities, refurbishment of nuclear weapons, and construction  of specialized scientific facilities. Our previous reports have found that  many of DOE’s major projects have experienced substantial cost overruns  and delays. As a result, since 1990, we have designated DOE’s contract  management—broadly defined to include contract administration and  project management—as a high-risk area for fraud, waste, abuse, and  mismanagement because of the department’s history of inadequate  management and oversight and failure to hold its contractors accountable.  In recent years, we have continued to find problems with DOE’s contract  management of its major projects. (See the list of related GAO products at  the end of this report.) For example, our 2002 assessment of DOE’s  contract reform initiatives found that 5 of the 16 major projects we  examined had more than doubled in cost—for billions of dollars in total  cost overruns—and experienced more than 5 years in delays.
In response to these problems, DOE has instituted contracting and project  management reforms. To better align DOE functions with challenges to  improve the planning, execution, and management of its contracts, the  department established the Office of Contract Management in 2000 with  responsibility for assisting the field and other DOE offices in the planning,  implementation, and oversight of (1) the contract award process, which  involves those front-end activities necessary to ensure that contracts are  structured in a way that best fulfills DOE’s needs, and (2) the contract  administration process, which encompasses all dealings between DOE and  its contractors from the time a contract is awarded until the work has been  completed and accepted or the contract has been terminated, payment  made, and any disputes resolved. To improve the contract award process,  the Office of Contract Management has, among other things, developed  updates to DOE’s Acquisition Guide, which amplifies upon the  requirements contained in the Federal Acquisition Regulation (FAR) that  governs governmentwide federal procurement activities. Each year, the  Office of Contract Management reviews a limited number of pending  contract actions, including any matters affecting a contract’s deliverables,  schedule, and cost. To improve the contract administration process, the  Office of Contract Management issued, in May 2000, a Reference Book for  Contract Administrators, a consolidated reference tool for DOE’s contract  administrators. The office also has performed periodic self-assessments of  contract administration practices.
As part of its project management reforms, DOE established the Office of  Engineering and Construction Management in 1999 to oversee the project  management process by developing project management policy, improving  project oversight systems, and implementing a career development  program for DOE project managers. In particular, the Office of Engineering  and Construction Management in 2001 implemented the Project  Assessment and Reporting System (PARS), a Web-based system for  collecting and analyzing current performance data for projects costing  more than $5 million. Although the department has other ways to  communicate project status information, including quarterly reports for  some projects, PARS monthly project status reports for the Deputy  Secretary of Energy (hereafter referred to as PARS reports) are DOE’s  primary tool for keeping senior managers apprised of a project’s  performance. In August 2003, the office initiated a certification program for  contractors’ project management systems that assesses the accuracy of the  cost and schedule performance data that contractors generate by  examining, in particular, whether each contractor’s system complies with  private industry’s standard for earned value management—a systematic  approach for integrating and measuring cost, schedule, and technical  (scope) accomplishments on a project.
You asked us to examine DOE’s contract management actions designed to  control cost growth and schedule slippage for its major projects.  Specifically, you asked us to assess, for major departmental projects, (1)  DOE’s use of performance incentives to effectively control costs and  maintain schedules, (2) the reliability of the data DOE uses to monitor and  assess contractor performance, and (3) the reliability of the PARS data that  senior managers use for project oversight. Our review focused primarily on  the 33 major projects that had passed, as of March 2004, DOE’s Critical  Decision 2 milestone—the point at which the department approves a  project’s cost, schedule, and scope baselines on the basis of an approved  conceptual design report and acquisition strategy. (See app. I for the 33  major projects that we reviewed.) The projects we reviewed include 28  projects that cost more than $400 million each and 5 projects that our 2002  assessment defined as major projects because their total costs exceeded  $100 million. DOE’s Office of Environmental Management (Environmental  Management), Office of Science, and National Nuclear Security  Administration (NNSA) are funding the 33 major projects.
To determine DOE’s guidance for using different performance incentives  for major projects, we examined the contract provisions for the 33 major  projects that have DOE-approved performance baselines, reviewed various  recent contract-related documents associated with these 33 projects, and  discussed the contents of these documents with key DOE contracting  officials. We then compared the contract provisions with requirements in  the FAR; the DOE Acquisition Regulation; and DOE’s contract-award  guidance, including the Performance-Based Contracting Guide and the  Acquisition Guide. To assess DOE’s oversight of contract administration  activities, we reviewed DOE’s Reference Book for Contract  Administrators, compared the findings made in DOE’s various contract  administration self-assessment reports with the actions the department has  taken, and interviewed contracting officials at the Department of Defense  and administrators at the National Aeronautics and Space Administration  because DOE officials acknowledged their leadership in implementing  such project and contract management reforms as earned value  management principles. To evaluate the reliability of PARS data, we  assessed the data’s accuracy, completeness, and timeliness by examining  the underlying data and the way data are reported to senior managers. For  example, we compared the information in monthly PARS reports from  January through September 2004 with project-specific data obtained from  various DOE program offices. When we identified discrepancies, we  assessed whether the discrepancies limited the reliability of data reported  in PARS. Given our review of the documentation provided by DOE and our  discussions with DOE officials, we have reservations about the reliability  of PARS data. These issues are discussed in this report. We conducted our  work between January 2004 and January 2005 in accordance with generally  accepted government auditing standards, which included an assessment of  data reliability and internal controls. (See app. II for further information  about the scope and methodology of our review.)
subsections: section_title: Background
paragraphs: DOE is the largest civilian contracting agency in the federal government;  about 90 percent of its annual budget is spent on contracts for carrying out  its activities and operating its facilities. In fulfilling their missions, DOE’s  program offices are responsible for contracting for and overseeing the  execution of the department’s major projects, many of which are first-of-a- kind efforts and thus involve substantial risk and may also be separate line  items in DOE’s budget. For example:  Environmental Management’s mission is to accelerate risk reduction  and cleanup of the environmental legacy of the nation’s nuclear  weapons program and government-sponsored nuclear energy research.  Environmental Management has used a single sitewide contract that  involves several major projects costing billions of dollars for cleaning up  some of its former facilities. In addition, Environmental Management  has undertaken many large-scale individual projects. For example, the  Hanford Tank Waste Treatment and Immobilization Plant project is an  important part of the cleanup effort at Hanford, Washington. The  project, which was initiated in December 2000, is intended to treat and  prepare for disposal 55 million gallons of high-level radioactive waste by  July 2011 at an estimated cost of $5.7 billion.
NNSA’s mission is to meet national security requirements by, among  other things, maintaining and enhancing the safety, reliability, and  performance of the U.S. nuclear weapons stockpile, which includes  maintaining the capability to design, produce, and test nuclear weapons.  To fulfill this mission, NNSA undertakes such projects as refurbishing  W-80 nuclear warheads to extend their operational lives. The W-80  refurbishment project was initiated in September 1998 and is expected  to be completed in fiscal year 2017 at an estimated cost of about $2.45  billion.
The Office of Science’s mission is to deliver the remarkable discoveries  and scientific tools that transform our understanding of energy and  matter and advance the national, economic, and energy security of the  United States. To fulfill this mission, the Office of Science has  constructed specialized scientific research facilities, such as the  Spallation Neutron Source at the Oak Ridge National Laboratory. This  project consists of an accelerator system that delivers short  (microsecond) pulses to a target/moderator system where neutrons are  produced by a nuclear reactor process called spallation. This project is  designed to provide the next-generation spallation neutron source for  neutron scattering and related research in broad areas of the physical,  chemical, materials, biological, and medical sciences. The Spallation  Neutron Source project began in October 1998 and is expected to be  completed in June 2006 at an estimated cost of about $1.4 billion.
DOE’s principal official responsible for the execution of a major project is  the federal project director, who is located at the project site and is  supported by project managers. The project director is responsible for  overseeing a project’s design, execution, budgeting, and performance. For  contracts with award fee provisions, senior DOE program office managers  consult with contracting and project officers to assess a contractor’s  performance and determine the appropriate award fees.
In addition to the contract management problems our prior reports have  identified, a recent series of reports by the National Research Council of  the National Academies identified weaknesses in DOE’s project  management. The council’s 2004 report cited several factors that have  contributed to the slow pace of project management improvements and  resulted in inconsistent project performance. These factors include the  desire of DOE site office personnel and contractors to be independent of  oversight from DOE headquarters, insufficient support for training,  inadequate numbers of DOE project managers to oversee contractors’  performance, and the absence of a champion for project managers and  process improvement who has the authority to ensure both adherence to  policies and procedures and the availability of necessary funding and  personnel resources.
During the past year, DOE has continued to implement contracting and  project management reforms. In particular, in December 2003, the  Secretary of Energy appointed an Associate Deputy Secretary with  responsibility, among other things, for both contract and project  management, addressing a key National Research Council concern. DOE  also entered into an agreement with the Defense Contract Management  Agency, within the Department of Defense, to support the certification of  contractors’ project management systems. More recently, DOE is  developing an action plan in response to the Civil Engineering Research  Foundation’s assessment of departmental project management that  recommended that DOE, among other things, develop a core group of  highly qualified project directors, require peer reviews for first-of-a-kind  and technically complex projects when the projects’ preliminary baselines  are approved, and enhance PARS by making the data more timely.   Furthermore, to improve its contract award process, DOE revised its  Acquisition Guide by adding chapter 16, which lists the various contract  types available and discusses their respective advantages and constraints.  To address future skill gaps in its procurement organization, DOE  established an acquisition career development program and has certified 90  percent of its procurement professionals as attaining mandatory training  and experience standards under this program. Within the Office of  Environmental Management, a series of contract and project management  improvements have occurred consisting of, but not limited to, providing  additional training and managing more of the cleanup work as projects.  Within the Office of Contract Management, a series of contract award and  administration initiatives have been completed. These initiatives include,  among other things, strengthening contract competition policies and  practices, improving acquisition workforce effectiveness, increasing small  business utilization throughout DOE, and strengthening DOE management  and fiscal effectiveness. For fiscal year 2005, the Office of Contract  Management has multiple initiatives planned, including identifying and  implementing follow-on actions related to the DOE management challenge  pertaining to contract competition.
section_title: DOE Could Use Performance Incentives More Effectively for Controlling Costs and Schedules
paragraphs: Because many of DOE’s major projects are first-of-a-kind and thus involve  substantial risk, DOE’s contracting decisions can be critical to the  successful completion of its major projects. However, DOE could use  performance incentives more effectively for controlling costs and  schedules for its major projects if the department developed criteria for  using different performance incentives and assigned responsibility for  reviewing a contract’s project management provisions prior to award. For  example, DOE has used contracts that have a technical, schedule, or other  performance incentive without an associated cost incentive or cost  constraint (other than the annual funding level for the contract). DOE also  has used cost-plus-incentive-fee contracts without certifying that  contractors’ project management systems generate reliable cost and  schedule data for measuring performance and awarding fees. In addition,  we found that the contract incentives for most of the 25 major  environmental restoration projects substantially differ from the “Gold  Chart” performance metrics that Environmental Management uses to  assess its performance and report its progress to the Congress.  Furthermore, for 11 major projects that are components of the  environmental cleanup of a DOE facility, Environmental Management has  not directly linked incentive fees to the successful completion of the  project, generally because the project is part of the contractor’s larger  cleanup responsibility. Finally, while Environmental Management has  decided that incentive fee determinations would consider only contractor  activities directly related to cleanup work, NNSA has, for at least 1 of its  major projects, considered a contractor’s indirect work-related activities in  awarding incentive fees.
subsections: section_title: DOE Has Not Fully Developed Performance Incentive Guidance to Effectively Control Costs and Maintain Schedules
paragraphs: Despite efforts in recent years to improve contract and project  management, DOE has not fully developed performance incentive guidance  to effectively control costs and maintain schedules. DOE has issued the  following guidance, order, and manual that are applicable to the contract  award process for major projects and that supplement the FAR and the  DOE Acquisition Regulation:  In the late 1990s, DOE issued its Acquisition Guide to, among other  things, supplement the FAR and the DOE Acquisition Regulation and be  a repository of best practices found throughout the department. Chapter  16 of the guide discusses contract types; however, the chapter notes that  it was not intended to provide a template for matching a contract type to  given contracting situations. While the guide’s index shows that chapter  34 is reserved for guidance to contracting officials related to major  projects, DOE has never drafted the chapter, according to the DOE  official responsible for maintaining revisions to the Acquisition Guide.
In October 2000, DOE issued Order 413.3, “Program and Project  Management for the Acquisition of Capital Assets,” to ensure that capital  assets, including major projects, would be delivered on schedule, within  budget, and fully capable of meeting mission needs. To accomplish  these goals, the order states, in part, that DOE officials are to develop an  acquisition plan during the acquisition process that includes such  elements as contracting options and a contractor incentive process. The  order, however, does not elaborate on the possible contracting and  performance incentive options whatsoever.
In March 2003, DOE issued manual 413.3-1, “Project Management for the  Acquisition of Capital Assets,” to improve the implementation of DOE  Order 413.3. The manual addresses various activities, including a  chapter on contracting that contains no direct reference to major  projects. The chapter states that the type of contract and incentives  proposed should be based on an overall view of the principal risks to the  project and provides a limited discussion of the types of contracts  available. For example, it states that fixed-price contracts are not  appropriate for research and development efforts or other complex  projects where there is a high degree of uncertainty in the execution or  DOE requirements. While the chapter mentions that DOE generally uses  a cost-plus-award-fee contract for contractors managing and operating  DOE sites, it does not address the other available types of contract.
Furthermore, DOE has not used its Acquisition Guide to identify best  practices, or lessons learned, based on its major project contracting  experiences. In our view, given DOE’s long history with major projects,  considerable information could be added to this guide detailing those  major project contracting approaches that worked and those that did not.  Improved guidance could help DOE better control costs and maintain  schedules for its major projects.
section_title: DOE Has Not Always Reviewed the Project Management Provisions of Its Major Contracts Prior to Award
paragraphs: Neither the Office of Contract Management nor the Office of Engineering  and Construction Management always reviews the project management  provisions of major project contracts prior to award to ensure that the  performance incentives are appropriately used. At the heart of this problem  is confusion over responsibility. The Director of the Office of Contract  Management and the Director of the Office of Engineering and  Construction Management each believe that the other office has  headquarters responsibility for reviewing the project management  provisions of contracts prior to approval.
The confusion exists because the chapter in DOE’s Acquisition Guide on  the headquarters review of contract and financial assistance actions is  silent on the role of the Office of Engineering and Construction  Management in the review process. This chapter indicates that packages  pertaining to contract actions will be sent to nine different DOE offices for  review, none of which is the Office of Engineering and Construction  Management. As a consequence, if this office has a role in the contract  review process, it has not been clearly defined.
According to the Director, Office of Contract Management, the Office of  Engineering and Construction Management should be responsible for  reviewing the project management provisions in major project contracts  because of its responsibility for project management matters. The director  told us that his office typically reviews from 60 to 70 pending contract  actions each year, and these reviews follow a general approach looking at  any matters that might affect timing, delivery, and cost—but no specific,  formalized list is followed.
According to the Director of the Office of Engineering and Construction  Management, his office reviews certain documentation that could affect  which company is selected for a contract, but his office has no role in  reviewing the actual provisions of the contract. While the Office of  Contract Management sends contract proposals to the Office of  Engineering and Construction Management for review, the director noted  that his office has only one staff person with contracting experience. The  director believes the solution to improving the review of major project  contracts is for contracting officials within the Office of Contract  Management to become more familiar with earned value management, a  DOE contracting requirement for integrating and measuring a contractor’s  performance.
section_title: Problems Have Developed Because DOE Has Not Effectively Used Performance Incentives
paragraphs: For many of the 33 major projects we reviewed, DOE has used  performance incentives that limit its ability to effectively control cost and  schedule performance. For example, almost all of DOE’s cost-plus-award- fee contracts for major projects have included a performance incentive  without also using an associated cost incentive or cost constraint (other  than the annual funding level for the contract). Also, DOE has used cost- plus-incentive-fee contracts without certifying that contractors’ project  management systems generate reliable cost and schedule data for  measuring performance and awarding fees. We also found that (1)  Environmental Management’s contracts included environmental cleanup  performance incentives that differed substantially from its new Gold Chart  performance metrics; (2) DOE did not always link its fee awards to  contractors’ performance on major projects; and (3) DOE’s program offices  have treated indirect work-related activities, such as providing timely and  accurate reports to DOE, differently in determining the contractors’  incentive award fees.
subsections: section_title: DOE’s Implementation of Performance Incentive Provisions Does Not Enable DOE to Effectively Control Costs
paragraphs: For 15 of the 17 major projects that use a cost-plus-award-fee contract, the  contract contained a technical, schedule, or other performance incentive  without including an associated cost incentive or cost constraint (other  than the annual funding level for the contract). Under such circumstances,  the potential exists that a contractor could meet all incentives and overrun  baseline costs but still receive full fees. The other 2 major projects used a  cost-plus-award-fee contract that included an associated cost incentive or  cost constraint for each technical, schedule, or other performance  incentive.
The FAR, the DOE Acquisition Regulation, and DOE guidance preclude the  inclusion of a schedule or other performance incentive without also  including a cost incentive or cost constraint. FAR § 16.402-1 states that no  incentive contract may provide for other incentives without also providing  for a cost incentive or cost constraint. Similarly, DOE Acquisition  Regulation § 970.5215-3 provides that requirements incentivized by other  than cost incentives must be performed within their specified cost  constraint. DOE’s Performance-Based Contracting Guide, dated October  2003, states that (1) cost incentives should be included if other incentives  are included because a schedule or other performance incentive may result  in the contractor paying little attention to the cost of achieving those  incentives unless cost is also a consideration and (2) DOE contracts, in  developing incentives and incentive programs, must comply with the  incentive contract provisions of the FAR and the DOE Acquisition  Regulation.
The Director of the Office of Contract Management told us that to  implement the FAR requirement to include a cost incentive or cost  constraint whenever a noncost incentive is in the contract, each noncost  incentive does not necessarily need an associated cost constraint dedicated  to that noncost incentive. According to the director, a single cost  constraint, which could be equivalent to the project’s annual funding level,  would fulfill the FAR requirement.
However, DOE contracting officials at Oak Ridge, West Valley, and  Savannah River believe that to implement the FAR and DOE Acquisition  Regulation requirements in a way that effectively controls costs, a contract  with a technical, schedule, or other noncost incentive should also have an  associated cost incentive to function as a constraint on the expenditure of  funds. One of these officials added that as the noncost incentives become  more objective and measurable, the cost constraint should be more clearly  defined in relation to each noncost incentive. Similarly, another one of  these officials told us that using the annual funding level or the project’s  cost baseline as the constraint is too vague and unworkable, and that some  funding levels and cost baselines do not track down to the performance  incentive level. As a result, neither the funding level nor the cost baseline  would indicate whether the performance incentive was accomplished  within the cost constraint.
These views are consistent with the findings from DOE’s 1997 assessment  of performance-based incentives, which found that DOE’s and contractors’  financial systems generally are budget-based and do not segregate and  track costs at the performance incentive level. The assessment added that  this limits DOE’s ability to establish meaningful cost baselines and to  monitor the cost of performance under specific incentivized work efforts in  relation to the total cost of the contract.
section_title: DOE Has Used Cost-Plus- Incentive-Fee Contracts without Certifying the Reliability of Contractors’ Performance Data
paragraphs: For 13 of the 33 major projects we reviewed, DOE used a cost-plus- incentive-fee contract that provides the contractor with an initially  negotiated fee that is subsequently adjusted by a formula based on the  relationship of total allowable costs to total target costs. The formula  provides, within limits, for fee increases when total allowable costs are less  than target costs. In recent years, DOE has made a major effort to move  toward the use of cost-plus-incentive-fee contracts.
Because a cost-plus-incentive-fee contract provides higher fee awards to  the extent that actual costs are lower than anticipated, it depends upon  reliable cost estimating at the outset in the form of a target cost and reliable  cost reporting later. In July 1997, the Office of Management and Budget  (OMB) issued requirements regarding the acceptability of contractors’  project management systems. However, DOE has not certified the  reliability of contractors’ project management systems that generate the  target cost data for the 13 major projects. As a result, a contractor might  receive a high fee payment because its project management system  generated an unreliable high initial cost estimate and subsequently  reported lower actual costs. A U.S. Army Corps of Engineers’ report, issued  in May 2004, concluded that it was not appropriate to use a cost-plus- incentive-fee contract for the Hanford Tank Waste Treatment and  Immobilization Plant project, in part because reliable cost data could not  be generated in advance.
Furthermore, DOE site personnel may not provide adequate surveillance of  the contractors’ cost records for these 13 projects. According to DOE’s  Performance-Based Contracting Guide, it is inappropriate to use a cost- plus-incentive-fee contract if there is an overreliance on contractor  accounting systems and contractor-collected data without significant  validation of those data. In such situations, the guide states, any potential  cost savings reported might be the result of a poor estimate of the amount  of labor or material required, the approach planned, or the associated  costs. The Office of Contract Management’s self-assessment of contract  administration in 2002 found that most of the DOE field locations visited  relied almost exclusively on the contractors’ data because they did not  have the staff resources capable of validating cost or technical baselines.  The report, however, did not identify the DOE field locations visited, and,  according to an Office of Contract Management official, no individual field  location reports were prepared.
section_title: Contracts’ Performance Incentives Differed from the Gold Chart Performance Metrics
paragraphs: For 16 of the 25 major environmental restoration projects that we  reviewed, the contracts’ performance incentives differed substantially from  the Gold Chart performance metrics that Environmental Management uses  to assess its performance and report its progress to the Congress.   Environmental Management developed the Gold Chart performance  metrics in October 2002 as a basis for clearly and objectively showing the  progress being made in the environmental cleanup program. We found,  however, that these Gold Chart metrics were not being used to measure  contractors’ performance or award fees. Instead, DOE measures  performance and awards fees on the basis of information from the  contractors’ project management systems, which DOE has not yet certified  as capable of producing reliable information.
For 4 projects at the Fernald Closure Site in Ohio, a lower performance fee  might have been appropriate if the Gold Chart metric had been used. For  fiscal year 2003, DOE awarded the contractor about $7.7 million of the $8  million in available fee, or 97 percent, on the basis of acceptable cost and  schedule performance toward closure of the entire site during fiscal year  2003. However, according to the fiscal year 2003 Gold Chart metrics, the  goal for the Fernald Closure Project was to accomplish four radioactive  facility completions and dispose of 2,568 cubic meters of radioactive  waste. According to Environmental Management information, the  contractor did not fully complete one of these tasks. Because the  contractor accomplished only three of the four radioactive facility  completions, Environmental Management might have given a different fee  amount if the two Gold Chart metrics had been used to determine award  fee.
Conversely, a different fee amount might have been warranted for the Solid  Waste Stabilization and Disposition project at Hanford, Washington. For  fiscal year 2003, DOE awarded the contractor about $2.2 million of about $3  million in available fee, or 73 percent, on the basis of the contractor’s  disposal of radioactive waste in accordance with an approved schedule  that DOE determined the contractor had met. In contrast, Environmental  Management data for fiscal year 2003, using Gold Chart metrics, show that  the contractor actually disposed of 3,634 cubic meters of waste as  compared with a goal of disposing 2,320 cubic meters of waste, or about  157 percent of the work intended. If the Gold Chart metrics had been used  to determine the award fee, the contractor might have received a different  fee amount.
For the Spent Nuclear Fuels project, at Hanford, Washington, the Gold  Chart metric and the contract’s performance incentive were so dissimilar  that it was difficult to determine how to gauge the contractor’s  performance. For fiscal year 2003, DOE awarded the contractor about $2.8  million of about $3.3 million in available fee, or 85 percent, on the basis of  the contractor’s removing 777 metric tons, or 87 percent, of the 890 metric  tons that had been planned. However, Environmental Management data for  fiscal year 2003, using the Gold Chart metrics, show the contractor  removed 805 units, or 94 percent, of the goal’s 855 units. Because the Gold  Chart metric and the contract’s performance incentive were so dissimilar,  we could not reconcile the information.
Environmental Management officials told us that the performance  incentives contained in environmental cleanup contracts and the Gold  Chart metrics should be aligned. In commenting on the draft report,  Environmental Management officials stated that the new Savannah River  cleanup contract incorporates Gold Chart metrics. They added, however,  that the contract renewals for the Oak Ridge, Fernald, and Rocky Flats  facilities do not contain the Gold Chart metrics because each is a cost-plus- incentive-fee contract that awards fee based on the final closure costs and  date for the site. It is unclear whether these cost-plus-incentive-fee  contracts will more effectively track contractors’ performance because  they rely on contractors’ project management systems that DOE has yet to  certify. In contrast, the Gold Chart metrics assess the accomplishment of  discrete amounts of work that is verifiable.
section_title: Incentive Fees Paid to Contractors Were Not Directly Tied to Performance for Some Major Projects
paragraphs: In 1996, we reported that a key factor inhibiting the successful completion  of DOE’s major projects was the lack of effective incentives. To the extent  that incentives are properly applied, they can help achieve agency goals. On  the other hand, if incentives are nonexistent or not effectively applied, a  project may not be successfully completed.
Sixteen of the 33 major projects we reviewed had no incentive fees directly  associated with the successful completion of work. Nine of these 16  projects involve closure work at the Fernald and Rocky Flats sites, where  the payment of incentive fees is based on an overall average of the cost and  schedule status for all site closure activities, including major projects and  other site activities. Environmental Management officials told us that  rather than awarding incentive fees specifically for completing any of the 9  major projects, or for other key interim milestones, the Fernald and Rocky  Flats contracts award provisional incentive fees for meeting or exceeding  overall targets for a fiscal year, provided the contractors successfully  achieve site closure on schedule.
However, it remains to be seen whether this approach will be effective in  completing major projects on time and within cost. For example, although  a major project at the Fernald site that we reviewed was experiencing cost  growth to the point where it was expected to exceed its cost baseline—the  total cost estimate to accomplish the project—DOE considered the overall  average of the cost and schedule status for all site activities at Fernald to be  acceptable and paid the contractor provisional incentive fees for fiscal year  2003. Similarly, a major project at the Rocky Flats site had overrun its  estimated cost by about $42 million through fiscal year 2003. However, this  overrun was offset by an underrun of about $46 million in activities such as  general counsel work and planning and integration that, according to DOE  information, had historically been understaffed. The net effect was that  DOE paid the contractor provisional incentive fees because the  contractor’s overall cost and schedule status for fiscal year 2003 was  considered to be acceptable.
section_title: DOE Program Offices Have Treated Indirect Work-Related Activities Differently in Awarding Incentive Fees
paragraphs: In addition to these other contracting problems, we found that DOE  program offices treated indirect work-related activities differently in  awarding incentive fees. In late 2002, Environmental Management decided  that award fee determinations will consider only contractor activities  directly related to cleanup work, while excluding such indirect work- related activities as providing timely and accurate reports to DOE,  providing support services to the government, and complying with the  contract because these activities are basic expectations of any contractor.  Environmental Management made this determination after its review of  contractors’ authorized fee incentives identified numerous examples of  incentive fee payments for indirect work-related activities. The review also  found that Environmental Management was paying some contractors  additional fees for performing work safely that the review concluded was a  basic expectation, and not exceptional performance worthy of additional  fee.
NNSA has not conducted a review similar to Environmental Management’s  assessing what, if any, indirect work-related activities are worthy of  incentive payments. The contractor for one NNSA major project received  incentive fee payments for providing timely and accurate reports to DOE  and other indirect work-related activities during fiscal year 2003.  Discrepancies in the treatment of various indirect work-related activities  have occurred because DOE’s guidance does not address the  appropriateness of including a contractor’s performance of indirect   work-related activities in determining incentive fee awards.
In commenting on the draft report, Environmental Management expressed  concern that it would be virtually impossible to develop meaningful  guidance that could be applied universally to DOE’s diverse programs. We  disagree. We believe that all DOE programs should use incentive fees to  reward contractors for achieving work-related activities, as opposed to  such indirect activities as providing the DOE programs with timely reports.
section_title: DOE Has Relied on Unvalidated Contractor Data to Monitor and Assess Contractors’ Performance for Major Projects
paragraphs: Because most of DOE’s operations are carried out through contracts,  contract administration is a significant part of DOE’s work. DOE has relied  on unvalidated contractor data to monitor contractors’ progress in  executing major projects and awarding fees for performance. This  reliance on unvalidated data limits the department’s ability to ensure it gets  what it is paying for. Specifically, DOE’s self-assessments of its contract  administration in 1997 and 2002 both found that field personnel overly  relied on contractor accounting systems and contractor-collected project  data without significant validation of these data. However, unlike the 1997  self-assessment, the one in 2002 made no recommendation to fix this  problem, and no subsequent self-assessment has been initiated to  determine if the problem has continued. DOE has begun to certify the  reliability of contractors’ project management systems that generate the  performance data used to monitor contractors’ progress; however, the  department has no time table for the completion of this certification  program. In addition, DOE has not required its contracting officers and  contracting officer representatives to receive training in earned value  management—a systematic approach for integrating and measuring cost,  schedule, and technical (scope) accomplishments on a project or task— even though these officials are required to determine whether contractors’  project management systems meet the private industry’s earned value  management standard.
subsections: section_title: DOE Has Not Used Self- assessments to Correct an Overreliance on Contractor Data
paragraphs: Self-assessment is an important tool for evaluating organizational  effectiveness. By taking a comprehensive look at itself, an organization can  identify weaknesses and plot a course of corrective action. DOE performed  comprehensive self-assessments of its contract administration practices in  1997, 1999, and 2002.
In 1997, DOE assessed 20 contracts to ensure that financial incentives  contained in those contracts were rational, linked to well-defined  performance objectives and measures, and properly administered. The self- assessment reported both positive and negative findings. For example, it  found that the use of performance-based objectives generally had been  effective in directing contractors’ management attention to desired  performance outcomes. However, it also found that field personnel overly  relied on contractor accounting systems and contractor-collected data  without significant validation of these data, and that DOE’s approval of fees  earned by the contractors relied upon contractor-generated documents. To  correct this deficiency, the self-assessment recommended (1) that the  cognizant DOE heads of contracting at each field location, as part of their  overall contract administration plan, identify the mechanisms,  responsibilities, and authorities for ensuring that contractor performance  against established objectives is appropriately monitored and (2) that  performance achievements are verified.
In 1999, DOE’s follow-up assessment of the effectiveness of the actions  taken in response to the 1997 self-assessment found that the  recommendation that contractor performance be monitored and  achievements verified had been implemented. Specifically, field offices  reported that their plans for administering contracts had been  appropriately modified and instituted. In addition, the follow-up  assessment stated that (1) early results indicated a substantial  improvement in the way incentives were being managed from DOE  headquarters and administered at DOE field contracting offices and (2)  anecdotal evidence suggested that contractor performance had improved.
In 2002, the Contract Administration Division again performed a self- assessment that examined, in part, how contract administration planning  and execution was conducted at various DOE field locations. The findings  and conclusions of this review were somewhat inconsistent with those of  the 1999 follow-up assessment. The 2002 review, like the 1997 assessment,  determined that few sites had the resources capable of validating  contractor cost or technical information and most sites must rely almost  exclusively on the contractor’s data. The review noted, in one instance, that  financial data provided by the contractor were generally accepted by DOE,  not on the basis of reasonableness and allowability, but on the basis of the  contractor’s “acceptable” self-assessment of the procedures used to collect  those data. However, unlike the 1997 assessment, the 2002 review  contained no specific recommendation to correct this overreliance on  contractor data.
According to the Director of DOE’s Contract Administration Division,  because of funding constraints and other factors, no broad self-assessment  of contractor administration has been done since 2002. The director added  that DOE now conducts individual site assessments as necessary rather  than conducting more comprehensive assessments. According to  information provided to us in April 2004, the last individual site assessment  was made in August 2003 and documented in December 2003. This site  assessment identified problems similar to those reported in the 2002 self- assessment. Specifically, the site assessment noted that, with respect to  one contract reviewed, there was no evidence of effective cost controls  and/or contract management. The site assessment contained no formal  recommendation to fix this problem. On the other hand, the site  assessment contained a recommendation to address the high rate of  expenditure on this contract over the remaining 2-year-option period. The  assessment recommended that the DOE site office review the scope and  cost of its current task orders for prioritization and inclusion in the  remaining option term.
In August 2003, DOE began to certify the reliability of contractors’ project  management systems that generate the performance data used to monitor  contractors’ progress. However, as of December 2004, the department has  assessed and certified project management systems for only 2 of the 33  major projects we reviewed and does not have a time table for completing  this certification program.
In commenting on the draft report, DOE noted that both Environmental  Management and the Office of Engineering and Construction Management  have been validating contractors’ cost and schedule performance baselines  for several years. In our view, DOE validation of contractor baselines will  not fully address the problems that have been identified. Validating  baselines is just the first step in performing adequate contractor oversight.  After baselines have been validated, DOE must not overly rely on  contractor accounting systems in reporting costs and on contractor- collected project data in awarding fees. That is the message from two DOE  self-assessments of performance-based contracting. With respect to DOE’s  experience in baseline validation, the Civil Engineering Research  Foundation’s July 2004 report for the Office of Engineering and  Construction Management found that some improvements in baseline  validation were needed. This report noted that many of the DOE projects it  reviewed were formulated with inadequate baseline estimates. In addition,  the report stated that periodic baseline changes were occurring that  masked the true status of certain projects. The report recommended that  DOE develop guidelines that appropriately control the rebaselining of  projects.
DOE further stated that the promulgation of contract management  planning guidance and the requirement for a contract management plan  addressed many of the issues that the 2002 self-assessment identified.  However, in our view, until a subsequent assessment is done, it remains  unclear whether this DOE action has adequately resolved the issues  identified in the 2002 self-assessment. For fiscal year 2005, DOE is planning  to examine the contract management plans and contractors’ purchasing  systems.
section_title: DOE Has Not Required Its Contracting Officers and Contracting Officer Representatives to Receive Earned Value Management Training
paragraphs: During the early 1990s, OMB issued several reports on civilian agencies’  contract administration practices that found that agencies frequently  experienced cost overruns and delays in receiving goods and services  because their contracting officials allocated more time to awarding  contracts than to administering existing ones. In response, OMB revised its  Circular A-11 to require that federal agencies assess and certify  contractors’ project management systems for proper use of earned value  management principles. OMB also identified several other deficiencies,  including a lack of proper training for agency officials performing contract  oversight. According to administrators at the National Aeronautics and  Space Administration, earned value management training is essential for  their contracting officers to adequately assess whether a contractor’s  project management system complies with the private industry’s standard.
We found that, with the exception of NNSA, DOE has not required its  contracting officers or contracting officer representatives to receive earned  value management training, even though they are responsible for  determining whether the contractor’s project management system  complies with the private industry’s earned value management standard  after the contract is awarded. The following three DOE documents contain  the contracting officer’s responsibilities, the standards against which those  responsibilities are to be discharged, and the training requirements for  contracting officers:  Chapter 1 of DOE’s Reference Book for Contract Administrators, issued  in 2000 and in effect through October 2004, outlines the contracting  officers’ many responsibilities, including a review of the adequacy of the  contractor’s project management system. The reference book states  that the system’s adequacy must be confirmed by the contracting officer  with the support of other DOE headquarters and field office personnel,  as appropriate. The reference book also indicates that corrective action  plans resulting from DOE reviews of contractor project management  systems are to be tracked until the DOE contracting officer confirms  that all open issues are closed.
DOE Order 413.3, “Program and Project Management for the Acquisition  of Capital Assets,” also issued in 2000, specifies that contractors’ project  management systems must comply with the American National  Standards Institute’s standard on earned value management. The order  states that this requirement applies only to systems involved in  controlling the performance of projects costing more than $20 million in  total. The order also requires that contractors’ systems provide cost and  schedule performance, milestone status, and financial status to DOE on  a monthly basis.
DOE Order 361.1A, “Acquisition Career Development Program,” issued  in April 2004, outlines the training and certification requirements for  DOE contracting officers and contracting officer representatives. The  order identifies a training curriculum for these officers by functional  area—including, among others, procurement contracts; interagency  agreements and sales contracts; grants and cooperative agreements;  loans and loan guarantees; and the government purchase card. The  order, however, does not require either the contracting officer or the  contracting officer representative to receive earned value management  training.
The Director of the Contract Administration Division corroborated our  assessment of DOE’s order for acquisition career development. The  director noted that the only reference to earned value management training  in DOE Order 361.1A requires that DOE project directors, not contracting  officers, complete a course on earned value management systems. Without  this training, however, it is unclear how DOE contracting officers and  contracting officer representatives can fulfill their responsibilities and  properly assess the adequacy of the project management systems of  departmental contractors. In providing us with exit conference comments,  DOE Office of Contract Management officials acknowledged that  contracting officers do have a responsibility in the area of earned value  management and will be receiving training on that subject in the future.  Subsequently, in December 2004, DOE provided contracting professionals  at DOE headquarters with a 1-hour course on earned value management.
DOE said that this training session, which was video recorded, is being  required nationwide for all DOE contracting officials. As opposed to this  1-hour course, we noted that NNSA requires its contracting officials to  participate in a 48-hour course on the fundamentals of earned value  management.
section_title: The Reliability of PARS Data Is Limited
paragraphs: The reliability of the project cost and schedule data that PARS provides to  senior DOE managers is limited by problems with the data’s accuracy,  completeness, and timeliness. In general, the accuracy of PARS report data  is uncertain because DOE (1) has assessed the reliability of contractors’  project management systems for only 2 of the 33 major projects we  reviewed, (2) generally measures projects’ cost and schedule performance  in PARS against the current DOE-approved cost and schedule baselines  without also tracking performance against the original targets, and (3) has  not provided most of its major project directors with the training needed to  ensure contractors are generating accurate performance data. PARS report  data are not complete because DOE program offices have not submitted  performance data to PARS for 3 major projects, as well as at least 2 smaller  projects, and PARS reports do not provide each project’s estimated cost at  completion or other helpful, forward-looking data. In addition, the Office of  Engineering and Construction Management stated that the June 2004 PARS  report’s performance data for 6 major projects and 5 smaller projects were  significantly out of date, primarily because contractors did not provide  updated project performance information. Senior managers have used  PARS data to take actions that averted cost increases for certain projects  that were experiencing cost or schedule challenges. Without reliable data,  however, PARS has not provided senior managers with information about  cost increases and schedule slippages for many projects, and the status of  many other projects is uncertain.
subsections: section_title: The Accuracy of Most PARS Data Is Uncertain
paragraphs: Three factors impair the accuracy of cost and schedule data reported in  PARS. First, DOE officials told us they have little assurance that the cost  and schedule data for most projects in PARS are accurate because DOE has  not assessed the reliability of contractors’ project management systems  that generate such data for data reliability, particularly those systems  believed to be using incorrect methods. Second, for almost all projects,  PARS reports compare cost and schedule performance against DOE’s  current baselines, without identifying the extent of cost or schedule  slippages that previously occurred. Third, most DOE project directors lack  the necessary training to evaluate and verify the accuracy of the  performance data that contractors generate, according to DOE officials.
subsections: section_title: DOE Has Assessed Few Contractors’ Systems That Generate Project Data
paragraphs: OMB Circular A-11 and DOE Order 413.3 require that DOE assess and  certify contractors’ project management systems for proper use of earned  value management principles in generating cost and schedule performance  data before the department approves a project’s cost and schedule baseline  at its Critical Decision 2 milestone. Earned value management, when used  correctly, produces data that reflect a contractor’s progress toward  completing a project within cost and schedule targets. In essence, earned  value management measures the value of work completed against the cost  and schedule of work planned, as opposed to comparing actual with  planned expenditures. To illustrate, assume a contract calls for 4 miles of  railroad track to be laid in 4 weeks at a cost of $4 million. After 3 weeks of  work, assume $2 million has been spent. By analyzing planned versus  actual expenditures, it appears the project is underrunning the estimated  costs. However, an earned value analysis reveals that the project is in  trouble because even though only $2 million has been spent, only 1 mile of  track has been laid; thus, the contract is only 25 percent complete. On the  basis of the value of work done, the project will cost $8 million ($2 million  to complete each mile of track), and the 4 miles of track will take a total of  12 weeks (3 weeks for each mile of track) to complete instead of the  originally estimated 4 weeks.
To ensure correct application of earned value management principles,  contractors must develop budgets and schedules based on measurable  components of a project, which include defined start points, end points,  and scopes of work. In addition, contractors must calculate the value of  work performed against the budgets and schedules for the measurable  project components. Experts in earned value management told us that  without defined start and end points and other measurable project  components, project performance data give little insight as to whether cost  and schedule performance are on track, and the data might mask more  serious problems.
DOE’s Office of Engineering and Construction Management and the  Defense Contract Management Agency assess whether a given contractor’s  project management system properly uses earned value management  principles by examining whether the contractor’s system complies with the  industry standards and verifying that the contractor is using the system to  manage the project. Once a contractor has fully addressed the concerns  identified by the assessment, DOE is to certify the project management  system, attesting that project performance data—data that convey progress  toward the approved cost and schedule targets—are generated reliably.  Assessment and certification of contractors’ earned value management  systems are critical components of DOE’s management of its performance- based contracting, according to DOE earned value management training  documents. While only three systems have been assessed since August  2003, Office of Engineering and Construction Management officials told us  that they and the Defense Contract Management Agency, working together,  could assess the project management systems for about 10 contractors in a  given year now that they are becoming more familiar with the process.
In August 2003, the Office of Engineering and Construction Management  and the Defense Contract Management Agency began the process of  assessing contractors’ project management systems as a basis for certifying  that they properly use earned value management principles. In September  2004, DOE certified Sandia National Laboratories’ project management  system for 1 major project, the Microsystems and Engineering Sciences  Applications project, and 6 smaller projects. DOE also plans to certify Oak  Ridge National Laboratory’s project management system for another major  project, the Spallation Neutron Source, once minor deficiencies are  corrected. Overall, however, DOE has assessed project management  systems for only 2 of the 33 major projects we reviewed—and 8 of the 73  projects in PARS—that have passed Critical Decision 2 with DOE-approved  cost and schedule baselines. (The remaining 65 projects in PARS whose  systems have not been assessed have baseline costs of nearly $75 billion.)   According to an Office of Engineering and Construction Management  official, the first three contractors’ systems were selected for assessment  on the basis of visibility, significance, and criticality to the Department’s  success, but also because cognizant DOE officials were confident that the  contractors’ project management systems would meet certification criteria.
The National Research Council’s 2004 report on DOE’s project management  found that the quality of earned value management across the department’s  projects was inconsistent and stated that senior DOE managers do not  know whether the reported data on cost and schedule performance are  accurate unless contractors’ systems are assessed and certified. Because  DOE has only recently begun to assess contractors’ project management  systems that feed data into PARS, DOE officials acknowledged to us that  they lack assurance regarding the accuracy of PARS performance data,  adding that they believe some of the project management systems not yet  assessed have important deficiencies. For example, a DOE expert in earned  value management noted that contractors for most Environmental  Management projects—about half of the projects in PARS that have passed  Critical Decision 2—have not properly implemented earned value  management principles because, among other things, many of the projects’  components lack defined start and end points. For example, the earned  value management expert believes, on the basis of his assessment of work  breakdown structures and other project components, that the contractor’s  project management system for the $10-billion Yucca Mountain Nuclear  Waste Repository project does not properly use earned value management  principles and generates performance data that cannot be regarded as  accurate. Consequently, senior DOE managers have no assurance that cost  and schedule targets will be met, even if the data suggest they will.
Similarly, for several major projects we examined, the contractors’ project  management systems do not seem to properly implement earned value  management principles to measure cost and schedule performance. For  example, the $2-billion East Tennessee Technology Park project at Oak  Ridge lacks measurable project components. In some instances, work is  categorized into activities such as “general operations” and “contractor  operations” that have no apparent defined start and end points. According  to the expert in earned value management, the categories of work for this  project make it difficult to accurately measure project performance  because there is no clear activity or time frame against which to measure  costs incurred or time spent. Instead, PARS data for this project seem to  measure only the project’s expenditures, which can conceal information on  the project’s cost and schedule status and progress toward completion. In  addition, the $5.7-billion Tank Waste Treatment and Immobilization Plant at  Hanford, Washington, lacks discrete, measurable project components  because work is categorized into activities such as “providing technology”  and “providing infrastructure” that lack defined start and end points. While  we recognize that it is appropriate, according to industry standards, to  categorize a small amount of work in this fashion, DOE project  management officials said the particular categories of work in these  instances reflected a poor comprehension of earned value management  and limited their confidence in the assessment of project performance.
Two Office of Engineering and Construction Management officials  acknowledged that the accuracy of data for these projects is uncertain  because DOE has not assessed whether the contractors’ project  management systems properly applied earned value management  principles. One of these officials suggested that the contractors’ project  management systems for such projects should be assessed as soon as  possible to correct deficiencies and improve the reliability of project  performance data provided to senior managers to oversee progress toward  cost and schedule targets. The Director of the Office of Engineering and  Construction Management agreed that DOE should develop a schedule that  would give priority to assessing these and other high-risk and high-cost  systems. As of January 2005, a schedule had not been developed, but the  director told us that he was in the process of doing so.
section_title: PARS Reports Generally Do Not Show Total Cost Overruns and Schedule Slippages
paragraphs: The accuracy of the PARS report data is further impaired because PARS  reports generally do not show total cost overruns and schedule slippages,  even though DOE requires each project team to estimate life-cycle costs  and assess project performance against established cost and schedule  baselines. Instead, a project’s DOE project director updates the cost and  schedule baselines in PARS when DOE approves a contract modification.  As a result, PARS reports show relatively small variances between a  project’s actual performance and its approved baselines, so that many of  the projects we reviewed appear not to have experienced problems when,  in fact, they did. For almost all projects, PARS reports do not provide data  that would enable senior DOE managers to assess (1) a contractor’s  performance against the project’s original DOE-approved baselines to  identify total cost overruns and schedule slippages or (2) the effect of any  DOE initiatives to control a project’s costs. The Civil Engineering Research  Foundation’s July 2004 report similarly found that PARS cost and schedule  data often do not convey the actual status of projects since their inception  because of periodic revisions of cost or schedule baselines. Furthermore,  for most Environmental Management projects, PARS measures project  performance from arbitrary dates, such as the beginning of the fiscal year,  which do not necessarily correspond to progress toward DOE-approved  targets. The following examples illustrate how PARS has masked problems  with projects by giving an incomplete picture of project costs or project  performance:  The January 2004 PARS report showed that the $1.6-billion Spent  Nuclear Fuels Stabilization and Disposition project at Hanford,  Washington, was on track to meet cost and schedule performance  targets. However, by April, total costs for the project increased by  nearly $150 million. DOE officials acknowledged that because the  January 2004 PARS report to senior DOE managers measured only  project performance from the beginning of the fiscal year, instead of  against the DOE-approved baselines, the PARS report concealed longer  term problems that threatened the project’s completion within costs.
In October 2002, the Tritium Extraction Facility at Savannah River,  South Carolina, had an approved total cost of about $400 million. Costs  for the project increased more than $100 million by September 2003, and  subsequent PARS reports showed that costs were on track to meet cost  targets, despite the 25 percent increase in the project’s costs.
In June 2004, Environmental Management restructured the PARS  reporting for 4 projects at Oak Ridge, Tennessee, by combining their  respective costs and schedules with those of other Oak Ridge projects.   As a result, Environmental Management stopped reporting project  performance data for each project, masking the fact that 2 of them,  totaling about $300 million, were significantly behind schedule. Two  Office of Engineering and Construction Management officials believe  the projects should be reported separately because combining projects’  respective cost and schedule data can inhibit the correct use of earned  value management.
The April 2004 PARS report showed that the total cost of the Soil and  Water Remediation project at Ashtabula, Ohio, would be $45 million,  although the performance data indicated the project would not likely  meet its baselines. However, this amount does not include about $109  million in expenditures on this project by October 1, 2003.  Environmental Management reports this project’s total costs to be about  $157 million—more than three times the amount reported in PARS.
PARS reports that total project costs for the Nuclear Facility  Deactivation and Decommissioning project at Columbus, Ohio, will be  about $31.5 million. However, this amount does not include about $106  million in expenditures prior to 2004. Environmental Management  estimates that this project’s total cost will exceed $163 million—more  than five times the amount reported in PARS.
The June 2004 PARS report showed that 90 percent of the 63 projects with  approved baselines were expected to meet their cost and schedule  baselines. However, this percentage may reflect project managers’ efforts  to keep the projects’ baselines up to date rather than improvements in  project management performance because PARS generally measures  projects’ performance against the most current DOE-approved baselines.  For example, as shown in table 1, the October 2002 PARS report’s  assessment of 2 major projects was red because both projects were  expected to breach their cost/schedule performance baselines. However,  the September 2003 PARS report’s assessment of these major projects was  green because total project costs were within the revised baseline that  DOE had subsequently approved. The September 2003 PARS report did not  indicate the extent to which each project’s total costs had exceeded the  costs that DOE approved at Critical Decision 2 on the basis of an approved  conceptual design report and acquisition strategy.
In addition to these projects, the 90 percent figure includes many  Environmental Management projects, whose performance is measured  over time frames that do not necessarily reflect performance against DOE- approved baselines. Further, the 90 percent figure does not reflect the 4  Oak Ridge projects whose performance data showed imminent  performance problems before being combined with the performance data  of other projects at the site.
DOE officials told us that the monthly PARS reports are the primary tool  for communicating project performance information to senior  management. However, for many projects—particularly those overseen by  Environmental Management—PARS does not report projects’ life-cycle  costs or performance against original baselines, even though DOE requires  each project team to estimate life-cycle costs and assess project  performance against established cost baselines and schedule milestones.  Office of Engineering and Construction Management officials  acknowledged that reporting life-cycle costs and project performance  against original cost and schedule baselines in PARS would make cost or  schedule challenges easier to identify, and Environmental Management  officials told us they plan to report life-cycle costs and project performance  against original baselines in PARS reports beginning by December 2004.
In addition to Environmental Management’s plans for PARS reporting, the  Office of Engineering and Construction Management intends to make  several upgrades to the PARS database, such as making the process for  entering monthly data more efficient and easier for users to understand and  ensuring that the correct data are being entered. Office of Engineering and  Construction Management officials reported that they are in the process of  implementing these improvements. However, these upgrades do not  address the limitations to reporting accurate data that we identified.  Furthermore, these improvements do not address limitations in the  reliability of data stemming from contractor’s project management systems  that have not been assessed or data that have not been reviewed.
section_title: Most DOE Project Directors Lack Certification in Earned Value Management
paragraphs: Project directors are DOE’s focal point for assessing the contractors’ cost  and schedule performance data that feed into PARS. However, most of  DOE’s project directors have not been certified in earned value  management, further reducing assurances that PARS data are accurate.  Because DOE believes that it is critical for project directors to understand  earned value management, the department informally designates its project  directors as “acting directors” if they have not completed the project  manager career development program, which includes training in earned  value management. Office of Engineering and Construction Management  officials told us that while some acting project directors are proficient in  earned value management and capable of evaluating the reliability of  contractor-generated data, other acting project directors are not.
DOE recently implemented the project management career development  program through which project directors are being trained in, among other  things, earned value management. However, DOE had trained only about 25  percent of them through this program as of July 2004, with plans to train  the remaining 75 percent by May 2006. A DOE official told us that the  appropriate level of earned value management training for acting project  directors depends on their experience in using earned value management.  While DOE aims to assess project directors’ capabilities in earned value  management to ensure that they are competent, validating the adequacy of  prior earned value management experience for acting project directors has  been time consuming. The lack of trained projects directors reviewing the  accuracy of a project’s performance data may, in some cases, adversely  affect the ability of senior DOE managers to properly assess the status of  major projects.
section_title: PARS Lacks Complete Information
paragraphs: In addition to reporting data of questionable accuracy, PARS provides  incomplete data, therefore senior DOE managers may not be aware of the  need to implement corrective actions to prevent cost overruns or schedule  slippages. We identified the following 5 projects—3 major projects to  refurbish nuclear weapons and 2 projects costing more than $100 million  each—that are not in the PARS database, despite DOE’s requirement that  projects costing more than $5 million provide monthly reporting: W80 Life Extension Program. NNSA recently increased the total cost of  this program, designed to extend the service life of the W80 nuclear  warhead by replacing components, from $1.3 billion to about $2.45  billion.
W76 Trident Missile Life Extension Program. NNSA expects this  project, designed to extend the service life of the W76 nuclear warhead  by replacing components, to cost about $680 million over the next 4  years.
B61 Alteration 357 Life Extension Program. NNSA expects this  project, designed to extend the service life of the B61 bomb, to cost  nearly $500 million. Our July 2003 report recommended that DOE  improve its oversight of the life extension program’s cost and schedule  status.
Purple and BlueGene/L Supercomputers under the Advanced  Simulation and Computing Program. NNSA expects this project, to  cost about $290 million and be completed in 2005.
Enterprise Project. NNSA increased the total cost of this project, which  will replace the accounting and management systems at Los Alamos  National Laboratory, from about $70 million when it was initiated in  2001 to nearly $160 million.
The National Research Council’s 2004 report found that DOE has not acted  in a timely fashion to include all projects costing more than $5 million in  PARS. Office of Engineering and Construction Management officials told us  DOE is still in the process of applying project management principles to  many of the department’s operational activities. While DOE’s program  offices are responsible for converting these activities to projects, many of  the program office personnel responsible for applying project management  principles do not have the necessary training, according to an Office of  Engineering and Construction Management official. While project  management training is available, DOE has required only project directors  and other senior-level employees to take this training. An Office of  Engineering and Construction Management official told us this training  would help expedite the application of project management principles to  DOE’s operational activities.
In addition, for many projects included in the PARS database, PARS reports  do not provide important performance information that senior DOE  managers need to assess the projects’ status. In some cases, project  performance data are not reported because the project is incorrectly listed  as being in the design phase when, in fact, it has passed Critical Decision 2.  For example, contractors have spent almost half of the approved funds for  2 projects at the Idaho National Engineering and Environmental Laboratory  projected to cost $4.3 billion without reporting performance data in PARS.   The PARS reports show that these projects are still in the design phase and,  therefore, are not subject to reporting performance data, but a DOE official  acknowledged that both projects have, in fact, passed Critical Decision 3  and other subsequent milestones. As a result, senior DOE managers cannot  rely on PARS for accurate and current performance information for these  projects, nor can they rely on PARS to determine whether these projects  require corrective actions.
For these and other projects, PARS also lacks forward-looking data, such  as scheduled work to be performed, the projects’ upcoming milestones,  and the projects’ estimated cost at completion. Without such data, PARS  cannot provide information on projects’ cost or schedule challenges and  DOE management does not have a basis for projecting progress or  identifying trends. While not in PARS, this information is available from  acting project directors. For example, although early cost savings for the  Microsystems and Engineering Sciences Applications project at Sandia  National Laboratories led to favorable performance data, DOE’s project  director identified supply imbalances in the steel market that would  increase the estimated construction costs. Using this information, the  project director revised the project’s estimated total cost. Currently, PARS  reports to senior DOE managers lack such forward-looking data that could  alert them to future cost or schedule challenges. The National Research  Council’s 2004 report stated that PARS reports should display forward- looking data to notify senior managers of upcoming milestones. In addition,  several acting project directors told us that forward-looking data, such as  data on estimated costs at completion, should be included in PARS to  identify project performance challenges for senior DOE managers.
To further illustrate this need, the total costs of some DOE projects are  projected to increase dramatically in the future, despite PARS reports  showing that they are expected to be completed on time and within budget.  For example, PARS report data show that the Hanford’s Tank Waste  Treatment and Immobilization Plant is projected to meet the DOE- approved baseline of $5.78 billion. However, PARS does not show that DOE  approved a $1.4-billion increase above the project’s original contract  estimate of $4.35 billion in April 2003, nor does it show that the U.S. Army  Corps of Engineers, in a May 2004 report, stated that project costs would  probably exceed the $5.78-billion cost baseline by $720 million.
Even though the DOE project management teams knew of cost and  schedule performance problems for the Tank Waste Treatment and  Immobilization Plant project, PARS reports have shown that this project  was on track for meeting cost and schedule targets. An Office of  Engineering and Construction Management official told us that PARS  monthly reports do not include forward-looking data and trend data to  minimize the amount of time necessary for senior managers’ review. As a  result, PARS did not provide senior DOE managers for this and other  projects with important information to analyze potential future challenges.  Forward-looking performance information, such as scheduled work to be  performed and estimated cost at completion, would better enable senior  managers to address project management challenges and minimize cost  overruns or schedule slippages.
section_title: PARS Lacks Timely Information
paragraphs: Further compounding reliability concerns, we identified problems with the  timeliness of PARS data that may limit the ability of senior DOE managers  to effectively identify and apply corrective actions. Specifically, we found  that cost and schedule performance data were significantly out of date at  some time during our review for 8 of the 33 major projects we reviewed and  20 smaller projects in PARS that had passed Critical Decision 2. In these  instances, data were out of date because DOE has not effectively enforced  requirements that contractors produce updated monthly cost and schedule  performance data, and that project directors ensure current performance  data are reported into PARS. For some projects, the lack of up-to-date data  masked problems that resulted in cost overruns and schedule slippages.  For instance:  The September 2003 PARS report showed that the Spent Nuclear Fuels  project at Hanford, Washington, was on track to meet its DOE-approved  total project cost of about $1.6 billion and its schedule completion date  of 2007; however, these data were 3 months out of date. Subsequently,  the April 2004 PARS report (1) showed that total project costs had  exceeded the project’s cost baseline by nearly $150 million and (2)  indicated that the project would exceed this revised total cost and the  scheduled completion date would slip. In June 2004, the contractor  requested additional funding from DOE because both cost and schedule  performance continued to worsen.
The September 2003 PARS report showed that the K25/27 Buildings  Deactivation and Decommissioning Removal project at Oak Ridge,  Tennessee, was on track to meet its DOE-approved total project cost of  about $265 million and its schedule completion date of 2008. However,  the contractor did not update the project’s performance data until April  2004, when the PARS report showed the project would still meet its cost  baseline. Environmental Management officials told us that although they  knew for several months that the K25/27 project’s total cost would  exceed its baseline, the PARS cost data were not updated because the  project was being combined with 5 other Oak Ridge projects. The total  cost of the K25/27 project could exceed $400 million—more than 50  percent above the DOE-approved total project cost.
In June 2004, the Soil and Water Remediation project at Pantex, Texas,  had a DOE-approved total project cost of about $175 million, but the  Office of Engineering and Construction Management could not assess  the project’s performance because data were not provided.  Subsequently, the September 2004 PARS report showed that the project  was at risk of exceeding its DOE-approved schedule target.
In addition to these timeliness problems, the monthly data in PARS reports  typically lag a project’s actual performance by 2 to 3 months because of the  time contractors need to generate the data and the time DOE project  managers need to review and incorporate the summary data into the PARS  database. The 2004 National Research Council report stated that the lack of  timely data prevents senior managers from using PARS to assess the  performance of projects in real time. Similarly, Department of Defense  officials familiar with project management have said that using such data to  assess project performance is like “overseeing by looking through a rear  view mirror” because performance problems have usually gotten worse by  the time departmental managers become aware of them.
We found that the Department of Defense requires all of its newer contracts  to use electronic data interchange to provide more timely information to  department program managers. In addition, some acting project directors  told us that electronically linking PARS to contractors’ project management  systems would improve timeliness because manually entering cost and  schedule data into the PARS database had often resulted in delays of 2 to 3  months to complete the process. In some instances, data were entered  incorrectly, although in each instance the data were corrected before being  reported to senior managers. While the DOE project directors we  contacted uniformly agree that manually entered data are correctly entered  by the time PARS monthly reports are delivered to senior managers,  electronically linking PARS to contractor systems could eliminate the  potential for such errors and enhance senior managers’ ability to address  potential cost or schedule challenges in real time. Alternatively, DOE might  include a provision requiring timely monthly reporting in all applicable  contracts.
When data can be relied upon, DOE senior managers have taken corrective  actions to address cost or schedule challenges while minimizing costs to  the government. For example, NNSA terminated the Sandia Underground  Reactor Facility project, which was intended to reduce the future  operational costs associated with securing a reactor, when management  learned that cost estimates had increased by more than 150 percent  between project conception and the final design phase. The project was  terminated before costs were incurred. In another instance, Environmental  Management approved a contractor’s recovery plan to complete the Melton  Valley Closure project at Oak Ridge, Tennessee, whose schedule  performance had slipped dramatically and required corrective actions. The  contractor lengthened work hours and modified its approach for  constructing a subproject. As a result, the recovery plan showed that the  scope of work could be accomplished without increasing project schedule.
section_title: Conclusions
paragraphs: Since 1990, we have designated DOE’s contract management, which we  have broadly defined to include contract administration and project  management, as a high-risk area for fraud, waste, abuse, and  mismanagement. Although DOE has implemented important contract  administration and project management reforms, problems persist and  many major projects continue to experience millions of dollars in cost  overruns and years of delays. Two deficiencies—the lack of contracting  criteria for major projects and the lack of reviews of the project  management terms in major project contracts—have resulted in  questionable DOE contracting decisions that limit its ability to effectively  control cost and schedule performance. For example, many of DOE’s  contracts for major projects have used performance incentives that have  used a technical, schedule, or performance incentive without an associated  cost incentive or cost constraint, thereby giving contractors an incentive to  pay limited attention to costs when working toward meeting technical or  performance levels in order to earn a higher award fee.
Furthermore, for major projects, DOE has given insufficient emphasis to  the oversight of contract administration, which begins after contracts are  awarded and helps ensure that the department gets what it pays for. DOE  needs to give increased emphasis to reviewing how it administers  contracts; correcting previously identified weaknesses, such as  overreliance on contractor data; and providing training to its contracting  officers. Without such actions, the department is totally dependent on its  contractors’ self-reports on their performance.
Because of problems with the accuracy, completeness, and timeliness of  the PARS data, senior DOE managers lack key project performance  information for assessing the progress of many major projects and making  decisions about corrective actions. In particular, because DOE has  assessed the reliability of only three contractors’ project management  systems that feed data into PARS, senior managers cannot be certain that  the contractor systems are producing reliable data. Such data are critical to  good project management and affect DOE’s assessment of contractor  performance. Absent reliable data from the contractor systems, DOE lacks  assurance that the fees it awards for a contractor’s project management  actions are well deserved.
section_title: Recommendations for Executive Action
paragraphs: To ensure the use of effective performance incentives for major projects,  we recommend that the Secretary of Energy direct the Associate Deputy  Secretary with responsibility for contract and project management to take  the following two actions:  develop a major projects chapter in the DOE Acquisition Guide that  specifies a systematic contracting approach, including, for example,  criteria for (1) ensuring that incentive fee awards are based on reliable  performance data, (2) using appropriate cost and schedule incentives,  (3) better linking fee awards to performance for major projects that are  part of larger site cleanups, and (4) determining which indirect work- related activities should and should not be considered in awarding  contractors’ fees, and  clarify roles and responsibilities for reviewing contracts prior to award  to ensure project management consistency.
To strengthen departmental oversight of contract administration for major  projects, we recommend that the Secretary of Energy direct the Associate  Deputy Secretary with responsibility for contract and project management  to take the following three actions:  conduct comprehensive self-assessments of contract administration at  least every 3 years,  identify corrective actions to reduce the overreliance on unvalidated  contractor data in awarding contract fees that was identified in previous  self-assessments, and  train contracting officials in earned value management.
To improve the reliability and usefulness of project performance data in  PARS, we recommend that the Secretary of Energy direct the appropriate  managers to take the following seven actions:  develop a schedule for assessing the reliability of the contractors’  project management systems, giving priority to major projects and those  projects with systems believed to be using incorrect methods to  generate PARS data;  revise DOE manual 413.3-1 to provide guidance that enhances the  accurate reporting of total cost and project performance data into PARS,  such as the reporting of life-of-project cost and schedule variances;  expedite training for major project directors in earned value   ensure that program office officials receive currently available project  management training so that they can better identify the elements of a  project, and apply the project management concepts necessary for them  to report performance data in PARS;  incorporate forward-looking trend data into PARS reports so that senior  managers can better identify negative trends and potentially take  corrective action;  explore options for ensuring that contractors provide cost and schedule  performance data to PARS on a monthly basis, such as making monthly  submissions a requirement in all applicable contracts; and  explore options for providing senior DOE managers with more timely  project performance data by, for example, electronically linking  contractors’ project management systems to PARS.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided DOE with a draft of this report for its review and comment. In  written comments, DOE generally concurred with our recommendations  but provided clarifying comments on four of the recommendations. (See  app. III.) First, concerning our recommendation that DOE develop a major  projects chapter in its Acquisition Guide, DOE stated that the department  has already developed an extensive body of material that constitutes a  “systematic contracting approach” for the acquisition and management of  departmental major projects, but added that the department will develop  an overview and summary of this information in a major projects chapter in  its Acquisition Guide. We believe this chapter will further enhance DOE’s  guidance, particularly if the department provides criteria that address each  of the four issues identified in our first recommendation. Second,  concerning our recommendation on DOE’s comprehensive assessment of  contract administration, DOE stated that the department did not stop  conducting comprehensive assessments. In response, we have revised our  recommendation to state that DOE should conduct these assessments at  least every 3 years. Third, concerning our recommendation that DOE  identify corrective actions for reducing overreliance on unvalidated  contractor data, DOE stated that the department had already taken positive  steps to reduce its overreliance on contractor data by, for example,  reviewing and validating such data and project baselines. DOE added that  the department would continue to identify any corrective actions necessary  to reduce overreliance on contractors’ data in awarding fees. While we  agree that validating project baselines is an important first step, we believe  that DOE’s efforts to ensure that contractor performance data are reliable  by certifying contractors’ project management systems is vital. Fourth,  concerning our recommendation that DOE link PARS and contractors’  project management systems, DOE stated that our recommendation is too  narrowly focused, particularly in light of DOE’s efforts to implement a  departmentwide enterprise architecture solution. We agree, and we have  revised our recommendation accordingly. In addition, DOE stated that it  believes the draft report contained a number of inaccuracies and provided  detailed comments. We have revised the report, where appropriate, in  response to these comments.
As arranged with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after the  date of this letter. At that time, we will send copies to the Secretary of  Energy and other interested parties. We will also make copies available to  others upon request. In addition, the report will be available at no charge  on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions on this report, please contact me at  (202) 512-3841. Key contributors to this report were Richard Cheston,  Robert Baney, Nathan Anderson, Bernice Dawson, Cynthia Norris, Judy  Pagano, and Doreen Feldman.
section_title: The Department of Energy’s 33 Major Projects That We Reviewed
paragraphs: Cost incentive or  cost constraint  for individual  project in  contract?
Schedule  performance  incentive for  individual  project in  contract?
Fee available for  individual  project?
Non-Nuclear Facility Decontamination and  Decommissioning  (OH-FN-0050)
Solid Waste Stabilization & Disposition  (OH-FN-0013) Cost plus incentive  Hanford Reservation:  River Protection  Hanford Tank Waste Treatment and Immobilization  Plant  (01-D-416)
Radioactive Liquid Tank Waste Stabilization &  Disposition  (ORP-0014)
Interim Tank Retrieval System  (94-D-407)
Tank Farm Restoration and Safe Operations  (97-D-402)
Hanford Reservation at Richland, Washington Nuclear Facility Decontamination and  Decommissioning—Fast Flux Test Facility Project  (RL-0042)
Nuclear Facility Decontamination and  Decommissioning—Remainder of Hanford  (RL-0040)
Nuclear Facility Decontamination and  Decommissioning—River Corridor Closure Project   (RL-0041)
Nuclear Material Stabilization & Disposition – PFP   (RL-0011)
Soil and Water Remediation – Vadose Zone (RL-0030) Cost plus award  Solid Waste Stabilization & Disposition—200 Area  (RL-0013)
Cost incentive or  cost constraint  for individual  project in  contract?
Schedule  performance  incentive for  individual  project in  contract?
Fee available for  individual  project?
Idaho National Engineering and Environmental Laboratory Advanced Mixed Waste Treatment Facility  (97-PVT-2) Fixed price Spent Nuclear Fuel Dry Storage  (98-PVT-2)
East Tennessee Technology Park Three-Building D&D  and Recycle Project  (OR-0040)
Facilities Capability Assurance Program  (88-D-122-27  & 88-D-122-42)
Rocky Flats Facility at Denver, Colorado Nuclear Facility D&D/North Side Facility Closures  (RF-0040)
Nuclear Facility D&D/South Site Facility Closures  (RF-0041)
Nuclear Material Stabilization & Disposition  (RF-0011) Cost plus incentive  Solid Waste Stabilization & Disposition  (RF-0013)
Microsystems and Engineering Science Application   (01-D-108)
High-Level Waste Removal from Filled Waste Tanks   (SR-0014C)
Cost incentive or  cost constraint  for individual  project in  contract?
Schedule  performance  incentive for  individual  project in  contract?
Fee available for  individual  project?
Although the September 2004 PARS report showed that this project would cost less than $400 million,  we included it in our review because it was included in our 2002 review of DOE’s major projects. (GAO,  Contract Reform:  DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved  Results, GAO-02-798 (Washington, D.C.:  Sept. 13, 2002).)
This project was designated as a major project when DOE’s threshold was $100 million.
NNSA stated that each of these life extension projects involved multiple management and operating  contractors (not 1 contract) in multiple locations, which is different from every other project that is listed  in this appendix.
section_title: Scope and Methodology
paragraphs: Our review focused primarily on 33 major projects that had passed, as of  March 2004, the Department of Energy’s (DOE) Critical Decision 2  milestone—the point at which the department approves a project’s cost,  schedule, and scope baselines on the basis of an approved conceptual  design report and acquisition strategy. The projects we reviewed include 28  projects that cost more than $400 million each and 5 projects that our 2002  assessment defined as major projects because their total costs exceeded  $100 million each. Our review did not include 46 major projects that, as of  March 2004, had not passed the Critical Decision 2 milestone. Since March  2004, at least 6 major projects have passed the Critical Decision 2 milestone  and now have approved baselines. The remaining major projects do not  have approved baselines for measuring performance.
To assess DOE’s use of performance incentives in contracts to effectively  control cost and maintain schedules, we reviewed relevant requirements in  the Federal Acquisition Regulation (FAR) and the DOE Acquisition  Regulation, as well as DOE Order 413.3, DOE manual 413.3-1, and DOE’s  Acquisition Guide, to obtain information on the factors that should be used  in determining a contractor’s fee. Through this effort, we identified whether  the department provided guidance on the appropriate circumstances for  using each contract type and the appropriate factors for determining a  contractor’s fee. In particular, we examined requirements regarding  contract provisions for award fees; cost, schedule, and performance  incentives; and fee determination plans.
We then compared government and departmental requirements with  project-specific elements found in the contracts for each of the 33 major  projects that have DOE-approved cost, schedule, and scope baselines to  determine whether DOE has used appropriate (1) types of performance  incentives, such as cost or schedule incentives, and (2) fee determination  plans and fee payments. For instance, to assess whether DOE’s contracts  used the appropriate incentives for each of three types of contracts, we  compared the types of incentives that DOE’s contracts and relevant  modifications used for each of the 33 major projects with the types of  incentives that the FAR and the DOE Acquisition Regulation, as well as  departmental orders and guidance, require. We then interviewed cognizant  DOE officials to discuss reasons for the inconsistencies we found. In  addition, we examined various contract-related documents associated with  the 33 major projects we reviewed, such as the “Gold Chart” metrics that  Environmental Management uses to measure its progress in DOE’s annual  budget submission to the Congress. Specifically, we compared the Gold  Chart’s performance metrics for each of Environmental Management’s 25  major projects with the performance measures in each projects’ contract.  Where differences were identified, we discussed the contents of the Gold  Chart and the associated projects’ contract with appropriate DOE  contracting officials. Furthermore, we interviewed officials in DOE’s Office  of Contract Management and officials in the Office of Engineering and  Construction Management to determine the extent to which DOE had  reviewed, prior to award, the contracts for the 33 major projects to ensure  that they included appropriate project management provisions.
To assess the reliability of the data DOE uses to monitor and assess  contractor performance, we reviewed the Office of Management and  Budget’s (OMB) directives, DOE’s Reference Book for Contract  Administrators, and other DOE documents and studies to identify relevant  requirements and departmental guidance. We identified the roles and  responsibilities of contract administration officials and examined the  extent to which these officials adhered to their responsibilities. More  specifically, we reviewed the department’s recent contract administration  self-assessments and the frequency with which they were conducted. In so  doing, we examined the department’s recommendations for improving  contract administration and determined whether the recommendations  were followed. If they were not followed, we discussed the reasons with  cognizant officials in the Contract Administration Division. We also  examined DOE’s order for acquisition career development, and other  related DOE directives, to assess training requirements for DOE’s  contracting officers and contracting officer representatives, particularly  regarding training in earned value management principles.
To determine the reliability of Project Analysis and Reporting System  (PARS) data used by senior managers for project oversight, we assessed  the accuracy, completeness, and timeliness of PARS data. To assess the  accuracy of the project performance data in PARS, we did the following:  Reviewed DOE Order 413.3, “Program and Project Management for the  Acquisition of Capital Assets,” and its implementing guidance; OMB  Circular A-11, part 7, “Planning, Budgeting, Acquisition, and  Management of Capital Assets”; and various documents outlining the  requirements in American National Standards Institute/Electronic  Industries Association-748-1998, which defines the requirements for  earned value management—the component of contractors’ project  management systems critical for producing reliable project  performance data.
Interviewed cognizant DOE officials in the Office of Engineering and  Construction Management, the Office of Environmental Management,  the Office of Science, and the National Nuclear Security Administration  on the extent to which the performance data that DOE contractors’  project management systems produced for PARS met earned value  management requirements. These officials included a DOE expert in  earned value management, who is responsible for assessing the  accuracy of the data that various projects’ systems produce. Where  specific deficiencies in a contractor’s project management system were  identified, we obtained relevant documents from the appropriate acting  DOE project director and analyzed whether the contractor generated  project performance data in accordance with the industry standard. We  also interviewed officials in two other major contracting agencies—the  Department of Defense and the National Aeronautics and Space  Administration—about their experience in implementing earned value  management requirements.
Compared data in monthly PARS reports provided to senior DOE  managers from January through September 2004 with project-specific  cost and schedule data obtained from earlier PARS reports, cognizant  program offices, project status reports, Inspector General reports, and  external reviews. When we identified total cost or project performance  data discrepancies between PARS and these other sources, we  contacted relevant project officials to determine their cause.
Identified the extent to which contractor-generated data in PARS were  sufficiently reviewed and verified by DOE by (1) identifying  requirements in DOE Order 413.3 and its implementing guidance for the  departmental review and verification of contractor project performance  data and (2) interviewing DOE project management officials to  determine whether the current breadth of review was adequate and  what plans, if any, DOE had for increasing the rigor of its review and  verification of contractor data.
To assess the completeness of PARS data, we determined whether the  PARS database included major DOE activities—those costing more than  $400 million or that DOE management had designated—identified in our  prior reports, Inspector General reports, DOE press releases, and printouts  from DOE’s Management Accounting and Reporting System. For projects  that were not included in PARS, we contacted headquarters project  management officials to determine if the projects met the criteria for PARS  reporting. For projects that were included in PARS, we examined the  completeness of reported data in various data fields by reviewing printouts  from the PARS database and by reviewing the reports of the National  Academies’ National Research Council and the Civil Engineering Research  Foundation, which also examined the completeness of PARS data. In  addition, we reviewed a 2004 report by the U.S. Army Corps of Engineers  on a major project at Hanford, Washington. Moreover, we discussed  options with DOE officials for reporting additional data that would improve  PARS’ ability to enable senior DOE managers to identify potential cost or  schedule challenges.
To assess the timeliness of PARS data, we reviewed PARS monthly reports  to senior DOE managers and identified projects whose performance data  were out-of-date. For many of these projects, we talked to headquarters  and project officials to determine the reasons for delay and explored  options with them on how timeliness could be improved. We also  interviewed numerous acting DOE project directors to learn how data from  their project management systems were summarized and incorporated into  the PARS database. In addition, we explored options with DOE  headquarters and project officials for improving the timeliness of all data  reported in PARS.
Given our review of the documentation provided by DOE and our  discussions with DOE officials, we have reservations about the reliability  of PARS data. These issues are discussed in this report.
We conducted our work from January 2004 through January 2005 in  accordance with generally accepted government auditing standards, which  included an assessment of data reliability and internal controls.
section_title: Related GAO Products
paragraphs: Nuclear Waste:  Absence of Key Management Reforms on Hanford’s  Cleanup Project Adds to Challenges of Achieving Cost and Schedule  Goals. GAO-04-611. Washington, D.C.:  June 9, 2004.
Nuclear Waste Cleanup:  DOE Has Made Some Progress in Cleaning Up  the Paducah Site, but Challenges Remain. GAO-04-457. Washington, D.C.:   April 1, 2004.
Nuclear Weapons:  Opportunities Exist to Improve the Budgeting, Cost  Accounting, and Management Associated with the Stockpile Life  Extension Program. GAO-03-583. Washington, D.C.:  July 28, 2003.
Nuclear Waste:  Challenges and Savings Opportunities in DOE’s High- Level Waste Cleanup Program. GAO-03-930T. Washington, D.C.:  July 17,  2003.
Nuclear Waste:  Challenges to Achieving Potential Savings in DOE’s  High-Level Waste Cleanup Program. GAO-03-593. Washington, D.C.:    June 17, 2003.
Department of Energy:  Status of Contract and Project Management  Reforms. GAO-03-570T. Washington, D.C.:  March 20, 2003.
Major Management Challenges and Program Risks:  Department of  Energy. GAO-03-100. Washington, D.C.:  January 1, 2003.
High-Risk Series:  An Update. GAO-03-119. Washington, D.C.:  January 1,  2003.
Contract Reform:  DOE Has Made Progress, but Actions Needed to Ensure  Initiatives Have Improved Results. GAO-02-798. Washington, D.C.:   September 13, 2002.
Nuclear Waste:  Technical, Schedule, and Cost Uncertainties of the Yucca  Mountain Repository Project. GAO-02-191. Washington, D.C.:    December 21, 2001.
Department of Energy:  Follow-Up Review of the National Ignition  Facility. GAO-01-677R. Washington, D.C.:  June 1, 2001.
Nuclear Cleanup:  Progress Made at Rocky Flats, but Closure by 2006 Is  Unlikely, and Costs May Increase. GAO-01-284. Washington, D.C.:   February 28, 2001.
High-Risk Series:  An Update. GAO-01-263. Washington, D.C.:  January 1,  2001.
Major Management Challenges and Program Risks:  Department of  Energy. GAO-01-246. Washington, D.C.:  January 1, 2001.