Department of Energy: Better Information Needed to Determine If Nonmajor Projects Meet Performance Targets
Published: 2012-12-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As of February 2011, EM and NNSA remained on GAO's high-risk list for contracting and project management. These two offices manage numerous construction and cleanup projects that each cost less than $750 million and are called nonmajor projects. DOE requires its program offices to establish performance targets for the expected scope, cost, and completion date of each project before starting construction or cleanup. GAO has encouraged federal agencies to use strategic workforce planning to help them meet present and future mission requirements. Two key elements of workforce planning are to identify mission-critical occupations and skills and any current and future shortfalls in these areas. GAO was asked to examine the (1) extent to which EM and NNSA nonmajor projects have met their scope, cost, and completion date targets, (2) factors affecting EM's and NNSA's management of nonmajor projects, and (3) extent to which EM's workforce plans identify mission-critical occupations and skills and any current and future shortfalls in these areas. GAO reviewed DOE documents and project data, examined EM workforce plans, toured selected DOE facilities, and interviewed DOE officials.
section_title: What GAO Found
paragraphs: Of the 71 nonmajor projects that the Department of Energy's (DOE) Office of Environmental Management (EM) and National Nuclear Security Administration (NNSA) completed or had under way from fiscal years 2008 to 2012, 21 met or are expected to meet their performance targets for scope, cost, and completion date. These projects included a $22 million EM project to expand an existing waste disposal facility at the Oak Ridge Reservation in Tennessee and a $199 million NNSA project to equip a radiological laboratory and office building at the Los Alamos National Laboratory in New Mexico. Another 23 projects did not meet or were not expected to meet one or more of their three performance targets for scope, cost, and completion date. Among these, 13 projects met or are expected to meet two targets, including a $548 million NNSA project to shut down a nuclear reactor in Russia for nonproliferation purposes; 8 projects met or are expected to meet one target; 1 project did not meet any of its targets; and 1 project was cancelled. Of the remaining 27 projects, many had insufficiently documented performance targets for scope, cost, or completion date, which prevented GAO from determining whether they met their performance targets. EM and NNSA often did not follow DOE requirements for documenting these performance targets, making it more difficult for GAO and DOE to independently assess project performance.
Several factors affected EM's and NNSA's management of their nonmajor projects that were completed or ongoing from fiscal years 2008 to 2012. These factors included the suitability of a project's acquisition strategy, contractor performance, and adherence to project management requirements. For example, EM officials managing an ongoing project to remediate soil and water at the Idaho National Laboratory used an acquisition strategy that tied incentives for the contractor to different performance milestones across the multiple subprojects within the contract, which will help the project meet its performance goals, according to EM officials. In contrast, NNSA encountered problems meeting its performance goals for a project to build an office building and radiological laboratory at the Los Alamos National Laboratory partly due to its acquisition strategy. According to NNSA project officials at the Los Alamos site office, the project team should have hired one contractor to design the project and solicited bids from other contractors to build the project rather than using the same contractor for both activities. The former strategy might have resulted in a more mature project design and more time to evaluate various contractors' qualifications to construct the project, according to the NNSA project officials.
EM's workforce plans do not consistently identify mission-critical occupations and skills and current and future shortfalls in these areas for its federal workforce. In addition, many EM workforce plans indicate that EM may soon face shortfalls in a number of important areas, including project and contract management. EM officials said that they recognize these issues and have taken a number of steps to address them, including conducting a skills assessment to identify key occupational series to target for succession planning. However, the inconsistent terms used to describe mission-critical occupations and skills in EM's workforce plans make it difficult for GAO and DOE to understand EM's most critical needs regarding its workforce. GAO recommends that EM and NNSA clearly define, document, and track the scope, cost, and completion date targets for each of their nonmajor projects and that EM clearly identify critical occupations and skills in its workforce plans. EM and NNSA agreed with GAO's recommendations.
section_title: What GAO Recommends
paragraphs: GAO recommends that EM and NNSA clearly define, document, and track the scope, cost, and completion date targets for each of their nonmajor projects and that EM clearly identify critical occupations and skills in its workforce plans. EM and NNSA agreed with GAOs recommendations.

section_title: Letter
paragraphs: Since 1990, we have reported that the Department of Energy (DOE) has  suffered from substantial and continual weaknesses in effectively  overseeing contractors and managing large, expensive, and technically  complex projects. As of February 2011, DOE’s two largest program  offices—the Office of Environmental Management (EM) and the National  Nuclear Security Administration (NNSA)—remained on our list of areas at  high risk of fraud, waste, abuse, and mismanagement for contracting and  project management. Our recent reviews of these offices focused on  major system acquisitions (or major projects), which include construction  and environmental cleanup projects that each cost $750 million or more.
However, these two offices also manage numerous projects that are  smaller, less expensive, and less technically complex, called nonmajor  projects, that each cost less than $750 million.
EM and NNSA must manage their projects, including nonmajor projects,  in accordance with DOE’s order on project management, Order 413.3.  The order defines DOE’s project management principles and process,  which consist of a series of management reviews and approvals, called  “critical decisions,” that are required to move a project forward from  project planning and design to implementation. One important step in this  process is the establishment and approval of a project’s performance  baseline, which represents DOE’s commitment to complete a project with  a specific scope of work at a certain cost and by a specific date. The  performance baseline consists of three interrelated targets—scope, cost,  and completion date. A project’s scope defines its technical goals and  includes targets for specific technical requirements that the project is to  deliver at completion. For example, for a construction project, the scope  defines in general terms what facilities and equipment will be purchased  or upgraded, as well as the project’s minimum capability to perform the  desired function at completion. DOE program offices are to establish and  approve a project’s performance baseline before work can start.
In our 2011 high-risk update, we noted that one of the top 10 issues  facing DOE is having sufficient people and other resources to resolve its  contract and project management problems. Specifically, we reported that, according to the department’s analysis, DOE lacks an adequate  number of federal contracting and project management personnel with  the appropriate skills (such as cost estimating, risk management, and  technical expertise) to plan, direct, and oversee the completion of a  project. To help ensure that federal agencies have the personnel with the  appropriate skills they need now and in the future, we and other  organizations have encouraged federal agencies to use strategic  workforce planning. In particular, in its Human Capital Assessment and  Accountability Framework, the Office of Personnel Management  specified, among its key elements of workforce planning, the need to  develop workforce plans that identify (1) “mission-critical” occupations  and skills that are essential to achieving an agency’s strategic goals and  (2) current and future potential shortfalls in these areas. In April 2012, we  reported on issues related to NNSA’s workforce planning efforts.  Specifically, we reported that NNSA and its site contractors face  shortages in qualified critically skilled candidates and an aging workforce.  NNSA and its site contractors told us that they are engaged in workforce  planning to avoid potential critical skill gaps, but NNSA did not expect to  complete NNSA-wide workforce plans until 2013.
In this context, you asked us to look at DOE’s management of nonmajor  projects. Our report objectives were to examine the (1) extent to which  EM and NNSA nonmajor projects that were completed or ongoing for  fiscal years 2008 to 2012 met their scope, cost, and completion date  targets; (2) factors affecting EM’s and NNSA’s management of nonmajor  projects during this period; and (3) extent to which EM’s workforce plans  identify mission-critical occupations and skills, as well as any current and  future shortfalls in these areas.
To determine the extent to which EM and NNSA nonmajor projects met  their scope, cost, and completion date targets, we reviewed project  performance information for 30 EM and 41 NNSA nonmajor projects that  were either completed or ongoing from fiscal year 2008 to fiscal year  2011 and for which EM and NNSA had established performance targets.  We also collected performance information for ongoing projects for fiscal  year 2012. The total estimated cost of these 71 projects is approximately  $10.1 billion. We excluded (1) EM projects funded by the American  Recovery and Reinvestment Act of 2009 because of a separate review  looking at these projects, (2) information technology projects, and (3)  operational activities. We relied on DOE Order 413.3 for requirements on  how scope, cost, and completion date targets are to be specified in a  project’s performance baseline. In assessing whether projects had  achieved their scope, cost, and completion date targets, we followed  Office of Management and Budget guidance and DOE performance  metrics. Taken together, these sources state that a project achieves a  satisfactory performance if it completes its scope target at less than 10  percent above its cost target and at less than 10 percent past its  completion date target. To evaluate factors affecting EM’s and NNSA’s  management of nonmajor projects, we conducted structured interviews  with EM and NNSA project officials for a subset of the 71 projects: a  nonprobability sample of 10 EM projects and 10 NNSA projects that  included completed and ongoing projects, with a wide range of project  costs. While we cannot generalize from these interviews to all EM and  NNSA projects, we chose these projects to provide examples of factors  that affected EM’s and NNSA’s management of these projects. The  interviews focused on aspects of project management, such as the  preparation of project designs, risk assessments, and cost and schedule  baselines, as well as adherence to DOE project management  requirements. As part of these interviews, we conducted tours of two  DOE facilities near Oak Ridge, Tennessee. To evaluate the extent to  which EM’s workforce plans identify mission-critical occupations and skills  and any current and future shortfalls in these areas, we examined EM’s  strategic workforce plans for its headquarters and site office staff, DOE’s  corrective action plan for contract and project management, and the  Office of Personnel Management’s Human Capital Assessment and  Accountability Framework. We also interviewed DOE and EM officials  with knowledge of EM’s practices in workforce planning, including officials  in EM’s Office of Acquisition and Project Management and its Office of  Human Capital and Corporate Services (Office of Human Capital).  Appendix I presents a more detailed description of our scope and  methodology.
We conducted this performance audit from June 2011 to December 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NNSA—a separately organized agency within DOE—has primary  responsibility for ensuring the safety, security, and reliability of the  nation’s nuclear weapons stockpile. NNSA carries out these activities at  eight government-owned, contractor-operated sites: three national  laboratories, four production plants, and one test site (see fig. 1). These  sites, taken together, have been a significant component of U.S. national  security since the 1940s. Contractors operate these sites under  management and operations (M&O) contracts.the contractor with broad discretion in carrying out the mission of the  particular contract, but grant the government the option to become much  more directly involved in day-to-day M&O. Currently, NNSA’s workforce is  made up of about 34,000 M&O contractor employees across the eight  sites, and about 2,400 federal employees directly employed by NNSA in  its Washington headquarters, at offices located at each of the eight sites,  and at its Albuquerque, New Mexico, complex.
section_title: Some Projects Have Met or Are Expected to Meet Their Performance Targets, but EM and NNSA Did Not Clearly Document Information Needed to Determine the Performance of Others
paragraphs: Of the 71 EM and NNSA nonmajor projects we reviewed that were  completed or ongoing for fiscal years 2008 to 2012, we were able to  determine performance for 44 projects. Among these 44 projects, 21 have  met or are expected to meet all three of their performance targets for the  scope of work delivered, cost, and completion date, while 23 have not  met or are not expected to meet one or more of their three targets. The  remaining 27 of the 71 projects we reviewed had insufficiently  documented performance targets or had modified scope targets, among  other things, which prevented us from determining whether they met or  were expected to meet their performance targets, according to our  analysis of DOE data. Determining whether projects fully met or partially  met performance targets was difficult because EM and NNSA did not  always follow DOE requirements for documenting these targets. DOE has  taken steps to ensure that EM and NNSA more clearly document  performance targets for their projects, but some problems persist.
subsections: section_title: EM and NNSA Have Met or Are Expected to Meet All Performance Targets for Some Nonmajor Projects
paragraphs: Of the 71 nonmajor projects we reviewed, 44 projects—17 EM projects  and 27 NNSA projects—had documented targets for scope, cost, and  completion date, enabling us to determine their performance. Table 2  shows the expected or completed performance of these 44 EM and  NNSA nonmajor projects.
As the table shows, of the 44 projects for which we were able to  determine performance, 21 projects met or are expected to meet their  performance targets for scope, cost, and completion date. Specifically, 17  completed projects—5 EM and 12 NNSA projects—met all three of their  performance targets. These projects included a $22 million EM project to  expand an existing waste disposal facility at the Oak Ridge Reservation in  Tennessee and a $469 million NNSA project to construct chemical,  electrical, and other laboratories and workspaces at the Sandia National  Laboratories in New Mexico. In addition, as of August 29, 2012, 4  ongoing projects—1 EM and 3 NNSA projects—were expected to meet all  three of their performance targets, according to DOE estimates. These  projects included a $77 million EM project to construct two disposal units  for storing waste at the Savannah River Site in South Carolina and a  $199 million NNSA project to equip the Radiological Laboratory/Utility/  Office Building at the Los Alamos National Laboratory in New Mexico to  make it suitable for performing programmatic work.
Table 2 also shows that 23 EM and NNSA projects did not meet or are  not expected to meet one or more of their three performance targets. Of  these 23 projects, 13 projects met or are expected to meet two of their  three performance targets, and eight met or are expected to meet one of  their three performance targets (see apps. II and III for more details). In  addition, one project did not meet any of its performance targets.  Specifically, EM’s project to decontaminate and decommission the Main  Plant Process Building in West Valley, New York, did not complete all of  its planned scope of work when it was completed in October 2011, almost  4 months after its completion date target and more than $50 million over  its cost target of $46 million. EM cancelled the remaining project— Uranium-233 Disposition project, at the Oak Ridge Reservation in  Tennessee—in December 2011 after spending approximately $225  million. (See apps. II and III for more details.)
In assessing whether projects had achieved their scope, cost, and  completion date targets, we followed DOE and Office of Management and  Budget performance metrics.must be completed within their original scope target. For cost targets,   For the scope, DOE states that projects  Office of Management and Budget guidance and DOE performance  metrics regard projects completed at less than 10 percent above their  original cost targets as having achieved satisfactory performance.  Regarding completion date, DOE’s performance metrics do not address  targets for completion. However, because Office of Management and  Budget guidance includes performance standards for project schedule,  we considered projects to be on time if they were or are expected to be  completed at less than 10 percent past their original completion date  targets.
section_title: Inconsistent Documentation of Scope Targets, Among Other Things, Made Assessment of the 44 Projects’ Performance Difficult
paragraphs: We encountered two major problems in assessing the performance of the  44 projects described above, which made it more difficult for us and DOE  to independently assess project performance. First, EM and NNSA did  not consistently follow DOE requirements for documenting scope targets  and tracking these targets using DOE’s performance database. Second,  EM did not always establish credible completion date targets or conduct  required independent reviews when it restructured its PBSs in 2010.
Establishing a clearly defined target for scope is critical for an agency to  accurately track and assess a project’s overall performance. In particular,  a project’s scope of work directly affects estimates of the project’s cost  and completion date. If the scope target is too broad or vaguely stated, it  can be difficult to track whether or to what extent certain aspects of  project scope were reduced or eliminated between CD-2 (when the  baseline was established) and CD-4 (when the project was completed),  potentially affecting cost and completion date targets. Accordingly, since  2003, DOE Order 413.3 has required that information on scope targets be  documented in a project execution plan as part of CD-2. The current  order also requires a project’s acquisition executive to sign a  memorandum approving CD-2 that contains this information. In addition,  the order requires that information on scope targets, as well as other  critical performance information, be entered into DOE’s centralized  database on project performance—the Project Assessment and  Reporting System (PARS).OAPM, is used to track and report on project performance.
This database, which is administered by  However, EM and NNSA did not always follow the order’s requirement on  documenting scope targets, and we encountered the following problems  when we attempted to identify scope targets for EM and NNSA projects:
Key project documents associated with CD-2 and identified in Order  413.3—project execution plans and CD-2 approval memorandums— often did not contain information on scope targets. Instead, we had to  obtain and review a variety of other project documents to try and  locate this information. For example, we obtained information on  scope targets for several EM and NNSA projects from briefing slides  (i.e., a PowerPoint presentation) prepared for a DOE advisory board  involved in reviewing and approving projects. We also used  independent project review reports, documents describing the  functional and operational requirements of the projects, and contractor  documents that provided detailed descriptions of a project’s scope of  work, among other documents. (See app. I for more details on our  scope and methodology.)
The additional project documents that EM and NNSA provided to  verify scope targets were often dated several months (or more) before  or after the approval of CD-2. Because these documents were often  not contemporaneous with the date when CD-2 was approved, we  had difficulty determining whether any scope targets had changed  during the interval. For example, we obtained information on scope  targets for two EM projects from a project execution plan that was  signed and dated almost 9 months after the CD-2 approval  memorandum had been signed. (If documents were not dated within  1 year of CD-2 approval, we did not consider them sufficient and  reliable for purposes of determining scope targets.)
EM and NNSA often did not clearly or uniformly identify scope targets  in their documents, instead providing this information in a variety of  ways. For example, a NNSA project provided this information in  briefing slides for a project advisory board under the headings  “programmatic requirements summary” and “physical design  summary.” In cases where we were unable to clearly identify scope  targets, we relied on EM and NNSA officials to identify the information  that they considered to represent scope targets.
OAPM has encountered similar problems in trying to identify scope  targets for projects at CD-2 and track how well completed projects had  met these targets at CD-4. Specifically, OAPM officials told us that DOE  program offices may have documented a project’s scope targets in a  variety of project documents, such as a project execution plan or an  acquisition strategy plan, rather than in an approval memorandum at CD- 2. As a result, OAPM officials said they occasionally must reconstruct a  project’s scope targets from other contemporaneous planning documents  near the time of the CD-2 approval. In a few instances, they said that no  audit trail exists to compare scope targets established at CD-2 with the  scope targets cited in a project’s CD-4 approval memorandum. In  addition, an OAPM official told us that OAPM has not completed the  process of locating and entering these data into PARS and continues to  work with EM and NNSA to reconstruct project scope targets near the  time of CD-2 approval.
We found two problems with the performance baselines of many projects  that EM restructured from its portfolio of PBS activities in 2009 and 2010.  First, several projects did not have a credible completion date target. For  example, EM’s April 2010 memorandum approving CD-2 for the Zone 1  Remedial Actions project, located in Oak Ridge, Tennessee, gave a  target completion date of fiscal year 2017, which meant that the target  completion date was at the end of the fiscal year (September 30, 2017),  according to EM officials. However, EM approved the formal completion  of this project, via a CD-4 approval memorandum, on September 30,  2011—6 years ahead of the completion date target identified in the CD-2  approval memorandum. In explaining this difference, EM officials stated  that they had linked the target completion date for this project to the end  of the existing PBS near-term baseline, which had been established  before the restructuring process. According to EM officials, EM used this  method because it already had a contract in place to conduct work  activities associated with the PBS near-term baseline. Since the end of  the PBS near-term baseline was to coincide with the end of the contract  period, EM officials said that they did not think it would be appropriate to  change the target completion dates. In addition, EM officials told us that  they were more focused on finishing the scope of work for a given project  within a specific dollar amount, and that it was not worth the additional  expenditure of time and dollars to modify contracts to change the  completion date. EM used this same methodology in establishing target  completion dates for several other projects we examined.
When we found that EM’s practice of establishing target completion dates  did not provide a meaningful benchmark for assessing project  performance, we had to locate additional documentation to establish a  more credible completion date target. For example, for the Zone 1  Remedial Actions project, we reviewed additional documentation and  decided that a more credible completion date target was December 15,  2011. As a result, for this and other projects, the completion date targets  we used to evaluate the performance of some projects are different than  the ones that DOE uses in its PARS database.
Second, EM often established a new performance baseline and approved  CD-2 for a project without having the baseline reviewed by an  independent team of experts, as DOE Order 413.3 requires. Among other  things, the review team is responsible for examining a project’s cost and  completion date targets to ensure that they are credible and valid.  However, EM did not conduct such reviews when it restructured some of  the projects we examined. Instead, according to EM officials, EM relied  on independent reviews conducted in the 2007 to 2008 time frame as part  of the CD-2 approval process for PBS activities. In addition, EM officials  said that it was not worth the additional expenditure of time and dollars to  conduct new independent reviews for these projects.
If EM had conducted new reviews as part of its restructuring process, it is  possible that these reviews would have uncovered problems with some of  the performance targets that EM had to correct later. Specifically, we  identified two projects—the decontamination and decommissioning of the  Paducah Gaseous Diffusion Plant in Paducah, Kentucky, and the Main  Plant Process Building in West Valley, New York—for which EM had to  significantly increase the cost targets it had approved 10 and 5 months  earlier, respectively. In both cases, these cost increases were due to  errors that EM project officials made in calculating total project cost. For  example, for the Paducah Gaseous Diffusion Plant project, the project  team did not incorporate additional project costs, including funds for  contingencies and the contractor’s fee, into the project’s cost estimate.  This omission resulted in underestimating the project’s cost target by  about $8 million, or about 21 percent. In both cases, these errors  increased the projects’ cost targets and caused EM to miss the original  cost targets, according to our assessment of performance.
section_title: Limited Documentation or Changing Performance Baselines Complicates Evaluation of 27 Projects’ Performance
paragraphs: We were unable to determine the extent to which 27 of the 71 nonmajor  projects that EM and NNSA completed or had under way from fiscal year  2008 through fiscal year 2012 had met their scope, cost, and completion  date targets for four reasons. First, EM and NNSA did not establish a  performance baseline for eight projects. Second, EM and NNSA did not  provide documentation that fully identified one or more performance  targets—including targets for scope, cost, and completion date—for eight  projects. Third, NNSA did not fully document a final project cost or a  current completion date for three projects. Fourth, EM and NNSA  modified the scope targets of eight projects after CD-2, rendering the  original performance targets unusable for purposes of assessing  performance.
EM and NNSA did not establish a performance baseline for a total of 8 of  the 71 nonmajor projects we reviewed that were completed or ongoing for  fiscal years 2008 to 2012. Without a performance baseline, a project’s  performance cannot be assessed. Specifically, we found the following:
EM. EM did not establish a performance baseline for 6 of the 30 EM  nonmajor projects we reviewed. According to EM documentation,  when the office established near-term baselines for its PBS activities  in the 2007 time frame, it decided that it would not establish a  baseline for a few projects that were near completion or for which  physical work was essentially complete, and remaining costs were  low. Because EM had essentially completed all physical work before  fiscal year 2008 on the 6 projects we identified, EM never established  a performance baseline for these projects, according to EM officials.  However, we included these projects in our review because EM did  not formally approve the completion of these projects (via a CD-4  approval memorandum) until the 2010 to 2011 time frame, which  meant that these projects would have been ongoing until that time.  The combined cost of these 6 projects is approximately $1.5 billion.  (See app. II for more details.)
NNSA. NNSA did not establish a performance baseline for 2 of the 41  NNSA nonmajor projects we reviewed. According to NNSA  documents and project officials, after a May 2000 wildfire damaged  lands and buildings at the Los Alamos National Laboratory, NNSA  formally authorized two emergency recovery efforts in July 2000.  Because this authorization was granted outside of the critical decision  process, NNSA did not establish formal performance targets for these  projects. The combined cost of these two projects is $145 million.  (See app. III for more details.)
For 8 of the 71 nonmajor construction projects we reviewed, EM and  NNSA did not fully define and document one or more performance targets  for scope, cost, and completion date when they established and approved  performance baselines for these projects at CD-2. Specifically, we found  the following:
For EM, 2 of the 30 EM nonmajor projects we reviewed did not have  clearly defined and documented targets for scope and completion  date, 1 project did not have a clearly defined and documented scope  target, and 1 project did not have a clearly defined and documented  completion date target. The combined cost of these 4 projects, 2 of  which are ongoing, is estimated to be at least $182 million. (See app.  II for more details.)
For NNSA, 4 of the 41 NNSA nonmajor projects we reviewed did not  have clearly defined and documented scope targets. The combined  cost of these 4 projects, 2 of which are ongoing, is estimated to be  $122 million. (See app. III for more details.)
For 2 of the 71 nonmajor projects we reviewed, NNSA did not fully  document the final project cost at CD-4. The final cost has not yet been  settled for these 2 projects due to pending litigation with the contractor.  NNSA estimated the combined cost of these 2 projects, both of which  have been completed, to be $195 million. In addition, for the Nuclear  Materials Safeguards and Security Upgrades, Phase II project at the Los  Alamos National Laboratory, NNSA and contractor officials have  determined that the project’s remaining construction costs will exceed the  existing funds for the project and have halted work on the project. As a  result, NNSA has not determined what the project’s revised completion  date target will be.
EM and NNSA modified the scope targets of 8 of the 71 projects we  reviewed after approving them at CD-2. EM and NNSA used procedures  to control and approve these modifications but did not establish new CD-2  performance targets. As a result, the scope modifications rendered the  original CD-2 performance targets unusable for assessing project  performance. We consider a project’s scope target to have been modified  if, among other things, EM or NNSA increased the scope of the project  after approving it at CD-2 or reduced the scope for programmatic reasons  and provided a sound justification for this reduction. In contrast, if EM or  NNSA reduced project scope solely to meet a project’s cost target, we did  not consider the scope target to have been modified; rather, we  considered the scope target not to have been met. (See apps. II and III  for more information.)
Projects for which EM or NNSA had modified the scope target sometimes  exceeded expectations. For example, an NNSA project to build a highway  at the Nevada National Security Site had an original scope target of 19.2  miles of highway and a cost target of about $14 million, but the project  team completed an additional 12 miles of highway at an incremental cost  of about $4 million, ahead of the original completion date target. However,  because the scope target changed, the total cost of the project also  changed, which made it unfair to judge the project’s performance against  its original cost target.
Table 3 provides an example of a NNSA project for which we consider the  scope target to have been modified.
We have previously reported on problems with the way DOE documents  and tracks the scope of its projects, and DOE has taken actions to  address this issue. In a 2008 report on DOE’s Office of Science, we noted  concerns within DOE that projects sometimes had overly broad definitions  of scope, making it difficult to determine the effects of a change in project   To address this issue, we recommended that DOE consider  scope.whether it could strengthen its project management guidance to help  ensure that each project’s scope is clearly and sufficiently defined. DOE  generally agreed with our recommendation and revised Order 413.3 in  November 2010 to establish clearer requirements for identifying and  documenting project scope at CDs 2 and 4. Specifically, the revised order  requires a project’s acquisition executive to clearly identify the scope  target in the documentation approving CD-2. In the documentation  approving CD-4, when a project is declared complete, this official must  clearly identify the scope accomplished and compare this scope with the  target established at CD-2.
To determine whether NNSA and EM had improved their documentation  of scope targets since DOE revised Order 413.3, we identified two  nonmajor projects for which NNSA and EM established performance  targets in 2011. These projects are NNSA’s Sanitary Effluent Reclamation  Facility Expansion project at the Los Alamos National Laboratory and  EM’s Purification Area Vault project at the Savannah River Site. In  reviewing project documentation, we found that both NNSA and EM had  provided information on targets for scope, cost, and completion date in  their memorandums approving CD-2. In particular, NNSA’s approval  memorandum identified the following scope targets: (1) expand the  existing Sanitary Effluent Reclamation Facility capacity to treat 300  gallons per minute of product water in an 18-hour day; (2) provide a  400,000-gallon product water storage tank, which provides a consistent  supply of water to the cooling towers in the event the facility is off-line for  maintenance; and (3) provide additional evaporation capacity. These  scope targets provide a quantitative measure of how the project is to  perform at completion, as required by DOE’s order.
However, we found problems with the way EM documented the scope  target for its Purification Area Vault project. EM’s approval memorandum  provided a high-level description of the project’s scope, stating that the  project will modify an existing portion of the K-Area Complex at the  Savannah River Site in South Carolina to accommodate a vault;  implement passive and active fire protection features as identified in the  project fire hazards analysis; and install a new heating, ventilation, and air  conditioning system. However, the scope target cited in the approval  memorandum did not provide sufficient detail for measuring scope  performance at project completion and, therefore, it may be difficult for an  independent reviewer to accurately assess project performance.  Specifically, we found the following:
The first part of the scope target—construct a secure storage location  for holding at least 500 containers—provides a quantitative measure  of how the project is to perform at completion, as required by DOE’s  order. The second part of the scope target—attain CD-4 approval for  storage of containers—does not provide a quantitative measure and  instead reflects a stage in DOE’s critical decision framework.  Therefore, only one part of the scope target can be used to  independently measure project performance regarding scope.
The scope target only captures some of the elements of scope  contained in the high-level scope description. As a result, the effect of  any changes to these other elements of scope on project performance  is unclear. For example, if EM decided not to fully implement the fire  protection features identified in its hazards analysis, it is not clear  whether EM or an independent reviewer would consider the project to  have met its scope target.
Only the scope target—as opposed to the other elements of scope in  the high-level scope description—is currently being tracked in DOE’s  centralized database for project performance. Given the other issues  we identified with this scope target, an independent reviewer relying  solely on information in DOE’s database may not have enough  information to assess the project’s performance accurately.
section_title: Several Factors Affect EM and NNSA in Managing Nonmajor Projects
paragraphs: Several factors affected EM and NNSA in managing their nonmajor  projects that were completed or ongoing from fiscal years 2008 to 2012.  According to our interviews with project officials, these factors included  the suitability of the acquisition strategy, contractor performance, and  adherence to project management requirements.
subsections: section_title: Acquisition Strategy
paragraphs: Because EM and NNSA carry out their work primarily through  agreements with private contractors, a project’s acquisition strategy is a  critical factor that affects the ability of these offices to properly manage  their nonmajor projects. According to DOE guidance, an acquisition  strategy is the high-level business management approach chosen to  achieve project objectives within specified resource constraints. The  acquisition strategy is the framework for planning, organizing, staffing,  controlling, and leading a project. As part of this framework, agency  officials have to choose the most appropriate contract alternative for a  given project. Alternatives can include the use of multiple contractors to  perform different tasks or the use of a prime contractor (such as the M&O  contractor at a DOE site), who would be responsible for awarding  subcontracts for different tasks. In addition, agency officials should  identify the use of special procedures, such as the use of a “design-build”  contract, whereby a single contract is awarded for both design work and  construction, or the use of a “design-bid-build” contract, whereby separate  contracts are awarded for the design and construction.
Some EM and NNSA officials told us that the acquisition strategy was an  important factor in the successful management of their projects. For  example, EM retained a prime contractor to manage the Soil and Water  Remediation–2012 project at the Idaho National Laboratory using a  contract containing incentives based on cost and schedule performance.  According to project officials, the fee structure under this acquisition  strategy is relatively simple and gives the project team flexibility to tie  incentives to different performance milestones across the multiple  subprojects within the contract. These officials said that this contract  structure has been a very effective tool in achieving performance goals.  EM expects this project to meet its cost target of $743 million and its  completion date target of September 2012, and officials said that they  expect the contractor to receive its incentive fee, as called for in the  contract.
Other EM and NNSA officials cited the existence of an inadequate  acquisition strategy as having a negative effect on the performance of  their projects. For example, according to NNSA project officials at the Los  Alamos site office, the M&O contractor at the Los Alamos National  Laboratory decided to construct the Radiological Laboratory/Utility/Office  Building project using a design-build acquisition strategy with a single  prime subcontractor responsible for both design and construction. This  approach was chosen based on the M&O contractor’s experience with  constructing office buildings. However, the project also involved the  construction of a radiological laboratory, which entailed the use of  rigorous documentation standards to show that the project can meet  nuclear quality assurance standards, among other things. Officials of the  NNSA site office said that one of their key lessons learned would be to  use a design-bid-build acquisition strategy if they had to manage a similar  project in the future. The use of a design-bid-build contract would have  offered several advantages over a design-build contract, according to  these officials. First, it would have allowed NNSA staff more time to  develop more robust project specifications and a more mature project  design before having contractors bid on the construction of that design.  Second, NNSA staff might have had more time to evaluate bids from  contractors to see if they had the skills to construct the project. Third, with  a more mature design, NNSA might have been able to reduce the number  of federal staff and the time spent overseeing the project. This project  was completed in June 2010, a few months after its completion date  target. Its cost target was $164 million; however, the final cost of this  project has not been determined because of ongoing litigation. According  to the officials of the site office, NNSA withheld the M&O contractor’s  performance incentive fee as a result of less than desirable contractor  and subcontractor management during the design and construction of the  facility.
DOE has previously identified ineffective acquisition strategies as being  among its top 10 management challenges. Specifically, in its 2008 root  cause analysis, DOE reported that its acquisition strategies and plans  were often ineffective.acquisition planning early enough in the process or devote the time and  resources to do it well.
section_title: Contractor Performance
paragraphs: Because contractors carry out the work associated with EM and NNSA  nonmajor projects, contractor performance is a fundamental factor  affecting EM’s and NNSA’s management of these projects. According to  EM and NNSA project officials, poor contractor performance was a  significant factor impeding their ability to successfully manage nonmajor  projects. Among other things, officials cited concerns with finding qualified  contractors that understood DOE’s nuclear safety requirements and  maintained adequate internal control processes. Examples are as follows:
Nuclear Facility Decontamination & Decommissioning – High Flux  Beam Reactor Project, Brookhaven National Laboratory, New York:  This project was completed in December 2010, more than a year  ahead of its completion date target, and at a cost of $16 million, which  was 31 percent higher than its cost target of $12 million. EM officials  stated that the major factor increasing costs was that the contractor  did not properly prepare for and pass internal safety reviews, which  were necessary to demonstrate the contractor’s readiness to begin  removal and disposal of key reactor components. Because the  contractor required more time than originally planned to prepare for  and pass internal safety reviews, work on the project was delayed,  and the total project cost increased. Officials did not explain why the  project was completed well ahead of its completion date target despite  the delays encountered. Officials stated that one of the most important  lessons learned was to better ensure earlier in the process that the  contractor had a rigorous process in place (e.g., procedures and  training) to demonstrate that their personnel were ready to perform  the decontamination and decommissioning work. Because EM’s  cleanup work at Brookhaven National Laboratory is performed under  the Laboratory Management and Operations Contract under the  purview of DOE’s Office of Science, EM officials said that they have  provided information to the Office of Science to be included in the  contractor’s overall performance evaluation.
Process Research Unit Project, Niskayuna, New York: EM  established a cost target of $79 million for this ongoing project. In  September 2010, a contamination incident occurred while the  contractor was performing open air demolition of a building at the site.  According to DOE’s incident report, the contamination incident had  two root causes: (1) the contractor failed to fully understand,  characterize, and control the radiological hazard; and (2) the  contractor failed to implement a work control process that ensured  facility conditions supported proceeding with the work. As a result of  this incident, as well as weather-related issues, the project has  exceeded its cost target, and the project’s final cost and completion  date depend on the outcome of negotiations between DOE and the  contractor, according to project officials.
Nuclear Materials Safeguards and Security Upgrades Project, Phase  II, Los Alamos National Laboratory, New Mexico: This ongoing project  is expected to meet its cost target of $245 million but not its  completion date target of January 2013. NNSA used a design-bid- build acquisition strategy for this project, with one contractor  responsible for designing the project, and another contractor  responsible for construction activities. According to project officials,  during the construction phase, the building contractor had to stop  work when it discovered errors with the design of the project.  Specifically, officials told us the designs contained an erroneous  elevation drawing that did not adequately account for the presence of  a canyon and a pipeline containing radioactive liquid waste on the  north side of the project site. In addition, other construction  subcontractors, whose work was to be performed in sequence, had to  wait to begin their work. As a result of these problems, the design  contractor spent considerable time redesigning the project, according  to project officials, and NNSA has had to award additional funding and  schedule time to the construction contractors to compensate for the  inadequate design. All told, officials told us the additional costs  resulting from redesign and the delay of construction ranged from $15  million to $20 million. In addition, NNSA and contractor officials  recently determined that the project’s remaining construction costs will  exceed the existing funds for the project and have halted work. As a  result, NNSA has not determined what the project’s revised  completion date target will be.
section_title: Adherence to Project Management Requirements
paragraphs: Effective project management also depends on having project officials  consistently follow DOE’s project management requirements. Among  other things, these requirements are aimed at ensuring that projects       (1) have a sufficiently mature design before establishing performance  targets and beginning construction activities; (2) have had their earned  value management systems certified for more accurate reporting on  performance; (3) undergo a review by an independent group of experts  before beginning construction activities; and (4) maintain an adequate  process to account for any significant changes to the project’s scope,  cost, or completion date targets (known as a change control process).  DOE has previously identified adherence to project management  requirements as among its top 10 management challenges, stating in its  2008 root cause analysis that the agency has not ensured that these  requirements are consistently followed. That is, in some instances,  projects are initiated or carried out without fully complying with the  processes and controls contained in DOE policy and guidance.
We found a similar problem with adherence to DOE project management  requirements in some of the projects we reviewed, although these  problems were more often associated with EM projects than with NNSA  projects. Specifically, in half of the 10 EM projects we reviewed in depth,  officials cited a lack of adherence to project requirements, particularly not  having a sufficiently mature design when establishing performance  targets and beginning work activities, as a significant factor impeding their  ability to manage projects within the performance baseline.
For example, EM’s project to convert depleted uranium hexafluoride into  a more stable chemical form at two locations—Paducah, Kentucky and  Portsmouth, Ohio—was completed in November 2010, more than 2 years  after its completion date target and more than $200 million over its cost  target of $346 million. A lessons-learned report, completed in 2009 at the  request of DOE, concluded that DOE’s critical decision process had  become a “mere rubber stamp of approval.”Project had results consistent with its level of definition at the time of  project commitment and execution start. Future DOE projects will likely  demonstrate similar performance unless they are better defined at the  start of detailed design and they follow not only the letter of DOE’s [critical  decision] process, but also its spirit.” According to EM officials, EM  withheld the construction contractor’s incentive fee due to its poor  performance.
It stated: “In the end, the …  In contrast, among the 10 NNSA projects we reviewed in depth, several  NNSA project managers credited adherence to project management  processes as contributing positively to project performance. The  advantages of adhering to project management processes are illustrated  by one of the projects we reviewed—the Ion Beam Laboratory project at  Sandia National Laboratories, New Mexico. This project—to use ion  beams to qualify electronics and other nonnuclear weapon components  for use in the nuclear stockpile—was completed ahead of schedule in  September 2011 at a cost of $31 million, which was 22 percent lower than  its cost target of $40 million. Project officials stated that implementing a  procedure to control any changes to the performance baseline and a  Baseline Change Control Board served as the foundation to manage all  changes to ensure that cost, schedule, and technical aspects were  evaluated to meet the mission of the project. In addition, project officials  made active use of earned value management data, with several officials  noting that applying earned value management principles on a regular  basis assisted the project team in taking management actions to keep the  project on track. For example, the Sandia Project Manager provided  monthly reports to Sandia senior managers and the federal project  director to communicate the project’s progress, the accomplishment of  milestones, financial outlays, project issues, and appropriate corrective  actions. Owing to the project’s success in meeting its performance  targets, NNSA did not withhold any fee from the contractor.
section_title: EM’s Workforce Plans Do Not Consistently Identify Mission- Critical Occupations and Skills or Shortfalls in These Areas
paragraphs: EM’s eight workforce plans for its federal workforce do not consistently  identify (1) mission-critical occupations and skills and (2) current and  future shortfalls in these areas. As shown in table 4, of the eight EM  workforce plans, one fully identifies both mission-critical occupations and  mission-critical skills; another four plans identify either mission-critical  occupations or mission-critical skills, but not both; and four of the eight  plans identify current and future shortfalls in mission-critical occupations.
EM’s workforce plans may not consistently identify mission-critical  occupations and skills and shortfalls in these areas in part because EM’s  Office of Human Capital has not established a consistent set of terms that  all EM sites use to define and describe mission-critical occupations and  skills, according to our analysis. Instead, the five EM workforce plans that  identified or partially identified these occupations and skills (as shown in  table 4) used different terms to identify them. The plans also differed in  the number and type of occupations or skills identified as mission-critical.  For example, two plans identified three such occupations and skills, while  another identified 20 different job series associated with 40 different  position titles. Table 5 shows the variations in mission-critical occupations  and skills identified in these five EM workforce plans.
When we brought the issue of inconsistent terminology to the attention of  EM officials, they agreed that it would be useful to establish a consistent  set of terms for mission-critical occupations and skills and told us that  they plan to address this issue in the fiscal year 2013 planning cycle.  However, we note that EM’s guidance to its site offices already instructed  them to describe shortfalls and surpluses in the skills most critical to site  performance; nonetheless, not all site offices did so.
Notwithstanding the variations in terms for mission-critical occupations  and skills in EM’s workforce plans, many of the plans indicate that EM’s  federal workforce may soon face shortfalls in a number of important  areas, including project and contract management. Examples are as  follows:
The Portsmouth/Paducah Project Office’s plan states that the office  will need more staffing, including in project management and  contracting, to meet mission needs in future years. Specifically, the  plan notes that 31 percent of its current federal workforce could retire  by fiscal year 2017, including up to 67 percent of its contract  specialists and up to 64 percent of its general engineers.
The workforce plan for EM headquarters, issued in July 2011, stated  that 26 percent of its federal workforce was currently eligible to retire,  with an additional 22 percent of the workforce projected to become  eligible for retirement by fiscal year 2015. The EM headquarters plan  projected that 60 percent of contracting officers would be eligible for  retirement by fiscal year 2015.
The Idaho Operations Office workforce plan states that a significant  number of federal employees in leadership and mission-critical  positions were already eligible for retirement at the end of fiscal year  2011, but the plan does not specify the number or positions of these  employees.
The Carlsbad Field Office workforce plan indicates that both of that  office’s “contract/procurement specialists” will be eligible to retire by  fiscal year 2017, along with 10 of its 15 general engineers.
The workforce plan for the Office of River Protection, which manages  the storage, retrieval, treatment, and disposal of tank waste at the  Hanford Site in Washington State, projects that the office will face a  61 percent shortfall in “contracting” and a 53 percent shortfall in  “project management” by fiscal year 2017.
EM officials said that they recognize the need to better identify mission- critical occupations and skills and shortfalls in these areas, and that they  have taken a number of steps to address these issues. For example,  officials in EM’s Office of Human Capital told us that they conducted a  skills assessment in 2010 that helped EM identify key occupational series  to target in its succession planning efforts. In addition, officials in this  office told us that they are actively engaged in mitigating the risk of having  a large number of EM federal employees retire in the near future by  developing a voluntary separation incentive plan and voluntary early  retirement plan. If employees eligible for retirement participate in this  plan, EM could fill vacated positions with younger employees who could  develop their skills in future years. Moreover, officials in EM’s Office of  Acquisition and Project Management told us that to ensure that each  project team has the skilled staff it needs to meet project goals, they  consult with the EM officials in charge of each project team, consider the  project’s execution plan, and use DOE staffing guidance as a tool to  inform staffing decisions.
EM officials also said that EM sites serve diverse functions and that,  therefore, some variation in the workforce plans and their descriptions of  mission-critical occupations and competencies is to be expected.  Nonetheless, without a workforce plan or summary document presenting  a consistent set of occupations and skills that are critical to every site  office’s mission, such as project and contract management, it is difficult  for DOE and us to understand EM’s most critical current and future  human capital needs.
section_title: Conclusions
paragraphs: The 71 nonmajor projects that we reviewed cost an estimated $10.1  billion and are critical to DOE’s mission to secure the nation’s nuclear  weapons stockpile and manage the radioactive waste and contamination  that resulted from the production of such weapons. EM and NNSA are  making some progress in managing these projects. For example, we  identified some NNSA and EM nonmajor projects that used sound project  management practices, such as the application of effective acquisition  strategies, to help ensure the successful completion of these projects.  However, some contract and project management problems persist.  Specifically, both EM and NNSA have approved the start of construction  and cleanup activities for some nonmajor projects without clearly defining  and documenting performance targets for scope, cost, or completion date  in the appropriate CD-2 documentation, as required by DOE’s project  management order. In addition, EM and NNSA have not consistently  tracked project performance, particularly for scope, in DOE’s centralized  database for tracking and reporting project performance, as required by  DOE’s project management order. Moreover, EM has approved new  performance targets for projects without ensuring that these targets are  reviewed by an independent team of experts, as required by DOE’s  project management order. Without clearly defining and documenting a  project’s performance targets and tracking performance against these  targets through project completion, and without ensuring that projects are  independently reviewed, neither DOE nor we can determine whether the  department is truly delivering on its commitments when its contractors  complete work on its projects.
Problems also persist regarding DOE’s workforce—specifically, its current  and potential shortfalls in federal personnel with the skills necessary to  manage its contracts and projects, an issue that has received attention in  our high-risk list. EM recognizes the need to address this issue and has  taken steps to do so, such as conducting succession planning based on  an assessment of key skills, as well as having EM’s Office of Project  Management consult with EM’s project teams to ensure that the project  teams have the skilled personnel they need to execute projects  successfully. However, EM does not consistently identify in its workforce  plans the occupations and skills most critical to the agency’s mission, as  well as current and future shortfalls in these areas. This issue is  compounded by EM’s decentralized planning process, in which site  offices produce their own workforce plans that do not use consistent  terminology and are not aggregated centrally by EM headquarters into a  single workforce plan or summary document. Some variation among site- specific workforce plans is to be expected, but EM officials have stated  that it would be useful to establish a consistent set of terms for mission- critical occupations and skills and told us that they plan to address this  issue in the fiscal year 2013 planning cycle. That said, previous EM  guidance for workforce planning specified that the plans describe  shortfalls and surpluses in the skills most critical to site performance, but  not all of EM’s plans did so. Without a summary document or single  workforce plan presenting a consistent set of occupations and skills that  are critical to every site office’s mission, such as project and contract  management, using consistent terms, it is difficult for DOE or us to  understand EM’s most critical current and future human capital needs.
section_title: Recommendations for Executive Action
paragraphs: To ensure that DOE better tracks information on its nonmajor projects,  including the extent to which these projects meet their performance  targets, and that EM consistently identifies mission-critical occupations  and skills, as well as any current and future shortfalls in these areas, in its  workforce plans, we recommend that the Secretary of Energy take the  following five actions:
Ensure that the department clearly defines performance targets— including targets for scope, cost, and completion date—for each of its  projects and documents the targets in appropriate CD-2  documentation, as is required by DOE’s project management order.
Ensure that the department tracks the performance of its projects  using the performance targets, particularly scope, it establishes for its  projects, as is required by DOE’s project management order.
Ensure that each project is reviewed by an independent team of  experts before the department approves performance targets, as is  required by DOE’s project management order.
Direct EM to develop a summary document or a single workforce plan  that contains information on mission-critical occupations and skills, as  well as current and potential future shortfalls in these areas, for all EM  sites.
Ensure that EM follows through on its plan to address the use of  consistent terms across all EM sites for mission-critical occupations  and skills.
section_title: Agency Comments
paragraphs: We provided a draft of this report to DOE for review and comment. In  written comments, DOE agreed with our recommendations. DOE’s written  comments are reprinted in appendix IV. DOE also provided technical  clarifications, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of Energy,  the appropriate congressional committees, and other interested parties. In  addition, the report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which the Department of Energy’s (DOE)  Office of Environmental Management (EM) and National Nuclear Security  Administration (NNSA) nonmajor projects have met their scope, cost, and  completion date targets, we obtained performance information on 71  nonmajor projects. These 71 nonmajor projects included 30 EM projects  and 41 NNSA projects that were either: (1) completed (i.e., reached  critical decision 4) from fiscal year 2008 to fiscal year 2011 or (2) ongoing  from fiscal year 2008 to fiscal year 2011 and for which EM and NNSA had  established performance baselines at critical decision 2. We also  collected performance information for ongoing projects for fiscal year  2012. The total estimated cost of these 71 projects is approximately $10.1  billion. The names and locations of these 71 projects are provided in  apps. II and III. We excluded the following types of projects from our  review: (1) major projects, or those projects that each cost more than  $750 million; (2) EM projects funded entirely by the American Recovery  and Reinvestment Act of 2009 because of a separate GAO review looking  at these projects; (3) information technology acquisitions; and (4)   We identified these projects using DOE’s Project  operational activities.Assessment and Reporting System (PARS). To assess the reliability of  PARS data, we interviewed officials about the system and reviewed  relevant documents. On the basis of this information, we determined that  the system has adequate and sound controls for entering and maintaining  data. We also conducted electronic testing on the specific data fields of  interest, including cost, schedule, and scope targets. We determined that  the cost and schedule data were complete and sufficiently reliable for our  purposes; however, we found the scope data to be incomplete. Through  interviews with officials, we ascertained that the scope data were not  missing because of a system or data entry problem; instead, because EM  and NNSA had not consistently identified and documented scope targets  for the 71 projects we reviewed, these data could not be entered into  PARS. Therefore, we obtained data on project scope, cost, and schedule  directly from EM and NNSA officials.
For the 71 nonmajor projects, we reviewed selected documents providing  information about the projects’ targets for scope, cost, and completion  date. We relied on DOE Order 413.3 for requirements on (1) specifying  the scope, cost, and schedule targets for a project’s performance  baseline and (2) documenting the performance baseline.requirements, we reviewed the relevant documentation (including critical  decision memoranda and project execution plans) and compared the  performance targets established for scope, cost, and schedule with the  actual performance of completed projects and the expected performance  of ongoing projects. For completed projects, we compared the  performance targets for scope, cost, and schedule—as documented in  critical decision 2 (CD-2) approval memorandum and project execution  plans—with the completed scope, actual costs, and approval dates as  documented in critical decision 4 (CD-4) approval memorandum. For  ongoing projects, we compared the performance targets for scope, cost,  and schedule with DOE project performance reports; we also had officials  from EM, NNSA, and DOE’s Office of Acquisition and Project  Management review performance information as of August 29, 2012. In  cases where key project documents—including the CD-2 and CD-4  approval memoranda and project execution plans—did not identify all  three performance targets for scope, cost, and completion, we requested  and reviewed alternative project documents. These included, among  other things: independent project review reports; briefing slides prepared  for DOE advisory boards; contractor work packages; DOE documents  listing the functional and operational requirements of projects;  memoranda used to request approval of changes to project baselines;  final acceptance reports documenting that contractors delivered project  requirements; and DOE quarterly and monthly status reports on ongoing  projects. When reviewing alternative project documents, we requested  documents dated as close to CD-2 and CD-4 as possible. If documents  were not dated within 1 year of CD-2 approval, we did not consider them  sufficient and reliable for purposes of determining scope targets.
In keeping with our prior work, and in recognition of Office of  Management and Budget guidance and DOE’s project performance  goals, we characterized nonmajor projects that met or exceeded (or are  expected to meet or exceed) their cost and schedule targets by less than  10 percent as completed within budget and on time, whereas we  considered projects that exceeded (or will exceed) their targets by 10  percent or more to be over cost or late.whether a project had successfully met its scope target. Projects that  reduced their scope target to meet their cost targets were considered not  to have met their scope targets. In a few cases, EM and NNSA increased  the scope of work associated with a project after establishing  performance targets at CD-2; in these cases, we noted that these projects  had been modified and did not calculate whether they had met or  exceeded their original cost and schedule targets.
In addition, we considered  To evaluate factors affecting EM’s and NNSA’s management of nonmajor  projects, we selected a nongeneralizable sample of 20 out of the 71  projects—including 10 EM projects and 10 NNSA projects—for more  detailed review. The names of these 20 projects are provided in apps II  and III. Results from nonprobability samples, including our sample of 20  projects, cannot be used to make inferences about EM’s and NNSA’s  overall project performance or generalized to projects we did not include  in our sample. We were interested in gathering information on the  selected projects to identify material factors that may not exist across all  projects but could help us understand EM’s and NNSA’s organization  strengths and potential challenges. We selected these 20 projects to  ensure that our sample included completed and ongoing projects, with a  wide range of project costs. Together, the 20 projects represented about  $4.1 billion, or approximately 41 percent, of the total value of the 71  projects.
For these 20 projects, we developed a structured interview template to  identify the key factors that affected the management of these projects.  We used three primary sources in developing this structured interview  template—GAO’s cost guide, DOE’s Order 413.3, and DOE’s guidance  document on conducting project reviews. The structured interview  template focused on certain aspects of project management, such as the  preparation of project designs, risk estimates, and cost and schedule  targets, as well as the adherence to DOE project management  requirements. We pretested the structured interview template during a  site visit to the Y-12 National Security Complex and the Oak Ridge  Reservation near Oak Ridge, Tennessee. At each site, we selected six  projects and interviewed relevant EM and NNSA federal project directors  and other knowledgeable staff using the structured interview template.  Based on our pretesting, we revised the structured interview template and  conducted 20 interviews with the relevant EM and NNSA federal project  directors and other knowledgeable staff to gather their perspectives on  their projects’ performance and reasons for it.
To evaluate the extent to which EM’s workforce plans identify mission- critical occupations and skills and any current and future shortfalls in  these areas, we examined EM’s strategic workforce plans for its  headquarters and site office staff, DOE’s corrective action plan for  contract and project management, and the Office of Personnel  Management’s Human Capital Assessment and Accountability  Framework. Specifically, we obtained the eight EM workforce plans,  prepared by EM headquarters, the Consolidated Business Center, the  Richland Operations Office and Office of River Protection (which manage  operations at the EM site in Hanford, Washington), the  Portsmouth/Paducah Site Office, the Savannah River Operations Office,  the Idaho Operations Office, the Carlsbad Field Office, and the Oak Ridge  Office. We reviewed these plans in their entirety, and also searched for  relevant terms, to determine the extent to which the plans identified  mission-critical occupations and skills and any current and future  shortfalls in these areas. In addition to our document review, we  interviewed DOE and EM officials with knowledge of EM’s practices in  workforce planning, including officials in EM’s Office of Acquisition and  Project Management and Office of Human Capital and Corporate  Services. We conducted these interviews to determine how EM develops  its workforce plans and to obtain EM officials’ points of view regarding the  state of the EM workforce.
We conducted this performance audit from June 2011 to December 2012  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Summary of Office of Environmental Management Nonmajor Projects Reviewed
paragraphs: We obtained and reviewed performance information on 30 EM nonmajor  projects that were either completed or ongoing from fiscal year 2008  through fiscal year 2012. Table 6 summarizes this information for 17 EM  projects for which we could determine performance. Table 7 summarizes  this information for 6 projects for which EM did not establish performance  targets. Table 8 summarizes this information for 4 EM projects with  incomplete documentation of their performance targets, which meant that  we could not determine performance. Table 9 summarizes this  information for 3 projects for which EM modified the scope after  establishing performance targets for these projects, rendering the original  performance targets unusable for purposes of assessing performance.
section_title: Appendix III: Summary of National Nuclear Security Administration Nonmajor Projects Reviewed
paragraphs: We obtained and reviewed performance information on 41 NNSA  nonmajor projects that were either completed or ongoing from fiscal year  2008 through fiscal year 2012. Table 10 summarizes this information for  27 NNSA projects for which we could determine performance. Table 11  summarizes this information for 2 projects for which NNSA did not  establish performance targets. Table 12 summarizes this information for 7  NNSA projects with incomplete documentation of their performance  targets or final cost, which meant that we could not determine  performance. Table 13 summarizes this information for 5 projects for  which NNSA modified the scope after establishing performance targets  for these projects, rendering the original performance targets unusable for  purposes of assessing performance.
section_title: Appendix IV: Comments from the Department of Energy
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Dan Feehan, Assistant  Director; Sandra Davis; Robert Grace; and Jason Holliday made key  contributions to this report. Also contributing to this report were John  Bauckman; Jennifer Echard; Cindy Gilbert; Steven Lozano; Minette  Richardson; Cheryl Peterson; and Carol Hernnstadt Shulman.