Nuclear Waste: Waste Isolation Pilot Plant Recovery Demonstrates Cost and Schedule Requirements Needed for DOE Cleanup Operations
Released: 2016-08-04T00:00:00

Letter

Background

This section describes (1) WIPP’s layout and base operations, (2) the February 2014 truck fire and radiological release accident investigation reports, (3) DOE’s requirements for operations activities and for project management applicable to the WIPP Recovery effort, (4) best practices for cost and schedule estimating, (5) best practices for conducting an AOA, (6) Revisions in 2014 and 2015 to DOE’s requirements for AOAs, and (7) DOE’s AOA process for WIPP’s new permanent ventilation system.

WIPP Facility Layout and Base Operations

As shown in figure 1, WIPP’s layout consists of surface facilities, underground facilities, and four shafts that connect the surface with the underground. WIPP’s underground facilities currently include seven waste disposal units or panels where waste containers are placed for final disposal, an area designated for constructing future disposal units, and an area for experimental research. The four shafts connecting the surface and underground facilities are (1) the air intake shaft, which is the primary air supply to the underground; (2) the exhaust shaft, through which all air exits the underground; (3) the salt handling shaft, which is used to remove mined salt from the underground and also supplies air to the underground; and (4) the waste handling shaft which is used to transport waste containers to the underground. Figure 1 also shows the locations of the February 2014 accidents that led to the suspension of WIPP’s operations.

Under normal operating conditions, base operations at WIPP include waste handling to receive waste from generator sites and prepare and place the waste, ongoing maintenance of the WIPP surface facilities and underground, as well as support functions such as program management, engineering, quality assurance, safety and security, and environmental management. In addition, NWP reviews and certifies waste containers on DOE’s behalf at DOE’s transuranic waste generator sites. During implementation of the WIPP recovery plan, DOE continued aspects of these base operations such as maintenance of the surface facilities and program management. In addition, NWP continued to review and certify waste packages at select sites.

WIPP Truck Fire and Radiological Release Accidents and Investigation Reports

The salt truck fire, which occurred on February 5, 2014, created substantial smoke and soot that damaged key equipment and facilities in the underground. On February 14, 2014, the radiological release occurred when a transuranic waste container was breached. The breach was caused by a chemical reaction inside the container between materials that DOE later determined should not have been packaged together. The reaction generated enough heat to increase pressure in the container. The pressure forced open the container’s lid and propelled its radioactive waste contents, combustible gases, and other materials into the air and on to adjacent waste containers. The radioactive contents, gases, and other materials ignited and triggered a fire in the disposal room by igniting other materials in the room. WIPP’s ventilation system failed to contain all of the airborne radiological material underground and allowed a small amount to enter the environment. As a result of the release, portions of the WIPP underground and the existing ventilation system were radiologically contaminated.

DOE issued an accident investigation report on the salt truck fire in March 2014 and an accident investigation report on the radiological release in April 2014 called “phase one report.” The phase one report focused on the release to the environment and the response at WIPP to the release. One year later, in April 2015, DOE issued a phase two report that focused on the radiological release from inside the waste container. The reports determined that the causes of the accidents included the degraded condition of critical equipment, inadequacies in the design and operations of WIPP’s ventilation system, and deficiencies in the management of WIPP’s safety programs that are intended to control and discipline operations to protect workers, the public, and the environment from radiological and other hazards. In total, the three reports made 100 recommendations to DOE and NWP to complete corrective actions. For example, the phase one report identified a number of weaknesses with the WIPP nuclear safety documentation and recommended that this documentation be revised prior to resuming limited waste disposal operations to ensure the repository can be operated safely with respect to workers, the public, and the environment.

DOE’s Requirements for Operations Activities and for Project Management Applicable to the WIPP Recovery Effort

DOE is managing the recovery activities to restart limited waste disposal operations at WIPP and to design and build the new permanent ventilation system capital asset project following two separate project management requirements. DOE requires that the recovery activities to restart limited waste disposal operations and overall base operations of WIPP be managed following EM’s protocol for operations activities. This protocol defines operations activities to include activities that are project- like with defined start and end dates and reoccurring facility or environmental operations. The new permanent ventilation system project must follow DOE’s Order 413.3B, which governs project management for the acquisition of capital assets. The following is a description of the two different requirements:

EM operations activities protocol. EM’s protocol establishes a framework for managing and reporting the progress of cleanup operations by requiring, among other things, that project performance be measured objectively and that management actions be taken to mitigate risks and manage costs. EM’s protocol directs EM sites to develop performance baselines—or estimates—for cost, schedule, and scope to use in assessing the project’s performance over the fiscal year, multiyear contract period of performance, and the life cycle of the project. The protocol directs that the project develop estimates but does not include specific requirements that must be followed, such as the steps that must be followed to develop cost and schedule estimates.

DOE Order 413.3B. DOE’s order provides program and project management direction for the acquisition of capital assets, with the stated goal of delivering fully capable projects within the planned cost, schedule, and performance baseline. The order establishes five critical decision processes of project development that each end with a major approval milestone—or “critical decision” point—that cover the life of a project. The order specifies the requirements that must be met, including for developing and managing project cost and schedule estimates to move a project past each critical decision milestone. DOE also provides suggested approaches for meeting the requirements contained in the order through a series of guides, such as a guide for cost estimating. In June 2015, the Secretary of Energy directed that this order be revised to require that cost and schedule estimates be developed, maintained, and documented in a manner consistent with methods and the best practices identified in GAO’s cost and schedule assessment guides and other published standards and best practices, such as the Federal Acquisition Regulation and Office of Management and Budget guidance.

GAO’s Best Practices for Cost and Schedule Estimating

The GAO Cost Estimating and Assessment Guide and the GAO Schedule Assessment Guide compiled best practices corresponding to the characteristics of high-quality and reliable cost and schedule estimates. A high-quality, reliable cost estimate has the following four characteristics: comprehensive (e.g., has enough detail to ensure that cost elements are neither omitted nor double counted), well-documented (e.g., allows for data it contains to be traced to accurate (e.g., is based on an assessment of most likely costs and has been adjusted properly for inflation), and credible (e.g., discusses any limitations because of uncertainty or bias surrounding data or assumptions).

Similarly, a high-quality, reliable schedule has four characteristics: comprehensive (e.g., captures all government and contractor activities necessary to accomplish a project’s objectives), well-constructed (e.g., sequences all activities using the most straightforward logic possible), controlled (e.g., is updated periodically to realistically forecast dates for activities), and credible (e.g., uses data about risks to predict the level of confidence in meeting a completion date and necessary schedule contingency and high-priority risks are identified based on conducting a robust schedule risk analysis).

The characteristics of a high-quality and reliable cost estimate are supported by best practices listed in appendix II, and the characteristics of a high-quality and reliable schedule are supported by best practices listed in appendix III.

GAO’s Best Practices for Analysis of Alternatives

In a December 2014 report we compiled 24 best practices for conducting an AOA. As noted above, the AOA is a key first step in acquisition of a capital asset such as WIPP’s new permanent ventilation system. The process entails identifying, analyzing, and selecting a preferred alternative. Conforming to these best practices helps ensure that the preferred alternative selected is the one that best meets the agency’s mission needs. We grouped these 24 best practices into the following four categories:

General principles. This category includes best practices to be applied in planning, conducting, and documenting the AOA, such as defining functional requirements based on the mission need and conducting the analysis without a predetermined solution.

Identifying alternatives. The identifying alternatives category includes best practices that help ensure the alternatives to be analyzed are sufficient, diverse, and viable.

Analyzing alternatives. The analyzing alternatives category contains best practices related to estimating the costs and benefits of each alternative over its life cycle.

Selecting a preferred alternative. The selecting a preferred alternative category includes best practices to help ensure a preferred alternative is selected that best meets the mission need by comparing alternatives based on their costs and benefits and independently reviewing the AOA process. Appendix IV lists the 24 best practices organized by the four categories.

Revisions in 2014 and 2015 to DOE’s Requirements for Analysis of Alternatives

DOE’s procedures for conducting an AOA have changed twice since DOE began the AOA process for the WIPP permanent ventilation system. DOE started the AOA process for this system in October 2014. At that time, DOE was operating under AOA requirements defined in Order 413.3B. The AOA process, as defined in the order, occurs during the span of the first two critical decisions—Critical Decision-0 and Critical Decision-1 (Approve Alternative Selection and Cost Range). In December 2014, the Secretary of Energy established a new requirement that for projects estimated to cost $50 million or more, the responsible program office is to conduct an AOA independent of the contractor responsible for the project. Later, in June 2015, the Secretary of Energy directed the department to develop guidance for conducting AOAs consistent with the AOA best practices that GAO has published. The Secretary further required that AOAs be conducted and documented in a manner consistent with the guidance when it is complete. A DOE official told us in June 2016 that the department expected the guidance to be completed by December 2016. The Secretary’s action followed a recommendation that we made in December 2014 that DOE revise its Order 413.3B to adopt AOA best practices.

DOE’s AOA Process for WIPP’s New Permanent Ventilation System

DOE’s changes in its requirements for the AOA process under Order 413.3B took effect in the midst of DOE’s AOA process for WIPP’s new permanent ventilation system. DOE’s process was conducted as follows:

DOE EM approved the mission need for the new ventilation system project in October 2014.

Also in October 2014, NWP started an initial AOA and completed it in January 2015 shortly after the Secretary’s December 2014 directive on the independence of AOAs and prior to the Secretary’s June 2015 directive about the use of the AOA best practices; therefore the new requirements did not apply to NWP’s AOA.

To respond to the Secretary’s December 2014 directive, DOE contracted in spring 2015 with a contractor unaffiliated with WIPP— Trinity Engineering Associates (Trinity)—to do a second AOA. DOE specifically directed Trinity in its contract to implement all of the AOA best practices that we compiled in our December 2014 report. The WIPP program manager in the Carlsbad Field Office told us that he included the best practices in the contract to provide more formality to their process. Trinity completed its AOA in October 2015.

DOE evaluated the preferred alternatives proposed by NWP and Trinity, and it selected and approved a final alternative to complete Critical Decision-1 in December 2015.

DOE Did Not Meet Its Initial Cost and Schedule Estimates for Restarting Waste Disposal Operations

DOE did not meet its initial cost and schedule estimates for the efforts needed to restart WIPP disposal operations, resulting in about $64 million in added costs and a delay of nearly 9 months. Two primary factors contributed to the cost increase and schedule delay. First, DOE only partially followed best practices in developing its initial cost and schedule estimates, which made them unreliable and increased the likelihood that they would be exceeded. Second, DOE did not successfully manage key project risks it had identified that had potential to cause delay.

DOE Incurred A Cost Increase of about $64 Million and a Nearly 9- month Delay in Its Efforts to Restart WIPP Operations

DOE incurred a cost increase of about $64 million and a nearly 9-month schedule delay in its efforts to restart WIPP waste disposal operations. Specifically, in January 2016, approximately a year after approving the initial project baseline, DOE approved a new project management baseline that increased the estimated costs for the recovery project by $2 million (from $242 million to $244 million) and added 8.5 months to the project schedule, extending the date when limited waste disposal operations might begin from March 2016 to December 2016. According to DOE officials, the project was unable to meet its estimated completion date due to delays, including delays associated with procuring components for the interim ventilation system upgrade that were found to be faulty, and delays associated with DOE’s decision to require the project to adhere to new nuclear safety requirements. According to the operations activity manager for WIPP in the Carlsbad Field Office, the recovery project only exceeded the $242 million baseline by $2 million in part because DOE overestimated the cost of some project activities and spent contingency funds to make up for cost increases in certain parts of the project. For example, DOE overestimated the cost for decontaminating the WIPP underground by more than $6 million, which offset a $5 million cost increase caused by delays associated with the faulty ventilation system components. However, we identified an additional $61.6 million cost increase in base operations that was attributable to the delay in completing the recovery project. Specifically, DOE’s justification supporting the President’s fiscal year 2016 budget request to Congress from February 2015 estimated about $130.6 million for WIPP’s base operations in fiscal year 2016. According to DOE officials, as DOE revised its cost and schedule estimates for the WIPP recovery project in the fall of 2015, DOE also revised its cost estimate for WIPP’s base operations and provided the updated information to the congressional appropriations committees. In December 2015, DOE received $192.1 million for fiscal year 2016 base operations, an increase of $61.6 million (47 percent) over the initial estimate for base operations in fiscal year 2016. According to the operations activity manager for WIPP in the Carlsbad Field Office and representatives from NWP, this increase to base operations occurred because DOE estimated the cost of base operations on the assumption of restarting WIPP in March 2016 following its initial schedule estimate. The delay in the recovery efforts required DOE to keep base operations running alongside the recovery efforts for a longer period of time than initially planned.

DOE’s Initial Cost and Schedule Estimates for Restarting Operations at WIPP Were Unreliable

DOE did not meet its initial schedule and cost estimates for restarting waste disposal operations at WIPP, in part because DOE did not develop the estimates following all best practices, rendering the estimates unreliable and increasing the likelihood that they would be exceeded. More specifically, of the four characteristics of a high-quality and reliable cost estimate, DOE’s initial cost estimate substantially met two characteristics—comprehensive and well-documented—but partially met or minimally met the other two characteristics—accurate and credible. We made the following observations:

DOE substantially met best practices for a comprehensive cost estimate by, for example, including a majority of life-cycle costs and identifying the cost estimating ground rules and assumptions.

DOE substantially met best practices for a well-documented estimate by including the actual sources for the cost data and documenting that the cost estimate was reviewed and accepted by DOE management.

DOE partially met best practices for an accurate estimate by, for example, only documenting a portion of the process they used to adjust costs to account for inflation.

DOE minimally met best practices for a credible estimate by, for instance, not developing an independent cost estimate. Had DOE completed an independent cost estimate, the department would have had an unbiased and objective benchmark to assess whether the cost estimate prepared by NWP could be achieved, and thus would have been positioned to reduce the risk that the project would proceed underfunded.

Regarding DOE’s initial schedule for restarting waste disposal operations, DOE substantially met two of the four characteristics of a high-quality schedule—comprehensive and controlled—but partially met the characteristics—well-constructed and credible—resulting in a schedule that was also unreliable and unrealistic. We made the following observations:

DOE substantially met best practices for a comprehensive schedule because, for example, the schedule reflected the activities in the recovery work breakdown structure, which defined the work necessary to accomplish the project’s objectives. In addition, work scope was assigned in the schedule as the responsibility of NWP or DOE.

DOE substantially met best practices for a controlled schedule by, for example, regularly updating its master schedule and using the schedule as the basis for measuring performance.

DOE partially met best practices for a well-constructed schedule because, for instance, a significant number of activities in the schedule had incorrect or missing logic relationships that are important for determining how delays or accelerations in one activity would affect the start or finish of other activities later in the schedule.

DOE partially met best practices for a credible schedule, in particular, because its schedule did not include extra time, or contingency, to account for known project risks. As noted above, DOE acknowledged in its WIPP recovery plan that the schedule did not include contingency that may be needed due to unanticipated difficulties or delays with the project. However, DOE did not acknowledge in the plan that the schedule also did not include contingency for the occurrence of known and quantified risks that had been anticipated in its project risk analysis—such as the risks that were the primary causes of the project delays. Notably, DOE’s risk analysis predicted that the March 2016 date to restart operations had a less than 1 percent confidence level, meaning in effect that DOE had a less than 1 percent chance of meeting the March 2016 deadline. EM officials said the department used a less than 1 percent confidence level because it wanted to have an aggressive goal for restarting operations. According to EM officials, when EM managers presented this schedule to senior DOE decision-makers for approval, they described the schedule as “aggressive” but did not clarify that there was less than a 1 percent chance of meeting the schedule. According to these officials, they did not think that senior managers would understand the project management terminology regarding confidence levels.

In January 2016, DOE approved revised cost and schedule estimates for restarting WIPP’s limited waste disposal operations. The operations activity manager for WIPP in the Carlsbad Field Office said the revised schedule included contingency and the revised restart date of December 2016 had an 80 percent confidence level. However, according to DOE officials, they did not follow other best practices. For example, DOE did not provide evidence of having an independent cost estimate which would have provided DOE an unbiased and objective benchmark to validate the estimates prepared by NWP. As noted above, we did not assess DOE’s revised estimates against all of the best practices. The full results of our analysis of the initial cost and schedule estimates can be found in appendixes II and III.

DOE did not follow all best practices in developing the initial cost and schedule estimates for the WIPP recovery project or in developing new estimates because, unlike DOE’s requirements for capital asset projects under Order 413.3B, DOE’s EM operations activities protocol that governs cleanup operations such as WIPP recovery does not require the use of best practices in developing such estimates. As mentioned above, DOE EM’s protocol requires that EM sites develop or approve baselines for each project’s cost and schedule to judge project performance. However, the protocol does not specify any best practices in terms of the steps to follow in developing the cost and schedule estimates in these baselines. The absence of a requirement to follow best practices for cost and schedule estimating is in contrast to a decision in June 2015 by the Secretary of Energy to require that all capital asset projects with an estimated cost of $10 million or more follow such practices under Order 413.3B. Operations activities such as the WIPP recovery activities are not considered capital asset projects and therefore are not required to follow these requirements even though EM’s operations activities share many of the same characteristics as capital asset projects. Without requiring EM’s operations activities such as WIPP to follow all best practices when developing cost and schedule estimates DOE cannot have confidence that it is producing reliable baselines needed to monitor its performance in managing these activities. Without reliable baselines, DOE will also continue to be at risk of cost overruns and delays in achieving its cleanup missions such as the permanent disposal of transuranic waste from sites across the country.

DOE has reported progress in completing the activities needed to restart limited waste operations using its revised cost and schedule estimates, although it still faces challenges and it remains unclear whether DOE identified and analyzed the risks associated with these challenges in revising the estimates. In terms of progress, for example, in December 2015, NWP submitted revisions to WIPP’s nuclear safety documentation to EM for review. In addition, DOE reported in March 2016 that NWP completed most of the construction of the interim ventilation system upgrade in the underground. However, as noted above, in March 2016, the Defense Nuclear Facilities Safety Board reported that additional work was needed to revise WIPP’s nuclear safety documentation to prevent recurrence of a radiological release accident. In addition, in April 2016, DOE’s Office of Enterprise Assessments, which provides independent internal oversight of DOE’s management of safety, issued two reports that found that although NWP had made improvements in its operational safety and emergency management programs and procedures, significant challenges remained to fully meet DOE requirements and effectively plan and implement these programs and procedures. As we noted above, DOE officials acknowledged that the department did not follow all best practices in developing the revised estimates to restart WIPP operations. Without having followed all best practices, including having an independent cost estimate conducted to validate the estimates, DOE cannot be confident that NWP sufficiently accounted for these challenges in revising its risk analysis and that DOE set an appropriate allowance for contingency to reduce the risk of cost overruns and delays in restarting WIPP’s operations.

DOE Did Not Successfully Manage WIPP Recovery Project Risks That Contributed to the Project Cost Increase and Schedule Delay but Has Revised Its Risk Management Process

DOE did not successfully manage key risks that it had identified and that contributed to the project’s cost increase and schedule delay but has taken steps to revise the risk management process for the WIPP recovery project. DOE’s EM operations activities protocol requires the use of a risk management process to identify and mitigate risks to completing a project within its baseline cost and schedule estimates. For the WIPP recovery project, DOE and NWP developed a project risk register to support the initial project management baseline for WIPP recovery. The risk register listed the risks that could increase costs or delay completion. DOE and NWP assessed the likelihood and consequence of each risk and identified a mitigation plan for each risk that described the actions to reduce the impact on the project if the risk were realized.

In a public statement explaining the need to revise its baseline for the recovery project, DOE cited problems with three activities that the department had previously identified in its project risk register: (1) completing revisions to WIPP’s nuclear safety documentation; (2) installing the interim ventilation system upgrade; and (3) completing the corrective actions that address the recommendations in the phase two accident investigation report. DOE had identified risks related to each of these activities in the project risk register but was not able to effectively mitigate them, as discussed below:

WIPP nuclear safety documentation. NWP identified the risk of a potential delay of up to 6 months in restarting operations if DOE decided that the WIPP nuclear safety documentation should be revised according to standards that were in the process of being updated rather than according to the existing standards. According to DOE, when NWP started to revise the safety documentation in the summer of 2014, the contractor assumed it should follow the existing standard. NWP added this risk to its project risk register to account for the possibility that EM could require NWP to revise the nuclear safety documentation according to the new standards. To mitigate this risk, NWP’s plan was to ensure they had concurrence with DOE to start the revisions using the existing standards. DOE issued the new standards in November 2014 and notified NWP in December 2014 that the WIPP safety documentation would need to comply with them. According to an NWP recovery project manager, this change in policy resulted in a 7-month delay in NWP’s schedule for revising the WIPP safety documentation.

Interim ventilation system. NWP identified the risk of a potential delay of 3 months to complete the installation of the interim ventilation system upgrade if NWP faced difficulties acquiring the components for the system. To mitigate this risk, NWP’s plan was to identify difficult- to-procure equipment early in the project, ensure it had a valid and up-to-date list of qualified suppliers, and monitor and review equipment purchases. According to NWP officials, this plan was partially effective. Specifically, NWP officials said that they discovered that components for the interim ventilation system were damaged when they inspected the components before formally accepting them. In addition, they said that the costs to return the components to the manufacturer and repair them were paid for by the manufacturer. However, the NWP officials said that as a result of receiving the faulty equipment, the installation of the interim ventilation system was delayed about 4 months because NWP needed to send the components back to the manufacturer and make corrections to its shipping process to prevent a reoccurrence of these issues.

Corrective actions from the phase two accident investigation report. NWP identified the risk of potential delay of up to 6 weeks regarding the length of time DOE needed to complete the phase two accident investigation. In the summer and fall of 2014, when NWP was developing the initial project management baseline, DOE’s phase two accident investigation was still underway. According to NWP officials, they needed to wait for DOE to complete the accident investigation and issue the final report with recommendations before they could begin developing and implementing corrective actions. DOE issued the final report in April of 2015. According to the operations activity manager for WIPP in the Carlsbad Field Office, completing the corrective actions in response to the report’s recommendations resulted in a 3-month delay.

NWP officials said that in recognition of the need for a more effective risk management process, they revised their process as part of revising the recovery cost and schedules estimates. These officials explained, for example, the revised process now involves more frequent discussions and updating the status of potential risks with managers at WIPP who oversee the departments where the project risks are generated. We did not conduct an assessment of the effectiveness of this new risk management process.

DOE Did Not Follow All Best Practices in Analyzing and Selecting an Alternative for the New Ventilation System at WIPP

DOE did not follow all best practices in analyzing and selecting an alternative for the new ventilation system at WIPP. As a result, DOE’s analysis was not reliable and DOE cannot be confident that the selected alternative will best provide the needed capabilities. Of the four categories of the best practices, DOE fully met the category for identifying alternatives. However, DOE partially or minimally met the other three categories of best practices—general principles, analyzing alternatives, and selecting the preferred alternative—because of key limitations. (See app. IV for the detailed results of our analysis of DOE’s AOA process.)

DOE Fully Met Best Practices in the Identifying Alternatives Category

DOE’s AOA process fully met the best practices in the identifying alternatives category. To identify alternatives, DOE relied on analyses conducted by NWP, the contractor responsible for the project, and the second contractor—Trinity. These analyses were completed in January and October 2015, respectively. NWP and Trinity identified a broad range of alternatives. Specifically, they identified nine ventilation system alternatives that the contractors determined should be analyzed because the alternatives could meet the mission need and other screening criteria that they defined: NWP identified six and Trinity identified three. The alternatives included continued use of the existing exhaust shaft as well as constructing a new shaft or more than one shaft, such as a new exhaust shaft and a new air intake shaft. In addition, the studies identified different ventilation filtration capacities and modes of operation that included continuous filtering of all exhaust air or filtering air only when a radiological release is detected underground.

DOE Partially Met Best Practices in the General Principles Category

DOE’s AOA process partially met the best practices in the general principles category, which covers how the AOA process is planned, conducted, and documented. In particular, DOE partially or minimally followed the best practices of defining the mission need for the project, analyzing alternatives without a predetermined solution, and defining the functional requirements based on the mission need. As the AOA related to the best practices of defining the mission need for the project and analyzing alternatives without a predetermined solution, DOE did not define the mission need for the new ventilation system by focusing only on the capabilities needed for the project, but instead defined the need for the system to include a particular solution, which included constructing a new ventilation exhaust shaft. Because DOE defined the mission need by mentioning a new exhaust shaft, both contractors appeared to have analyzed alternatives with a preference for the alternatives that included constructing a new shaft. Specifically, NWP completed its analysis of the six alternatives by proposing two for further consideration by DOE, one that did not include a new shaft and one that did. NWP’s alternative that did not include a new shaft was its highest-scoring alternative and the alternative that included a new shaft was its third-highest scoring alternative. NWP officials told us that they proposed the second alternative along with the highest-scoring one because they believed the mission need statement discussed a new shaft. Regarding Trinity, in its final AOA report, Trinity stated that the mission need statement endorsed a new exhaust shaft. Trinity then analyzed three alternatives, one of which included using the existing shaft, and proposed an alternative to DOE that included constructing a new shaft. In its final assessment, DOE assessed the two alternatives proposed by NWP and the one proposed by Trinity and selected NWP’s alternative that included constructing a new shaft, shown as the second of the three alternatives in table 1. According to best practices, the AOA process should be an unbiased inquiry into the costs, benefits, and capabilities of all alternatives. By conducting the AOA with a predetermined solution, DOE undermined the credibility of its final decision.

Another limitation under the general principles category was that DOE partially followed the best practice of defining the functional requirements that the ventilation system would need to satisfy. DOE did not consistently define two key functional requirements of the project—(1) the rate of airflow needed to support full operations at WIPP, and (2) the expected operational life of the new system—which limited the reliability of the overall AOA process as follows:

Regarding the estimated airflow, DOE did not specify a functional requirement in its mission need statement for the rate of airflow that the new ventilation system must be capable of providing for full operations at WIPP. Instead, the mission need statement described the minimum airflow required by WIPP’s hazardous waste facility permit, which was 260,000 cfm, and the actual airflow at WIPP before the accidents, which was 425,000 cfm. In conducting its AOA, NWP calculated its own airflow that it thought would be necessary for full operations at WIPP, which they estimated was about 540,000 cfm. NWP estimated the higher airflow based on its analysis of underground operations and used this amount to analyze a range of alternatives. In contrast, Trinity used the airflow associated with the WIPP hazardous waste facility permit, 260,000 cfm, as the estimated airflow in conducting its AOA. The federal project director in the Carlsbad Field Office told us that he did not direct Trinity to redo its analysis using NWP’s higher airflow because he believed that airflow was needed for examining the scale of the system and would be defined lower during design. Moreover, he said that he wanted to maintain Trinity’s independence from NWP to be consistent with the December 2014 DOE requirement for conducting independent AOAs.

However, as a result of not defining the same functional airflow requirement for both contractors, DOE officials explained that they needed to conduct additional analysis to compare the alternatives proposed by NWP and Trinity.

Regarding the estimated operational life of the new ventilation system, DOE did not specify a functional requirement in its mission need statement for the estimated operational life of WIPP. Without such a requirement, NWP used 2030 as the estimated end date for WIPP operations—this date was based on DOE’s approved life-cycle plan for ending WIPP’s operations current at the time NWP conducted its analysis. NWP completed its AOA in January 2015 when it presented two alternatives to DOE for further evaluation—as described above, NWP had proposed these two alternatives from an initial group of six alternatives. In February 2015, EM revised the estimated operational life of WIPP, extending it from 2030 to 2050 to more accurately reflect DOE’s schedules for transuranic waste cleanup at DOE sites. Trinity did not start its AOA process until after DOE had revised the estimated operational life of WIPP to 2050 and therefore used the 2050 date in conducting its AOA. In analyzing NWP’s two proposed alternatives, DOE used the new estimated operational life of 2050 and did not reassess the four previously eliminated alternatives using the revised date.

By not following the best practice of specifying functional requirements at the start of the AOA process, including specifying the airflow that the system needed to deliver and the estimated end date of operations for WIPP, it is unclear whether the AOAs conducted by NWP and Trinity and DOE’s subsequent analyses of these AOAs allowed DOE to select the alternative that best meets mission needs.

DOE Minimally Met Best Practices in the Analyzing Alternatives Category

DOE’s AOA process minimally met the best practices in the analyzing alternatives category. Significant limitations in this category included DOE and its contractors not consistently examining life-cycle costs for each alternative and not quantifying their benefits. Regarding examining life- cycle costs:

NWP did not examine full life-cycle cost estimates for all of the alternatives it examined. Specifically, NWP’s AOA completed in January 2015 examined the estimated costs for the design and construction of each of the six alternatives, but did not examine the full life-cycles of these alternatives. As a result, NWP eliminated four of its six principal alternatives from further evaluation before examining each of them in terms of full life-cycle costs. According to the NWP project manager who led NWP’s AOA team, the team did not have sufficient time to examine the full life-cycle costs of each alternative. In addition, according to DOE officials, NWP was not required to examine the full life-cycle costs of each alternative under DOE Order 413.3B because they completed the analysis in January 2015, which was several months before the Secretary’s June 2015 directive to incorporate AOA best practices in the order. According to AOA best practices, the team conducting the AOA should be given enough time to complete the AOA process to ensure a robust and complete analysis.

NWP and Trinity used different assumptions in developing life-cycle cost estimates, which prevented DOE from directly comparing the life- cycle estimates for the three proposed alternatives. We found that DOE relied on NWP and Trinity to define the assumptions they used rather than providing the contractors consistent guidance or direction. As a result, NWP assumed certain costs should be excluded in estimating the life-cycle costs of its two proposed alternatives. These costs included the costs for major equipment replacements or upgrades which NWP assumed would not be needed during the 30- year operational life of the system; costs for providing facility security, quality assurance, and nuclear safety that are provided to all facilities at WIPP; and costs for the final closure of the new exhaust shaft after 2050 when the system will be decommissioned. In contrast, Trinity assumed these costs should be included in estimating the life-cycle costs of its alternatives. As a result of including these additional costs, Trinity’s estimated life-cycle cost was substantially more than NWP’s estimate—Trinity’s estimate for its proposed alternative was $3.45 billion and NWP’s estimates for its two proposed alternatives were $368.8 million and $467.6 million.

Regarding quantifying benefits, DOE and its contractors did not provide any measures of the benefits or effectiveness for each of the alternatives in their analyses. By not ensuring that the life-cycle costs of each alternative were developed in a consistent manner and not ensuring that the benefits or effectiveness of each alternative were quantified, DOE did not have an accurate and complete picture of all the alternatives to make reliable comparisons between them.

DOE Partially Met Best Practices in the Selecting a Preferred Alternative Category

DOE’s AOA process partially met the best practices in the selecting a preferred alternative category which covers selecting an alternative that best meets the mission need and also independently reviewing the overall AOA process. Most notably, DOE and its contractors did not follow the best practice to select the alternatives based on a cost-benefit analysis and only partially followed the best practice to independently review the AOA process, as follows:

Regarding cost-benefit analysis, DOE and its contractors did not select the alternatives based on such an analysis which compares the life-cycle costs and benefits or effectiveness of each alternative. As noted above, DOE and its contractors did not consistently examine life-cycle cost estimates for each alternative and did not quantify their benefits. Therefore, DOE did not produce the information needed for a cost-benefit analysis. By not selecting the alternatives based on a cost-benefit analysis, DOE did not adequately justify that the selected final alternative would best provide the capabilities needed at WIPP.

Regarding independently reviewing the AOA process, DOE conducted an independent review of the AOA process (which assessed whether best practices were followed) but did not implement the recommendation identified by the review. Specifically, an independent review conducted by DOE’s Office of Project Management Oversight and Assessments found that the project team did not adequately document a cost-benefit analysis and that, as a result, the selection of the preferred alternative was not supported by compelling information. The review recommended that DOE perform a cost-benefit analysis to be consistent with best practices and support the selection of the final alternative. However, DOE and NWP did not conduct the recommended cost-benefit analysis and document it before DOE selected the final alternative. According to the DOE officials who led the independent review of the AOA process, the project team was not required by DOE’s Order 413.3B to implement the recommendation or justify and document the reasons for not doing so. Therefore, DOE EM approved the selection of the preferred alternative for the permanent ventilation system project in December 2015 without implementing the independent review’s recommendation.

Conducting an independent review of the AOA is a best practice because it is one of the most reliable means to ensure that bias does not influence the AOA process and that the AOA is sufficiently thorough to ensure that a preferred solution is chosen and not a favored solution. DOE’s independent review was an internal control to help DOE meet its goal for the capital asset project, in this case selecting the alternative that will best provide the capabilities to meet the mission need. Under federal standards for internal control, management is to evaluate and document internal control issues and determine appropriate corrective actions. By not implementing the recommendation from the independent review to do a cost-benefit analysis or not justifying and documenting the reason for not doing so, DOE cannot provide assurance that the final alternative selected would best provide the capabilities needed at WIPP.

Conclusions

Restarting WIPP’s waste disposal operations is a top priority for DOE, and it has made progress in its efforts to restart limited waste disposal operations. However, DOE did not meet its initial cost and schedule estimates to restart operations and incurred a cost increase of about $64 million—$2 million to the recovery project and $61.6 million in base operations—and a delay of nearly 9 months. DOE incurred the cost increase and schedule delay, in part, because DOE’s initial estimates did not follow all best practices and were therefore unreliable. Notably, DOE did not include any contingency in its schedule, giving itself less than a 1 percent chance of success—said another way DOE gave itself a 99 percent chance of failure in meeting its schedule to restart operations. Moreover, when DOE revised its cost and schedule estimates for WIPP recovery it still did not follow all best practices. DOE did not develop its estimates for WIPP recovery following best practices, in part because DOE does not require its cleanup operations activities, such as WIPP, to follow them. This lack of a requirement is in contrast to a new policy put into effect in June 2015 by the Secretary of Energy, which established the requirements that DOE develop cost and schedule estimates for its capital asset projects following best practices—such projects include the new permanent ventilation system at WIPP. Without similar requirements for EM operations activities to follow best practices, DOE cannot have confidence that it is producing reliable baselines needed to monitor its performance in managing these activities and reduce the risk of cost overruns and delays in achieving its cleanup missions, such as the permanent disposal of transuranic waste from DOE sites across the country. By also requiring cleanup operations to follow best practices, DOE would have more confidence in the estimates for its cleanup operations activities and its capital asset projects.

In selecting the preferred alternative for the new permanent ventilation system at WIPP, DOE relied on an AOA process that did not follow all best practices. In particular, DOE only partially followed or did not follow best practices to define functional requirements based on the mission need, compare alternatives using a cost-benefit analysis, or independently review the AOA process. By not following these and other practices, DOE’s AOA process was not reliable. Notably, DOE conducted an independent review of the process which found that the project team did not adequately document a cost-benefit analysis and that, as a result, the selection of the preferred alternative was not supported by compelling information. Nonetheless, DOE did not implement the recommendation of the review to conduct a cost-benefit analysis before the department selected the preferred alternative. DOE officials explained that the project team was not required by DOE’s Order 413.3B to implement the recommendation. By not implementing the recommendation of the independent review or not justifying and documenting the reason for not doing so, DOE cannot provide assurance that the final selected alternative would best provide the capabilities needed at WIPP. More broadly, without requiring in its Order 413.3B that recommendations from independent reviews of AOAs be implemented or that the reason for not doing so be justified and documented, DOE cannot have assurance that it selects, for all capital asset projects, preferred alternatives from its AOAs that best meet the mission need.

Recommendations for Executive Action

To help ensure that DOE develops and uses reliable cost and schedule estimates and AOAs, we recommend that the Secretary of Energy take the following three actions:

Direct EM to revise its protocol governing cleanup operations activities to require use of best practices in developing cost and schedule estimates.

Direct EM to implement the recommendation made by DOE’s Office of Project Management Oversight and Assessments in its independent review of the AOA for WIPP’s new permanent ventilation system to perform a cost-benefit analysis consistent with best practices for conducting an AOA, or justify and document why the office does not intend to do so.

Direct DOE to revise its Order 413.3B to require that DOE offices implement any recommendations from an independent review of the extent to which an AOA followed best practices, or justify and document the rationale for not doing so.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE for review and comment. In written comments, reproduced in appendix V, DOE concurred with the report’s recommendations. DOE stated that our recommendations were consistent with its commitment to continuous improvement in project management. In addition, DOE stated that it had two issues that the department believed needed to be addressed. DOE also provided technical comments that were incorporated, as appropriate.

DOE identified the following two issues:

DOE stated that WIPP recovery activities were under way concurrent with DOE-wide efforts to improve project management, including revisions to its project management order (DOE Order 413.B). DOE stated that, consequently, EM project management practices were separate from evolving department project management guidance and that it is now in the process of bringing EM practices into conformance with the department’s project management guidance.

DOE stated that the report needs to be clear, in each instance, regarding the $64 million cost increase for WIPP attributed to schedule delays in recovery activities, to indicate that $2 million of the cost increase was WIPP recovery project-related and that $61.6 million was due to reexamination and assessment of the cost of base operations.

We do not believe changes are needed to the report for either of these issues. The report explains that WIPP recovery activities were based on EM’s operations activities protocol and that actions were under way within the department to improve project management, including revisions to DOE’s project management order. In addition, the report includes a breakout of the cost increase. We did, however, revise the conclusions to reflect this breakout.

As noted above, DOE concurred with the report’s three recommendations. Regarding the first recommendation—that EM revise its protocol governing cleanup operations activities to require use of best practices in developing cost and schedule estimates—DOE stated in its written comments that it concurred with clarification. DOE stated that EM is transitioning from the operations activities protocol to a new directive that is expected to include a key decision approving a cost and schedule baseline. As EM develops the guidance for this key decision, it will include the use of cost and schedule best practices. DOE stated that EM plans to finalize this new directive by September 2016 and seek departmental approval by December 2016.

Regarding the second recommendation— that EM implement the recommendation made by DOE in its independent review of the AOA for WIPP’s new ventilation system to perform a cost-benefit analysis consistent with best practices, or justify and document why it does not intend to do so—DOE stated in its written comments that it concurred with clarification. DOE stated that in accordance with GAO best practices, further cost-benefit analysis will be conducted on the project prior to approval of Critical Decision-2 (Approve Performance Baseline). DOE stated that several alternatives remain to be evaluated including the size of the ventilation system and the location of the exhaust shaft.

Regarding the third recommendation—that DOE revise its project management (Order 413.3B) to require that DOE offices implement recommendations from independent reviews of the extent to which an AOA followed best practices, or justify and document the rationale for not doing so—DOE concurred with the recommendation. DOE stated that it will prepare a project management policy on how DOE offices should respond to recommendations from independent reviews. DOE stated that it will prepare this policy by December 2016 and update DOE Order 413.3B at the next available opportunity.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our report examined the extent to which the Department of Energy (DOE) (1) met its initial cost and schedule estimates for restarting waste disposal operations at the Waste Isolation Pilot Plant (WIPP) and (2) followed best practices in analyzing and selecting an alternative for the new ventilation system. To address both objectives, we conducted a site visit to WIPP in January 2015. During the site visit we obtained documentation and interviewed officials from DOE’s Office of Environmental Management (EM), which is responsible for oversight of WIPP and exercises this responsibility primarily through its Carlsbad Field Office. We also interviewed representatives at WIPP from Nuclear Waste Partnership LLC (NWP), which is the private contractor that manages and operates WIPP for DOE.

To examine the extent to which DOE met its initial cost and schedule estimates for restarting waste disposal operations, we compared DOE’s initial cost and schedule estimates to restart limited waste disposal operations contained in its February 2015 WIPP recovery project management baseline with DOE’s revised estimates in its January 2016 integrated project management baseline for WIPP. We reviewed DOE’s budget justification supporting the President’s fiscal year 2016 budget request to Congress from February 2015 and the amounts DOE received for WIPP in the 2016 Consolidated Appropriations Act. We reviewed DOE’s reports on the reasons it exceeded the initial estimates for restarting waste disposal operations and DOE’s risk management plans for WIPP recovery prepared by NWP and used in developing the February 2015 estimates. We also compared DOE’s initial cost and schedule estimates to restart limited waste disposal operations from the February 2015 WIPP recovery project management baseline with the best practices described in our cost and schedule guides that identified the characteristics of high-quality, reliable cost and schedule estimates because these were the approved estimates when we conducted our analysis. Specifically, we compared DOE’s initial WIPP recovery cost estimate presented in DOE’s project management baseline document and supporting documents and data with the best practices in our Cost Estimating and Assessment Guide. In addition, we compared DOE’s initial WIPP recovery schedule presented in DOE’s project management baseline document and supporting documents and data with the best practices in our Schedule Assessment Guide. We interviewed the Carlsbad Field Office officials who oversee the recovery project and NWP’s cost estimator and scheduler. We provided a draft of our cost and schedule assessments to the Carlsbad Field Office and NWP and revised the draft, as appropriate, after discussing our assessment with the federal officials and the contractor. We reviewed documentation on the revised (January 2016) baseline and interviewed DOE and NWP officials on the approach followed to develop the revised cost and schedule estimates to restart WIPP’s waste disposal operations in the baseline, but we did not assess the revised estimates against the best practices because of the time frame of our review.

To examine the extent to which DOE followed best practices in analyzing and selecting an alternative for WIPP’s new ventilation system, we used as criteria the best practices for conducting an AOA identified in GAO-15-37 issued in December 2014. GAO developed the best practices identified in this report by reviewing AOA policies and guidance used by seven public and private-sector entities with experience in the AOA process, and verified these practices with subject-matter experts. DOE’s AOA process for the WIPP ventilation system consisted of three elements: NWP’s January 2015 AOA that resulted in the selection of two preferred alternatives proposed for further analysis, a second AOA completed by Trinity Engineering Associates (Trinity) in October 2015 which resulted in a single preferred alternative proposed to DOE, and DOE’s final alternative evaluation process led by the Carlsbad Field Office that considered the three alternatives proposed by the initial studies. Our analysis assessed the overall AOA process considering each element. We compared the process with the best practices and determined a score for the overall process. We reviewed project documentation from the Carlsbad Field Office, NWP, and Trinity and interviewed Carlsbad Field Office officials and NWP representatives in charge of the AOA. In addition, in October 2015, as we were conducting our engagement, EM and DOE’s Office of Project Management Oversight and Assessments completed separate assessments of the WIPP AOA for the new ventilation system. We reviewed documentation of these reviews and interviewed the DOE officials who worked on them. We examined the extent that the independent assessments followed the best practice to have an independent entity assess the extent that a project’s AOA followed all best practices.

To score DOE’s AOA process, a GAO analyst examined the AOA documentation received from the agency and then assigned a score for each of the 24 best practices. Following this, a GAO AOA specialist independent of the engagement team reviewed the AOA documentation and the scores assigned by the analyst for accuracy and cross-checked the scores in all the analyses for consistency. We used a five-point scoring system to determine the extent to which DOE’s AOA process conformed to the best practices. After determining a score for each individual best practice, we calculated the score for each category—(1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative—by calculating the average of the scores for the best practices that fall under each category. If the score for each best practice and the average score for each category was “fully met” or “substantially met,” we concluded that the AOA process conformed to best practices and therefore could be considered reliable. In contrast, if the score was “partially met,” “minimally met,” or “not met,” we concluded that the AOA process did not conform to best practices and therefore could not be considered reliable. We shared our analysis with DOE officials and representatives from NWP for review and incorporated their technical comments and any additional evidence they provided in our analysis, as appropriate. We also interviewed officials from DOE’s Office of Enterprise Assessments, which provides internal oversight of DOE facilities; the Defense Nuclear Facilities Safety Board, which provides external oversight of DOE defense nuclear facilities; the U.S. Mine Safety and Health Administration, which provides external oversight of mining activities at WIPP; as well as the U.S. Environmental Protection Agency and the New Mexico Environment Department, both of which provide external regulation of WIPP.

We conducted this performance audit from November 2014 to August 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Assessment of DOE’s Cost Estimate for the WIPP Recovery Project Compared with Best Practices

Table 2 below assesses the Department of Energy’s (DOE) initial cost estimate in its February 2015 project management baseline for the Waste Isolation Pilot Plant (WIPP) recovery project compared to best practices for cost estimating published in the GAO Cost Estimating and Assessment Guide. Overall, of four characteristics of a high-quality and reliable cost estimate, DOE’s initial cost estimate substantially met two characteristics—comprehensive and well-documented—but partially met or minimally met the other two characteristics—accurate and credible.

Appendix III: Assessment of DOE’s Schedule for the WIPP Recovery Project Compared with Best Practices

Table 3 below assesses the Department of Energy’s (DOE) initial schedule in its February 2015 project management baseline for the Waste Isolation Pilot Plant (WIPP) recovery project compared to best practices for developing a schedule estimate published in the GAO Schedule Assessment Guide. Overall, DOE’s schedule substantially met two of the four characteristics of a high-quality schedule—comprehensive and controlled—but partially met the characteristics of well-constructed and credible.

Appendix IV: Assessment of DOE’s Analysis of Alternatives for the WIPP Ventilation System Project Compared with Best Practices

Table 4 below compares the Department of Energy’s (DOE) analysis of alternatives (AOA) process completed in December 2015 for the new permanent ventilation system at the Waste Isolation Pilot Plant (WIPP) to best practices for conducting an AOA published in a December 2014 GAO report. DOE’s overall AOA process consisted of an initial AOA by DOE’s WIPP contractor, Nuclear Waste Partnership LLC, completed in January 2015; a second AOA by a contractor unaffiliated with WIPP, Trinity Engineering Associates, completed in October 2015; and DOE’s final evaluation of the three preferred alternatives proposed by the contractors’ analyses. Overall, DOE’s AOA process fully met the category for identifying alternatives and partially or minimally met the other three categories of best practices—general principles, analyzing alternatives, and selecting the preferred alternative.

Appendix V: Comments from the Department of Energy

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

David C. Trimble, (202) 512-3841 or trimbled@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Daniel Feehan, Assistant Director; Cheryl Arvidson; Mark Braza; Richard P. Burkard; Jennifer Echard; Brian M. Friedman; Carly Gerbig; Jason Lee; Eli Lewine; Cynthia Norris; and Katrina Pekar-Carpenter made key contributions to this report.