Nuclear Waste: DOE Needs a Comprehensive Strategy and Guidance on Computer Models that Support Environmental Cleanup Decisions
Released: 2011-03-14T00:00:00

Letter

Background

Since the 1940s, DOE and its predecessors have operated a nationwide complex of facilities used to research, design, and manufacture nuclear weapons and related technologies. The environmental legacy of nuclear weapons production at dozens of these sites across the United States includes contaminated buildings, soil, water resources, and large volumes of radioactive and hazardous wastes that require treatment, storage, and disposal. The two sites that account for the majority of the costs of the cleanup effort—Hanford and SRS—were established in the 1940s and 1950s, respectively, to produce plutonium and other nuclear materials needed to manufacture nuclear weapons. EM manages cleanup projects at these and other sites that involve multiple activities to treat and dispose of a wide variety of radioactive and hazardous wastes.

Under federal and state laws, EM must clean up radioactive and hazardous substances in accordance with specified standards and regulatory requirements. EM carries out its cleanup activities under the requirements of federal environmental laws that include, among others, CERCLA and NEPA. CERCLA requires EM to evaluate the nature and extent of contamination at the sites and determine what cleanup remedies, if any, are necessary to protect human health and the environment into the future. Under NEPA, EM must prepare an environmental impact statement that assesses the environmental effects for a proposed agency action, all reasonable alternatives, and the no-action alternative. Under both the CERCLA and NEPA processes, EM analyzes proposed remedial action alternatives according to established criteria, invites and considers public comment, and prepares a Record of Decision that documents the selected agency action. If the cleanup method selected under CERCLA or NEPA will result in disposal of waste at an on-site disposal facility, EM is then required, under DOE’s radioactive waste management order—DOE Order 435.1—to ensure that waste management activities at each disposal facility are designed, constructed, and operated in a manner that protects workers, the public, and the environment. EM does this by completing a “performance assessment” of the selected cleanup method. To guide the implementation of selected cleanup methods, EM and its contractors may prepare a “system plan” that provides the basis for scheduling cleanup operations and preparing budget requests. For example, both Hanford and SRS have prepared system plans for treating and disposing of liquid radioactive waste stored in aging and leak-prone underground tanks.

EM officials at DOE headquarters and field offices oversee cleanup activities at the sites, but the work itself is carried out primarily by private firms contracting with DOE. EM applies different approaches to managing cleanup activities, depending on the type and extent of contamination and state and/or federal requirements with which it needs to comply. In addition, DOE has agreements with state and federal regulators, known as Federal Facility Agreements, to clean up the Hanford and SRS sites. The agreements lay out legally binding milestones for completing major steps in the waste treatment and cleanup process. EPA officials, as well as officials with environmental agencies in the states where EM sites are located, enforce applicable federal and state environmental laws and oversee and advise EM on its cleanup efforts.

One tool EM uses in support of cleanup decision analyses is computer modeling. Although the computer models used across EM sites vary, they have certain common characteristics. In general, computer models are based on mathematical formulas that are intended to reflect physical, biogeochemical, mechanical, or thermal processes in simplified ways. For example, a computer model can simulate the movement of contamination through the soil and groundwater or simulate the transfer of high-level radioactive waste from underground storage tanks to facilities where the waste will be treated. Appendix II details the key computer models used in the cleanup decisions we reviewed at Hanford and SRS.

Computer Models Provide Critical Information to EM’s Environmental Cleanup Decision- Making Process

EM uses computer models to provide critical information for its decision- making process. First, computer models provide information that EM uses to analyze the effectiveness of alternative actions to clean up radioactive waste. Second, once a cleanup strategy has been selected, computer modeling provides information that EM needs to assess the performance of the selected cleanup strategy in reducing risks to human health and the environment. Third, EM uses computer models to simulate operations in the cleanup process, providing the basis for planning cleanup efforts and for making annual budget requests.

EM Uses Computer Models to Analyze the Potential Effectiveness of Cleanup Alternatives

EM’s decision making for its cleanup efforts is based on meeting federal and state requirements; input from state, local, and regional stakeholders; and other considerations, including the costs of cleanup actions. Computer models provide critical information that EM needs to assess compliance with regulatory requirements when seeking to identify and select alternatives for cleaning up radioactive and hazardous wastes, as well as contaminated soil and groundwater at its sites.

EM’s cleanup decisions are guided by several federal and state environmental laws, including CERCLA and NEPA, which both set forth processes related to cleanup decisions. In the case of CERCLA, EM determines the nature and extent of the contamination, assesses various cleanup alternatives, and selects the best alternatives according to evaluation criteria that include, among other things, protection of human health and the environment, ease of implementing the alternative, state and community acceptance, and cost. To accomplish these steps, EM uses computer modeling to, among other things, simulate the movement of contaminants through soil and groundwater over many years assuming no cleanup action is taken. Projected contamination levels, migration pathways, and contamination travel timelines are provided by simulations and are evaluated to determine whether regulatory standards will likely be exceeded in the future. If action is needed, then modeling simulations may be conducted for a number of different cleanup alternatives. For example, EM used modeling to assess contamination and the potential effectiveness of various cleanup strategies at SRS’s C-Area Burning/Rubble Pit. Used during the 1960s as a trash pit to dispose of organic solvents, waste oils, paper, plastics, and rubble, SRS burned the contents of the pit periodically to reduce its volume. Eventually, SRS used the pit for the disposal of inert rubble, finally covering it with two feet of soil in the early 1980s. However, the disposal of these materials and periodic burning resulted in hazardous substance contamination of the surrounding soil and groundwater. Between 1999 and 2004, EM implemented several actions to clean up the majority of the area’s contamination. Following these actions, EM used computer models to simulate the movement of the remaining contamination through the soil and groundwater over the next 1,000 years. Information provided by this modeling helped EM to identify the remaining risks to human health and the environment and to identify actions to clean up the remaining contamination. Using this information, in conjunction with other criteria such as additional site data, input from federal and state regulators and the public, and the availability of an appropriate cleanup technology, EM selected a final cleanup remedy. This remedy, which is ongoing and combines several different cleanup technologies, was estimated in 2008 to cost, in present-worth dollars, about $1.9 million over a 70-year period.

In implementing CERCLA, DOE focuses on discrete facilities or areas within a site that are being remediated, making limited assessments of cumulative impacts. By contrast, under NEPA, EM generally prepares environmental impact statements that assess the environmental impacts— including cumulative impacts—of a proposed cleanup action, all reasonable alternatives, and taking no action. For example, the environmental impact statement for closing underground liquid radioactive waste tanks at Hanford—which, as of November 2010 was still in draft form—includes an analysis of the potential environmental impact of various options for treating and disposing of about 55 million gallons of mixed radioactive and hazardous waste and closing 149 underground radioactive waste tanks. The draft environmental impact statement includes an analysis of 11 tank waste treatment and closure alternatives, including a no-action alternative. These alternatives range in cost from about $3 billion to nearly $252 billion, excluding the costs associated with the final disposal of the treated waste.

In the draft environmental impact statement, EM used computer models to simulate the movement of contamination through soil and groundwater over a period of 10,000 years for each of the cleanup alternatives. As with CERCLA modeling, the results of the computer models were used to estimate the remaining risks to human health and the environment following the completion of each cleanup alternative and these risks were then compared with requirements. The results of these models will be used along with other information such as input from regulators and the public and the costs of each alternative when EM selects the alternative it will eventually implement.

EM Uses Computer Models to Assess the Performance of Selected Cleanup Activities

After a particular cleanup alternative is selected, EM also uses computer modeling to demonstrate that the cleanup activity will result in reduced future contamination levels that meet regulatory requirements. If the cleanup method selected under CERCLA or NEPA will result in disposal of waste at an on-site disposal facility, EM is then required, under DOE’s radioactive waste management order—DOE Order 435.1—to ensure that waste management activities at each disposal facility are designed, constructed, and operated in a manner that protects workers, the public, and the environment. To meet the requirements of the order, EM completes a “performance assessment” of the selected cleanup method. Under the order, this performance assessment is to document that the disposal facility is designed, constructed, and operated in a manner that protects workers, the public, and the environment. The performance assessment also is to project the release of contamination into the soil and groundwater from a site after cleanup and must include calculations of potential chemical doses to members of the public in the future.

For example, in March 2010, SRS issued a performance assessment of a cleanup and closure strategy for a group of 20 underground liquid radioactive waste tanks, known as the F-Tank Farm. The performance assessment evaluated closing the underground waste tanks and filling them with a cement-like substance called grout—the alternative selected following completion of SRS’s 2002 environmental impact statement. Computer modeling was used extensively to prepare this performance assessment. Specifically, computer modeling was performed using two different types of models. The first computer model was used to perform human health and environmental risk calculations and to calculate radiation doses that could be compared to the maximum level allowed by federal and state requirements. The second model was used to analyze sensitivities and uncertainties in the results of the first model.

EM Uses Computer Modeling to Help Plan and Budget Cleanups

EM also uses computer models for lifecycle planning, scheduling, and budgeting for its cleanup activities. Computer models provide important information that EM and its contractors use to develop system plans that outline the schedules for cleanup activities at EM sites. Outputs from computer models and databases are used to create tables, charts, and schedules that are published in the system plans and inform annual budget requests for cleanup activities.

For example, at Hanford, a computer model known as the Hanford Tank Waste Operations Simulator is designed to track the retrieval and treatment of over 55 million gallons of radioactive waste held in underground storage tanks. According to the most recent Hanford tank waste system plan, which was issued in November 2010, the model projects the chemical and radiological characteristics of batches of waste that are to be sent to a $12.2 billion waste treatment plant that is being built at Hanford to treat this waste. The model also provides scheduling information the contractor uses to project near- and long-term costs and schedules.

Similarly, SRS uses a computer model known as SpaceMan Plus™ to support the site’s liquid waste system plan, which was issued in January 2010. For example, project work schedules for SRS’s tank waste program are guided by this model. The model also simulates how the tank farms integrate with waste processing facilities and tracks the movement of waste throughout the liquid waste system. Output from the model was used to provide tables and schedules found in the appendixes of SRS’s system plan that details the specific cleanup activities that are to be accomplished. These tables and schedules are used as part of the basis for determining the costs of completing those activities. This information, in turn, allows DOE and its contractors to generate annual budget requests.

EM Has General Quality Policies for Its Computer Models, but It Has Not Regularly Assessed Contractors’ Implementation of Quality Assurance Procedures

Although EM uses general departmental quality assurance policies and standards that apply to computer models and relies on contractors to implement specific procedures that reflect these policies and standards, these policies and standards do not specifically provide guidance on ensuring the quality of the computer models used in cleanup decisions. Moreover, EM officials have not regularly performed periodic quality assurance assessments, as required by DOE policy to oversee contractors’ development and use of cleanup models and the models’ associated software. In addition, DOE and others have identified quality assurance problems. For example, the state of Washington has cited flaws in a model EM uses to analyze soil and groundwater contamination and has told EM that it will no longer accept the use of this model for chemical exposure analysis at Hanford.

Although EM Has General Quality Assurance Standards, Its Oversight Is Not Sufficient to Ensure the Quality of Cleanup Models

DOE addresses quality through various departmental policies and industry standards; however these policies and standards do not specifically provide guidance on ensuring the quality of the computer models used in cleanup decisions. Specifically, DOE’s primary quality assurance policy— DOE Order 414.1C—provides general requirements EM and its contractors must meet to ensure all work at the cleanup sites is carried out correctly and effectively, including the development and use of computer models. These requirements include developing a quality assurance program, training staff how to check the quality of their work, and providing for independent assessments of quality. A manual accompanying this order describes acceptable, nonmandatory methods for specifically ensuring quality of “safety software.” Safety software is described in the manual as software used to design, manage, or support nuclear facilities. However, the manual is less clear on how to assure quality in computer models. Furthermore, it does not clearly address the use of computer software not considered as safety software, such as those used by computer models that support DOE’s cleanup decisions.

DOE’s quality assurance order also requires contractors to select and comply with an appropriate set of industry standards for all work, including computer modeling. One common set of standards was developed by the American Society of Mechanical Engineers and provides the requirements necessary to ensure safety in nuclear facilities, including the development and validation of computer models and software that is used to design and operate such facilities. Initially, the American Society of Mechanical Engineers standards were not mandatory for computer models and software used for cleanup decisions, many of which are considered nonsafety software. These standards were but one of many standards that contractors could choose to use. However, as of November 2008, EM made the American Society of Mechanical Engineers standards mandatory for all cleanup activities, including modeling.

EM’s contractors are to implement DOE’s quality assurance requirements using specific policies and procedures they develop. The specifics of implementation vary from contractor to contractor. In the case of computer software quality, a contractor is to include procedures for testing and validating the software, ensuring changes to software are properly documented, and correcting any errors. EM allows its contractors to take a “graded approach” to quality procedures for computer software, which means the contractor may adjust the rigor of the quality procedures to match the importance of the software to overall operations. According to documents we reviewed, computer software that controls systems in a nuclear facility, for example, would require more rigorous quality procedures than an administrative payroll system, as any failure in the software controlling a nuclear facility could result in potentially hazardous consequences to workers, the public, and/or the environment.

EM is to oversee its contractors’ implementation of quality standards for computer models by performing periodic quality assurance assessments, according to DOE’s quality assurance order. These quality assurance assessments are intended to ensure that computer models meet DOE and accepted industry quality standards. In our review of eight cleanup decisions at Hanford and SRS, we found EM had conducted only three quality assurance assessments that addressed quality standards for the models used in those decisions. For example, for three of the four decisions we reviewed at SRS, DOE officials at SRS could not provide quality assurance assessments that specifically addressed whether the models used in those decision processes met DOE’s quality assurance requirements. DOE officials at SRS provided three general quality assurance assessments, but these quality assurance assessments did not specifically look at the cleanup models. In contrast, the models for a March 2010 performance assessment selecting a cleanup strategy to close underground liquid waste tanks at SRS did receive a quality assurance assessment by a DOE headquarters group established to review performance assessment decisions. In particular, as part of the review, among other things, the DOE group conducted a quality assurance assessment that evaluated the quality of the computer models used in the performance assessment and the degree to which the models complied with DOE requirements and industry standards. A DOE quality assurance official at SRS noted that the site relies primarily on its contractors to perform quality assurance assessments of computer models and their associated software.

Similarly, in our review of four cleanup decisions at Hanford, we found that EM had performed assessments that addressed quality standards for the models used in those decisions in only two cases. In fact, one quality assurance assessment was only undertaken after a contractor discovered data quality errors in 2005 in a computer model used to support a prior environmental impact statement at Hanford. According to a DOE quality assurance manager at Hanford, his office conducts quality assurance assessments primarily on those computer models and the associated software for which the failure would result in significant safety consequences to workers, the public, and/or the environment.

Some Reviews Have Revealed Quality Assurance Problems

Concerns have been raised by DOE and others that EM does not have complete assurance of the quality of the models. For example: Citing a number of flaws in a model DOE uses to analyze soil and groundwater contamination at Hanford, the Washington state Department of Ecology told DOE in February 2010 that it would no longer accept the use of this model for chemical exposure analysis at Hanford. For example, Ecology cited previous concerns that the model was not robust enough to capture complexities of the movement of contamination through the subsurface soil. We found that DOE had conducted no specific quality assurance reviews on the model and its associated software.

EM headquarters officials conducted two technical reviews in 2009 of planning models used for tank waste operations at Hanford and SRS. The review of the Hanford planning model found that the model has limited ability to sufficiently predict the composition of the contaminated waste as it is prepared for the treatment processes. The review team cautioned that this limitation raised a significant risk that, when actual waste treatment operations started at the site, the waste may not meet the acceptance requirements for processing by Hanford’s treatment facility. In addition, the review of SRS’s planning model found that, although the data the model provided on tank waste operations were reasonable, the model did not have the ability to optimize operating scenarios, which hampered the site’s long-term planning abilities.

A March 2010 independent review commissioned by a Hanford citizen’s group raised concerns about a model used in the preparation of a draft environmental impact statement of alternatives for closing Hanford’s waste tanks. These concerns, based on reviewing the draft statement, included insufficient documentation of the quality assurance processes followed for the model and that modeling uncertainties were inadequately quantified. The review concluded that the environmental impact statement was insufficiently precise to be used to make a cleanup decision.

Where DOE has conducted quality assurance assessments, it has found that contractors did not always implement quality requirements consistently. Furthermore, in their own internal reviews, contractors have noted problems with the implementation of quality assurance requirements. Problems noted in DOE’s and contractors’ quality assurance assessments include: Inadequate documentation. A 2007 software quality review conducted by DOE at Hanford found implementation problems, including inadequate documentation and improper training for personnel in quality procedures. At SRS, two general software quality assurance reviews performed by DOE in 2004 found that while contractors generally met quality requirements, documentation was sometimes lacking or improperly prepared. A similar 2007 DOE review at SRS found a good software quality program overall, but listed a number of deficiencies including inadequate software plans and procedures.

Not following correct procedures. A 2007 DOE review of a Hanford contractor’s software quality assurance program found, among other things, that not all contractor personnel fully understood software quality requirements. The report stated that, although software quality assurance training had been provided, personnel did not follow procedures in managing, maintaining, and overseeing software quality. For example, the report cited an example of a spreadsheet in which data input cells were not properly locked, in violation of procedures. In addition, the report noted that software documentation was not periodically updated, as required, because staff did not fully understand the procedures.

Incorrect quality assurance grading. In some cases, contractors did not always correctly determine the level of rigor needed to ensure the quality of computer models and their associated software. For example, a 2007 internal contractor review at Hanford found that 23 of 138 software codes registered in a central repository were incorrectly designated as nonsafety software, when in fact they should have been considered safety software. As a result, the quality assurance procedures appropriate for a given level of risk may not have always been applied.

EM Does Not Have an Overall Strategy and Guidance for Managing Its Cleanup Models

Although EM has recently begun some efforts to promote consistency in the use of models across its various sites, these efforts are still in early stages and, to date, some have had limited involvement of modeling officials at the sites and federal, state, and local stakeholders who are affected by decisions made using the output of computer models. In addition, these efforts are not part of a comprehensive, coordinated effort to improve the management of computer models across EM. In the absence of such a strategy, EM also does not have overarching guidance promoting consistency in modeling management, development, and use across EM’s sites.

EM Has Some Initiatives to Improve Management of Its Cleanup Models, but They Are Not Part of a Comprehensive, Coordinated Strategy

EM has begun some efforts to improve the use of computer models across its various sites. For example, EM, in fiscal year 2010, began developing a set of state-of-the-art computer models to support soil and groundwater cleanup across the nuclear weapons complex. According to EM officials and documentation they provided, this initiative, called the Advanced Simulation Capability for Environmental Management, will allow EM to provide more sophisticated analysis of soil and groundwater contamination for cleanup decisions. Although the initiative’s director told us that the goal is to encourage all sites to use these models for all of their soil and groundwater analysis, he noted that there are no plans to make using these models mandatory.

Moreover, SRS has created a forum for improving consistency in groundwater computer modeling performed at the site. According to the charter document, the forum, called the Groundwater Modeling Consistency Team, was formed in 2006 following the discovery of inconsistencies in the data used in groundwater computer modeling conducted at Hanford in support of the preparation of an environmental impact statement under NEPA. The group, which is made up of DOE and contractor officials, reviews software codes, model inputs, and model assumptions to promote sitewide consistency in the management of computer models.

Although these efforts may help improve EM’s use of computer models, they are largely still in early stages. In addition, according to EM officials, some of these efforts have, to date, had limited involvement of modeling officials at EM’s sites and of federal, state, and local stakeholders who are affected by decisions made using the output of computer models. Furthermore, they are not part of a comprehensive, coordinated effort to improve the consistency of computer models and reduce duplication across EM’s various sites. For example, we found that different models are used to perform similar functions not only between EM sites, but also within sites. At SRS, one contractor uses a set of models to perform soil and groundwater analyses when evaluating the potential effectiveness of cleanup alternatives under CERCLA and NEPA, while another contractor uses a different set of models to perform similar analyses for performance assessments under DOE’s radioactive waste management order. Each contractor has its own set of procedures for developing and using each computer model. Officials from both contractors told us that they use different models because state and federal regulators have only approved the use of certain models for specific types of cleanup decisions. Issues with consistency and duplication of effort in the use of computer models have also been noted by others. For example, a February 2010 DOE review noted that five major DOE sites use 28 different models to analyze groundwater and subsurface contamination when preparing performance assessments under DOE’s radioactive waste management order. DOE officials told us that past modeling practices have resulted in conflicting assumptions and data sets, as well as different approaches to uncertainty analyses. In addition, a September 2009 DOE technical review of the Hanford tank waste modeling system raised concerns that two models at Hanford that share data use different assumptions that could lead to inconsistencies between the two. As a result, the Hanford waste treatment system plan, which is based on the output of one of these models, may not reflect the most current information.

In contrast, other federal agencies and DOE offices have taken steps to improve consistency and reduce duplication as part of a comprehensive, coordinated strategy to manage the use of computer models. For example, EPA organized a Center for Regulatory Environmental Modeling in 2000 as part of a centralized effort to bring consistency to model development, evaluation, and usage across the agency. The Center brings together senior managers, modelers, and scientists from across the agency to address modeling issues. Among its tasks are to help the agency (1) establish and implement criteria so that model-based decisions satisfy regulatory requirements; (2) implement best management practices to use models consistently and appropriately; (3) facilitate information exchange among model developers and users so models can be continuously improved; and (4) prepare for the next generation of environmental models. According to a DOE official, EM does not have a central coordination point similar to EPA’s.

Within DOE, the Office of Nuclear Energy recently established an initiative—the Nuclear Energy Modeling and Simulation Energy Innovation Hub—that provides a centralized forum for nuclear energy modelers. According to the director of the Office of Nuclear Energy’s Office of Advanced Modeling and Simulation, the hub will provide a more centrally coordinated effort to bring together modeling and simulation expertise to address issues associated with the next generation of nuclear reactors. Similar comprehensive, coordinated efforts are lacking within EM and, as a result, EM may be losing opportunities to improve the quality of its models, reduce duplication, keep abreast of emerging computer modeling and cleanup technologies, and share lessons learned across EM’s sites.

Other Federal Agencies and DOE Offices Have Recognized the Need for Comprehensive Modeling Guidance

The need for specific guidance for ensuring the careful management of computer models used in decision making is not new. As early as 1976, we reported on the government’s use of computer models and found that the lack of guidance contributed to ineffective and inefficient use of computer models. We noted that guidance should define the problem to be solved, specify the assumptions and limitations of the model, and provide methods to test whether the model reasonably describes the physical system it is modeling.

More recently, a 2007 National Research Council study of modeling at EPA laid out guidelines to improve environmental regulatory computer modeling. The study noted that adoption of a comprehensive strategy for evaluating and refining EPA’s models could help the agency add credibility to decisions based on modeling results. It also noted several key principles to follow for model development, evaluation, and selection. Moreover, the study recommended that peer review be considered as an important tool for improving model quality. According to the study, a peer review should entail not only an evaluation of the model and its output, but also a review of the model’s origin and its history. The study also made recommendations on quantifying and communicating uncertainty in model results to better communicate a model’s limitations to stakeholders affected by decisions made using the results of computer models.

EPA has taken action to develop specific guidance, issuing a guide in 2009 addressing the management, development, and use of computer modeling used in making environmental regulatory decisions. In this guidance, EPA developed a set of recommended best practices to help mode lers effectively use computer models. The guidance defines the role of computer models in the public policy process, discusses appropriate ways of dealing with uncertainty, establishes criteria for peer review, and addresses quality assurance procedures for computer modeling.

Even within DOE, another office outside of EM has recognized the need for specific guidance for managing computer models. Specifically, DOE’s Office of Civilian Radioactive Waste Management specified in its quality assurance requirements several requirements for computer models. These requirements included clearly defining the model’s objective, documenting alternative models that could be used and the rationales for not using them, and discussed a model’s limitations and uncertainties. In addition, the office specified in its requirements that, among other things, a computer model receive a technical review through a peer review or publication in a professional journal.

Although the importance of comprehensive guidelines for managing computer models is well established, according to its officials, EM does not have such overarching guidance. As previously discussed, EM does have a manual accompanying its quality assurance order that describes acceptable methods for specifically ensuring the quality of safety software. However, the manual does not generally address models used in cleanup decisions. EM also has guidance addressing the management of computer models used in conducting performance assessments under its radioactive waste management order. Specifically, a DOE headquarters group that is charged with reviewing decisions made under this order—the Low-Level Waste Disposal Facility Federal Review Group—has developed a manual that contains guidance on, for example, ensuring that input data to computer models are described and are traceable to sources derived from, among other things, field data from the site and referenced literature that is applicable to the site. However, this guidance does not apply to computer models used to analyze the potential effectiveness of cleanup alternatives under CERCLA or NEPA or to computer models used for planning, scheduling, and budgeting purposes. As a result, computer models developed at various DOE sites do not have consistent criteria to define the role of the model in the decision-making process, consistent ways of dealing with uncertainties and a model’s limitations, and mechanisms to ensure computer model quality, such as quality assurance assessments and peer review.

Conclusions

EM’s computer models provide critical information that is needed to make significant decisions about how to clean up the radioactive and hazardous legacy waste across the country. However, EM’s oversight of the quality of these models and its management of the development, evaluation, and use of the models has not always been commensurate with the models’ importance. Because the decisions EM makes must protect human health and the environment for thousands of years into the future, it is critical that the models on which EM bases its decisions are of the highest quality possible. In addition, because these cleanup efforts will take decades and cost billions of dollars, it is also important that models used for planning, scheduling, and budgeting purposes provide the most accurate data possible for EM and Congress to make informed decisions on cleanup activities.

EM’s failure to fully oversee its contractors’ implementation of quality assurance procedures has led to a reduced level of confidence that the models reasonably represent the conditions they are meant to simulate. In several cases, we found necessary quality assurance reviews were not conducted. In others, reviews found that quality assurance procedures were inadequately implemented. Because existing quality assurance requirements that are applied to EM’s computer models have not been adequately implemented and, in some cases, are insufficiently understood by its contractors, EM and its contractors do not have an effective mechanism to provide the public and other EM stakeholders with assurance of a model’s quality.

To its credit, EM is beginning to undertake efforts to improve the consistency of models across the nuclear weapons complex. However, some of these efforts are still in their infancy, and it remains to be seen whether any improvements in EM’s management of its models will result. We recognize that every site has its unique conditions and challenges and that a one-size-fits-all approach to modeling would not be appropriate. Nevertheless, there is room for additional consistency in model development and implementation, as well as a mechanism for sharing lessons learned among DOE’s various sites. For a number of years, other federal agencies and offices within DOE have recognized the importance of a comprehensive guidance for managing computer models. Without a comprehensive strategy and modeling guidance, EM may miss opportunities to improve the quality of computer models, promote consistency, reduce duplication across DOE sites, and share lessons learned.

Recommendations for Executive Action

To help EM increase confidence in the quality of information provided to the public and its stakeholders resulting from the use of computer modeling, we recommend the Secretary of Energy take the following three actions: Clarify specific quality assurance requirements for computer models used in environmental cleanup decisions, including to analyze the potential effectiveness of cleanup alternatives, assess the performance of selected cleanup activities, and assist in planning and budgeting cleanup activities.

Ensure that the models are assessed for compliance with these requirements.

Develop a comprehensive strategy and guidance for the management of computer models to promote consistency, reduce duplication, and ensure sharing of lessons learned.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE for its review and comment. In its written comments, DOE agreed with our recommendations and stated that modeling is an important component of management analysis and decision making for the department. DOE noted that it is committed to continuous improvement in model development and application and commented that our recommendations will strengthen its modeling efforts.

DOE stated in its comments that it disagreed with the draft report’s assertion that its directives and standards fall short for the development and management of computer models. DOE commented that its quality assurance directives apply directly to the development, coding, and validation of safety and nonsafety computer models used in cleanup decisions and that EM has interpreted and applied these directives and accompanying standards to develop its quality program. We agree with DOE, and our draft report noted, that DOE addresses quality through various departmental policies and industry standards. However, these directives do not provide specific guidance to EM on assuring quality of the cleanup models themselves, guidance that other agencies and offices within DOE have developed. In particular, DOE’s primary quality assurance policy—DOE Order 414.1C—addresses general standards that EM and its contractors must meet to ensure all work at its sites is carried out effectively, but is vague on the specific steps that must be followed to ensure the quality of models used in cleanup decisions. In addition, as our draft report noted, a manual accompanying this order describes acceptable, nonmandatory methods for specifically ensuring quality of safety software. However, the manual is less clear on the use of computer software not considered as safety software, such as those used by computer models that support DOE’s cleanup decisions. Our recommendation that DOE clarify the specific quality assurance requirements for computer models used in environmental cleanup decisions is intended to address these problems.

DOE’s comments also provided additional information on the department’s oversight of computer models, initiatives it is undertaking to improve its modeling efforts, and the specific steps it plans to take to address our recommendations. DOE also provided technical comments that we incorporated in the report as appropriate. DOE’s written comments are presented in appendix III.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees; the Secretary of Energy; the Director, Office of Management and Budget; and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions regarding this report, please contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Scope and Methodology

To determine how the Department of Energy’s (DOE) Office of Environmental Management (EM) uses computer modeling in cleanup decisions, we focused on cleanup decisions EM has made at its Hanford Site in Washington state and Savannah River Site (SRS) in South Carolina because together these two sites account for more than one-half of EM’s annual cleanup spending and approximately 60 percent of the total estimated cost of approximately $275 billion to $329 billion to clean up the entire nuclear weapons complex. We focused our review on decisions made in two major areas that represent the largest and most significant elements of the cleanup program at these two sites. The first is cleanup of radioactive and hazardous waste stored in underground tanks, which DOE has determined poses the most significant environmental safety and health threat in the cleanup program. DOE estimates cleaning up tank waste at the sites will cost between $87 billion and $117 billion, making it the largest cost element of EM’s cleanup program. Second, both sites have significant contamination to soil and groundwater, which DOE estimates will cost more than $12 billion to remediate. For each site, we selected three types of decisions that were representative of major decisions made at these sites between 2002 and 2010—(1) decisions made under environmental statutes, including the Comprehensive Environmental Response, Compensation, and Liability Act of 1980, as amended (CERCLA)—which addresses specific environmental remediation solutions for a cleanup site—and the National Environmental Policy Act, as amended (NEPA)—under which DOE evaluates the impacts to human health and the environment of proposed cleanup strategies and possible alternatives; (2) performance assessments under DOE orders governing radioactive waste management; and (3) cleanup budgeting and planning decisions. We reviewed publicly available information from regulators and interviewed DOE officials and contractor staff to identify the most recent decisions for each of the three types of decisions selected for review at each site. We reviewed these decisions to identify the most recent decision that included the use of computer modeling. We then selected, based on input by EM officials, the main models used to support these decisions at the two sites. We visited both Hanford and SRS and spoke with both EM officials and contractor staff there to better understand the use of models in planning and cleanup decisions and DOE’s oversight of the models. We obtained demonstrations of these models, as well as information on how they were used in decision making. We obtained and reviewed the decision documents, as well as modeling studies, notes of meetings between DOE and its regulators to develop models, and other documentation showing how the models were used in decisions. We interviewed officials from DOE headquarters and the two sites, as well as contractor staff, to determine how the models work and how they were used in these decisions. We analyzed this information to determine how the results of computer models were used in making cleanup decisions, the importance of modeling in the selection of a cleanup strategy, and other factors that contributed to the selection of a cleanup strategy.

To evaluate how EM determines the quality of the computer models used in cleanup decision making, we obtained and reviewed documentation showing the standards the models were required to meet. We gathered documentation on DOE standards, as well as policies and procedures from contractors overseeing the models. We discussed computer model and software standards with EM officials from EM’s sites, contractors at the sites, and headquarters officials. We also interviewed officials from the Defense Nuclear Facilities Safety Board, the National Research Council, the Environmental Protection Agency, and the Washington state Department of Ecology about existing standards for the use and implementation of computer modeling and its associated software. We analyzed EM policies and contractor procedures to determine what quality assurance standards exist to address the quality of computer models. We also requested from EM and its contractors all assessments that were conducted on computer models used in the decisions we were reviewing, indicating whether quality standards were met. In general, the assessments we reviewed were largely conducted by the contractors, regulators, or external sources, such as consultants. These reviews ranged from contractor-performed assessments of the implementation of quality standards for software, to federal and state regulator comments on the modeling output used to develop alternatives in a regulatory package, to an outside consultant-performed review on the appropriateness of modeling for selecting a preferred alternative from an environmental impact statement prepared under NEPA. We analyzed these assessments to understand the level of oversight EM provided to assure model and software quality, as well as the extent to which contractors were implementing quality procedures.

To address EM’s overall strategy for managing computer models that are used in cleanup decisions, we interviewed DOE officials from headquarters and from each site. We also interviewed officials from the Environmental Protection Agency, National Research Council, DOE’s Office of Nuclear Energy, and DOE’s Office of Civilian Radioactive Waste Management about the implementation of computer modeling guidance and modeling coordination strategies. We reviewed modeling guidance from these organizations, as well as from the Office of Management and Budget. We focused our review on model quality assurance standards and the use of models in decision making, not on the quality of the models themselves or of their output.

We conducted this performance audit from October 2009 to February 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Functions of Key Models Used in Cleanup Decisions GAO Reviewed at EM’s Hanford and Savannah River Sites

Models used at Hanford

Appendix III: Comments from the Department of Energy

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, Ryan T. Coles, Assistant Director; Ivelisse Aviles; Mark Braza; Dan Feehan; Nancy Kintner-Meyer; Jonathan Kucskar; Mehrzad Nadji; Kathryn Pedalino; Thomas C. Perry; and Benjamin Shouse made key contributions to this report.

Related GAO Products

Nuclear Waste: Actions Needed to Address Persistent Concerns with Efforts to Close Underground Radioactive Waste Tanks at DOE’s Savannah River Site. GAO-10-816. Washington, D.C.: September 14, 2010.

Recovery Act: Most DOE Cleanup Projects Appear to Be Meeting Cost and Schedule Targets, but Assessing Impact of Spending Remains a Challenge. GAO-10-784. Washington, D.C.: July 29, 2010.

Department of Energy: Actions Needed to Develop High-Quality Cost Estimates for Construction and Environmental Cleanup Projects. GAO-10-199. Washington, D.C.: January 14, 2010.

Nuclear Waste: Uncertainties and Questions about Costs and Risks Persist with DOE’s Tank Waste Cleanup Strategy at Hanford. GAO-09-913. Washington, D.C.: September 30, 2009.

Department of Energy: Contract and Project Management Concerns at the National Nuclear Security Administration and Office of Environmental Management. GAO-09-406T. Washington, D.C.: March 4, 2009.

Nuclear Waste: DOE Lacks Critical Information Needed to Assess Its Tank Management Strategy at Hanford. GAO-08-793. Washington, D.C.: June 30, 2008.

Hanford Waste Treatment Plant: Department of Energy Needs to Strengthen Controls over Contractor Payments and Project Assets. GAO-07-888. Washington, D.C.: July 20, 2007.

Nuclear Waste: DOE Should Reassess Whether the Bulk Vitrification Demonstration Project at Its Hanford Site Is Still Needed to Treat Radioactive Waste. GAO-07-762. Washington, D.C.: June 12, 2007.

Hanford Waste Treatment Plant: Contractor and DOE Management Problems Have Led to Higher Costs, Construction Delays, and Safety Concerns. GAO-06-602T. Washington, D.C.: April 6, 2006.

Nuclear Waste: Absence of Key Management Reforms on Hanford’s Cleanup Project Adds to Challenges of Achieving Cost and Schedule Goals. GAO-04-611. Washington, D.C.: June 9, 2004.

Nuclear Waste: Challenges to Achieving Potential Savings in DOE’s High-Level Waste Cleanup Program. GAO-03-593. Washington, D.C.: June 17, 2003.

Nuclear Waste: Department of Energy’s Hanford Tank Waste Project— Schedule, Cost, and Management Issues. GAO-RCED-99-13. Washington, D.C.: October 8, 1998.