Nuclear Waste: DOE Needs a Comprehensive Strategy and Guidance on Computer Models that Support Environmental Cleanup Decisions
Published: 2011-02-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Energy's (DOE) Office of Environmental Management (EM) is responsible for one of the world's largest cleanup programs: treatment and disposal of radioactive and hazardous waste created as a by-product of nuclear weapons production and energy research at sites across the country, such as EM's Hanford Site in Washington State and the Savannah River Site (SRS) in South Carolina. Computer models--which represent physical and biogeochemical processes as mathematical formulas--are one tool EM uses in the cleanups. GAO was asked to (1) describe how EM uses computer models in cleanup decisions; (2) evaluate how EM ensures the quality of its computer models; and (3) assess EM's overall strategy for managing its computer models. GAO analyzed the use of selected models in decisions at Hanford and SRS, reviewed numerous quality assurance documents, and interviewed DOE officials as well as contractors and regulators.
section_title: What GAO Found
paragraphs: EM uses computer models to support key cleanup decisions. Because the results of these decisions can cost billions of dollars to implement and take decades to complete, it is crucial that the models are of the highest quality. Computer models provide critical information to EM's cleanup decision- making process, specifically to: (1) Analyze the potential effectiveness of cleanup alternatives. For example, computer models at SRS simulate the movement of contaminants through soil and groundwater and provide information used to predict the effectiveness of various cleanup strategies in reducing radioactive and hazardous material contamination. (2) Assess the likely performance of selected cleanup activities. After a particular cleanup strategy is selected, EM uses computer modeling to demonstrate that the selected strategy will be designed, constructed, and operated in a manner that protects workers, the public, and the environment. (3) Assist in planning and budgeting cleanups. EM also uses computer models to support lifecycle planning, scheduling, and budgeting for its cleanup activities. For example, a Hanford computer model simulates the retrieval and treatment of radioactive waste held in underground tanks and provides information used to project costs and schedules. EM uses general departmental policies and industry standards for ensuring quality, but they are not specific to computer models used in cleanup decisions. EM has not regularly performed periodic quality assurance assessments, as required by DOE policy, to oversee contractors' development and use of cleanup models and the models' associated software. In our review of eight cleanup decisions at Hanford and SRS that used computer modeling as a critical source of information, GAO found EM conducted required assessments of the quality of computer models in only three cases. In addition, citing flaws in a model EM uses to analyze soil and groundwater contamination, regulators from Washington state have told EM that it will no longer accept the use of this model for chemical exposure analysis at Hanford. EM does not have an overall strategy for managing its computer models. EM has recently begun some efforts to promote consistency in the use of models. For example, it is developing a set of state-of-the-art computer models to support soil and groundwater cleanup decisions across its sites. However, these efforts are still in early stages and are not part of a comprehensive, coordinated effort. Furthermore, although other federal agencies and DOE offices have recognized the importance of comprehensive guidance on the appropriate procedures for managing computer models, EM does not have such overarching guidance. As a result, EM may miss opportunities to improve the quality of computer models, reduce duplication between DOE sites, and share lessons learned across the nuclear weapons complex.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOE (1) clarify specific quality assurance requirements for computer models used in environmental cleanup decision making; (2) ensure that the models are assessed for compliance with these requirements; and (3) develop a comprehensive strategy and guidance for managing its models. DOE agreed with our recommendations.

section_title: Letter
paragraphs: The Department of Energy’s (DOE) Office of Environmental Management  (EM) is responsible for one of the world’s largest environmental cleanup  programs, the treatment and disposal of radioactive and hazardous waste  created as a by-product of producing nuclear weapons and energy  research. The largest component of the cleanup mission is the treatment  and disposal of millions of gallons of highly radioactive waste stored in  aging and leak-prone underground tanks. In addition, radioactive and  hazardous contamination has migrated through the soil into the  groundwater, posing a significant threat to human health and the  environment. EM spends about $6 billion annually to clean up its sites. As  of February 2010, DOE estimated that the overall cost to complete the  entire cleanup mission will be between $275 billion and $329 billion. Two  DOE sites—the Hanford Site in southeastern Washington state and the  Savannah River Site (SRS) in South Carolina—account for more than one- half of these annual costs and about 60 percent of the total projected cost  of the overall cleanup of nuclear waste at DOE sites. As with nearly all of  DOE’s missions, the majority of the work at these two sites is performed  by private firms under contract with DOE.
One tool EM uses to help decide how to clean up this radioactive and  hazardous waste is computer simulation modeling—hereafter referred to  as computer models—where the behavior of physical and biogeochemical  processes are described through the use of mathematical formulas. For  example, computer models may be used to simulate a process such as the  transport of contamination through the soil and groundwater or to predict  how long it will take to empty waste tanks in a certain sequence. The  results from these models often contribute to the basis for cleanup  decisions that can cost hundreds of millions of dollars to implement.
The set of processes used to ensure the quality of computer software and  models—known as “quality assurance”—has been a concern in the past. In  2000 and again in 2002, the Defense Nuclear Facilities Safety Board raised  concerns that DOE did not have adequate controls to ensure the reliability  of software used in nuclear facilities. The Board noted that many systems  used to maintain safety in nuclear or hazardous facilities, such as  ventilation system controls, rely on the smooth operation of software to  prevent accidents. Another concern regarding software and modeling was  raised at Hanford in 2006, when a DOE headquarters review team found  that the absence of quality assurance oversight activities and the lack of  formal data validation and verification led to data inaccuracies in  modeling used to support the development of an environmental impact  statement. These problems prompted DOE to undertake a new modeling  effort, delaying the environmental impact statement.
In response to your request, this report (1) describes how EM uses  computer models in cleanup decisions; (2) evaluates how EM ensures the  quality of its computer models; and, (3) assesses EM’s overall strategy for  managing its computer models. To address these objectives, we gathered  and reviewed information on the types of cleanup decisions DOE has  made at Hanford and SRS. For each site, we selected examples of three  types of decisions that were representative of major decisions DOE has  made at these sites between 2002 and 2010—(1) decisions made under  environmental statutes, including the Comprehensive Environmental  Response, Compensation, and Liability Act of 1980, as amended  (CERCLA)—which addresses specific environmental remediation  solutions for a cleanup site—and the National Environmental Policy Act  (NEPA), as amended— under which DOE evaluates the impacts to human  health and the environment of proposed cleanup strategies and possible  alternatives; (2) performance assessments under DOE orders governing  radioactive waste management; and (3) budgeting and planning decisions  for liquid tank waste treatment and disposal. We then selected, based on  input from EM officials, the main models that were used to support these  decisions at the two sites. We obtained and reviewed documentation on  the computer models used and decisions made, and interviewed officials  from DOE headquarters to determine how the models were used in these  decisions. We analyzed this information to determine how the results of  computer models were used in making cleanup decisions and the  importance of the results. We also obtained and reviewed documentation  showing the standards the models were required to meet, as well as DOE,  contractor, and other quality assurance assessments indicating whether  these standards were met. We also interviewed officials from the  Environmental Protection Agency (EPA), the National Research Council,  the Defense Nuclear Facilities Safety Board, and other organizations about  existing standards for the use and implementation of computer modeling  software and modeling coordination strategies. We visited both Hanford  and SRS and spoke with EM officials and contractor staff at both locations  to better understand the use of models in planning and cleanup decisions,  as well as EM oversight of the models. We focused our review on model  standards and the use of models in decisions, not on the quality of the  models themselves or of their output.
We conducted this performance audit from October 2009 to February  2011, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Since the 1940s, DOE and its predecessors have operated a nationwide  complex of facilities used to research, design, and manufacture nuclear  weapons and related technologies. The environmental legacy of nuclear  weapons production at dozens of these sites across the United States  includes contaminated buildings, soil, water resources, and large volumes  of radioactive and hazardous wastes that require treatment, storage, and  disposal. The two sites that account for the majority of the costs of the  cleanup effort—Hanford and SRS—were established in the 1940s and  1950s, respectively, to produce plutonium and other nuclear materials  needed to manufacture nuclear weapons. EM manages cleanup projects at  these and other sites that involve multiple activities to treat and dispose of  a wide variety of radioactive and hazardous wastes.
Under federal and state laws, EM must clean up radioactive and hazardous  substances in accordance with specified standards and regulatory  requirements. EM carries out its cleanup activities under the requirements  of federal environmental laws that include, among others, CERCLA and  NEPA. CERCLA requires EM to evaluate the nature and extent of  contamination at the sites and determine what cleanup remedies, if any,  are necessary to protect human health and the environment into the  future. Under NEPA, EM must prepare an environmental impact statement  that assesses the environmental effects for a proposed agency action, all  reasonable alternatives, and the no-action alternative. Under both the  CERCLA and NEPA processes, EM analyzes proposed remedial action  alternatives according to established criteria, invites and considers public  comment, and prepares a Record of Decision that documents the selected  agency action. If the cleanup method selected under CERCLA or NEPA  will result in disposal of waste at an on-site disposal facility, EM is then  required, under DOE’s radioactive waste management order—DOE Order  435.1—to ensure that waste management activities at each disposal facility  are designed, constructed, and operated in a manner that protects  workers, the public, and the environment. EM does this by completing a  “performance assessment” of the selected cleanup method. To guide the  implementation of selected cleanup methods, EM and its contractors may  prepare a “system plan” that provides the basis for scheduling cleanup  operations and preparing budget requests. For example, both Hanford and  SRS have prepared system plans for treating and disposing of liquid  radioactive waste stored in aging and leak-prone underground tanks.
EM officials at DOE headquarters and field offices oversee cleanup  activities at the sites, but the work itself is carried out primarily by private  firms contracting with DOE. EM applies different approaches to managing  cleanup activities, depending on the type and extent of contamination and  state and/or federal requirements with which it needs to comply. In  addition, DOE has agreements with state and federal regulators, known as  Federal Facility Agreements, to clean up the Hanford and SRS sites. The  agreements lay out legally binding milestones for completing major steps  in the waste treatment and cleanup process. EPA officials, as well as  officials with environmental agencies in the states where EM sites are  located, enforce applicable federal and state environmental laws and  oversee and advise EM on its cleanup efforts.
One tool EM uses in support of cleanup decision analyses is computer  modeling. Although the computer models used across EM sites vary, they  have certain common characteristics. In general, computer models are  based on mathematical formulas that are intended to reflect physical,  biogeochemical, mechanical, or thermal processes in simplified ways. For  example, a computer model can simulate the movement of contamination  through the soil and groundwater or simulate the transfer of high-level  radioactive waste from underground storage tanks to facilities where the  waste will be treated. Appendix II details the key computer models used in  the cleanup decisions we reviewed at Hanford and SRS.
section_title: Computer Models Provide Critical Information to EM’s Environmental Cleanup Decision- Making Process
paragraphs: EM uses computer models to provide critical information for its decision- making process. First, computer models provide information that EM uses  to analyze the effectiveness of alternative actions to clean up radioactive  waste. Second, once a cleanup strategy has been selected, computer  modeling provides information that EM needs to assess the performance  of the selected cleanup strategy in reducing risks to human health and the  environment. Third, EM uses computer models to simulate operations in  the cleanup process, providing the basis for planning cleanup efforts and  for making annual budget requests.
subsections: section_title: EM Uses Computer Models to Analyze the Potential Effectiveness of Cleanup Alternatives
paragraphs: EM’s decision making for its cleanup efforts is based on meeting federal  and state requirements; input from state, local, and regional stakeholders;  and other considerations, including the costs of cleanup actions.  Computer models provide critical information that EM needs to assess  compliance with regulatory requirements when seeking to identify and  select alternatives for cleaning up radioactive and hazardous wastes, as  well as contaminated soil and groundwater at its sites.
EM’s cleanup decisions are guided by several federal and state  environmental laws, including CERCLA and NEPA, which both set forth  processes related to cleanup decisions. In the case of CERCLA, EM  determines the nature and extent of the contamination, assesses various  cleanup alternatives, and selects the best alternatives according to  evaluation criteria that include, among other things, protection of human  health and the environment, ease of implementing the alternative, state  and community acceptance, and cost. To accomplish these steps, EM uses  computer modeling to, among other things, simulate the movement of  contaminants through soil and groundwater over many years assuming no  cleanup action is taken. Projected contamination levels, migration  pathways, and contamination travel timelines are provided by simulations  and are evaluated to determine whether regulatory standards will likely be  exceeded in the future. If action is needed, then modeling simulations may  be conducted for a number of different cleanup alternatives. For example,  EM used modeling to assess contamination and the potential effectiveness  of various cleanup strategies at SRS’s C-Area Burning/Rubble Pit. Used  during the 1960s as a trash pit to dispose of organic solvents, waste oils,  paper, plastics, and rubble, SRS burned the contents of the pit periodically  to reduce its volume. Eventually, SRS used the pit for the disposal of inert  rubble, finally covering it with two feet of soil in the early 1980s. However,  the disposal of these materials and periodic burning resulted in hazardous  substance contamination of the surrounding soil and groundwater.  Between 1999 and 2004, EM implemented several actions to clean up the  majority of the area’s contamination. Following these actions, EM used  computer models to simulate the movement of the remaining  contamination through the soil and groundwater over the next 1,000 years.  Information provided by this modeling helped EM to identify the  remaining risks to human health and the environment and to identify  actions to clean up the remaining contamination. Using this information,  in conjunction with other criteria such as additional site data, input from  federal and state regulators and the public, and the availability of an  appropriate cleanup technology, EM selected a final cleanup remedy. This  remedy, which is ongoing and combines several different cleanup  technologies, was estimated in 2008 to cost, in present-worth dollars,  about $1.9 million over a 70-year period.
In implementing CERCLA, DOE focuses on discrete facilities or areas  within a site that are being remediated, making limited assessments of  cumulative impacts. By contrast, under NEPA, EM generally prepares  environmental impact statements that assess the environmental impacts— including cumulative impacts—of a proposed cleanup action, all  reasonable alternatives, and taking no action. For example, the  environmental impact statement for closing underground liquid  radioactive waste tanks at Hanford—which, as of November 2010 was still  in draft form—includes an analysis of the potential environmental impact  of various options for treating and disposing of about 55 million gallons of  mixed radioactive and hazardous waste and closing 149 underground  radioactive waste tanks. The draft environmental impact statement  includes an analysis of 11 tank waste treatment and closure alternatives,  including a no-action alternative. These alternatives range in cost from  about $3 billion to nearly $252 billion, excluding the costs associated with  the final disposal of the treated waste.
In the draft environmental impact statement, EM used computer models to  simulate the movement of contamination through soil and groundwater  over a period of 10,000 years for each of the cleanup alternatives. As with  CERCLA modeling, the results of the computer models were used to  estimate the remaining risks to human health and the environment  following the completion of each cleanup alternative and these risks were  then compared with requirements. The results of these models will be  used along with other information such as input from regulators and the  public and the costs of each alternative when EM selects the alternative it  will eventually implement.
section_title: EM Uses Computer Models to Assess the Performance of Selected Cleanup Activities
paragraphs: After a particular cleanup alternative is selected, EM also uses computer  modeling to demonstrate that the cleanup activity will result in reduced  future contamination levels that meet regulatory requirements. If the  cleanup method selected under CERCLA or NEPA will result in disposal of  waste at an on-site disposal facility, EM is then required, under DOE’s  radioactive waste management order—DOE Order 435.1—to ensure that  waste management activities at each disposal facility are designed,  constructed, and operated in a manner that protects workers, the public,  and the environment. To meet the requirements of the order, EM  completes a “performance assessment” of the selected cleanup method.  Under the order, this performance assessment is to document that the  disposal facility is designed, constructed, and operated in a manner that  protects workers, the public, and the environment. The performance  assessment also is to project the release of contamination into the soil and  groundwater from a site after cleanup and must include calculations of  potential chemical doses to members of the public in the future.
For example, in March 2010, SRS issued a performance assessment of a  cleanup and closure strategy for a group of 20 underground liquid  radioactive waste tanks, known as the F-Tank Farm. The performance  assessment evaluated closing the underground waste tanks and filling  them with a cement-like substance called grout—the alternative selected  following completion of SRS’s 2002 environmental impact statement.  Computer modeling was used extensively to prepare this performance  assessment. Specifically, computer modeling was performed using two  different types of models. The first computer model was used to perform  human health and environmental risk calculations and to calculate  radiation doses that could be compared to the maximum level allowed by  federal and state requirements. The second model was used to analyze  sensitivities and uncertainties in the results of the first model.
section_title: EM Uses Computer Modeling to Help Plan and Budget Cleanups
paragraphs: EM also uses computer models for lifecycle planning, scheduling, and  budgeting for its cleanup activities. Computer models provide important  information that EM and its contractors use to develop system plans that  outline the schedules for cleanup activities at EM sites. Outputs from  computer models and databases are used to create tables, charts, and  schedules that are published in the system plans and inform annual budget  requests for cleanup activities.
For example, at Hanford, a computer model known as the Hanford Tank  Waste Operations Simulator is designed to track the retrieval and  treatment of over 55 million gallons of radioactive waste held in  underground storage tanks. According to the most recent Hanford tank  waste system plan, which was issued in November 2010, the model  projects the chemical and radiological characteristics of batches of waste  that are to be sent to a $12.2 billion waste treatment plant that is being  built at Hanford to treat this waste. The model also provides scheduling  information the contractor uses to project near- and long-term costs and  schedules.
Similarly, SRS uses a computer model known as SpaceMan Plus™ to  support the site’s liquid waste system plan, which was issued in January  2010. For example, project work schedules for SRS’s tank waste program  are guided by this model. The model also simulates how the tank farms  integrate with waste processing facilities and tracks the movement of  waste throughout the liquid waste system. Output from the model was  used to provide tables and schedules found in the appendixes of SRS’s  system plan that details the specific cleanup activities that are to be  accomplished. These tables and schedules are used as part of the basis for  determining the costs of completing those activities. This information, in  turn, allows DOE and its contractors to generate annual budget requests.
section_title: EM Has General Quality Policies for Its Computer Models, but It Has Not Regularly Assessed Contractors’ Implementation of Quality Assurance Procedures
paragraphs: Although EM uses general departmental quality assurance policies and  standards that apply to computer models and relies on contractors to  implement specific procedures that reflect these policies and standards,  these policies and standards do not specifically provide guidance on  ensuring the quality of the computer models used in cleanup decisions.  Moreover, EM officials have not regularly performed periodic quality  assurance assessments, as required by DOE policy to oversee contractors’  development and use of cleanup models and the models’ associated  software. In addition, DOE and others have identified quality assurance  problems. For example, the state of Washington has cited flaws in a model  EM uses to analyze soil and groundwater contamination and has told EM  that it will no longer accept the use of this model for chemical exposure  analysis at Hanford.
subsections: section_title: Although EM Has General Quality Assurance Standards, Its Oversight Is Not Sufficient to Ensure the Quality of Cleanup Models
paragraphs: DOE addresses quality through various departmental policies and industry  standards; however these policies and standards do not specifically  provide guidance on ensuring the quality of the computer models used in  cleanup decisions. Specifically, DOE’s primary quality assurance policy— DOE Order 414.1C—provides general requirements EM and its  contractors must meet to ensure all work at the cleanup sites is carried  out correctly and effectively, including the development and use of  computer models. These requirements include developing a quality  assurance program, training staff how to check the quality of their work,  and providing for independent assessments of quality. A manual  accompanying this order describes acceptable, nonmandatory methods for  specifically ensuring quality of “safety software.” Safety software is  described in the manual as software used to design, manage, or support  nuclear facilities. However, the manual is less clear on how to assure  quality in computer models. Furthermore, it does not clearly address the  use of computer software not considered as safety software, such as those  used by computer models that support DOE’s cleanup decisions.
DOE’s quality assurance order also requires contractors to select and  comply with an appropriate set of industry standards for all work,  including computer modeling. One common set of standards was  developed by the American Society of Mechanical Engineers and provides  the requirements necessary to ensure safety in nuclear facilities, including  the development and validation of computer models and software that is  used to design and operate such facilities. Initially, the American Society  of Mechanical Engineers standards were not mandatory for computer  models and software used for cleanup decisions, many of which are  considered nonsafety software. These standards were but one of many  standards that contractors could choose to use. However, as of November  2008, EM made the American Society of Mechanical Engineers standards  mandatory for all cleanup activities, including modeling.
EM’s contractors are to implement DOE’s quality assurance requirements  using specific policies and procedures they develop. The specifics of  implementation vary from contractor to contractor. In the case of  computer software quality, a contractor is to include procedures for  testing and validating the software, ensuring changes to software are  properly documented, and correcting any errors. EM allows its contractors  to take a “graded approach” to quality procedures for computer software,  which means the contractor may adjust the rigor of the quality procedures  to match the importance of the software to overall operations. According  to documents we reviewed, computer software that controls systems in a  nuclear facility, for example, would require more rigorous quality  procedures than an administrative payroll system, as any failure in the  software controlling a nuclear facility could result in potentially hazardous  consequences to workers, the public, and/or the environment.
EM is to oversee its contractors’ implementation of quality standards for  computer models by performing periodic quality assurance assessments,  according to DOE’s quality assurance order. These quality assurance  assessments are intended to ensure that computer models meet DOE and  accepted industry quality standards. In our review of eight cleanup  decisions at Hanford and SRS, we found EM had conducted only three  quality assurance assessments that addressed quality standards for the  models used in those decisions. For example, for three of the four  decisions we reviewed at SRS, DOE officials at SRS could not provide  quality assurance assessments that specifically addressed whether the  models used in those decision processes met DOE’s quality assurance  requirements. DOE officials at SRS provided three general quality  assurance assessments, but these quality assurance assessments did not  specifically look at the cleanup models. In contrast, the models for a  March 2010 performance assessment selecting a cleanup strategy to close  underground liquid waste tanks at SRS did receive a quality assurance  assessment by a DOE headquarters group established to review  performance assessment decisions. In particular, as part of the review,  among other things, the DOE group conducted a quality assurance  assessment that evaluated the quality of the computer models used in the  performance assessment and the degree to which the models complied  with DOE requirements and industry standards. A DOE quality assurance  official at SRS noted that the site relies primarily on its contractors to  perform quality assurance assessments of computer models and their  associated software.
Similarly, in our review of four cleanup decisions at Hanford, we found  that EM had performed assessments that addressed quality standards for  the models used in those decisions in only two cases. In fact, one quality  assurance assessment was only undertaken after a contractor discovered  data quality errors in 2005 in a computer model used to support a prior  environmental impact statement at Hanford. According to a DOE quality  assurance manager at Hanford, his office conducts quality assurance  assessments primarily on those computer models and the associated  software for which the failure would result in significant safety  consequences to workers, the public, and/or the environment.
section_title: Some Reviews Have Revealed Quality Assurance Problems
paragraphs: Concerns have been raised by DOE and others that EM does not have  complete assurance of the quality of the models. For example:    Citing a number of flaws in a model DOE uses to analyze soil and  groundwater contamination at Hanford, the Washington state Department  of Ecology told DOE in February 2010 that it would no longer accept the  use of this model for chemical exposure analysis at Hanford. For example,  Ecology cited previous concerns that the model was not robust enough to  capture complexities of the movement of contamination through the  subsurface soil. We found that DOE had conducted no specific quality  assurance reviews on the model and its associated software.
EM headquarters officials conducted two technical reviews in 2009 of  planning models used for tank waste operations at Hanford and SRS. The  review of the Hanford planning model found that the model has limited  ability to sufficiently predict the composition of the contaminated waste as  it is prepared for the treatment processes. The review team cautioned that  this limitation raised a significant risk that, when actual waste treatment  operations started at the site, the waste may not meet the acceptance  requirements for processing by Hanford’s treatment facility. In addition,  the review of SRS’s planning model found that, although the data the  model provided on tank waste operations were reasonable, the model did  not have the ability to optimize operating scenarios, which hampered the  site’s long-term planning abilities.
A March 2010 independent review commissioned by a Hanford citizen’s  group raised concerns about a model used in the preparation of a draft  environmental impact statement of alternatives for closing Hanford’s  waste tanks. These concerns, based on reviewing the draft statement,  included insufficient documentation of the quality assurance processes  followed for the model and that modeling uncertainties were inadequately  quantified. The review concluded that the environmental impact statement  was insufficiently precise to be used to make a cleanup decision.
Where DOE has conducted quality assurance assessments, it has found  that contractors did not always implement quality requirements  consistently. Furthermore, in their own internal reviews, contractors have  noted problems with the implementation of quality assurance  requirements. Problems noted in DOE’s and contractors’ quality assurance  assessments include:  Inadequate documentation. A 2007 software quality review conducted by  DOE at Hanford found implementation problems, including inadequate  documentation and improper training for personnel in quality procedures.  At SRS, two general software quality assurance reviews performed by  DOE in 2004 found that while contractors generally met quality  requirements, documentation was sometimes lacking or improperly  prepared. A similar 2007 DOE review at SRS found a good software quality  program overall, but listed a number of deficiencies including inadequate  software plans and procedures.
Not following correct procedures. A 2007 DOE review of a Hanford  contractor’s software quality assurance program found, among other  things, that not all contractor personnel fully understood software quality  requirements. The report stated that, although software quality assurance  training had been provided, personnel did not follow procedures in  managing, maintaining, and overseeing software quality. For example, the  report cited an example of a spreadsheet in which data input cells were  not properly locked, in violation of procedures. In addition, the report  noted that software documentation was not periodically updated, as  required, because staff did not fully understand the procedures.
Incorrect quality assurance grading. In some cases, contractors did not  always correctly determine the level of rigor needed to ensure the quality  of computer models and their associated software. For example, a 2007  internal contractor review at Hanford found that 23 of 138 software codes  registered in a central repository were incorrectly designated as nonsafety  software, when in fact they should have been considered safety software.  As a result, the quality assurance procedures appropriate for a given level  of risk may not have always been applied.
section_title: EM Does Not Have an Overall Strategy and Guidance for Managing Its Cleanup Models
paragraphs: Although EM has recently begun some efforts to promote consistency in  the use of models across its various sites, these efforts are still in early  stages and, to date, some have had limited involvement of modeling  officials at the sites and federal, state, and local stakeholders who are  affected by decisions made using the output of computer models. In  addition, these efforts are not part of a comprehensive, coordinated effort  to improve the management of computer models across EM. In the  absence of such a strategy, EM also does not have overarching guidance  promoting consistency in modeling management, development, and use  across EM’s sites.
subsections: section_title: EM Has Some Initiatives to Improve Management of Its Cleanup Models, but They Are Not Part of a Comprehensive, Coordinated Strategy
paragraphs: EM has begun some efforts to improve the use of computer models across  its various sites. For example, EM, in fiscal year 2010, began developing a  set of state-of-the-art computer models to support soil and groundwater  cleanup across the nuclear weapons complex. According to EM officials  and documentation they provided, this initiative, called the Advanced  Simulation Capability for Environmental Management, will allow EM to  provide more sophisticated analysis of soil and groundwater  contamination for cleanup decisions. Although the initiative’s director told  us that the goal is to encourage all sites to use these models for all of their  soil and groundwater analysis, he noted that there are no plans to make  using these models mandatory.
Moreover, SRS has created a forum for improving consistency in  groundwater computer modeling performed at the site. According to the  charter document, the forum, called the Groundwater Modeling  Consistency Team, was formed in 2006 following the discovery of  inconsistencies in the data used in groundwater computer modeling  conducted at Hanford in support of the preparation of an environmental  impact statement under NEPA. The group, which is made up of DOE and  contractor officials, reviews software codes, model inputs, and model  assumptions to promote sitewide consistency in the management of  computer models.
Although these efforts may help improve EM’s use of computer models,  they are largely still in early stages. In addition, according to EM officials,  some of these efforts have, to date, had limited involvement of modeling  officials at EM’s sites and of federal, state, and local stakeholders who are  affected by decisions made using the output of computer models.  Furthermore, they are not part of a comprehensive, coordinated effort to  improve the consistency of computer models and reduce duplication  across EM’s various sites. For example, we found that different models are  used to perform similar functions not only between EM sites, but also  within sites. At SRS, one contractor uses a set of models to perform soil  and groundwater analyses when evaluating the potential effectiveness of  cleanup alternatives under CERCLA and NEPA, while another contractor  uses a different set of models to perform similar analyses for performance  assessments under DOE’s radioactive waste management order. Each  contractor has its own set of procedures for developing and using each  computer model. Officials from both contractors told us that they use  different models because state and federal regulators have only approved  the use of certain models for specific types of cleanup decisions. Issues  with consistency and duplication of effort in the use of computer models  have also been noted by others. For example, a February 2010 DOE review  noted that five major DOE sites use 28 different models to analyze  groundwater and subsurface contamination when preparing performance  assessments under DOE’s radioactive waste management order. DOE  officials told us that past modeling practices have resulted in conflicting  assumptions and data sets, as well as different approaches to uncertainty  analyses. In addition, a September 2009 DOE technical review of the  Hanford tank waste modeling system raised concerns that two models at  Hanford that share data use different assumptions that could lead to  inconsistencies between the two. As a result, the Hanford waste treatment  system plan, which is based on the output of one of these models, may not  reflect the most current information.
In contrast, other federal agencies and DOE offices have taken steps to  improve consistency and reduce duplication as part of a comprehensive,  coordinated strategy to manage the use of computer models. For example,  EPA organized a Center for Regulatory Environmental Modeling in 2000 as  part of a centralized effort to bring consistency to model development,  evaluation, and usage across the agency. The Center brings together senior  managers, modelers, and scientists from across the agency to address  modeling issues. Among its tasks are to help the agency (1) establish and  implement criteria so that model-based decisions satisfy regulatory  requirements; (2) implement best management practices to use models  consistently and appropriately; (3) facilitate information exchange among  model developers and users so models can be continuously improved; and  (4) prepare for the next generation of environmental models. According to  a DOE official, EM does not have a central coordination point similar to  EPA’s.
Within DOE, the Office of Nuclear Energy recently established an  initiative—the Nuclear Energy Modeling and Simulation Energy  Innovation Hub—that provides a centralized forum for nuclear energy  modelers. According to the director of the Office of Nuclear Energy’s  Office of Advanced Modeling and Simulation, the hub will provide a more  centrally coordinated effort to bring together modeling and simulation  expertise to address issues associated with the next generation of nuclear  reactors. Similar comprehensive, coordinated efforts are lacking within  EM and, as a result, EM may be losing opportunities to improve the quality  of its models, reduce duplication, keep abreast of emerging computer  modeling and cleanup technologies, and share lessons learned across EM’s  sites.
section_title: Other Federal Agencies and DOE Offices Have Recognized the Need for Comprehensive Modeling Guidance
paragraphs: The need for specific guidance for ensuring the careful management of  computer models used in decision making is not new. As early as 1976, we  reported on the government’s use of computer models and found that the  lack of guidance contributed to ineffective and inefficient use of computer  models. We noted that guidance should define the problem to be solved,  specify the assumptions and limitations of the model, and provide  methods to test whether the model reasonably describes the physical  system it is modeling.
More recently, a 2007 National Research Council study of modeling at EPA  laid out guidelines to improve environmental regulatory computer  modeling. The study noted that adoption of a comprehensive strategy for  evaluating and refining EPA’s models could help the agency add credibility  to decisions based on modeling results. It also noted several key principles  to follow for model development, evaluation, and selection. Moreover, the  study recommended that peer review be considered as an important tool  for improving model quality. According to the study, a peer review should  entail not only an evaluation of the model and its output, but also a review  of the model’s origin and its history. The study also made  recommendations on quantifying and communicating uncertainty in model  results to better communicate a model’s limitations to stakeholders  affected by decisions made using the results of computer models.
EPA has taken action to develop specific guidance, issuing a guide in 2009  addressing the management, development, and use of computer modeling  used in making environmental regulatory decisions. In this guidance,  EPA developed a set of recommended best practices to help mode lers  effectively use computer models. The guidance defines the role of  computer models in the public policy process, discusses appropriate ways  of dealing with uncertainty, establishes criteria for peer review, and  addresses quality assurance procedures for computer modeling.
Even within DOE, another office outside of EM has recognized the need  for specific guidance for managing computer models. Specifically, DOE’s  Office of Civilian Radioactive Waste Management specified in its quality  assurance requirements several requirements for computer models.  These requirements included clearly defining the model’s objective,  documenting alternative models that could be used and the rationales for  not using them, and discussed a model’s limitations and uncertainties. In  addition, the office specified in its requirements that, among other things,  a computer model receive a technical review through a peer review or  publication in a professional journal.
Although the importance of comprehensive guidelines for managing  computer models is well established, according to its officials, EM does  not have such overarching guidance. As previously discussed, EM does  have a manual accompanying its quality assurance order that describes  acceptable methods for specifically ensuring the quality of safety software.  However, the manual does not generally address models used in cleanup  decisions. EM also has guidance addressing the management of computer  models used in conducting performance assessments under its radioactive  waste management order. Specifically, a DOE headquarters group that is  charged with reviewing decisions made under this order—the Low-Level  Waste Disposal Facility Federal Review Group—has developed a manual  that contains guidance on, for example, ensuring that input data to  computer models are described and are traceable to sources derived from,  among other things, field data from the site and referenced literature that  is applicable to the site. However, this guidance does not apply to  computer models used to analyze the potential effectiveness of cleanup  alternatives under CERCLA or NEPA or to computer models used for  planning, scheduling, and budgeting purposes. As a result, computer  models developed at various DOE sites do not have consistent criteria to  define the role of the model in the decision-making process, consistent  ways of dealing with uncertainties and a model’s limitations, and  mechanisms to ensure computer model quality, such as quality assurance  assessments and peer review.
section_title: Conclusions
paragraphs: EM’s computer models provide critical information that is needed to make  significant decisions about how to clean up the radioactive and hazardous  legacy waste across the country. However, EM’s oversight of the quality of  these models and its management of the development, evaluation, and use  of the models has not always been commensurate with the models’  importance. Because the decisions EM makes must protect human health  and the environment for thousands of years into the future, it is critical  that the models on which EM bases its decisions are of the highest quality  possible. In addition, because these cleanup efforts will take decades and  cost billions of dollars, it is also important that models used for planning,  scheduling, and budgeting purposes provide the most accurate data  possible for EM and Congress to make informed decisions on cleanup  activities.
EM’s failure to fully oversee its contractors’ implementation of quality  assurance procedures has led to a reduced level of confidence that the  models reasonably represent the conditions they are meant to simulate. In  several cases, we found necessary quality assurance reviews were not  conducted. In others, reviews found that quality assurance procedures  were inadequately implemented. Because existing quality assurance  requirements that are applied to EM’s computer models have not been  adequately implemented and, in some cases, are insufficiently understood  by its contractors, EM and its contractors do not have an effective  mechanism to provide the public and other EM stakeholders with  assurance of a model’s quality.
To its credit, EM is beginning to undertake efforts to improve the  consistency of models across the nuclear weapons complex. However,  some of these efforts are still in their infancy, and it remains to be seen  whether any improvements in EM’s management of its models will result.  We recognize that every site has its unique conditions and challenges and  that a one-size-fits-all approach to modeling would not be appropriate.  Nevertheless, there is room for additional consistency in model  development and implementation, as well as a mechanism for sharing  lessons learned among DOE’s various sites. For a number of years, other  federal agencies and offices within DOE have recognized the importance  of a comprehensive guidance for managing computer models. Without a  comprehensive strategy and modeling guidance, EM may miss  opportunities to improve the quality of computer models, promote  consistency, reduce duplication across DOE sites, and share lessons  learned.
section_title: Recommendations for Executive Action
paragraphs: To help EM increase confidence in the quality of information provided to  the public and its stakeholders resulting from the use of computer  modeling, we recommend the Secretary of Energy take the following three  actions:    Clarify specific quality assurance requirements for computer models used  in environmental cleanup decisions, including to analyze the potential  effectiveness of cleanup alternatives, assess the performance of selected  cleanup activities, and assist in planning and budgeting cleanup activities.
Ensure that the models are assessed for compliance with these  requirements.
Develop a comprehensive strategy and guidance for the management of  computer models to promote consistency, reduce duplication, and ensure  sharing of lessons learned.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOE for its review and comment. In  its written comments, DOE agreed with our recommendations and stated  that modeling is an important component of management analysis and  decision making for the department. DOE noted that it is committed to  continuous improvement in model development and application and  commented that our recommendations will strengthen its modeling  efforts.
DOE stated in its comments that it disagreed with the draft report’s  assertion that its directives and standards fall short for the development  and management of computer models. DOE commented that its quality  assurance directives apply directly to the development, coding, and  validation of safety and nonsafety computer models used in cleanup  decisions and that EM has interpreted and applied these directives and  accompanying standards to develop its quality program. We agree with  DOE, and our draft report noted, that DOE addresses quality through  various departmental policies and industry standards. However, these  directives do not provide specific guidance to EM on assuring quality of  the cleanup models themselves, guidance that other agencies and offices  within DOE have developed. In particular, DOE’s primary quality  assurance policy—DOE Order 414.1C—addresses general standards that  EM and its contractors must meet to ensure all work at its sites is carried  out effectively, but is vague on the specific steps that must be followed to  ensure the quality of models used in cleanup decisions. In addition, as our  draft report noted, a manual accompanying this order describes  acceptable, nonmandatory methods for specifically ensuring quality of  safety software. However, the manual is less clear on the use of computer  software not considered as safety software, such as those used by  computer models that support DOE’s cleanup decisions. Our  recommendation that DOE clarify the specific quality assurance  requirements for computer models used in environmental cleanup  decisions is intended to address these problems.
DOE’s comments also provided additional information on the  department’s oversight of computer models, initiatives it is undertaking to  improve its modeling efforts, and the specific steps it plans to take to  address our recommendations. DOE also provided technical comments  that we incorporated in the report as appropriate. DOE’s written  comments are presented in appendix III.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees; the Secretary of Energy; the  Director, Office of Management and Budget; and other interested parties.  In addition, the report will be available at no charge on the GAO Web site  at http://www.gao.gov.
If you or your staffs have any questions regarding this report, please  contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine how the Department of Energy’s (DOE) Office of  Environmental Management (EM) uses computer modeling in cleanup  decisions, we focused on cleanup decisions EM has made at its Hanford  Site in Washington state and Savannah River Site (SRS) in South Carolina  because together these two sites account for more than one-half of EM’s  annual cleanup spending and approximately 60 percent of the total  estimated cost of approximately $275 billion to $329 billion to clean up the  entire nuclear weapons complex. We focused our review on decisions  made in two major areas that represent the largest and most significant  elements of the cleanup program at these two sites. The first is cleanup of  radioactive and hazardous waste stored in underground tanks, which DOE  has determined poses the most significant environmental safety and health  threat in the cleanup program. DOE estimates cleaning up tank waste at  the sites will cost between $87 billion and $117 billion, making it the  largest cost element of EM’s cleanup program. Second, both sites have  significant contamination to soil and groundwater, which DOE estimates  will cost more than $12 billion to remediate. For each site, we selected  three types of decisions that were representative of major decisions made  at these sites between 2002 and 2010—(1) decisions made under  environmental statutes, including the Comprehensive Environmental  Response, Compensation, and Liability Act of 1980, as amended  (CERCLA)—which addresses specific environmental remediation  solutions for a cleanup site—and the National Environmental Policy Act,  as amended (NEPA)—under which DOE evaluates the impacts to human  health and the environment of proposed cleanup strategies and possible  alternatives; (2) performance assessments under DOE orders governing  radioactive waste management; and (3) cleanup budgeting and planning  decisions. We reviewed publicly available information from regulators and  interviewed DOE officials and contractor staff to identify the most recent  decisions for each of the three types of decisions selected for review at  each site. We reviewed these decisions to identify the most recent decision  that included the use of computer modeling. We then selected, based on  input by EM officials, the main models used to support these decisions at  the two sites. We visited both Hanford and SRS and spoke with both EM  officials and contractor staff there to better understand the use of models  in planning and cleanup decisions and DOE’s oversight of the models. We  obtained demonstrations of these models, as well as information on how  they were used in decision making. We obtained and reviewed the  decision documents, as well as modeling studies, notes of meetings  between DOE and its regulators to develop models, and other  documentation showing how the models were used in decisions. We  interviewed officials from DOE headquarters and the two sites, as well as  contractor staff, to determine how the models work and how they were  used in these decisions. We analyzed this information to determine how  the results of computer models were used in making cleanup decisions,  the importance of modeling in the selection of a cleanup strategy, and  other factors that contributed to the selection of a cleanup strategy.
To evaluate how EM determines the quality of the computer models used  in cleanup decision making, we obtained and reviewed documentation  showing the standards the models were required to meet. We gathered  documentation on DOE standards, as well as policies and procedures from  contractors overseeing the models. We discussed computer model and  software standards with EM officials from EM’s sites, contractors at the  sites, and headquarters officials. We also interviewed officials from the  Defense Nuclear Facilities Safety Board, the National Research Council,  the Environmental Protection Agency, and the Washington state  Department of Ecology about existing standards for the use and  implementation of computer modeling and its associated software. We  analyzed EM policies and contractor procedures to determine what quality  assurance standards exist to address the quality of computer models. We  also requested from EM and its contractors all assessments that were  conducted on computer models used in the decisions we were reviewing,  indicating whether quality standards were met. In general, the assessments  we reviewed were largely conducted by the contractors, regulators, or  external sources, such as consultants. These reviews ranged from  contractor-performed assessments of the implementation of quality  standards for software, to federal and state regulator comments on the  modeling output used to develop alternatives in a regulatory package, to  an outside consultant-performed review on the appropriateness of  modeling for selecting a preferred alternative from an environmental  impact statement prepared under NEPA. We analyzed these assessments  to understand the level of oversight EM provided to assure model and  software quality, as well as the extent to which contractors were  implementing quality procedures.
To address EM’s overall strategy for managing computer models that are  used in cleanup decisions, we interviewed DOE officials from  headquarters and from each site. We also interviewed officials from the  Environmental Protection Agency, National Research Council, DOE’s  Office of Nuclear Energy, and DOE’s Office of Civilian Radioactive Waste  Management about the implementation of computer modeling guidance  and modeling coordination strategies. We reviewed modeling guidance  from these organizations, as well as from the Office of Management and  Budget. We focused our review on model quality assurance standards and  the use of models in decision making, not on the quality of the models  themselves or of their output.
We conducted this performance audit from October 2009 to February  2011, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Functions of Key Models Used in Cleanup Decisions GAO Reviewed at EM’s Hanford and Savannah River Sites
subsections: section_title: Models used at Hanford
section_title: Appendix III: Comments from the Department of Energy
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Ryan T. Coles, Assistant  Director; Ivelisse Aviles; Mark Braza; Dan Feehan; Nancy Kintner-Meyer;  Jonathan Kucskar; Mehrzad Nadji; Kathryn Pedalino; Thomas C. Perry;  and Benjamin Shouse made key contributions to this report.
section_title: Related GAO Products
paragraphs: Nuclear Waste: Actions Needed to Address Persistent Concerns with  Efforts to Close Underground Radioactive Waste Tanks at DOE’s  Savannah River Site. GAO-10-816. Washington, D.C.: September 14, 2010.
Recovery Act: Most DOE Cleanup Projects Appear to Be Meeting Cost and  Schedule Targets, but Assessing Impact of Spending Remains a  Challenge. GAO-10-784. Washington, D.C.: July 29, 2010.
Department of Energy: Actions Needed to Develop High-Quality Cost  Estimates for Construction and Environmental Cleanup Projects.  GAO-10-199. Washington, D.C.: January 14, 2010.
Nuclear Waste: Uncertainties and Questions about Costs and Risks  Persist with DOE’s Tank Waste Cleanup Strategy at Hanford.  GAO-09-913. Washington, D.C.: September 30, 2009.
Department of Energy: Contract and Project Management Concerns at  the National Nuclear Security Administration and Office of  Environmental Management. GAO-09-406T. Washington, D.C.: March 4,  2009.
Nuclear Waste: DOE Lacks Critical Information Needed to Assess Its  Tank Management Strategy at Hanford. GAO-08-793. Washington, D.C.:  June 30, 2008.
Hanford Waste Treatment Plant: Department of Energy Needs to  Strengthen Controls over Contractor Payments and Project Assets.   GAO-07-888. Washington, D.C.: July 20, 2007.
Nuclear Waste: DOE Should Reassess Whether the Bulk Vitrification  Demonstration Project at Its Hanford Site Is Still Needed to Treat  Radioactive Waste. GAO-07-762. Washington, D.C.: June 12, 2007.
Hanford Waste Treatment Plant: Contractor and DOE Management  Problems Have Led to Higher Costs, Construction Delays, and Safety  Concerns. GAO-06-602T. Washington, D.C.: April 6, 2006.
Nuclear Waste: Absence of Key Management Reforms on Hanford’s  Cleanup Project Adds to Challenges of Achieving Cost and Schedule  Goals. GAO-04-611. Washington, D.C.: June 9, 2004.
Nuclear Waste: Challenges to Achieving Potential Savings in DOE’s  High-Level Waste Cleanup Program. GAO-03-593. Washington, D.C.: June  17, 2003.
Nuclear Waste: Department of Energy’s Hanford Tank Waste Project— Schedule, Cost, and Management Issues. GAO-RCED-99-13. Washington,  D.C.: October 8, 1998.