Nuclear Safety: DOE Needs to Determine the Costs and Benefits of Its Safety Reform Effort
Published: 2012-04-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOE carries out many of the nations most critical missions, including stewardship of the nations nuclear weapons stockpile and the environmental remediation of radioactive and hazardous legacy waste left over from the Cold War. DOE uses a system of regulations and internal directives that lay out requirements and guidance for ensuring the safety of staff and contractors, the public, and the environment. Over the past 10 years, GAO and others have repeatedly made recommendations for DOE to improve safety performance. In March 2010, DOE announced a reform effort to revise safety-related directives to increase productivity and reduce costs while maintaining safety.
This report examines (1) how DOE revised safety directives under its reform effort, (2) the costs of the reform effort and the benefits DOE hoped to achieve, and (3) the extent to which its reform effort addresses safety concerns GAO and others have identified. GAO reviewed relevant DOE reform effort documents, visited selected DOE sites to interview site office and contractor officials, and analyzed past GAO and other reports on DOEs safety problems.
section_title: What GAO Found
paragraphs: Under its safety reform effort, the Department of Energy (DOE) reduced the number of safety directives by eliminating or combining requirements it determined were unclear, duplicative, or too prescriptive and by encouraging the use of industry standards. DOE reduced the number of its safety directives from 80 to 42, and for some of the directives DOE retained, it made extensive revisions. For example, DOE deleted requirements from its quality assurance directive addressing a corrective action program because another safety directive adequately covered these requirements. DOE obtained comments on its proposed revisions from DOE and contractor staff and from the Defense Nuclear Facilities Safety Board (Safety Board).
The benefits of DOEs reform effort are not clear. DOE intended to enhance productivity and reduce costs while maintaining safety, but DOE did not determine how the original requirements contained in safety directives impaired productivity or added costs before undertaking the reform effort. Moreover, DOE did not assess whether the cost to implement the revised directives would exceed the benefits, but officials said they had launched an initial study to determine, among other things, the costs associated with implementing selected safety requirements. DOE also did not develop performance measures in order to assess how the reform effort will lead to improved productivity or lower costs while maintaining safety. Instead, DOE is measuring success by using output-oriented measures, such as the number of directives eliminated, and not outcome measures, such as specific productivity improvements or cost savings. In the absence of clear measures linking the reform effort to productivity and safety improvements, DOE is not well positioned to know that its reform effort will achieve the intended benefits.
DOEs reform effort did not fully address safety concerns GAO and others have identified in three key areas: (1) quality assurance, (2) safety culture, and (3) federal oversight. Regarding quality assurance, DOE strengthened its quality assurance directive by clarifying that contractors must follow specific industry quality assurance standards, but quality assurance problems persist. For example, DOE proposed a nearly $250,000 fine against a contractor in July 2011 after identifying quality assurance problems in an incident where a worker punctured his hand with a sharp object contaminated with plutonium. With regard to safety culture, DOE revised its Integrated Safety Management directives to attempt to strengthen the safety culture at its sites, but DOE removed requirements for contractors to follow the directives because contractors already had to comply with safety management requirements in federal regulation. Safety Board officials raised concerns that the requirements in federal regulation are less detailed and, as a result, contractors may not implement safety practices as rigorously as if they were subject to the more specific requirements in DOEs directives. Finally, regarding federal oversight, DOE revised its approach to place greater emphasis on having its independent oversight staff review safety design documents before facilities are constructed, rather than after they are built. Other changes, however, such as requiring oversight staff to coordinate their assessment activities with DOE site office and contractor staff, raise concerns about the oversight staffs ability to provide a critical review of safety at DOEs sites that is independent from DOE site office and contractor staff.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOE analyze the costs and benefits of its safety reform effort and identify how the effort will help address safety concerns. DOE agreed with the recommendations but commented that it had significant concerns about the accuracy of the reports findings and conclusions. GAO stands by its findings and conclusions for the reasons discussed in the report.

section_title: Letter
paragraphs: The Department of Energy (DOE) carries out many of the nation’s most  critical missions, including stewardship of the nation’s nuclear weapons  stockpile and the environmental remediation of radioactive and hazardous  legacy waste left over from the Cold War era. Work carried out at DOE’s  sites across the country involves, among other things, (1) research on  dangerous nuclear materials, such as plutonium, and (2) the handling and  disposal of radioactive and hazardous waste that, if not handled safely,  could cause nuclear accidents or expose the public and the environment  to heavy doses of radiation. DOE’s sites also conduct a wide range of  other activities, including construction and routine maintenance and  operation of equipment and facilities that also run the risk of accidents,  such as those involving heavy machinery or electrical mishaps. The  consequences of such accidents could be less severe than those  involving nuclear materials, but they could also lead to long-term  illnesses, injuries, or even deaths among workers.
DOE relies on contractors and subcontractors to perform day-to-day  operations at most of its nuclear and nonnuclear facilities located at  approximately 40 sites across the country. To help ensure safety at  these sites, the department requires its contractors to adhere to federal  safety laws; departmental regulations; and DOE requirements that are  provided in the department’s system of directives—including DOE’s  policies, orders, guides, and manuals. Among other things, DOE  directives address the safe operation of nuclear facilities as well as  worker safety. The department incorporates directives into contracts and  holds contractors accountable for meeting them. DOE’s program offices— both at headquarters and its sites—are responsible for directly overseeing  the contractors to ensure safety requirements are being met. The  department’s Office of Health, Safety, and Security independently  oversees and enforces, among other things, nuclear and worker safety at  DOE’s sites.
Over the years, federal officials, Congress, and others have expressed  concerns about safety problems at DOE’s sites. For example, the  Defense Nuclear Facilities Safety Board (Safety Board)—an independent  executive branch agency created by Congress to independently assess  safety conditions and operations at defense nuclear facilities at DOE’s  sites—has held 11 public hearings over the past 10 years to address  concerns about DOE’s safety practices. Additionally, in January 2005, the  director of DOE’s Lawrence Livermore National Laboratory suspended all  programmatic work at the laboratory’s plutonium-handling facility, largely  because of numerous unresolved safety issues and a failure to address  these issues adequately. Moreover, in October 2007, we reported that  nearly 60 serious accidents or near misses had occurred at DOE’s  nuclear weapons laboratories since 2000. For instance, at DOE’s Los  Alamos National Laboratory in July 2004, an undergraduate student who  was not wearing required eye protection was partially blinded in a laser  accident. Accidents such as this one and nuclear safety violations  focused attention on needed improvements in safety performance across  DOE’s sites.
In March 2010, the Deputy Secretary of Energy announced a new effort— the 2010 Safety and Security Reform Plan (reform effort)—to revise  DOE’s safety and security directives and reform its oversight approach to  “provide contractors with the flexibility to tailor and implement safety and  security programs without excessive federal oversight or overly  prescriptive departmental requirements” while “maintaining the highest  standards of safe and secure operations” at DOE’s sites. In the  memorandum announcing this effort, the Deputy Secretary noted that  burdensome safety requirements were affecting the productivity of work  at DOE’s sites and that reducing this burden on contractors would lead to  measurable productivity improvements.
In this context, you asked us to evaluate the department’s current reform  effort. Although DOE’s 2010 Safety and Security Reform effort addresses  both safety and security directives, this report focuses on revisions to  DOE’s safety-related directives, which constitute 80 of the 107 directives  included in the reform effort. Our objectives were to examine (1) how  DOE revised safety directives under its reform effort, (2) the costs of the  reform effort and the benefits DOE hoped to achieve, and (3) the extent to  which DOE’s reform effort addresses safety concerns we and others have  identified.
To conduct this work, we reviewed documents on DOE’s reform effort,  including the overall strategy and implementation plan for the reform  effort. DOE’s reform effort addresses directives both in the safety and  security areas. This report focuses on safety, including nuclear safety,  worker safety, quality assurance, oversight, and operating experience.  We interviewed DOE officials at DOE headquarters in the Office of  Health, Safety, and Security and in the National Nuclear Security  Administration (NNSA), a semiautonomous agency within the department  that is responsible for, among other things, the management and security  of the nation’s nuclear weapons programs. To gain an understanding of  the potential impact of the reform effort on DOE’s sites, we visited sites  that were in various stages of implementing revised safety requirements.  These sites were the Kansas City Plant in Missouri, the Nevada National  Security Site in Nevada, the Lawrence Livermore National Laboratory in  California, and the Hanford Site in Washington state. We interviewed  both DOE and contractor officials at these sites. We also reviewed  documents relating to safety performance measures, efforts to address  safety problems, and oversight. We interviewed officials at other  governmental agencies, including the Nuclear Regulatory Commission  (NRC), which licenses and regulates commercial nuclear power plants,  and the Department of Labor’s Office of Occupational Health and Safety  (OSHA), which generally regulates worker safety and commercial  industrial facilities. We also interviewed officials with the Safety Board to  gain an understanding of the process of changing regulatory safety  requirements. In addition, we interviewed officials from the American  Society of Industrial Standards, an industry standards-making  organization, to determine how industry standards are developed. To gain  an understanding of DOE’s long-standing safety problems, we reviewed  51 reports from the past 10 years by GAO, DOE’s Office of Inspector  General, and the Safety Board. Appendix I presents a detailed description  of our scope and methodology.
We conducted this performance audit from February 2011 through April  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOE generally regulates the safety of its own nuclear facilities and  operations at its sites. In contrast, NRC generally regulates commercial  nuclear facilities, and OSHA generally regulates worker safety at  commercial industrial facilities. DOE manages the safety of its facilities  and operations at its sites through federal safety laws; departmental  regulations; and a system of internal directives—that is, DOE policies,  orders, guides, and manuals. DOE policies communicate top-level  expectations for safety issues, such as conducting safe operations at  high-hazard nuclear facilities. DOE orders are used as the primary  directives for communicating requirements. DOE guides detail  acceptable, but not mandatory, methods for implementing requirements  contained in DOE orders, while manuals may contain both requirements  and guidance.followed and guidance for implementing requirements to help ensure the  safety of workers, the public, and the environment at DOE’s sites.
In total, DOE’s directives lay out requirements to be  To ensure adherence to the requirements in DOE’s directives, the  department oversees its own operations. This oversight has three main  components. First, DOE’s program offices—such as DOE’s Office of  Environmental Management and NNSA—have primary responsibility to  safely carry out their program missions. Second, the site offices oversee  the contractors that manage and operate DOE’s sites. DOE’s contractors  are responsible for safely designing, constructing, and operating the  facilities at these sites that they use to carry out their work. Third, DOE’s  Office of Health, Safety, and Security is responsible for, among other  things, the development of safety policy and conducting independent  oversight of compliance with DOE’s safety regulations and directives, and  enforcement activities. Within DOE’s Office of Health, Safety, and  Security, the Office of Independent Oversight conducts periodic  appraisals of the environment, safety, and health programs at DOE’s sites  to determine if DOE officials and contractors are complying with DOE’s  safety regulations and directives.
DOE also receives safety assessments and recommendations from other  organizations, most prominently the Safety Board. The Safety Board  provides independent safety oversight of DOE’s defense nuclear facilities,  which are located at 13 DOE sites across the United States. The Safety  Board has broad oversight responsibilities at defense nuclear facilities  located at DOE’s sites and seeks to use informal interactions with DOE to  resolve safety issues that are of concern to the Safety Board. However,  the Safety Board also uses formal communications, such as  recommendations, to address more substantial safety issues at a facility  or site as well as broader safety issues across DOE’s sites. Other  organizations that provide assessments to DOE on the management of its  sites include DOE’s Office of Inspector General, the National Academy of  Sciences, and GAO.
In March 2010, the Deputy Secretary of Energy issued a memorandum  that announced a reform effort aimed at revising DOE’s 80 safety-related  directives and the department’s approach to regulating and overseeing  safety at its sites. In announcing the reform effort, the Deputy Secretary  noted that it was aimed at (1) providing increased recognition of program  management’s responsibility for safety, (2) reviewing opportunities to  streamline safety requirements, and (3) eliminating safety directives that  do not add value. The reform effort was intended to provide contractors  with the flexibility to tailor and implement safety programs without  excessive federal oversight or overly prescriptive departmental  requirements, while also maintaining safety at DOE’s sites. Key elements  of the reform effort included reviewing the department’s safety directives  to (1) clarify and streamline wording in them; (2) increase the use of  external (non-DOE) standards, such as industry standards and those  from other regulatory agencies; (3) focus oversight activities on high-risk  activities and contractors with poor safety records; and (4) provide more  reliance on contractor systems that reliably measure the effectiveness of  their safety management systems and ability to implement effective  corrective actions.
DOE’s reform effort mirrors another reform effort underway at NNSA.  Known as “governance transformation,” NNSA’s effort involves revising  the agency’s business model to, among other things: more clearly establish roles of federal oversight staff and contractors; place more reliance on contractor’s self-oversight through its  contractor assurance systems to ensure such things as safety  performance; allow contractors to tailor safety and other requirements to their work  without overly prescriptive DOE requirements; and promote the use of national and international standards as  appropriate, rather than departmentally unique requirements.
NNSA’s Kansas City Plant has completed implementation of this new  model, and other NNSA sites—such as the Nevada National Security Site  in Nevada and Y-12 National Security Complex at Oak Ridge,  Tennessee—are currently implementing the changes.
section_title: DOE’s Reform Effort Streamlined Directives, but Views on the Revisions Were Not Sought from the Public or Federal Agencies Other than the Safety Board
paragraphs: Under its reform effort, DOE reduced the number of safety directives by  eliminating or combining, among other things, requirements it determined  were unclear, duplicative, or too prescriptive; allowing contractors  additional flexibility; and encouraging the use of industry standards. In  revising its directives, DOE followed a systematic process that included  obtaining the views of staff from within DOE and its contractor community  and from the Safety Board, but it did not seek input from the public or  other federal agencies.
subsections: section_title: DOE Revised Its Safety Directives to Reduce Duplication, Allow Contractors Additional Flexibility, and Encourage the Use of Industry Standards
paragraphs: In total, under its reform effort, DOE reduced the number of its safety  directives from 80 to 42. When deciding whether to keep, revise, or  eliminate a safety directive, DOE identified a justification for the  directive—that is, why the directive was first established and what it  should accomplish. For example, DOE’s safety directive requiring the  development of a departmentwide system to collect, communicate, and  evaluate lessons learned about operating nuclear and other facilities was  established in 2006 in response to, among other things, the 2003  explosion of the Space Shuttle Columbia where inadequate collecting and  sharing of operating lessons learned was found to have contributed to the  accident. According to DOE documents, the department determined that  it would not eliminate this directive because it helped establish good work  practices and enabled the sharing of information across DOE’s sites. In  contrast, DOE eliminated its policy directive on recruiting and training  federal employees that ensured the employees had sufficient technical  knowledge of safety issues. This directive was established in 1998 to  provide departmental expectations for maintaining sufficient federal  technical safety capability. In undertaking its reform effort, DOE  concluded this policy had served its purpose and was no longer needed  as a stand-alone directive because a companion directive implementing  the requirements for recruiting and training staff to safely carry out work  was sufficient.
If DOE decided to retain rather than eliminate an entire safety directive,  DOE officials undertook a line-by-line examination of the specific  requirements contained within the directive. This examination was  intended to determine whether revisions to the requirements were  necessary to make them more easily understandable and to eliminate  confusion arising from having similar requirements in multiple directives.  As mentioned, the examination included determining why each  requirement was initially established. For example, some specific  requirements in a DOE directive may have been the result of  recommendations from the Safety Board or the result of a long-standing  DOE or industry practice. Establishing the reason why a requirement was  initially created assisted the DOE officials who were undertaking the line- by-line examination to determine whether revisions were necessary.
Within the safety directives that DOE retained, DOE eliminated some  requirements that department officials determined were unclear,  duplicative, or too prescriptive. In particular, DOE removed requirements  that it determined already existed in other safety directives or regulations.  In its quality assurance directive, for example, DOE deleted 61 of 245  requirements for a corrective action program because it determined that  the requirements were adequately addressed in revisions to other  directives on oversight practices. DOE also eliminated requirements that  it identified as being too prescriptive. For example, in revising its oversight  directives, DOE modified requirements that defined specific components  that a contractor’s oversight system must have. DOE revised the  requirements to allow contractors the flexibility to develop their own  oversight systems, as long as those oversight systems could generally  provide assurance that work was being performed safely.
DOE also revised directives to promote greater use of industry standards.  For example, DOE modified its quality assurance directive to require its  contractors to follow industry standards—or its equivalent—for nuclear  facility quality assurance that were developed by the American Society of  Mechanical Engineers.
In addition, DOE revised directives to refocus federal oversight on the  performance of high-risk activities and placed greater responsibility on  contractors to oversee lower-risk activities. For example, DOE revised a  directive detailing how accident investigations should be carried out by  eliminating one of two categories under which a federally led accident  investigation would take place. Under the revision, a federal investigation  would be initiated using some, but not all, of the criteria previously  established under the two categories. Accidents not meeting these  criteria, including some that would have initiated federal accident  investigations, are the responsibility of the contractors themselves to  investigate.
section_title: DOE Obtained Views on Proposed Changes from DOE and Contractor Staff and from the Safety Board, but Did Not Seek the Views of the Public or Other Federal Agencies
paragraphs: As part of its process to revise its directives, DOE obtained comments  from federal and contractor officials on proposed revisions. For example,  DOE used “red teams”—groups of management-level staff who reviewed  each of the proposed revisions before a revised directive received  departmentwide review. DOE’s red teams reviewed proposed revisions to  directives to determine (1) whether the original requirements provided  adequate safety protection, (2) the impacts of the proposed revisions and  whether unintended consequences were analyzed, and (3) whether the  original requirements provided sufficient flexibility in implementation  without increasing safety risks or whether further revisions were needed.  The red teams were intended to provide an independent review of  proposed revisions to ensure they met the objectives of the reform effort.
DOE generally obtained comments on proposed revisions to its directives  from affected DOE federal and contractor staff by using its departmentwide  review system. This review system—called RevCom—is an online system  that disseminates proposed revisions to DOE federal and contractor staff  for their review and comment on the revisions’ content, relevance,  applicability, accuracy, impact, and implementation cost.
DOE also obtained comments from the Safety Board, which is  responsible for overseeing nuclear safety at DOE and for independently  reviewing proposed revisions to relevant safety directives. DOE officials  told us they generally discussed the reform process with trade unions  whose members are employed at DOE’s sites. However, DOE did not  provide other stakeholders or the public with an opportunity to comment  on proposed revisions to the directives, according to DOE officials we  spoke with. Other external stakeholders that could have provided input  included agencies that have similar safety expertise, such as NRC or  OSHA. NRC and OSHA officials told us that they are required by law to  obtain the public’s views on any changes in their safety regulations, which  generally contain requirements for private employers. Although DOE is  required to obtain comments on changes to its safety regulations, DOE is  not required to obtain the public’s views on its internal directives, many of  which contain requirements for DOE’s contractors. NRC and OSHA  officials told us that obtaining a wide range of comments on proposed  revisions can be very helpful in developing quality regulations that  balance cost with the expected benefits of the regulations. They said that  obtaining a wide range of comments also assists them to build support for  the changes.
section_title: Benefits of DOE’s Reform Effort Are Unclear
paragraphs: Under its reform effort, DOE intended to enhance productivity and reduce  costs while maintaining safety, but it is unclear whether its effort will  achieve these benefits, or whether the benefits will outweigh the costs to  implement the reform. For example, DOE did not determine how the  original requirements contained in safety directives impaired productivity  or added costs before undertaking the reform effort. Moreover, DOE did  not establish clear goals and performance measures to determine the  potential benefits of its reform effort. As a result, DOE is not well  positioned to know that its reform effort will achieve its intended benefits.
subsections: section_title: DOE Sought to Enhance Productivity and Reduce Costs but Did Not First Analyze Burden or Costs of the Original Requirements
paragraphs: According to the Deputy Secretary’s March 2010 memorandum  announcing the reform effort and discussions with DOE officials, DOE  undertook the reform effort to realize productivity and cost benefits while  maintaining safety. According to the memorandum, achieving these  benefits would be accomplished, in part, by revising requirements  considered to be burdensome and overly prescriptive.
To identify safety requirements that were burdensome and overly  prescriptive, DOE’s Associate Deputy Secretary told us that DOE and  contractor officials used their experience and judgment. It is unclear,  however, whether the safety requirements these officials identified were  indeed burdensome because DOE and contractor officials we spoke with  could not provide clear examples of how these requirements negatively  affected productivity or costs or criteria that they used for making a  determination that they were burdensome. Specifically:
At the request of the Secretary of Energy, senior managers at DOE’s  national laboratories participated in an informal May 2011 study of the  DOE policies they considered to be the most burdensome. Four of the  18 most burdensome policies the laboratory managers identified in  this study addressed safety-related issues—excessive oversight;  duplication between directives and industry standards; directives with  requirements that provide no value, including DOE’s quality  assurance directive; and excessive safety reporting. In the study,  however, the laboratory managers did not provide specific examples  of burdensome safety requirements or criteria that could be used for  making a determination of what is burdensome. For example, the  senior laboratory official who led the study could not provide us  examples of specific safety requirements that were burdensome from  the survey data collected from the laboratories’ managers. The official  also could not provide an explanation of how burdensome  requirements affected productivity or costs or how changes to them  would be an improvement.
In response to the concerns of several senior contractor and federal  staff in NNSA that DOE’s nuclear safety requirements may be  burdensome or overly prescriptive, NNSA launched a study to  determine the strength of these concerns. In April 2011, as a result of  the study, NNSA reported that the original requirements did not place  an undue burden on the contractors. The study found that senior  contractor managers considered that DOE nuclear safety  requirements provided a sound framework to safely operate nuclear  facilities. These managers noted, however, that DOE had not clearly  communicated its safety requirements, which resulted in uneven  implementation of these requirements across NNSA’s sites.
In terms of productivity, DOE officials indicated that productivity  improvements from reducing burdensome requirements will be achieved.  For example, several senior DOE officials told us that increases in  productivity would result from raising the threshold at which a nuclear  facility, such as a facility treating radioactive waste, would be required to  undergo a rigorous review process before starting up to ensure it will  operate safely. This review process involves federal staff review of a  contractor’s detailed set of facility startup procedures, contractor  preparation of a detailed plan to correct any deficiencies found during that  review, and preparation of a detailed report indicating how any  deficiencies were resolved. This rigorous review was previously required  whenever any facility’s operations were restarted, even after routine  maintenance. Under the reform effort, this level of review is now only  required for facilities and activities posing a higher risk, such as starting  up a newly constructed nuclear facility, and not required for facilities and  activities posing lesser risk, such as restarting facilities after routine  maintenance. According to two officials from DOE’s Office of Health,  Safety, and Security, the revised requirement will help free up federal  resources to focus on higher-risk activities, such as the startup of a newly  constructed nuclear waste processing facility, instead of lower-risk  activities, which are the responsibility of the contractor to manage safely.
DOE did not analyze the costs of the original requirements or attempt to  quantify how revising or eliminating requirements would reduce costs.  DOE officials told us that cost savings are expected as a result of having  a more streamlined, clear, and nonduplicative set of requirements.  However, according to DOE officials, the department had not developed  estimates of the cost savings it expected. According to DOE’s Associate  Deputy Secretary, the department has launched an initial study of  selected directives to determine how changes to these directives have  impacted mission work, schedule, and costs, among other things.
Moreover, it is not clear whether the reform’s benefits will exceed the  costs of implementing the revised directives, as DOE’s reform effort is  likely to add costs in the short term. Specifically, several officials at DOE’s  Hanford Site told us that contractors will have to perform cost-impact  analyses prior to implementing the revised directives in their contracts.  These officials estimated that it can cost $20,000 to $50,000 to conduct  this type of analysis for each revised directive.
section_title: DOE Did Not Determine the Potential Benefits of Its Reform Effort by Establishing Clear Goals and Performance Measures
paragraphs: As we have previously reported, high-performing organizations typically  follow four key practices to ensure management improvement initiatives  are successfully implemented and achieve their intended benefits.  These practices include: (1) establishing clearly defined goals, (2)  developing an implementation strategy that sets milestones and  establishes responsibility, (3) establishing results-oriented outcome  measures early in the process, and (4) systematically using results- oriented data to evaluate the effectiveness of the initiative and make  additional changes where warranted. Table 1 describes characteristics of  these four key practices, as identified by us and others, including DOE.
DOE did not completely follow these key practices to assess benefits  when revising or planning for the implementation of the department’s  safety directives. Specifically:
Clearly defined goals. DOE’s stated goals—enhancing productivity  and reducing costs while maintaining safe operations—are defined so  generally that they do not lead to the development of tangible,  measurable objectives against which achievement can be compared.  For example, the goal of reducing costs does not specify those safety  areas where DOE believes savings can be achieved. Moreover,  because the goals are so general, DOE site office and contractor  officials expressed concern about the lack of effective communication  about what the reform effort is trying to achieve. For example, some  NNSA officials at the Nevada National Security Site told us that the  goals of the reform effort are unclear. Without clear goals that are  objective, measurable, and effectively communicated to its sites, DOE  will find it difficult to fully assess whether the reform effort is making  progress toward achieving its intended results.
Implementation strategy. DOE developed a near-term implementation  strategy for its reform effort—which included how safety directives  would be revised—but did not develop a long-term strategy on how  those revised directives would be implemented. For near-term  implementation, DOE developed a project management plan, which  focuses on the process to revise the directives but not on how to  implement most of them. In the plan, DOE identifies priority actions  and milestones to be achieved; establishes a process for the  directives reform; and assigns responsibility for leading and managing  the directives reform to DOE’s Office of Health, Safety, and Security.  DOE’s implementation strategy, however, did not provide further  information on how these changes would be implemented over the  long term at DOE’s sites. According to DOE officials, it is up to each  program office within DOE to determine how best to implement the  revised directives. DOE’s Office of Health, Safety, and Security,  however, has developed an implementation strategy for changing its  approach to independent oversight that details how these changes will  be implemented at DOE’s sites.
Outcome measures. DOE has not developed results-oriented  outcome measures for productivity, cost, or safety under its reform  effort. Instead, DOE’s reform effort has focused on output-oriented  measures, such as the number of directives cancelled or revised, and  not on outcome measures, such as specific improvements in  productivity or cost savings. DOE indicated that the reform is  expected to result in more than a 50 percent reduction in the number  of directives but has not explained how this reduction will improve  productivity, reduce costs, or maintain or improve safety. DOE’s focus  on output-oriented measures such as the number of directives  reduced does not align with its own guidance on performance  measures. DOE’s guidance states that there is a danger when  performance measures become numerical quotas because the setting  of numerical goals and quotas does nothing to accomplish process  improvements. For safety performance, DOE has measures that it uses to monitor safety problems at its sites. However, according to  DOE officials, the department has not developed a plan showing how  it could use these measures, if at all, to evaluate the benefits of its  reform effort.
Effectiveness evaluation. DOE did not develop a strategy for  evaluating the effectiveness of its reform effort and therefore is not  collecting the results-oriented performance data necessary to identify  benefits achieved or areas where further changes are needed. To  date, DOE officials have only provided anecdotes on the benefits the  reform is expected to achieve. For example, in response to written  questions we provided to the department, DOE’s Office of Health,  Safety, and Security stated that it expects measurable productivity  improvements because it significantly streamlined the requirements  for contractor oversight systems, which provide information on safety  performance and other management issues. Some officials at DOE  sites we visited, however, stated that DOE’s reform effort may not  produce any productivity or safety improvements and may have little  impact on the way they conduct their work. For instance, DOE and  contractor officials at DOE’s Hanford Site told us that, while they are  still working on implementing the revised directives at their site, they  have neither seen any changes in project safety performance, nor do  they expect to see any changes in the future as a result of the reform  effort. Additionally, as discussed previously, DOE has launched an  initial study of selected directives to determine how changes to these  directives have impacted mission work, schedule, and costs, among  other things. However, without results-oriented performance data, it  will be difficult for DOE to identify benefits achieved by the reform  effort.
section_title: DOE’s Reform Effort Did Not Fully Address Safety Concerns We and Others Have Identified
paragraphs: DOE’s reform effort did not fully address safety concerns that we, DOE’s  Office of Inspector General, and the Safety Board have repeatedly  identified in three key areas—(1) quality assurance, (2) safety culture,  and (3) federal oversight. In each of these three areas, some changes  that DOE has made to its safety directives may begin to address safety  concerns, but other changes it made are potentially problematic.
subsections: section_title: DOE Strengthened Its Quality Assurance Directive by Requiring Contractors to Follow Established Industry Standards, but Some Problems Persist
paragraphs: Quality assurance entails ensuring that proper work processes and  procedures are in place to ensure all work meets certain standards of  quality. Such processes include correcting identified problems and  following up to ensure that these actions are effective. As we, DOE’s  Office of Inspector General, and the Safety Board have reported in 20  reports since 2001, DOE has experienced numerous quality assurance  problems, ranging from improper documentation detailing quality  assurance procedures to insufficient monitoring of actions to correct  identified problems.
Under its reform effort, DOE revised a key quality assurance directive in  April 2011 to update and clarify how the directive applies to facilities with  and without nuclear materials as well as to encourage the use of industry  quality assurance standards. Specifically:  In its revised directive, DOE clarified responsibilities to help ensure  that contractors at facilities without nuclear materials at DOE’s sites  are better aware that they can tailor implementation of their quality  assurance programs to their nonnuclear needs. According to DOE  officials, because problems at nonnuclear facilities may pose different  levels of risk than those at nuclear facilities, quality assurance  programs can be more flexible at DOE’s nonnuclear facilities and  sites.
To support the reform effort’s goal of greater use of industry safety  quality assurance standards, the revised directive includes a new  requirement for contractors overseeing high-hazard nuclear facilities  to use a uniform version of a quality assurance standard developed by  the American Society of Mechanical Engineers. Previously, some  sites were using different versions of the standard. As a result,  according to DOE documents we reviewed, assessments of the  quality assurance practices at DOE’s sites sometimes resulted in  citations that did not concern safety issues but rather arose from  differences in the way that the different versions of the quality  assurance standard were written. By eliminating the use of multiple  versions of the quality assurance standard, DOE believes it can save  time and costs when doing assessments.
The revised quality assurance directive now helps to clarify quality  assurance requirements for DOE’s safety software, which is  consistent with commitments DOE made in response to the Safety  Board’s September 2002 recommendation to improve the quality of  DOE’s safety software and our February 2011 recommendation to  clarify specific quality assurance requirements for computer models  used in environmental cleanup decisions. DOE expects this  clarification to improve the performance of software used on safety- related systems, according to DOE documentation we reviewed.
These revisions appear to have strengthened the directive, but it is not  yet clear if they will fully address quality assurance problems, some of  which continue. For example, DOE’s Office of Enforcement identified  quality assurance problems following a June 2010 incident during which a  worker was exposed to radiation after puncturing his hand with a sharp  object contaminated with plutonium at DOE’s Savannah River Site. As a  result, in July 2011, this office proposed a fine of $243,750 against the  contractor. Furthermore, in June 2011, DOE’s Office of Inspector General  reported that both the Livermore Site Office and the laboratory’s  contractor lacked adequate quality assurance processes to ensure that  corrective actions were effective. Inspector General recommended that the site office perform more timely  inspections to verify that corrective actions had been instituted prior to  closing the finding that led to the corrective actions. In addition, at a  nuclear facility at NNSA’s Nevada National Security Site, NNSA found in  August 2011 that the contractor had not effectively identified, controlled,  or prevented the recurrence of quality assurance problems related to the  installation and inspection of fire safety and other components in nuclear  testing facilities, as required in nuclear safety regulations, and proposed a  nearly $180,000 fine against the contractor.
DOE, Office of Inspector General, Implementation of Beryllium Controls at Lawrence  Livermore National Laboratory, DOE/IG-0851 (Washington, D.C.: June 17, 2011).
section_title: DOE’s Revisions to Its Directives Were Intended to Strengthen Its Safety Culture, but the Safety Board Raised Concerns That Less Vigorous Safety Practices Could Result
paragraphs: A strong safety culture includes the quality that health and safety are  integrated into all work decisions so that workers and managers are  better aware of and better implement safety practices, which helps to  minimize accidents. DOE defines safety culture as an organization’s  values and behaviors modeled by leadership and practiced by employees  to strive to make the safe performance of work a priority. To promote a  strong safety culture at its sites, DOE has federal regulations and a set  of directives—collectively known as Integrated Safety Management—for  implementing the department’s key safety management practices. These  directives lay out high-level expectations and implementing methods for  safety management and include several guides and manuals with  suggested methods for implementing safety management practices.  Many of these directives resulted from DOE’s response to October 1995  and September 1998 recommendations by the Safety Board to establish  clear and systematic processes for addressing and resolving safety  issues and to establish clear lines of responsibility for safety.
Nevertheless, DOE has experienced problems maintaining an effective  safety culture at some of its sites, as we, DOE’s Office of Inspector  General, and the Safety Board have repeatedly reported. Problems have  included not following required safety practices, inadequate testing of key  nuclear waste processing systems to ensure they operate safely, and not  developing safety strategies for extremely hazardous events. For  example, in October 2007, we identified weaknesses in the safety culture  at DOE’s three nuclear research laboratories, and recommended that  DOE strengthen safety management at its weapons laboratories and  ensure that safety improvements are carried out in a systematic  manner. DOE generally agreed with our recommendations, but it noted that, in the department’s view, the safety management systems DOE had  in place adequately addressed safety culture issues and, as a result, the  department took no further action to address those recommendations. In  addition, in September 2009, DOE’s Office of Inspector General reported  that fire fighters who respond to incidents at Los Alamos were not  provided with enough opportunity to tour and become familiar with the  site, which would impede response time in an emergency. Furthermore,  in October 2009, the Safety Board reported that Los Alamos lacked a  safety strategy to minimize the potential consequences of earthquakes to  stored nuclear material and recommended actions to mitigate these  potential consequences.
Under its reform effort, DOE revised its Integrated Safety Management  directives to attempt to strengthen the safety culture at its sites. For  example, DOE’s revisions more clearly emphasized DOE and contractor  senior management roles and responsibilities for building a strong safety  culture at DOE’s sites. Specifically, the new directives lay out tasks that  each level of management should complete, such as developing and  maintaining safety documents, approving site office safety initiatives, and  establishing safety expectations across DOE’s sites. These revisions  address past issues, such as lack of clarity in roles and responsibilities for  ensuring safety practices at DOE’s sites. In addition, DOE has  undertaken several efforts aimed at improving safety practices across its  sites. For example, the department has worked closely with a contractor  group—Energy Facilities Contractors Group—to issue guidance on  assessing and strengthening safety culture. In addition, DOE issued a  new Integrated Safety Management guide in September 2011 to provide  additional and extensive information to assist the programs in  implementing integrated safety management requirements. Finally, in  2011, DOE established its Nuclear Safety and Security Council, which  advises the department’s senior management on nuclear safety issues  with the aim to improve safety performance.
However, some revisions DOE made to its Integrated Safety  Management directives raised concerns with Safety Board officials that  safety culture may be undermined. For example:
When it revised its Integrated Safety Management directives, DOE  removed language that required contractors to follow the directives  and instead revised the directives to apply only to federal employees.  This is because contractors already had to comply with the  department’s acquisition regulations, which the Safety Board noted  contain similar but less detailed requirements. According to Safety  Board officials, because the acquisition regulation does not contain  the specificity of the Integrated Safety Management directives,  contractors may not implement the requirements as vigorously as they  would if they had to follow the more specific practices in the original  directives.
Because DOE removed language for contractors to follow the  department’s Integrated Safety Management directives, contractors  will no longer need to consider best practices for implementing a  safety culture based on lessons learned over the years that are  integrated into the directives. Specifically, Safety Board officials stated  that they are concerned that contractors will therefore overlook these  lessons learned since the contractors are no longer required to follow  the Integrated Safety Management directives.
It is also not clear how revisions to DOE’s Integrated Safety Management  directives will affect safety culture problems that persist at DOE’s sites.  For example, in June 2011, the Safety Board reported that workers at  DOE’s Waste Treatment and Immobilization Plant at its Hanford Site  hesitated to raise safety concerns for fear of management reprisal and  recommended that the Secretary of Energy act to ensure a strong safety  culture at the site. DOE’s Office of Health, Safety, and Security has also  recently reported on continuing safety culture problems at the Waste  Treatment and Immobilization Plant. Furthermore, DOE’s Office of  Enforcement issued a notice of violation in September 2011 to the  contractor at DOE’s Separations Process Research Unit site in  Niskayuna, New York, after a building demolition accident potentially  exposed at least 100 employees to low-level radioactive contamination in  September 2010. DOE’s Office of Enforcement found that the accident  was tied to the contractor’s failure to consistently use safe work practices  at the site and management’s failure to assess work processes to ensure  a safe working environment.
section_title: DOE Clarified Roles and Responsibilities for Its Oversight of Contractor Activities, but Its Reform May Result in Weakened Federal Oversight
paragraphs: We and others have reported on DOE’s problems overseeing contractors’  performance. These problems have ranged from the department not  ensuring that it had sufficient resources to perform oversight at its sites to  not consistently applying such oversight. For example, in October 2008,  we reported that DOE’s authority to impose civil penalties on contractors  for violating nuclear safety requirements had not reduced some recurring  safety problems, and we recommended that DOE increase its  independent oversight presence at DOE’s sites to provide more frequent  observations to facilitate enforcement actions.
In response to our recommendations, and under its reform effort, DOE  has changed its approach to conducting independent oversight that is  reflected in its revised oversight directives. For example:
DOE has placed greater emphasis on having its Office of Independent  Oversight staff review safety design documents, which lay out the  safety systems for facilities at DOE’s sites, before their construction.  Our October 2008 report noted that lack of oversight meant that these  documents did not receive an independent review and therefore the  safe operation of newly constructed facilities at DOE’s sites could not  be reasonably assured. In response to our recommendations and  under its reform effort, DOE’s Office of Independent Oversight staff  now review the safety design documents before the facility is  constructed, rather than after it is built, and when the facility  undergoes major modification.
DOE implemented a new approach to oversight that designates  headquarters staff responsibility for oversight at key sites, such as  DOE’s Hanford and Savannah River sites and its Idaho National  Laboratory. These “site leads” monitor and assess all aspects of a  site’s operations in order to assist the Office of Independent Oversight  in planning assessment activities. According to Office of Independent  Oversight officials, under the site lead approach, oversight staff now  visit a site several times per year rather than approximately once  every 3 years under the previous approach. These officials said that  they believe this change will result in more frequent reviews at these  key sites, enabling them to catch potential safety problems earlier.
However, some of the revisions DOE made to its oversight directives  under its reform effort raise concerns about the department’s ability to  effectively execute independent oversight. According to DOE’s  independent oversight directive, independent oversight is an integral part  of DOE’s ability to help ensure the effectiveness of DOE and contractor  safety performance. However, some of DOE’s revisions place  accountability for key oversight functions with the site offices and  contractors, which may reduce confidence in an independent oversight  process. For example:
DOE revised its oversight approach to give site office, rather than  Office of Independent Oversight staff, increased responsibility for  managing actions to correct problems raised in independent  assessments. Previously, issues identified during Independent  Oversight assessments had to be addressed with a formal corrective  action plan submitted to and tracked by the Office of Independent  Oversight. Under the reform, site offices have responsibility for  determining which issues will be formally addressed and monitored,  and which problems are dealt with without the need for a formal plan.  According to some Office of Independent Oversight officials, under  this revised approach, independent oversight staff try to limit the  concerns raised during assessments to higher priority issues in order  to better ensure these receive site management attention. Site office  determinations of whether issues required more formal contractor  responses may be influenced by the fact that the site offices also have  responsibility for keeping costs under control and work on schedule.
In giving greater responsibility to the site offices, DOE has allowed  them to use staff from the Office of Independent Oversight to help  assess contractor performance. In addition, Office of Independent  Oversight staff must now coordinate their assessment activities with  site office management in order to maximize the use of resources.
This potentially raises concerns about whether Office of Independent  Oversight staff will be sufficiently independent from site office  management. Furthermore, Safety Board officials told us that the  coordination requirement could reduce the effectiveness of Office of  Independent Oversight assessments because surprise “drop-in”  assessments would be lost, allowing site office and contractor staff to  prepare ahead of the assessment. We have raised similar concerns in  the past regarding the independence of DOE’s oversight, including  concerns that the department’s focus on providing program offices a  greater role in overseeing contractor oversight resulted in weakening  independent review.
Even after DOE’s oversight directives were revised, effective oversight of  contractor activities continues to be a problem for DOE. For example, in  June 2011, DOE’s Office of Inspector General reported that the Livermore  site office was not sufficiently overseeing the contractor to ensure that  corrective actions were fully and effectively implemented. In September 2011, DOE’s Office of Enforcement proposed a fine of $412,500 against  the Separations Process Research Unit contractor in Niskayuna, New  York, due, in part, to oversight inadequacies during the September 2010  accident discussed earlier in this report. Moreover, the Office of  Enforcement proposed a $150,000 fine in early October 2011 against the  Idaho National Laboratory contractor, in part, because of inadequate  management oversight that resulted in the tipping over of a hoisting rig  loaded with 7,800 pounds.
section_title: Conclusions
paragraphs: DOE’s reform effort was driven by a belief that some requirements in its  safety directives have placed excessive burdens on its contractors and  that revisions would give contractors more flexibility in operating safely.  The elimination of potentially burdensome, duplicative, and costly  requirements that offer little or no contribution to improved safety is a  worthy goal. If high levels of safety could be assured at DOE’s sites while  unnecessary requirements were eliminated, then DOE’s reform effort  would be considered a success. However, DOE did not first determine  whether the directives it planned to revise were indeed burdensome or  costly. DOE also does not have results-oriented outcome measures to  help the department gauge the effectiveness of its effort. Simply counting  the number of directives revised or eliminated does not indicate the  benefit of the reform on productivity and safety performance at DOE’s  sites. Safety should not be measured by the amount of paper that is  saved but by actual improvements in safety performance across the  department.
Before embarking on management reforms, high-performing  organizations (1) set clear goals, (2) develop an implementation strategy  that sets milestones and establishes responsibility, (3) establish results- oriented outcome measures, and (4) use results-oriented data to evaluate  the reform’s effectiveness and make additional changes where needed.  However, DOE did not fully follow these practices to assess the costs and  benefits of revising its safety directives. Furthermore, our discussions with  DOE and contractor officials at DOE’s sites indicate that there is some  question about the benefits the reform effort can actually achieve. It is  also unclear how revising safety directives will address past and recurring  safety concerns with quality assurance, safety culture, and federal  oversight of contractor activities that we and others have identified over  the years. Many of the directives DOE revised were originally developed  to correct problems in these areas. In revising these directives, DOE  could be undermining hard-won progress over the years in safety  performance at its sites. DOE has not effectively communicated with its  site officials or its contractors about the overall goals of the reform effort,  how progress will be measured, and whether benefits of the reform will  exceed the costs involved with revising directives. In addition, changes to  DOE’s oversight directives raise concerns about the ability of DOE’s  Office of Independent Oversight staff to provide a critical review of safety  at DOE’s sites that is independent from DOE site office and contractor  staff, a concern we have raised in the past. As a result, it is unclear  whether DOE’s reform effort will result in actual productivity or safety  improvements at its sites.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that DOE’s reform of its safety directives results in  improved productivity and safety at its sites, we recommend that the  Secretary of Energy take the following four actions prior to fully  implementing revisions to its directives across all of the department’s  sites:  systematically analyze the costs and benefits associated with  implementing the revised safety directives to ensure that the costs do  not exceed the benefits that the department expects to achieve; provide DOE’s sites and contractors with a plan that details (1) the  reform effort’s goals, (2) the effort’s long-term implementation  strategy, (3) results-oriented outcome measures, and (4) how DOE  will use results-oriented data to evaluate the reform’s effectiveness  and to determine whether additional changes are needed; ensure that the plan developed for DOE’s sites and contractors  identifies how the reform effort will help address past and recurring  safety concerns with quality assurance, safety culture, and federal  oversight of contractor activities; and  clearly define the oversight roles and responsibilities of DOE’s Office  of Independent Oversight staff to ensure that their work is sufficiently  independent from the activities of DOE site office and contractor staff.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided DOE with a draft of this report for its review and comment. In  its written comments, DOE generally agreed with our recommendations  and noted that it will use the report to build on its commitment to  continually improve its approach to safety management. However, DOE  commented that it had significant concerns with the accuracy of the  report’s findings and the validity of its conclusions.
Specifically, DOE commented that it never assumes its existing directives  are perfect and cannot be improved. Instead, DOE stated that its  management principles require it to constantly evolve, engaging with its  employees, contractors, and stakeholders to recognize impediments and  work to mitigate them. We agree with DOE and, as our draft report noted,  the elimination of potentially burdensome, duplicative, and costly  requirements that offer little or no contribution to improved safety is a  worthy goal. Our draft report also clearly described the process DOE  followed and the involvement of its employees; contractors; and  stakeholders, such as the Safety Board. However, in our view, it is not  possible to identify impediments and work to mitigate them without first  determining whether and how existing requirements are burdensome. As  our draft report stated, an April 2011 NNSA study found that DOE’s  nuclear safety requirements did not place an undue burden on the  contractors. Instead, NNSA reported that these requirements provided a  sound framework to safely operate nuclear facilities. Officials we spoke  with at some of DOE’s sites also told us that they lacked sufficient data to  determine whether the safety requirements the department was revising  were burdensome. We agree that continuous improvement to remove  impediments is beneficial, but DOE did not determine whether its safety  directives were, in fact, impediments prior to revising them.
In addition, DOE’s letter provided information on the roles of its Office of  Health, Safety, and Security and noted that its oversight and regulatory  enforcement is independent from line management, guaranteeing the  office’s autonomy without potential conflicts of interest. Our draft report  clearly described the roles and responsibilities of DOE’s Office of Health,  Safety, and Security and its Office of Independent Oversight. However,  some of DOE’s revisions to its directives place accountability for key  oversight functions with DOE site offices and contractors, which may  reduce confidence in an independent oversight process. For example, as  we noted in our draft report, Office of Independent Oversight staff must  now coordinate assessment activities with site office management in  order to maximize the use of resources. Safety Board officials told us that  the coordination requirement could reduce the effectiveness of  assessments because surprise “drop-in” assessments would be lost,  allowing site offices and contractor staff to prepare ahead of the  assessment. This problem echoes similar concerns we have raised in the  past regarding the independence of DOE’s oversight, including that the  department’s focus on providing program offices a greater role in  overseeing contractor oversight resulted in weakening independent  review.
DOE also enclosed a longer summary of its concerns with the draft  report’s findings and conclusions. DOE’s letter, summary, and our  detailed responses are reproduced in appendix II. DOE also provided  technical comments, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of to the appropriate  congressional committees; the Secretary of Energy; the Director, Office of  Management and Budget; and other interested parties. In addition, the  report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions regarding this report, please contact  me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To examine how the Department of Energy (DOE) revised safety  directives under its reform effort, we reviewed DOE documents describing  the directives review process, including the March 2010 Safety and  Security Reform Plan, the June 2010 Project Management Plan, and the  September 2010 HSS Requirements Framework and Systems Approach:  Integrating Requirements Framework and Overview. We also reviewed  documents used to revise safety directives, including justification  memorandums and crosswalks between the original directives and their  revisions. We also reviewed comments to directive revisions that were  submitted to the department’s directives review system. This review  system—called RevCom—is an online system that disseminates  proposed revisions to DOE federal and contractor staff for their review  and comment on the revisions’ content, relevance, applicability, accuracy,  impact, and implementation cost. We interviewed officials at DOE  headquarters, including officials in DOE’s Office of Health, Safety, and  Security and the National Nuclear Security Administration (NNSA), about  the directives review process. In addition, we visited the Kansas City  Plant in Kansas City, Missouri; the Nevada National Security Site in Las  Vegas, Nevada; the Lawrence Livermore National Laboratory in  Livermore, California; and the Hanford Site near Richland, Washington  state; and interviewed DOE and contractor officials at those sites to  determine how the safety directives and oversight reform affected the  sites and contractors, if at all. Each of these sites is in different stages of  implementing revised directives, from full implementation of revised  directives at the Kansas City Plant to revised directives not yet  implemented at the Lawrence Livermore National Laboratory. We also  interviewed officials at other federal regulatory agencies, including the  Nuclear Regulatory Commission and the Department of Labor’s Office of  Occupational Health and Safety, to gain an understanding of the process  of changing safety requirements and federal regulations. We reviewed  comments on proposed revisions to DOE’s safety directives made by the  Defense Nuclear Facilities Safety Board (Safety Board)—which was  created by Congress to independently assess safety conditions and  operations at defense nuclear facilities at DOE’s sites and interviewed  Safety Board officials about those changes. We also interviewed officials  from the American Society of Industrial Standards to determine how  industry standards are developed.
To examine the benefits DOE hoped to achieve through its reform effort,  we reviewed the reform documents mentioned previously, among others.  We interviewed officials at DOE headquarters, including the Associate  Deputy Secretary of Energy and staff from the Office of Health, Safety,  and Security, to discuss how DOE intended to gauge the success of its  reform effort. We also interviewed officials with NNSA to discuss the DOE  reform effort as it affected NNSA’s simultaneous governance and  oversight reform effort. NNSA’s effort involves revising the agency’s  business model to, among other things, place more reliance on  contractor’s self-oversight through its contractor assurance systems.  Furthermore, we visited the Kansas City Plant, the Nevada National  Security Site, Lawrence Livermore National Laboratory, and the Hanford  Site and interviewed DOE site office and contractor officials about the  2010 reform, site performance measures, expected productivity  improvements, and whether cost savings were anticipated. To identify  and develop best practices criteria for managing improvement initiatives  and performance measures, we reviewed our prior work, DOE guidance  on managing improvement initiatives and performance measures, and  documentation on performance measures and performance evaluation by  industry groups such as the Energy Facility Contractors Group. We then  compared these best practices criteria to DOE’s documents describing its  reform effort, including the March 2010 Safety and Security Reform Plan,  the June 2010 Project Management Plan, and the August 2010 Program  Plan for Enhancing Independent Oversight of Nuclear Safety, in order to  determine how these best practices criteria were applied to these  documents. We also verified this information by speaking with DOE  officials about these best practices criteria to obtain their perspectives on  how these criteria were applied to the reform effort. To identify other  safety performance measures DOE uses to monitor safety performance  at DOE’s sites, we reviewed DOE’s Office of Health, Safety, and Security  worker safety performance reports and data, as well as performance  measures, performance data, and performance evaluation plans provided  to us by site office and contractor officials during our site visits. We did  not validate these data, but we did examine the reasonableness of these  data based on information in our prior work and DOE reports and  concluded that these were sufficient for purposes of our report. We also  interviewed officials from the Safety Board on issues pertaining to  performance measures and performance evaluation.
To examine the extent to which DOE’s reform effort will address concerns  we and others identified, we reviewed reports over the past 10 years by  GAO, DOE’s Office of Inspector General, and the Safety Board that  contain recommendations related to safety and oversight issues at the  department. To identify the safety-related concerns of these report  recommendations, we conducted a content analysis of the  recommendations. Specifically, we developed categories for identifying  safety-related concerns—including quality assurance, nuclear safety,  worker safety, and oversight—using DOE’s HSS Requirements  Framework and Systems Approach: Integrating Requirements Framework  and Overview document. After discussions with our methodologist and a  review of DOE safety directives, we combined the nuclear safety and  worker safety categories under the category of safety culture. Two  analysts independently reviewed the report recommendations and coded  them into the three categories. In cases where differences between the  two reviewers regarding the coding of the recommendations into these  safety-related categories were found, all differences were resolved  through reviewer discussion. To determine how DOE’s reform effort  addressed past safety concerns, we reviewed key safety directives,  crosswalks between the previous version of the directive and the current  version, comments provided on those changes from DOE staff and  contractors, and comments provided by the Safety Board. We also visited  and interviewed staff at the Kansas City Plant, the Nevada National  Security Site, Lawrence Livermore National Laboratory, and the Hanford  Site to determine how the reform effort was affecting their day-to-day  operations and strategies to address past concerns on-site. We also  interviewed DOE officials, including at the Office of Health, Safety, and  Security, as well as Safety Board officials.
We conducted this performance audit from February 2011 through April  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Energy
subsections: section_title: GAO Comments
paragraphs: 1.  DOE is incorrect that the draft report did not identify any significant  flaws in the revised directives that resulted from the department’s  safety reform effort. On the contrary, the draft report discussed  concerns that the Defense Nuclear Facilities Safety Board (Safety  Board) and we had with revisions to DOE’s Integrated Safety  Management and independent oversight directives. In addition, DOE  is confident that the directives will, among other things, effectively  support mission requirements in a safe and efficient manner. In our  view, however, DOE cannot demonstrate that the reform effort will  result in benefits because it failed to develop outcome-based  performance metrics.  2.  Our draft report clearly described the role of the Safety Board in  DOE’s safety reform effort, including the Safety Board’s concerns with  DOE’s revisions to its Integrated Safety Management directive.  However, contrary to DOE’s comment, Safety Board officials with  whom we spoke stated that, during the reform process, they had  expressed concerns about some of the changes DOE made to its  safety directives, including its Integrated Safety Management and  independent oversight directives. The Safety Board staff indicated that  they did not believe those original concerns had been fully addressed  and that they continue to have concerns about some of the changes  made. Our draft report discussed these concerns, particularly as they  relate to independent oversight.  3.  Our draft report did not suggest requiring the department to seek  public comment when revising its internal directives. Instead, our draft  report clearly stated that DOE is not required to obtain the public’s  views on its internal directives. However, we did note that officials at  NRC and OSHA—federal agencies with significant experience in  nuclear and worker safety, respectively—told us that obtaining a wide  range of comments on proposed revisions can be very helpful in  developing quality requirements that balance cost with benefits and  assist them to build support for the changes. We revised our report to  further emphasize that DOE is not required to obtain public comments  when making changes to its internal directives. However, we maintain  that obtaining a wide range of comments from a variety of  stakeholders is beneficial.  4.  DOE agrees that it did not conduct an analysis of the costs and  benefits of its requirements prior to revising them, stating that such an  approach would be inconsistent with the department’s approach to  safety and to its management principles. In our view, DOE’s  comments indicate that the department’s priority when revising its  directives was to act quickly rather than to use the key practices of  high-performing organizations when undertaking management  improvement initiatives. As our draft report noted, high-performing  organizations set clearly defined goals that are targeted levels of  performance, expressed as tangible, measurable objectives against  which achievement can be compared. DOE did not do this—as DOE  itself notes in its comments—opting for speed instead of  effectiveness. We maintain, as our draft report noted, that simply  counting the number of directives revised or eliminated does not  indicate the benefit of the reform on productivity and safety  performance at DOE’s sites and that safety should not be measured  by the amount of paper that is saved but by actual improvements in  safety performance. In our view, when it comes to rigorous safety  requirements, the speed of a reform effort is far less important than  the effectiveness of the reform.  5.  DOE states that its safety reform resulted in more concise safety  directives with requirements that are clearer and add value in  protecting its workers and the public, and that the results are  consistent with the goals established by the Deputy Secretary for the  safety reform effort. As our draft report noted, however, one of the  goals expressed by the Deputy Secretary in his March 2010  memorandum announcing the reform effort was that “measurable  productivity improvements” would result from reducing burdensome  safety requirements. Because DOE made no attempt to determine  whether these requirements were burdensome—which DOE itself  acknowledges in its comments (see comment 4)—and has no  outcome-oriented metrics to assess the performance of the safety  reform effort, there is no way for the department to demonstrate that  measurable productivity improvements will result from its revisions to  its safety directives.  6.  DOE’s comment that it does not measure success by the number of  directives eliminated is not supported by the evidence. For example,  the Deputy Secretary’s March 2010 memorandum announcing the  reform effort stated that the reform was expected to result in more  than a 50 percent reduction in the number of directives. Furthermore,  in numerous meetings, DOE officials cited examples of the  accomplishments of the safety reform effort. In particular, during a  November 2011 meeting, the Associate Deputy Secretary of Energy  provided us with a copy of DOE’s fiscal year 2011 “Management and  Operational Excellence Performance Results” that cited the  elimination of more than 1,650 pages from DOE directives, the  cancellation of 22 directives, and the revision of 26 other directives as  an accomplishment of the reform effort.  7.  Our draft report did not assert that inadequate directive requirements  caused a variety of performance weakness at DOE sites over the last  several years. Our point was not that inadequate safety directives  caused these weaknesses, but that DOE’s revisions to these  directives did not correct these weaknesses. We agree with DOE that  safety performance problems are often not related to deficiencies in  requirements, but rather in the implementation of the requirements. In  our view, this increases the importance of ensuring directives are  adequately implemented rather than spending inordinate time and  staff energy revising those directives to eliminate “burdens” that have  not been clearly defined.  8.  DOE is correct that safety management and safety culture are  different, but related, concepts. Nevertheless, as DOE itself  acknowledges in its comments, the department has used the revisions  to its Integrated Safety Management directives as one means to  further clarify and strengthen expectations and requirements for safety  culture. For example, our draft report stated DOE revised its  Integrated Safety Management directives to, among other things,  attempt to strengthen safety culture at its sites by more clearly  emphasizing DOE and contractor senior management roles and  responsibilities for building a strong safety culture at DOE’s sites.  Nevertheless, as our draft report noted, Safety Board officials raised  concerns that some of these changes—in particular the removal of  language that required contractors to follow the directives and  allowing them to follow similar but less detailed requirements in DOE’s  acquisition regulations—could weaken safety culture at DOE’s sites.  However, to address DOE’s concerns that the report does not  adequately acknowledge the department’s efforts to improve safety  culture, we revised the report to add additional examples of recent  DOE safety culture improvement initiatives.  9.  DOE stated that it has been the department’s long-standing practice  to coordinate oversight activities with line organizations to ensure  adequate planning and that this coordination is not a change from  previous order expectations. However, the revised order states that  “Independent Oversight appraisal activities must be coordinated with  affected DOE line management and staff offices to promote efficient  and effective use of resources.” The previous order that the revised  order replaced did not contain similar language. The previous order  only required coordination between its oversight activities and DOE’s  sites on the development of corrective actions to any deficiencies  found. Although both orders note that independent oversight  activities must be independent of all DOE elements that have line  management responsibilities for safety and security programs, Safety  Board officials with whom we spoke stated that they were concerned  that this revision would result in surprise “drop-in” assessments being  lost, allowing site office and contractor staff to prepare ahead of the  assessment.  10. We revised the report to discuss the establishment of the Nuclear  Safety and Security Council.  11. As DOE itself acknowledges in its comments (see comment 4), the  department did not analyze the costs and benefits of its requirements  prior to revising them. Therefore, we maintain that the draft report’s  title is accurate.  12. Our draft report discussed in detail the process DOE followed in  revising its directives, including the line-by-line review of each safety  directive and its consultations with Safety Board staff, and we  therefore made no changes to the report in response to this comment.  13. Our draft report noted that DOE has measures that it uses to monitor  safety problems at its sites. However, the department has not been  able to demonstrate how it could use these measures, if at all, to  evaluate the benefits of its reform effort, nor has it developed other  outcome-based metrics that could also be used. In our view, this is  not consistent with one of the Deputy Secretary’s goals for the reform  effort—namely, that measurable improvements in productivity would  result from the elimination of burdensome requirements while  maintaining safety. Because DOE never defined “burdensome”  through an analysis of costs and benefits and never developed  outcome-based measures of the performance of the safety reform  effort, we are left with merely DOE’s assurances that improved  productivity and safety performance will result rather than objective  and verifiable data that could adequately demonstrate improvement.  14. We acknowledge DOE’s acceptance of our recommendations and will  monitor the department’s implementation of them.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Ryan T. Coles, Assistant  Director; Pamela Davidson; Carlos Gomez; Nancy Kintner-Meyer; Cheryl  Peterson; Kevin Remondini; and Carol Herrnstadt Shulman made key  contributions to this report.
section_title: Related GAO Products
paragraphs: 2012 Annual Report: Opportunities to Reduce Duplication, Overlap and  Fragmentation, Achieve Savings, and Enhance Revenue.   GAO-12-342SP. Washington, D.C.: February 28, 2012.
National Nuclear Security Administration: Observations on NNSA’s  Management and Oversight of the Nuclear Security Enterprise.   GAO-12-473T. Washington, D.C.: February 16, 2012.
Excess Uranium Inventories: Clarifying DOE’s Disposition Options Could  Help Avoid Further Legal Violations. GAO-11-846. Washington, D.C.:  September 26, 2011.
Nuclear Material: DOE’s Depleted Uranium Tails Could Be a Source of  Revenue for the Government. GAO-11-752T. Washington, D.C.:   June 13, 2011.
Nuclear Waste: Disposal Challenges and Lessons Learned from Yucca  Mountain. GAO-11-731T. Washington, D.C.: June 1, 2011.
DOE Nuclear Waste: Better Information Needed on Waste Storage at  DOE Sites as a Result of Yucca Mountain Shutdown. GAO-11-230.  Washington, D.C.: March 23, 2011.
Nuclear Waste: DOE Needs a Comprehensive Strategy and Guidance on  Computer Models that Support Environmental Cleanup Decisions.   GAO-11-143. Washington, D.C.: February 10, 2011.
Nuclear Waste: Actions Needed to Address Persistent Concerns with  Efforts to Close Underground Radioactive Waste Tanks at DOE’s  Savannah River Site. GAO-10-816. Washington, D.C.:   September 14, 2010.
Recovery Act: Most DOE Cleanup Projects Appear to Be Meeting Cost  and Schedule Targets, but Assessing Impact of Spending Remains a  Challenge. GAO-10-784. Washington, D.C.: July 29, 2010.
Department of Energy: Actions Needed to Develop High-Quality Cost  Estimates for Construction and Environmental Cleanup Projects.   GAO-10-199. Washington, D.C.: January 14, 2010.
Nuclear Waste: Uncertainties and Questions about Costs and Risks  Persist with DOE’s Tank Waste Cleanup Strategy at Hanford.   GAO-09-913. Washington, D.C.: September 30, 2009.
Department of Energy: Contract and Project Management Concerns at  the National Nuclear Security Administration and Office of Environmental  Management. GAO-09-406T. Washington, D.C.: March 4, 2009.
Nuclear Waste: DOE Lacks Critical Information Needed to Assess Its  Tank Management Strategy at Hanford. GAO-08-793. Washington, D.C.:  June 30, 2008.
Hanford Waste Treatment Plant: Department of Energy Needs to  Strengthen Controls over Contractor Payments and Project Assets.   GAO-07-888. Washington, D.C.: July 20, 2007.
Nuclear Waste: DOE Should Reassess Whether the Bulk Vitrification  Demonstration Project at Its Hanford Site Is Still Needed to Treat  Radioactive Waste. GAO-07-762. Washington, D.C.: June 12, 2007.
Hanford Waste Treatment Plant: Contractor and DOE Management  Problems Have Led to Higher Costs, Construction Delays, and Safety  Concerns. GAO-06-602T. Washington, D.C.: April 6, 2006.
Nuclear Waste: Absence of Key Management Reforms on Hanford’s  Cleanup Project Adds to Challenges of Achieving Cost and Schedule  Goals. GAO-04-611. Washington, D.C.: June 9, 2004.
Nuclear Waste: Challenges to Achieving Potential Savings in DOE’s  High-Level Waste Cleanup Program. GAO-03-593. Washington, D.C.:  June 17, 2003.
Nuclear Waste: Department of Energy’s Hanford Tank Waste Project— Schedule, Cost, and Management Issues. GAO-RCED-99-13.  Washington, D.C.: October 8, 1998.