Yucca Mountain: Persistent Quality Assurance Problems Could Delay Repository Licensing and Operation
Published: 2004-04-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Energy (DOE) must obtain a license from the Nuclear Regulatory Commission (NRC) to construct a nuclear waste repository at Yucca Mountain, Nevada. In licensing, a quality assurance program helps ensure that the information used to demonstrate the safety of the repository is defensible and well documented. DOE developed a corrective action plan in 2002 to fix recurring problems with the accuracy of such information. This report assesses the status of corrective actions and the adequacy of DOE's plan to measure the effectiveness of actions taken.
section_title: What GAO Found
paragraphs: DOE has reportedly implemented most of the actions in its 2002 corrective action plan, but recent audits and assessments have identified lingering quality problems with data, models, and software and continuing management weaknesses. Audits revealed that some data sets could not be traced back to their sources, model development and validation procedures were not followed, and some processes for software development and validation were inadequate or not followed. DOE believes these problems have not affected the technical basis of the project; however, they could adversely affect the licensing process. Recent assessments identified continuing management weaknesses in the areas of roles and responsibilities, quality assurance policies and procedures, and a work environment that did not foster employee confidence in raising concerns without fear of reprisal. NRC has acknowledged DOE's effectiveness in identifying quality problems, but recently concluded that quality problems could delay the licensing process. DOE cannot assess the effectiveness of its 2002 plan because the performance goals to assess management weaknesses lack objective measurements and time frames for determining success. The goals do not specify the amount of improvement expected, how quickly the improvement should be achieved, or how long the improvement should be sustained before the problems can be considered corrected. DOE recently developed a measurement tool that incorporates and revises some of the goals from the action plan, but most of the revised goals continue to lack the necessary time frames needed to determine whether the actions have corrected the recurring problems. A recently completed DOE review of the 2002 plan found that the corrective actions have been fully implemented. However, the review also noted the effectiveness of the actions could not be evaluated because many of the plan's goals lacked the level of objectivity and testing needed to measure effectiveness.

section_title: Letter
paragraphs: High-level nuclear waste, created as a by-product of the nuclear power  process in reactors, can remain highly radioactive for hundreds of  thousands of years, endangering the public if not properly disposed.   Storing this waste safely is therefore of vital interest to the nation.   Currently, more than 50,000 metric tons of this waste is being stored at 72  sites across the country.  In 2002, Congress approved the President’s  recommendation of Yucca Mountain, Nevada, 90 miles from Las Vegas, as a  suitable site for the Department of Energy (DOE) to construct and operate  a geologic repository to safely and permanently dispose of this waste.  To  construct and operate the repository, DOE must obtain a license from the  Nuclear Regulatory Commission (NRC).  As part of the license application,  DOE must, among other things, demonstrate an effective quality assurance  program that ensures the safe construction and operation of a repository,  protecting public health and safety.  DOE plans to submit a license  application by December 2004 and is following a demanding schedule to  meet this date.  NRC is reviewing an extensive amount of data as part of a  prelicensing agreement with DOE.
Before granting a license, NRC requires nuclear facilities to develop a  quality assurance program that ensures that the technical information  submitted in support of a license application—such as scientific data,  models, and details on design and construction—is well documented and  defensible.  The quality assurance program involves a two-part process that  (1) requires program staff to follow procedures to help ensure the  reliability of information and (2) uses quality assurance auditors to verify  that the procedures have been followed.  Both program staff and quality  assurance auditors are required to identify when procedures are not being  followed or when they encounter problems with the procedures.  DOE and  contractor quality assurance auditors periodically assess compliance with procedures.  In cases where a significant problem is found (DOE’s criteria  refer to significant problems as significant conditions adverse to quality),  quality assurance personnel and program managers follow specific steps to  analyze and correct the problem: 1. Quality assurance auditors or program personnel complete a corrective  action report that describes the problem and the need for corrective  action. 2. Program managers conduct a root-cause analysis of the problem.   3. Program managers identify corrective actions that address the root  cause(s) to prevent the problem from recurring—these actions are  included in a corrective action plan.   4. Program managers implement these corrective actions and quality  assurance personnel verify that they have been implemented. 5. Quality assurance personnel close the corrective action report, and  program managers later conduct an effectiveness review.   a.
If actions are determined ineffective, the process begins again with  the issuance of a new corrective action report. b.
In cases involving more significant problems, an effectiveness  review may be conducted prior to closing the corrective action  report.
In 1998, DOE’s quality assurance auditors identified significant problems  with data sources, validation of scientific models, and software  development and issued three corrective action reports.  For data sources,  DOE reported that it could not ensure that all data needed to support the  scientific models could be tracked back to original sources or that the data  had been properly collected.  For validation of models, DOE reported that  it had no standardized process to develop the scientific models needed to  simulate geological events.  For software, DOE reported that it had no  process for ensuring that the software being developed to support the  models would work.  As required by DOE’s quality assurance procedures,  the department conducted a root-cause analysis and issued a corrective  action plan in 1999 that identified the needed corrective actions.  Following  implementation of the actions, DOE considered the issues resolved and  closed the corrective action reports.  However, problems with models and  software resurfaced during 2001 quality assurance audits.  As a result, new  corrective action reports were completed in May and June 2001, beginning  another iteration of the corrective action process.
Recognizing the need to correct these recurring problems, DOE (1)  conducted a comprehensive root-cause analysis that included reviews of  numerous past self-assessments and independent program assessments  and (2) identified weaknesses in management systems, quality processes,  and organizational roles and responsibilities.  As a result, DOE issued a  corrective action plan in July 2002 that addressed both the quality problems  with data and models and the management weaknesses.  In addition to the  37 actions in the 2002 plan that addressed models and software, DOE  added 35 corrective actions to address management weaknesses that it  found in five key areas:   corrective action plans, and  a work environment that allows employees to raise quality concerns  without fear of reprisal.
To correct these weaknesses, DOE completed a management  reorganization and issued new policy statements to clarify roles and  responsibilities, revised the primary quality assurance implementing  document, reviewed and revised program procedures, revised the system  to correct quality problems, and provided new training for employees to  encourage them to raise concerns about quality.  To assess the  effectiveness of its actions in correcting the management weaknesses,  DOE developed 13 goals to determine whether the corrective actions were  successful, such as achieving decreasing trends in problems attributed to  unclear roles and responsibilities, reducing the time required to revise  procedures and complete corrective actions, and reducing the number of  employee concerns related to the work environment.  Because of the  significance of these problems, DOE stated in the 2002 corrective action  plan that an effectiveness review would be completed prior to closing the  corrective action reports and reporting the results to NRC.
In May 2003, at a congressional field hearing, we provided preliminary  observations on the Yucca Mountain quality assurance program.   Specifically, we noted DOE’s poor track record in correcting recurring  quality assurance problems and provided preliminary observations on  recent actions taken to correct these problems.  You requested that we  continue our evaluation of the quality assurance program at Yucca  Mountain, focusing on DOE’s actions to correct the recurring quality  problems.  As agreed with your offices, this report (1) assesses the status of  DOE’s corrective actions to resolve recurring problems and (2) determines  the adequacy of DOE’s plan to measure the effectiveness of these actions.
In conducting our work, we met with DOE and contractor officials,  assessed the status of DOE’s corrective actions, reviewed audits and  deficiency reports, and visited the Yucca Mountain project office.  We met  with NRC officials and reviewed NRC-prepared documents, including  observation audits, on-site representative reports, and correspondence  between DOE and NRC.  We attended several DOE-NRC quarterly quality  assurance meetings and met with representatives of the State of Nevada  Agency for Nuclear Projects and with representatives of the Nuclear Waste  Technical Review Board.  Our work was performed from April 2003 to April  2004 in accordance with generally accepted government auditing  standards.  Our scope and methodology for this review are presented at the  end of this letter.
subsections: section_title: Background
paragraphs: In 2002, after more than 15 years of scientific investigation, Congress  approved the Yucca Mountain site in Nevada as a suitable location for the  development of a long-term permanent repository for high-level nuclear  waste.  DOE is responsible for developing and operating the repository, and  NRC is responsible for licensing the repository.  DOE is currently preparing  an application to submit to NRC by December 2004 for a license to  construct the repository.  To obtain a license, DOE must, among other  things, demonstrate to NRC that the repository will not exceed  Environmental Protection Agency health and safety standards over a  10,000-year period.  An ineffective quality assurance program runs the risk  of introducing unknown errors into the design and construction of the  repository that could lead to adverse health and safety consequences.
To demonstrate compliance with the health standards over this 10,000-year  period, DOE must rely primarily on a “performance assessment” computer  model that incorporates over 1,000 data sources, approximately 60  scientific models, and more than 400 computer software codes to simulate  the performance of the repository.  Given the prominence of computer  modeling in the licensing of the repository, one of DOE’s most important  tasks is to demonstrate the adequacy of the data, models, and software  used to perform the simulation.  In addition, as part of the licensing  process, DOE must demonstrate that its quality assurance program can  effectively identify and correct deficiencies in areas important to the safe  operation and long-term performance of the repository, such as the natural  and engineered barriers of the repository and the program’s data, models,  and software.  See appendix I for more information on the role of quality  assurance in the licensing process.
DOE has a long-standing history of attempting to correct quality assurance  problems.  In 1988, we identified significant problems with the quality  assurance program, noting that NRC had identified many specific concerns  about the Yucca Mountain program, including    DOE’s heavy reliance on contractors and inadequate oversight would  increase the likelihood that DOE might encounter quality-related  problems;  the possibility that Nevada would contest the licensing proceedings,  thereby increasing the probability that DOE would have to defend its  quality assurance program;  additional expense and time-consuming delays to correct program  weaknesses if DOE could not properly defend the quality of its work;  and  DOE staff’s and contractors’ negative attitude toward quality assurance.
Since the late 1990s, DOE has attempted to correct continuing quality  assurance problems in three areas critical to the repository’s successful  performance: the adequacy of the data sources, the validity of scientific  models, and the reliability of computer software that have been developed  at the site.  These problems surfaced in 1998 when DOE began to run the  initial versions of its performance assessment model.  Specifically, DOE  was unable to ensure that critical project data had been properly collected  and tracked back to original sources.  In addition, the department lacked a  standardized process for developing scientific models used to simulate a  variety of geologic events and an effective process for ensuring that  computer software used to support the scientific models will work  properly.  DOE implemented actions in 1999 to correct these deficiencies  and prevent their recurrence.
In 2001, similar deficiencies associated with models and software  resurfaced.  DOE attributed the recurrence to ineffective procedures and  corrective actions, improper implementation of quality procedures by line  managers, and personnel who feared reprisal for expressing quality  concerns.  To ensure that it adequately addressed the problems to prevent  future recurrence, DOE developed a more comprehensive corrective action  plan in July 2002, concentrating on actions needed to address the causes of  the recurring problems while improving the organizational culture and  instilling a strong commitment to quality in all project personnel.  The plan  detailed specific actions for both DOE and its contractor, Bechtel/SAIC  Company, LLC (Bechtel), to strengthen the roles, responsibilities,  accountability, and authority of project personnel; develop clearer quality  assurance requirements and processes; improve program procedures;  create an improved programwide corrective action process; and improve  processes for ensuring that employees can raise project concerns without  fear of reprisals.
section_title: Quality Assurance Problems Persist at the Yucca Mountain Project
paragraphs: DOE reports that it has implemented almost all of the actions identified in  its 2002 corrective action plan; however, recent audits and assessments  indicate that recurring quality assurance problems have not been  corrected.  In 2003, DOE conducted three audits to evaluate the  effectiveness of the corrective actions taken to address recurring problems  with data, models, and software.  Because each audit identified additional  quality assurance problems, DOE concluded that there was insufficient  evidence to demonstrate that the recurring problems had been corrected.   DOE recently closed the corrective action reports for data and software,  but did so without determining whether corrective actions have been  effective.  To examine actions taken to correct some of the management  weaknesses identified in the 2002 corrective action plan, DOE conducted  four management assessments late in 2003.  Collectively, these assessments  found continuing management weaknesses that DOE had identified as root  causes of the recurring problems.  NRC also conducted an assessment that  was issued in April 2004.  NRC’s assessment noted some improvements but  also found continuing weaknesses and noted that quality assurance  problems could hinder the licensing process.
subsections: section_title: Audits Have Found Recurring Problems with Data, Models, and Software
paragraphs: In 2003, DOE’s audits of data, models, and software identified continuing  quality problems that could impede DOE’s license application.  As a result,  DOE could not close corrective action reports for models and software for  nearly 3 years.  In a June 2003 audit, DOE found quality problems in  developing and validating software.  In September 2003, DOE quality  assurance auditors found that some data sets were still not qualified or  traceable to their sources.  In October 2003, a DOE audit found continuing  quality problems in model documentation and validation.  DOE officials  have stated that these findings represent problems with procedures and  documentation and do not invalidate the technical products produced  using the data, models, and software.  In March 2004, DOE closed the  corrective action reports for data and software but did so without  evaluating the effectiveness of corrective actions—according to agency  officials, they will evaluate effectiveness at a later date.  DOE anticipates  closing the corrective action report for models in August 2004 but also  plans to do so without evaluating the effectiveness of corrective actions.
subsections: section_title: Data Qualification and Traceability Problems Are Still Being Corrected
paragraphs: In April 2003, DOE again reported significant problems similar to those  originally identified in 1998 with the qualification and traceability of data  sets.  At the time, DOE implemented corrective actions to recheck all of its  data sets to confirm that they were traceable and qualified.  However, a  September 2003 audit identified similar data problems and new problems in  addition to those noted in the corrective action report.  The audit found  that some data sets did not have the documentation needed to trace them  back to their sources; the critical process of data control and management  was not satisfactory; and, as in 1998, faulty definitions were developed for  data procedures, which allowed unqualified data to be used.  In addition,  DOE found that overall compliance with procedures was unsatisfactory.   Similarly, the April 2003 corrective action report also noted a lack of  management leadership, accountability, and procedural compliance, issues  which are closely related to the key improvement area of roles and  responsibilities.  DOE officials noted that these findings represented  noncompliance with procedures, and that the procedures and processes  were effective in producing defensible technical products if properly  followed.  As of February 2004, DOE had not finished rechecking all of its  data sets or correcting problems in its data sets.  However, DOE closed the  corrective action report in March 2004 by making the rechecking process a  continuing part of the Yucca Mountain repository’s work.  The corrective  action report was closed without DOE evaluating the effectiveness of the  rechecking process in correcting problems with data.  DOE officials stated  that they plan to evaluate effectiveness at a later date.
section_title: Models Still Lack Proper Validation
paragraphs: An October 2003 DOE quality assurance audit found continuing problems  with the documentation and validation of models that DOE plans to use in  its license application.  Although auditors reported that processes were  effective in producing defensible models to support the license application,  they found that for some models sampled, project personnel did not  properly follow model validation procedures.  These problems were similar  to those identified by audits conducted in 2001.  Auditors compared results  from the 2003 audit with actions taken to correct problems identified in  2001 and found that procedures still were not being satisfactorily  implemented in the areas of model documentation and traceability, model  validation, and checking and review.  For example, an action was taken in  2001 to improve the self-identification of problems before issuing new  model reports by allowing for sufficient scheduling time for model  checking and review.  However, the 2003 audit concluded that instances of  new errors in model reports were evidence that the previous actions may  not have been fully implemented.  As a result, DOE has been unable to  close the May 2001 model corrective action report for almost 3 years.  DOE  recently directed a team of industry experts to review its models and revise  them to ensure consistency, traceability, and procedural compliance.  DOE  anticipates closing the corrective action report in August 2004 but will do  so without conducting another audit of models to determine if corrective  actions have been effective.
section_title: Software Development Problems Persist
paragraphs: In a June 2003 audit, DOE auditors discovered recurring software problems  that could affect confidence in the adequacy of software codes.   Specifically, the auditors found ineffective software processes in five areas:  technical reviews, software classification, planning, design, and testing.   The auditors found several of the software development problems to be  similar to previously identified problems, indicating that previous actions  were ineffective in correcting the problems.  For example, auditors again  noted instances of noncompliance with software procedures.  They also  concluded that technical reviews during software development were  inadequate, even though documentation indicated that corrective actions  for this condition had been completed 3 months before the 2003 audit.   Auditors also noted poorly defined roles and responsibilities as a cause of  problems identified in the technical review of software, even though DOE  had taken actions under its 2002 corrective action plan to clarify roles and  responsibilities.  Because of these results, DOE was unable to close the  June 2001 software corrective action report.  DOE employed a team of  industry professionals in the fall of 2003 to examine software quality  problems identified from 1998 through 2003.  The professionals’ February  2004 report concluded that software problems recurred because DOE did  not assess the effectiveness of its corrective actions and did not adequately  identify the root causes of the problems.  In a January 2004 follow-up audit  of software, auditors verified that unqualified software was used to run  approved models, and noted that procedural controls for determining the  adequacy of software were inadequate.  In March 2004, without evaluating  the effectiveness of corrective actions, DOE closed the software corrective  action report.  DOE officials plan to evaluate the effectiveness of its  corrective actions for software sometime in the future.
section_title: Assessments Indicate Continuing Management Weaknesses
paragraphs: DOE reported in the fall of 2003 that it had implemented most of the  actions identified in the plan focusing on management weaknesses, but  four DOE management assessments of the Yucca Mountain project  completed between September and November 2003 found that some of the  identified management weaknesses had yet to be properly addressed.   These assessments included one requested by project management  comparing DOE’s management practices at Yucca Mountain with external  industry best practices, one required as an annual assessment of the  adequacy and effectiveness of the quality assurance program, one  requested by the project director that examined the effectiveness of  selected DOE and contractor management systems, and one examining  the project work environment.  Collectively, these assessments identified  continuing weaknesses in the areas of roles and responsibilities, quality  assurance procedures, and a work environment that did not foster  employee confidence in raising concerns without fear of reprisal.  DOE  officials stated that they are presently reviewing the findings of these  assessments, and have recently initiated additional corrective actions.
subsections: section_title: Unclear Roles and Responsibilities
paragraphs: Three of the four management assessments conducted late in 2003  identified significant continuing problems with the delineation and  definition of roles and responsibilities for carrying out the quality  assurance program.  In its 2002 corrective action plan, DOE stated that it  was not possible to build accountability into management without clearly  and formally defining roles and responsibilities for DOE and its  contractors.  DOE’s planned actions included clarification of roles and  responsibilities within DOE and Bechtel through policy statements,  communications, a new program manual, and realignment of the  organization to support performance accountability.  DOE reported that it  had completed all corrective actions in this area by May 2003.  The  assessments noted that these actions had resulted in some improvements,  but that some management weaknesses remained.  The assessments found  that the Yucca Mountain project  lacked formal mechanisms for defining and communicating roles and  responsibilities that meet both DOE and NRC requirements;  did not have a systematic process for assigning authorities to DOE and  Bechtel organizations and individuals;  relied on program managers who had not fully assumed ownership and  responsibility for quality assurance;  lacked formal control of documents outlining roles and responsibilities,  ensuring that they reflect the organization;  lacked clear reporting relationships between the project and supporting   had not adequately established processes for reviewing procedures   had few systematic and effective approaches in place for assigning  accountability to individuals and organizations; and   did not effectively plan and communicate reorganizations and assign  appropriate authority levels, in the opinion of many project employees.
As a result of findings from these assessments, DOE is pursuing further  corrective actions.  For example, DOE plans to formally control the high- level document that defines its organizational structure.  Also, Bechtel has  initiated a management system improvement project, which includes  issuing a new document defining roles and responsibilities.  DOE officials  expect that roles and responsibilities will continue to be a challenge in the  future, but that efforts will continue.
section_title: Ineffective Procedures
paragraphs: Three of the four management assessments identified continuing problems  with project procedures, one of the areas of management weaknesses  addressed by the 2002 corrective action plan.  Although the assessments  noted that DOE and Bechtel had made improvements in the procedure  management system and DOE had reportedly reviewed existing  procedures, issued new or revised procedures, and ensured that personnel  using the procedures were properly trained, the assessments noted that  procedures were overly prescriptive,  procedures did not cover all required processes, and   continuing noncompliance with procedures remained a problem.
Although DOE completed actions under the 2002 plan to revise project  procedures, DOE has initiated further corrective actions, including a plan  to again revise Yucca Mountain project procedures by June 2005.
section_title: Inadequate Work Environment
paragraphs: Three of the four assessments identified continuing problems with efforts  by DOE and Bechtel to ensure a work environment in which employees can  freely raise concerns without fear of reprisal—one of the key areas of  management weaknesses identified in the corrective action plan.  DOE and  Bechtel implemented corrective actions to improve the work environment  by revising and expanding policies, modifying DOE contracts to require  implementation of program requirements, decreasing the backlog of  employee concerns, and providing programwide training that is based on  industry practices.  However, the assessments revealed continuing  problems with the work environment, including both DOE’s and Bechtel’s  employee concerns programs, which provide personnel with an  opportunity to formally raise concerns about the project outside the  normal chain-of-command without fear of reprisal.  Appendix II describes  the requirements of the Yucca Mountain employee concerns programs.   Although the assessments noted ongoing management actions to  strengthen the implementation of the concerns programs, they also noted  that  neither DOE nor Bechtel have effectively controlled corrective actions  under the employee concerns programs, sometimes closing cases on the  basis of anticipated actions;  both DOE and contractor employee concerns programs are not being  utilized to their fullest;  there is a general lack of employee confidence in reporting safety issues   DOE and Bechtel have not made effective resources available for  determining the root causes of problems identified;   DOE and Bechtel have not established a climate of trust despite  communication mechanisms and messages; and  a majority of DOE and contractor employees either do not consider the  project’s corrective action process to be effective or are not sure of its  effectiveness.
Although the plan’s actions to improve the work environment were  completed in November 2003, DOE plans to take additional actions to  improve employee confidence in raising issues without fear of reprisal.
section_title: NRC Is Concerned That Recurring Problems Could Adversely Affect Licensing
paragraphs: NRC has commented on DOE’s lack of progress in making improvements to  the quality assurance program.  At an April 2003 management meeting with  DOE, an NRC official commented that the quality assurance program had  not produced the outcomes necessary to ensure that the program is  compliant with NRC requirements.  In response, DOE outlined the steps it  was taking to ensure that its license application would meet NRC  expectations for completeness, accuracy, and compliance with quality  assurance requirements.  The steps included additional actions to improve  performance in five areas:  license application, procedural compliance, the  corrective action program, the work environment, and accountability.  In  October 2003, DOE reported to NRC that it had completed some of the  actions and was making progress in the remaining open action items.   While NRC officials noted that DOE’s actions might enhance performance,  they found that significant implementation issues persist.  NRC officials  stated that they were seeking evidence of incremental DOE progress in the  implementation of the quality assurance program in order to gain  confidence in the adequacy of data, models, and software supporting the  potential license application.  In a November 2003 management meeting  with DOE, NRC officials expressed encouragement with DOE’s progress in  implementing an improved corrective action process and the continued  performance of effective audits and the identification of areas for  improvement.  However, the NRC staff continued to express concerns with  DOE’s lack of progress in correcting repetitive quality problems with  models and software. “…if DOE continues to use its existing policies, procedures, methods, and practices at the  same level of implementation and rigor, the license application may not contain information  sufficient to support some technical positions in the application.  This could result in a large  volume of requests for additional information in some areas which could extend the review  process, and could prevent NRC from making a decision regarding issuing a construction  authorization to DOE within the time required by law.”
section_title: Corrective Action Plan Lacks Measurable Goals
paragraphs: DOE cannot formally assess the overall effectiveness of its 2002 corrective  action plan because the performance goals to assess management  weaknesses in the plan lack objective measurements and time frames for  determining success.  For example, the goals do not specify the amount of  improvement expected, how quickly the improvement should be achieved,  or how long the improvement should be sustained before the problems can  be considered corrected.  For example, whereas 1 goal calls for a  decreasing trend in the average time needed to make revisions in  procedures, it does not specify the desired amount of the decrease, the  length of time needed to achieve the decrease, or how long the decrease  must be sustained.  DOE recently developed a management tool to measure  overall project performance that includes more than 200 performance  indicators with supporting goals, including 17 goals linked to the 13 goals  included in the 2002 corrective action plan.  These 17 goals specify the  desired amount of improvement, but most still lack the time frames needed  for achieving and sustaining the goals.  DOE officials told us they intend to  use this performance measurement tool to track the progress of the  project, including actions taken under the 2002 corrective action plan.  A  DOE independent review of the corrective action plan completed in March  2004 found that the corrective actions from the 2002 plan to address  management weaknesses have been fully implemented.  However, the  review also noted the effectiveness of corrective actions under the plan  could not be evaluated because many of the goals in the performance  measurement tool that are linked to the 2002 plan lacked the level of  objectivity and testing needed to measure effectiveness.
subsections: section_title: Goals Are Not Objectively Measurable and Lack Specific Time Frames
paragraphs: DOE’s 2002 plan included 13 goals to be used to determine the  effectiveness of the corrective actions that addressed the five areas of  management weaknesses.  However, these goals were poorly defined, thus  limiting DOE’s ability to evaluate the effectiveness of actions taken.  Both  GAO and the Office of Management and Budget (OMB) have stated that  performance goals need to be measurable, and time frames need to be  established in order to track progress and demonstrate that deficiencies  have been corrected.  Of the 13 goals in the corrective action plan, 3  indicated how much improvement was expected.  For example, 1 of the 3  goals specified that the number of significant quality problems self- identified by program managers should be at least 80 percent of all  significant quality problems, including those identified by program  managers, quality assurance auditors, or other employees.  In contrast, 1 of  the other 10 goals called for the achievement of a decreasing trend in the  time needed for revising procedures, but did not specify how much of a  decrease is expected.  Further, none of the 13 goals specified the length of  time needed to reach and maintain the desired goal to demonstrate that the  actions taken were effective.  For example, the goal calling for self- identified significant quality problems to be at least 80 percent of all  significant quality problems did not indicate the length of time needed to  achieve the goal or how long this goal should be sustained in order to  demonstrate effectiveness.  DOE does not intend to revise the goals of the  2002 corrective action plan to include quantifiable measures and time  frames.  Without such quantifiable measures to determine whether a goal  has been met, and without a specified time for the goal to be maintained,  DOE cannot use these goals to determine the effectiveness of the actions  taken.
section_title: Subsequent Efforts to Improve Goals Still Lack Time Frames
paragraphs: DOE’s recent efforts to improve performance measurement have not  allowed it to adequately measure the effectiveness of its corrective action  plan.  DOE has developed a projectwide performance measurement tool to  assess project performance that includes over 200 performance indicators  with supporting goals related to the project.  At our request, Bechtel was  able to link 17 of the supporting goals to 12 of the 13 goals of the 2002  corrective action plan.  Although these linked goals improved quantifiable  measurement for 11 of the plan’s goals by specifying the amount of  improvement expected, most did not include the necessary time frames for  meeting the goals and sustaining the desired performance.  DOE officials  stated that this tool was not specifically tailored to evaluate the corrective  action plan’s effectiveness, but that they have decided to use it in lieu of the  original 13 goals to monitor improvements and progress in correcting the  management weaknesses identified in the plan.  Table 1 provides a  comparison of the supporting goals in the performance tool with the 2002  corrective action plan goals.
section_title: DOE Is Unable to Evaluate the Effectiveness of Corrective Actions
paragraphs: DOE has recently assessed the implementation of corrective actions, but it  has not yet assessed the effectiveness of these actions in correcting  recurring problems.  In December 2003, DOE outlined the approach it used  to determine whether corrective actions have been implemented.  This  approach is part of the overall process described in the 2002 action plan— appendix III provides an overview of the action plan and the status of the  process.  To determine if corrective actions had been implemented, DOE  relied on the collective judgment of project managers regarding how  effectively they have incorporated corrective actions into their regular  project activities.  A March 2004 DOE review analyzed the implementation  of corrective actions for each of the management weaknesses but was not  able to evaluate the effectiveness of the corrective actions.
DOE’s March 2004 review noted strong management commitment to  improvement and described recent actions taken to ensure that work  products meet quality objectives for a successful license application.   However, the review identified continuing weaknesses in DOE’s ability to  determine the effectiveness of the actions it has taken.  The review team  attempted to measure how effectively DOE had met each of the plan’s  original 13 goals.  The team was unable to measure whether 10 of the 13  goals had been met, but concluded that the project had met 2 of the goals  and made progress toward another goal, based on an analysis of trends in  quality problems identified.  However, these conclusions were not based on  an evaluation of quantifiable goals with time frames for meeting and  sustaining the desired performance.  The review also concluded that the  performance indicators developed to evaluate the success of the actions  lacked the level of objectivity and testing needed to measure effectiveness  and that some lacked the data needed to assess effectiveness.  The review  recommended that DOE continue its corrective actions and refine  performance indicators so that the effectiveness of corrective actions in  meeting the plan’s goals can be more readily measured.
In April 2004, DOE notified NRC that it had completed, validated, and  independently assessed the commitments it made in the 2002 corrective  action plan, institutionalized the corrective actions, and established a  baseline to foster and sustain continuous improvement.  DOE officials  stated they have achieved the initial goals of the 2002 plan through these  actions.  These officials indicated they would continue to refine and  improve project tools used to evaluate the effectiveness of corrective  actions.  However, because of the limitations noted in its March 2004  review, DOE has not yet evaluated the effectiveness of corrective actions.
section_title: Conclusions
paragraphs: Despite working nearly 3 years to address recurring quality assurance  problems, recent audits and assessments have found that problems  continue with data, models, and software, and that management  weaknesses remain.  As NRC has noted, quality assurance problems could  delay the licensing process.  Despite recurring quality problems, DOE has  recently closed the corrective action reports for data and software and  intends to close the corrective action report for models in August 2004  without first evaluating the effectiveness of the corrective actions taken to  address the problems in these areas.  DOE also does not intend to improve  the goals of the 2002 plan associated with management weaknesses so that  they can be adequately measured.  Instead, DOE continues to plan and  implement further actions to correct its quality problems and management  weaknesses.  This approach provides no indication regarding when DOE  may be in a position to show that corrective actions have been successful.   Entering into the licensing phase of the project without resolving the  recurring problems could impede the application process, which at a  minimum could lead to time-consuming and expensive delays while  weaknesses are corrected and could ultimately prevent DOE from  receiving authorization to construct a repository.  Moreover, recurring  problems could create the risk of introducing unknown errors into the  design and construction of the repository that could lead to adverse health  and safety consequences.  Because of its lack of evidence that its actions  have been successful, DOE is not yet in a position to demonstrate to NRC  that its quality assurance program can ensure the safe construction and  long-term operation of the repository.
section_title: Recommendations for Executive Action
paragraphs: To better evaluate the effectiveness of management actions in correcting  recurring quality problems, we recommend that the Secretary of Energy  direct the Director, Office of Civilian Radioactive Waste Management, to   revise the performance goals in the 2002 action plan to include  quantifiable measures of the performance expected and time frames for  achieving and maintaining this expected level of performance and   close the 2002 plan once sufficient evidence shows that the recurring  quality assurance problems and management weaknesses that are  causing them have been successfully corrected.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOE and NRC for their review and  comments.  DOE’s written comments, which are reproduced in appendix  IV, expressed disagreement with the report’s findings and  recommendations.  DOE commented that the report did not properly  acknowledge improvements the department has made in the quality  assurance program; failed to properly characterize the 2002 Management  Improvement Initiatives as a “springboard” to address management issues;  did not consider DOE’s use of the full range of performance indicators  related to quality assurance; and mischaracterized the results of several  independent, external reviews, taking a solely negative view of the findings.
We disagree with most of DOE’s comments.  Our draft report  acknowledged that DOE has taken a number of actions to address past  problems in the quality assurance program, but to ensure clarity on this  point, we have incorporated additional language to this effect in the report.   However, our primary focus for this review was to evaluate the  effectiveness of DOE’s corrective actions in addressing the recurring  quality problems.  Despite the many actions taken to improve the quality  assurance program, the management weaknesses and quality problems  with data, models, and software have continued, indicating that the  corrective actions have not been fully effective.  Regarding DOE’s views on  our treatment of the 2002 Management Improvement Initiatives, DOE itself  characterized the initiative as a “comprehensive corrective action plan.”    DOE stated that the implementation of the plan has been successful based  on the evidence that responsible managers have taken agreed-upon action.   This approach can be misleading, however, because it does not incorporate  a determination of whether these actions have been effective.  In fact, DOE  has not evaluated the effectiveness of these actions in solving recurring  problems.   DOE further stated that we did not consider the full range of  performance indicators related to quality assurance that DOE uses to  manage the project.  We agree.  We asked DOE staff to compare their new  performance indicators to the goals in the 2002 plan, and those are the  goals that we presented for comparison in table 1 of our report.  A  discussion of the remainder of the hundreds of other goals was beyond the  scope of our review and would not have added to an understanding of the  overall problems with DOE’s goals.  Finally, we disagree with DOE’s  comment that we mischaracterized the results of recent independent  reviews.  We noted instances in these reports where improvements were  found.  However, we also devoted appropriate attention to evidence in  these reports that address whether DOE’s corrective actions have been  effective.  As our report states, these reports consistently found that these  actions have not yet had their intended effect.
In NRC’s written comments, reproduced in appendix V, the agency agreed  with our conclusions but suggested that DOE be given the flexibility to  choose alternative approaches to achieve and measure quality assurance  program performance.  We agree that alternative approaches could be used  to measure performance; however, to ensure the success of any  approaches, DOE must include objective measurements and time frames  for reaching and sustaining desired performance and include an end point  for closing out the corrective action plan.
section_title: Scope and Methodology
paragraphs: To assess the status of DOE’s corrective actions to resolve recurring quality  problems, we reviewed audits and deficiency reports written by the  program over the past 5 years that identified problems with data, models,  and software.  We did not independently assess the adequacy of data,  models, and software, but rather relied on the results of the project’s  quality assurance audits.  In addition, we reviewed numerous documents  that NRC prepared as part of its prelicensing activities at Yucca Mountain,  including observations of quality assurance audits, NRC on-site  representative reports, and correspondence between NRC and DOE on  quality matters.  We also observed an out-briefing of a quality assurance  audit to obtain additional knowledge of how quality problems are identified  and reported.  To document the status of actions taken, we reviewed  evidence used by DOE’s Office of Civilian Radioactive Waste Management  to prove corrective actions had been implemented and interviewed officials  with DOE, at the Yucca site and in headquarters, and officials with Bechtel,  the primary contractor.  We also reviewed the results of four DOE  assessments completed in the fall of 2003 that included the quality  assurance program, interviewing the authors of the assessment reports to  obtain a clear understanding of the problems identified.  We attended  quarterly meetings held between DOE and NRC to discuss actions taken  under the plan and met with representatives of the State of Nevada Agency  for Nuclear Projects and with representatives of the Nuclear Waste  Technical Review Board, which was established to advise DOE on  scientific and technical aspects of the Yucca Mountain project.
To determine the adequacy of DOE’s plan to measure the effectiveness of  the actions it has taken, we examined the July 2002 corrective action plan  and subsequent project performance measurement documents to  determine how DOE intended to use goals and performance measures to  evaluate the plan’s effectiveness.  We asked Bechtel officials to assist us in  identifying and matching performance goals in the projectwide  performance measurement tool with those in the 2002 corrective action  plan.  We compared DOE’s approach in its corrective action plan and  subsequent projectwide tool with GAO and OMB guidance on performance  measurement.  We discussed the implementation of the corrective action  plan and methods for measuring its effectiveness with various DOE and  NRC officials and DOE contractors in Washington, D.C., and at the Yucca  Mountain project office in Las Vegas, Nevada.  We also interviewed other  GAO personnel familiar with performance measurement to more fully  understand the key elements needed for effective assessments.
We will send copies of this report to the appropriate congressional  committees, the Secretary of Energy, and the Chairman of the Nuclear  Regulatory Commission. We will also make copies available to others on  request.  In addition, this report will be available at no charge on the GAO  Web site at www.gao.gov.
If you or your staffs have any questions about this report, please call me on  (202) 512-3841.  Major contributors to this report are listed in appendix VI.
section_title: Role of Quality Assurance in the Licensing Process
paragraphs: After the Department of Energy (DOE) submits its license application to  the Nuclear Regulatory Commission (NRC), NRC will review it to  determine whether all NRC requirements have been met and whether the  repository is likely to operate safely as designed.  NRC’s review will be  guided by its Yucca Mountain Review Plan, which NRC developed to  ensure the quality, uniformity, and consistency of NRC reviews of the  license application and of any requested amendments.  The review plan is  not a regulation, but does contain the licensing criteria contained in federal  regulations.  DOE’s application is to include general, scientific, and  administrative information contained in two major sections:  (1) a general  information section that provides an overview of the engineering design  concept for the repository and describes aspects of the Yucca Mountain  site and its environs that influence repository design and performance, and  (2) a detailed safety analysis section that provides a review of compliance  with regulatory performance objectives that are based on permissible  levels of radiation doses to workers and the public, established on the basis  of acceptable levels of risk.  The general information section covers such  topics as proposed schedules for construction, receipt, and emplacement  of waste; the physical protection plan; the material control and accounting  program; and a description of site characterization work.  The detailed  safety analysis is the major portion of the application and includes DOE’s  detailed technical basis for the following areas:  the repository’s safety performance before permanent closure in 100 to   the repository’s safety performance in the 10,000 years after permanent  closure, on the basis of the “performance assessment” computer model;   a research and development program describing safety features or  components for which further technical information is required to  confirm the adequacy of design and engineered or natural barriers;   a performance confirmation program that includes tests, experiments,  and analyses that evaluate the adequacy of information used to  demonstrate the repository’s safety over thousands of years; and   administrative and programmatic information about the repository, such  as the quality assurance program, records and reports, training and  certification of personnel, plans for start-up activities, emergency  planning, and control of access to the site.
After DOE submits the license application (currently planned for  December 2004), NRC plans to take 90 days to examine the application for  completeness to determine whether DOE has addressed all NRC  requirements in the application.  One of the reviews for completeness will  include an examination of DOE’s documentation of the quality assurance  program to determine if it addresses all NRC criteria.  These criteria  include, among other things, organization, design and document control,  corrective actions, quality assurance records, and quality audits.  If it  deems any part of the application incomplete, NRC may either reject the  application or require that DOE furnish the necessary documentation  before proceeding with the detailed technical review of the application.  If  it deems the application complete, NRC will docket the application,  indicating its readiness for a detailed technical review.
Once the application is docketed, NRC will conduct a detailed technical  review of the application over the next 18 months to determine if the  application meets all NRC requirements, including the soundness of  scientific analyses and preliminary facility design, and the NRC criteria  established for quality assurance.  If NRC discovers problems with the  technical information used to support the license application, it may  conduct specific inspections to determine the extent and effect of the  problem.  Because the data, models, and software used in modeling  repository performance are integral parts of this technical review, quality  assurance plays a key role since it is the mechanism used to verify the  accuracy of the information DOE presents in the application.  NRC may  conduct inspections of the quality assurance program if technical problems  are identified that are attributable to quality problems.  According to NRC,  any technical problems and subsequent inspections could delay the  licensing of the repository or, in a rare instance, lead to ultimate rejection  of the application.  NRC will hold public hearings chaired by its Atomic  Safety and Licensing Board to examine specific topics.  Finally, within 3 to 4 years from the date that NRC dockets the application, NRC will make a  decision to grant the application, reject the application, or grant it with  conditions.  Figure 1 presents the licensing process and timeline.
section_title: Employee Concerns Programs at the Yucca Mountain Project
paragraphs: DOE and Bechtel/SAIC Company, LLC (Bechtel), have each established an  employee concerns program to allow employees to raise concerns about  the work environment without fear of reprisal.  NRC requires licensees to  establish a safe work environment where (1) employees are encouraged to  raise concerns either to their own management or to NRC without fear of  retaliation and (2) employees’ concerns are resolved in a timely and  appropriate manner according to their importance.  DOE and contractor  employees at Yucca Mountain have various means through which to raise  concerns about safety, quality, or the work environment, including  a corrective action program—a process in which any employee can  formally cite a problem on the project, including the work environment,  that needs to be investigated and corrective actions taken;  a DOE or contractor employee concerns program; or   filing a concern directly with NRC.
NRC encourages, but does not require, licensees to establish employee  concerns programs.  Both the DOE and Bechtel concerns programs at  Yucca Mountain have three main steps:   1. An employee notifies concerns program staff about an issue that he/she  feels should be corrected, such as safety and health issues, free from  harassment, retaliation, or quality assurance problems.   2. The concerns program staff documents and investigates the employee’s  concern.   3. The concerns program notifies the employee of the results of the  investigation and notifies management of any need for corrective  actions.
DOE and Bechtel each have established a communication network to allow  employees to register concerns.  These networks include brochures and  regular newsletters on the program and numerous computer links to the  program on the contractor’s intranet where employees can obtain concerns  program forms on line.
Recent statistics released by DOE show that most of the 97 concerns  investigated by the DOE and Bechtel concerns programs in 2003 related to  complaints against management.  A summary of the concerns investigated  in 2003 is shown in table 2.
section_title: 2002 Corrective Action Plan Process and Status
paragraphs: DOE has established a process for completing corrective actions  associated with the 2002 corrective action plan and evaluating their  effectiveness.  According to this process, after managers report they have  taken actions to correct management weaknesses and specific problems  with models and software, a confirmation team of DOE and contractor  personnel verify that the actions have been completed.  Once this step is  completed, DOE conducts internal and external effectiveness reviews to  determine if the actions have been effective in correcting the reported  conditions.  After the reviews of effectiveness, the results are assessed and  reported to the Director of the Office of Civilian Radioactive Waste  Management (OCRWM).  The director then notifies NRC officials of the  results of the effectiveness reviews, and the 2002 corrective action plan is  closed.  Figure 2 shows the corrective action plan process and the status of  each step.
section_title: Comments from the Department of Energy
paragraphs: The following are GAO’s comments on the Department of Energy’s letter  dated April 19, 2004.
subsections: section_title: GAO Comments
paragraphs: 1. We disagree.  Our report states that the recent independent  assessments have shown improvements in the key management areas  identified in the 2002 corrective action plan.  However, the assessments  also showed that problems remain in these areas and thus the  corrective actions have not yet been successful in correcting these  weaknesses.  DOE’s examples of progress illustrate our point regarding  improperly specified goals.  For example, DOE states in its comments  that line management’s self-identification of conditions adverse to  quality has increased approximately 100 percent in the last 15 months  (as opposed to the identification of such conditions by quality  assurance auditors).  However, despite this seemingly dramatic  increase, DOE has yet to meet its goal of line management’s self- identifying 80 percent of all quality problems.  (DOE’s 100 percent  increase brought them up to about 50 percent of all quality problems  being self-identified by line managers.)  Further, the goal continues to  lack a time frame for when the 80 percent goal should be attained and  for how long it should be sustained before the corrective actions can be  judged successful.  As our report points out, without such specificity,  improvements cannot be evaluated in terms of overall success.   2. We disagree.  The 2002 Management Improvement Initiatives clearly  state that it was a “comprehensive corrective action plan necessary to  address weaknesses in the implementation of  quality  assurance requirements and attain a level of performance expected of  an NRC license applicant.” Contrary to DOE’s assertion, the initiative  does not indicate it was a “springboard effort to address management  issues and transition improvements into day-to-day line management  activities.” Although the transitioning of improvements to the line is  laudable, the initiative focused on implementing corrective actions and  evaluating the effectiveness of the actions in correcting problems.  This  approach is consistent with DOE’s criteria for correcting significant  conditions adverse to quality, and it is the criteria we relied on to  determine whether the corrective actions specified in the initiatives  were successful.   3. We agree.  We did not include the full range of performance indicators  (goals) that have recently been developed, and continue to change, to  assess the 2002 plan.  Instead, of the hundreds of indicators that are  being developed to manage the project, we relied on those few that  Bechtel officials told us were connected to the goals of the 2002 plan.   As table 1 shows, some improvements have been made in specifying the  quantitative aspects of the goals, but weaknesses continue to exist in  the new goals.  In fact, table 1 shows that DOE no longer has a goal in  its performance tool that specifically tracks the trend in problems  related to roles and responsibilities.  This omission is particularly  important because the area of roles and responsibilities was noted in  the 2002 plan as one of the biggest sources of problems in the quality  assurance process, and, as the recent assessments have found, this is  an area with continuing problems. 4. We disagree.  We acknowledge that these reviews found positive  improvements in a number of management areas.  However, we also  note that continuing problems were found with management  weaknesses despite all corrective actions having been implemented in  2003.   5. While DOE believes that it has achieved the objectives of the 2002 plan,  it lacks evidence that its actions have been effective in addressing the  management weaknesses and correcting the recurring quality problems  with data, models, and software.  Evaluating performance against  measurable goals with time frames for meeting and sustaining the goals  would provide the needed evidence.   6. The draft report that we sent to DOE for review included reviews of 9  of the 12 documents listed in the enclosure of DOE’s letter.  We have  since reviewed the 3 remaining documents.  The information in the 3  documents did not change our assessment of DOE’s efforts to correct  its quality assurance program.
After full consideration of the information included in DOE’s comments,  we believe that our findings are complete and our conclusions are  accurate.
section_title: Comments from the Nuclear Regulatory Commission
paragraphs: The following is GAO’s comment on the U.S. Nuclear Regulatory  Commission’s letter dated April 16, 2004.
subsections: section_title: GAO Comment
paragraphs: 1. We agree that alternative approaches could be used to measure  performance; however, to ensure the success of any approaches, DOE  must include objective measurements and time frames for reaching and  sustaining desired performance and include an end point for closing  out the corrective action plan.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Robert Baney, Lee Carroll,  Thomas Kingham, Chalane Lechuga, Jonathan McMurray, Judy Pagano,  Katherine Raheb, Anne Rhodes-Kline, and Barbara Timmerman made key  contributions to this report.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to   e-mail alerts” under the “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs