Yucca Mountain: DOE's Planned Nuclear Waste Repository Faces Quality Assurance and Management Challenges
Released: 2006-04-25T00:00:00

Letter

Background

The Congress enacted the Nuclear Waste Policy Act of 1982 to establish a comprehensive policy and program for the safe, permanent disposal of commercial spent nuclear fuel and other highly radioactive wastes in one or more mined geologic repositories. The act charged DOE with (1) establishing criteria for recommending sites for repositories; (2) “characterizing” (investigating) three sites to determine each site’s suitability for a repository (1987 amendments to the act directed DOE to investigate only the Yucca Mountain site); (3) recommending one suitable site to the President, who would submit a recommendation of such site to the Congress if he considered the site qualified for a license application; and (4) upon approval of a recommended site, seeking a license from NRC to construct and operate a repository at the site. The Yucca Mountain project is currently focused on preparing an application for a license from NRC to construct a repository. DOE is compiling information and writing sections of the license application, conducting technical exchanges with NRC staff, and addressing key technical issues identified by NRC to ensure that sufficient supporting information is provided.

In February 2005, DOE announced that it does not expect the repository to open until 2012 at the earliest, which is more than 14 years later than the 1998 goal specified by the Nuclear Waste Policy Act of 1982. More recently, the conference report for DOE’s fiscal year 2006 appropriations observed that additional significant delays to submitting a license application are likely. In October 2005, the project’s Acting Director issued a memorandum calling for the development of wide-ranging plans for the “new path forward” to submitting the license application. The plans address the need to review and replace USGS work products, establish a lead national laboratory to assist the project, and develop a new simplified design for the waste canisters and repository facilities, among other things. In addition, DOE announced, in April 2006, that it was proposing legislation intended to accelerate licensing and operations. For example, the legislation provides that if NRC authorizes the repository, subsequent licensing actions would be conducted using expedited, simplified procedures.

Given the delays, the Congress has considered other options for managing existing and future nuclear wastes, such as centralized interim storage at one or more DOE sites. In addition, the conference report for DOE’s fiscal year 2006 appropriations directed DOE to develop a spent nuclear fuel recycling plan to reuse the spent fuel. However, according to the Nuclear Energy Institute, which represents the nuclear energy industry, none of technological options being considered will eliminate the need to ultimately dispose of nuclear waste in a geologic repository.

DOE Has a Long History of Quality Assurance Problems at Yucca Mountain

DOE has had a long history of quality assurance problems at the Yucca Mountain project. In the project’s early stages, DOE had problems assuring NRC that it had developed adequate quality assurance plans and procedures. By the late 1990s, DOE had largely addressed NRC’s concerns about its plans and procedures, but its own audits identified quality assurance problems with the data, software, and models used in the scientific work supporting its potential license application. While recently resolving these quality problems, DOE is now relying on costly and time- consuming rework to ensure the traceability and transparency of several technical work products that are key components of the license application.

As we reported in 1988, NRC reviewed DOE’s quality assurance program for the Yucca Mountain project and concluded that it did not meet NRC requirements and that DOE’s quality assurance audits were ineffective. In 1989, NRC concluded that DOE and its key contractors had yet to develop and implement an acceptable quality assurance program. However, by March 1992, NRC determined that DOE had made significant progress in improving its quality assurance program, noting among other things, that all of the contractor organizations had developed and were in the process of implementing quality assurance programs that met NRC requirements, and that DOE had demonstrated its ability to evaluate and correct deficiencies in the overall quality assurance program.

By the late 1990s, however, the DOE quality assurance program began detecting new quality problems in three areas critical to demonstrating the repository’s successful performance over time: data management, software management, and scientific models.

Data management. In 1998, DOE identified quality assurance problems with the quality and traceability of data, specifically that some data had not been properly collected or tested to ensure its accuracy and that data used to support scientific analysis could not be properly traced back to its source. DOE found similar problems in April and September 2003.

Software management. DOE quality assurance procedures require that software used to support analysis and conclusions about the performance and safety of the repository be tested or created in such a way to ensure that it is reliable. From 1998 to 2003, multiple DOE audits found recurring quality assurance problems that could affect confidence in the adequacy of software.

Model validation. In 1998, a team of project personnel evaluated the mathematical models used to simulate natural and environmental conditions and determined that 87 percent of them did not comply with validation requirements to ensure they accurately predict geologic events. In 2001, and again in 2003, DOE audits found that project personnel were not properly following procedures, specifically in the areas of model documentation, model validation, and checking and review. Further, the 2003 audit concluded that previous corrective actions designed to improve validation and reduce errors in model reports were not fully implemented.

After many years of working to address these quality assurance problems with data, software, and models, DOE had mostly resolved these problems by February 2005.

As DOE prepares to submit the Yucca Mountain project license application to NRC, it is relying on costly and time-consuming rework to ensure that the documents supporting its license application are accurate and complete. Although the department had known for years about quality assurance problems with the traceability and transparency of technical work products called Analysis and Model Reports (AMR)—a key component of the license application—-DOE did not initiate a major effort to address these problems until 2004. AMRs contain the scientific analysis and modeling data that demonstrate the safety and performance of the planned repository and, among other quality requirements, must be traceable to their original source material and data and be transparent in justifying and explaining their underlying assumptions, calculations, and conclusions. In 2003, based in part on these problems, as well as DOE’s long-standing problems with data, software, and modeling, NRC conducted an independent evaluation of three AMRs to determine if they met NRC requirements for being traceable, transparent, and technically appropriate for their use in the license application. In all three AMRs, NRC found significant problems with both traceability and transparency. NRC concluded that these findings suggested that other AMRs may have similar problems and that such problems could delay NRC’s review of the license application, as it would need to conduct special inspections to resolve any problems it found with the quality of technical information.

To address problems of traceability and transparency, DOE initiated an effort in the spring of 2004 called the Regulatory Integration Team (RIT) to perform a comprehensive inspection and rework of the AMRs and ensure they met NRC requirements and expectations. According to DOE officials, the RIT involved roughly 150 full-time personnel from DOE, USGS, and multiple national laboratories such as Sandia, Los Alamos, and Lawrence Livermore. The RIT decided that 89 of the approximately 110 AMRs needed rework. According to DOE officials, the RIT addressed or corrected over 3,700 problems, and was completed approximately 8 months later at a cost of about $20 million. In a February 2005 letter to DOE, the site contractor stated that the RIT effort had successfully improved the AMRs’ traceability and transparency.

Subsequently, however, DOE identified additional problems with traceability and transparency that required further inspections and rework. DOE initiated a review of additional AMRs that were not included in the scope of the 2004 RIT review after a March 2005 discovery of e-mails from USGS employees written between May 1998 and March 2000 implying that employees had falsified documentation of their work to avoid quality assurance standards. These additional AMRs contained scientific work performed by the USGS employees and had been assumed by the RIT to meet NRC requirements for traceability and transparency. However, according to DOE officials, DOE’s review determined that these AMRs did not meet NRC’s standards, and rework was required. DOE identified similar problems as the focus of the project shifted to the design and engineering work required for the license application. In February 2005, the site contractor determined that, in addition to problems with AMRs, similar traceability and transparency problems existed in the design and engineering documents that constitute the Safety Analysis Report—the report necessary to demonstrate to NRC that the repository site will meet the project’s health, safety, and environmental goals and objectives. In an analysis of this problem, the site contractor noted that additional resources were needed to inspect and rework the documents to correct the problems.

DOE Cannot Be Certain Its Efforts to Improve Quality Assurance Have Been Effective Because of Weaknesses in Tracking Progress and Identifying Problems

DOE’s management tools for the Yucca Mountain project have not enabled it to effectively identify and track progress in addressing significant and recurring quality assurance problems. Specifically, its panel or one-page summary of selected performance indicators did not highlight the areas of management concern covered by its Management Improvement Initiatives (Initiatives) and had weaknesses in assessing progress because the indicators kept changing. Its trend reports also did not focus on tracking these management concerns, had technical weaknesses for identifying significant and recurrent problems, and has inconsistently tracked progress with problems. Furthermore, the trend reports have sometimes been misleading as to the significance of the problems being presented because their significance ratings tend to be lower if corrective actions were already being taken, without considering the effectiveness of the actions or the problem’s importance to the project.

In April 2004, DOE told us it expected that the progress achieved with its Initiatives for improving quality assurance would continue and that its performance indicators would enable it to assess further progress and direct management attention as needed. By that time, the actions called for by the Initiatives had been completed and project management had already developed a “panel” of indicators to use at monthly management meetings to monitor project performance. The panel was a single page composed of colored blocks representing selected performance indicators and their rating or level of performance. For example, a red block indicated degraded or adverse performance warranting significant management attention, a yellow block indicated performance warranting increased management attention or acceptable performance that could change for the worse, and a green block indicated good performance. The panel represented a hierarchy of indicators where the highest-level indicators were visible, but many lower-level indicators that determined the ratings of the visible indicators were not shown. Our review analyzed a subset of these indicators that DOE designated as the best predictors in areas affecting quality.

We found that the panel was not effective for assessing continued progress because its indicators poorly represented the management concerns identified by the Initiatives. The Initiatives had raised concerns about five key areas of management weakness as adversely affecting the implementation of quality assurance requirements, and had designated effectiveness indicators for these areas. (These areas of concern are described in app. I.) However, two of the Initiatives’ five key areas of concern—roles and responsibilities as well as work procedures—were not represented in the panel’s visible or underlying indicators. In other cases, the Initiatives’ effectiveness indicators were represented in underlying lower-level indicators that had very little impact on the rating of the visible indicator. For example, the Initiatives’ indicator for timely completion of employee concerns was represented by two lower-level indicators that together contributed 3 percent of the rating for an indicator visible in the panel.

Another shortcoming of the panel was that frequent changes to the indicators hindered the ability to identify problems for management attention and track progress in resolving them. The indicators could change in many ways, such as how they were defined or calculated. Such changes made it difficult to measure progress because changes in indicator ratings could reflect only the changes in the indicators rather than actual performance changes. Some of the indicators tracking quality issues changed from one to five times during the 8-month period from April 2004 through November 2004. Even after a major revision of the panel in early 2005, most of the performance indicators tracking quality issues continued to change over the next 6 months—that is, from March 2005 through August 2005. Only one of these five indicators did not change during this period. One indicator was changed four times during the 6- month period, resulting in it being different in more months than it remained the same. Moreover, the panel was not always available to track problems. It was not created for December 2004 through February 2005, and it has not been created since August 2005. In both cases, the panel was undergoing major revisions. In December 2005, a senior DOE official told us that the project would begin to measure key activities, but without use of the panel.

According to DOE, a second management tool, the project’s quarterly trend evaluation reports, captured some aspects of the Initiatives’ areas of concern and their associated effectiveness indicators that were not represented in the performance indicators. However, the trend reports are designed more to identify emerging and unanticipated problems than to monitor progress with already identified problems, such as those addressed by the Initiatives. In developing these reports, trend analysts seek to identify patterns and trends in condition reports, which document problematic conditions through the project’s Corrective Action Program. For example, analysis might reveal that most occurrences of a particular type of problem are associated with a certain organization.

In practice, DOE missed opportunities to use trend reports to assess progress in the Initiatives’ areas of concern. For example, DOE missed an opportunity to use trend reports to discuss the Initiatives’ goal that the project’s work organizations become more accountable for self-identifying significant problems. The August 2005 trend report briefly cited an evaluation of a condition report highlighting the low rate of self- identification of significant problems during the previous quarter and reported the evaluation’s conclusion that it was not a problem warranting management attention. However, the trend report did not mention that about 35 percent of significant problems were self-identified during the previous quarter, while the Initiatives’ goal was that 80 percent of significant problems would be self-identified.

Beyond whether they effectively track the Initiatives’ areas of concern, trend reports generally face serious obstacles to adequately identifying recurrent and significant problems. For example, trend analysis tends to focus on the number of condition reports issued, but the number of reports does not necessarily reflect the significance of a problem. For example, the number of condition reports involving requirements management decreased by over half from the first quarter to the second quarter of fiscal year 2005. However, this decrease was not a clear sign of progress. Not only did the number rise again in the third quarter, but the May 2005 trend report also noted that the number of all condition reports had dropped during the second quarter. According to the report, the volume of condition reports had been high in the first quarter because of reviews of various areas, including requirements management.

Due, in part, to these obstacles, trend reports have not consistently determined the significance of problems or performed well in tracking progress in resolving them. For example, trend reports have questionably identified human performance as a significant problem for resolution and ineffectively tracked progress in resolving it because there was (1) no clearly appropriate or precise benchmark for performance, (2) a changing focus on the problem, and (3) unreliable data on cause codes. The February 2004 trend report identified a human performance problem based on Yucca Mountain project data showing the project’s proportion of skill-based errors to all human performance errors was two times higher than benchmark data from the Institute of Nuclear Power Operations (INPO). Interestingly, the report cautioned that other comparisons with these INPO data may not be appropriate because of differences in the nature, complexity, and scope of work performed, but did not explain why this caution did not apply to the report’s own comparison. While this comparison has not appeared in trend reports since May 2004, a November 2004 trend report changed the focus of the problem to the predominance of human performance errors in general, rather than the skill-based component of these errors. (Later reports reinterpreted this predominance as not a problem.) The report cited an adverse trend based on the fact that the human performance cause category accounted for over half of the total number of causes for condition reports prepared during the quarter. Nevertheless, by February 2005, trend reports began interpreting this predominance as generally appropriate, given the type of work done by the project. That is, the project’s work involves mainly human efforts and little equipment, while work at nuclear power plants involves more opportunities for errors caused by equipment. In our view, this interpretation that a predominance of human performance errors would be expected implies an imprecise benchmark for appropriate performance.

Further, although trend reports continued to draw conclusions about human performance problems, the February 2005 report indicated that any conclusions were hard to justify because of data reliability problems with cause coding. For example, the majority of problems attributed to human performance causes are minor problems, such as not completing a form, that receive less-rigorous cause analysis. This less-rigorous analysis tends to reveal only individual human errors—that is, human performance problems—whereas more-rigorous analysis tends to reveal less-obvious problems with management and procedures.

Another shortcoming of the trend reports was that their rating categories made it difficult to adequately determine the significance of some problems. Specifically, trend reports sometimes assigned a problem a lower significance than justified because corrective actions were already being taken. The rating categories for a problem’s significance also involve an assessment of the need for management action. In their current formulation, DOE’s rating categories cannot accurately represent both these assessments, and the designated rating category can distort one or the other assessment. For instance, a November 2005 trend report rated certain requirements management issues as a “monitoring trend”—defined as a small perturbation in numbers that does not warrant action but needs to be monitored closely. However, this rating did not accurately capture the report’s simultaneous recognition that significant process problems spanned both BSC and DOE and the fact that the numbers and types of problems were consistently identified over the previous three quarters. A more understandable explanation for why the problem received a low rating is that designating the problem at any higher level of significance would have triggered guidelines involving the issuance of a condition report, which, according to the judgment expressed in the report, was not needed. Specifically, the report indicated that existing condition reports have already identified and were evaluating and resolving the problem, thereby eliminating the need to issue a new condition report.

However, by rating the problem at the lowest level of significance, the trend report did not sufficiently draw management’s attention to the problem. At about the same time the trend report judged no new condition reports were necessary, a separate DOE investigation of requirements management resulted in 14 new condition reports—3 at the highest level of significance and 8 at the second-highest level of significance. These condition reports requested, for instance, an analysis of the collective significance of the numerous existing condition reports and an assessment of whether the quality assurance requirement for complete and prompt remedial action had been met. As a result of the investigation and a concurrent DOE root cause analysis, DOE stated during the December 2005 quarterly management meeting with NRC that strong actions were required to address the problems with its requirements management system and any resulting uncertainty about the adequacy of its design products.

I would now like to update you on the project’s February 2006 stop-work order, which occurred too late to be included in our report. We believe this incident is an example of how the project’s management tools have not been effective in bringing quality assurance problems to top management’s attention. After observing a DOE quality assurance audit at the Lawrence Livermore National Laboratory in August 2005, NRC expressed concern that humidity gauges used in scientific experiments at the project were not properly calibrated—an apparent violation of quality assurance requirements. According to an NRC official, NRC communicated these findings to BSC and DOE project officials on six occasions between August and December 2005, and issued a formal report and letter to DOE on January 9, 2006. However, despite these communications and the potentially serious quality assurance problems involved, the project’s acting director did not become aware of the issue until January 2006, after reading about it in a news article. Due to concerns that quality assurance requirements had not been followed and the length of time it took top management to become aware of the issue, BSC issued a February 7, 2006, stop-work order affecting this scientific work. Project officials have begun a review of the issue.

DOE’s New Path Forward to Submitting a License Application Faces Substantial Quality Assurance and Other Challenges

In pursuing its new path forward, DOE faces significant quality assurance and other challenges, including (1) determining the extent of problems and restoring confidence in the documents supporting the license application after the discovery of e-mails raising the potential of falsified records, (2) settling the design issues and the associated problems with requirements management, and (3) replacing key personnel and managing the transition of new managers and other organizational challenges.

The early 2005 discovery of USGS e-mails suggesting possible noncompliance with the project’s quality assurance requirements has left lingering concerns about the adequacy of USGS’s scientific work related to the infiltration or flow of water into the repository and whether other work on the project has similar quality assurance problems. As part of its new path forward, DOE has taken steps to address these concerns. It is reworking technical documents created by USGS personnel to ensure that the science underlying the conclusions on water infiltration is correct and supportable. In addition, DOE is conducting an extensive review of approximately 14 million e-mails to determine whether they raise additional quality assurance concerns. According to NRC on-site representatives, screening these millions of e-mails to ensure that records were not falsified will be challenging. Further, many of the e-mails were written by employees who no longer work at the project or may be deceased, making it difficult to learn their true meaning and context. Moreover, if additional e-mails raise quality assurance concerns, DOE may have to initiate further review, inspections, or rework.

DOE officials have stated that it will need to resolve long-standing quality assurance problems involving requirements management before it can perform the design and engineering work needed to support the revised project plans called for by its new path forward. According to a 2005 DOE root cause analysis report, low-level documents were appropriately updated and revised to reflect high-level design changes through fiscal year 1995. However, from 1995 through 2002, many of these design documents were not adequately maintained and updated to reflect current designs and requirements. Further, a document that is a major component of the project’s requirements management process was revised in July 2002, but has never been finalized or approved. Instead, the project envisioned a transition to a new requirements management system after the submission of the license application, which at that time was planned for December 2004. However, for various reasons, the license application was not submitted, and the transition to a new requirements management system was never implemented. The DOE report described this situation as “completely dysfunctional” and identified the root cause of these conditions as DOE’s failure to fund, maintain, and rigidly apply a requirements management system. According to an NRC on-site representative, repetitive and uncorrected issues associated with the requirements management process could have direct implications for the quality of DOE’s license application.

In December 2005, DOE issued a stop-work order on design and engineering for the project’s surface facility and certain other technical work. DOE stated that a root cause analysis and an investigation into employee concerns had revealed that the project had not maintained or properly implemented its requirements management system, resulting in inadequacies in the design control process. The stop-work order will be in effect until, among other things, the lead contractor improves the requirements management system, validates that processes exist and are being followed, and requirements are appropriately traced to implementing mechanisms and products. Further, DOE will establish a team to take other actions necessary to prevent inadequacies in requirements management and other management systems from recurring.

Finally, DOE continues to be challenged to effectively manage a changing and complex program and organization. The significant project changes initiated under the new path forward create the potential for confusion over accountability as roles and responsibilities change—a situation DOE found to contribute to quality assurance problems during an earlier transition period. An important part of this challenge is ensuring that accountability for quality and results are effectively managed during the transition to the new path forward. For example, DOE’s plan to establish a lead laboratory to assist the project would not only have to be effectively managed, but also would introduce a new player whose accountability DOE would have to ensure. According to one DOE manager, transitioning project work to a lead laboratory under a direct contract with DOE could pose a significant challenge for quality assurance because the various laboratories assisting with the project are currently working under BSC quality assurance procedures and will now have to develop their own procedures.

In addition, the project faces management challenges related to ensuring management continuity at the project. DOE has experienced turnover in 9 of 17 key management positions since 2001. For example, in the past year, the project has lost key managers through the departures of the Director of Project Management and Engineering, the Director of the License Application and Strategy, the Director of Quality Assurance, and the contractor’s General Manager. To ensure the right managers move the project forward to licensing, the project has a recruitment effort for replacing key departing managers. Further, the director position for the project has been occupied by three individuals since 1999 and is currently filled by an acting director. The current Acting Director took his position in summer 2005, and initiated the new path forward in October 2005. DOE is currently awaiting congressional confirmation of a nominee to take the director position. However, the current Acting Director told us he expects that the new path forward will be sustained after the new director assumes the position because it has been endorsed by the Secretary of Energy.

Conclusions

DOE has a long history of trying to resolve quality assurance problems at its Yucca Mountain project. Now, after more than 20 years of work, DOE once again faces serious quality assurance and other challenges while seeking a new path forward to a fully defensible license application. Even as DOE faces new quality assurance challenges, it cannot be certain that it has resolved past problems. It is clear that DOE has not been well served by management tools that have not effectively identified and tracked progress on significant and recurring problems. As a result, DOE has not had a strong basis to assess progress in addressing management weaknesses or to direct management attention to significant and recurrent problems as needed. Unless these quality assurance problems are addressed, further delays on the project are likely.

Mr. Chairman, this concludes my prepared statement, I would be happy to respond to any questions that you or other Members of the Subcommittee may have at this time.

GAO Contact and Staff Acknowledgments

For further information about this testimony, please contact Jim Wells at (202) 512-3841 or wellsj@gao.gov. Casey Brown, John Delicath, Terry Hanford, and Raymond Smith also made key contributions to this statement.

Appendix I: The Management Improvement Initiatives’ Key Areas of Concern

The Department of Energy’s Management Improvement Initiatives (Initiatives) perceived five key areas of management weakness as adversely affecting the implementation of quality assurance requirements at the Yucca Mountain project: 1. Roles and responsibilities were becoming confused as the project transitioned from scientific studies to activities supporting licensing. The confusion over roles and responsibilities was undermining managers’ accountability for results. The Initiatives’ objective was to realign DOE’s project organization to give a single point of responsibility for project functions, such as quality assurance and the Corrective Action Program, and hold the project contractor more accountable for performing the necessary work in accordance with quality, schedule, and cost requirements. 2. Product quality was sometimes being achieved through inspections by the project’s Office of Quality Assurance rather than being routinely implemented by the project’s work organizations. As a result, the Initiatives sought to increase work organizations’ responsibility for being the principle means for achieving quality. 3. Work procedures were typically too burdensome and inefficient, which impeded work. The Initiatives sought to provide new user- friendly and effective procedures, when necessary, to allow routine compliance with safety and quality requirements. 4. Multiple corrective action programs existed, processes were burdensome and did not yield useful management reports, and corrective actions were not completed in a timely manner. The Initiatives sought to implement a single program to ensure that problems were identified, prioritized, and documented and that timely and effective corrective actions were taken to preclude recurrence of problems. 5. The importance of a safety-conscious work environment that fosters open communication about concerns was not understood by all managers and staff, and they had not been held accountable when inappropriately overemphasizing the work schedule, inadequately attending to work quality, and acting inconsistently in practicing the desired openness about concerns. Through issuing a work environment policy, providing training on the policy, and improving the Employee Concerns Program, the Initiatives sought to create an environment in which employees felt free to raise concerns without fear of reprisal and with confidence that issues would be addressed promptly and appropriately.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.