Enclosure 

U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION AUDIT REPORT OAR-07-03, 

OBSERVATION AUDIT OF THE U.S. DEPARTMENT OF ENERGY, 

OFFICE OF CIVILIAN RADIOACTIVE WASTE MANAGEMENT, 

OFFICE OF QUALITY ASSURANCE AUDIT OQA-SNL-07-06, 

OF THE INFILTRATION MODEL REPORT 

 /RA/ 11/16/07 /RA/ 11/08/07

 Jack D. Parrott Eugene Peters 

 Division of High-Level Waste Division of High-Level Waste 

 Repository Safety Repository Safety 

 Office of Nuclear Material Safety Office of Nuclear Material Safety 

 and Safeguards and Safeguards 

 

 /RA/ 11/16/07 /RA/ 11/06/07

 Randall Fedors Vincent Everett 

 Division of High-Level Waste Division of Nuclear Material Safety 

 Repository Safety Region IV 

 Office of Nuclear Material Safety 

 and Safeguards 

 

 /RA/ 12/17/07 /RA/ 11/07/07

 Stuart Stothoff Rodney Weber 

 Center for Nuclear Waste Center for Nuclear Waste 

 Regulatory Analyses Regulatory Analyses 

 

 

 

 Reviewed and Approved by: 

 /RA/ 12/21/07

 Aby Mohseni, Deputy Director 

 Licensing and Inspection Directorate 

 Division of High-Level Waste 

 Repository Safety 

 Office of Nuclear Material Safety 

 and Safeguards 
2

1.0 INTRODUCTION

On June 4-18, 2007, staff from the U.S. Nuclear Regulatory Commission’s (NRC’s), Division of 

High-Level Waste Repository Safety, and the Center for Nuclear Waste Regulatory Analyses 

(CNWRA) observed the U.S. Department of Energy (DOE), Office of Civilian Radioactive Waste 

Management (OCRWM), Office of Quality Assurance (OQA), quality assurance (QA) audit 

OQA-SNL-07-06, in Albuquerque, New Mexico, and Las Vegas, Nevada. The DOE audit team 

evaluated the infiltration model developed by the Sandia National Laboratory, OCRWM’s Lead 

Laboratory (Lead Lab) for the Yucca Mountain Project. The DOE audit team transmitted its 

report on this audit to the Lead Lab Program Director by letter dated August 15, 2007. It is 

available on NRC’s Licensing Support Network (accession numbers DN2002454003 and 

DN2002456224). 

The primary objective of the DOE audit was to evaluate the MDL-NBS-HS-000023, “Simulation 

of Net Infiltration for Present-Day and Potential Future Climates” (Infiltration Model Report) 

relative to defensibility in the potential license application process. The DOE audit team 

performed an overall assessment of the technical adequacy, product quality, and 

implementation effectiveness of the “Quality Assurance Requirements and Description” 

(QARD), DOE/RW-0333P, requirements for the Infiltration Model Report. The objective of the 

NRC observation was to assess whether the audit team effectively met the objectives of the 

audit. 

2.0 MANAGEMENT SUMMARY 

The DOE auditors evaluated the Infiltration Model Report through staff technical interviews and 

examination of quality records. They identified nine conditions adverse to quality and eight 

technical issues but determined that overall, the Infiltration Model Report is adequate for its 

intended purpose and meets established criteria for overall effectiveness in QARD 

implementation, technical adequacy, and product quality. The DOE audit team also identified 

three noteworthy (i.e., good) practices and eight recommendations for improvement. 

The NRC observers determined that the DOE audit team performed effectively and met the 

objectives of the audit. However, the NRC observers identified five areas where the objective of 

the audit may have been better achieved. In accordance with the “Agreement Between 

DOE/OCRWM and NRC/Office of Nuclear Material Safety and Safeguards (NMSS) Regarding 

Prelicensing Interactions,” five NRC recommendations for these areas are summarized below: 

The primary NRC recommendation is to: 

1) Avoid grouping multiple audit-identified technical issues into one condition report (CR), [see 

Section 4.2.3]. 

Of secondary importance are NRC recommendations to: 

2) Conduct audits earlier in the model development process, so that audit-identified issues can 

be addressed before the final revision of the technical product is produced, [see Section 4.2.1]; 
3

3) Have the most current version of the technical products being audited available to 

performance-based audit teams before the audit, [see Section 4.2.2]; 

4) Include, in the initial scope of performance-based audits of technical products, any 

subordinate model input reports that discuss the collection, qualification, reduction, and 

manipulation of input data critical to the technical products, [see Section 4.1]; and, 

5) More effectively use the NRC’s Yucca Mountain Review Plan (YMRP) acceptance criteria, as 

performance-based audit criteria, at the beginning of the audit of technical products that support 

the license application, [see Section 4.1]. 

3.0 AUDIT PARTICIPANTS

DOE Audit Team Members

Marilyn Kavchak, OQA, Audit Team Leader 

Patrick Auer, Project Enhancement Corporation (PEC), Audit Team Member 

Christian Palay, OQA, Audit Team Member 

Mark Kachun, PEC, Audit Team Member 

Lester Wagner, PEC, Audit Team Member 

Harvey Dove, PEC, Technical Specialist 

Thomas Doe, Golder and Associates, Technical Specialist 

John McCray, Management and Technical Support Services (MTS), Technical Specialist 

Peter Wierenga, MTS, Technical Specialist 

NRC Observers

Jack Parrott, NRC, Observation Team Leader 

Randall Fedors, NRC, Technical Observer 

Eugene Peters, NRC, Technical Observer 

Vincent Everett, NRC, Quality Assurance Specialist 

Rodney Weber, CNWRA, Quality Assurance Specialist 

Stuart Stothoff, CNWRA, Technical Observer

Affected Units of Local Government Observers

Susan Lynch, State of Nevada 

Engelbrecht Von Teisenhausen, Clark County, Nevada 

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION

The DOE auditors conducted the audit in accordance with Line Procedure 18.5Q-OCRWM, 

“OCRWM Contractor Surveys/Audits.” They identified adverse conditions and 

recommendations in accordance with Administrative Procedure 16.1Q, “Condition Reporting 

and Resolution.” The NRC observers followed NRC Manual Chapter 2410, “Conduct of 

Observation Audits,” and/or the “Agreement Between DOE/OCRWM and NRC/NMSS 

Regarding Prelicensing Interactions,” in the conduct and documentation of the audit 

observation. 
4

4.1 Scope of the Audit

The DOE auditors evaluated the Infiltration Model Report in regard to the requirements of the 

QARD. The DOE audit team evaluated the technical adequacy and product quality against 

critical process steps that include planning and procedural compliance, control of data and 

input, validity of calculation and calculation documentation, document checking and comment 

resolution, and document accuracy and compliance. 

The DOE audit focused on reviewing the hydrological aspects and critical process steps applied 

in the development of the Infiltration Model Report. The DOE audit also evaluated use of 

material from report ANL-NBS-HS-000027, Revision 1, “Analysis of Infiltration Uncertainty,” to 

determine whether inputs (i.e., data) were adequately integrated into the Infiltration Model 

Report. 

Although the planned scope of the audit addressed the Infiltration Model Report and the 

“Analysis of Infiltration Uncertainty” report, the NRC observers noted that there were nine other 

key subordinate input reports that discussed the collection, qualification, reduction, and 

manipulation of input data critical to the infiltration model (e.g., soil thickness and bedrock 

hydraulic properties). The NRC observers noted that the objective of the audit would have been 

better achieved if the original scope had included these nine subordinate input reports. The 

NRC observers recommend that, for similar technically focused audits, the scope be expanded 

to include these types of key supporting documents, particularly documents like these that were 

created for the sole purpose of supporting, and providing input to, the audited document. The 

subordinate input reports for the Infiltration Model Report are particularly significant in that they 

seek to substantiate the quality of the data used in the original U.S. Geological Survey net 

infiltration modeling report. 

The NRC observers also noted that the audit plan stated, “Overall the Model Report, MDL-NBSHS-000023, will be evaluated relative to the defendability [defensibility] in the license application 

process.” However, the performance-based audit criteria identified in the audit plan and initially 

used in the audit were not specifically tied to NRC’s YMRP acceptance criteria, which NRC will 

use in the potential license application review process. During the audit, the DOE audit team 

recognized the utility of evaluating the Infiltration Model Report against the applicable criteria in 

the YMRP, to help determine the report’s defensibility as a support document in a potential 

licensing process. The NRC observers recommend that the YMRP acceptance criteria be used 

in future performance-based audits of technical products used to support the licensing basis of a 

potential license application. 

4.2 Conduct and Timing of the Audit

The DOE audit team members conducted the audit through records reviews and interviews, 

challenged and questioned responses when appropriate, and effectively employed their 

checklists. The DOE auditors caucused daily with the NRC observers to discuss the current 

audit status and identified issues. 
5

The NRC observers determined that the DOE audit team performed the audit effectively and 

demonstrated adequate knowledge of the applicable implementing procedures and QA program 

requirements. However, NRC observers identified three issues related to the conduct and 

timing of the audit as discussed in the following three sections. 

4.2.1 Audit Scheduling

Attachment 3 of Procedure LP-18.5Q-OCRWM, “OCRWM Contractor Surveys/Audits,” under 

the section, “Scheduling and Planning OCRWM Contractor Surveys/Audits,” states, “Audits of 

OCRWM contractors shall be scheduled in a manner to provide coverage, consistency, and 

coordination with ongoing work as follows: Begin the audit as early in the life of the work as 

practical and at a frequency commensurate with the status and importance of the work.” 

The work on the Infiltration Model Report originally began on September 22, 2005; however, the 

June 2007 audit was the first audit of this activity. Conducting this audit when work is 

essentially complete impacts the objective of a performance-based audit because it becomes 

more difficult to incorporate the findings to improve the quality (i.e., defensibility) of the product. 

Also, a DOE audit team finding, that several QA program requirements were not effectively 

implemented during the development of the Infiltration Model Report, indicates that auditing of 

this work should have been done sooner, or that additional audits or surveillances of this work 

should have been done while the work was in progress. NRC recommends that OQA evaluate 

the timing of performance-based audits so as to be able to more effectively use the results to 

improve the quality of the products audited. 

4.2.2 Coordination Between the Auditing and Audited Organization

The checklist the DOE auditors used was developed using a draft version of the Infiltration 

Model Report, with the understanding that this would be the audited product. However, as 

described in the DOE audit report, the finalized version was approved on May 24, 2007, but not 

provided to the DOE audit team until the audit started. This led to inefficient use of the checklist 

and some delay at the beginning of the audit. NRC recommends that better coordination 

between the auditing and the audited organizations be undertaken before audits begin so that 

the audited version of products is available to the auditing team somewhat before the audit 

begins. 

4.2.3 Processing of Technical Issues Identified

The NRC observers noted that, as described in the DOE audit report, the eight technical issues 

identified by the DOE technical specialists were combined into a single CR, after the audit, and 

described as “recommendations.” However, the uniqueness of each individual technical issue, 

the potential significance of each issue relative to the objective of the audit plan, its potential 

impact on the quality of the document, and the ability to trend the individual issues in the 

Corrective Action Program, or track them in other programs, demonstrate that these issues 

should have been identified as individual CRs. NRC recommends that distinct technical issues 

with potentially different levels of significance not be grouped together into one CR. 
6

4.3 Audit Team Qualifications and Independence

The NRC observers reviewed the qualifications of the auditors with respect to LP-18.4Q, “Audit 

Personnel Qualification.” Each of the DOE auditors had been qualified as lead auditor, and the 

NRC observers verified their independence from the areas reviewed in this audit. The DOE 

technical specialists had also been appropriately qualified. The DOE audit team leader 

identified several discrepancies with the qualification documentation in the auditor qualification 

records before presenting them to the NRC observers for review. However, these represented 

documentation discrepancies—not actual qualification discrepancies. These documentation 

discrepancies were documented in a CR. 

4.4 Examination of QA Elements

The DOE auditors prepared checklists to evaluate adherence to QARD requirements for the 

development of the Infiltration Model Report. The checklists were organized according to the 

critical process steps identified in the audit plan as follows: (1) planning and procedural 

compliance; (2) control of data and input; (3) validity of calculation and calculation 

documentation; (4) document checking and comment resolution; and (5) document accuracy 

and compliance. The QA auditors also followed up on previously closed CRs related to the 

Infiltration Model Report and identified one CR as having inadequate completion of 

corrective action. In addition, the DOE auditors reviewed online records to assess areas such 

as: (a) contracting; (b) purchasing; (c) training documentation; (d) data inputs; (e) checking and 

review; (f) approvals; and (g) change control. They identified areas where the Lead Lab had 

failed to effectively implement some QA program elements. 

The NRC observers agreed with the audit results in these areas. However, the NRC observers 

noted that a DOE auditor review of the Lead Lab audit, surveillance, and self-assessment 

schedules for the Infiltration Model Report work indicated that scheduled audit and selfassessment activities had not been conducted (see Section 4.2.1). Had they been conducted, 

the issues regarding implementation of some QA program elements might have been identified 

earlier. 

4.5 Examination of Technical Activities

The DOE technical specialists prepared checklists based on detailed reviews of the Infiltration 

Model Report and associated technical work plans, before the audit. During the audit, the nine 

subordinate input reports were identified and also reviewed, and further audit questions were 

generated. The NRC observers noted that these nine subordinate input reports are integral for 

supporting input values used in the net infiltration model and should have been reviewed before 

the audit to facilitate the objective of the audit. During the audit, the DOE audit team 

interviewed a primary contributor and other key contributors to the Infiltration Model Report, as 

needed. 

With a view toward defensibility of the Infiltration Model Report in a potential repository licensing 

process, two areas of focus for the DOE audit team were: (1) control of data and inputs; and (2) 

validity of calculation and calculation documentation. For the control of data and inputs, the 

DOE technical specialists identified the sensitive parameters for the net infiltration estimates 

and focused questions during the audit accordingly. For the validity of calculation and 
7

calculation documentation, the DOE technical specialists focused on traceability of supporting 

bases for inputs and models, including both software validation and validation of results. 

Within these focus areas, the DOE technical specialists identified eight technical issues, which 

were summarized in one CR as recommendations. The NRC observers noted that the 

recommendations, if implemented, will improve confidence in the model results and therefore 

the quality of the Infiltration Model Report. However, the NRC observers noted that technical 

recommendation 6 discussed a bias in the calculations towards an overestimation of infiltration, 

caused by the tendency to select conservative parameters or assumptions. It should be noted 

that biases in assumptions and model inputs can lead to overestimates or underestimates. The 

YMRP acceptance criteria leads NRC reviewers to ensure that parameter values and modeling 

results provide a conservative estimate of risk or do not cause an overall reduction of the risk 

estimate. Therefore, technical evaluations of risk inputs should focus on reducing the 

uncertainty of parameters or assumptions, or on providing enough information to evaluate the 

effects of biases on results, rather than highlighting conservative inputs or assumptions. 

5.0 NRC STAFF FINDINGS 

The NRC observers determined that the DOE audit team conducted the audit effectively and 

demonstrated acceptable knowledge of the applicable implementing procedures and QA 

requirements. The DOE auditors conducted thorough interviews, challenged and questioned 

responses when appropriate, and effectively employed their checklists. 

The NRC observers agreed with the DOE audit team’s conclusions and findings, but note that 

the objectives of the audit would have been better achieved had: 

• The original scope of the audit identified, for review, the nine subordinate documents 

that provide support for direct inputs to the model; 

• The YMRP acceptance criteria been more effectively used as audit criteria at the 

beginning of the audit, to help determine model defensibility; 

• The timing of the audit relative to document finalization been better coordinated to allow 

for pre-audit review and checklist development on the version of the document that was 

actually audited; 

• The technical issues not been grouped into one CR as that might have masked the 

uniqueness and significance of the issues and might have precluded model quality 

improvement; 

• An audit been conducted earlier in the model development process, since at least some 

of the issues identified in this audit could have been addressed before the Lead Lab’s 

final revision of the Infiltration Model Report. 

The observers also identified several good practices during the audit. These included: 

• Good issue discovery through review of objective evidence, even in areas that were not 

strictly compliance issues; 
8

• Willingness of QA auditors to use conservative judgments regarding interpretation 

of requirements; and 

 

• Improved facilitation of NRC’s observation by the presence of observers from QA and 

 licensing.