May 6, 2003

Mr. Ronald A. Milner, Chief Operating Officer

Office of Civilian Radioactive Waste Management

U.S. Department of Energy

1000 Independence Avenue, SW

Washington, DC 20585

Subject: U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION AUDIT

REPORT OAR–03–02, OBSERVATION AUDIT OF OFFICE OF QUALITY

ASSURANCE AUDIT OQAP–BSC–03–05 OF BECHTEL SAIC COMPANY, LLC

Dear Mr. Milner:

I am transmitting the U.S. Nuclear Regulatory Commission’s (NRC’s) Observation Audit Report

OAR–03–02. Staff from the NRC, Division of Waste Management, observed a U.S.

Department of Energy’s (DOE), Office of Quality Assurance (OQA), data verification audit

OQAP–BSC–03–05 on March 17–27, 2003, at the Bechtel SAIC Company, LLC (BSC), facility

in Las Vegas, Nevada. The objectives of this performance-based audit were to assess data

quality (including data accuracy, adequacy, consistency, traceability, transparency,

appropriateness, and overall integrity) supporting the Yucca Mountain Project license

application (LA), and BSC’s implementation of the Office of Civilian Radioactive Waste

Management (OCRWM) Quality Assurance Requirements and Description (QARD),

DOE/RW–0333P, Revision 12, regarding controlling data. 

During the Post Audit Conference on March 27, 2003, the DOE Audit Team Leader concluded

that the DOE audit team was unable to determine the quality of data used to support the LA. 

The Audit Team Leader made this conclusion because a broad sample of completed technical

products in support of LA were in draft form and were, therefore, not available for review by the

audit team. Furthermore, some of the technical products not yet completed are critical to the

LA and should be a part of the audit in order to determine the overall quality of data. The

Director, OQA, stated at the conclusion of the Post Audit Conference that DOE intends to

conduct a second data verification audit to verify the integrity of data used to support the LA.

The observers found that the DOE audit of BSC to determine the quality of data was not fully

effective. The observers noted that certain audit team members were not familiar with the

automated Tracking Data Management System (TDMS), which was a critical tool in

accomplishing the objectives of this audit. This resulted in some delays and inefficiency in

performing data verification activities. The NRC observers concluded that the actual conduct of

this audit may have been more appropriately performed at a time when a larger sample of

completed technical products critical to the LA were available. Based on the conduct and

results of the audit, the NRC observers concluded that the quality of data to support the LA is

indeterminate. A description of the conduct of the audit, and the observers’ findings, are

provided in the enclosed Observation Audit Report.
The audit team identified three potential deficiencies: 1) Inadequate documentation for

evaluation criteria in technical work plans (TWPs); 2) a failure to adequately document the

results of data qualifications, a lack of rationale for abandoning selected evaluation criteria, and

failure to sufficiently identify subject matter experts; and 3) a lack of traceability and

transparency of data to the Technical Data Management System. 

The audit team also identified four potential quality observations: 1) Inadequate control of

electronic data in one scientific notebook (SN) related to a TWP; 2) a procedural inadequacy in

AP-2.27, Revision 0, Planning for Science Activities; 3) an incorrect document reference where

a SN number was transposed; and 4) a procedure, AP-SIII.2Q, Unqualified Data and the

Documentation of Rationale for Accepted Data, does not correctly reflect QARD requirements. 

The observers agreed with the audit team’s conclusions and findings presented at the Post

Audit Conference. However, during the conduct of the audit, the observers initiated an Audit

Observer Inquiry (AOI) OQAP–BSC–03–05-01 regarding the identification of trends associated

with data-related deficiency reports written in the last four years that the audit team did not

identify during this audit. The observers provided this AOI to the Audit Team Leader at the Post

Audit Conference and requested a response to the AOI within 45 days. 

A written response to this letter and the enclosed report is not required. The staff will continue

to interface with OCRWM and follow the action that BSC is taking to address the issues

identified during this audit. If you have any questions, please contact Ted Carter of my staff at

301-415-6684.

Sincerely,

/RA/

Janet Schlueter, Chief

High-Level Waste Branch

Division of Waste Management

Office of Nuclear Material Safety

 and Safeguards

Enclosure: U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION

AUDIT REPORT OAR–03–02, OBSERVATION AUDIT OF

OFFICE OF QUALITY ASSURANCE AUDIT OQAP–BSC–03–05

OF BECHTEL SAIC COMPANY, LLC

cc: See attached distribution list
R. A. Milner

The audit team identified three potential deficiencies: 1) Inadequate documentation for

evaluation criteria in technical work plans (TWPs); 2) a failure to adequately document the

results of data qualifications, a lack of rationale for abandoning selected evaluation criteria, and

failure to sufficiently identify subject matter experts; and 3) a lack of traceability and

transparency of data to the Technical Data Management System. 

The audit team also identified four potential quality observations: 1) Inadequate control of

electronic data in one scientific notebook (SN) related to a TWP; 2) a procedural inadequacy in

AP-2.27, Revision 0, Planning for Science Activities; 3) an incorrect document reference where

a SN number was transposed; and 4) a procedure, AP-SIII.2Q, Unqualified Data and the

Documentation of Rationale for Accepted Data, does not correctly reflect QARD requirements. 

The observers agreed with the audit team’s conclusions and findings presented at the Post

Audit Conference. However, during the conduct of the audit, the observers initiated an Audit

Observer Inquiry (AOI) OQAP–BSC–03–05-01 regarding the identification of trends associated

with data-related deficiency reports written in the last four years that the audit team did not

identify during this audit. The observers provided this AOI to the Audit Team Leader at the Post

Audit Conference and requested a response to the AOI within 45 days. 

A written response to this letter and the enclosed report is not required. The staff will continue

to interface with OCRWM and follow the action that BSC is taking to address the issues

identified during this audit. If you have any questions, please contact Ted Carter of my staff at

301-415-6684.

Sincerely,

/RA/

Janet Schlueter, Chief

High-Level Waste Branch

Division of Waste Management

Office of Nuclear Material Safety

 and Safeguards

Enclosure: U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION

AUDIT REPORT OAR–03–02, OBSERVATION AUDIT OF

OFFICE OF QUALITY ASSURANCE AUDIT OQAP–BSC–03–05

OF BECHTEL SAIC COMPANY, LLC

cc: See attached distribution list

DISTRIBUTION:

FILE CENTER HLWB r/f DWM r/f ACNW PUBLIC OSRs

LSN CNWRA CWReamer JSchleuter KStablein LCampbell

MYoung ACampbell BSpitzberg KBrockman DDambly D. Esh

MEhnstrom DHiggs TCarter TMatula BMabrito RLatta

TAhn JParrott SFlanders BSchlapper

S:DWM\HLWB\THC\OAR-03-02-DataRev5.wpd ML

OFFICE HLWB Tech Ed HLWB HLWB HLWB

NAME T. Carter E. Kraus T. Matula K. Stablein J. Schlueter

DATE / /03 / /03 / /03 / /03 / /03

OFFICIAL RECORD COPY
Letter to R. Milner from J. Schlueter dated : May 30, 2003

A. Kalt, Churchill County, NV M. Corradini, NWTRB

R. Massey, Churchill/Lander County, NV J. Treichel, Nuclear Waste Task Force

I. Navis, Clark County, NV K. Tilges, Shundahai Network

E. von Tiesenhausen, Clark County, NV M. Chu, DOE/Washington, D.C.

G. McCorkell, Esmeralda County, NV G. Runkle, DOE/Washington, D.C.

L. Fiorenzi, Eureka County, NV C. Einberg, DOE/Washington, D.C.

A. Johnson, Eureka County, NV S. Gomberg, DOE/Washington, D.C.

A. Remus, Inyo County, CA W. J. Arthur, III , DOE/ORD

M. Yarbro, Lander County, NV R. Dyer, DOE/ORD

D. Carriger, Lincoln County, NV J. Ziegler, DOE/ORD

M. Baughman, Lincoln County, NV A. Gil, DOE/ORD

L. Mathias, Mineral County, NV W. Boyle, DOE/ORD

L. Bradshaw, Nye County, NV D. Brown, DOE/OCRWM

D. Chavez, Nye County, NV S. Mellington, DOE/ORD

D. Hammermeister, Nye County, NV C. Hanlon, DOE/ORD

J. Larson, White Pine County, NV T. Gunter, DOE/ORD

J. Ray, NV Congressional Delegation S. Morris, DOE/ORD

B. J. Gerber, NV Congressional Delegation J. Mitchell, BSC

F. Roberson, NV Congressional Delegation M. Mason, BSC

T. Story, NV Congressional Delegation S. Cereghino, BSC

J. Reynoldson, NV Congressional Delegation N. Williams, BSC

L. Hunsaker, NV Congressional Delegation M. Voegele, BSC/SAIC

S. Joya, NV Congressional Delegation D. Beckman, BSC/B&A

K. Kirkeby, NV Congressional Delegation W. Briggs, Ross, Dixon & Bell

R. Loux, State of NV P. Johnson, Citizen Alert

S. Frishman, State of NV R. Holden, NCAI

S. Lynch, State of NV B. Helmer, Timbisha Shoshone Tribe

M. Paslov Thomas, Legislative Counsel Bureau R. Boland, Timbisha Shoshone Tribe

J. Pegues, City of Las Vegas, NV J. Birchim, Yomba Shoshone Tribe

M. Murphy, Nye County, NV R. Arnold, Pahrump Paiute Tribe
cc: (Continued)

R. Clark, EPA C. Meyers, Moapa Paiute Indian Tribe

R. Anderson, NEI R. Wilder, Fort Independence Indian Tribe

R. McCullum, NEI D. Vega, Bishop Paiute Indian Tribe

S. Kraft, NEI J. Egan, Egan & Associates, PLLC

J. Kessler, EPRI J. Leeds, Las Vegas Indian Center

D. Duncan, USGS R. M. Saulque, Benton Paiute Indian Tribe

R. Craig, USGS C. Bradley, Kaibab Band of Southern Paiutes

W. Booth, Engineering Svcs, LTD R. Joseph, Lone Pine Paiute-Shoshone Tribe

L. Lehman, T-Reg, Inc. L. Tom, Paiute Indian Tribes of Utah

S. Echols, ESQ . E. Smith, Chemehuevi Indian Tribe

M. Smurr, BNFL, Inc. V. McQueen, Sr., Ely Shoshone Tribe

J. Bococh, Big Pine Paiute Tribe of the

Owens Valley 

R. Quintero, Inter-Tribal Council of NV

 (Chairman, Walker River Paiute Tribe)

R. Mike, Duckwater Shoshone Tribe D. Eddy, Jr., Colorado River Indian Tribes

T. Kingham, GAO H. Jackson, Public Citizen

D. Feehan, GAO J. Wells, Western Shoshone National Council

E. Hiruo, Platts Nuclear Publications D. Crawford, Inter-Tribal Council of NV

G. Hernandez, Las Vegas Paiute Tribe I. Zabarte, Western Shoshone National Council

K. Finfrock, NV Congressional Delegation NRC On-Site Representatives

L. Tomero, NV Congressional Delegation 

 
U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION AUDIT REPORT OAR–03–02,

OBSERVATION AUDIT OF OFFICE OF QUALITY ASSURANCE AUDIT OQAP–BSC–03–05

OF BECHTEL SAIC COMPANY, LLC

 /RA/ 5/ 29 /03 /RA/ 5/ 29 /03

Ted Carter Thomas Matula

Projects and Engineering Section Projects and Engineering Section 

High-Level Waste Branch High-Level Waste Branch

Division of Waste Management Division of Waste Management

 /RA/ 5/ 30 /03 /RA/ 5 / 30 /03

David Esh Beth Schlapper

Performance Assessment Section Repository Site Section

Environmental and Performance High-Level Waste Branch

 Assessment Branch Division of Waste Management

Division of Waste Management

 /RA/ 5/ 29 /03

Rodney Weber

Center for Nuclear Waste Regulatory Analyses

Reviewed and Approved by:

 /RA/ 5/ 30 /03

N. King Stablein, Chief

Projects and Engineering Section

High-Level Waste Branch

Division of Waste Management
2

1.0 INTRODUCTION

Staff from the U.S. Nuclear Regulatory Commission (NRC), Division of Waste Management,

observed a U.S. Department of Energy’s (DOE), Office of Quality Assurance (OQA), data

verification audit, OQAP–BSC–03–05, on March 17-27, 2003, at the Bechtel SAIC Company,

LLC (BSC), facility in Las Vegas, Nevada. The objectives of this performance-based audit were

to assess data quality (including data accuracy, adequacy, consistency, traceability,

transparency, appropriateness, and overall integrity) supporting the Yucca Mountain Project

license application (LA), and BSC’s implementation of the Office of Civilian Radioactive Waste

Management’s Quality Assurance Requirements and Description (QARD), DOE/RW–0333P,

Revision 12, regarding the control of data. The DOE OQA audit team (audit team) assessed

data used in specific technical products (including model, analysis, and technical reports)

against established measurement criteria described in the BSC Audit Plan OQAP-BSC-03-05,

Revision 1. The NRC observers (observers) assessed the effectiveness of the audit team and

the audit process in achieving the audit objectives. 

2.0 MANAGEMENT SUMMARY

The audit team’s goal was to assess the quality of data used to support the LA by verifying

implementation of appropriate sections of the QARD regarding data, and by reviewing data

used in completed technical products. Within the areas evaluated, the audit team identified

three potential deficiencies: 1) Inadequate documentation for evaluation criteria in technical

work plans (TWPs) as required by Administrative Procedure (AP)-SII.2Q, Revision 1, ICN 0,

Qualification of Unqualified Data and the Documentation of Rationale for Accepted Data; 2) a

failure to adequately document the results of data qualifications, a lack of rationale for

abandoning selected evaluation criteria, and failure to sufficiently identify subject matter

experts; and 3) a lack of traceability and transparency of data to the Technical Data

Management System. 

The audit team also identified four potential quality observations: 1) Inadequate control of

electronic data in one scientific notebook (SN) related to a TWP; 2) a procedural inadequacy in

AP-2.27, Revision 0, Planning for Science Activities; 3) an incorrect document reference where

a SN number was transposed; and 4) a procedure, AP-SIII.2Q, does not correctly reflect QARD

requirements. 

However, during the Post Audit Conference on March 27, 2003, the DOE Audit Team Leader

concluded that the DOE audit team was unable to determine the quality of data used to support

the LA. The Audit Team Leader made this conclusion because a broad sample of completed

technical products in support of LA were in draft form and were, therefore, not available for

review by the audit team. Furthermore, some of the technical products not yet completed are

critical to the LA and should be a part of the audit in order to determine the overall quality of

data. The Director, OQA, stated at the conclusion of the Post Audit Conference that DOE

intends to conduct a second data verification audit to verify the integrity of data used to support

the LA.

The observers noted that certain audit team members were not familiar with the automated

Tracking Data Management System (TDMS), which was a critical tool in accomplishing the
3

objectives of this audit. This resulted in some delays and inefficiency in performing data

verification activities. The NRC observers concluded that the actual conduct of this audit may

have been more appropriately performed at a time when a larger sample of completed technical

products critical to the LA were available. Based on the conduct and results of the audit, the

observers concluded that the quality of data to support the LA is indeterminate. The observers

determined that the DOE audit of BSC to determine data quality was not fully effective as

described in Section 4.0 below. 

The observers reviewed the qualifications for the Audit Team Leader and the auditors and

determined that they were qualified and independent of the areas reviewed. The audit team

found the implementation of the QARD regarding data to be adequate. The observers agreed

with the audit team’s conclusions, findings, and recommendations presented at the Post Audit

Conference.

3.0 PARTICIPANTS

DOE Audit Team Members

Marilyn Kavchack, Navarro Quality Services (NQS), Audit Team Leader

James Voigt, NQS, Auditor

Bruce Foster, NQS, Auditor

Christian Palay, NQS, Auditor

Mary Bennington, OQA, Auditor

Harvey Dove, NQS, Technical Specialist

Eric Zwahlen, Management and Technical Support/Golder Associates (MTS), Technical

Specialist

Ronald Linden, MTS, Technical Specialist

Observers

Ted Carter, NRC, Team Leader

Tom Matula, NRC, QA Engineer

David Esh, NRC, Technical Specialist

Beth Schlapper, NRC, Technical Specialist

Rodney M. Weber, Center for Nuclear Waste Regulatory Analyses, QA Specialist

Jon White, DOE, Office of Repository Development

4.0 REVIEW OF AUDIT AND AUDITED ORGANIZATION

The audit team performed the data verification audit by following AP–18.3Q, Internal Audit

Program, and AP–16.1Q, Management of Conditions Adverse to Quality. The audit team used

the QARD, Scientific Process Guidelines Manual, TDR–WIS–MD–000001 R–01, and applicable

QARD implementing procedures to generate the audit checklist. The NRC observers followed

NRC Manual Chapter 2410, Conduct of Observation Audits, July 12, 2000, while observing the

audit.
4

4.1 Scope of the Audit

The scope of the audit included ascertaining the overall quality of data used to support the LA. 

The audit team was to determine the adequacy of data used in completed technical products

and verify implementation of appropriate sections of the QARD. The scope of the audit

included the following QARD elements: 1) QA Program; 2) Design Control; 3) Implementing

Documents; 4) Corrective Action; 5) QA Records; 6) Scientific Investigation; and 7) Control of

Electronic Management of Data. 

The scope of the audit also included evaluating the implementation of the following procedures

regarding data:

AP-2.14Q, Review of Technical Products and Data

AP-3.15Q, Managing Technical Product Inputs

AP-3.27Q, Planning for Science Activities

AP-5.1Q, Plan and Procedure Preparation, Review, and Approval

AP-SIII.1Q, Scientific Notebook

AP-SIII.2Q, Unqualified Data and the Documentation of Rationale for Accepted Data

AP-SIII.3Q, Submittal and Incorporation of Data to the Technical Data Management System

AP-SIII.9Q, Scientific Analysis

AP-SIII.10Q, Models

AP-SV.1Q, Control of the Electronic Management of Data

YMP/93-09, Technical Data Management Plan

4.2 Conduct and Timing of the Audit

During the Post Audit Conference on March 27, 2003, the DOE Audit Team Leader concluded

that the DOE audit team was unable to determine the quality of data used to support the LA. 

The Audit Team Leader made this conclusion because a broad sample of completed technical

products in support of LA were in draft form and were, therefore, not available for review by the

audit team. Furthermore, some of the technical products not yet completed are critical to the

LA and should be a part of the audit in order to determine the overall quality of data. The

Director, OQA, stated at the conclusion of the Post Audit Conference that DOE intends to

conduct a second data verification audit to verify the integrity of data used to support the LA.

The audit team caucused at the end of each day during the audit to discuss the audit status and

any new and developing issues. The audit team met with BSC management, as appropriate,

each morning, to discuss the current audit status and potential issues. The observers attended

these meetings to determine the audit team’s effectiveness in communicating issues to

interested project personnel.

4.3 DOE Audit Team Qualification and Independence

The observers reviewed the qualifications for the Audit Team Leader and the auditors and

determined that they were qualified and independent of the areas reviewed. 
5

4.4 Examination of QA Elements

The audit team assessed the development, use, and control of data in each of the following

process steps to determining data quality and qualification status:

1) Planning;

2) Data and input development;

3) Data control;

4) Technical product input selection; 

5) Analysis, documentation, and management of data; 

6) Corrective action review;

7) Evaluation of QARD Procedural Sufficiency

4.4.1 Planning

The observers determined that the planning and the timing of the audit was not adequate for

the team to effectively determine overall data quality for the LA. Specifically, an insufficient

number of completed technical products were available to evaluate in order for the auditors to

determine the overall quality of data, and some of the technical products that are critical to the

LA were not available. From a preliminary list of approximately 92 technical products for LA,

DOE originally identified the following technical products that were scheduled to be completed

and available for review during the audit:

• ANL-EBS-MD-000003, General and Localized Corrosion of WP Outer Barrier

• ANL-EBS-MD-000049, Multiscale Thermohydrologic Mode

• ANL-NBS-HS-000002, Analysis of Hydrologic Properties Data

• ANL-WIS-MD-000018, DSNF and Other WF Degradation Abstraction

• MDL-EBS-PA-000004, AMR (WF Colloid Source Term)

• MDL-MGR-GS-000002, Atmospheric Dispersal and Deposition of Tephra

• MDL-MGR-MD-000001, Biosphere Model Report

• MDL-NBS-GS-000005, Thermal Conductivity of the Potential Repository Horizon Model

Report

• MDL-NBS-HS-000004, Seepage Calibration Model and Testing Data

However, many of the technical products originally identified were not completed at the time of

the audit, so the audit team actually reviewed the following technical products during the audit:

• ANL-EBS-MD-000006, Hydrogen Induced Cracking of Drip Shield

• ANL-NBS-GS-000008, Future Climate Analysis

• ANL-NBS-HS-000015, Development of Numerical Grids for UZ Flow and Transport

Modeling

• ANL-MGR-MD-000005, Characteristics of the Receptor for the Biosphere Model

• ANL-MGR-MD-000006, Agricultural and Environmental Input Parameters for the Biosphere

Model

• MDL-NBS-GS-000002, Geologic Framework Model

• MDL-NBS-GS-000005, Thermal Conductivity of the Potential Repository Horizon Model

Report
6

• MDL-NBS-HS-000003, Calibrated Properties Model

• TDR-EBS-MD-000022, Update and Revision of Geochemical Thermodynamic Database

• TDR-MGR-GE-000003, Update and Revision of Geochemical Thermodynamic Database

• TDR-MGR-GE-000004, Intact Rock Properties Data on Poisson’s Ratio and Young’s

Modulus

• TDR-MGR-GE-000005, Intact Rock Propertied Data on Tensile Strength, Schmidt Hammer

Rebound Hardness, and Rock Triaxial Creep

• TDR-NBS-HS-000005, Mineralogy Data for Use on the Yucca Mountain Project

• TDR-WHS-CI-000001, 1:1200 Scale Topographic Maps for Use on the Yucca Mountain

Project

The audit team concluded that the broad sample of completed technical products in support of

LA were in draft and therefore not available as originally planned. The audit team reviewed only

one of the technical products originally identified during the planning phase of the audit (MDLNBS-GS-000005). Therefore, the audit team had to identify and evaluate substitute technical

products during the audit. 

The Audit Team Leader concluded that a broad sample of completed technical products in

support of LA were in draft form, and some of the technical products critical to the LA were not

available for review by the audit team. Therefore, the audit team was unable to effectively

determine the quality of data used to support the LA. The observers agreed with the audit

team’s conclusions that quality of data for the LA is indeterminate. The NRC observers

concluded that the actual conduct of this audit may have been more appropriately performed at

a time when a larger sample of completed technical products critical to the LA were available.

4.4.2 Data and Input Development

The audit team reviewed three procedures regarding data development, qualification, and

management: 1) AP-3.15, Managing Technical Product Inputs; 2) AP-SIII.2Q, Qualification of

Unqualified Data and the Documentation of Rationale for Accepted Data; and 3) AP-SIII.3Q,

Submittal and Incorporation of Data to the Technical Data Management Systems. The audit

team identified one potential deficiency in this area. The auditors’ review of one Qualification

Data Report found 24 technical publication inputs that were not documented using sufficient

evaluation criteria. The auditors determined that the summary evaluation provided in the report

was insufficient.

The observers agreed with the audit team findings in this area.

4.4.3 Data Control

The audit team reviewed three procedures regarding data control: 1) AP-2.14Q, Review of

Technical Products and Data; and 2) AP-SV.1Q, Control of the Electronic Management of Data,

formed the basis of this review. The audit team reviewed the adequacy of the procedures with

respect to the QARD; adequacy of Technical Product Review packages; information protection;

control of backup media; data integrity, security, and retrievability; and the adequacy of the
7

Technical Data Management Plan. The audit team did not identify any potential deficiencies in

this area.

The observers agreed with the audit team’s findings in this area.

4.4.4 Technical Product Input and Selection

The audit team reviewed two procedures regarding technical product input and selection:

1) AP-SIII.9Q, Scientific Analysis; and 2) AP-SIII.10Q, Models. The auditors reviewed

documents for QARD implementation; input/output transparency and traceability, overall

technical adequacy; checking; and review adequacy related to data inputs. The audit team

identified one potential deficiency in this area. The audit team found errors in the traceability of

data to the TDMS and in references cited for three of the models reviewed.

The observers agreed with the audit team’s findings in this area.

4.4.5 Analysis, Documentation, and Management of Data

The audit team reviewed AP–3.15Q, Managing Technical Product Inputs regarding analysis,

documentation, and management of data. The auditors reviewed documents for QARD

implementation; data identification and control; data status; reference agreement/traceability

with the various document information systems and technical data bases; and the appropriate

and consistent use of data. The audit team did not identify any potential deficiencies in this

area. 

The observers agreed with the audit team is findings in this area.

The observers noted that certain audit team members were not familiar with the automated

TDMS, which was a critical tool in accomplishing the objectives of this audit. This resulted in

some delays and inefficiency in performing data verification activities. 

4.4.6 Corrective Action Review

The audit team evaluated the effectiveness of corrective actions taken regarding data problems

identified over the previous two years on the following Deficiency Reports (DRs):

• BSC–02–D–074, regarding the use of data not obtained from the TDMS (two occurrences)

• BSC–01–D–063, regarding incorrect data labeling (eight sets)

• BSC(B)–03–D–054, regarding road maps for data sets (seven sets)

• BSC(B)–02–D–191, regarding the use of data from uncontrolled sources (two occurrences)

• BSC(O)–02–D–123, regarding road maps for data sets (three sets)

• BSC–02–D–009, regarding technical data review records for qualified data (three sets)

• BSC–01–D–055, regarding impact review not initiated for a superceded data set (one set)
8

The audit team incorporated specific data problems identified on the above listed DRs into the

audit checklist. Each auditor reviewed the effectiveness of the corrective actions taken

regarding the DRs as the auditors progressed through the checklist. The audit team discussed

these corrective actions during daily caucuses as they related to specific checklist questions. 

The audit team did not identify any potential deficiencies in this area. 

During the audit, the observers determined that BSC had reviewed DRs regarding data

problems for the previous four years, grouped similar deficiencies, and found trends indicating

ineffective corrective action. BSC documented this ineffective corrective action on a draft

Corrective Action Report (CAR) BSC(B)-03-107. However, the audit team did not identify this

potential significant condition adverse to quality during its review of data problems documented

on DRs. The observers prepared an audit inquiry for the purposes of acknowledging BSC’s

intent related to data discrepancies and to provide for timely follow-up (Refer to Section 5.3). 

4.4.7 Evaluation of QARD Procedural Sufficiency

The audit team assessed BSC procedure adequacy against the requirements of the QARD and

found that the procedures adequately reflect the requirements of the QARD.

The observers agreed with the audit team’s findings in this area.

4.5 Examination of Technical Activities

DOE’s TDMS was observed to be a powerful tool to enable evaluation of data traceability and

data verification. However, approximately 9000 Data Tracking Numbers (DTNs) reside in the

system, and some of the DTNs represent large amounts of information. The total number of

DTNs set for evaluation in the course of the audit was on the order of 100 or approximately

1 percent of the total in the system. As an example, a DTN associated with the calibrated

properties model (MO0012MWDGFM02.002) contained 3081 files and over a gigabyte of

information. NRC technical specialists’ evaluation of the transparency and traceability of the

DTNs, themselves, was rather straightforward, using the electronic information management

system (i.e., the correct information source was referenced). However, verification of the

technical information used from or contained within some of the larger DTNs was very difficult

to determine in anything more than a superficial manner, during the course of the audit. 

A more extensive audit, directed at verifying the technical adequacy and transparency of the

data extracted from both simple and more difficult DTNs (and a larger sample of DTNs over a

larger range of documents), is needed, to make conclusions pertaining to the adequacy of data

for a license application.

An audit checklist item was to evaluate the consistency between information/references cited in

the audited AMR and other Project documents. This type of evaluation is much more timeconsuming and requires that documents which may have used consistent types of information be

evaluated during the course of the audit. Because of the state of development of many of the

project documents, the NRC observers agree with the DOE audit team that consistent use of

common sets of information could not be conclusively evaluated during the course of this audit.
9

4.5.1 “Calibrated Properties Model,” MDL–NBS–HS–000003

The purpose of this report is to document the model that provides calibrated property sets for

unsaturated zone flow and transport process models. The calibrated property sets contain

matrix and fracture parameters for use in various natural system models. The calibration

process includes inversions using iTOUGH V5.0.

The technical specialists’ evaluation focused on 17 DTNs. The DTNs contained information on

saturations, water potentials, pneumatic pressure, infiltration maps, numerical grids,

uncalibrated fracture data, and various other types of information. In general, it was apparent

that the correct DTNs were referenced as sources of information. However, many of the

natural-system DTNs contained large amounts of information, which made evaluation of the

accurate selection/transfer of information (from the DTNs to the report or into calculations)

infeasible during the course of the audit. Table 4 (“Uncalibrated Fracture Property Data”) and

Table 15 (“Calibrated Mountain-Scale Fracture Permeabilities”) in the model were observed to

have errors, compared to the information contained in the referenced DTNs.

The audit team determined that scientific notebook LBNL–SC–SCI–229 was incorrectly

referenced as LBNL–SC–SCI–299 and that the incorrect reference was carried forward from

the notebook review into the report. This has been classified as a potential quality observation. 

The observers agreed with the audit team’s findings in this area. 

4.5.2 “Development and Qualification of the Thermodynamic Database Geochemical

Modeling Calculations and Analyses,” TDR–EBS–MD–000022

The purpose of the document was to incorporate unqualified data sets into the database,

perform an extension of the information in the database from 25C to higher temperatures,

where possible, and to develop a qualified database for use in geochemical modeling 

calculations.

The technical specialists evaluated the criteria for which qualification was determined and the

backgrounds of the members of the qualification team. The qualification team members had

extensive backgrounds in geochemical modeling and appeared to be highly qualified for the

task. The primary data qualification method was technical assessment. Many of the data

values to populate or use in creation of the database were brought in from handbooks as

accepted data. The qualification lead stated that geochemical data can be quite uncertain, and

therefore the selection of a representative single value in a data qualification effort can be

challenging. The primary method of technical assessment was to look at variability in a data

set, and qualitative judgement was used to select a particular data value for the qualified

database. The qualification lead stated that uncertainties in geochemical modeling mainly

come from modeling approaches and not necessarily from the thermodynamic data itself.

The audit team thoroughly checked all the extractions of data and manipulations by following

the AP–SIII.9Q, ”Scientific Analyses,” checking process, even though the report was not written

to AP–SIII.9Q. Aside from a list of the persons who checked various parts of the database, no

objective evidence was available to support the assertion of thorough checking. In addition, the
10

list did not identify a checker for some of the Microsoft® Excel files (approximately 79 files of

moderate to high complexity).

The audit team determined that the 24 technical assessments were not robust enough to

support data qualification. The audit team documented this issue as a potential deficiency. 

The observers agreed with the audit team’s findings in this area.

4.5.3 “Hydrogen-Induced Cracking of the Drip Shield,” ANL–EBS–MD–000006

The purpose of this report was to evaluate the potential for hydrogen-induced cracking of the

drip shield. The drip shield is designed to prevent or minimize water contact with the waste and

may also limit the potential for mechanical damage of the engineered system.

The technical specialists focused on the treatment and adequacy of the data in the report. Three

primary sources of information were used in the model for evaluation of hydrogen- induced

cracking of the drip shield: (i) the general corrosion rate of the material; (ii) a hydrogen

absorption factor; and (iii) a critical threshold. Two of the three data sources were assumptions. 

The critical hydrogen concentration during the regulatory time period was calculated as 750 mg/g

whereas the assumed threshold was 1000 mg/g. Therefore, hydrogen-induced cracking was not

included in the performance assessment calculations. Galvanic coupling from carbon steel drift

support materials contacting the titanium was also evaluated. Galvanic coupling was evaluated to

create approximately 100 to 400 mg/g hydrogen. The NRC observers asked whether combined

effects of hydrogen-induced cracking from both general corrosion and galvanic coupling were

considered. The author stated they had not been but that he believed when galvanic processes

were occurring, the general corrosion would be negligible.

The auditors evaluated the AP–2.14Q, “Review of Technical Products and Data,” materials

produced during the document review process and concluded that a thorough technical review

had been performed on the product.

The data from the DTN for general corrosion was corrected for the effect of silica precipitation

during the weight-loss measurements. The correction factor was not identified in the report

and, therefore, there was not transparency and traceability of the information used in the report,

without recourse to the originator. The audit team identified this issue as a potential deficiency.

The observers agreed with the audit team’s findings in this area.

4.5.4 “Rock Properties Qualification Report,” TDR–MGR–GE–000003, Revision 00

The purpose of the qualification report was to document the process and rationale through

which unqualified mechanical rock property data were consolidated into a single qualified

data set.

The audit team technical specialists focused on the criteria through which qualification was

achieved. The data being evaluated represented empirical measurements from up to six

different institutions. Some qualified data were available; therefore, a combination of the
11

corroboration method and the technical assessment method was used to determine whether

the unqualified data could be qualified. The auditors questioned why criteria for qualification, as

specified in AP–SIII.2Q, “Qualification of Unqualified Data and the Documentation of Rationale

for Accepted Data,” were not clearly presented in the report. The author said it was especially

difficult to specify qualification criteria at the planning stage when dealing with natural-system

rock-property data. Overall, because the technical assessment determined there was

agreement of a “substantially positive fashion,” the data were determined to be qualified. None

of the data was rejected in the qualification effort. Some subsets of the data had small sample

sizes and large variability in the data; therefore qualification was primarily subjective in nature. 

The issue of data qualification criteria resulted in a potential deficiency report (DR).

The observers agreed with the audit team’s findings in this area. 

4.5.5 “Development of Numerical Grids for UZ Flow and Transport Modeling,”

ANL–NBS–HS–000015, Revision 01G

The purpose of the grid report was to describe the methods used to develop numerical grids of

the unsaturated hydrogeologic system beneath Yucca Mountain. 

As part of the data checklist, the audit team performed a review of the various data sets,

notebooks, correspondence, and plans, via the TDMS, for the associated “Record

Roadmap”—DTN GS960808312231.004. The audit team used the various records to assess

the data qualification level used to prepare the grid report. The observers again noted that the

electronic data management system was a powerful tool for use in data tracking.

The observers agreed with the audit team’s findings in this area.

4.6 Additional NRC Observations

4.6.1 Technical Assessment and Data Qualification

Of the data qualification reports evaluated in this audit, technical assessment was strongly

relied as the method for qualification of unqualified data. NUREG–1298 “Qualification of

Existing Data for High-Level Nuclear Waste Repositories” is a Generic Technical Position that

identifies four alternative methods acceptable for qualifying existing data. One of these

methods is peer review, in accordance with NRC’s generic technical position elucidated in

NUREG–1297, “Peer Review for High-Level Nuclear Waste Repositories.” A peer review is

defined as “... an in-depth critique of assumptions, calculations, extrapolations, alternative

interpretations, methodology, and acceptance criteria employed, and of conclusions drawn in

the original work. Peer reviews confirm the adequacy of work.” Peer review is appropriate in

many circumstances, including when “Data adequacy is questionable—such as, data may not

have been collected in conformance with an established QA program.” In some cases it

appears that DOE’s technical assessment process is directed at trying to assess the adequacy

of highly uncertain data, with much less emphasis on evaluating the adequacy of the work that

generated the unqualified data. 
12

4.6.2 Use of Different Types of Information to Support a License Application

In some instances, data that were originally assigned a to-be-verified designation were

converted to an assumption and then inserted directly into an analysis. In addition, many data

sets for use in the performance assessment or supporting models are classified as accepted

data. Accepted data can be established fact (e.g., handbooks) or not-established-fact data. 

For not-established-fact data, a documented rationale for accepted data may include (per

AP–SIII.2Q) various options such as: (i) best available or only data source; (ii) historical

precedence; and (iii) inclusion in a refereed or peer-reviewed journal or publication. From an

NRC technical specialist perspective, adequate confidence is needed in data used to support a

license application regardless of the type of information (e.g., qualified, assumption, accepted). 

Use of not-established-fact accepted data and data-oriented assumptions would appear to be 

areas that should be the focus of an audit, to see if these data classes are being appropriately

handled. If only one unqualified data source is available, this would appear to be a reason to

be cautious of directly incorporating the data into the analysis as an accepted-data source. 

This type of audit would need to be heavily weighted toward technical specialists familiar with

the data being evaluated.

4.6.3 Technical Checking

In some cases there was good evidence of technical review of the audited materials. However,

in other cases there was sparse evidence of technical checking of calculations and data. For

example, one case of this was DTN MO0302SPATHDYN.001, associated with “Data

Qualification: Update and Revision of the Geochemical Thermodynamic Database. The file

name Data0.ymp contains 79 Excel spreadsheets as well as many other files. Some of the

spreadsheets contain fairly complicated calculations and many unit conversions. It is unlikely

that a technical checker would be able to effectively evaluate the accuracy of the information

contained in the spreadsheets during the document review. This is not an isolated example, as

there are many difficult DTNs contained in the data tracking system. The NRC technical

specialists consider it important to have a robust technical checking process, but it was unclear

during the course of this audit that robust technical checking would be feasible, given the

current management of information in sometimes large sets of files.

5.0 NRC STAFF FINDINGS

The audit team was to assess data quality used to support the LA by verifying implementation

of appropriate sections of the QARD regarding data and in reviewing data used in completed

AMRs. Within the areas evaluated, the audit team identified three potential deficiencies and

four potential quality observations. The observers agreed with the audit team’s findings. 

However, during the conduct of the audit, the observers initiated an AOI documenting concerns

about the audit team not identifying a potential significant condition adverse to quality regarding

ineffective corrective action on data problems over the last four years.

During the Post Audit Conference, the DOE Audit Team Leader concluded that the audit team

was unable to determine the quality of data used to support the LA. The Audit Team Leader

made this conclusion because a broad sample of completed AMRs in support of LA were in
13

draft form and were not available for review by the audit team and some of the completed

AMRs reviewed by the audit team were found to be not critical to the LA. 

The observers noted that certain audit team members were not familiar with the automated

Tracking Data Management System (TDMS), which was a critical tool in accomplishing the

objectives of this audit. This resulted in some delays and inefficiency in performing data

verification activities. The NRC observers concluded that the actual conduct of this audit may

have been more appropriately performed at a time when a larger sample of completed technical

products critical to the LA were available. Based on the conduct and results of the audit, the

observers concluded that the quality of data to support the LA is indeterminate. The observers

determined that the DOE audit of BSC to determine data quality was not fully effective as

described above.

During the post audit conference, the observer team leader expressed appreciation for the

cooperation and responsiveness the observation team received during its observation activities. 

In addition, the Observer Team Leader stated that the observers agreed with the audit team’s

findings and recommendations, as presented at the post audit conference. Also, during the

post audit conference, the Observation Team Leader discussed the following points:

• The overall assessment of the observation audit team is that the audit was indeterminate.

• The audit scope was limited because of the small number of technical products evaluated. 

• It is noted that this was the first audit regarding data verification in technical products

intended for licensing. As a result, the approach, level of detail, and time necessary to

access the data records may not have been fully anticipated.

• The audit was limited in its determination of the overall adequacy, appropriateness, and

suitability of the supporting data. 

• Based on the audit findings, potential issues remain concerning input, both programmatic

and technical. But because of the limited scope of the audit, the extent of these issues

remains indeterminate and additional evaluation activities, to be performed by BSC and

OQA, are necessary.

• An AOI was written by the observation audit team requesting additional information on

Corrective Action Report (CAR), BSC (B)–03–107. BSC indicated during the audit that they

will document this CAR as a result of a review of approximately 46 data-related deficiency

reports issued from 1998 to the present. Because of the importance of the matter, the NRC

requests a response to this audit inquiry within 45 days of the date March 27, 2003, of this

AOI.

6.0 NRC AUDIT OBSERVER INQUIRIES

6.1 NRC generated one inquiry as a result of observing audit OQAP-BSC-03-05 (see Exhibit 1). 

6.2 The following three inquiries remain open.

6.2.1 BQAP-BSC-03-02 No. 1

DOE/BSC used qualified, verification level 2 (QL-2), and unqualified data as inputs for modeling

and analysis purposes, for low risk significant applications supporting site recommendation. 
14

Given that unqualified DTNs are being used in the development of TPOs, how will DOE/BSC

assure that only qualified and verified data and software are used for high risk significant

applications supporting license application?

6.2.2 BQAP-BSC-03-02 No. 2

The audit team identified an instance where, apparently because of time and schedule pressure, a

BSC qualified checker and a BSC Quality Engineering Representative approved the Thermal

Testing Measurement Report (U0220) without reviewing all of the associated data. How will DOE

and BSC management create an environment to assure that personnel performing checking and

quality assurance assignments will be afforded adequate time to perform their assigned tasks as

time and schedule become even more important leading up to license application? What metric

will be developed and used to assure that quality activities are not influenced by cost and

schedule?

6.2.3 LLNL-ARC-02-07 No. 1

This inquiry referenced welded metal plates and services supplied by Framatome. The NRC has

received the response and is actively evaluating this response.