ENCLOSURE

NRC OBSERVATION AUDIT REPORT NO. OAR–03–04, “OBSERVATION AUDIT OF OFFICE

OF QUALITY ASSURANCE AUDIT OQAP–BSC–03–14 OF BECHTEL SAIC COMPANY, LLC” 

 /RA/ 10/14/03 /RA/ 10/15/03

Ted Carter David Esh 

Projects and Engineering Section Performance Assessment Section 

High-Level Waste Branch Environmental and Performance

Division of Waste Management Assessment Branch

Division of Waste Management

Reviewed and Approved by:

 /RA/ 10/17/03

N. King Stablein, Chief

Projects and Engineering Section

High-Level Waste Branch

Division of Waste Management
2

1.0 INTRODUCTION

Staff from the U.S. Nuclear Regulatory Commission (NRC) Division of Waste Management,

observed the U.S. Department of Energy (DOE), Office of Quality Assurance (OQA), audit

OQAP–BSC–03–14, on September 8–19, 2003, at the Bechtel SAIC Company, LLC (BSC)

facility in Las Vegas, Nevada. The two objectives of this performance-based audit were to

assess the technical product input integrity relating to selected model and analysis reports

associated with the Yucca Mountain Project License Application (LA), and the BSC

implementation of the Office of Civilian Radioactive Waste Management Quality Assurance

Requirements and Description (QARD), DOE/RW–0333P, Revision 12, regarding control of

data. This audit followed up OQA data verification audit OQAP–BSC–03–05 (conducted in

March 2003). In addition, data-related deficiencies not previously addressed or included in

Corrective Action Report (CAR) BSC(B)–03–C–107 were reviewed and evaluated for

effectiveness of the actions taken. The NRC observers (observers) assessed the effectiveness

of the audit team and the audit process in achieving the audit objectives.

2.0 MANAGEMENT SUMMARY

The audit team goal was to assess the quality of data used to support the LA by verifying

implementation of the data-related portions of the QARD and review of data used in completed

technical products. Twelve technical products, representing approximately 25 percent of the

products available at the time of the audit, were evaluated for:

• Adequacy (of the data for the intended use) 

• Implementation (of data management and control procedures during technical product

development)

• Effectiveness (of the implementation of critical process steps)

The technical product evaluations formed the basis for conclusions regarding the adequacy,

implementation, and effectiveness of the critical process steps. The audit team identified

11 potential Deficiency Reports during the audit and concluded:

• Effectiveness was determined unsatisfactory for 3 of the 12 technical products audited.

• Implementation was determined unsatisfactory for 1 of the 12 technical

products audited.

• The Technical Product Input and Selection critical process step was determined

unsatisfactory for adequacy, implementation, and effectiveness.

• The Data Input and Development critical process step was determined unsatisfactory for

implementation.

• The Data Control and Data Management critical process step was determined

unsatisfactory for implementation.

Overall, the audit team determined that control of technical product input integrity was:

• Satisfactory in regard to adequacy

• Unsatisfactory in regard to implementation

• Satisfactory in regard to effectiveness (the audit team determined the unsatisfactory

finding regarding implementation did not significantly impact the overall effectiveness)
3

The observers determined BSC’s audit was effective in determining technical product input

integrity. The observers agreed with the audit team conclusions, findings, and process

improvement recommendations.

The observers reviewed the qualifications of the Audit Team Leader and the auditors and

determined all were qualified and all were independent of the areas being audited. The lessons

learned from the previous audit of the same focus were used effectively to plan and conduct

this audit.

The observers noted four areas of concern in the data management and control processes:

• Subjective and vague data definitions

• Unclear descriptors of data value derivation and selection

• Missing Records Road Maps (a repeat issue)

• Inappropriate use of technical information resulting in unqualified data being introduced

into the data management system

The fourth item was the subject of an Audit Observation Inquiry (see Section 5.2). 

3.0 AUDIT PARTICIPANTS

DOE Audit Team Members

James Voigt, Navarro Quality Services (NQS), Audit Team Leader

James Harper, NQS, Audit Team Leader in Training

Gerard Heaney, BSC, Auditor

Roxie VanDillen, BSC, Auditor

Christian Palay, NQS, Auditor

Harvey Dove, NQS, Technical Specialist

Raymond Keeler, University of Nevada, Las Vegas, Technical Specialist

Keith Kersch, Shaw Group, Technical Specialist

Observers

Ted Carter, NRC, Team Leader

David Esh, NRC, Technical Specialist

Ernesto Quiñones, NRC, Technical Specialist (Intern)

Randy Fedors, Center for Nuclear Waste Regulatory Analyses (CNWRA), Technical Specialist

Rodney Weber, CNWRA, QA Specialist

Jon White, DOE, Repository Engineering and Design Division

Englebrecht von Tiesenhausen, Clark County, Nevada

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION

The audit was conducted in accordance with AP–18.3Q, Internal Audit Program, and

AP–16.1Q, Management of Conditions Adverse to Quality. The audit team used the Quality

Assurance Requirements and Description (QARD) and applicable BSC implementing

procedures to generate the audit checklist. The observers followed the NRC Manual Chapter

2410, Conduct of Observation Audits, July 12, 2000, while observing the audit.
4

4.1 Scope of the Audit

The audit focused on technical product input integrity relating to selected Analysis and Model

Reports (AMRs) associated with the LA. The audit team evaluated the adequacy of technical

product input based on established measurement criteria and the processes associated with

technical product input. In addition, the audit team followed up those issues identified

as observations and deficiencies subsequent to CAR BSC(B)–03–C–107, which is associated

with audit OQAP–BSC–03–05.

The audit included evaluating the following procedures regarding data:

• AP–2.14Q, Review of Technical Products and Data

• AP–3.15Q, Managing Technical Product Inputs

• AP–3.11Q, Technical Reports

• AP–SIII.2Q, Unqualified Data and Documentation of Rationale for Accepted Data

• AP–SIII.3Q, Submittal and Incorporation of Data to the Technical Data

Management System

• AP–SIII.9Q, Scientific Analysis

• AP–SIII.10Q, Models

This audit focused on 12 AMRs selected for their status of recently issued or draft/nearing

completion in review:

• Saturated Zone Transport Method and Component Integration, MDL–NBS–HS–000010

(In review)

• Saturated Zone Flow and Transport Model Abstraction, MDL–NBS–HS–000021 (Issued)

• Seepage Model for PA Including Drift Collapse, MDL–NBS–HS–000002 (Issued)

• Number of Waste Packages Hit by Igneous Intrusion, ANL–MGR–GS–000003 (In

review)

• CSNF Waste Form Degradation: Summary Abstraction, ANL–EBS–MD–000015

(Issued)

• HLW Glass Degradation, ANL–EBS–MD–000016 (Issued)

• Isotropic Model Report for Commercial SNF Burnup Credit, MDL–DSU–NU–000001

(Issued)

• Stress Corrosion Cracking of Dripshield, Waste Package Outer Barrier and Stainless

Steel Structural Material, ANL–EBS–MD–000005 (Issued)

• Characterize Eruptive Processes at Yucca Mountain, Nevada, ANL–MGR–GS–000002

(In review)

• In-Drift Precipitates/Salts Model, ANL–EBS–MD–000045 (Draft)

• Development of Earthquake Ground Motion Input, MDL–MGR–GS–000003 (In review)

• Seismic Consequence Abstraction, MDL–WIS–PA–000003 (Issued)

4.2 Conduct and Timing of the Audit

This was the second data-related audit, and the approach, overall planning, access to data

records, and overall organization was significantly improved relative to the previous

data-related audit.
5

The observers determined the timing of the audit was appropriate, particularly because a

sufficient population of technical products was available for review. The number of products

available during this audit, in addition to the products reviewed during the previous data-related

audit, was considered sufficient to determine the status of technical product input integrity.

The audit team and observers caucused at the end of each day to discuss the audit status and

any new and developing issues. The audit team met with BSC management, as appropriate,

each morning to discuss the current audit status and potential issues.

At the beginning of the audit, the audit team gave a presentation to the observers to explain the

Technical Data Management System (TDMS) and the linkages to other databases in the Yucca

Mountain Program. The presentation enhanced the observers’ understanding of the DOE data

management system. Several areas were identified of potential improvement to the data

management and control system.

4.3 Audit Team Qualification and Independence

The observers reviewed the qualifications for the Audit Team Leader and the auditors and

determined all were qualified and independent of the areas reviewed. Although the technical

products evaluated were diverse in subject matter, the technical specialists’ qualifications were

appropriate and adequate for the scope of this audit.

4.4 Examination of QA Elements

4.4.1 Data-Related Definitions

The following data-related definitions were used during the audit and appear throughout

this report. When used in the context of these definitions, the terms are underlined.

• Data: Factual information obtained in the field or the laboratory from investigation

activities such as sample collection, physical measurements, and testing and analyses.

• Direct Input: Input used to develop and produce product output subject to the QARD.

Direct Input must be qualified data.

• Technical Information: Information that does not meet the definition of Data and is used

for Direct Input. It may include, but is not limited to, information from controlled Yucca

Mountain Project reports and design products, Established Fact, or documented and

substantiated information, including electronic databases. Technical Information may or

may not be qualified.

• Reference Only: Technical product input not directly used to address safety and waste

isolation issues. Reference Only information needs no qualification.

• Model: Representation of a system, process, or phenomenon, along with any

hypotheses required, to describe the process or system or explain the phenomenon,

often mathematically.
6

• Established Fact: Outside data that has been reviewed and accepted by project staff

and is currently included as a subset of Technical Information. Outside data refers to

data collected outside the Yucca Mountain Program, though the term also has been

interpreted as data collected from areas outside of Yucca Mountain proper. Audit team

members noted definitions should not rely on geographic boundaries.

The audit team noted data definitions and quality requirements from the DOE procedures were

inconsistent between NUREG–1298 and the QARD. The problem with using Technical

Information and Reference Only as Data Input was that unqualified data could be brought into a

technical product and become considered as qualified data output.

Technical Information is any Direct Input that does not meet the definition of Data. This term

was used by some authors to interpret the distinction between Technical Information and Data

as the level of processing required when making measurements. For example, porosity can be

directly measured (e.g., weight of wet and dry cores) or indirectly measured (e.g., interpreted

from geophysics). Thus, the latter method would mistakenly lead to categorization of data as

Technical Information rather than Data that are Direct Input. In addition, the example in the

definition of Technical Information of “documented and substantiated” leaves a wide degree of

latitude for introducing Data from the general literature (e.g., journal articles and conference

proceedings). Often, sufficient details are not presented in articles and proceedings to

determine methodology and extent of calibration used to obtain Data from measurements.

The definition of Reference Only information leads to vagueness and ambiguity. The question

of how the data are used in the document is central to the categorization of data

input/information as Reference Only. Information that is “not directly used to address safety

and waste isolation issues” has been interpreted as not directly used in a model. Because the

interpretation that Reference Only required the information not be used in a Model, the

definition of Model also becomes important. Clear criteria are needed to define what

constitutes a Model. In addition, a problem arises because Reference Only information allows

the use of unqualified data to exclude model components.

Numerous issues were identified during the course of the audit regarding the appropriate and

consistent use of technical product input. The audit team identified that unqualified data were

introduced into technical documents and models supporting LA via the misuse of two types of

information categories: Technical Information and Reference Only. An Audit Observer Inquiry

(AOI) was generated that requested clarification of the path forward for data definitions and

data qualification status (see Section 5.2).

In most instances, the AMR authors had requested assistance from management to determine

the categorization of information as Technical Information and not Data. The confusion lay in

the definitions, their interpretations by authors, and the training guidance. The fundamental

issue concerns the definition and categorization of information. AP–3.15Q contains a checklist

to assist authors in the categorization of information data input as Direct Input, Technical

Information, and Reference Only.
7

4.4.2 Critical Process Steps

Four critical process steps were evaluated by the audit team during their investigations of

technical products: 

• Data Input and Development

• Technical Product Input and Selection

• Analysis and Documentation

• Data Control and Data Management

The four critical process steps were evaluated for

• Adequacy (of the data for the intended use)

• Implementation (of data management and control procedures during technical

product development)

• Effectiveness (of the implementation of critical process steps)

Based on their evaluation of the 12 technical products, the audit team determined

• The Technical Product Input and Selection critical process step was determined

unsatisfactory for adequacy, implementation, and effectiveness.

• The Data Input and Development critical process step was determined unsatisfactory for

implementation.

• The Data Control and Data Management critical process step was determined

unsatisfactory for implementation.

Data Input and Development

The audit team found a deficiency in the Record Road Maps for qualified data. This issue was

previously identified in condition reports BSC(B)–03–D–152, BSC(B)–03–D–227, and

BSC(B)–03–D–242, all issued after CAR BSC(B)–03–C–107 had been closed. This

continuation of identified deficiencies for Records Road Maps in individual areas led to the

suggestion by the audit team that a global solution was warranted to address the problem.

A Records Road Map is required for all qualified data used in a document for LA. AP–17.1Q

requires a roadmap be completed within 1 year, whereas 60 days is specified for all

documentation supporting submittal of qualified data.

Technical Product Input Selection

The auditors reviewed documents for QARD implementation; input/output transparency and

traceability, overall technical adequacy, checking, and review adequacy related to data inputs. 

The audit team identified an issue with the use of the Technical Information designation for

Direct Input; a situation that could lead to unqualified data being used as Direct Input. Three

documents extensively used the Technical Information category for Direct Input after requesting

assistance and clarification from management. 
8

Analysis and Documentation

The audit team reviewed documents for QARD implementation, data identification and control,

data status, reference agreement/traceability with the various document information systems

and technical databases, and the appropriate and consistent use of data. The audit team

identified no areas of deficiency in the analysis and documentation process step.

Data Control and Management of Data

This category was intended to reflect the overall use, control, and documentation of data in

Yucca Mountain Program reports. The audit team determined that Technical Information used

in three of the reviewed documents, and in numerous Records Road Maps were incomplete. 

The audit team found instances of incomplete Records Road Maps associated with data in the

audited documents. In addition, the BSC data management team recently developed a much

larger list of inadequate or missing Records Road Maps, though the list was not yet considered

complete.

The observers agreed with the audit team findings in this area, however, the classifications of

adequacy, implementation, and effectiveness had not been defined clearly or procedurally. 

These three classifications also were applied to the technical products, but the applicability of

the same definitions to technical product evaluations also was not clear. Questions from the

audit team members during process step evaluations indicated the need for definitions in

relation to assessing the acceptability of critical process steps. 

4.4.3 Examination of Corrective Actions

During audit activities, the audit team referred to previous observations and deficiency reports. 

Repeated deficiencies were noted in the potential Deficiency Reports resulting from this audit. 

In addition, CAR BSC(B)–03–B–107, Repeating Deficiencies in the Management and Utilization

of Data, was issued in May 2003 because of ongoing data-related deficiencies. This CAR was

the subject of NRC observer inquiry BQAP–BSC–03–02 No. 1. The response to the CAR had

not been completed at the time of the audit; therefore, the audit team chose to defer decisions

regarding any additional corrective actions until a completed response is issued. An outline of

these repeated conditions, including a historical perspective, was included in the root cause

determination for CAR BSC(B)–03–B–107. 

No potential deficiencies were noted in this area.

The observers agreed with the audit team findings in this area.

4.4.4 Evaluation of Procedural Sufficiency

The observers noted the scope of the audit did not include a direct assessment of procedures. 

The audit team’s evaluation of technical products, however, required the products be evaluated

against current procedures or those in-place at the time of technical product preparation,

review, or both.
9

The audit team findings indicated recurring issues related to technical product input and the

proper categorization and use of inputs. The complexity of procedural requirements and the

continual changes to procedures to address and clarify requirements contributed to these

discrepancies. The observers noted the numerous revisions, some of which were significant,

and their frequency appeared to have created an environment of uncertainty and confusion in

some cases. The root cause determination for CAR BSC(B)–03–B–107 discusses procedure

effectiveness and other issues related to data definition. 

No deficiencies were identified related to the flow-down of QARD requirements to

BSC procedures.

The observers agreed with the audit team findings in this area.

4.5 Examination of Technical Activities

Criteria for determining the overall adequacy and fitness of technical product input included:

• Accuracy of input

• Consistency of input use, as defined in supporting documents

• Adequacy of documentation to support inputs

• Traceability of technical product inputs

• Implementation of corresponding QARD requirements and satisfactory completion of

critical process steps (see Section 4.4.2)

In a bottom-up approach, the AMRs were first reviewed in regard to the criteria and evaluated

for adequacy, implementation, and effectiveness. Then, considering all the AMR evaluations,

the four critical process steps were likewise evaluated for adequacy, implementation,

and effectiveness.

Overall, the audit team technical specialists were effective in their audit. The observers believe

that additional preparation by the audit team prior to the beginning of the audit could have

increased the efficiency and a larger number of datasets could have been reviewed during the

audit. 

The following sections describe the audit team criteria in more detail and include comments

from the audit team during the AMR evaluations together with comments related to each AMR.

4.5.1 Measurement Criteria

This section describes the criteria and gives examples of the audit team efforts in evaluating

AMRs using the criteria.
10

Accuracy

The audit plan described this criterion as accuracy and appropriateness of input (e.g., inputs

used as represented; direct inputs qualified as required; inputs supporting assumptions are not

direct inputs). The examples given in the parentheses, however, referred only to accuracy in

following procedures for the use of qualified and unqualified data and in accurately transcribing

Data Tracking Numbers (DTN). Appropriateness of input appears to fall under the

second criterion. 

In following input data for documents, the audit team consistently checked how the Data was

used in the document. The issues pertaining to the misuse of Technical Information and

Reference Only were identified by the audit team as inconsistent with the procedures. The

authors, uncertain about interpreting the procedures, followed the advice of the procedure

training sessions or requested assistance from appropriate management and data

management personnel. The misuse of the Technical Information category in three of the

reviewed AMRs was considered the most important finding of the audit (Potential Audit Findings

No.1 and No. 4, see Section 4.6).

In addition, typographical errors and inconsistencies between the AMRs and the databases

were identified. Some inconsistencies were caused by expected delays in updating the

databases and the timing of document completions. For documents still in the review process,

corrections were requested, and the issue was classified as closed during the audit. For

completed documents, Potential Audit Finding No. 9 was initiated (see Section 4.6).

Consistency

The audit plan defined this criterion as “consistency of input use, as defined in supporting

documents.” This definition appears to allude to appropriateness of input data. Appropriate

use of input data refers to a subjective decision that the input data are relevant to the modeling

or analysis scale, conditions, and processes and considers the model and the study objectives. 

Appropriate use of data requires a justification or basis for the use of technical product input in

a model. The implementing procedures governing the development of models (AP–SIII.10Q)

and the development of analyses (AP–SIII.9Q) require the supporting documentation to provide

a justification for the technical product input selected as well as those omitted.

Included in the audit checklist was an item to evaluate the consistency between

information/references cited in the audited AMR and other project documents. This type of

evaluation requires cross-checking multiple documents, which is time consuming. Because of

the state of development of many of the other project documents, the observers agreed with the

audit team that consistent use of common sets of information could not be evaluated

conclusively during the course of this audit.

The audit team noted that appropriate use of data was a topic that overlapped with the

objectives of the models audit scheduled for October 2003. Evaluating the appropriateness of

input data requires thorough reading and understanding of the document and produces a

subjective question on which technically competent researchers could disagree. The observers

were hampered in their activities because the AMRs were not available to them prior to the
11

audit, and the observers did not have a great deal of time to evaluate these reports during the

audit. The audit team also had little time to evaluate the appropriateness of input data because

of the aggressive schedule (four AMRs per audit subteam). Also, because of the specialized

knowledge required in specific technical areas, pertinent datasets that may have been omitted

from a model or analyses were difficult to identify by the audit team. Appropriateness of input

should be evaluated in more detail during the models audit scheduled for October 2003.

For the few instances checked by the audit team, the supporting documentation did provide

adequate justification for the input selected. The justifications were usually clear and well

written. One question regarding appropriate use of input was identified in

ANL–EBS–MD–000005, however; the issue was passed on to the forthcoming models audit

because of the subjective technical judgment needed to evaluate the issue. In regard to report

ANL–EBS–MD–000005, externally obtained coefficients for an empirical relation for stress

corrosion cracking of steel were applied to Alloy 22 without sufficient justification presented in

the document.

Transparency

In AP–SIII.9Q, REV 01, transparency is defined as the “attribute of producing documents that

are sufficiently detailed as to purpose, method, assumptions, inputs, conclusions, references,

and units, such that a person technically qualified in the subject can understand the documents

and ensure their adequacy without recourse to the originator.” Clear descriptions of data

selection refer to defensible and transparent documentation of bases supporting the intended

use of the data.

Technical product inputs are required to be transparent with respect to their source, technical

content, and qualification status. The audit team evaluated subsets of the input datasets

supporting each product. In general, the technical product inputs were transparent with respect

to their source and qualification status. Processing input data to estimate parameters in the

document generally was described clearly. A few instances of inadequate transparency were

identified in documents still being reviewed. Authors readily addressed the issues and made

the changes requested by the auditors during the audit. Issues addressed during the audit are

not carried forth as potential findings, but rather are tracked in the auditor checklists.

The observers noted, at the time of the audit, some products were supported by few fully

qualified datasets. For example, in a sample of four of the AMRs evaluated, 292 references

were listed on the Document Reference Input System (DIRS):

• Four references were fully qualified data sets (Direct Input)

• Seventeen references were to be verified data sets (Direct Input)

Fewer than 20 percent of the Direct Input for these AMRs was qualified at the time of the audit.

The balance of the references for these AMRs was:

• Qualified software (2 references)

• Technical Information (7 references)

• Reference Only (262 references)
12

Traceability

In AP–SIII.9Q, REV 01, traceability is defined as the “ability to trace the history, application, or

location of an item, data, or sample using recorded documentation.”

The audit subteams identified four qualified datasets with incomplete or missing Records Road

Maps. Data qualified, acquired, or developed after June 30, 1999, and not technical product

output, must have a Records Road Map developed following AP–3.15Q, REV 04. Because this

issue had been previously identified, the audit team discussed the need to address the global

problem (Potential Audit Finding No.10, see Section 4.6).

Implementation

This criterion pertains to the compliance with corresponding QARD requirements and

satisfactory completion of critical process steps (described in Section 4.4.2).

4.5.2 Examination of Technical Products

An important finding of the audit team was that technical product input was being categorized

as Technical Information or Reference Only when the audit team believed it should be

considered Direct Input, which must be qualified. The subjective nature of these terms and

their definitions are discussed in detail in Section 4.4.1.

No issues were identified for the following documents: 

• Isotopic Model Report for Commercial SNF Burnup Credit, REV 00,

MDL–DSU–NU–000001, REV 00, Issued 6/30/03 

• In-Drift Precipitates/Salts Model, ANL–EBS–MD–000045, REV 01, Draft

• Development of Earthquake Ground Motion Input, MDL–MGR–GS–000003, REV 00,

In Review

• Seismic Consequence Abstraction, MDL–WIS–PA–000003, REV 00, Issued 8/14/03

Short descriptions and linkage to the Potential Audit Findings in Section 4.6 for the remaining

eight documents are included next.

Saturated Zone Transport Method and Component Integration, MDL–NBS–HS–000010,

REV 01C, In Review.

The audit team noted the use of Technical Information as input to the model. Sorption

coefficients, surface complexation binding constants, and silica surface areas were categorized

as Technical Information and used as Direct Input to the Model. The audit team position was

these input parameters meet the definition of Data in AP–3.15Q, and therefore, should not be

categorized as Technical Information (Potential Audit Finding No. 1, see Section 4.6). 
13

An instance of lack of transparency was identified and corrected during the audit. An

incomplete description was contained in the AMR of the process for selecting a portion of an

input dataset. Mineralogical data from below the water table from different boreholes were

selected for further processing without providing the water table position needed to distinguish

records in the files that were from the unsaturated and saturated zones.

Saturated Zone Flow and Transport Model Abstraction, MDL–NBS–HS–000021, REV 00,

Issued 8/1/03.

The audit team noted the use of Technical Information as input to the model. Porosity, effective

porosity, and flowing interval porosity were categorized as Technical Information and used as

Direct Input to the Model. The audit team position was these input parameters meet the

definition of Data in AP–3.15Q.

Seepage Model for PA Including Drift Collapse, MDL–NBS–HS–000002, REV 02,

Issued 7/3/03.

Two issues were identified, but closed during the audit. In the first issue, qualified data were

introduced as Reference Only input. Qualified air permeability data were used to justify the

range of permeability to use in developing a response surface that was technical product

output. The range used in the analysis was larger than the average values of measured air

permeability. The justification for the range was said to be determined in consultation with the

downstream user of the response surface. The audit team did not have enough time to track

down the adequacy of the justification for the range in the downstream document. The concern

was this range could be interpreted to include a range representative of Yucca Mountain. 

For the second issue, a superceded dataset was used as input. The submission of the new

data occurred 2 weeks prior to completion of the AMR, a time period considered too close to

completion to warrant corrective action. 

Number of Waste Packages Hit by Igneous Intrusion, ANL–MGR–GS–000003, REV 00H,

In Review.

A dataset used in the AMR was superceded and no Impact Review Action Notice (IRAN) was

initiated. Because this document was in review, the problem was corrected during the audit. It

was also noted that a recent shift of the IRAN system to the data management group and

eventual link to the other databases will correct the problem of inconsistent initiation of IRANs. 

The IRAN system was intended to notify all managers or authors of reports and datasets that a

superceded dataset is in place. 

An exemplary practice was identified by the audit team for the summary description of the

contents for each input dataset. The description helped to establish the consistency or

appropriateness of the data for use in the document. This practice also helped in traceability

for locating specific portions of the input datasets used in the document by knowing in advance

the contents of the input dataset.
14

CSNF Waste Form Degradation: Summary Abstraction, ANL–EBS–MD–000015, REV 01,

Issued 7/2/03.

The audit team identified a missing Records Road Map (Potential Audit Finding No. 10, see

Section 4.6) and a disagreement between the value listed in the document as technical product

output and the value in the dataset submitted to the TDMS (Potential Audit Finding No. 9, see

Section 4.6).

An issue closed during the audit pertained to an introduced parameter value for the upper

bound of a probability distribution for specific surface area. This parameter distribution is a

Direct Input to the calculation of the release rates from spent nuclear fuel. The supporting

dataset was not listed as document input; thus, the dataset was not listed on the technical data

information sheet of the information system. 

HLW Glass Degradation, ANL–EBS–MD–000016, REV 01, Issued 8/22/03.

Three issues were identified by the audit team. The first issue pertained to typographical errors

in a completed document. The errors were in the cross-referencing of sections and in units

associated with parameter values (Potential Audit Finding No. 9, see Section 4.6). The second

issue related to values of a cracking factor for high-level waste glass from the Reference

Information Base (RIB) database, a project-accepted database of parameter values lying

outside the range of cracking factors developed in the document. The RIB database was not

searched by the author of the document. Further analysis revealed the cracking factor in the

RIB database originated from a previous revision of the document, hence; this was identified as

a problem with updating the RIB database (Potential Audit Finding No. 3, see Section 4.6).

The third issue identified by the audit team pertained to unqualified, developed input data used

in the document without the required justification. The introduced parameter used in the

development was the specific surface area of degrading high-level waste glass, which was

introduced as Technical Information. The specific surface area is directly used in the

calculation of the release rates of radionuclides from high-level waste glass (Potential Audit

Finding No. 6, see Section 4.6).

SCC of DS, WP Outer Barrier and SS Structural Material, ANL–EBS–MD–000005, REV 01,

Issued 7/16/03.

Eight issues related to six different Potential Audit Findings were identified by the audit team for

this document. Three issues were grouped as part of Potential Audit Finding No. 9 on errors

and omissions. These three issues include (1) typographical errors in DTNs, (2) a reference

listed in the reference section of the document, but not listed in the DIRS or the body of the text,

and (3) an incorrect Technical Information Center (TIC) number for a reference included in the

DIRS.
15

The forth issue pertained to a previously qualified data item being changed to Technical

Information, which has no qualification status. This change could lead to a problem in

traceability to the qualification status. The problem was related to the misuse of the Technical

Information category for input (Potential Audit Finding No. 1, see Section 4.6).

The fifth issue was that two input datasets were listed also as output datasets. This was an

implementation issue and a problem because of the circular reference created in the database

(Potential Audit Finding No. 5, see Section 4.6). 

The sixth issue was that Reference Only information was used as Direct Input to a model or

analysis and then incorporated into an output dataset. Thus, Reference Only data were

changed to qualified status. A value for incipient crack size from an outside source was

incorporated directly into an output dataset that was used by the waste package degradation

model (Potential Audit Finding No. 2, see Section 4.6).

The seventh issue identified the problem with the RIB database. The RIB database originally

intended to contain project-accepted values of parameters, however; the RIB database had not

been updated as new analyses were completed (Potential Audit Finding No. 4, see Section 4.6)

The eighth issue was concerned with input sources for the effects of lead on stress corrosion

cracking categorized as Reference Only, but the audit team believed the input was related to

waste isolation. Reference Only is for technical products not directly used to address safety

and waste isolation. The data source supported the conclusion there would be no stress

corrosion cracking at low pH in the presence of lead, thus removing a model component from

consideration. The essence of this issue was how direct does input need to be considered

Direct Input (Potential Audit Finding No. 7, see Section 4.6).

A exemplary practice was identified that provided an excellent example of data traceability. The

AMR had extracted information from a dataset that contained files created with a spreadsheet

program. In addition to providing the DTN number, the report also referenced the filename, 

spreadsheet name, and cell numbers within the DTN that contained the information used in the

report. 

Characterize Eruptive Processes at Yucca Mountain, Nevada, ANL–MGR–GS–00002,

REV 01E, In Review.

Four instances of personal communication were used to introduce data into the document

without the data going through a qualification process (Potential Audit Finding No. 8, see

Section 4.6). 

The audit team also identified two issues that were corrected during the audit, which was

permitted because the document was in the review process. The first issue was the lack of

transparency in the processing of data from a DTN. A subset of 12 parameters of 41 total in

the DTN was selected for use as Direct Input. The second issue was an incorrect DIRS citation

in the report.
16

4.5.3 Adequacy, Implementation, and Effectiveness of Technical Products

The AMRs were determined satisfactory or unsatisfactory for each of the categories of

adequacy, implementation, and effectiveness as described in Section 4.4.2. 

Three documents, MDL–NBS–HS–000010, MDL–NBS–HS–000021, and

ANL–EBS–MD–000016, had unsatisfactory determinations in the effectiveness category

because of the use of Technical Information for technical product input that instead fit the

definition of Data. Because the authors had followed procedural requirements, the

unsatisfactory result was put in the effectiveness category to reflect a management and

guidance (i.e., training) problem. 

One document, ANL–EBS–MD–000005, was graded unsatisfactory for implementation based

on eight issues identified during the audit. These eight issues were mapped to six different

Potential Audit Findings.

4.6 Potential Audit Findings

As a result of the audit, the audit team identified 11 potential Deficiency Reports. 

Potential Audit Finding No. 1: Technical Information

The audit team believed that Direct Input were incorrectly being categorized as Technical

Information in three AMRs. The team noted porosity data used in MDL–NBS–HS–000021 met

the definition for Data (collected) as defined in the BSC procedures. In accordance with BSC

procedures, this is incorrect as a reference for these data. Similarly, Direct Input from

ANL–NBS–HS–000010, including sorption coefficients, surface complexion binding constants,

and silica sample surface areas, also meet the BSC procedure definition for Data (collected). 

Also, five qualified datasets were introduced into ANL–EBS–MD–000005 as Technical

Information, thus potentially confounding traceability of their qualification status. The DIRS lists

these items as Technical Information. The audit team also noted this is the same issue

described in open Deficiency Report BSC(O)–03–D–214. 

Potential Audit Finding No. 2: Technical Information Carried Through Product Output

The audit team noted ANL–EBS–MD–00005 used Technical Information cited in the input as

output for the AMR. The information was originally qualified but is now listed in the TDMS as

Technical Information Product Output. The output DTN for this AMR passes on a

recommended value to another AMR. This value was from an unqualified source. Use of the

term Technical Information Product Output, however, inferred a qualified data status and,

hence, bypassed the qualification process. 
17

Potential Audit Finding No. 3: RIB Not Current

The audit team noted a cracking parameter for high-level waste glass obtained from an outside

source was referenced in ANL–EBS–MD–000016. This parameter appears to meet the

definition of Data. A different value for the same parameter was available in the RIB, though

the RIB database was not searched during preparation of this AMR. Further analysis by the

audit team led to the conclusion this issue appeared to be a failure to update the RIB database

with an available updated value. 

Potential Audit Finding No. 4: Consequences of Technical Information Definition Change

Three instances of developed or qualified data were supported only by Technical Information. 

Further discussions with BSC data management staff led to the conclusion this issue was an

unintended consequence of a change in the database software system. The software change

was necessitated by the categories of data that were now considered by definition to be

Technical Information. 

Potential Audit Finding No. 5: Circular Reference

The audit team noted two DTNs cited in the DIRS for ANL–EBS–MD–000005 listed the DTNs

as input. The DTN system lists them as product output for the same AMR. This circular

reference was an implementation issue.

Potential Audit Finding No. 6: Unqualified Developed Data Used Without Justification

The audit team noted that unqualified, developed data were used in ANL–EBS–MD–000016

without justification. An unqualified data item was found in Table 4.1-2 of this AMR without

providing justification in Section 4.1 of the report.

Potential Audit Finding No. 7: Definition of Direct Input

The audit team noted in ANL–EBS–MD–000005 a discussion indicates that lead (Pb) has no

effect on stress corrosion cracking for waste isolation. Two datasets used to support this

discussion were listed as Reference Only. Reference Only is intended for technical product

input that are not directly used to address safety and waste isolation issues. If the outcome of

the discussion would affect the report results (i.e., Direct Input, the supporting data would

require qualification), further consideration regarding “how direct is direct” was needed.

Potential Audit Finding No. 8: Personal Communications Not Documented in Document

Input Reference System

The audit team noted in ANL–MGR–GS–000002 that references were made to personal

communications of information. These communications contained data values transmitted 

without documentation in the DIRS, along with the associated qualification status. 
18

Potential Audit Finding No. 9: Errors in Completed Documents 

Three products were found to contain errors. One example: the audit team noted several

typographical errors related to parameter units and table references in ANL–EBS–MD–000016. 

One value listed in the conclusions section in ANL–EBS–MD–000015 did not match the value in

the output dataset located in the TDMS. Also, for ANL–EBS–MD–00005, the DIRS listed an

incorrect TIC catalog number for one of the references cited in the report.

Potential Audit Finding No. 10: Records Road Maps

The audit team found many records packages submitted without Records Road Maps. For

example, four qualified and final DTNs did not have Records Road Maps for

ANL–EBS–MD–000015. Others have been found to be incomplete. This issue was previously

identified and addressed in three open Condition Reports. 

Potential Audit Finding No. 11: (Closed During the Audit) IRAN

The audit team noted that ANL–MGR–GS–00003 contained a data input superceded by Model

MO0307SPADKBSM.000 without generating the required impact review. Two similar

deficiencies were previously reported and closed. 

5.0 NRC STAFF FINDINGS

5.1 NRC OBSERVATION SUMMARY

The NRC observers determined the audit team was effective regarding the assessment of

technical product input integrity and in determining the level of compliance of BSC activities. 

The NRC observers agreed with the audit team conclusions, findings, and process

improvement recommendations within the limited scope of the audit. The observers determined

the audit team members were qualified and all were independent of the areas being audited. 

Specific concerns of the observers included:

• The audit team identified discrepancies related to missing Records Road Maps, a

previously identified issue. Because of the importance of the Records Road Maps in the

data qualification process, the repetitive nature of this finding is of concern to the NRC,

and it warrants additional management attention.

• The inappropriate use of Technical Information to introduce unqualified data into the

data management system was a significant area of concern identified by the audit team. 

This issue was the subject of an AOI provided to the audit team.

• Data definitions were subjective and vague. These definitions need to be consistent,

clear, and must unambiguously reflect the requirements of NUREG–1298 and

Supplement III of the QARD.
19

• Transparency issues were noted and must be addressed. AMRs must describe clearly

how data values were derived. 

• Because of the cumulative significance of these issues, BSC must pursue timely and

effective corrective actions because these issues may adversely impact the quality of

the LA.

• The observers were encouraged that BSC has plans to link the IRAN and the TDMS.

• When the data confirmation process is completed and CAR BSC(B)–03–B–107 has

been fully addressed, the NRC staff expect the corrective action system will ensure the

stability and maintainability of the data management system (i.e., data will be

appropriately qualified and suitably used in products).

• The lessons learned from the previous data audit were used effectively to plan and

conduct this audit.

• All auditors were familiar with the TDMS functions, a critical tool in accomplishing the

objectives of this audit. 

5.2 NRC AUDIT OBSERVER INQUIRY 

The NRC generated one AOI as a result of observing this audit. The inquiry states 

“NRC is concerned with the path forward for defining and qualifying data, particularly the

consistency between NUREG–1298 and DOE AP–3.15Q, Rev. 4, ICN 2 and AP–III.3Q. The

Root Cause Determination for Corrective Action Report, BSC(B)–03–B–107, Reporting

Deficiencies in the Management and Utilization of Data, discusses the differences in the data

definitions and the DOE history of data definitions. Of particular interest are

• Qualification status of Reference Only data as used to exclude model

components (address possibility of bias in data selection).

• Qualification of Technical Information, particularly in regard to the

category of documented and substantiated information, including

electronic databases (please clarify).”