September 22, 2000

Mr. James H. Carlson, Acting Director

Program Management and Administration

Office of Civilian Radioactive Waste Management

U. S. Department of Energy

1000 Independence Avenue, SW

Washington, DC 20585

SUBJECT: U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION AUDIT

REPORT NO. OAR-00-08, “OBSERVATION AUDIT OF THE OFFICE OF

CIVILIAN RADIOACTIVE WASTE MANAGEMENT QUALITY ASSURANCE

DIVISION AUDIT NO. M&O-ARP-00-013"

Dear Mr. Carlson:

I am transmitting the U.S. Nuclear Regulatory Commission’s (NRC’s) Observation Audit Report

No. OAR-00-08 of the U.S. Department of Energy (DOE), Office of Civilian Radioactive Waste

Management (OCRWM), Office of Quality Assurance (OQA), Yucca Mountain Quality

Assurance Division (YMQAD), audit of the processes and activities supporting the Total System

Performance Assessment-Site Recommendation (TSPA-SR) performed by the OCRWM

Management & Operating Contractor (M&O). The audit, M&O-ARP-00-013, was conducted on

July 9–19, 2000, at the M&O facilities in Las Vegas, Nevada.

The purpose of this performance-based audit was to evaluate the quality of TSPA-SR inputs,

the adequacy of the TSPA-SR model, and the effectiveness of the TSPA-SR approach in

demonstrating compliance with the overall performance objective and applicable regulatory

criteria.

The audit team concluded that the OCRWM quality assurance (QA) program had been

satisfactorily implemented for the analysis model reports (AMRs) supporting the TSPA-SR with

the exception of model validation. Since the TSPA-SR Model Report was in draft during the

audit, the report effectiveness will be determined during a second audit. Within the areas

evaluated, the audit team identified potential deficiencies for: 1) failure to document the impact

of “to be verified” inputs on analysis and models; 2) failure to document rationale for exclusion

of uncertainties, assumptions, and alternative conceptual models for process level AMRs;

3) failure to implement appropriate methodology to validate the TSPA-SR model; 4) failure to

maintain model information in the Model Warehouse; and 5) failure to follow planning

documents or make changes when appropriate. A number of recommendations were offered

for improvements and enhancements to the AMRs. The deficiencies are discussed throughout

Sections 4.4 and 4.5 of the enclosed report.

The NRC observers (observers) determined that this audit was effective in identifying potential

deficiencies in the AMRs and the TSPA-SR Model Report. During the conduct of the audit,
J. Carlson -2-

both the audit team and the observers reviewed data, analysis model reports, and software

within the scope of the audit to determine whether they were properly qualified. The audit team

and the observers determined that the software supporting the AMRs, with a few exceptions,

had been qualified. The observers agreed with the audit team’s conclusions, findings, and

recommendations.

However, the observers noted that, when reviewed collectively, the potential deficiencies may

indicate programmatic problems with the implementation of the QA Program. Specifically, the

potential deficiencies identified by the audit team included failure to revise planning documents,

failure to maintain the Model Warehouse, and failure to properly validate models. These were

all examples where the M&O failed to follow procedures.

The observers were concerned that failure to follow procedures continues to be a weakness.

This problem was most recently documented by DOE in its “OCRWM QA Trend Report for

Quality Program Deficiencies First Semester 2000,” dated August 10, 2000. The report reviews

trends for deficiencies identified between January 1, and June 30, 2000. In that report DOE

concluded that “the majority of this semesters issues continue to be personnel error related to

failure to follow procedure and inattention to detail.” The observers recommend that DOE

management continue to focus attention on procedural compliance.

A written response to this letter and the enclosed report is not required. If you have any

questions, please contact Timothy J. Kobetz of my staff at (301) 415-7285.

Sincerely, /RA by N. King Stablein /

Acting For/

Janet Schlueter, Chief (Acting)

High-Level Waste Branch

Division of Waste Management

Office of Nuclear Material Safety

and Safeguards

Enclosure: U.S. Nuclear Regulatory Commission Observation Audit Report No. OAR-00-08,

“Observation Audit of the Office of Civilian Radioactive Waste Management

Quality Assurance Division Audit No. M&O-ARP-00-013"
J. Carlson -2-

both the audit team and the observers reviewed data, analysis model reports, and software

within the scope of the audit to determine whether they were properly qualified. The audit team

and the observers determined that the software supporting the AMRs, with a few exceptions,

had been qualified. The observers agreed with the audit team’s conclusions, findings, and

recommendations.

However, the observers noted that, when reviewed collectively, the potential deficiencies may

indicate programmatic problems with the implementation of the QA Program. Specifically, the

potential deficiencies identified by the audit team included failure to revise planning documents,

failure to maintain the Model Warehouse, and failure to properly validate models. These were

all examples where the M&O failed to follow procedures.

The observers were concerned that failure to follow procedures continues to be a weakness.

This problem was most recently documented by DOE in its “OCRWM QA Trend Report for

Quality Program Deficiencies First Semester 2000,” dated August 10, 2000. The report reviews

trends for deficiencies identified between January 1, and June 30, 2000. In that report DOE

concluded that “the majority of this semesters issues continue to be personnel error related to

failure to follow procedure and inattention to detail.” The observers recommend that DOE

management continue to focus attention on procedural compliance.

A written response to this letter and the enclosed report is not required. If you have any

questions, please contact Timothy J. Kobetz of my staff at (301) 415-7285.

Sincerely,

/RA by N. King Stablein /

Acting For

Janet Schlueter, Chief (Acting)

High-Level Waste Branch

Division of Waste Management

Office of Nuclear Material Safety

and Safeguards

Enclosure: U.S. Nuclear Regulatory Commission Observation Audit Report No. OAR-00-08,

“Observation Audit of the Office of Civilian Radioactive Waste Management

Quality Assurance Division Audit No. M&O-ARP-00-013"

DISTRIBUTION:

File Center PUBLIC ACNW LSS CNWRA NMSS r/f HLWB r/f

DWM r/f SWastler CGlenn BMabrito WBelke RLatta JHolonich

KHooks TCarter DBrooks

ADAMS ACCESSION NUMBER: ML003747947 *See attached concurrence

OFC HLWB HLWB HLWB HLWB

NAME TJKobetz:vlm* LLCampbell* NKStablein JSchlueter

DATE 09/14/00 09/14/00 09/22 /00 09/ /00

OFFICIAL RECORD COPY

ACNW: YES X NO Delete file after distribution: Yes No

1) This document should/ should not be made available to the PUBLIC TJK1 9/21/00

(Initials) (Date)

2) This document is related to the HLW program. It should be placed in the LSS. Yes TJK1 9/21/00
Letter to J. Carlson from J. Schlueter dated: September 22, 2000

cc: R. Loux, State of Nevada

S. Frishman, State of Nevada

L. Barrett, DOE/Wash, DC

A. Brownstein, DOE/Wash, DC

S. Hanauer, DOE/Wash, DC

C. Einberg, DOE/Wash, DC

J. Carlson, DOE/Wash, DC

N. Slater, DOE/Wash, DC

A. Gil, DOE/Las Vegas, NV

R. Dyer, YMPO

S. Brocoum, YMPO

R. Clark, YMPO

C. Hanlon, YMPO

T. Gunter, YMPO

G. Dials, M&O

J. Bailey, M&O

D. Wilkins, M&O

M. Voegele, M&O

S. Echols, Winston & Strawn

B. Price, Nevada Legislative Committee

J. Meder, Nevada Legislative Counsel Bureau

D. Bechtel, Clark County, NV

E. von Tiesenhousen, Clark County, NV

A. Kalt, Churchill County, NV

G. McCorkell, Esmeralda County, NV

L. Fiorenzi, Eureka County, NV

A. Remus, Inyo County, CA

B. Duke, Lander County, NV

J. Pitts, Lincoln County, NV

J. Wallis, Mineral County, NV

L. Bradshaw, Nye County, NV

M. Murphy, Nye County, NV

J. McKnight, Nye County, NV

B. Ott, White Pine County, NV

D. Weigel, GAO

W. Barnard, NWTRB

R. Holden, NCAI

A. Collins, NIEC

R. Arnold, Pahrump County, NV

J. Lyznicky, AMA

R. Clark, EPA

F. Marcinowski, EPA

R. Anderson, NEI

R. McCullum, NEI

S. Kraft, NEI

J. Kessler, EPRI

R. Wallace, USGS

R. Craig, USGS

W. Booth, Engineering Svcs, LTD

J. Curtiss, Winston & Strawn
U.S. NUCLEAR REGULATORY COMMISSION

OBSERVATION AUDIT REPORT NO. OAR-00-08

OBSERVATION AUDIT OF THE

OFFICE OF CIVILIAN RADIOACTIVE WASTE MANAGEMENT

QUALITY ASSURANCE DIVISION

AUDIT NO. M&O-ARP-00-013

/S/ 09/21/00 /S/ 09/21/00

Richard Codell David Esh

Performance Assessment Section Performance Assessment Section

Environmental and Performance Environmental and Performance

Assessment Branch Assessment Branch

Division of Waste Management Division of Waste Management

/S/ 09/21/00

Tim Kobetz

Project and Engineering Section

High-Level Waste Branch

Division of Waste Management

/S/ 09/14/00 /S/ 09/14/00

Robert Brient Sitakanta Mohanty

Center for Nuclear Waste Regulatory Center for Nuclear Waste Regulatory

Analyses Analyses

/S/ 09/14/00 /S/ 09/14/00

Michael Smith Osvaldo Pensado

Center for Nuclear Waste Regulatory Center for Nuclear Waste Regulatory

Analyses Analyses

Reviewed and Approved by:

/RA/ 09/21/00

N. King Stablein, Chief

Projects and Engineering Section

High-Level Waste Branch

Division of Waste Management
2

1.0 INTRODUCTION

Staff from the U.S. Nuclear Regulatory Commission (NRC) Division of Waste Management

observed the U.S. Department of Energy (DOE), Office of Civilian Radioactive Waste

Management (OCRWM), Office of Quality Assurance (OQA), Yucca Mountain Quality

Assurance Division (YMQAD) audit of the processes and activities supporting the Total System

Performance Assessment-Site Recommendation (TSPA-SR) performed by the OCRWM

Management & Operating Contractor (M&O). The audit, M&O-ARP-00-013, was conducted on

July 9–19, 2000, at the M&O facilities in Las Vegas, Nevada.

The objective of this audit was to evaluate the implementation of the applicable provisions

contained in the OCRWM Quality Assurance Requirements and Description (QARD), DOE/RW0333P, Revision 9, by reviewing selected analysis model reports (AMRs) supporting the TSPASR. During the audit, selected AMRs and the draft TSPA-SR Model Report were subjected to

technical reviews as well as reviews to assess whether the applicable programmatic

requirements contained in the QARD and implementing procedures were met.

The NRC staff objective was to assess whether the M&O and OQA were properly implementing

the provisions contained in the QARD and the requirements contained in Subpart G, Quality

Assurance, to Part 60 of Title 10 of the Code of Federal Regulations (10 CFR Part 60).

Because of the anticipated DOE submittal of the site recommendation (SR) in November 2000,

the following observation activities were emphasized: 1) confirming that data, software, and

models supporting site recommendation are properly qualified; and 2) reviewing the progress

being made by DOE and its contractors in meeting the qualification goals for SR.

2.0 MANAGEMENT SUMMARY

OQA Audit M&O-ARP-00-013 was the first of two audits planned for the TSPA-SR. This audit

evaluated the early-phase TSPA-SR activities, particularly inputs to the TSPA-SR. The second

audit will evaluate outputs from TSPA-SR. The NRC staff determined that OQA Audit M&OARP-00-013 was effective and conducted in a professional manner. Audit team members were

independent of the activities they audited and were knowledgeable in the quality assurance

(QA) and technical disciplines within the scope of the audit.

The audit team concluded that the OCRWM QA program had been satisfactorily implemented

for the AMRs supporting the TSPA-SR with the exception of model validation. Since the TSPASR Model Report was in draft during the audit, the report effectiveness will be determined

during the second audit. Five potential deficiency reports were initiated during the audit.

Specifically, the audit team identified potential deficiencies for: 1) failure to document the

impact of “to be verified” inputs on analysis and models; 2) failure to document rationale for

exclusion of uncertainties, assumptions, and alternative conceptual models for process level

AMRs; 3) failure to implement appropriate methodology to validate the TSPA-SR model; 4)

failure to maintain model information in the Model Warehouse; and 5) failure to follow planning

documents or make changes when appropriate. A number of recommendations were offered

for improvements and enhancements to the AMRs. The deficiencies and recommendations are

discussed throughout sections 4.4 and 4.5 of this report.
3

The NRC observers (observers) determined that this audit was effective in identifying potential

deficiencies, and recommending improvements, in the AMRs and TSPA-SR Model Report.

During the conduct of the audit, both the audit team and the observers reviewed data, analysis

model reports, and software within the scope of the audit to determine whether it was properly

qualified. The audit team and the observers determined that the software supporting the

AMRs, with a few exceptions, had been qualified. However, some of the data still required

verification.

However, the observers noted that, when reviewed collectively, the potential deficiencies may

indicate programmatic problems with the implementation of the QA Program. Specifically,

deficiencies identified by the audit team included failure to revise planning documents, failure to

maintain the Model Warehouse, and failure to validate all models which were all examples

where the M&O failed to follow procedures.

The observers were concerned that failure to follow procedures continues to be a weakness.

This issue was discussed with the audit team during the audit, however, not specifically

discussed at the audit exit meeting. Subsequent to the audit, this problem was documented by

DOE in its “OCRWM QA Trend Report for Quality Program Deficiencies First Semester 2000,”

dated August 10, 2000. The report reviews trends for deficiencies identified between January

1, and June 30, 2000. In that report DOE states “it is concluded that the majority of this

semesters issues continue to be personnel error related to failure to follow procedure and

inattention to detail.” The observers recommend that DOE management continue to focus

attention on procedural compliance.

In addition to the above observations, the observers provided detailed observations regarding

the implementation of the quality assurance program (see Section 4.6.1 of this report) and a

review of the technical adequacy of the AMRs and TSPA-SR (see Section 4.6.2 of this report).

These observations cover topics discussed during the audit that are of particular concern

regarding the technical adequacy of the AMRs and potential concern over the acceptability of

DOE’s upcoming site recommendation report.

3.0 AUDIT PARTICIPANTS

3.1 Nuclear Regulatory Commission Observers

Robert Brient Team Leader (Quality Assurance) CNWRA

David Esh Technical Specialist (Performance Assessment) NRC

Richard Codell Technical Specialist (Performance Assessment) NRC

Tim Kobetz QA Engineer (Quality Assurance) NRC

Sitakanta Mohanty Technical Specialist (Performance Assessment) CNWRA

Michael Smith Technical Specialist (Performance Assessment) CNWRA

Osvaldo Pensado Technical Specialist (Performance Assessment) CNWRA
4

3.2 OQA Audit Team

Kristi Hodges Audit Team Leader OQA/Quality Assurance Technical Support

Services (OQA/QATSS)

James Blaylock Auditor OQA/DOE

Harvey Dove Technical Specialist OQA/QATSS

Michael Eshleman Auditor Yucca Mountain Site Characterization

Office (YMSCO)/Management and

Technical Services (MTS)

Mark Nutt Technical Specialist YMSCO/MTS

Richard Powe Auditor OQA/QATSS

James Voigt Auditor OQA/QATSS

Alf Wikjord Technical Specialist YMSCO/Atomic Energy of Canada, Limited

Frank Wong Technical Specialist YMSCO/MTS

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION

This OQA audit of the M&O was conducted in accordance with OCRWM Quality Assurance

Procedure (QAP) 18.2, Revision (Rev.) 6, Interim Change Notice (ICN) 0, “Internal Audit

Program,” and QAP-16.1Q, Rev. 4, ICN 1, “Performance/Deficiency Reporting.” The NRC

staff’s observation of this audit was based on the NRC procedure, “Conduct of Observation

Audits,” issued October 6, 1989.

4.1 Scope of the Audit

The audit team conducted a limited scope, performance based, audit of activities and

processes related to the development of the AMRs and the TSPA-SR Model Report supporting

the TSPA-SR. AMRs, software, and data were evaluated during the audit process. The audit

included review of the programmatic controls governing the AMRs and technical issues

discussed in the AMRs. The following procedures and AMRs supporting the TSPA-SR were

reviewed by the audit team and the observers during the audit:

Procedures

a) AP-SI.1Q, “Software Management,” Revision 2, Interim Change Notice 4

b) AP-SIII.2Q, “Qualification of Unqualified Data and the Documentation of Rationale for

Accepted Data,” Rev. 0, ICN 2

c) AP-2.13Q, “Technical Product Development Planning,” Rev. 0, ICN 4

d) AP-2.14Q, “Review of Technical Products,” Rev 0, ICN 1

e) AP-3.10Q, “Analysis and Models,” Rev. 2, ICN 2

f) AP-3.15Q, “Managing Technical Product Inputs,” Rev. 1, ICN 1
5

Reports

a) MDL-WIS-PA-000002, “Total System Performance Assessment (TSPA) Site

Recommendation Model Report” in draft

b) ANL-EBS-PA-000001, “WAPDEG Analysis of Waste Package and Drip Shield

Degradation,” Revision 00

c) ANL-WIS-MD-000010, “Summary of Dissolved Concentration Limits,” Revision 00

d) ANL-NBS-MD-000005, “Abstraction of Drift Seepage,” Revision 00

e) ANL-NBS-MD-000007, “Abstraction of BDCF Distribution for Irrigation Period,”

Revision 00

f) Features, Events, and Processes (FEPs) AMRs: 1) Engineered Barrier System FEPs

AMR (ANL-WIS-PA-000002); 2) Waste Package FEPs (ANL-EBS-PA-000002 and

ANL-WIS-MD-000008); 3) Waste Form FEPs (ANL-WIS-MD-000009); 4) UZFT FEPs

AMR (ANL-NBS-MD-000001); and 5) Biosphere FEPs AMR (ANL-MGR-MD-000009).

The FEPs Database (Revision 1)

4.2 Conduct and Timing of the Audit

The audit was performed in a professional manner and the audit team demonstrated a sound

knowledge of the applicable M&O and DOE programs and procedures. Audit team personnel

were persistent in their interviews, challenged responses when appropriate, and performed an

acceptable audit.

The audit team and observers caucused at the end of each day to discuss new and developing

issues. Also, the audit team met with M&O management, with the observers present, each

morning to discuss the current audit status and preliminary findings. The observers determined

that the timing of the audit was appropriate for the audit team to evaluate ongoing TSPA-SR

activities.

4.3 Audit Team Qualification and Independence

The observers determined that the qualifications of the audit team leader and the OQA audit

team members met the requirements of QAP-18.1, “Auditor Qualification.” The audit team

members did not have prior responsibility for performing the activities they audited. Curriculum

vitae of the audit team Technical Specialists were reviewed by the observers and found to be

acceptable.

4.4 Examination of Quality Assurance Elements

The OQA programmatic and technical audit activities were conducted separately. The limited

scope audit focused on the QA elements associated with the development of the AMRs. The

observers evaluated the audit team’s review of the following QA areas and agreed with the

audit team’s findings and conclusions in these areas.
6

4.4.1 AP-2.13Q “Technical Product Development Planning”

The audit team reviewed technical development plans and work product planning sheets

applicable to the subject AMRs. The audit team found that, in two instances, the M&O failed to

follow AP-2.13, Step 5.3, which requires changes to a plan to be documented. The audit team

considered this issue a potential deficiency report siting AP-3.10Q, Steps 5.2 and 5.6, which

require a plan to be followed in the development of AMRs and the technical checker to verify

that the plan was followed (see Sections 4.5.1 and 4.5.2 for specific findings). The observers

agreed with this finding.

4.4.2 AP-SI.1Q “Software Management”

The audit of software management focused on control of software routines. Rather than

requiring the full qualification process expected for more complex software, the procedure

requires that routines must be verified (usually by hand calculation) for each application. The

audit team was concerned that some verification was by calculation using different software

(e.g., MATHCAD) rather than hand calculations and some routines may have been complex

enough to justify full qualification as a computer code. The observers agreed with this concern.

4.4.3 AP-3.15Q “Managing Technical Product Inputs”

The audit team identified a concern with the tracking of input data. In the iterative performance

assessment process, the outputs from lower tier AMRs (such as process and abstraction

models) become the inputs for higher tier AMRs and eventually the TSPA-SR model. Outputs

were classified under the current procedure as “N/A: Technical Product Output” regardless of

any unqualified or TBV input data that may have been used in the lower tier AMR. The audit

team identified this concern as a potential deficiency report involving the requirements of AP3.10Q to evaluate the impact of unqualified data on the validation of a model. The observers

agreed with this finding.

4.4.4 AP-SIII.2Q “Qualification of Unqualified Data and the Documentation of Rationale

for Accepted Data”

The data status and traceability were evaluated, however, the data qualification process was

not specifically audited.

4.4.5 AP-3.10Q “Analysis and Models”

Independent of the technical audit, the technical checking process was evaluated for the

subject AMRs. The draft TSPA-SR Model Report had not yet been through this verification

step. No discrepancies were identified.

However, the audit team identified potential deficiency reports with regard to the following areas

of AP-3.10Q:

• The M&O failed to assess the impact of TBV data input to the AMRs, particularly in the

impact of unqualified data on model validity (see Section 4.4.3, of this report).
7

• AMRs did not adequately address model validation, (i.e., validation criteria and methods,

validation tests conducted, and results). Specifically, the model validation approach for

the TSPA-SR model was taking credit for process level AMR validations which may or

may not have occurred, rather than establishing a formal model validation for the TSPASR model. The audit team determined this is a potential deficiency report citing AP3.10Q, Step 5.3. The observers agreed with this finding.

• The audit team identified that AP-SIII.3Q required inputs and outputs of models to have

been submitted to a “Model Warehouse.” A potential deficiency citing failure to follow

AP-3.10Q was identified because, in several examples, certain information required by

the procedure was not transferred to the Model Warehouse. The observers agreed with

this finding.

4.4.6 AP-2.14Q “Review of Technical Products”

According to the procedure, technical reviews were required when organizations outside of the

originating organization, in this case, Performance Assessment, were affected by the report.

Since Performance Assessment was the ultimate user of these AMRs, technical reviews were

generally not performed on the subject AMRs. The audit team noted that more extensive

reviews may improve the products and may have corrected some of the concerns that were

found during the audit. The observers agreed with this finding.

4.5 Examination of Technical Activities

The audit team technical specialists prepared detailed checklists for each of the AMRs being

evaluated. Technical activities examined by the audit team are summarized below for each of

the AMRs.

4.5.1 Total System Performance Assessment Site for Recommendation

(MDL-WIS-PA-000002, in draft)

The report on the TSPA-SR documented the model for analyzing the performance of the

repository system in isolating waste for long periods of time. The objective of the TSPA-SR

model was to integrate information from other process models into one comprehensive model.

The individual parts of the TSPA-SR model are called component models. The M&O selected

to segment the repository system into the following main component models:

• Unsaturated zone flow and transport

• Thermal hydrology

• In-drift geochemical environment

• Waste package degradation

• Waste form degradation

• Saturated zone flow and transport

• Biosphere

• Disruptive events

The audit of the TSPA-SR included an assessment of whether the quality assurance

procedures were followed in the development of the document and whether the technical
8

content of the product was acceptable. The audit team inquired about numerous technical and

procedural areas of the report generation process. These included: 1) planning and

implementation; 2) qualification status of data utilized; 3) assumptions used; 4) data acquisition

and traceability; 5) data uncertainty; 6) integration of the TSPA-SR model with other

components; 7) code verification; and 8) model implementation.

The observers concurred with the audit team that the technical content of the report was

appropriate. The audit team determined the M&O had not followed the plans for the AMR.

Specifically, the M&O had not updated the AMR to address all of the acceptance criteria

provided in the NRC Issue Resolution Status Reports (IRSRs) and the M&O had not addressed

Peer Review comments in the AMR. The M&O stated that, in both cases, it had addressed the

items implicitly. The audit team noted that these types of comments should have received more

explicit treatment. The observers concurred with the audit team that a more explicit treatment

of IRSR acceptance criteria and peer review comments was required by the plan and would

improve transparency and traceability. The audit team identified this issue as a failure to follow

AP-2.13Q, Step 5.3, which requires changes to a plan to be documented. The audit team

identified this issue as a potential deviation report citing AP-3.10Q, Steps 5.2 and 5.6, for failure

to implement a plan and failure by the technical checker to verify the plan was followed.

The audit team found that the qualification status of the data inputs to the TSPA-SR model

were in various states of verification with many data inputs having a TBV quality status.

Therefore, the results (output) from this model were also TBV. The audit team found that every

major component of the TSPA-SR model produced output that should have been categorized

as TBV. The observers noted that a significant amount of data still required verification and

that the M&O’s schedule for completing data verification appeared ambitious.

GoldSim is a software program used by the M&O to integrate models and collectively evaluate

the repository performance for the TSPA-SR. The GoldSim software program was not yet

qualified.

The observers concurred with the audit team that, compared to documentation reviewed in past

audits, an improved effort has been made by the M&O to explicitly list assumptions that apply to

the TSPA-SR model. In addition, the model document lists assumptions that were specifically

generated via the component model abstraction process. However, all of the assumptions

applying to supporting AMRs (those supporting abstraction AMR’s) were not propagated into

the higher-level documents. The audit team noted, and the observers agreed, that a complete

listing of applicable assumptions and either the technical basis or references to the technical

basis supporting the assumptions would improve the document.

The audit team found that verification of calculations had been performed in the TSPA-SR

model document. The audit team reviewed a number of the calculations in detail. The audit

team was not able to determine the requirements for the number of calculations required to

verify a model. In addition, the audit team was unable to identify acceptance criteria for model

verification. Specifically, some calculations were exact to two significant digits while other

calculations of an identical type were exact to five significant digits. It was not clear to the audit

team whether verification to two significant digits was acceptable or if all calculations should be

performed to five significant digits. Since the TSPA-SR model was still in draft no technical

review had been performed. The audit team commented that, in the future, the technical

reviewer should identify and resolve this type of potential discrepancy.
9

The audit team discussed the component verification of the WAPDEG (see Section 4.5.2 of this

report) with the M&O. Specifically, the audit team noted that the WAPDEG calculations and

transfer of information to and from the TSPA-SR model should be discussed in the TSPA-SR

model document. At the conclusion of the audit this had not been completed.

The audit team identified that some basic steps in model testing were not documented and/or

completed. The TSPA model is a mathematical model; therefore, it should be appropriately

tested and the results documented. For example, the audit team determined that sensitivity of

the TSPA-SR results, with regard to variations in TSPA time-step-size and to variations in

resolution at the component-level (i.e., number of infiltration bins, number of thermohydrology

bins, number of stream tubes, etc.), had not been presented. The M&O stated that such testing

did take place, but the results of the testing were not formally presented at the system-level.

The audit team determined, and the observers agreed, that a description of TSPA-SR model

stability was essential to achieving confidence in the model.

The audit team identified that data usage by the TSPA-SR model from the component models

was appropriate, with one exception. Dissolved concentration limits were generated for a more

narrow range of chemical conditions than the expected environments generated by the

in-package chemistry component model. Therefore, the TSPA-SR model could potentially

generate chemical conditions for which the dissolved concentration limit abstraction does not

apply. While these extreme chemical conditions are not expected, they are possible as defined

by the in-package chemistry component model. No objective evidence existed that the current

procedures were able to identify the aforementioned problem. The observers agreed with this

concern.

The audit team discussed the implementation of seismic effects on cladding in the TSPA-SR

model. The observers concurred with the audit team that the current implementation in the

TSPA model did not appear to be correct. Inclusion of seismic events should be consistent with

the implementation of intrusive or extrusive igneous activity in computing risk.

The audit team also identified problems with the implementation of model validation. The

QARD states in part that as part of scientific investigation, model validation must be planned

and implemented. The QARD provided a number of options to model validation. The TSPASR model document outlined an approach to model validation where the component models are

validated individually and the transfer of information within the TSPA is verified. The audit team

determined, and the observers agreed, that the approach taken by the M&O is not in

accordance with the information contained in the QARD. Almost all of the major abstraction

AMRs have not been validated. The audit team considered this issue a potential deficiency

report for failing to follow AP-3.10Q which implements the requirements of the QARD.

4.5.2 Analysis Model Report WAPDEG Analysis of Waste Package and Drip Shield

Degradation (ANL-EBS-PA-000001, Rev 00)

This AMR documented the abstraction of drip shield and waste package (WP) degradation from

the code “WAPDEG,” for use in DOE’s TSPA-SR. The WAPDEG model itself is composed of

several sub-models to determine general and localized corrosion of the drip shield, WP outer

and inner barriers, manufacturing defects, stress corrosion cracking, material aging, and

microbial induced corrosion.
10

The audit of the WAPDEG AMR included a combination of procedural and technical inquiries to

verify whether the quality assurance procedures were followed in the development of the

document and the technical quality of the product was acceptable. The audit team inquired

about technical areas of the report and procedural areas of the report generation process,

including: 1) planning and implementation; 2) qualification status of data utilized;

3) assumptions used; 4) data acquisition and traceability; 5) data uncertainty; 6) integration of

this model component with other model components; 7) rationales for the types of abstractions;

and 8) use of the generated technical output by other groups or system components.

The observers concurred with the audit team that the technical content of the report was

appropriate. However, as with several of the other audits, there appeared to be inadequate

tracking of comments generated in DOE’s external peer review of the TSPA. The WAPDEG

planning document states that they will address the peer review comments, but there was no

mention of them in the AMR. This AMR acknowledged the issues from the NRC IRSRs that

pertained to waste package degradation, but did not appear to explicitly address them. The

audit team identified this issue as a failure to follow AP-2.13Q, Step 5.3, which requires

changes to a plan to be documented. The audit team identified this issue as a potential

deficiency report citing AP-3.10Q, Steps 5.2 and 5.6, for failure to implement a plan and failure

by the technical checker to verify the plan was followed.

The audit team also identified a number of technical problems and inconsistencies with the

WAPDEG model:

• The AMR did not provide sufficient detail to demonstrate how stress corrosion cracking

(SCC) was incorporated into the model for degradation of the end-cap welds. This

included a lack of detail on how stress mitigation techniques were incorporated into the

SCC models. This issue was identified as a potential deficiency report for failure to

implement Supplement III.2.6B, of the QARD.

• It was not clear how aging effects of the alloy 22 material were taken into account.

• The audit team noted an inconsistency in the way that the corrosion rate was sampled

for SCC of the end cap for the WPs. The end cap model has two layers of alloy 22

representing a single layer of material in the as-designed waste package. The

WAPDEG code samples the corrosion rate twice even though the corrosion of a single

layer of material is being calculated. The audit team felt that there should have been a

single corrosion rate to cover both of the layers.

• Failure time is described as the time of first penetration of the WP. However, the

penetration time for SCC is orders of magnitude shorter than the time for general

corrosion, yet the SCC failure would allow only diffusive release. The audit team felt

that the AMR should make this distinction very clear, so that the short times for failure

due to SCC are not misinterpreted.

The audit team identified concerns with the use of “Gaussian Variance Partitioning” (GVP) in

the AMR. GVP was utilized in an attempt to separate uncertainty and variability in the

distributions of parameters used in the WAPDEG model. The concept was that a distribution of

a parameter, such as the corrosion rate for alloy 22, contains both variability and uncertainty,

and that this dichotomy should be recognized in the performance analysis. For corrosion rate,
11

“variability” would represent the real differences in corrosion rate from place to place in the

repository, or from place to place on the waste package, caused by mechanistic differences in

material properties or the environment. This information could be gathered in corrosion

experiments by subjecting coupons to a range of chemical environments, and taking coupons

for the experiments from potentially different materials such as welds and open areas.

Uncertainty, on the other hand, would be random, non-mechanistic variations caused by

measurement errors or unquantifiable processes. The AMR author stated that most of the

distribution is caused by uncertainty in experimental measurements rather than mechanistic

differences in the samples or environments. Randomly sampling the partitioning of uncertainty

and variability is not appropriate if the result is a reduction in risk.

The author of the AMR noted that increasing the proportion of the distribution attributed to

variability gave conservatively shorter times for the first package breach, but it gave lower peak

release rates overall. Therefore, the current application of GVP could result in a significant

underestimation of the peak dose if the application of the GVP technique is not statistically

supported by the data. The observers commented that the way in which the peak dose is

reported (i.e., the peak of the mean dose), makes it unclear whether increased variability or

increased uncertainty is more conservative. The observers recommended that the WAPDEG

model results should not be looked at in isolation, and that the overall TSPA-SR results would

have to be examined to make this determination.

4.5.3 Analysis Model Report Summary of Dissolved Concentration Limits

(ANL-WIS-MD-000010, Rev 00)

The AMR for the summary of dissolved concentration limits documented the M&O staff

abstraction of solubility limits of radioactive elements from the process-level models provided by

Natural Environment Program Operations and Waste Package Operations. The product of the

abstraction was to develop solubility limits as functions, distributions, or constants for all

transported radioactive elements identified by the Performance Assessment Operation

radioisotope screening. The results of the analyses were generated for performance

assessment calculations.

The audit of the summary of dissolved concentration limits AMR included a combination of

procedural and technical inquiries to verify that the quality assurance procedures were followed

in the development of the document and the technical quality of the product was acceptable.

The audit team inquired about many technical areas of the report and procedural areas of the

report generation process, including: 1) planning and implementation; 2) qualification status of

data utilized; 3) assumptions used; 4) data acquisition and traceability; 5) data uncertainty;

6) integration of this model component with other model components; 7) rationales for the types

of abstractions [probability distribution functions, response surfaces, constants]; and 8) use of

the generated technical output by other groups or system components.

The observers concurred with the audit team that the technical content of the report was

appropriate. The audit team determined the plans for the document had been followed with two

exceptions. Specifically, NRC IRSR acceptance criteria and TSPA-SR Peer Review comments

were not addressed (consistent with all technical documents reviewed for this audit). The

document authors stated they did address the aforementioned items but did so implicitly. The

audit team was looking for a more explicit treatment. The observers concurred with the audit
12

team that a more explicit treatment of IRSR acceptance criteria and peer review comments

would improve transparency and traceability. This AMR acknowledged the issues from the

NRC IRSRs that pertained to waste package degradation, but did not appear to explicitly

address them. The audit team identified this issue as a failure to follow AP-2.13Q, Step 5.3,

which requires changes to a plan to be documented. The audit team identified this issue as a

potential deficiency report citing AP-3.10Q, Steps 5.2 and 5.6 for failure to implement a plan

and failure by the technical checker to verify the plan was followed.

The audit team found that the qualification status of the data inputs to the TSPA model were in

various states of verification with the main input to the EQ3/6 modeling effort being TBV.

Therefore, the results (output) from this abstraction were also TBV.

As discussed in Section 4.5.1 of this report, the observers concurred with the audit team that an

improved effort has been made to explicitly list assumptions that apply to an abstraction.

However, all of the assumptions applying to supporting AMRs (those supporting abstraction

AMRs) are not propagated into the higher-level documents. The audit team and observers

agreed that a complete listing of applicable assumptions and either the technical basis or

references to the technical basis supporting the assumptions would improve the document.

Some potential technical problems were identified with the integration of this model component

with other model components and the use of technical output generated by other groups or

system components. For example, solubility limits were generated for a range of potential

chemical environments. The in-package chemistry AMR was not completed at the time of

preparation of the dissolved concentration limits AMR. Therefore, the dissolved concentration

limits AMR developed probability distribution functions (pdfs) and response surfaces based on

their best estimate for expected in-package environmental conditions. However, the ranges

selected for key chemical variables by the dissolved concentration limits AMR were more

narrow than those generated by the in-package chemistry model. Therefore, the result was

that the TSPA-SR model would potentially generate chemical conditions for which no data has

been produced for the solubility limits. The authors did evaluate the impact of using the

response surfaces outside of the ranges for which they were developed but did not do the same

for the pdfs. In addition, it was not clearly identified that the dissolved concentration limits were

developed based on the assumption of long-package lifetime (J-13 water as the starting fluid

concentration). The abstraction would not apply for an under performance-type calculation.

The audit team identified that little technical basis was provided for the decoupling of system

components. For example, in-package chemistry is one-way coupled to the dissolved

concentration limits abstraction, such that, there is no feedback to in-package chemistry

resulting from the solubility limits abstraction. The source-term degradation resulting in the

release of uranium was discussed in this context. The uranium will be present in solution

potentially up to its solubility limit (and may be deposited as secondary phases). The M&O

stated that high uranium concentrations (determined via the solubility limit) could affect pH,

thereby creating a fully-coupled system at the abstraction-level.

The audit team also identified problems with the implementation of model validation. The

QARD states in part that as part of scientific investigation, model validation must be planned

and implemented. The QARD provides a number of options for model validation. The

document authors acknowledged that they had not completed model validation but that they
13

were compiling the information and it should appear in a future revision to the document. The

audit team determined, and the observers agreed, that the approach taken by the M&O is not in

accordance with the information contained in the QARD. Almost all of the major abstraction

AMRs have not been validated. The audit team considered this issue a potential deficiency

report for failure to follow AP-3.10Q which implements the requirements of the QARD.

4.5.4 Analysis Model Report Abstraction of Drift Seepage

(ANL-NBS-MD-000005, Rev 00)

The AMR documented the M&O’s abstraction of the process-level models for drift seepage for

use in DOE’s TSPA-SR. It was based both on process-model results (CRWMS M&O, 2000s,

section 6.1) and calibration of seepage tests from one niche in the Exploratory Studies Facility

(CRWMS M&O 2000d). The model also takes into account increased seepage flow to the drift

to account for uncertainties in rock mechanics, parameter correlations, and channelized flow in

the rock. The abstracted model is needed for the TSPA-SR.

The audit of this AMR included a combination of procedural and technical inquiries to assess

whether the quality of the product was acceptable. The audit team inquired about several

technical and procedural areas of the report generation process, including:

1) planning and implementation; 2) assumptions used; 3) data acquisition and traceability; 4)

data uncertainty; 5) integration of the present model with other models; 6) rationales for the

types of abstraction; and 7) use of the generated technical output by other groups or system

components.

The audit team commented that the authors did a good job of tracking assumptions. The

abstraction model incorporated all of the original assumptions in the background model, and

several additional ones made necessary by simplification. The observers agreed with the audit

team’s assessment that the assumptions were carefully stated.

The audit team assessed how much data used in the models was from expert elicitation and

how much was from data collected at YM or an appropriate analog. The author stated that

there were problems with the underlying analyses that supported the abstraction. Specifically,

the author stated that he was not entirely comfortable with the level of justification of the

models. The observers reviewed the results of an audit report on the performance-based QA

audit on activities related to the Unsaturated Flow and Transport Process Model Report,

conducted by DOE January 24–28, 2000. The observers also reviewed the NRC observation

report which discussed that audit. The DOE report identified deficiencies in the bases used to

derive the abstraction and the NRC report agreed with this finding. The present AMR did not

justify the underlying models, but simply abstracted the behavior into a model suitable for the

TSPA code. The observers are concerned that this problem has not been corrected.

The audit team identified several other weaknesses during the audit:

• The report had no explicit mention of any open issues or acceptance criteria relating to

seepage from NRC’s IRSRs for Container Life and Source Term, Evolution of the Near

Field Environment, or Unsaturated/Saturated Flow under Isothermal Conditions.
14

• There were several recommendations from a DOE peer review conducted on the TSPASR, but there was no apparent tracking of these comments.

• The AMR did not mention alternative conceptual models. The author stated that the

underlying basis models did not consider alternatives either, and this was basically an

abstraction of that work. Other AMRs had alternative conceptual models for seepage,

but these were not considered in the abstraction.

The audit team found that the AMR author did not have high confidence in the abstraction,

mainly because the underlying AMR contained insufficient justification. The author commented

that he felt a need for more seepage tests at the site before they would have sufficient

confidence. Some data from air-permeability tests in the niches were used in the model, but

the data were taken in close proximity, less than 1 meter, from the drift wall and were likely to

be influenced by the mining operations, stress-induced cracking and ventilation effects.

4.5.5 Analysis Model Report Abstraction of BDCF Distribution for Irrigation Period

(ANL-NBS-MD-000007, Rev 00)

The AMR for abstraction of biosphere dose conversion factor (BDCF) distributions for irrigation

periods documented the M&O staff’s derivation of abstractions for the time evolution of the

BDCFs due to radionuclide build-up effects in soil. These abstractions are to be used in

TSPA-SR. The analyses for radionuclide build-up in soil were conducted using GENII-S, which

includes effects from previous irrigation, harvest removal, radioactive decay, and leaching. The

M&O investigated and added the effects of soil erosion to their analyses prior to performing

distribution fitting to develop the abstractions.

The audit of the abstraction of BDCF distributions for irrigation periods AMR included a

combination of procedural and technical inquiries to verify that the quality assurance

procedures were followed in the development of the document and that the technical quality of

the product was acceptable. The audit team inquired about many technical areas of the report

and procedural areas of the report generation process, including: 1) planning and

implementation; 2) data acquisition and traceability; 3) assumptions used; 4) rationales for the

types of abstractions; 5) qualification status of data utilized; 6) model designation; 7) data

uncertainty; and 8) integration of this model component with other model components.

The observers concurred with the audit team that the technical content of the report was

appropriate. The audit team determined the plans for the document had been followed. The

authors’ specific use of NRC’s acceptance criteria was noted by the audit team and referred to

as a potential model for other AMRs.

The observers concurred with the audit team that improvements could be made in the area of

data acquisition and traceability. The audit team focused on the procedures used for notifying

affected parties when data had been superceded. The author appeared familiar with the

procedure (affected parties would be notified when data used was changed) but had never

been informed of such a data change. The audit team was concerned that notification to

affected AMRs of superceded data may not be occurring. The auditor noted that the

traceability of the mathematical contribution of BDCFs to human dose in TSPA would be

verified later.
15

The observers agreed with the audit team that final assumptions had been clearly stated.

Alternate assumptions and options were reported in supporting AMRs, but are not explicitly

discussed in the abstraction of BDCF’s AMR. The audit team noted that some of the

assumptions stated in this AMR were not carried forward into higher-level documents.

Specifically, the auditor noted that only one of the two primary assumptions listed in the

abstraction of BDCF’s AMR was carried forward to the TSPA-SR. No explanation was provided

since the AMR author was not scheduled to review the TSPA-SR until after the audit.

The observers concurred with the audit team that the scientific approach appeared sound and

defensible but improved documentation in some areas was needed. The author acknowledged

that areas needing improvements included selection of distribution types, erosion calculations,

and the survey conducted to obtain critical group characteristics.

The observers concurred with the audit team that additional work may be needed in the area of

data and model validation, specifically, in the use of GENII. The author was aware of this issue

and stated that work had already begun for validating the use of GENII for YM dose

assessments. Most of the data used for this AMR have been developed. The use of expert

elicitation was minimal, possibly none, and most of the data were accepted. Limited data for

soil-to-plant transfer factors is available for the specific conditions at YM (i.e. plants, soil types,

pH).

The audit team also reviewed the reporting of this type of work (abstraction of BDCFs) in an

AMR rather than in a technical report (TR). The audit team questioned whether this material

was a model and warranted the QA scrutiny that is associated with an AMR, or whether the

AMR should be reviewed to the lesser requirements of a technical report. The audit team was

also satisfied with the subjective nature of the goodness-of-fit used and that the BDCFs utilized

for each radionuclide reflected the true nature of uncertainty seen in the biosphere model. The

author acknowledged that additional work was required for the parameters related to the critical

group and biosphere.

4.5.6 Analysis Model Reports for Features, Events, and Processes (FEPs)

The reports reviewed included: 1) Engineered Barrier System FEPs AMR

(ANL-WIS-PA-000002); 2) Waste Package FEPs (ANL-EBS-PA-000002 and

ANL-WIS-MD-000008); 3) Waste Form FEPs (ANL-WIS-MD-000009); 4) UZFT FEPs AMR

(ANL-NBS-MD-000001); and 5) Biosphere FEPs AMR (ANL-MGR-MD-000009). The FEPs

Database (Revision 1) was also audited.

The purposes of these AMR documents and the electronic database are to identify and

document the analyses and resolution of the primary FEPs affecting the repository

performance. The process-level FEP AMRs identify subject-specific FEPs and provide

screening arguments. The overall FEPs AMR contains FEPs identified from various sources

and describes screening methodology. DOE prepared these AMRs to aid in the resolution of

the FEP inclusion/exclusion process and the screening methodology used in the process.

These documents were developed to: 1) identify which FEPs are to be considered explicitly in

the TSPA (called included FEPs); and 2) identify FEPs not to be included in the TSPA (called

excluded FEPs) and provide justification for why these FEPs do not need to be a part of the

TSPA model.
16

The biosphere AMR for evaluation of the applicability of biosphere-related FEPs had a more

expanded scope than the other FEP AMR documents audited. The biosphere FEP AMR

documented two areas of work conducted by the M&O staff: 1) the screening analysis for FEPs

that are potentially biosphere related; and 2) the adequacy of the scientific bases for the Yucca

Mountain Project (YMP) biosphere model. The screening analysis included the screening

decision, screening argument, and recommended TSPA utilization for biosphere-related FEPs.

Validation of the YMP biosphere model, GENII-S, was performed in accordance with AP-3.10Q

to ensure that the model is appropriate and adequate for its intended use for Yucca Mountain.

The audit did not cover the second part of the AMR, which discussed the adequacy of the

scientific bases for the YMP biosphere model. That section of the AMR described the validation

of the YMP biosphere model, GENII-S, to ensure that the model is appropriate and adequate

for its intended use for Yucca Mountain.

The scope of all of the FEP AMR audits included the evaluation of the FEPs screening process,

screening decision, screening argument, and recommended TSPA utilization. The audit

evaluations included a combination of procedural and technical inquiries to verify that the quality

assurance procedures were followed in the development of the documents and that the

technical quality of the products was acceptable. The procedural inquiry primarily focused on

areas of 1) planning and implementation; and 2) integration. The technical areas of inquiry

included the 1) assumptions and criteria used; and 2) rationale for inclusion and exclusion of

FEPs.

The audit team reviewed the process used to create the FEPs reports. For all process-level

AMRs, the existing overall list of FEPs was used from which recommendations for change and

modification were made. In several reports, FEPs were added or modified, but in others

several FEPs were only shifted to and from other locations (e.g., microbial corrosion moved out

of biosphere and soil type moved into biosphere).

The audit team reviewed the qualifications of the document authors and the processes that

were followed in identifying included and excluded FEPs in the reports and the database. The

team also examined the technical basis/rationale supporting inclusion and exclusion of these

FEPs. The audit team also inquired about how the planning document was used in tracing

FEPs screening arguments and decisions. The audit team focused the discussions on the

M&O’s rationale for inclusion and exclusion of various FEPs. The audit team stated that the

scope was not to investigate the adequacy of logic used to screen FEPs, since that was

reviewed during the review of each individual AMR.

The overall FEPs AMR (ANL-WIS-PA-000002) was audited first, followed by the audit of

remaining process-level FEP AMRs. The audit team performed reviews for: 1) assumptions

and criteria used and 2) rationale for inclusion and exclusion of FEPs. In doing so, the audit

team selected both included and excluded FEPs. Neither the audit team nor the observers

identified any problems with the selection of FEPs.

Overall, the audit team was satisfied with the implementation of the process and rationale for

inclusion and exclusion of FEPs. The audit team, however, found deficiencies in interaction

among various groups to ensure consistency in rationale for inclusion or exclusion of a FEP.

For example, there was a lack evidence of interactions among safety, design, and

implementation groups. The observers considered the audit to be effective, and concurred with

the findings of the audit team.
17

4.6 Nuclear Regulatory Commission Staff Findings

The observers determined that OQA Audit M&O-ARP-00-013 was effective. The observers

agreed with the audit team conclusion that the OCRWM QA program was effectively

implemented except for model validation for the AMRs supporting the TSPA-SR, and that the

effectiveness of the TSPA Model Report (in draft during this audit) will be determined during the

second phase audit. The observers agreed with all other audit team conclusions, potential

deficiency reports, findings, and recommendations.

The observers agreed with the technical findings of the audit team. In addition, the observers

identified one other issue that was discussed with the audit team but not specifically discussed

at the audit exit meeting. The observers noted that, when reviewed collectively, the potential

deficiencies indicate a potential programmatic breakdown with the implementation of the QA

Program. Specifically, deficiencies identified by the audit team included failure to revise

planning documents in accordance with AP-2.13Q and AP-3.10Q, failure to maintain model

information the Model Warehouse in accordance with AP-3.10Q, and failure to validate all

models in accordance with AP-3.10Q. The observers were concerned that failure to follow

procedures continues to be a weakness.

This problem was most recently documented by DOE in its “OCRWM QA Trend Report for

Quality Program Deficiencies First Semester 2000,” dated August 10, 2000. The report reviews

trends for deficiencies identified between January 1, and June 30, 2000. In that report DOE

states “it is concluded that the majority of this semesters issues continue to be personnel error

related to failure to follow procedure and inattention to detail.” The observers recommend that

DOE management continue to focus attention on procedural compliance.

The following were technical findings identified by the audit team. These findings relate to

technical issues in the documents reviewed and are not reflective of DOE’s implementation of

its QA program. These items were discussed in Section 4.5 of this report and are being

highlighted again in this section to stress the observers’ agreement on the importance of the

issues.

4.6.1 General Technical Findings

1. Uncertainties and assumptions identified in lower tier TSPA-related level AMRs were not

communicated in the successive tier documents, such as the abstraction AMRs and

TSPA model report.

2. TSPA-SR model validation has not been completed nor documented with the exception

of the seepage model abstraction. Model validation of scientific investigation is a

requirement of the QARD.

3. Some TSPA-SR components (e.g., dissolved concentration limits) were not sufficiently

integrated into the TSPA-SR model. The reports did not provide an adequate technical

basis for not fully coupling the dissolved concentration limits with the in-drift

geochemical environment and the in-package chemistry.
18

4. The conservatism of assumptions and the conservatism of selection among alternative

conceptual models did not appear to be based on comparison to the peak mean dose

(i.e., the risk metric). The conservative elements of the assumptions were not clearly

identified, nor were they always intuitive.

5. Uncertainty was not consistently addressed in performance assessment component

models stochastic analyses. When data are limited:

• uncertainty should be assigned a high value to reflect the statistical uncertainty in

the parameter,

• conservative values should be used, and/or

• more data should be collected to reduce uncertainty.

In addition, the confidence in the selection of parameter ranges does not appear to have

been statistically tested.

4.6.2 Specific Technical Findings

1. Analysis Model Report—Abstraction of Seepage into Drifts (ANL-EBS-000005)

Open issues, such as those identified in NRC’s IRSR and in DOE’s peer review of

TSPA, should be directly addressed within the content of the affected AMR.

2. Analysis Model Report—WAPDEG Analysis of Waste Package and Drip Shield

Degradation (ANL-EBS-PA-000001)

The conceptual model of the waste package has a great deal of uncertainty, particularly

in terms of material properties and fundamental mechanisms of waste package

corrosion. Therefore, the underlying base of data and understanding of the conceptual

models may be inadequate for the purposes to which WAPDEG will be applied. The

GVP, while a reasonable approach for examining the importance of variability and

uncertainty in key data, should be carried through to the final results of the system

performance assessment. In addition, the hypothesis of uncertainty and variability

should be statistically tested for each set of data where it may be applied for Type I or II

errors.

3. FEP AMR Activities

• The status of whether the FEP database is ‘quality-affecting’ or not should be

resolved. While the database is abstracted data from approved AMRs, appropriate

control to assure consistency between the database and contributing AMRs

(especially as the AMRs are revised) may be important.

·

• Limited interaction between FEP AMR developers from different disciplines and

between FEP AMR and ‘process’ AMR developers may lead to inconsistent criteria

applied for including or excluding an FEP.
19

• The definitions for FEP inclusion and exclusion need to be clearer. Additional FEP

inclusion/exclusion categories may be necessary. For example, the FEP ‘radiation

damage’ is excluded; however, radiation damage is used as a basis for selecting

amorphous phases for solubility controls.

4.6.3 Audit Observer Inquiries

No audit observer inquiries were issued.

4.6.4 Open NRC Audit Observer Inquires

No NRC audit observer inquiries were open at the conclusion of this observation.

5.0 REFERENCES

DOE/RW-0333P, “Quality Assurance Requirements Document,” Revision 9.