NRC OBSERVATION AUDIT REPORT NO. OAR–04–01, “OBSERVATION AUDIT OF OFFICE

OF QUALITY ASSURANCE AUDIT OQAP–BSC–03–10 OF BECHTEL SAIC COMPANY, LLC”

 /RA/ 12/5/03 /RA/ 12/5/03

Robert Latta Richard B. Codell

Repository Site Section Performance Assessment Section 

High-Level Waste Branch Environmental and Performance

Division of Waste Management Assessment Branch

Division of Waste Management

Reviewed and Approved by:

 /RA/ 12/9/03

N. King Stablein, Chief

Projects and Engineering Section

High-Level Waste Branch

Division of Waste Management
2

1.0 INTRODUCTION

Staff from the U.S. Nuclear Regulatory Commission (NRC), Division of Waste Management,

observed the U.S. Department of Energy (DOE), Office of Quality Assurance (OQA), audit

OQAP–BSC–03–10, on October 21–31, 2003, at the Bechtel SAIC Company, LLC (BSC) facility

in Las Vegas, Nevada. This performance-based audit assessed the Analysis and Model Reports

(AMRs) and AMR development processes associated with the Yucca Mountain Project License

Application (LA) and the BSC implementation of the Office of Civilian Radioactive Waste

Management Quality Assurance Requirements and Description (QARD), DOE/RW–0333P,

Revision 12, regarding technical product development. The NRC observers (observers)

assessed the effectiveness of the audit team and the audit process in achieving the two

audit objectives.

2.0 MANAGEMENT SUMMARY

The audit team assessed the effectiveness and implementation of applicable AMR procedures,

processes, and products used to support the LA. The audit team performed a vertical-slice

evaluation of six AMRs followed by a horizontal review of critical processes drawing from the

total population of completed AMRs. The audit of the model validation process contributed

findings for the follow-up of Corrective Action Request (CAR) BSC–01–C–01.

At the time of the audit exit meeting, the audit team identified 17 conditions adverse to quality

and 6 less severe conditions, including 2 corrected during the audit, and concluded that, overall

• The adequacy of procedures in implementing QARD requirements was satisfactory.

• The implementation of procedures was unsatisfactory.

• The effectiveness of processes (when implemented as specified) and the effectiveness

of AMRs in meeting their intended objectives were satisfactory with some instances of

minor nonconformance.

At the audit exit meeting, the Director of OQA indicated that CAR BSC–01–C–01 would not be

closed because of the findings identified during the audit and during other followup activities.

The overall conclusion of unsatisfactory implementation of procedures by the audit team was

due, in part, to the unsatisfactory implementation of the critical processes of (1) documentation

and traceability of data, (2) model validation, and (3) AMR checking and review. In addition, one

of the six AMRs reviewed was determined to be unsatisfactory for both implementation

and effectiveness.

The observers determined the audit team was effective in meeting the objectives established in

the audit plan. The observers agreed with the audit team conclusions, findings, and

opportunities for process improvement. The observers expressed concern regarding the

repetitive nature of model validation issues.
3

3.0 AUDIT PARTICIPANTS

DOE Audit Team Members

Bruce Foster, Navarro Quality Services (NQS), Audit Team Leader

Don Harris, NQS, Auditor

John Doyle, NQS, Auditor

Marlin Horseman, NQS, Auditor

Judith Shipman, BSC, Auditor

James Voight, NQS, Auditor

Harvey Dove, NQS, Technical Specialist

John Savino, Management and Technical Support (MTS), Technical Specialist

Paul LaPointe, Golder and Associates, Inc., Technical Specialist

Steve Marks, Golder and Associates, Inc., Technical Specialist

Don West, Golder and Associates, Inc., Technical Specialist

Arthur Stein, Shaw Stone and Webster, Technical Specialist

Tracy Ikenberry, Dade Moeller and Associates, Technical Specialist

Observers

Robert Latta, NRC, Observation Team Leader

Abou-Bakr Ibrahim, NRC, Technical Specialist (week 1)

Richard Codell, NRC, Technical Specialist 

Robert Brient, Center for Nuclear Waste Regulatory Analyses (CNWRA), Quality

Assurance (QA) Specialist

Thomas Trbovich, CNWRA, QA Specialist

Yi-Ming Pan, CNWRA, Technical Specialist

Gary Walter, CNWRA, Technical Specialist (week 1)

Brittain Hill, CNWRA, Technical Specialist (week 1)

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION

The audit was conducted in accordance with AP–18.3Q, Internal Audit Program, and AP–16.1Q,

Management of Conditions Adverse to Quality. The audit team used the QARD and applicable

QARD and BSC implementing procedures to generate the audit checklist. The observers

followed NRC Manual Chapter 2410, Conduct of Observation Audits, July 12, 2000, while

observing the audit.

4.1 Scope of the Audit

The audit focused on the Scientific Investigation (Supplement III) element of the QARD. The

audit included evaluating the following two procedures:

• AP–SIII.9Q, Scientific Analysis

• AP–SIII.10Q, Models
4

The period October 21–24, 2003, involved a vertical-slice audit of six AMRs selected for their

status as recently issued or having completed the checking process:

• Biosphere Model Report, MDL–MGR–MD–000001 (Issued)

• Site-Scale Saturated Zone Transport Model, MDL–NBS–HS–000010 (through checking

process only)

• Seepage Calibration Model and Testing Data, MDL–NBS–HS–000004 (Issued)

• Atmospheric Dispersal and Deposition, MDL–MGR–GS–000002 (through checking

process only)

• Stress Corrosion Cracking of Drip Shield, Waste Package Outer Barrier, and Stainless

Steel Structural Material, ANL–EBS–MD–000005 (Issued)

• Seismic Consequence Abstraction, MDL–WIS–PA–000003 (Issued)

The horizontal-slice assessment of the adequacy, implementation, and effectiveness of critical

processes included samples of (approximately 40) completed AMRs. The critical

processes were:

• Planning and Development

• Documentation and Traceability (of data)

• Use of Data

• Use of Software

• Model Validation

• AMR Checking and Review

• Procedure Adequacy

The audit of the model validation process was used, in part, to provide information for the

evaluation of the corrective actions associated with CAR BSC–01–C–01.

4.2 Conduct and Timing of the Audit

The observers determined the timing of the audit was appropriate because an adequate number

of technical products were completed or sufficiently through checking and review. The audit had

been rescheduled from July 2003 so that more products would be available for review. Auditors

used “Performance-Based Audit Worksheets” to identify the important steps and the methods to

measure performance. Portions of the audit checklists were prepared based on these

worksheets. Checklists were provided to BSC a few days in advance of the audit. This practice

is not typical and may have adverse impacts (i.e., potential bias in materials presented for

review) on the audit process, so an Audit Observer Inquiry (AOI) was issued (see Section 5.2.3).

The audit team and observers caucused at the end of each day to discuss the audit status and

any new and developing issues. The audit team met with BSC management daily, as

appropriate, to discuss the audit status and potential issues. During the final caucus of the

audit, the audit team jointly classified the Condition Reports and the conclusions related to

adequacy, implementation, and effectiveness. Condition Reports were defined as

• Level A: Significant condition adverse to quality;

• Level B: Condition adverse to quality;

• Level C: Minor adverse condition with no stated noncompliance with the QARD or

conditions closed during the audit; and

• Level D: Opportunity for improvement.
5

4.3 Audit Team Qualifications and Independence

The observers reviewed the qualifications for the Audit Technical Specialists and determined the

Audit Technical Specialists were qualified and independent of the areas reviewed. For one of

the AMRs (see Section 4.5.4), the full scope of the AMR was not covered by the qualifications of

the assigned Technical Specialist, so the audit results were limited to his range of expertise. 

During previous audits, qualifications of the Audit Team Leader and Auditors had been reviewed

and were determined acceptable.

4.4 Examination of Quality Assurance Elements

Examination of the QA elements occurred during both phases of the audit, during the

vertical-slice audits of six AMRs in the first week and during the horizontal-slice audits of critical

processes in the second week. The population of AMRs from which the horizontal-slice audits

sampled included all completed AMRs.

4.4.1 Planning and Development

The audit included planning and integration of modeling and analysis activities; implementation

of planning documentation; identification and inclusion/exclusion of features, events, and

processes (FEP); and the extent to which Key Technical Issues (KTI) were addressed in

the AMR. The audit team findings for both phases are listed in section 4.4.1through 4.4.7.

The auditors: 

• Compared the QARD requirements with those contained in AP–2.27Q, Revision 1,

ICN 1, Planning for Scientific Activities;

• Selected seven AMRs at random from the listing of completed AMRs in the audit plan for

a compliance review to AP–2.27Q requirements;

• Reviewed the Technical Work Plans (TWPs) to assure incorporation of FEPs and KTIs

• Reviewed the selected AMRs for incorporation of TWP requirements; and

• Selected 10 personnel involved with the selected AMRs and confirmed the Verification of

Education and Experience had been properly completed for each.

The auditors identified:

• The Stress Corrosion Cracking AMR did not meet specified TWP requirements, and

variances from TWP requirements were not explained adequately in the AMR (Condition

Report Level B); and

• One opportunity for improvement concerning consistency of the KTI information in the

TWPs and AMRs (Condition Report Level D).

The auditors determined this process to be:

• Satisfactory for adequacy of procedures;

• Satisfactory for implementation; and

• Satisfactory for effectiveness of the process.
6

The observer determined the audit of this process was effective and agreed with the conclusions

of the auditors.

4.4.2 Documentation and Traceability

The audit evaluated preparation of documentation, records development and submittal, and

traceability to other technical products, data, and issues. The audit also evaluated the

transparency and traceability of identified model features serving as input to LA products. The

AMR procedures (AP–SIII.9Q and AP–SIII.10Q) were used as the basis for checklist

requirements.

The auditors:

• Selected five AMRs at random from the list of completed AMRs provided in the audit plan

and tracked the version/configuration controls through a review of the selected AMR

records packages;

• Reviewed the AMR sections for transparency in purpose, analysis, and results

(i.e., software use defined; product inputs defined; uncertainties and restrictions

discussed and evaluated; and assumptions, constraints, bounds, or limits defined and

impacts described);

• Selected references in the AMR and verified specific sections within the references

support statements made in the AMR;

• Reviewed records packages for proper completion of reviews or concurrences; and

• Verified any Technical Error Reports initiated had been properly closed.

The auditors identified:

• Several instances of transparency/defensibility problems in the Stress Corrosion

Cracking AMR (Condition Report Level B);

• Assumptions lacking documentation within the specified section of the Site-Scale

Saturated Zone and Seepage Calibration Model AMRs (Two Condition Reports Level B);

• Pertinent validation of test results appended to the test plan rather than to the test report 

(Condition Report Level B);

• Statements of fact and cited information not confirmed in the identified references for the

Atmospheric Dispersal and Deposition AMR (Condition Report Level B);

• All required records not contained in the records package for the Seismic Consequence

Abstraction AMR (Condition Report Level B);

• Fault selection criteria basis not provided in the Geologic Framework Model AMR 

(Condition Report Level B); and

• One opportunity for improvement (Condition Report Level D).

The auditors determined this process to be:

• Satisfactory for adequacy of procedures;

• Unsatisfactory for implementation; and

• Satisfactory with minor discrepancies for effectiveness of the process.

The observer determined the audit of this process was effective and agreed with the conclusions

of the auditors.
7

4.4.3 Use of Data

The audit included data selection and use to support analyses or model objectives. Data

sufficiency and the technical bases to support model development were evaluated and

confirmed. Data uncertainty was evaluated through consideration of the technical bases for

parameter values, assumed ranges, probability distributions, and bounding values to support

expressed conservatism.

The auditors selected AMRs for evaluation to match the technical expertise of the Technical

Specialists, namely, materials science and engineering geology.

The auditors:

• Reviewed the AMRs to evaluate how the data were being used;

• Searched the Technical Data Management System to track the data and data

qualification statuses; and

• Discussed data issues with AMR teams and other BSC staff.

The auditors identified:

• Values meeting the definition of data were classified as Technical Information (i.e., not

requiring qualification) (Condition Report Level B);

• A data tracking number (DTN) qualification status was listed as Technical Information,

which is not a valid status (Condition Report Level B);

• A DTN providing borehole elevation data was not referenced in the Geologic Framework

AMR that used the data (Condition Report Level B); and

• Multiple data problems were included in the Effects of Fault Displacement on

Emplacement Drifts AMR (Condition Report Level B).

The auditors determined this process to be:

• Satisfactory for adequacy of procedures;

• Satisfactory with minor discrepancies for implementation; and

• Satisfactory for effectiveness of the process.

The observer determined the audit of this process was effective and agreed with the conclusions

of the auditors.

4.4.4 Use of Software

The audit evaluated software selection and use to support analyses or model objectives. The

checklist was developed from the Performance-Based Audit Worksheet and generally was

based on software control procedures. The auditors selected five AMRs having a cross section

of software [i.e., developed (by BSC), commercially available, process level, and abstraction

level]. In all, six software items were audited.

The auditors:

• Reviewed Software Management Reports;

• Had discussions with principal investigators and software control staff; and

• Investigated training of software developers and users.
8

The auditors identified:

• No documentation supporting software use within its validated range for two AMRs 

(Condition Report Level B);

• No verification of education and experience available for a user/developer of the

Geologic Framework Model (Condition Report Level B); and

• One opportunity for improvement (Condition Report Level D).

The auditors determined this process to be:

• Satisfactory for adequacy of procedures;

• Satisfactory for implementation; and

• Satisfactory for effectiveness of the process.

The observer determined the audit of this process was effective and agreed with the conclusions

of the auditors.

4.4.5 Model Validation

The audit determined if models performed consistently with their documented purpose,

limitations, and bounding conditions. In addition, the extent to which the models were

adequately supported by data and information was audited, as was the model uncertainty

through examination of model parameters in light of available data, natural analog information,

and process model studies to support expressed conservatism. The audit included followup of

CAR BSC 01–C–01 regarding model validation and discussions with staff from the Chief

Science Office regarding model accuracy, level of confidence, and uncertainty. The audit

checklist used AP–SIII.10Q and CAR BSC–01–C–01 as sources of requirements. The audit

team chose to evaluate 20 out of the approximately 40 completed AMRs. 

The corrective action for CAR BSC–01–C–01 involved classifying AMRs into bins related to the

types of deficiencies. The sample of AMRs audited included all eight of the Bin 2 (models

containing technical issues) AMRs, seven of the Bin 3 (models with technical issues and

validation documentation incomplete) AMRs and five new AMRs not included in the binning.

The auditors:

• Reviewed TWPs and records packages;

• Had discussions with principal investigators and Chief Science Office management; and

• Investigated training of modelers and checkers.

The auditors identified:

• The redistribution model in the Atmospheric Dispersal and Deposition AMR was not

validated as required in the TWP (Condition Report Level B);

• The model used in the Stress Corrosion Cracking AMR had not been validated for Yucca

Mountain Conditions or for Alloy C-22 (Condition Report Level B);

• TWPs and AMRs did not consistently describe the criteria for model validation (Condition

Report Level B); and

• One model used in the Site-Scale Saturated Zone Model AMR needed additional

validation (Condition Report Level C).
9

The auditors determined this process to be:

• Satisfactory for adequacy of procedures;

• Unsatisfactory for implementation; and

• Satisfactory with minor discrepancies for effectiveness of the process.

The observer determined the audit of this process was effective and agreed with the conclusions

of the auditors.

4.4.6 Analysis and Model Checking and Review

This audit area addressed the process of documenting the interdisciplinary reviews of AMRs,

technically checking data, and documenting the impacts of changed inputs on the products.

The auditors:

• Interviewed the originator, checker, Quality Engineering Representative reviewer, and

AP–2.14Q reviewers for each AMR;

• Evaluated the technical competency and training of the checker and the review criteria

specified for the applicable AMR; and

• Reviewed the record packages to determine that all mandatory comments were resolved

and incorporated into each AMR.

The auditors identified:

• Requirements to perform checking activities using criteria from the TWP were not

followed for the Atmospheric Dispersal and Deposition AMR (Condition Report Level B);

• Technical and editorial errors were found in five AMRs; one error was corrected during

the audit (Five Condition Reports Level C); and

• One opportunity for improvement (Condition Report Level D).

The auditors determined this process to be:

• Satisfactory for adequacy of procedures;

• Unsatisfactory for implementation; and

• Satisfactory with minor discrepancies for effectiveness of the process.

In part, the auditors based their conclusion of unsatisfactory implementation on the large number

of errors in documentation and traceability and in model validation not detected during checking

and review.

The observer determined the audit of this process was effective and agreed with the conclusions

of the auditors.

4.4.7 Procedure Adequacy

This audit area evaluated the extent that the QARD requirements were addressed by the

procedural controls.
10

The auditors:

• Assessed development, review, effectiveness and training performed with the revision of

procedures AP–SIII.9Q, Scientific Analyses; and AP–SIII.10Q, Models, through a review

of the records packages and discussions with the Chief Science Officer and author;

• Verified incorporation of the QARD requirements into the revised procedures; and

• Discussed use of the uncontrolled Scientific Processes Model Manual with the Chief

Science Officer.

The auditors identified:

• Lack of QA program controls throughout the widespread use of informal reviews 

(Condition Report Level B); and

• One opportunity for improvement (Condition Report Level D).

The auditors determined procedures to be adequate.

The observer determined the audit of this process was effective and agreed with the conclusion

of the auditors.

4.5 Examination of Technical Activities

The conclusions of the audit team regarding the six AMRs evaluated during the first week were

expressed regarding (1) the implementation of procedure requirements and (2) the effectiveness

of the AMRs in meeting their intended objectives. Procedure adequacy was not evaluated in the

examination of technical activities, as this is only relevant in the context of the examination of the

QA elements. All of the findings in this section are also listed in sections 4.4.1 through 4.4.7. 

Cross-references to those sections are also provided.

4.5.1 Biosphere Model Report

The Biosphere Model Report described and justified the equations and construction of the

Environmental Radiation Model for Yucca Mountain Nevada (ERMYN) and provided validation of

the model. ERMYN has been implemented through the commercially available GoldSim®

software. Execution of the mathematical model has been performed and reported in five AMRs

for biosphere input parameters and two AMRs for Biosphere Dose Conversion Factors (BDCF). 

The BDCF AMRs provide input to the Total System Performance Assessment–Licence

Application (TSPA-LA). Because the mathematical execution of the model was accomplished in

other AMRs, the Biosphere Model Report used no Direct Input data.

The auditors prepared a checklist containing technical and QA-related items, the technical items

being the majority. Noteworthy was the use of the Yucca Mountain Review Plan as one of the

resources for the technical checklist items. The QA checklist items generally followed the critical

process steps appropriate to the specific scope of the Biosphere Model Report. 

The audit process included:

• Discussions with the AMR development team and completion of the checklist items;

• Demonstration of the ERMYN code and its depiction of the model;

• Use of the data management system to trace references and data used in the AMR;

• Review of records packages, including checking and review documentation; and

• Review of several associated parameter and BDCF AMRs for consistency.
11

The auditors identified:

• Exemplary practices in (1) transparency of the representation of ERMYN as implemented

in GoldSim® and (2) sensitivity analyses that should provide additional confidence in

the model; and

• Three opportunities for improvement to enhance clarity of the AMR (Condition Report

Level D).

The auditors determined this AMR to be:

• Satisfactory for implementation; and

• Satisfactory for effectiveness for its intended use.

The observer determined the audit of this AMR was effective and agreed with the conclusions of

the auditors.

4.5.2 Site-Scale Saturated Zone Transport Model

The Site-Scale Saturated Zone Transport Model report:

• Provided an update to the site-scale advection-dispersion transport model with

matrix diffusion;

• Provided a description and validation of the transport model;

• Described numerical methods for simulating radionuclide transport;

• Documented parameters and uncertainty distributions; 

• Documented parameters for modeling colloid-facilitated transport;

• Described alternative conceptual models.

The report provided a basecase simulation of cumulative mass flow breakthrough curves across

the 18-km compliance boundary and simulations of the breakthrough curves based on ranges of

the various transport parameters. The model was intended for use in TSPA. The flow field used

in the transport model was derived from the site-scale saturated zone flow model. At the time of

the audit, the AMR had been processed only through checking.

The auditors prepared a checklist containing both technical and QA-related items. The checklist

emphasized the identification of assumptions and proper qualifications of data. The checklist

also identified specific needs for additional validation of the colloid facilitated transport portions

of the model. The audit process included:

• Discussion of the checklist items with the principal author and key members of the AMR

development team;

• Discussion of the guidance used to develop the AMR content, specifically with respect to

identification of assumptions;

• Use of the data management system to trace and verify references; and

• Inspection of documents to support model validation.

The auditors identified:

• Two conditions adverse to quality related to use of data (Condition Report Level B, see

Section 4.4.3);

• One condition adverse to quality related to documentation and traceability (Condition

Report Level B, see Section 4.4.2);
12

• Three minor conditions adverse to quality, one of which was corrected during the audit

(Condition Report Level C); and

• One opportunity for improvement for additional clarity in the report (Condition Report

Level D). 

The auditors determined this AMR to be:

• Satisfactory for implementation with minor discrepancies; and

• Satisfactory for effectiveness for its intended use.

The observer determined:

• The auditors probed adequately into the items identified in the checklist; and

• The auditors adequately pursued questions regarding model verification raised by

the observer.

The observer agreed with the conclusions of the auditors but has residual concerns regarding

the transparency of the AMR with respect to model validation.

4.5.3 Seepage Calibration Model and Testing Data

The seepage calibration model developed parameters for the seepage models to predict

seepage of meteoric water into the emplacement drifts for at least 10,000 years. Results of the

predictive model would then be reduced to a look-up table for subsequent use in the TSPA-LA. 

The report describes the model, computer codes, experimental techniques, assumptions in the

calibration, and analyses of data used to develop parameters from the data.

The auditors prepared a checklist containing technical and QA-related items. The checklist

emphasized the identification of assumptions and proper qualifications of data. The audit

process included:

• Discussion of the checklist items with the principal author and key members of the AMR

development team;

• Discussion of the guidance used to develop the AMR content, specifically with respect to

identification of assumptions;

• Use of the data management system to trace and verify references, scientific notebooks,

and data;

• Inspection of documents to support model validation; and

• Review of previous versions of documents to identify what was changed and why.

The auditors identified inconsistencies between the guidance provided to authors and the QA

requirements with respect to the documentation of assumptions. 

The auditors identified:

• One condition adverse to quality related to documentation and traceability (Condition

Report Level B, see Section 4.4.2); and

• One minor condition adverse to quality (Condition Report Level C). 
13

The auditors determined this AMR to be:

• Satisfactory for implementation; and

• Satisfactory for effectiveness for its intended use.

The observer determined:

• The auditor and technical expert probed adequately into the items identified in

their checklist; and

• The auditors adequately pursued questions regarding model validation raised by

the observer.

The observer determined the audit was effective and agreed with the conclusions of

the auditors.

4.5.4 Atmospheric Dispersal and Deposition

The Atmospheric Transport and Deposition AMR described and justified use of the DOE

conceptual model for potential eruption and transport of volcanic tephra, along with numerical

relationships used to develop some model parameters. In addition, this AMR presented a

conceptual model for the remobilization of tephra deposits following a potential volcanic eruption

at the proposed repository site. Output from these models will provide input for the TSPA-LA. 

Model parameters were developed in other AMRs, with the majority of information derived from

an in-process revision to the AMR, Characterize Eruptive Processes in the Yucca

Mountain Region. 

Observers reviewed the qualifications of the audit team. An AOI (see Section 5.2.1) was

generated because of concern with an apparent gap in the technical expertise of the audit team. 

The audited AMR included development of a conceptual model for posteruption remobilization of

potential volcanic tephra deposits, which is a model with apparent high risk significance. A

technical specialist with expertise appropriate to review this particular model was not included in

the audit team. As a result of the AOI, the scope of the audit and the audit conclusions were

limited to the ASHPLUME model and the remobilization model was excluded.

The auditors prepared a checklist containing technical and QA-related items. Compared with

other checklists in this audit, QA-related items appeared to represent the majority. A noteworthy

audit question (#11-22) regarded demonstration of compliance with 10 CFR 63.114 and Yucca

Mountain Review Plan Acceptance Criteria for Airborne Transport of Radionuclides. The

audit process included:

• Discussions with lead authors of the AMR;

• Independent review of the AMR;

• Discussion with the TSPA staff on implementing the ASHPLUME model in TSPA-LA;

• Data management system use to trace references and data used in the AMR;

• Records package reviews, including checking and review documentation; and

• Data consistency checks with supporting AMRs.

The auditors identified:

• Three conditions adverse to quality (all Condition Report Level B), one related to

documentation and traceability (see Section 4.4.2), one related to model validation (see

Section 4.4.5), and one related to checking and review (see Section 4.4.6);
14

• One technical error (Condition Report Level C); and

• One opportunity for improvement (Condition Report Level D).

The auditors determined the ASHPLUME model presented in the Atmospheric Transport and

Deposition AMR, to be:

• Satisfactory for implementation of the procedures; and

• Satisfactory for effectiveness for its intended use.

The observer determined the audit of this AMR was effective and agreed with the conclusions of

the auditors.

4.5.5 Stress Corrosion Cracking of Drip Shield, Waste Package Outer Barrier, and

Stainless Steel Structural Material

This AMR developed process-level models for assessing waste package outer barrier and drip

shield degradation because of stress corrosion cracking. The stress corrosion cracking model

abstraction included a waste package manufacturing defects abstraction model, a residual

closure weld stress finite element model, a crack growth model based on the slip dissolution/film

rupture mechanism, and a determination of the threshold stress and stress intensity for crack

initiation. Provided no time delay for crack propagation, through-wall stress corrosion cracking

of the waste package outer barrier was assumed to occur once the very conservative initiation

and propagation threshold criteria were exceeded. The stress corrosion cracking degradation

mode, however, was precluded because tensile residual stresses were assumed to be removed

in the stress mitigation process. Output from this AMR can be used as input for the Integrated

Waste Package Degradation Model AMR. This AMR also will provide input for TSPA analysis. 

The auditors prepared a checklist containing technical and QA-related items. The checklist

emphasized the proper qualifications of data and model validation. The checklist also identified

specific needs for assessing the effects of temperature, compositional variation, and fabrication

processes on stress corrosion cracking. The audit process included:

• Interviewing the AMR originator and key team members to discuss the checklist items;

• Interviewing the test material vendor for alloy fabrication;

• Discussing model validation with BSC management and Chief Science Office staff;

• Checking TWP requirements for acceptance criteria, FEPs, and model validation criteria;

• Using the Technical Data Management System (TDMS) to trace DTNs, references, and

review comment sheets; and

• Reviewing records packages to determine data consistency and check documentation.

The auditors identified:

• Three conditions adverse to quality (all Condition Report Level B) related to planning and

development (see Section 4.4.1), documentation and traceability (see Section 4.4.2), and

model validation (see Section 4.4.5);

• One minor condition related to checking and review (see Section 4.4.6) (Condition Report

Level C); and

• One opportunity for improvement (Condition Report Level D).
15

The auditors determined this AMR to be:

• Unsatisfactory for implementation; and

• Unsatisfactory for effectiveness for its intended use.

The observer determined the audit was effective and agrees with the conclusions of the auditors. 

During numerous interview discussions, the observer noted BSC considers further stress

corrosion testing to be unnecessary to evaluate the effects of compositional variation and

fabrication processes. BSC believes further evaluation is not appropriate for this AMR, but

believes evaluation should be handled as part of fabrication and weld process design. The

observer recommends BSC consider adding data and analysis to strengthen the defensibility of

the AMR and to fulfill the DOE and NRC agreements. 

4.5.6 Seismic Consequence Abstraction

This abstraction developed the response of the Engineered Barrier System components to

seismic hazards (vibratory ground motion, fault displacement, and cladding) at Yucca Mountain,

Nevada. Also, this AMR defined the methodology for using these abstractions in the seismic

scenario class for TSPA-LA. Results from this abstraction also will address integrated subissues

dealing with mechanical disruption of the engineered barriers.

The auditors prepared checklists concentrating mainly on technical issues discussed in the

abstraction. The audit process included:

• Interviewing the principal investigator and management and Chief Science Office staff

from BSC;

• Checking that the AMR refers to the appropriate sections of the Yucca Mountain

Review Plan;

• Using the TDMS to trace DTNs used in the AMR;

• Checking review procedures used by informal and formal reviewers;

• Examining the credentials and qualifications of the technical checkers; and

• Examining whether the AMR followed the last issued TWP.

The auditors identified:

• An exemplary practice in the effective interface between the AMR team and the TSPA-LA

modeling team;

• One condition adverse to quality, the AMR records package did not contain all required

review records (Condition Report Level B, see Section 4.4.2); and

• Three opportunities for improvement for additional clarity in the AMR (Condition Report

Level D).

The auditors determined this AMR to be:

• Satisfactory for implementation; and

• Satisfactory for effectiveness for its intended use.

The observer agreed with the conclusions of the auditors. The observer recommends all

informal comments on this AMR be kept in TDMS to ensure procedures in developing the AMR

are followed accurately (also see Section 5.2.2).

5.0 NRC STAFF FINDINGS
1

The high risk significance to this conceptual model is supported by (i) the credit apparently taken in TSPA

for redirection of potential airborne contaminant plumes away from the location of the reasonably maximally exposed

individual in approximately 80 prcent of the calculations, (ii) the Biosphere Technical Basis Document that credits

rapid reduction in airborne particle concentrations and resulting inhalation dose in posteruption risk calculations

based on results of remobilization modeling, and (iii) the high risk ranking of this topic in NRC Key Technical Issue

Agreement 2.17.

16

5.1 NRC OBSERVATION SUMMARY

• The audit team was effective regarding the assessment of technical products, the

assessment of technical product development processes, and determination of the level

of compliance of the BSC activities.

• The NRC observers agreed with the audit team conclusions, findings, and opportunities

for improvement.

• The observers determined the audit team members were qualified and independent of

the areas being audited.

• NRC is concerned with the repetitive nature of the conditions regarding model validation

that indicate corrective actions are ineffective.

• The observers determined closure of CAR BSC–01C–01 did not appear adequately

justified based on results of the verification activities.

• Organization and coordination of the audit and audit team were excellent. Audit status

communication was clear and timely. Auditor access to needed records was facilitated

by having several computers on-line in the audit room.

• Except for one period with extremely heavy demand, the BSC Technical Information

Center was able to respond to requests quickly.

• The unsatisfactory implementation of procedures suggests that corrective actions should

focus on compliance rather than on revising procedures.

5.2 NRC AUDIT OBSERVER INQUIRIES 

NRC generated three AOIs as a result of observing the audit. Responses to the AOIs are

requested within 45 days of completion of the audit.

5.2.1 Technical Specialist Coverage of the Audit Scope

• Long-term remobilization of potential volcanic tephra deposits is a process of apparent

high risk significance1

 to calculations of postclosure repository performance. The

conceptual model for tephra remobilization is presented in the Atmospheric Transport

and Deposition AMR [MDL–MGR–GS–000002 REV00F].

• Evaluation of potential tephra remobilization processes requires expertise in the general

field of surface-process geology, with specific knowledge of sedimentary processes in

arid terrains. Although the audit Technical Specialist assigned to audit this AMR has the

technical expertise to audit the atmospheric transport processes also included in this

AMR, his expertise does not include (by his own admission) processes associated with

tephra redistribution.

• The NRC staff are concerned this AMR evaluation may be inadequate to ascertain the 

technical adequacy of the conceptual model for potential tephra remobilization

processes. Specific concerns include

— Traceability in TWP for model development

— Transparency in AMR to support model adequacy

— Validity of model assumptions

— Tests of model hypotheses

— Appropriate evaluation of model uncertainties

— Consideration of alternative conceptual models
17

5.2.2 Informal Reviews 

During review of the Seismic Consequence Abstraction AMR, the audit team became aware of

the inappropriate use of undocumented technical reviews of a quality-affecting document. 

Specifically, the audit team determined that during the initial development of the Seismic

Consequence Abstraction AMR, the principal investigator forwarded the draft document to DOE

and MTS for comment, as well as to two additional unqualified reviewers. As established by the

audit team, the incorporation of these unofficial comments resulted in substantial changes to the

draft AMR. There was no record, however, of the review comments for this document or the

review efforts associated with this quality-affecting activity.

This practice is an apparent departure from the requirements of QARD Section 5.2, which states

quality-affecting “work shall be performed in accordance with controlled implementing

documents;” Section 6.2.4, which states “documents shall be reviewed in accordance with

Subsection 2.2.10, Document Review;” which states, in part, that implementing documents and

documents that specify technical or quality requirements “shall be reviewed in accordance with

established criteria using pertinent background information by technically competent individuals

other than the preparer, considering all aspects of the document.”

5.2.3 Checklist Distribution

In recent audits, the audit teams have provided checklists to auditees prior to the start of the

audit. NRC is concerned about the potentially adverse impacts of this practice on the audit

process and results.
18