Enclosure

U.S. NUCLEAR REGULATORY COMMISSION OBSERVATION AUDIT REPORT OAR–06–06,

OBSERVATIONS OF BECHTEL SAIC COMPANY, LLC, AUDIT BQA–BSC–06–09 OF

SCIENTIFIC INVESTIGATIONS FOR THE SEISMIC CONSEQUENCE ABSTRACTION

MODEL AND MAJOR INPUT CALCULATIONS

 /RA/ 10/06/06 /RA/ Thomas Matula for 10/06/06

Thomas Matula Jack Parrott

Division of High-Level Waste Division of High-Level Waste

 Repository Safety Repository Safety

Office of Nuclear Material Office of Nuclear Material

 Safety and Safeguards Safety and Safeguards

 /RA/ 10/06/06 /RA/ Thomas Matula for 10/06/06

Abou-Bakr Ibrahim Robert Brient

Division of High-Level Waste Center for Nuclear Waste 

 Repository Safety Regulatory Analyses

Office of Nuclear Material

 Safety and Safeguards

/RA/ Thomas Matula for 10/06/06 /RA/ Thomas Matula for 10/06/06

Luis Ibarra Michael Simpson

Center for Nuclear Waste Center for Nuclear Waste 

 Regulatory Analyses Regulatory Analyses

/RA/ Thomas Matula for 10/06/06

Thomas Wilt

Center for Nuclear Waste

 Regulatory Analyses

Reviewed and Approved by:

/RA/ N. King Stablein for 10/06/06

Frank Akstulewicz, Acting Deputy Director

Licensing and Inspection Directorate

Division of High-Level Waste

 Repository Safety

Office of Nuclear Material

 Safety and Safeguards
-2-

1.0 INTRODUCTION

On August 21–September 7, 2006, staff from the U.S. Nuclear Regulatory Commission (NRC),

Division of High-Level Waste Repository Safety, and the Center for Nuclear Waste Regulatory

Analyses (CNWRA) observed the Bechtel SAIC Company, LLC (BSC), internal quality

assurance audit BQA–BSC–06–09 in Las Vegas, Nevada. The title of the audit was Audit of

Scientific Investigations for the Seismic Consequence Abstraction Model and Major Input

Calculations. The scope of the audit was to evaluate model development and scientific

investigation activities regarding the Seismic Consequence Abstraction and to evaluate four

technical reports providing direct input to the abstraction. The technical reports subject to this

audit are identified in the table below. 

Technical Report Audited Abbreviated Title*

Seismic Consequence Abstraction

(MDL–WIS–PA–000003, Revision 02)

Abstraction Report

Mechanical Assessment of the Waste Package Subject to

Vibratory Ground Motion (CAL–WIS–AC–000001,

Revision 0B)

Waste Package Report

Mechanical Assessment of the Drip Shield Subject to

Vibratory Ground Motion and Dynamic and Static Rock

Loading (CAL–WIS–AC–000002, Revision 00A)

Drip Shield Report

Peak Ground Velocities for Seismic Events at Yucca

Mountain, Nevada (ANL–MGR–GS–000004, Revision 00)

Peak Ground Velocities Report

Development of Earthquake Ground Motion Input to

Preclosure Seismic Design and Postclosure Performance

Assessment of a Geologic Repository at Yucca Mountain,

Nevada (MDL–MGR–GS–000003, Revision 01) 

Ground Motion Report

* Abbreviated titles of the technical reports audited are used in this report.

The primary objective of this audit was to determine whether the inputs provided to the

Abstraction Report had been developed and calculated using methods that are technically

sound and defensible with adequate transparency and traceability. The audit team’s Technical

Specialists also evaluated the work to determine whether it met selected elements of the five

criteria in Section 2.2.1.3.2.3 of the Yucca Mountain Review Plan, Final Report, NUREG–1804,

Revision 2. The objective of the NRC observation was to assess whether the BSC audit team

effectively met the objective of the audit.

2.0 MANAGEMENT SUMMARY

During the conduct of the audit, the NRC observers identified a weakness in the audit planning

and preparation. The observers determined that the BSC auditors apparently reduced the

scope of the audit, and did not prepare sufficiently to meet the primary objective of the audit. 
-3-

Specifically, the observers concluded after the fourth day of the audit that the auditors were

deviating from the audit plan by not evaluating the Ground Motion Report, one of the technical

reports identified in the Audit Plan. The observers noted a lack of questioning of technical staff

responsible for the Ground Motion Report by the audit team’s Technical Specialists during

interviews, and the absence of associated checklist questions that should have been developed

by the Technical Specialists while preparing for the audit. The observers expressed concern to

the Audit Team Leader whether this reduced level of technical evaluation would be adequate to

meet the audit objective. As a result of the expressed observers’ concern, the Audit Team

Leader issued Condition Report 9035 reflecting inadequate audit planning. BSC immediately

extended the audit end date from August 31 to September 7, 2006, and added one additional

auditor to provide sufficient time and resources to evaluate the Ground Motion Report. With the

inclusion of evaluating the Ground Motion Report during the audit as originally planned, the

observers determined that the BSC audit team effectively met the objective of the audit. It is

noted that during the August 15-31, 2005, BSC audit BQAP–BSC–05–07, the NRC observers

identified a similar weakness in audit planning and preparation. The observers note that

improvement is needed in BSC’s planning and preparing for performance-based audits. 

At a Post-Audit Meeting with DOE and BSC management, the BSC audit team presented its

findings. The audit team found that, overall, the scientific investigations that provide input to the

Abstraction Report had been developed and calculated using methods that were technically

sound and defensible with adequate transparency and traceability. The audit team did not

identify significant issues related to scientific investigations supporting the Abstraction Report

but did identify documentation and transparency issues related to calculations which provide

input to the Abstraction Report. The observers agreed with the audit team’s findings,

recommendations, and overall conclusions. 

3.0 AUDIT PARTICIPANTS

BSC Audit Team Members

Roxanna VanDillen, Audit Team Leader

Kenneth Gilkerson, Auditor

Gerard Heaney, Auditor

Richard Maudlin, Auditor

Stephen Schuermann, Auditor

Charles Warren, Auditor

Quazi Hossain, Lawrence Livermore National Laboratory (LLNL), Technical Specialist

Lawrence Hutchings, LLNL, Technical Specialist

Observers

Thomas Matula, NRC, Observation Team Leader

Jack Parrott, NRC, Observation Team Co-Leader

Abou-Bakr Ibrahim, NRC, Technical Specialist

Robert Brient, CNWRA, Quality Assurance (QA) Specialist

Michael Simpson, CNWRA, QA Specialist

Luis Ibarra, CNWRA, Technical Specialist

Thomas Wilt, CNWRA, Technical Specialist
-4-

U.S. Department of Energy (DOE) Office of Quality Assurance

Bruce Foster, QA Observer, Navarro Quality Systems

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION

The auditors conducted the audit in accordance with BSC procedure QA–PRO–1046, QA

Internal Audit Program. The auditors identified issues (adverse conditions and

recommendations) in accordance with Administrative Procedure (AP)–16.1Q, Condition

Reporting and Resolution. The observers followed NRC Manual Chapter 2410, Conduct of

Observation Audits.

4.1 Scope of the Audit

The scope of the audit was to evaluate model development and scientific investigation activities

related to the Abstraction Report and utilize a vertical slice technique to evaluate activities with

a focus on four technical reports that provided input to the Abstraction Report. The primary

objective of the audit was to determine whether inputs provided to the Abstraction Report had

been developed and calculated using technically sound and defensible methods with adequate

transparency and traceability. The observers concluded after the fourth day of the audit that

the auditors were deviating from the audit plan by not sufficiently evaluating the Ground Motion

Report, one of the technical reports identified in the audit scope. The observers noted a lack of

questioning of technical staff responsible for the Ground Motion Report by the audit team’s

Technical Specialists during interviews, and the absence of associated checklists questions that

should have been developed by the Technical Specialists while preparing for the audit. The

observers expressed concern to the Audit Team Leader whether this reduced level of technical

investigation would be adequate to meet the audit objective. As a result of the expressed NRC

concern, the Audit Team Leader issued Condition Report 9035 reflecting inadequate audit

planning. BSC immediately extended the audit end date from August 31 to September 7, 2006,

and added one additional auditor to provide sufficient time and resources to evaluate the

Ground Motion Report.

The observers noted that some of the technical reports being audited were planned for revision

in the near future. Therefore, the auditors focused on calculation, modeling, and scientific

investigation processes. The observers also noted that the audit team developed Noteworthy

Practices, as presented in the Attachment to this report, based on a focused audit using a

vertical slice technique and limited sample size. The Programmatic and Technical Issues also

presented in the Attachment indicate that additional work may be required to assure that the

Abstraction Report is of high quality.

4.2 Conduct and Timing of the Audit

The observers determined that the audit team performed demonstrated sound knowledge of the

applicable implementing procedures and QA program requirements. The audit team members

conducted thorough interviews, challenged and questioned responses when appropriate, and

effectively employed their checklists. The auditors caucused daily with the observers to discuss

the current audit status and potential issues. The auditors, BSC management, and observers

met daily to discuss the audit status and any new and developing issues. The timing of the
-5-

audit was appropriate in relation to ongoing seismic consequence-related activities.

4.3 Audit Team Qualifications and Independence

The observers reviewed the qualifications of the audit team members and verified their

qualifications and independence of the areas reviewed. Technical Specialists had technical

qualifications appropriate for the activities audited and had received training in appropriate audit

procedures and processes.

4.4 Examination of Quality Assurance Elements

QA auditors prepared checklists to evaluate the Abstraction Report and the technical reports

providing direct input to the Abstraction Report identified in Section 1.0 above. As discussed in

Section 4.1 above, an additional auditor was assigned to the audit team to evaluate the Ground

Motion Report. During the audit, the newly added auditor prepared a detailed checklist to

evaluate the Ground Motion Report, which should have been accomplished while planning for

the audit.

The QA auditors organized their checklists according to the critical process steps identified in

the audit plan: planning, development and documentation of models (including inputs,

assumptions, and software control), checking and review, product output, approvals, and

change control. The QA auditors also followed up on previously closed Condition Reports

related to the technical reports audited and identified no recurrences of those conditions.

QA auditors and Technical Specialists interviewed BSC technical staff together. They also

reviewed online records to assess areas such as data inputs, checking and review, approvals,

and change control. 

4.5 Examination of Technical Elements

The Abstraction Report provided direct inputs to the Total System Performance Assessment

(TSPA) code associated with the mechanical response of engineered barrier system

components to seismic hazards. The abstractions used results from calculations of kinematic

and structural response to vibratory ground motions. The Abstraction Report received direct

input from the Drip Shield, Waste Package, and Peak Ground Velocities Reports, which are

included in the audit scope, and the Drift Degradation Analysis, which was not within the scope

of this audit. The Ground Motion Report provided the time histories used in several of the

reports contributing to the Drip Shield Report, and the Drift Degradation Analysis Report

provides loading input for the detailed drip shield structural calculations.

The Waste Package Report included an overview of previous calculation reports (e.g., the

Structural Calculation of Waste Package Exposed to Vibratory Ground Motions), as well as

additional calculations performed with more complex representations of the engineered barrier

system components. The Drip Shield Report summarized the results of a series of supporting

engineering calculations developed to evaluate the effect of static and dynamic loads on the

mechanical performance of the drip shield. The Peak Ground Velocities Report described an

analysis that bounds the horizontal peak ground velocity at the waste emplacement level

300 meters below the surface at Yucca Mountain and presented results as a probability
-6-

distribution for horizontal peak ground velocities to represent uncertainties in the analysis. 

The Ground Motion Report described a site response model and its implementation using the

RASCALS software and provided spectral acceleration including peak ground acceleration,

peak ground velocity, and dynamically induced strains as a function of depth. The Ground

Motion Report also described the development of model inputs, implementation of the model,

its results, and the development of earthquake time history inputs based on model results.

The audit Technical Specialists prepared checklists based on detailed reviews of the subject

reports and associated Technical Work Plans. The Technical Specialists developed several 

checklists during the conduct of the audit to address reports not covered by other checklists

and to address the evaluation of the Ground Motion Report. These checklists should have

been developed while planning for the audit. The audit team interviewed the lead authors of the

technical reports evaluated and key contributors.

4.6 Potential Audit Findings

As presented in the Post-Audit Meeting, the audit team concluded that, overall, scientific

investigations that provide input to the Abstraction Model had been developed and calculated

using methods that were technically sound and defensible with adequate transparency and

traceability. The audit team did not identify significant issues related to scientific investigations;

however, the audit team identified several documentation and transparency issues related to

calculations providing input to the Abstraction Model.

Detailed descriptions of the audit team’s findings presented during the Post-Audit Meeting,

including adverse conditions, recommendations, and noteworthy practices, are provided in the

attachment to this report. These findings form the basis for the audit team’s determination that

the Abstraction Model appeared to meet the five acceptance criteria of NUREG–1804, 

Section 2.2.1.3.2.3, with the exception of the following:

1. Criterion 1, Subcriterion 3, transparency issues

2. Criterion 2, Subcriterion 1, data (modulus damping reduction curves) interpretation

and use

3. Criterion 3, Subcriterion 1, H2 (horizontal transverse ground motion) component

not considered

4. Criterion 3, Subcriterion 2, drip shield separation issue

5. Criterion 5, Subcriterion 3, unacceptable Engineered Barrier System structural models

The audit team determined that the modeling process satisfactorily addressed applicable critical

process steps, with the exception of the development and documentation of the models. 

Specifically, the audit team identified several technical issues in the areas of transparency,

conservatism, and data inconsistencies. Overall, the audit team determined the modeling

process to be effective.

The audit team determined that the methodology and structural models used in the mechanical
-7-

assessment calculations (i.e., Waste Package and Drip Shield Reports) were not always

conservative. However, since the calculations had been performed using extremely

conservative peak ground velocity values, the audit team judged the Seismic Consequence

Abstraction Model, which used the calculation results, to be adequate.

5.0 NRC STAFF FINDINGS

5.1 NRC Observation Summary

During the conduct of the audit, the NRC observers identified a weakness in the audit planning

and preparation. The observers determined that the BSC auditors apparently reduced the

scope of the audit, and did not prepare sufficiently to meet the primary objective of the audit. 

After the mid-audit adjustments to include evaluating the Ground Motion Report, the audit team

adequately evaluated technical activities to meet the audit objective. It is noted that during the

August 15-31, 2005, BSC audit BQAP–BSC–05–07, the NRC observers identified a similar

weakness in audit planning and preparation. The observers note that improvement is needed

in BSC’s planning and preparing for performance-based audits. 

The observers determined that the audit team demonstrated sound knowledge of the applicable

implementing procedures and QA requirements. The auditors conducted thorough interviews,

challenged and questioned responses when appropriate, and effectively employed their

checklists. The observers determined that the audit team identified substantive adverse

conditions that, when addressed, should provide for improved processes and improved

technical products. The observers agreed with the audit team’s findings, recommendations,

and overall conclusions. 

5.2 NRC Audit Observer Inquiry

The observers initiated no Audit Observer Inquiries during this audit.
A-1

ATTACHMENT

Audit Team Findings

Programmatic Issues—Adverse Conditions

• Verification of assumptions in calculations that were issued as preliminary and later

updated to committed status lacked transparency.

• Procedure inadequacy relating to software problem reports for failures encountered

during independent verification and validation of legacy software codes that were not

required to be issued.

• Engineering design calculation and analysis procedures did not describe the interface

with engineering calculations to support final scientific documents.

• All inputs directly used to develop the model had not been listed in the report. Input

information only identified one of two values used as direct input to the

Abstraction Report.

• Information presented in products was not consistent with planning documents for

the products.

• An exemption regarding software exemption criteria was misstated in a

technical product.

Programmatic Issues—Recommendations

• Provide justification for acceptability of software regarding issues in results that had not

been resolved.

• Ensure that the integrity of the technical review process is not compromised due to the

perceptions of conflicts of interest in the forthcoming revisions of the Abstraction Report.

• Revise the Technical Work Plan to be consistent with the Ground Motion Report.

Technical Issues—–Abstraction Report Adverse Conditions

• The description of the development of the abstractions and the methodology for using

these abstractions in the Abstraction Report was not transparent enough for a technical

reader to understand without recourse to the originator.

• Deficiencies regarding insufficient duration of vibratory ground motions were identified in

the Abstraction Report, Waste Package Report, and Drip Shield Report.
A-2

Technical Issues—–Abstraction Report Recommendations

• Use LS-DYNA to model and analyze impact scenarios involving deformable structural

and mechanical systems in future revisions to the Abstraction, Waste Package, and Drip

Shield Reports.

Technical Issues—Drip Shield Report Adverse Conditions

• The Drip Shield Report did not consider the H2 (horizontal transverse) ground motion

component when evaluating the fragility of the drip shield components and determining

the potential for separation of drip shield segments.

• The Drip Shield Report did not consider the rubble-covered scenario (the dynamic

analysis) when evaluating the fragility of the drip shield components and determining the

potential for separation of drip shield segments.

• Inconsistencies existed between the Abstraction Report and the Drip Shield Report.

Section 1.2 of the Abstraction Report states that “the structural response calculations

include corrosion of the waste package over a 20,000 year time frame.” Section 1.3 of

the Drip Shield Report has been considered for only 10,000 years.

• For drip shield separation as a result of vibratory ground motion, the structural integrity

of the drip shield to drip shield connector component (failure of which would result in drip

shield separation) was evaluated for the loads predicted from a simplistic UDEC

kinematic analysis model that did not consider all three spatial components of vibratory

ground motion.

• Because Section 5.2.3.1.4 of the Drip Shield Report states that computational meshes

used for the damaged area calculations might be too coarse to realistically simulate a

puncture or tear and that the predicted stresses might be unconservative, the model is

unreliable for predicting a puncture or tear.

Technical Issues—Waste Package Report Adverse Conditions

• In Section 6.5 of the Abstraction Report, two waste package impact scenarios and the

resulting damage prediction results were assumed to be independent of each other

even though both scenarios were consequences of the same vibratory ground motion.

• Potential damage from waste package to drip shield impacts was not included in the

seismic damage abstractions in the Abstraction Report and Waste Package Report.

Technical Issues—Peak Ground Velocities Report Adverse Conditions

• The peak ground velocity scaling approach used to develop time histories for structural

response calculations in the Abstraction Report cannot be performed theoretically or

empirically to represent earthquakes with larger peak ground velocities without changing

the magnitude of the earthquake.
A-3

• The Peak Ground Velocities Report does not address how incremental strains used in

numerical modeling and laboratory testing can represent in-situ behavior due to cyclic

shear strains associated with seismic wave propagation.

Technical Issues—Peak Ground Velocities Report Recommendations

• Revise Section 6.3 of the Peak Ground Velocities Report to address geologic

observations regarding seismically induced damage transparency issues in the

development of arguments and presentation of observations.

• Eliminate the extrapolated values from the second part of Table 6-3 in the Peak Ground

Velocities Report because they are not used in the Peak Ground Velocities Report or

the Abstraction Report.

• Regarding inputs of peak ground velocity, provide a stronger clarification of the use of

two values on page 4-1 of the Peak Ground Velocities Report and further clarify with

the document.

Technical Issues—Ground Motion Report Adverse Conditions

• Validation of modulus reduction and damping curves was only partially accomplished in

the Ground Motion Report by comparison to laboratory data and modeling nonlinear

observed records.

• Regarding RASCALS calculations in the Ground Motion Report, validate the possibility

that two-dimensional/three-dimensional effects at Yucca Mountain have particular

combinations that are not easily approximated as one dimensional.

• The Ground Motion Report validations had model values using different input

parameters for vertical and horizontal components for the same earthquake.

Technical Issues—Ground Motion Report Recommendations

• Identify studies in the Ground Motion Report that have validated the “Spectral Analysis

of Surface Waves” method with actual velocity profile data.

Noteworthy Practices 

• The Technical Specialist found the quality of the Ground Motion Report to be

noteworthy. It was stated that the document provides the most comprehensive,

validated, scientifically sound, and careful work ever performed for a site response

model and input ground motion calculation and that, in that sense, it is a historical

document that will be a template for future hazard studies.

• The Abstraction Report presents the evaluation that was performed to assess the

adverse consequences of seismic events during the postclosure period. The evaluation

was performed to meet the regulatory requirement that is extremely conservative and

unprecedented. Compliance to such an extreme requirement necessitated the use of

the seismic hazard determination methodology for annual exceedance frequencies well
A-4

beyond the range for which the methodology could be validated. This resulted in

extremely high peak ground velocity for which the Engineered Barrier System

components are required to be evaluated. At these high peak ground velocity values,

the drift materials and the Engineered Barrier System components behave nonlinearly

and the determination of their responses becomes very complex and uncertain. 

However, the seismic consequence abstraction performed accounted for most of these

complexities, uncertainties, and nonlinearities. Even though some of the methodology

and dynamic models used in assessing the damage to the Engineered Barrier System

components are not necessarily conservative, in an overall sense, the rigor and

thoroughness used in the seismic consequence abstraction process is remarkable

and noteworthy.