September 5, 2002

Mr. Bruce Mabrito, Director

Quality Assurance

Center for Nuclear Waste Regulatory Analyses

6220 Culebra Road, Building 189

San Antonio, TX 78238-5166

SUBJECT: OBSERVATION OF THE JULY 2002 INTERNAL AUDIT OF CENTER FOR

NUCLEAR WASTE REGULATORY ANALYSES

Dear Mr. Mabrito:

From July 23 through 26, 2002, members of the U.S. Nuclear Regulatory Commission (NRC)

Office of Nuclear Material Safety and Safeguards staff participated as observers in the Center

for Nuclear Waste Regulatory Analyses (CNWRA) Internal Quality Assurance (QA) Audit 2002-

1, conducted in San Antonio, Texas. The NRC staff evaluated this audit to determine whether

the CNWRA is effectively implementing the requirements of its QA program. This letter

transmits NRC Observation Audit Report Number OAR-02-09-CNWRA.

The audit evaluated the adequacy and effectiveness of the CNWRA QA program and its

implementation. Eighteen applicable QA programmatic areas and seven technical areas were

audited. The technical areas included: 1) Licensing Support Network; 2) Container Life and

Source Term; 3) Repository Design and Thermal Mechanical Effects; 4) Mixed Oxide Fuel

Fabrication Facility Technical Assistance; 5) Reuse Soil Scenario Analysis; 6) Integrated Issue

Resolution Status Report; and 7) Total System Performance Assessment and Technical

Integration. This report addresses the effectiveness of the audit and the procedural adequacy

and effectiveness of implementation of QA programs controls in the audited areas.

The NRC observers (hereafter, observers) based their evaluation of the audit process and the

CNWRA QA program on: 1) discussions with, and direct observations of: a) the audit team,

and b) CNWRA staff being audited; and 2) reviews of pertinent audit documentation, such as

the audit plan, the audit checklist, CNWRA deliverables, and other CNWRA documents. The

observers determined that, overall, CNWRA Audit 2002-1 achieved its purpose, and was

thorough, effective, and performed in a professional manner. In general, the observers agree

with the audit team’s findings and that, overall, the CNWRA QA program controls are being

adequately implemented in the areas that were evaluated. 

During the audit, the audit team identified two major deficiencies in the areas of data

interpretation and analysis (Ref: CQAM Section 3.3.10) and verification of calculations (Ref:

QAP-014) and one minor deficiency dealing with inconsistency in the frequency for trending

adverse conditions (Ref: QAP-010). Also three isolated incidents were identified and corrected

during the audit.
B. Mabrito -2-

The observers discussed their observations with the audit team and CNWRA management

during the conduct of the audit and at the post-audit meeting.

We will continue to monitor the CNWRA progress in correcting its QA-related problems. 

Although a written response to this letter or the enclosed report is not required, we request that

the CNWRA provide us with copies of its response to the audit team’s findings. If you have any

questions, please call Ted Carter at (301) 415-6684.

Sincerely,

/RA/

Deborah A. DeMarco

NRC CNWRA Deputy Program Manager

Program Management, Policy Development

 and Analysis Staff

Office of Nuclear Material Safety

 and Safeguards

Enclosure: NRC Observation Audit Report No. OAR-02-09-CNWRA, “Observation

Audit of Center for Nuclear Waste Regulatory Analyses”
B.Mabrito -2-

The observers discussed their observations with the audit team and CNWRA management

during the conduct of the audit and at the post-audit meeting.

We will continue to monitor the CNWRA progress in correcting its QA-related problems. 

Although a written response to this letter or the enclosed report is not required, we request that

the CNWRA provide us with copies of its response to the audit team’s findings. If you have any

questions, please call Ted Carter at (301) 415-6684.

Sincerely,

/RA/

Deborah A. DeMarco

NRC CNWRA Deputy Program Manager

Program Management, Policy Development

 and Analysis Staff

Office of Nuclear Material Safety

 and Safeguards

Enclosure: NRC Observation Audit Report No. OAR-02-09-CNWRA, “Observation Audit of

Center for Nuclear Waste Regulatory Analyses”

DISTRIBUTION: CNWRA NMSS r/f HLWB r/f TCarter JGreeves

TBloomer RLatta JParrot JPearson DEsh WBrach

JLinehan

ADAMS Accession #: ML

DOCUMENT NAME: S/DWM/HLWB/THC/CNWRA August 2002 Internal audit.wpd

OFC HLWB Tech.Ed. HLWB HLWB

NAME TCarter EKraus NKStablein JSchlueter

DATE 08/22/02 08/22/02 08/26/02 08/30/02

OFFICIAL RECORD COPY

ACNW: YES X Delete file after distribution: Yes No 

1) This document should/should not be made available to the PUBLIC. Yes, thc1

2) This document is related to the HLW program. It should be placed in the LSS. thc1 08/ /01

 (Initials) (Date)
Enclosure

U.S. NUCLEAR REGULATORY COMMISSION

OBSERVATION AUDIT REPORT NO. OAR-02-09-CNWRA,

“OBSERVATION AUDIT OF THE 

CENTER FOR NUCLEAR WASTE REGULATORY ANALYSES

AUDIT 2002-1"

 /RA/ Date: 08/22/02 /RA/ Date: 08/22/02 

Ted Carter Tamara Bloomer

Projects and Engineering Section Projects and Engineering Section

High-Level Waste Branch High-Level Waste Branch

Division of Waste Management Division of Waste Management

 /RA/ Date: 08/22/02 /RA/ Date: 08/22/02 

David Esh Jim Pearson

Performance Assessment Section Transportation and Storage Safety and

Environmental and Performance Inspection Section

 Assessment Branch Spent Fuel Project Office

Division of Waste Management

Reviewed and Approved by:

 /RA/ Date: 08/26/02 

N. King Stablein, Chief

Projects and Engineering Section

High-Level Waste Branch

Division of Waste Management
-2-

1.0 INTRODUCTION

From July 23-26, U. S. Nuclear Regulatory Commission (NRC) staff participated as observers

in the Center for Nuclear Waste Regulatory Analyses (CNWRA) internal Quality Assurance

(QA) Audit 2002-1 conducted in San Antonio, Texas. The NRC staff evaluated this

performance-based audit to determine whether the CNWRA is continuing to effectively

implement the requirements of its QA program.

The audit evaluated the adequacy and effectiveness of the CNWRA QA program and its

implementation. Eighteen applicable QA programmatic areas and seven technical areas were

audited. The technical areas included: 1) Licensing Support Network; 2) Container Life and

Source Term; 3) Repository Design and Thermal Mechanical Effects; 4) Mixed-Oxide Fuel

Fabrication Facility Technical Assistance; 5) Reuse Soil Scenario Analysis; 6) Integrated Issue

Resolution Status Report; and 7) Total System Performance Assessment and Technical

Integration. This report addresses the effectiveness of the audit and the procedural adequacy

and effectiveness of implementation of QA programs controls in the audited areas.

2.0 OBJECTIVES

The CNWRA objective for this audit was to evaluate whether the implementation of QA

controls, associated with CNWRA QA programmatic and technical activities, met the applicable

requirements of Appendix B to Title 10, U. S. Code of Federal Regulations (10 CFR), Part 50,

as required by its contract with NRC. The NRC staff’s objectives were to determine: 1) if the

audit was performed in such a manner as to provide confidence in the CNWRA audit process

and 2) whether CNWRA staff were adequately implementing QA program requirements

specified in the CNWRA Quality Assurance Manual (CQAM), thus meeting contractual QA

requirements.

3.0 SUMMARY AND CONCLUSIONS

The NRC observers’ evaluation of the audit process and the CNWRA QA program was based

on: 1) discussions with and direct observations of: a) the auditors and technical specialists of

the audit team and b) CNWRA staff being audited; and 2) reviews of pertinent audit

documentation such as the audit plan, the audit checklist, and other CNWRA documents. 

The NRC observers determined that CNWRA Audit 2002-1 achieved its purpose of evaluating

the implementation of the QA controls of programmatic and technical activities. The audit team

was qualified and familiar with the QA requirements of the CNWRA program. The audit was

conducted in a professional manner. The audit schedule and individual assignments were

adequately described in the audit plan, and the audit checklist was detailed and complete.

The audit team identified two major deficiencies, one minor deficiency, and three isolated

incidents which were corrected during the audit. All the findings were discussed with CNWRA

management during daily debriefs and at the post-audit meeting. 

In general, the observers agreed with the audit team findings that the CNWRA Quality

Assurance Program is being effectively implemented and appears to provide effective controls
-3-

for technical product development. The audit team also indicated that the deficiencies identified

during the audit are unlikely to have adverse impacts on the technical products.

4.0 AUDIT PARTICIPANTS

The audit team was staffed with NQA-1 qualified lead auditors and auditors from the Southwest

Research Institute (SwRI) QA Department and by technical specialists from SwRI and a local

university. The lead auditor, auditors, and technical specialists were independent of the

activities audited, technically qualified, and trained in auditing techniques. In each case, a subteam of a Technical Specialist and a QA Auditor performed the audit of each technical activity.

4.1 Nuclear Regulatory Commission Observers

Ted Carter Observer (Team Leader -NRC)

James Pearson Observer (QA Specialist - NRC)

Tamara Bloomer Observer (Technical Specialist - NRC)

David Esh Observer (Technical Specialist -NRC)

Deborah DeMarco Observer (CNWRA Deputy Program Manager -NRC)

4.2 Audit Team

Robert Brient, Audit Team Leader SwRI

Thomas Trbovich QA Auditor SwRI

Rodney Weber QA Auditor SwRI

Mike Enright Technical Specialist SwRI

Steve Hudak Technical Specialist SwRI

John Hageman Technical Specialist SwRI

William Thomann Technical Specialist University of the Incarnate Word

Alex Bernardo Technical Specialist SwRI

Aaron DeWispelare Technical Specialist SWRI

Bob Mason Technical Specialist SWRI

5.0 CONDUCT OF THE AUDIT AND OBSERVATION

The CNWRA audit was conducted in accordance with CNWRA Quality Assurance Procedure

(QAP)-011, “Audits.” The NRC staff’s observation of the CNWRA audit was based on the NRC

procedure, “Conduct of Observation Audits,” issued October 6, 1989 (Draft).

5.1 Scope of the Audit

The audit evaluated the CNWRA QA program to determine whether it meets the applicable

requirements of Title 10, U. S. Code of Federal Regulations (10 CFR) Part 50, Appendix B.

Such evaluation was accomplished by verifying the QA program implementation and

effectiveness. The audit was performance-based and evaluated programmatic requirements in

light of their application to technical activities. In addition, the corrective action process was

reviewed to determine its effectiveness.
-4-

5.1.1 QA Programmatic Elements

10 CFR Part 50, Appendix B, Criteria CQAM Chapter

I Organization 1

II QA Program 2

III Design Control N/A

Scientific Investigation and Analysis Control 3

IV Procurement Document Control 4

V Instructions, Drawings, and Procedures 5

VI Document Control 6

VII Control of Purchased Material 7

VIII Identification and Control of Items 8

IX Control of Special Processes 9

X Inspection 10

XI Test Control 11

XII Control of Measuring and Test Equipment 12

XIII Handling, Storage, and Shipping 13

XIV Inspection Test and Operating Status 14

XV Nonconformance Control 15

XVI Corrective Action 16

XVII Records Control 17

XVIII Audits 18

5.1.2 Technical Areas

The technical and programmatic areas selected for audit were determined by the Audit Team

Leader and the CNWRA QA Director. The determination was made based on the level of

activity, technical risks involved, programmatic risks involved, and the time since each area had

last been audited. Since design-related activities are not performed by the CNWRA, criterion III

of 10CFR Part 50, Appendix B, is not applicable to this audit. 

A performance-based approach was used by the auditors during the audit to evaluate the

effectiveness of the QA program in ensuring product quality. The auditors used direct

evaluation of selected activities, assessment of products, and evaluations of product

development processes. The performance-based approach was implemented by the use of

teams composed of programmatic auditors and technical specialists evaluating activities from

their individual perspectives, as well as evaluating implementation of procedures and plans

associated with product development. The following areas were selected for this audit:

1) Licensing Support Network (LSN)

2) Container Life and Source Term (CLST)

3) Repository Design and Thermal Mechanical Effects (RDTME)

4) Mixed-Oxide Fuel Fabrication Facility Technical Assistance (MOX)

5) Reuse Soil Scenario Analysis (RSSA)

6) Integrated Issue Resolution Status Report (IIRSR)

7) Total System Performance Assessment and Technical Integration (TSPA&I)

5.2 Conduct and Timing of the Audit

The audit was performed in a professional manner and the audit team was well-prepared and

demonstrated a sound knowledge of the CNWRA program. The auditors and technical

specialists used the checklist effectively during discussions with CNWRA personnel and review
-5-

of documents. The audit plan was thorough in describing the scope of the audit. Based on the

length of time required during the audit for the CLST area, the NRC staff recommends that

consideration be given to re-allotment of time for similar areas during the next audit to allow for

the audit process to be thoroughly implemented without interruption. 

The audit team and the NRC observers caucused at the end of the day. Also, a meeting of the

audit team, observers, and CNWRA management was held each morning to discuss the

current audit status and preliminary findings. In general the NRC staff believes the timing of the

audit was appropriate.

5.3 Audit Team Qualification and Independence

The QA audit team was staffed with auditors from the SwRI QA Department and lead auditors

qualified to the CNWRA QA manual. The qualifications of the Audit Team Leader and the audit

team members were found to be acceptable in that each met the requirements of SwRI

procedure No. NQAP 2.0-1, “Qualification and Certification of QA Auditors,” dated November

1989. A sub-team of a Technical Specialist and a QA Auditor performed each technical audit

activity and used the checklist to cover the subject matter well. The team members had no

involvement with, or responsibility for, performing any of the activities they audited.

5.4 Examination of QA Programmatic and Technical Activities

CNWRA Audit 2002-1 was conducted as a performance-based audit. The audit team focused

on the technical activities and evaluated the QA programmatic controls applicable to those

activities. The NRC staff observed that each of the auditors reviewed related documentation

and interviewed a representative sample of CNWRA personnel to determine their

understanding of implementing procedures and processes. Checklists were used effectively,

and issues were provided, beyond the checklists, when appropriate. The NRC observers were

provided ample opportunities to provide comments and ask questions.

The audit of all of the above seven technical areas listed in section 5.1.2 was performed. Out

of the seven technical areas audited by the audit team, all were observed by the observation

team, with the exception of RSSA and IIRSR.

Training, education, and experience records were reviewed for personnel conducting scientific

studies, to ensure such personnel were in compliance with their individual position descriptions.

Objective evidence was provided to, and reviewed by, the audit team, and it was determined

that all personnel were in compliance.

During the audit, the audit team identified deficiencies in the program, which resulted in two

deficiencies in the areas of justification of assumptions and verification of calculations and one

deficiency dealing with inconsistency in the frequency for trending adverse conditions. These

findings were discussed with CNWRA management at the post-audit meeting.

The audit was effective in determining CNWRA compliance with procedural controls in the

areas examined. The audit was thorough and effective in determining CNWRA personnel

knowledge of, and compliance with, procedural controls. The NRC staff agrees with the audit

team’s assessment that, overall, the CNWRA is acceptably implementing its QA program.
-6-

6.0 NRC STAFF FINDINGS AND RECOMMENDATIONS

The QA programmatic and technical portions of the audit were conducted in a professional

manner and the audit team adequately evaluated activities and objective evidence. The Audit

Team Leader was effective in his daily presentation to the CNWRA management and staff and

provided guidance to the audit team. Both the auditors and CNWRA staff were knowledgeable

in their respective disciplines. The checklist was an excellent tool for providing a sound basis

from which to conduct the performance-based audit. The NRC observers believed that the

CNWRA audit was thorough and effective. They determined that the audit was effective in

determining CNWRA compliance with procedural controls in the areas examined and that the

CNWRA QA program controls are being adequately implemented. The technical qualifications

of CNWRA staff were satisfactory. The technical adequacy of the work products and

procedures was found satisfactory and is subject to continuing in-depth evaluation by NRC

technical staff. While the observers do not agree with all the audit team findings (see CLST

under 6.1), overall the observers agree with the audit team’s assessment that the CNWRA is

implementing its QA program satisfactorily.

CNWRA Audit 2002-1 was conducted as a performance-based audit. The audit team focused

on the technical activities and evaluated the QA programmatic controls applicable to those

activities. Out of the seven technical areas audited by the audit team, all were observed by the

observation team, with the exception of RSSA and IIRSR.

6.1 NRC Findings

License Support Network (LSN)

An interview with the LSN element manager indicated the significance of a high level of

accuracy during the population of the LSN, as well as the remaining labor-intensive activity that

will be required to accomplish the task. Responses to audit interview questions included some

of the challenges of populating the network with multiple types of media and development of

acceptable references to the physical samples that may be required for license support. 

Additional discussion included the high level of accuracy required for indexing the large volume

of supporting documentation which will populate the LSN.

The following strengths were indicated during audit interviews with the ongoing work in this

element. 1) A good relationship with NRC counterparts allows frequent and candid interactions,

2) CNWRA staff are able to raise and resolve issues with NRC in a timely fashion.

Two technical suggestions were offered during the audit: 1) Develop a documented string of

plans that provide continuity from the operations plan to products for the staff. 2) Develop a

high-level process description of how LSN issues are established and resolved to support longterm external visibility into the LSN.

The observers agreed with the audit team’s findings and recommendations.

Container Life Source Term (CLST)

The audit team reviewed two CNWRA deliverables in the CLST area; the first was Intermediate

Milestone 01402.571.210, “Evaluation of Analogs for Performance Assessment of High-Level

Waste Container Materials” (CNWRA 2002-02); and the second was Intermediate Milestone
-7-

01412.571.170, “Effect of Environment on the Corrosion of Waste Package and Drip Shield

Materials” (CNWRA 2001-3).

The audit team found that overall the report preparation and review process were adequate,

and that the reports had well thought-out technical approaches (such as long-term verification

to confirm short-term experiments). The audit team’s review of several scientific notebooks

from the CLST laboratories found that compliance has improved over previous years. 

However, minor deficiencies were identified and corrected during the audit. The audit team

considered this finding a minor nonconformance. The observers also reviewed several

scientific notebooks and agreed with these findings. 

The observers agreed with the audit team’s assessment that the CNWRA is implementing its

QA program in the procedure area.

The observers do not agree that the audit team’s evaluation of the deliverable was appropriate.

The audit team’s evaluation of the deliverables led to several technical suggestions. The audit

team explained that these findings are not considered as recommendations and no action is

required by the CNWRA. However, the observer found that the questions during this part of the

audit team’s inspection appeared outside the scope of a quality program audit. The observers

also found that the technical suggestions were more in the arena of a programmatic review and

should not have been presented to the CNWRA. The observers discussed this concern with

the audit team lead, while the questions were in progress, and with the audit team as a whole,

when the suggestions were written during the daily caucus. In addition to the above concern,

the observers suggested to the audit team, for future audits, that more time be allocated to the

review of areas that incorporate laboratory investigations. The auditor only had time to review

one scientific notebook within the time allotted for the CLST. One notebook is not necessarily a

representative sample of all of the notebooks produced since the last audit.

Repository Design and Thermal Mechanical Effects (RDTME)

The audit team reviewed the processing and technical quality of several selected CNWRA

deliverables in the RDTME area. These included “Mechanical Response of Drip Shields Under

Repository Environment - Progress Report 2, Repository Design and Thermal-Mechanical

Effects Key Technical Issue Intermediate Milestone No. 20.01402.671.240, November 2001";

“DECOVALEX III Task 2C: Thermal-Mechanical Modeling of the Drift-Scale Heater Test at

Yucca Mountain - A Progress Report, December 2001"; and “Assessment and Process-Level

Waste Package and Drip Shield Studies for Input to SEISMO Module of the TPA Code.”

The audit team found that the deliverables had adequately met the QA requirements. 

Numerous positive practices were identified. An error was identified with an ASME code. A

detailed discussion was provided in the deliverable, describing the error and why an earlier

version of the ASME code was used. Scientific notebook reviews were being completed more

frequently than required and also were being copied every 6 months, to reduce the risk of

losing data. Software documentation was complete and readily available.

Two technical suggestions that resulted from the audit were to provide: (1) quantitative

validation for a finite element model being used in some of the analyses, and (2) a broader

range of temperatures in the evaluation of the mechanical response of the drip shields. The

investigators had selected a pessimistic value (160 C) understanding that there could be

significant variability in temperatures both temporally and spatially. It was difficult for the
-8-

investigators to select a more realistic value with associated uncertainty as a result of DOE’s

frequent changes to repository design.

The observers agreed with the audit team’s findings and recommendations.

Mixed Oxide (MOX) Fuel Fabrication Facility Technical Assistance

The interview of the MOX element manager, during the audit, showed that the project was well

in control. The additional conservatism, noted through the use of criteria based on NRC

documents for nuclear power plant licensing during the review process, was identified as a

good practice. No scientific notebooks were initiated nor were any calculations performed in

regard to the MOX final review plan; therefore none were available for review during the audit.

The observers agreed with the audit team’s findings and recommendations.

Total System Performance Assessment and Technical Integration (TSPA&I)

The audit team reviewed the processing and technical quality of several selected CNWRA

deliverables in the TSPA area: These included (1) “Response to the External Peer Review of

the Total-System Performance Assessment Version 3.2 Code, September 2001", “Transmittal

of revised version of the Total-System Performance Assessment and Integration Issue

Resolution Blueprint - Letter Report,” April 5, 2002, and “Software Requirements Description for

the TPA Version 5.0 Code, December 2001.”

The audit team found that the deliverables were thorough and detailed. The software

requirements document provided a good description of the requirements for modifications to the

TPA code. Review packages were evaluated and were found to include comprehensive

comments from the technical and programmatic reviewers.

Two deliverables contributed to major deficiencies. First, the TSPA issue resolution blueprint

letter report contained inadequate and inconsistent referencing. CQAM section 3.3.10 “Data

Interpretation and Analysis” requires that a technically qualified person may review, understand,

and verify the analysis without recourse to the originator (including references). It was

determined that QAP-002, instructions to technical reviewers, does not convey the full scope of

the requirements identified in CQAM 3.3.10. Second, the review package for the response to

the external peer review deliverable did not provide information on how sample (calculation)

verification was accomplished, nor the conclusions regarding the calculation accuracy. The

audit team determined that QAP-002,” Instructions to Technical Reviewers,” does not provide

for the specification of the extent and type of over-checks, and does not instruct the reviewer

regarding the level of documentation required for verification. It was discussed and accepted

that the intent of the requirements was for technical reviewers to produce objective information

documenting calculation verification.

The response to the external peer review for TPA version 3.2 was developed to address

formally elicited comments from a group of experts. The original comments from the experts

were in some cases written verbatim, in other cases paraphrased, and sometimes combined

with other comments. The report was developed providing a response to the extracted

comment with a link to the original text (located in an appendix). The response to the

comments was traceable when proceeding from the response to the experts’ original

comments. However, it was very difficult for an independent reviewer to proceed from the

experts’ original comments to a specific response. Therefore it was very difficult to verify that

all the concerns presented by the experts had been adequately addressed. A technical
-9-

suggestion was made to provide for clear accounting of all comments and their resolution

(consider using cross-walk tables) in a future revision to the document.

The observers agreed with the audit team’s findings and recommendations.

6.2 NRC Recommendations

The following three recommendations are made by the observers:

 Based on the length of time required during the audit for the CLST area, the NRC staff

recommend that consideration be given to re-allotment of time for similar areas next

audit to allow for the audit process to be thoroughly implemented without interruption. 

 The observers found that most of the audit team questions based in the CLST

deliverables appeared outside the scope of a quality program audit. The NRC staff

recommends that further guidance be provided to the audit team in preparing check list

to ensure that questions are appropriate QA audit.

 When an analyst/scientist is going to leave CNWRA (because of the end of his/her term

of employment there), the appropriate CNWRA personnel should thoroughly review all

work the departing analyst/scientist did on NRC projects, and also verify if it was

completed accurately. Assumptions the analyst/scientist made, regarding the project(s)

should also be reviewed, if possible, before his/her departure.