IC REGOJ 

0 UNITED STATES 

0 oNUCLEAR REGULATORY COMMISSION 

WASHINGTON, D C. 20555-00O1 

"February 4, 2000 

Mr. Dwight Shelor, Acting Director 

Program Management and Administration 

Office of Civilian Radioactive Waste Management 

U. S. Department of Energy 

1000 Independence Avenue, SW 

Washington, DC 20585 

SUBJECT: U.S. NUCLEAR REGULATORY COMMISSION'S OBSERVATION AUDIT 

REPORT NO. OAR- 00-03, "OBSERVATION AUDIT OF OFFICE OF THE 

CIVILIAN RADIOACTIVE WASTE MANAGEMENT, QUALITY ASSURANCE 

DIVISION, AUDIT NO. M&O-ARP-00-002" 

Dear Mr Shelor: 

I am transmitting the U.S. Nuclear Regulatory Commission's (NRC's) Observation Audit Report 

No. OAR-00-03 of the U.S. Department of Energy (DOE), Office of Civilian Radioactive Waste 

Management (OCRWM), Office of Quality Assurance (OQA), Yucca Mountain Quality 

Assurance Division (YMOAD), audit of the Biosphere Process Model Report (PMR) activities 

performed by the OCRWM Management and Operating Contractor (M&O). The audit, M&O

ARP-00-002, was conducted on November 15-19, 1999, at the M&O facilities in Las Vegas, 

Nevada. 

This audit was limited in scope and evaluated the effectiveness of the implementation of the 

OCRWM QA Program described in the Quality Assurance Requirements and Description 

(QARD) and its implementing procedures for selected analysis model reports (AMRs) 

supporting the Biosphere PMR. 

The NRC staff determined that this audit was effective in identifying deficiencies and 

recommending improvements in the AMR process. During the conduct of the audit, both the 

audit team and the NRC observers reviewed data, analysis reports, and software within the 

scope of the audit to confirm that it was properly qualified. The NRC observers determined 

that: a) the software supporting the AMRs had been properly qualified; and b) certain data 

categorized as "accepted data" were determined to be controlled in accordance with 

established procedures and properly categorized as "accepted data." 

The NRC staff generally agrees with the audit team conclusions, findings, and 

recommendations. However, as noted in Section 4.7 of this report, the NRC staff expressed a 

concern about the adequacy of the process controlling the preparation and use of procedures 

for the AMR process. Further, as discussed in various sections of this report, the NRC staff is 

concerned about the lack of data qualification activities for the AMRs reviewed during this audit 

and the two previous audits. Accordingly, this condition appears to be a condition requiring 

DOE's management attention. 

As discussed in Section 4.7.1 of the attached report, the NRC observers generated two audit 

observer inquiries (AOIs) questioning the process used for the validation of analysis and 

models, and concerning documenting the resolution of a reviewer's comments Also, we would
February 4. 2000 

like to point out that we have not received a response to an AOI, dated September 22, 1999, which addressed the qualification status and use of the waste stream profiles 

A 

OQA 

written response to this letter and the enclosed report is not required. However, we do expect to provide replies to the open AOIs. If you have any questions, please contact Larry L. Campbell of my staff at (301) 415-5000. 

Sincerely, 

C. William Reamer, Chief 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management 

Office of Nuclear Material Safety 

and Safeguards 

Enclosure: NRC 

Civilian 

Observation Audit Report No. OAR-00-03, "Observation Audit of the Office of Radioactive Waste Management, Quality Assurance Division, Audit No. 

M&O-ARP-00-002"
D. Shelor -2- February 4, 2000 

like to point out that we have not received a response to an AOI, dated September 22, 1999, which addressed the qualification status and use of the waste stream profiles. 

A 

OQA 

written response to this letter and the enclosed report is not required. However, we do expect to provide replies to the open AOIs. If you have any questions, please contact Larry L. Campbell of my staff at (301) 415-5000. 

Sincerely, 

C. William Reamer, Chief 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management Office of Nuclear Material Safety 

and Safeguards 

Enclosure: 

Civilian 

NBRC Observation Audit Report No. OAR-00-03, "Observation Audit of the Office of Radioactive Waste Management, Quality Assurance Division, Audit No. M&O-ARP-00-002" 

DISTRIBUTION: 

File 

SWastlerl 

Center PUBLIC ACNW LSS CNWRA NMSS r/f HL-WB rif CGlenn BMabrito WBelke RLatta JHolonich DWM r/f KHooks TCarter DBrooks 

DOCUMENT NAME: S/DWM/HLWB/lIc/biosphereobsrpt.wpd

ACNW: YES V_ NO Delete file after distribution: Yes No 1) This document:ýý LldhýJýJbe made available to the PUBLIC ",¢.- l/7 ,• 

This document (Initials) (Date)2) is related to the HLW program. It should be placed in the LSS. 7I
U S NUCLEAR REGULATORY COMMISSION

OBSERVATION AUDIT REPORT NO. OAR-00-03 

OBSERVATION AUDIT OF THE 

OFFICE OF CIVILIAN RADIOACTIVE WASTE MANAGEMENT 

QUALITY ASSURANCE DIVISION 

AUDIT NO. M&O-ARP-00-002 

01I•OH/00 Ke ~ j. 

Larry Campbell 

High-Level Waste and Performance High-Level Waste and Perfor anc/e 

Assessment Branch Assessment Branch 

Division of Waste Management Division of Waste Managemen

Robert Brient 

Center for Nuclear Waste Regulatory 

Analyses 

Reviewed and Approved by. 

"O• O//00 

"g Stablein, Chief 

P cts and Engineering Section 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management

01/2-7/00 

Patrick LaPlante 

Center for Nuclear Waste Regulatory Analyses
1.0 INTRODUCTION

Staff from the U.S. Nuclear Regulatory Commission (NRC) Division of Waste Management and 

contractors from the Center for Nuclear Waste Regulatory Analyses (CNWRA) observed the 

U.S. Department of Energy (DOE), Office of Civilian Radioactive Waste Management (OCRWM), Office of Quality Assurance (OQA), Yucca Mountain Quality Assurance Division 

(YMQAD) audit of the Biosphere Process Model Report (PMR) activities performed by the 

OCRWM Management & Operating Contractor (M&O). The audit, M&O-ARP-00-002, was 

conducted on November 15-19, 1999, at the M&O facilities in Las Vegas, Nevada. 

The objective of this audit was to evaluate the implementation of the applicable provisions contained in the OCRWM Quality Assurance Requirements and Description (QARD), DOE/RW

0333P, Revision 8, by reviewing selected analysis model reports (AMRs) supporting the Biosphere PMR. During the audit, selected AMRs were subjected to a technical review as well 

as review to ensure that the applicable programmatic requirements contained in the QARD and 

implementing procedures were met. 

The NRC staff objective was to gain confidence that the M&O and OQA are properly implementing the provisions contained in the QARD and the requirements contained in Subpart G, Quality Assurance, to Part 60, of Title 10 of the Code of Federal Regulations (10 CFR Part 

60). Because of the anticipated DOE submittal of the site recommendation (SR) in November 

2000, the following observation activities were emphasized: 1) confirming that data, software, and models supporting SR are properly qualified; and 2) reviewing the progress being made by DOE and its contractors in meeting the qualification goals for SR. 

This report addresses the NRC staff determination of the effectiveness of the OQA audit and 

the adequacy of implementation of QARD controls by the M&O in the audited areas of AMR 

development. 

2.0 MANAGEMENT SUMMARY 

The NRC staff has determined that OQA Audit M&O-ARP-00-002 was useful, effective, and conducted in a professional manner. Audit team members were independent of the activities 

they audited and appeared to be knowledgeable in the QA and technical disciplines within the 

scope of the audit. The audit team members' qualifications were reviewed and the members 

were found to be qualified in their respective disciplines. 

The audit team concluded that the OCRWM QA program had been satisfactorily implemented in the areas evaluated. Seven deficiency documents were generated during the audit. Two 

deficiencies were documented on deficiency reports (DRs) and four were documented on deficiency identification and referral (DIR) documents that add the conditions identified in this audit to those previously identified in currently open corrective action requests (CARs) or DRs. 

One deficiency was corrected during the conduct of the audit. Eight recommendations were 

offered for improvements and enhancements to the AMRs and to the procedures controlling various elements of the AMR process.

I
The NRC staff determined that this audit was effective in identifying deficiencies and 

recommending improvements in the AMR process. During the conduct of the audit, both the 

audit team and the NRC observers reviewed data, analysis reports, and software within the 

scope of the audit to confirm that it was properly qualified. The audit team and the NRC 

observers determined that the software supporting the AMRs had been properly qualified. The 

audit team and the NRC observer's also determined that certain data, categorized as "accepted 

data," were controlled in accordance with procedures and properly categorized as "accepted 

data." 

The NRC staff generally agrees with the audit team conclusion's, findings, and 

recommendations. However, as noted in Section 4.7 of this report, the NRC staff expressed a 

concern about the adequacy of the process controlling the preparation and use of procedures 

controlling the AMR process. Further, as discussed in various sections of this report, the NRC 

staff is concerned about the lack of data qualification activities for the AMRs reviewed during 

the audit and the two previous audits. This appears to be a condition requiring DOE's 

management attention. 

3.0 AUDIT PARTICIPANTS 

3.1 Nuclear Regulatory Commission Observers

Robert Brient 

Kien Chang 

Larry Campbell 

Patrick LaPlante

Team Leader 

Technical Specialist 

Senior QA Engineer 

Technical Specialist

CNWRA 

NRC 

NRC (Part time audit observer) 

CNWRA

3.2 OQA Audit Team

Donald Harris 

Kenneth McFall 

Larry Abenathy 

Harvey Dove 

Brenda Bowlby 

Chag-Hsiung Tung

Audit Team Leader 

Auditor 

Auditor 

Technical Specialist 

Technical Specialist 

Technical Specialist

OQA/Quality Assurance Technical 

Support Services (OQA/QATSS) 

OQA/QATSS 

OQA/QATSS 

OQA/QATSS 

Management and Technical 

Services(MTS) 

M&O

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION 

This OQA audit of the M&O was conducted in accordance with OCRWM Quality Assurance 

Procedure (QAP) 18.2, "Internal Audit Program," and QAP 16.1Q, "Performance/Deficiency 

Reporting." The NRC staff's observation of this audit was based on the NRC procedure, 

"Conduct of Observation Audits," issued October 6, 1989.

2
4.1 Scope of the Audit 

The audit team conducted a limited scope, performance based audit of activities and processes 

related to the development of the AMRs supporting the Biosphere PMR. AMRs, software, and 

data were evaluated during the audit process. The audit included review of the programmatic 

controls governing the AMRs and technical requirements contained in the AMRs. The following 

procedures and AMRs supporting the Biosphere PMR were reviewed by the audit team and the 

NRC observers during the audit: 

Procedures 

a) AP-2.13Q, 'Technical Product Development Planning," Revision 0, with Interim Change 

Notice (ICN) No. 1 

b) AP-SI.1Q, "Software Management," Revision 2, with ICN No. 0 

c) AP-3.1.5Q, "Managing Technical Product Inputs," Revision 0, with ICN No. 1 

d) AP-SIII.2Q, "Qualification of Unqualified Data and the Documentation of Rationale for 

Accepted Data," Revision 0, with ICN No. 0 

e) AP-3.1 0Q, "Analysis and Models," Revision 1, with ICN No. 0 

f) AP-2.140, "Review of Technical Products," Revision 0, with ICN No. 0 

g) AP-SlII.3Q, "Submittal and Incorporation of Data to the TDMS," Revision 0 

h) YAP-SV.1Q, "Control of the Electronic Management of Data," Revision 0, with ICN No. 1 

i) QAP-SIII-1, "Scientific Investigations", Revision 3 

Analysis Model Reports 

a) ANL-MGR-MD-000008, "Transfer Coefficient Analysis," Revision 00 

b) ANL-MGR-MD-000002, "Dose Conversion Factor Analysis: Evaluation of GENII-S Dose 

Assessment Methods," Revision 00 

c) ANL-MGR-MD-000003, "Disruptive Event Biosphere Dose Conversion Factor (BDCF) 

Analysis" (Draft) 

4.2 Conduct and Timing of the Audit 

The audit was performed in a professional manner and the audit team demonstrated a sound 

knowledge of the applicable M&O and DOE programs and procedures. Audit team personnel 

were persistent in their interviews, challenged responses when appropriate, and performed an 

acceptable audit. The NRC staff believes the timing of the audit was appropriate for the 

auditors to evaluate ongoing Biosphere PMR activities. However, the audit team was unable to 

confirm that data supporting the AMRs had been properly qualified because no qualification

3
activities had been initiated by M&O for this data. The NRC staff considers the lack of data 

qualification activities during this audit and the two previous PMR audits to be a condition 

requiring OQA management attention. The NRC staff suggests that OQA management 

evaluate the need to conduct audits specifically to evaluate the qualification of data. 

The DOE audit team and NRC observers caucused at the end of each day. Also, meetings of 

the audit team and M&O management (with the NRC observers present) were held each 

morning to discuss the current audit status and preliminary findings. 

4.3 Audit Team Qualification and Independence 

The qualifications of the audit team leader and the OQA audit team members were found to be 

acceptable in that they met the requirements of QAP 18.1, "Auditor Qualification," as verified by 

the NRC observation audit lead. The audit team members did not have prior responsibility for 

performing the activities they audited. In addition, training, education and experience records 

for audit team members were reviewed and found acceptable. 

4.4 Examination of Quality Assurance Elements 

The OQA programmatic and technical audit activities were conducted simultaneously using 

sub-audit teams consisting of a technical specialist and a QA auditor. The limited scope audit 

focused on the QA elements closely associated with the development of the AMRs. The NRC 

observation team evaluated the audit team's review of the following QA elements. 

4.4.1 AP-2.13Q "Technical Product Development Planning" 

The auditors reviewed technical development plans (TDPs) and work product planning sheets 

(WPPS) applicable to the subject AMRs. A deficiency in the implementation of planning was 

identified regarding electronic management of data not being addressed in the TDP and a TDP 

with content not meeting specified requirements. 

4.4.2 AP-SI.1Q "Software Management" 

GENII-S Version 1.485 is the computer software that will be used for many of the Biosphere 

AMRs, including those AMRs subject to this audit. The auditors reviewed its qualification 

documentation which was determined to meet the requirements of the software management 

procedure. This software had also been re-verified after general software qualification 

concerns were identified in the previously issued CAR-006. The NRC observers agreed with 

the audit team that the GENII-S software had been properly qualified. 

4.4.3 AP-3.15Q "Managing Technical Product Inputs" 

Each of the AMRs examined included document input reference sheets that list the inputs to 

and references cited in the AMR. The document input reference sheets also identify the status 

of the input, (e.g., qualified, to be verified (TBV)). At the time of the audit, the TBV status had 

not been removed for any of the Biosphere AMR input documents.

4
Documents cited as references or as corroborating data were given the status of not applicable 

along with a brief explanation. However, AP-3.15Q does not have provisions for this. The AMR 

developers consulted with the author of AP-3.15Q and came up with the N/A designation. 

During the audit, M&O staff initiated a change request to AP-3.15Q to attempt to clarify the use 

of references that are not directly used as inputs. 

The auditors noted that one reference was inadvertently omitted from the document input 

reference sheets for the draft AMR for disruptive event BDCF analysis. This deficiency was 

corrected during the audit. 

The status of the input documents for the three AMRs is summarized as follows: 

a) Transfer Coefficient Analysis: Most of the documents were classified as unqualified 

corroborating data (N/A). One input is classified as TBV because it is unconfirmed after 

a CAR cast uncertainty about the qualification status of data. A TBV tracking number 

(3059) has been assigned to this document as required by AP-3.15Q. 

b) Dose Conversion Factor Analysis: Evaluation of GENII-S Dose Assessment Methods: 

Several documents listed in the document input reference sheets are classified as N/A, 

used for reference only. One document is classified as 'Accepted,' its source being a 

Federal Guidance Report issued by the U.S. Environmental Protection Agency (EPA). A 

document tracked by TBV tracking number 3059 was used in this AMR. 

c) Disruptive Event Biosphere Dose Conversion Factor Analysis: All of the inputs used in 

this AMR are from other Biosphere AMRs, most of which have not been issued. These 

inputs are classified as TBV Several references are identified and given the N/A 

classification. 

4.4.4 AP-SIII.2Q "Qualification of Unqualified Data and the Documentation of Rationale 

for Accepted Data" 

Requests to qualify two reports conceming food consumption surveys had been initiated. At 

the time of the audit, no data qualification had been completed. The audit team confirmed that 

two sets of data from EPA Federal Guidance Reports used in the biosphere AMRs have 

completed the process for "accepted data" in accordance with AP-SIII.2Q. "Accepted data," as 

defined by the QARD, are data considered as established fact (e.g., engineering handbooks, 

density table, gravitational laws, or other physical constants) or data generally accepted by the 

scientific and engineering community and found to be technically defensible by those using it. 

The NRC observers agreed with the audit team that the subject data had been properly 

categorized. 

4.4.5 AP-3.100 "Analysis and Models" 

The three AMRs evaluated during this audit are considered analyses. AP-3.1 OQ provides 

control for both analysis and models. The development and technical checking processes 

described in AP-3.1OQ have been completed for the Transfer Coefficient Analysis and Dose 

Conversion Factor Analysis: Evaluation of GENII-S Dose Assessment Methods AMRs. The 

Disruptive Event Biosphere Dose Conversion Factor Analysis AMR was in review and comment 

resolution during the audit.

5
None of the three AMRs had been subjected to model validation at the time of the audit. An 

interview with Biosphere PMR management indicated that they were not certain when or if 

model validation was necessary. The audit team identified the lack of model validation as a 

deficiency and recommended that AP-3.10Q be revised to clarify the criteria for determining if 

an activity is an analysis, a model (therefore requiring validation), or both. The NRC observers 

initiated an audit observer inquiry (AOI) to the audit team to assure that the NRC staff is aware 

of the resolution of this issue ( see Section 4.7.1 of this report). The NRC staff believes that to 

properly support licensing decisions, calculations must be performed using validated model(s) 

as well as qualified data and software. 

The two completed AMRs had been subjected to the technical checking process. While 

described as checking, the review and reviewer criteria suggests that this activity represents the 

substantive review by a subject matter expert. The audit team noted that AP-3.10Q requires 

only that the checker document comments on a "check copy" of the document. AP-3.10Q does 

not require that the resolution of comments be documented except for the checker's signature. 

While responses were provided for some comments in the "check copy," many were not 

responded to. - In several cases, the auditors could not trace the checker's comment through to 

a revision in the document being reviewed. This condition was identified as a deficiency and 

the audit team made a strong recommendation that available comment resolution forms be 

used. The auditors also found one occasion where the checker failed to address one of the 

specified review criteria and the document was issued despite failing to meet the objective of its 

development plan. 

The audit team determined that the AMR author had not included sufficient detail in the 

justification of technical judgements in the Dose Conversion Factor Analysis: Evaluation of 

GENII-S Dose Assessment Methods AMR. This was included in the deficiency concerning 

AP-3.1 OQ. 

4.4.6 AP-2.14Q "Review of Technical Products" 

The AMRs reviewed were subjected to the technical review process. The AP-2.14Q technical 

reviews are performed by organizations that are external to the organization that prepared 

the AMR and serve primarily as interface reviews. 

AP-2.14Q allows several options for documenting comments and their resolution. For the 

AMRs in this audit, all used a markup of the document rather than requiring comment 

resolution. The audit team strongly recommended the use of the OCRWM comment sheet. 

The NRC staff concurs with this recommendation. 

4.5 PRIORITIZATION OF QUALIFICATION ACTIVITIES 

During the observation of the Biosphere PMR audit, the NRC observers met with DOE and 

M&O in order to obtain information on the process being used to prioritize the qualification of 

data, software, and models supporting its site recommendation (SR). 

4.5.1 Background 

The latest information provided by DOE at the time of the audit on its qualification of the data, 

software, and models supporting the SR was that: a) the qualification and validation of inputs

6
for the SR will be prioritized and evaluated in order of their importance; b) approximately 50 

percent of the inputs for the SR will be qualified by the end of May 2000; c) approximately 80 

percent of the inputs for the SR will be qualified mid-January 2001, and d) at the time the SR is 

issued in proposed form for NRC review in November 2000, the most critical inputs for the SR 

would be qualified with approximately 20 (+) percent of the inputs not qualified. Also, DOE 

informed the NRC that the inputs for the SR would be the basis for the license application (LA) 

and that all inputs for the LA would be qualified prior to its transmittal to the NRC. 

NOTE: [Subsequent to the audit, on December 16, 1999, NRC and DOE 

management met and discussed a number of issues including 

prioritization of data used as inputs for SR. At this meeting DOE informed 

the NRC that it intended too only to qualify data that was initially qualified 

and later determined to be "suspect" data if such data was categorized as 

high-risk significant. Further, DOE emphasized that low-risk significant 

data that was initially qualified and later determined to be "suspect" data 

would not be subject to any additional qualification.] 

4.5.2 Qualification Methodology 

The NRC observers were provided the following information by DOE and M&O: 

Data, software, and models supporting SR and LA will be prioritized based on their importance 

to waste isolation and to safety using the broad criteria contained in the "Repository Safety 

Strategy (RSS)," Revision 3 (currently issued by M&O and presently under review by DOE). 

The RSS contains the plan for preparing the post-closure safety case to support SR and the 

LA. The RSS evaluated the natural and engineered barrier systems relative to their roles in 

preventing or mitigating the release and migration of radionuclides to the public. 

The RSS identifies seven principal factors, disruptive events, and 20 other factors that 

contribute to the performance of the proposed high-level waste repository at Yucca Mountain 

(YM). The seven principal factors represent those repository performance features which 

provide the preponderance of waste isolation performance. The seven principal factors are: 1) 

seepage into drifts; 2) performance of the drip shield; 3) solubility limits of dissolved 

radionuclides; 4) retardation of radionuclide migration in the unsaturated zone; 5) retardation of 

radionuclide migration in the unsaturated zone; 6) retardation of radionuclide migration in the 

saturated zone; and 7) dilution of radionuclide concentrations during migration. Disruptive 

events include earthquakes and volcanism. 

The prioritization process groups the data, software, and models into categories by their use. If 

data, software, or models directly support the analysis used for a principal factor or a disruptive 

event having waste isolation significance (e.g., related to the 7 principal factors in the RSS), it is 

placed in the first priority category (high priority) and the qualification of these items will be 

identified for first priority qualification or verification, ahead of those related to the 20 other 

factors. 

Data Tracking Numbers (DTN) will be assigned to data once the data has been confirmed as 

inputs to analyses, calculations, software, and models required to support SR and LA. Because 

the entry of data into the DTN system continues to occur well into the analytical development

7
process, the actual inventory of DTNs subject to qualification and verification is dynamic and 

will not be finalized until near the end of the AMR and PMR completion process. 

4.5.3 Completeness of Site Recommendation Qualification Activities 

DOE and M&O informed the NRC observers that they expect to have 80 percent of the data, 80 

percent of the software, and 80 percent of the models supporting SR fully qualified by mid

January 2001. 

NOTE: [As previously noted, DOE informed the NRC during a December 16, 1999, 

public meeting that only data, categorized as high-risk significant that had 

not been initially qualified or was "suspect" data, requiring re-qualification, 

would be subject to the qualification process. Further, DOE stated that 

this decision was based on the sample of "suspect" data re-qualified to 

date. At the December 16, 1999, meeting the staff stated that it would 

evaluate bases for DOE's decision not to re-qualify the "suspect," low-risk 

-significant data.] 

[DOE now plans to have 80 percent of the high-risk significant data 

qualified by mid-January 2001.] 

4.5.4 Conclusions on Prioritization of Qualification Activities 

Based on the discussions with DOE and M&O, the NRC observers concluded that the 

priontization process used for qualifying data, software, and models supporting SR appears to 

be reasonable. However, to fully understand this priontization process, the NRC staff needs to 

review its implementation. The NRC staff will review the implementation of the prioritization 

process and document the results in future NRC staff observations of DOE audits for PMRs 

and through the NRC Onsite Representative's activities. 

The NRC staff will also continue to review the progress being made by DOE and M&O in 

meeting its qualification percent completion goals. This review will also be accomplished as 

part of the NRC staff observations of DOE audits of the PMRs and through the NRC Onsite 

Representative's activities. 

4.6 Examination of Technical Activities 

NRC staff observed the audit team technical specialists conducting detailed checks of the 

technical adequacy of the subject AMRs. At the start of the audit, NRC observers reviewed the 

technical specialists' qualifications (resumes) and found that the technical specialists had 

sufficient technical education, training and experience related to the AMRs reviewed. The 

technical specialists used a combination of technical issue probing and procedural compliance 

checks and verifications to thoroughly consider both the technical adequacy of the AMRs and 

the effectiveness of implementation of the QA program.

8
4.6.1 Analysis Model Report Transfer Coefficient Analysis (ANL-MGR-MD-000008, 

Rev 00) 

The AMR for transfer coefficient analysis documents the M&O staff analyses to select transfer 

coefficients. Transfer coefficients (i.e., factors that determine concentrations of radionuclides in 

plants and animal products from radionuclide concentrations in soil and feed) are data inputs 

for the GENII-S Version 1.485 code used to calculate BDCFs for total system performance 

assessment (TSPA) calculations. Because transfer factors for plants vary with plant type and 

soil chemistry, wide variation exists in published values. The purpose of the AMR is to establish 

criteria for selection of transfer coefficients and apply the criteria to a number of data sources to 

select a set of transfer coefficients applicable to YM that can be qualified in accordance with 

procedures. 

The audit of the transfer coefficient AMR included a combination of procedural and technical 

inquiries to verify that procedures were followed and that the technical quality of the product 

was satisfactory. The audit team inquired about the technical basis for the report including: a) 

planning and implementation of the technical approach; b) assumptions used; c) data 

acquisition and traceability; d) qualification of source data; e) treatment at data uncertainties; f) 

data selection criteria; g) rationales for data exclusion; and h) rationales for defining data as 

accepted. Selected key issues concerning source data analyzed to select transfer coefficients 

were investigated with extensive questioning and technical discussion. Discussions 

emphasized that data sources were summaries of available literature and these summaries 

constituted unqualified data. However, the selected transfer coefficients would eventually be 

qualified according to procedure even though the original sources would remain unqualified. 

After extensive discussions, the audit team and observers agreed that the selected transfer 

coefficients could be qualified according to procedures using the source data to corroborate the 

transfer coefficient selection. 

In checking the collection of site-specific data associated with this AMR, auditors investigated 

the use of the food consumption survey results Although the audit team determined that the 

data from the food consumption survey has not been qualified, no problems were identified with 

the use of the information. 

Resolution of technical comments from checkers and reviewers was assessed by auditors 

thoroughly reviewing a number of examples in the records package. The qualifications of the 

document originator and checkers were checked (resumes reviewed) by the auditors and 

verified by observers and all were found to have sufficient technical experience to conduct the 

assigned work. The records package for the AMR was extensively reviewed to confirm that 

checkers had provided comments, that the comments were technically adequate, and that the 

comments were resolved by the originator. The auditors and NRC observers noted a variety of 

comments that were both editorial and detailed on technical issues. At least one of the 

checkers was found to have provided very detailed technical comments. While the final report 

could be checked to determined that comments had been resolved (in a number of cases the 

originator provided written responses to comments in the text of the report), the auditors and 

NRC observers noted that formal comment/response forms were not required by the procedure. 

This condition made comment resolution traceability difficult for the auditors. Nonetheless, the 

auditors and observers verified that procedures for review had been followed correctly. The 

inclusion of comment/response forms for report checking was noted by NRC observers as an 

inquiry at the conclusion of the audit (see Section 4.7.1 of this report).

9
4.6.2 Analysis Model Report Dose Conversion Factor Analysis: Evaluation of GENII-S 

Dose Assessment Methods (ANL-MGR-MD-000002, Rev 00) 

The purpose of the dose conversion factor analysis AMR is to document analyses confirming 

the selection of internal and external dose conversion factors for use in the 

GENII-S Version 1.485 code as data inputs. GENII-S Version 1.485 is used to calculate 

Biosphere Dose Conversion Factors (BDCFs) for the TSPA. This AMR was necessary because 

the GENII-S code contains the dose conversion factors in binary data files that cannot be 

modified by the user. Because the GENII-S code was initially developed for analyses at the 

Hanford site in Washington State, the dose conversion factors in GENII-S Version 1.485 are 

based on material properties consistent with the waste materials existing at Hanford in the late 

1980's. Since that time, other accepted sources of external and internal dose conversion 

factors have been published, for example, by the EPA in Federal Guidance Reports 11 and 12 

(Environmental Protection Agency, 1988, 1993). Therefore, the AMR was prepared to address 

whether default dose conversion factors in GENII-S are consistent with currently accepted 

sources. 

The audit of the dose factor analysis AMR included a combination of procedural and technical 

inquiries to verify that procedures were followed and the technical quality of the product was 

satisfactory. The audit team inquired about the technical basis for the report including: a) 

planning and implementation of the technical approach; b) assumptions used; c) data inputs, 

acquisition, and traceability; d) data selection criteria; e) rationales for data exclusion; f) 

software validation; and g) justification for conclusions. Upon initial inspection of the AMR 

report, one of the auditors commented that the bases for key assumptions needed to be stated 

clearly in the report. The auditor noted that many assumptions appeared to be considered 

common knowledge by the originator and not thoroughly explained. The NRC observers 

concurred that more explicit bases for assumptions would improve the report. However, it was 

recognized that the omissions were due to the originators familiarity with the material and were 

not intentional. Auditors also asked questions about the bases for selecting a limited set of 

radionuclides for the analysis and the decision was traced back to viability assessment 

conclusions. Inquiries about the sources for data clarified that most were taken from the 

available literature and some were unqualified but did not need TBV because of their use as 

corroborative evidence. 

Resolution of technical comments from checkers and reviewers was assessed by auditors by 

thoroughly reviewing a number of examples in the records package. The qualifications of the 

document originator and checkers were checked (resumes reviewed) by auditors and verified 

by observers and all were found to have sufficient technical experience to conduct the assigned 

work. The records package for the AMR was extensively reviewed to confirm that the checker's 

comments were technically adequate and appropriately resolved by the originator. The auditors 

and NRC observers reviewed a variety of comments that were both technical and editorial. At 

least one of the checkers was found to have provided very detailed technical comments. Thus, 

the checker and technical reviews appeared to be adequate to ensure the technical quality of 

the report even though the procedures for the checker hampered traceability of comment 

resolutions (see discussion in section 4.6.1).

10
References: 

Environmental Protection Agency. Limiting Values of Radionuclide Intake and Air 

Concentration and Dose Conversion Factors for Inhalation, Submersion, and Ingestion: 

Federal Guidance Report No. 11. EPA 520/1-88-20. Washington DC: Environmental 

Protection Agency, September, 1988. 

Environmental Protection Agency. External Exposure to Radionuclides in Air, Water, 

and Soil: Federal Guidance Report No. 12. EPA 402-R-93-081. Washington DC: 

Environmental Protection Agency, September, 1993. 

4.6.3 Analysis Model Report Disruptive Event Biosphere Dose Conversion Factor 

Analysis (ANL-MGR-MD-000003, Draft) 

The purpose of the AMR for disruptive event BDCF analysis is to document DOE environmental 

pathway/dose calculations for radionuclides deposited in the biosphere following a volcanic 

eruption at the proposed YM repository site. The model assumes that radionuclides are 

released to the air and transported to the critical group location. The BDCFs are used in TSPA 

calculations to convert radionuclide concentrations deposited in soil to annual dose to the 

critical group. 

The audit of the transfer coefficient AMR included a combination of procedural and technical 

inquiries to verify that procedures were followed and the technical quality of the product was 

satisfactory. The audit team inquired about the technical basis for the report; planning and 

implementation of the technical approach; assumptions; data acquisition and traceability; 

qualification of source data; data selection criteria; rationales for data exclusion; and software 

qualification. The planning documentation (Work Package Planning Summary) described in 

AP-2.15Q was in draft form. Auditors asked a number of detailed questions regarding 

assumptions for the work which led to a similar concern as with the dose conversion factor 

analysis (e.g., need more explicit rationales for modeling assumptions per AP-3.10Q). The 

auditors asked technical questions about assumptions. The NRC observers found these 

questions to be insightful and indicated that the auditors had conducted a detailed review of the 

material in formulating the QA checklist and were familiar with pertinent technical issues. NRC 

observations of the discussions about technical assumptions indicated the report originator had 

a comprehensive understanding of the important parameters and limitations of available data. 

Many of the difficult-to-determine parameter inputs were found to be the under the jurisdiction 

of other AMR reports. The auditors and NRC observers emphasized the importance of future 

audits for tracing data sources and key assumptions that support more than one AMR. 

Auditors tracked data successfully to its source by use of DTN and the technical document 

management system. Auditors requested input transmittal records for those parameters that 

did not have accession numbers. Data obtained from other AMR reports were tracked by 

auditors to those referenced reports. All software packages used for the technical analyses 

were checked by auditors for their qualification status. 

In response to auditor and observer questioning about the qualification and validation status of 

the GEN1I-S Version 1.485 software, the AMR originator indicated that the software had been 

validated according to the procedure for software qualification (AP-SI.1Q); however, there was 

no attempt at model validation. The software qualification documentation (that includes 

software validation) was extensively reviewed by the auditors and observers. The auditors and

1I
NRC observers found that the qualification documentation conformed to AP-SI-1Q. In 

response to further questioning about why the model in GENII-S was not validated, the 

originator noted that the procedures in AP-3.10Q do not require model validation if the report is 

considered to be an analysis rather than a model. The auditors and NRC observers noted this 

as a limitation of the procedures because the procedures intended models to be validated, 

however, originators could bypass the requirements for model validation by selecting the option 

to call the report an analysis rather than a model. NRC observers presented the lack of clarity 

of the procedures regarding model validation to the audit team as an inquiry (see Section 4.7.1 

of this report). 

Resolution of technical comments from checkers and reviewers was assessed by auditors 

thoroughly reviewing a number of examples in the records package. The qualifications of the 

document originator and checkers were checked (resumes reviewed) by auditors and verified 

by observers and all were found to have sufficient technical experience to conduct the assigned 

work. The records package for the AMR was extensively reviewed to confirm that checker's 

comments were resolved by the originator. The checker's comments were technically sound 

and adequate; The auditors and NRC observers noted a variety of depth in comments 

(from editorial to detailed technical issues). At least one of the checkers was found to have 

provided very detailed technical comments Thus, the checker and technical reviews appeared 

to be adequate to ensure the technical quality of the report even though the procedures for the 

checker hampered traceability of comment resolutions (see discussion in section 4.6.1) 

4.7 NUCLEAR REGULATORY COMMISSION STAFF FINDINGS 

The NRC staff has determined that OQA Audit M&O-ARP-00-002 was effective in determining 

the level of compliance of M&O activities associated with the subject AMRs. The NRC staff 

agrees with the audit team conclusion that the OCRWM QA program had been satisfactorily 

implemented. 

However, the NRC staff expressed the following concerns during the conduct of the audit: 

a) There was no objective evidence that data qualification activities had been initiated for 

data supporting the selected AMRs for this audit and the previous two AMR audits. 

b) The NRC staff questioned the adequacy of the process controlling the preparation and 

use of procedures for the AMR process. During this audit, OQA identified one 

deficiency and made five recommendations about the adequacy and clarity of the these 

procedures During the previous two audits of selected AMRs, ten recommendations 

and one deficiency identified similar conditions. The NRC staff expressed a concern 

that the M&O management and its senior staff responsible for the supervision and use 

of these procedures should have recognized the need for the more apparent procedure 

clarifications prior to OQA making these recommendations.

12
4.7.1 Audit Observer Inquiries

Two NRC audit observer inquiries (AOIs) were presented to the audit team: 

1. AOI No. M&O-ARP-00-02-1 

AP-3.1 0Q, "Analysis and Modeling" and the QARD are not specific regarding which 

calculations/analyses are subject to model validation and the timing of model validation. 

M&O Environmental, Safety, and Regional Programs Office involved with the biosphere 

AMRs do not appear to have an understanding or strategy of model validation as it 

applies to the biosphere AMRs/PMR. 

2. AOI No. M&O-ARP-00-02-2 

Documented resolution of individual comments is not required for checks of analysis 

and models (AP-3.10Q) and is optional for reviews of technical products (AP-2.14Q). A 

lack of documented resolution is inconsistent with the QARD section 2.2.10 (f) which 

requires that mandatory comments shall be documented and resolved before approving 

the document. Note that the audit of the Integrated Site Model (ARP-99-009) also 

identified several recommendations concerning the review processes of AP-3.100 and 

AP-2.14Q. 

The two NRC Staff inquiries follow recommendations made by the audit team. The NRC staff is 

interested in DOE's and the M&O's resolution of these issues because of their potential 

significance in licensing. These inquiries remain open at the time of this report. 

4.7.2 Closure of Previous NRC Audit Observer Inquiries 

AOI No. M&O-ARP-99-009-1 was closed during the conduct of this observation. This AOI 

questioned aspects of the data qualification process (see the discussion is Section 4.5 of this 

report). 

4.7.3 Open NRC Audit Observer Inquires (AOIs) 

The following NRC AQIs remain open: 

a) AOI No OCRWM-ARC-99-015-1, dated September 22, 1999: OQA agreed to provide 

information to the NRC on the qualification status and use of the "Waste Stream 

Profiles" addressed in the "Design Basis Waste Stream for Interim Storage and 

Repository" and the "Waste Quantity, Mix and Throughput Study" documents. 

b) AOI No. M&O-ARP-00-02-1, dated November 18, 1999: (See Section 4.7.1 of this 

report for a description of this AOI) 

c) AOI No. M&O-ARP-00-02-2, dated November 18, 1999: (See Section 4.7.1 of this 

report for a description of this AOI)

13