- UNITED STATES 

NUCLEAR REGULATORY COMMISSION 

zt WASHINGTON D C 20555-0001 

***4 14eDecember 14, 1999 

Mr Dwight Shelor, Acting Director 

Program Management and Administration 

Office of Civilian Radioactive Waste Management 

U S Department of Energy 

1000 Independence Avenue, SW 

Washington, DC 20585 

SUBJECT- U. S NUCLEAR REGULATORY COMMISSION'S AUDIT OBSERVATION AUDIT 

REPORT NO. OAR-00-01, "OBSERVATION AUDIT OF THE CIVILIAN 

RADIOACTIVE WASTE MANAGEMENT QUALITY ASSURANCE DIVISION 

AUDIT M&O-ARP-99-009" 

Dear Mr. Shelor. 

I am transmitting the U.S. Nuclear Regulatory Commission (NRC) Observation Audit Report No 

OAR-00-01 of the U S. Department of Energy (DOE), Office of Civilian Radioactive Waste 

Management (OCRWM), Office of Quality Assurance (OQA), Yucca Mountain Quality Assurance Division (YMQAD) limited scope, performance based audit of the OCRWM Quality Assurance (QA) program of the Civilian Radioactive Waste Management System Management and Operating (M&O) Contractor. The audit, M&O-ARP-99-009, was conducted on October 11

15, 1999, at the M&O offices in Las Vegas, NV. 

This audit evaluated the activities that constitute scientific, engineering and performance 

assessment analyses and models pertaining to the Integrated Site Model (ISM) and the 

technical activities associated with processes and controls related to the ISM. The following Analysis Model Reports (AMRs) were evaluated: "Mineralogical Model (MM3.0)" AMR (Draft); "Rock Properties Model (RPM3.1)" AMR (Draft); Geologic Framework Model (GFM3.1) AMR 

(Draft) and the ISM Process Model Report (PMR) (Draft). The audit also included a review of 

appropriate procedures directly related to the AMRs and PMRs and evaluated the effectiveness 

of the analysis and model processes for the ISM and the quality of the resultant end products identified. This was done by verifying implementation adequacy of the critical process steps relative to analysis and models for the ISM. 

The audit team concluded that the M&O documentation of activities that constitute scientific, 

engineering and performance assessment analysis and models pertaining to the ISM were 

satisfactory. Three deficiencies were identified during the audit. One of the three deficiencies 

was corrected during the audit. Nine recommendations were offered as improvements to the 

program. 

The NRC staff agrees with the audit team's conclusion, findings, and recommendations. The 

NRC staff determined that this audit was effective and that the M&O ISM process implementation was adequate. NRC observers also reviewed data and software packages, 

within the scope of the audit, and found them properly qualified.
D Shelor

A written response to this letter and the enclosed report is not required If you have any 

questions, please contact Ted Carter of my staff at (301) 415-6684 

Sincerely, 

C. William Reamer, Chief 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management 

Office of Nuclear Material Safety 

and Safeguards 

Enclosure: U.S. Nuclear Regulatory Commission 

Observation Audit Report, Audit M&O-ARP-99-009

cc: See attached list

-2-
Letter to D Shelor from C W Reamer dated

cc R Loux, State of Nevada 

S. Frishman, State of Nevada 

L. Barrett, DOE/Wash, DC 

A. Brownstein, DOE/Wash, DC 

S. Hanauer, DOE/Wash, DC 

C. Einberg, DOE/Wash, DC 

N. Slater, DOE/Wash, DC 

R. Dyer, YMPO 

S. Brocoum, YMPO 

R. Clark, YMPO 

A. Gil, YMPO 

G. Dials, M&O 

J. Bailey, M&O 

D. Wilkins, M&O 

M. Voegele, M&O 

S. Echols, M&O 

B. Price, Nevada Legislative Committee 

J. Meder, Nevada Legislative Counsel Bureau 

D. Bechtel, Clark County, NV 

E. von Tiesenhousen, Clark County, NV 

J. Regan, Churchill County, NV 

H. Ealey, Esmeralda County, NV 

L. Fiorenzi, Eureka County, NV 

A. Remus, Inyo County, CA 

T. Manzini, Lander County, NV 

E. Culverwell, Lincoln County, NV 

J. Wallis, Mineral County, NV 

L. Bradshaw, Nye County, NV 

M. Murphy, Nye County, NV 

J. McKnight, Nye County, NV 

N. Stellavato, Nye County, NV 

D. Kolkman, White Pine County, NV 

D. Weigel, GAO 

W. Barnard, NWTRB 

R. Holden, NCAI 

R. Arnold, Pahrump County, NV 

J. Lyznicky, AMA 

R. Clark, EPA 

F. Marcinowski, EPA 

R. Anderson, NEI 

R. McCullum, NEI 

S. Kraft, NEI 

J. Kessler, EPRI 

G. McKnight, Pahrump, NV 

R. Wallace, USGS 

R. Craig, USGS 

W. Booth, Engineering Svcs, LTD 

S. Trubatch, Winston & Strawn

Decemiber 14, 1999
U S NUCLEAR REGULATORY COMMISSION 

OBSERVATION AUDIT REPORT OAR-00-01 

OF THE YUCCA MOUNTAIN QUALITY ASSURANCE DIVISION 

AUDIT M&O-ARP-99-009 

OF THE 

CIVILIAN RADIOACTIVE WASTE MANAGEMENT SYSTEM 

MANAGEMENT AND OPERATING CONTRACTOR

K~(~12/18/99 Ted Carter 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management 

,_7/99 

Bruce Mabrito 

Center for Nuclear Waste Regulatory 

Analyses 

Reviewed and Approved by: 

S1__2//,W99 

'7iStablein 

Se fon Leader 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management

v,..,._.12/1/99 

William Dam 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management 

7,446ý,ý12/f /99 

Nerry Stirewaltn 

MANDEX, Inc

David Brooks" 

Section Leader 

High-Level Waste and Performance 

Assessment Branch 

Division of Waste Management

Enclosure
1.0 INTRODUCTION 

Staff and consultants of the U.S Nuclear Regulatory Commission (NRC) Division of Waste 

Management observed the U S Department of Energy (DOE), Office of Civilian Radioactive 

Waste Management (OCRWM), Office of Quality Assurance (OQA), Yucca Mountain Quality 

Assurance Division (YMQAD), limited scope, performance based audit of the Integrated Site 

Model (ISM) at the Management & Operating Contractor (M&O) facilities. The audit, M&O-ARP

99-009, was conducted on October 11-15, 1999, for work being performed at the facilities in Las 

Vegas, Nevada. 

The objective of this limited-scope performance based audit by YMQAD was to evaluate the 

implementation of the OCRWM program requirements and the technical activities associated 

with development of the ISM/Process Model Report (PMR) 

The NRC staff objective was to gain confidence that OQA and the M&O are properly 

implementing the requirements of their QA programs in accordance with OCRWM Quality 

Assurance Requirements and Description (QARD): DOE/RW-0333P and Title 10 of the Code of 

Federal Regulations (1OCFR), Part 60, Subpart G (which references 10 CFR Part 50, Appendix 

B). Also, the NRC staff observers were provided an opportunity to determine if data, software, 

and models supporting the ISM were properly qualified. 

This report addresses the effectiveness of the OQA audit and the adequacy of implementation 

of QA controls in the audited areas of the M&O PMR development. 

2.0 MANAGEMENT SUMMARY 

The NRC observers have determined that OQA Audit M&O-ARP-99-009 was useful and 

effective. The audit was organized and conducted in a professional manner. Audit team 

members were independent of the activities they audited. The audit team was qualified in their 

respective disciplines, and its assignments and checklist items were adequately described in the 

audit plan. 

The audit team concluded that the OCRWM QA program had been satisfactorily implemented in 

the areas evaluated. Three deficiencies were identified and one was corrected during the audit; 

therefore, two Deficiency Reports will be issued. Nine recommendations were offered as 

improvements/enhancements to the program. The data, software, and models supporting the 

ISM were reviewed and found to be properly qualified. 

The NRC observers agree with the audit team conclusion, findings, and recommendations. The 

NRC observers determined that this audit was effective and that the QA program 

implementation was adequate. Further, the NRC observers reviewed data and software 

packages, within the scope of the audit, and found them properly qualified.

I
3.0 AUDIT PARTICIPANTS

3.1 NRC

Ted Carter 

William Dam 

Bruce Mabrito 

Gerry Stirewalt 

3.2 Audit Team

Observer (Team Leader) 

Observer (Technical Specialist) 

Observer (QA Specialist - Center for Nuclear Waste Regulatory Analyses) 

Observer (Technical Specialist - MANDEX, Inc )

Daniel Klimas 

Robert Hasson 

Steve Harris 

Victor Barish 

Kristi Hodges 

John Doyle 

Keith Kersch

Audit Team Leader (ATL) 

Auditor 

Auditor 

Auditor 

Auditor 

Auditor 

Technical Specialist

OQA/Quality Assurance Technical 

Support Services (OQA/QATSS)

MACTEC 

OQA/QATSS -MACTEC 

OQA/QATSS -MACTEC 

OQA/QATSS -MACTEC 

OQA/QATSS -MACTEC 

OQA/QATSS -MACTEC 

OQA/QATSS-Science Applications 

International Corporation (SAIC)

4.0 REVIEW OF THE AUDIT AND AUDITED ORGANIZATION 

This OQA audit of the M&O was conducted in accordance with OCRWM Quality Assurance 

Procedure (QAP) 18.2, Internal Audit Program, and QAP 16.1 Q, Performance/Deficiency 

Reporting. The NRC staffs observation of this audit was based on the NRC procedure, "Conduct 

of Observation Audits," issued October 6, 1989. 

4.1 Scope of the Audit 

The audit team conducted a limited *scope, performance based audit of activities and processes 

supporting the development of the PMR for the ISM. Analysis Modeling Reports (AMRs), 

software, and supporting data for the ISM were evaluated as part of the audit process. The NRC 

representatives observed the audit team's evaluation of the following draft AMR deliverables:

S 

S 

0

Geologic Framework Model Report (GFM3.1) 

Rock Properties Model Report (RPM3.1) 

Mineralogic Model Report (MM3.0)

The process steps which were examined in relationship to accomplishing this assessment 

included the following processes as described in Administrative Procedure AP-3.1OQ "Analysis 

and Models":

S 

S 

S

Planning 

Developing and Documentation of Analysis and Models 

Validation of Models

2
"* Use of Software or Models 

"* Documentation Check and Review 

"* Approvals 

"* Editorial Corrections 

"* Analysis or Model Revisions or Change 

"* Submittal of Data and Models to the Technical Data Management System (TDMS) 

4.2 Conduct and Timing of the Audit 

The audit was performed in a professional manner and the audit team was well prepared and 

demonstrated a sound knowledge of the M&O and DOE QA programs. Audit team personnel 

were persistent in their interviews, challenged responses when appropriate, and performed an 

acceptable audit. The NRC staff believes the timing of the audit was appropriate for the auditors 

to evaluate the ongoing activities and implementation of the QA program. 

The DOE audit team and NRC observers caucused at the end of each day. Also, meetings of the 

audit team and M&O management (with the NRC observers present) were held each morning to 

discuss the current audit status and preliminary findings. 

4.3 Audit Team Qualification and Independence 

The qualifications of the Audit Team Leader and audit team members were found to be 

acceptable in that they each met the requirements of QAP 18 1, Auditor Qualification, as checked 

by the NRC Observation Audit Lead. The audit team members did not have prior responsibility for 

performing the activities they audited. In addition, training, education and experience records for 

audit team members were reviewed and found acceptable. 

The audit team members were prepared in the areas they were assigned to audit and were 

knowledgeable of applicable procedures. The checklist was adequately formulated and covered 

the subject matter well. 

4.4 Examination of Quality and Administrative Requirements 

Programmatic audit activities were conducted in accordance with the OCRWM QA Audit Plan for 

Audit M&O-ARP-99-009. The auditors reviewed documents identified in the audit plan and used 

checklists as a basis for inquiries. In addition, related documentation supporting report 

conclusions was reviewed to verify data source and status of qualification. Personnel directly 

responsible for document products or appropriate representatives with sufficient levels of 

knowledge were interviewed by the auditors. The checklists used were effective and additional 

inquiries were made beyond specific checklist items, when appropriate. The NRC observers were 

briefed on audit conduct procedures, including the inquiry process and method for raising . 

concerns. The NRC observers were given ample opportunity to provide comments and ask 

questions. 

The audit team reviewed training, education, and experience records for personnel conducting 

scientific studies to assure such personnel were in compliance with their individual position 

descriptions. Objective evidence was provided and reviewed by the auditors, and it was

3
determined that all personnel were appropriately qualified and assigned design tasks 

commensurate with their qualifications 

The NRC observers found that the requirements of QA Procedure 18 2 were implemented in an 

effective and satisfactory manner These planning and implementation activities were 

accomplished and observed by the NRC observers as follows: 

(1) distlibution of a quality assurance audit plan, M&O-ARP-99-009, 

(2) development of a performance based audit checklist, 

(3) coordination and communications with all team members; 

(4) conduct of an introductory pre-audit kickoff meeting with the audit team and observers; 

(5) conduct of a kickoff meeting with M&O personnel including high level management; 

(6) daily caucus meetings held for the audit team and observers; 

(7) daily management status meetings held for M&O management, and 

(8) conduct of a post audit meeting with M&O management, audit teams members and 

NRC observers. 

4.5 Examination of Technical Activities 

The DOE Integrated Site Model (ISM 3 1) is a 3-D visual computer model of Yucca Mountain that 

displays, and enables analysis of, geological information. ISM provides input into hydrologic flow, 

radionuclide transport, waste package design, and performance assessment modeling More 

specifically, ISM incorporates into the geologic framework, represented by models of structural 

geology and stratigraphy, porosity and matrix mineralogy models. 

ISM combines the 3D Geologic Framework Model (GFM3.1) developed by the M&O contractor to 

the DOE, the Rock Properties Model (RPM3 1) developed at Sandia National Laboratory, and the 

Mineralogy Model (MM3.0) developed at Los Alamos National Laboratory. 

4.5.1 Process and Analysis Model Reports (PMRIAMR) 

The following four unpublished, unapproved, draft reports (one PMR and three AMR's) were 

delivered to the NRC technical observers two work days prior to the audit. These reports are 

expected to be finalized in December 1999. 

1. TRW Environmental Systems, Inc., 1999. Integrated Site Model, Process Model 

Report, TDR-NBS-GS-000001, Rev. OOA. 

2. Clayton, R., 1999. Geologic Framework Model (GFM3.1), Analysis Model Report, 

MDL-NBS-GS-000002, Rev OOC. 

3. Carey, J.W. 1999. Mineralogy Model (MM3.0), Analysis Model Report, MDL-NBS-GS

000003, Rev OOB. 

4. Rautman, C.A. 1999. Rock Properties Model (RPM3.1), Analysis Model Report, MDL

NBS-GS-000004, Rev OOC. 

Additional unpublished reports supporting audit observations were briefly examined by NRC staff 

on the following subjects:

4
1 Geophysical forensics report creating single curves from borehole logs. by Howard 

Rael 

2. Surveillance report to review core data supporting lithostratigraphic contacts, by Ken 

Gilkerson (LVMO-SR-99-021) 

4.5.2 Administrative Procedures Observed 

A comprehensive checklist and associated Administrative Procedures were assembled by the 

DOE audit team to evaluate the reports describing data, software, and models. The checklist and 

procedures were provided to the NRC observers on the first day of the audit, thus, a critical review 

of the procedures was beyond the scope of the NRC observation. Approximately eight DOE 

contractors were interviewed during the audit process. These individuals were primarily 

responsible for interpreting data, building models, and preparing or reviewing the four draft reports 

listed above. 

The NRC technical observers noted that the DOE audit team examined procedures for the 

following items: 1) analyses and models (AP-3.10Q); 2) software codes (AP-SI.1Q); 3) technical 

reports (AP-3.11Q); and 4) scientific notebooks (AP-SIII.1Q). The auditors carefully and throughly 

evaluated the reports based on the AP procedures. 

The independence of the technical checker review was determined to be adequate by the DOE 

audit team and NRC observers. The observers noted that the reviewer's scope focused on 

AMR/PMR reports, including careful checks on the data presentations, and not on the ISM model 

Errors were found in data plotted in the GFM3.1 report, but these discrepancies did not affect the 

quality of the Geologic Framework Model. 

Procedures for using software were carefully considered as part of the DOE audit. EarthVision, 

developed by Dynamic Graphics, Inc., was used to construct GFM3.1 and ISM3.1. EarthVision 

has been qualified by DOE and throughly reviewed by the NRC. However, four codes used in the 

development of MM3.0 and RPM3.1 have not been qualified: STRATAMODEL, GSLIB, QUAL2, 

and WILDSOFT. These software codes are now in the process of being qualified. The use of 

unqualified software in the AMRs resulted in a deficiency report (DR No. LVMO-O0-D-12) issued 

by the auditors as described in section 4.7.1. 

Procedures for collecting qualified borehole geophysical logs were described by the DOE auditors 

and an M&O expert on geophysical log interpretation. The logs enable determining elevations of 

lithostratigraphic contacts and generating porosity data for the lithostratigrahic units which are 

essential input into ISM3.1. Several boreholes have been repeatedly logged with similar results 

including G2, G10, G12, WT2. A problem in logging borehole SD-6 was described where the 

logging tool may have become stuck near the water table. The tool was eventually removed. 

Review of scientific notebooks was conducted. The notebooks for GFM3.1 and MM3.0 had not 

been processed for the technical review which is required before the AMR/PMR can be released. 

The NRC observers briefly examined the scientific notebooks and found that sufficient 

documentation was adequately compiled by the technical investigators.

5
4.5.3 Data Review and Qualification 

The DOE audit technical specialist and NRC observers examined geologic data supporting 

lithostratigraphic contact elevations, which is essential input for GFM3.1. Although it appears that 

most of the Data Tracking Numbers (DTNs) have been qualified for input into GFM3.1, the NRC 

observers are concerned that one unqualified DTN (M0981 1 MWDGFM03.000) contained over 

2,000 data values that are in the process of qualification. This data file was examined and 

contained lithostratigraphic contact elevations derived from core and geophysical log 

interpretations. It was not apparent that individual DTNs have been assigned for each data value 

within this file. The PMR report should better reflect the contents of this unqualified DTN. 

The NRC observers suggested that determining the percentage of data qualified should perhaps 

be based on the data contents within the DTN and not on the individual DTNs. For example, if 

nine of 10 DTNs contained within GFM3.1 are qualified, a value of 90% qualification would not be 

representative of the data that are qualified if the one unqualified DTN contains a large amount of 

data supporting the model. 

To qualify old, unqualified core samples (following procedure AP-SIII.2Q) from 10 boreholes, a 

very comprehensive analysis is being conducted using cores, core photos, geophysical and 

caliper logs, cuttings and downhole video logs to support lithostratigraphic contacts. (A previous 

surveillance was observed and reported by an NRC staff member the week before this audit). 

NRC observers requested a special meeting after the audit was complete to review core at the 

Sample Management Facility. The LANL geologist demonstrated his process to qualify old core 

samples to enable qualification of mineralogy data supporting MM3.0. Observers examined core 

from borehole G4 which was carefully keyed to an Excel spreadsheet. Numerous methods to 

determine stratigraphic elevations were presented with associated uncertainty measurements. 

The NRC observers were impressed by comments from the LANL geologist that his procedure 

was more stringent than the AP-SIII.2Q procedure requirements by using recently collected and 

qualified geophysical logs to compare with unqualified logs. The procedure allows corroboration 

of data between different sets of unqualified data. The LANL geologist also stated that borehole 

geophysical depth data collected in English units has been converted into metric units by various 

investigators. The accuracy of the data is influenced by the number of significant figures used in 

the feet/meter conversion. Investigators are not always accurately converting data by using fewer 

than seven significant figures. 

The process of updating the database for newly qualified data was examined. The NRC 

observers were provided a copy of one DTN that was searched by a DOE auditor. The DTN 

(SNL01A05059301.005) pertains to lab thermal conductivity data for four boreholes. The 

auditors determined that the qualification status was changed prematurely in the Automated 

Technical Data Tracking System. A verbal authorization had been given to change the status to 

qualified data but the written documentation had not been completed, contrary to required 

procedures. 

4.5.4 Observation-of ISM in Operation 

The NRC technical observer who is most familiar with the EarthVision software application 

requested a meeting with the individual at the M&O responsible for ISM3.1 construction to 

evaluate the visual software program. During this brief but detailed and fruitful examination of

6
ISM3 1 the NRC observer was able to determine that the model is running in a functional manner 

on EarthVision software and that it is constructed so that each rock property (e g, porosity) and 

mineralogical constituent included in the model is developed as a separate" face" file which can 

be separately analyzed with the EarthVision 3D viewer The NRC observer was able to 

interactively manipulate the rock property and mineralogic files and examine the final product in 

enough detail to gain an understanding of how the ISM was constructed and how readily it can be 

analyzed using EarthVision software This software package is in use at the NRC in the 

Computerized Risk Assessment and Data Analysis Lab (CRADAL). 

4.5.5 NRC Staff Recommendations 

R-00-01 -1 

Determining the percentage of data that are qualified should be based on the total data content of 

the total number of DTNs and not on the DTN only, since it is important to focus on the data for 

qualification. Clearly DTNs contain different types and amounts of data which vary in relative

importance in the final product. 

R-00-01-2 

The scope of future AMR/PMR audits could be expanded to include demonstration and technical 

review of the computer-based, final technical product In this case, ISM3.1 was examined by the 

DOE technical auditors and NRC technical observers after the request was made by the 

observers. 

R-00-01-3 

The technical checker/reviewer's scope could be expanded to include review of the final software 

product (e.g., the 3-D ISM in this case, which was developed using EarthVision software from 

Dynamic Graphics, Inc.). 

4.6 NRC Staff Findings 

The NRC staff has determined that OQA Audit M&O-ARP-99-009 was useful and effective. The 

audit was organized and conducted in a professional manner. Audit team members were 

independent of the activities they audited. The audit team was well qualified in the QA discipline, 

and its assignments and checklist items were adequately described in the audit plan. 

The audit team concluded that the OCRWM QA program had been satisfactorily implemented. 

Three deficiencies were identified and one was corrected during the audit; therefore, two 

Deficiency Reports will be issued. Nine recommendations were offered as 

improvementslenhancements to the program. The data, software, and models supporting the ISM 

were reviewed and found to be properly qualified. The NRC observers agree with the audit team 

conclusion, findings, and recommendations. 

The NRC observers determined that this audit was effective and that the QA program 

implementation was adequate. The NRC observers also reviewed data and software, within the 

scope of the audit, and found them properly qualified.

7
4.6.1 Audit Observer Inquiries (AOIs)

The following NRC observer questions were documented regarding performance of the audit or 

the audited organization's program The NRC Observers received a briefing to AOI No M&O

ARP-99-009-1, but at the time of writing this report, formal written responses have not been 

completed by the auditors. Responses to each question on AOl No. M&O-ARP-99-009-2 and AOl 

No. M&O-ARP-99-009-3 documented verbal questions raised during the audit and subsequent 

responses have been provided by the technical auditor. The AOI forms provide an important 

record of the NRC observer questions and auditors' responses. The timing of the questions 

helped to influence the course of the audit and in some cases incomplete answers (such as AOI 

No. M&O-ARP-99-009-3, question 6) indicate that technical issues remain a concern to the NRC 

technical observers. 

AOI No. M&O-ARP-99-009-1 

Question 1: 

How are the 1,200-1,300 DTNs supporting the AMRs prioritized? 

Response 1: 

Awaiting formal written response from OQA Audit Team. 

Question 2: 

What criteria are used to determine the significant/important DTNs? 

Response 2: 

Awaiting formal written response from OQA Audit Team. 

Question 3: 

How are the more significant DTNs being scheduled for completion? 

Response 3: 

Awaiting formal written response from OQA Audit Team. 

Question 4: 

It is our understanding that 80-percent of the data to be used for the License Application will 

be available at the time of Site Recommendation. Will this include all of the scientific and 

engineering software issued? 

Response 4: 

Awaiting formal written response from OQA Audit Team. 

Closure of AOI No. M&O-ARP-99-009-1 will be in a future NRC observation audit report. The 

NRC tracking number for open AOI No. M&O-ARP-99-009-1 is NRC Open Item No. 00-1-1. 

AOl No. M&O-ARP-99-009-2 

Question 1: 

The ISM is developed by combining the GFM, RPM, and MM. Considering that QA/QC should 

assure the quality and consequent usability of the final ISM product, will the ISM be 

reviewed/viewed during the audit as well as the separate pieces (i.e. GFM, RPM, MM)? 

Response 1: 

A detailed demonstration of the models was provided by M&O staff, showing the functional 

relations between the models.

8
Question 2. 

Assuming the ISM is a 3D property model running on EarthVision software. is R Clayton 

responsible for assembling the 3 pieces into a single ISM?' 

Response 2 

Yes 

Question 3 

Is it not practical to determine that implementation of procedures has led to generation of a 

functional ISM by examining the 3D model with EarthVision Software? 

Response 3: 

R. Clayton demonstrated the capabilities to be functional to the satisfaction of those present. 

AOl No. M&O-ARP-99-009-2 was closed during the conduct of this audit. 

AO! No. M&O-ARP-99-009-3 

Question 1: 

The audit team focused on reviewing technical aspects of geophysical logs. Has other data 

that are used to correlate GFM lithostratigraphic contacts been audited? These data include 

cores, cuttings, and downhole videos? 

Response 1: 

These issues were discussed with R. Clayton. Contacts were originally picked using cores, 

then confirmed using logs 

Question 2: 

Does this data (from cores, etc.) correlate with geophysical logs? 

Response 2: 

Because logs average a large volume, it is common to see a few feet variability between logs 

and cores. 

Question 3: 

Has porosity and mineralogy data represented in RPM and MM been used to improve or verify 

the horizons (or other information) in GFM to fully integrate ISM? 

Response 3: 

Core samples are often too widely spread to obtain accurate contas.---- . 

Question 4: 

Has data from RPM and MM been restricted or removed due to limitations of GFM? A detailed 

list of data used and not used in ISM is needed. 

Response 4: 

All available data were examined. Lists of data are included in AMRs. 

Question 5: 

The Calico Hills unit in GFM is represented by 2 units that are not based on vitric or zeolitic 

percentages. Can the data from MM improve the representation of these vitric and zeolitic 

sequences within the ISM?

9 -
Response 5 

The Calico Hills is actually one unit and the Calico Hills Bedded Tuff is another in the GFM In 

the MM it was divided into four equal units for better definition There are no uniform 

mineralogic differences to justify other descriptions 

Question 6" 

Has the audit team evaluated the sufficiency of data present in the 3 models ? Has the ISM 

been evaluated by statistical methods to determine spatially varying uncertainty as was done 

in RPM (see p. 117 of AMR)? 

Response 6: 

Statistical methods were not used 

The response to Question 6 did not address sufficiency of data. This will be further evaluated by 

the NRC technical staff and does not require "open item" status. AOI No. M&O-ARP-99-009-3 

was closed during the conduct of this audit 

4.6.2 NRC Summary Comments 

The scope of the audit as specified by the DOE audit team was successfully met..The final 

computer model product, ISM3.1, was examined by the auditors when the NRC technical 

observers requested that this be done. The technical observers recommend that DOE consider 

expanding the scope of audits and the checking process to include the final model product. 

4.7 Summary of YMQAD/OQA Findings 

The audit team identified three deficiencies of which one was corrected during the audit. 

Additionally, there were nine recommendations resulting from the audit. The NRC staff agrees 

with the audit team findings and recommendations. 

4.7.1 Deficiency Reports (DRs)of Audit M&O-ARP-99-009: 

DR No. LVMO-00-D-011 

Condition: Technical Change Requests (TCRs); 1999-0087 "Initial Rock Properties Model 

(RPM.3.1) Analysis Model Report, T1999-0088 "Initial Mineralogical Model (MM3.0) Analysis 

Model Report", T1999-0133 "Initial Geologic Framework Model (GFM3.1) Analysis Report" and 

T1999-0216 "PMR-Integrated Site Model" have been initiated and submitted to the CCB 

Secretary. The above Model Reports are being developed without specifically satisfying AP-3.4Q 

paragraphs 5.2.1 through 5.2.6. 

DR No. LVMO-00-D-12 

Condition: Several software packages have been used to support the AMRs that support the ISM. 

These software packages have not been qualified by following the steps of the procedure in that 

they are either at or about the Control Point 1 or have not been qualified as a routine or macro 

The following software was found not to be qualified: 1) Wildsoft v.1.65, 2) QLA2/GES v.1.0, 3) 

Stratamodel v.4.1.1, 4) VARIO v.1.16 and v.1.20, 5) VARIOFIT v. 1.20.

10
4.7.2 Deficiencies Corrected During the Audit (CDA):

A memo was issued to lift a "Global' To Be Verified (TBV) condition, however, the "Data or 

Technical Information Checklist" for reverification was not yet completed There was an 

unresolved question regarding the statuslqualification of the software identified in the reverification 

checklist There was no negative impact since the TBV number had not been removed from the 

TDMS National Environment Program Operation (NEPO) personnel performed a review and 

determined that there were no other Document Traceability Numbers with TBVs lifted prior to 

completion of the reverification checklists This was considered to be an isolated condition and 

was corrected during the audit. 

4.7.3 OQA Audit Team Recommendations: 

Nine recommendations were offered as improvementslenhancements to the program. 

a) Project has consolidated qualification of personnel under a single procedure (AP-2.2Q) 

However, personnel supporting the various PMRs, including the ISM, have been qualified 

under superseded procedures, e.g., laboratory qualification procedures Because AP

2.2Q requires the responsible manager to evaluate personnel qualifications, there needs 

to be a determination of who is the responsible manager. Secondly, if previous 

qualification documentation is to be relied upon as evidence of personnel qualification, the 

functional manager needs to assure that assigned personnel have the appropriate 

education and/or experience to perform the assigned tasks. 

b) The Data Qualification Plan, "Logs of Geophysical Recording Runs," was developed in 

accordance with a previous procedure. It is recommended that it be revised in accordance 

with the current procedure. 

c) Procedures AP-3.1OQ, AP-2.14Q, and AP-SI.1Q all speak about performing technical 

reviews. There is also a QAP-6.2 review. There seems to be confusion and conflicting 

direction as to which procedures are applicable. It is recommended that the review 

process be evaluated to be made as uniform as possible among procedures. 

d) Procedure AP-3.10Q is not clear in addressing independent technical reviews. It is not 

clear as to the justification by the AMR/PMR Leads as to by pass the AP-2.14Q technical 

review for AMRs. Additionally, if the Checker is the function for independent technical 

review, it is recommended that the procedure clearly state this. 

e) In the Mineralogy report, mineral abundance is calculated using an equation referenced in 

the AMR. It is recommended that justification be provided in the AMR and Scientific 

Notebook as to why this equation was used to further strengthen the selection. 

f) AP-SIII.1Q, Scientific Notebooks, requires a technical review on the notebook when it 

supports a deliverable. AP-310Q also has review requirements. It is recommended that 

this review be completed prior to issuing the AMRs and make sure all procedure steps are 

met in both procedures for review on the notebooks. 

g) The AP-SI.1Q definition for software routine can be a stand alone code, unchanged from 

the supplier. However, the text expects the source code to be available. It is 

recommended the procedure be changed to permit use of vendor supplied software 

unchanged by the user. 

h) Some procedures have been found that have requirements that overlap the requirements 

and processes of others, and the procedures have been in a considerable state of flux. It 

is recommended that there be a single entity or organization section assigned the task of

11] -
creating a matrix of the requirements of one procedure to the others when there are 

requirements that overlap and clarify the processes such that the staff have clear direction 

in the relationship and implementing requirements for their specific tasks 

i) There is a Guidance List that was developed by the NEPO to assist Responsible Manager 

and personnel in implementing the AP-3 10Q process, Analysis and Modeling This is a 

step-by-step guide to implement the AP-3 10Q process with a cross-reference to other 

applicable procedure requirements. It is recommended that this guidance list be 

distributed to all organizations developing AMRs to assist in assuring full implementation of 

procedure requirements.

12