Issue Date: 07/12/00 - 1 - 2410

ˇ ̨ ̋ ̧ ̊ˇ ̇ ̆ ̄ ̋ ̃ ̊ˆˇ ̧ıÙˇÛÙÚ ˇ ̨ ̋ ̋

 ̨ˇ

CONDUCT OF OBSERVATION AUDITS

2410-01 PURPOSE

The purpose of this procedure is to provide guidance to the U. S. Nuclear Regulatory Commission’s

(NRC) Division of Waste Management (DWM) staff on how to conduct observation audits of

quality assurance (QA) audits performed by the U. S. Department of Energy (DOE), its contractors,

and its subcontractors.

2410-02 OBJECTIVE

02.01 General. The objective of the NRC observation audit program is to gain confidence that

the DOE and its contractors and subcontractors are properly implementing the provisions contained

in its QA program description document, the Quality Assurance Requirements and Description

(QARD); the requirements contained in Subpart G, “Quality Assurance,” to Part 60, of Title 10 of

the U. S. Code of Federal Regulations (10 CFR Part 60); and the requirements contained in

Subpart G, Quality Assurance, of 10 CFR Part 63, when it has been approved by the Commission.

02.02 Specific

a. Evaluate the effectiveness of the audit team to identify and evaluate deficiencies in the QA

program of the organization being audited by:

1. Reviewing the activities of the DOE audit team, including audit planning, preparation

of auditors, scope of the audit, performance of the auditors, and conclusions of the

audit team concerning the organization or activity being audited.

2. Forming conclusions about the acceptability and effectiveness of the audit.

b. Evaluate the implementation of the QA program of the organization or activity being

audited by:

1. Reviewing the procedures, work activities, and work products sampled by the DOE

auditors.

2. Forming conclusions about the effectiveness and acceptability of the organization

being audited in implementing its QA program.

c. Based on Sections 02.02 a and b above, determine the overall acceptability of DOE QA

program for high-level waste (HWL).

2410-03 DEFINITIONS

03.01 Affected Local Units of Government. Any borough, city, county, parish, town, township,

village, or other general-purpose political subdivision of a State.
2410 - 2 - Issue Date: 07/12/00

03.02 Audit Observer Inquiry (AOI). A question or request for information concerning an activity

or item for which the NRC requires further explanation, in order to judge its acceptability or to

obtain additional information, to better understand the rationale for an action or decision. For the

purpose of this document, an AOI can become a finding and is also used to track an open item.

03.03 Audit. The review or examination of a limited sample of work activities and work products.

03.04 Deficiency. A departure from specific requirements. This includes the failure to satisfy a

written commitment or the failure to conform to the provisions of applicable codes, standards,

guides, or accepted industry or scientific practices that the DOE or an organization being audited

have imposed.

03.05 Finding. A deficiency, observation, good practice, weakness, or request for information

(also see definition for AOI).

03.06 Good Practice. Any activity or item, that the NRC observers have identified or adopted,

which, although not a requirement, enhances the audit or the QA program of the organization being

audited.

03.07 Observation. A deficiency, the NRC observers have identified or adopted, within the DOE

audit team or QA program of the audited organization.

03.08 Observation Audit. The process of over viewing and evaluating an audit and the QA

program, work activities, and work products of the audited organization.

03.09 NRC Observer. One who performs NRC observation audits. A representative(s) sent

primarily to observe; but not to participate, substantially, in an activity (as in a meeting, audit, or

surveillance). Observers may furnish questions, observations, and recommendations generally at

the beginning and end of meetings.

03.10 Weakness. An activity or item, identified or adopted by the NRC observers, which - though

not a requirement - is less than optimum for the audit or for the organization being audited.

2410-04 RESPONSIBILITIES

04.01 Branch Chief, High-Level Waste & Performance Branch (HLWB). Review the observation

audit report and sign transmittal letter.

04.02 Section Chief, Project and Engineering Section, High-Level Waste & Performance Branch

(HLWB)

a. Approve the observation audit schedule.

b. Approve and sign the observation audit report after ensuring that it is technically correct

and that sufficient information is included to justify the conclusion.

04.03 Section Chief. Approve and sign the observation audit report after ensuring that it is

technically correct and that sufficient information is included to justify the conclusion.

04.04 QA Engineer

a. Prepare, and revise as needed, the observation audit schedule.

b. Designate an Observation Audit Team Leader.

c. Assure that observers are adequately indoctrinated and trained for the observation audit.

d. Keep DWM management informed of progress and results of the observation audit.

e. Determine an appropriate course of action if there are significant differences between the

DOE audit team’s findings reported at the exit meeting and those that are documented in

the DOE audit report.
Issue Date: 07/12/00 - 3 - 2410

f. Issue current DOE audit/surveillance schedules (and subsequent revisions thereto) to the

Section Chief, in advance, to keep the Section Chief informed of the technical aspects

being audited/surveilled; to allow sufficient lead time for planning and personnel resources.

04.05 Observation Audit Team Leader

a. Coordinate the arrangements for the observation audit, including, but not limited to, air

travel, arrival and departure times, hotel accommodation, vehicle transportation,

directions, and meetings.

b. Select the NRC staff observers, with input from the QA Engineer and Section Chiefs.

c. Ensure that the observers are given sufficient time free from other duties to adequately

prepare for the observation audit.

d. Research prior NRC observer and DOE audit findings for the past two audits and provide

this information to NRC observer team members to alert them to potential recurrences or

ineffective corrective action implementation.

e. Distribute the documents received from the DOE to the team members and resolve

questions concerning the documents.

f. Lead observation audit team during the audit.

g. Emphasize to all observers that they are there to observe and not audit.

h. Keep QA Engineer and QA Section Chief informed of progress and results during and

after the observation audit is completed.

i Integrate the input from the observation audit team members.

j. Serve as principal spokesperson for the NRC during the audit. The team leader may rely

on observation audit team members to explain their findings or discuss other topics within

their disciplines.

k. Recommend to the DOE Audit Team Leader (ATL) any items of interest that should be

added to the audit scope.

l. Ensure that all observation audit team members’ findings, methods, etc. are consistent

with NRC and HLW policies and procedures.

m. Integrate NRC observers’ evaluations of the DOE audit team and the audited

organization’s QA program.

n. Prepare “Item of Interest to the Commission” (Exhibit 1).

o. Forward “Item of Interest to the Commission” through the QA Section Chief.

p. Coordinate preparation of the Observation Audit Report in accordance with Section 2410-

08 of this document.

q. Track each AOI through satisfactory resolution.

r. Document the actions taken to resolve each AOI.

s. Investigate differences between the audit team’s findings reported at the exit meeting and

those included in the audit report. Report significant differences to the QA Engineer and

QA Section Chief.

t. Prepare, sign, and issue the observation audit report.

u. Obtain the appropriate NRC signatures and concurrences.

04.06 Observation Audit Team Members (Observers)

a. Prepare for the observation audit in accordance with Section 06.03c of this document.
2410 - 4 - Issue Date: 07/12/00

b. Evaluate the audit in accordance with Sections 06.04 a and b of this document.

c. Evaluate the QA program of the audited organization in accordance with Section 06.06 a

and b of this document.

d. Explain observation findings to the DOE audit team.

e. Prepare input for the Observation Audit Report (for area of responsibility) in accordance

with Section 2410-08 of this document. Include all findings.

f. Provide input to the Observation Audit Team Leader within 5 working days after

completing the observation audit.

g. Sign the final observation audit report. If the observation audit team member is not

available, the Observation Audit Team Leader or the member’s supervisor can sign for

the team member.

h. Obtain specialize training, if necessary, to perform observation audit.

QA observers will be primarily responsible for evaluating the DOE audit team’s operation and its

assessment of the QA program of the audited organization. Technical observers will be primarily

responsible for evaluating the effectiveness of the DOE audit team in assessing the quality of the

technical work and evaluating the technical work activities and work of the audited organization.

Because these areas overlap and because observers may possess qualifications in areas outside

their specific responsibilities, technical and QA observers shall coordinate and integrate their

review.

2410-05 QUALIFICATION AND SELECTION OF OBSERVERS

05.01 Observers are selected by the Observation Audit Team Leader with input from the QA

Engineer and Section Chiefs. Observers shall have experience or training commensurate with the

scope, complexity, or special nature of the activities to be audited. For example, technical

observers shall be selected based on their education and experience in the technical area being

audited.

05.02 Observers should be trained in the following topics before the audit. The QA Engineer will

determine the type of training.

a. 10 CFR Parts 60 and 63 QA Requirements.

b. Audit fundamentals.

c. Applicable requirement documents.

d. DOE/NRC protocol for observers.

e. Conduct of observers.

f. The DOE QARD and audit procedures.

2410-06 OBSERVATION AUDIT GUIDANCE

NRC observation audits may be performed on audits by the DOE, its contractors, its

subcontractors, and other organizations participating in quality-affecting HLW activities. For

example, the NRC staff may observe a United States Geologic Survey audit of one of its

contractors (Note: USGS is a DOE contractor in the HLW program.

06.01 The selection of audits for observation should be based on the following:

a. The importance to public radiological health and importance to safety and/or waste

isolation of the activity being audited.
Issue Date: 07/12/00 - 5 - 2410

b. The results of previous audits, observation audits, or other reviews by the NRC or the

DOE, particularly those that identified major concerns.

c. The audit planned by the DOE, as shown in the Office of Civilian Radioactive Waste

Management QA Internal Audit Schedule.

d. Availability of staff resources and the time since the last audit .

06.02 Areas to be Observed. NRC observers are necessarily constrained in the activities and

work products that they can observe by the scope and conduct of the audit being observed.

However, within these constraints, there are two major areas to be observed. These are: 1) the

activities of the auditors; and 2) the work activities/products of the organization being audited. The

primary focus of observers should be on items and activities that are most important to public

radiological health and safety and waste isolation.

The DOE audit team will normally hold daily team meetings to review the auditor’s findings, and the

ATL will normally hold daily meetings with the organization being audited to appraise it of the status

of the audit. The observation audit team should be represented at these meetings. The

observation audit team should note findings reported by the audit during the daily audit team

meetings, including any deficiencies resolved during the course of the audit.

06.03 Preparation for the Observation Audit

a. In preparation for the audit, the Observation Audit Team Leader will coordinate a pre-audit

meeting conference call(s) involving NRC and its contractor and DOE audit team to

accomplish the following:

1. Coordinate audit observer team assignments.

2. Discuss audit scope, protocol, logistics, schedule, and security requirements.

3. Discuss and resolve all questions that the NRC observation team might have

pertaining to the audit or documentation supporting the audit.

4. Introduce the members of the DOE audit team and NRC observation audit team and

discuss the role and responsibility of each member.

Depending on the scope of the audit, the Observation Audit Team Leader may need to

hold team and planning sessions to prepare for the audit observation.

b. The Observation Audit Team Leader will obtain the following information from the DOE in

a timely manner and before the observation audit:

1. The audit notification letter.

2. The audit plan, including the QA and technical checklists.

3. The QA and technical procedures that will be audited (when requested by the staff).

4. Previous applicable audit reports, including adverse findings and resulting corrective

actions (when requested by the staff).

5. As applicable, technical documents to be reviewed during the audit or those that

serve as the basis for the audit check list.

c. Observers shall accomplish the following before the audit.

1. Complete required audit observer training.

2. Review QA plan and program under which the DOE audit team will be auditing.

3. Review a sample of implementing procedures (when they have been requested and

received from the DOE) for the work activities to be audited.
2410 - 6 - Issue Date: 07/12/00

4. Review applicable reports of past audits and surveillances (up to 18 months old) of

the organization to be audited.

5. Determine areas in which observers should be focused, based on past surveillance

reports, audit reports, observation audit reports, status reports, etc.

6. Review the DOE audit plan and checklists to determine whether they are sufficient

to establish overall QA program effectiveness.

7. Recommend to the Observation Audit Team Leader items of interest that should be

added to the audit scope.

8. Review applicable DOE technical documents.

06.04 Conduct and Evaluation of the Observation Audit

Observers should place more emphasis on team performance than on the team‘s compliance with

audit procedures, plans, and checklists. Observers must evaluate whether the DOE audit team

verified that the audited organization’s QA program is producing quality work products such as

reviews, reports, software, data, and test procedures and the objective evidence (usually

documentation) to defend that work in licensing.

a. Observers should evaluate the effectiveness of the audit in verifying the quality of work

activities and end products. For example:

1. Were the appropriate samples of work activities and work products selected for the

audit?

2. Were findings identified by the auditors during the course of the audit investigated

to determine their significance and effect on public radiological health and safety

and waste isolation?

3. Were work activities verified as adequately controlled to ensure the quality of the

work products?

4. Was the acceptability of the work products and their documentation verified?

5. Did the audit result in meaningful, substantive, and valid findings?

6. Is the audit program adequate?

b. Observers should evaluate whether the DOE audit team determined whether the audited

organization’s QA program was implemented effectively. For example:

1. Were the appropriate work activities and work products selected to be audited?

2. Were adverse findings identified by the DOE audit team followed up during the

course of the audit to determine their significance and effect on items important to

public radiological health and safety and waste isolation?

3. Did the DOE audit team verify that the technical activities being audited were being

performed and documented acceptably?

4. Did the DOE audit team evaluate the adequacy of field and laboratory data?

5. Did the DOE audit team evaluate the source and correctness of design inputs?

6. Did the DOE audit team evaluate the calculation techniques and mathematical

models used?

7. Did the DOE audit team evaluate the appropriateness and application of the

conclusions of the audited organization’s technical staff?

06.05 Conduct of Observation Surveillances. Observation Surveillances may be performed using

the guidelines specified in this document. However, surveillances are limited in scope and all
Issue Date: 07/12/00 - 7 - 2410

provisions stated in this document may not apply. The scope and schedule of surveillances will be

determined by the QA Engineer, with approval from the Section Chief, HLWB.

06.06 Evaluation of the Audited Organization’s QA Program

a. Observers should evaluate the implementation of the audited organization’s QA program

based on the findings identified by the DOE audit team and on their own findings.

b. Observers should emphasize evaluation of the audited organization’s QA program

effectiveness, timely identification, and correction of deficiencies. For example:

1. Do findings identified by the audited organization’s QA program significantly affect

activities and items important to public radiological health and safety or waste

isolation?

2. Are the findings singular events or do they indicate a programmatic problem?

3. Are the findings identical or similar to findings previously identified?

4. Were the findings identified before initiation of, or early in the work activity, or after

completion of the activity? (Is first-line QA finding the problem?)

c. Technical observers should emphasize evaluation of the audited organization’s QA

program effectiveness in ensuring a quality product. For example:

1. Are sufficient procedures in place to ensure technically adequate work activities and

end products and adequate documentation of the quality?

2. Are the procedures appropriate and reasonable to accomplish the activity and have

they been properly implemented?

3. Are field and laboratory data adequate?

4. Is the source of design input identified and acceptable, and is the design input

correct?

5. Are the calculation techniques and mathematical models acceptable?

6. Are appropriate conclusions drawn by the technical staff of the audited

organization?

7. Does the audit indicate that the technical work being audited is being performed and

documented in an adequate fashion?

8. Is the performance of each audit activity consistent with good practice?

9. Are conclusions based on the work being reviewed for consistency with the input

data?

10. Are designs being reviewed adequately to ensure the designed items will meet their

design criteria?

06.07 Significance of Observations

a. Some of the questions that observers should consider in determining the significance of

their observations are:

1. Is the basis of the observation a procedural deficiency or an individual error?

2. Is the basis of the observation a recurring, programmatic problem or an isolated

event?
2410 - 8 - Issue Date: 07/12/00

3. Does the observation indicate an inability of the management of the DOE audit

team or the audited organization to identify or correct problems?

4. Is the observation significant in its effect on the quality of work activities or work

products?

5. Can the NRC defend the overall adequacy of the audit and of the audited

organization’s QA program in light of the observation?

2410-07 PROTOCOL DURING THE OBSERVATION AUDIT

07.01 General. Observers should conduct themselves in accordance with the guidelines given

in the “Agreement between DOE/OCRWM and NRC/NMSS Regarding Prelicensing Interactions,”

dated November 16, 1998 until Manual Chapters are developed.

07.02 Specific. During the audit, observers shall conduct themselves in a professional and

cooperative manner. They shall coordinate with the Observation Audit Team Leader to ensure that

the effectiveness of the DOE audit team is not decreased. The observers should attend the

entrance and exit meetings and audit team caucuses whenever possible. The observers should

try to minimize direct questions of the audited organization.

Observers shall not take over the audit. Observers are encouraged to participate by documenting

their request for information to the DOE ATL, in writing, through the use of the AOI form (Exhibit

2) or equivalent. Responses to observers’ questions should be timely; if not, the Observation Audit

Team Leader should notify the ATL.

Before the exit meeting, usually the night before, the NRC observers, or Observation Audit Team

Leader, or both, may meet with the other observers (if any from the State, Affected Local Units of

Government, etc.) and brief them on the NRC’s preliminary findings and conclusions. The

Observation Audit Team Leader will also brief the DOE ATL or audit team or both on the same

matter. There should be no surprises in the NRC Observation Report.

All findings should be communicated to the ATL in a clear and timely manner. All findings should

be based on facts. Personal opinions should be avoided. Observers should report their findings

regarding the audited organization’s QA program to the ATL. Observer findings regarding the

conduct of the audit should be addressed through the Observation Team Leader to the ATL unless

directed otherwise by the Observation Audit Team Leader. Every attempt should be made to report

findings daily to the ATL.

The ATL should be given the opportunity to respond to observation audit team findings. Observers

should consider any new information to determine if findings are still valid. Efforts should be made

to reach agreement with the ATL on the nature of each finding and, when necessary, on

appropriate corrective action being taken.

The Observation Audit Team Leader shall report any findings as well as the acceptable areas of

the audit program to the ATL before leaving the audit site.

2410-08 REPORTING OBSERVATION AUDITS

08.03 General. Within 30 days, the HLWB staff shall prepare an Observation Audit Report

(Exhibit 3) after observation audits. HLWB management will be informed of progress and the result

of the observation audit. An “Item of Interest to the Commission” will be issued the week after the

audit.

08.04 Report Format and Contents. Observation audit reports should contain a description of

the observation audit scope; identification of the audited organization; the date and location of the

observation audit; identification of persons performing the observation audit; summary of the

observation audit results; description of each significant deficiency in detail; a statement regarding

the adequacy/effectiveness of the audited organization’s QA program implementation; the

signature of each member of the observation audit; and the approval of the Section Chiefs.
Issue Date: 07/12/00 - 9 - 2410

Observation audit reports will also include an assessment of the DOE audit team’s effectiveness.

Observation audit reports shall contain the sections described below.

08.05 Cover Page. The cover page of the observation audit report shall include the report title,

including the organization (U.S. Nuclear Regulatory Commission) observing the audit and the

observation audit report number, and the organization (Office of Civilian Radioactive Waste

Management Quality Assurance Division) being audited and its audit report number. The cover

page shall also include signature blocks with the signature of the Observation Audit Team Leader,

each observation audit team member, and the approval signatures of the Section Chiefs.

08.06 Introduction. The introduction shall include the date of the audit, the date of the

observation audit, name and location of the audited organization, the objective of the audit, the

objective of the observation audit, and general background.

08.07 Management Summary. This section shall include a summary of the audit effectiveness,

DOE audit team members’ independence and qualifications, overall implementation of the audited

organization’s quality program, and NRC agreement/disagreement with the DOE audit team

findings, recommendations and conclusions.

08.08 Participants. This section shall list all members of the NRC observation team, DOE audit

team, and Affected Local Units of Government participating as observers.

08.09 Review of Audit and Audited Organizations. This section shall include the basis for the

audit and observation audit.

08.10 Scope of the Audit. This section shall include the type (performance- based or

compliance) audit performed, and products and documents reviewed.

08.11 Conduct and Timing of the Audit. This section shall include the NRC observation team’s

conclusion regarding the DOE audit team’s performance and also comments on the timing of the

audit, in order to perform an acceptable audit.

08.12 DOE Audit Team Qualification and Independence. This section shall include a statement

summarizing the acceptability of the DOE ATL and audit team members’ qualifications and

independence, as verified by the NRC observation team leader and in accordance with

requirements of the DOE QARD, current revision.

08.13 Examination of QA Elements. This section shall focus on the QA programmatic elements

observed during the observation audit and supporting evidence.

08.14 Examination of Technical Activities. This section shall focus on the Technical Products

observed during the observation audit and supporting evidence.

08.15 NRC Staff Findings. This section shall include a NRC statement of

agreement/disagreement with the DOE audit team findings, conclusions, and recommendations.

This section shall also include NRC observation team findings, conclusions, and recommendations

and any open carryover items/findings from previous observation audits.

2410-09 DISTRIBUTION

Observation Audit Reports of DOE audits will be transmitted to the DOE, DOE

contractors/suppliers, the State of Nevada, Affected Local Units of Government and other

interested parties as an attachment to a transmittal letter. Exhibit 4 is a sample transmittal letter

for an observation audit of the DOE. The transmittal letter will be signed by the Branch Chief,

HLWB.

2410-10 FOLLOW-UP

NRC AOIs which remain open will be listed and discussed in the observation audit report.

Preliminary CARs, identified by the audit team, that are not resolved during the course of the audit,

should be discussed by the DOE ATL, at the exit meeting, with the organization being audited. The

NRC observers should record and understand these preliminary CARs before the audit is
2410 - 10 - Issue Date: 07/12/00

completed. The preliminary CARs will be noted in the NRC Observation Audit Report and

compared with the final CARs described in the DOE Audit report. Differences between the

preliminary and final CARs may require NRC follow-up action.

The NRC may elect to observe follow-up audits or surveillances that are used to verify that the

audited organization is implementing the necessary corrective action to resolve and prevent the

recurrence of deficiencies. Likewise, follow-up audits by the NRC staff may be necessary to

ensure that audit team recommendations and observed QA program weaknesses are being

addressed acceptably.

2410-11 REFERENCES

10 CFR Part 60 “Disposal of High Level Radioactive Wastes in Geological Repositories.”

10 CFR Part 63, “Disposal of High-Level Radioactive Wastes in a Geologic Repository at Yucca

Mountain, Nevada,” (draft as of March 15, 2000).

“Agreement Between DOE/OCRWM and NRC/NMSS Regarding Prelicensing Interactions

(November 16, 1998)” - included in DOE Memo from Alan B. Brownstein to C. William Reamer

dated March 1999.

DOE QARD, current revision.

END

EXHIBITS

1. Sample - “Item of Interest to the Commission”

2. Sample - “Audit Observer Inquiry Form”

3. Sample - “Observation Audit Report”

4. Sample - “Observation Audit Report Transmittal Letter”