POLICY ISSUE

INFORMATION

March 9, 2001 SECY-01-0038

FOR: The Commissioners

FROM: William D. Travers

Executive Director for Operations

SUBJECT: ANNUAL STATUS REPORT ON THE ADMINISTRATION OF NRC'S

REQUALIFICATION PROGRAM AND THE INITIAL OPERATOR

LICENSING EXAMINATIONS (WITS 8800098)

PURPOSE:

To inform the Commission of the status of the NRC's licensed operator requalification program

and the results of NRC's initial licensing examinations for reactor operator (RO) and senior

reactor operator (SRO) applicants.

BACKGROUND:

The staff has submitted periodic reports since August 1989 on the status of the NRC's licensed

operator requalification program oversight activities. These reports also contained results

of initial RO and SRO licensing examinations. The most recent of these reports was

SECY-00-0059 (dated March 6, 2000).

DISCUSSION:

NRC Requalification Program and Inspection Summary for Fiscal Year 2000

During fiscal year (FY) 2000, the staff continued to monitor facility licensees’ licensed operator

requalification training and examination programs. The staff inspected the licensed operator

requalification programs at 41 power reactor facilities during FY 2000. The inspections were to

(1) confirm that the requalification programs contributed to safe power plant operation by

evaluating how well individual operators and crews mastered training program objectives;

CONTACT: David C. Trimble, NRR/DIPM/IOHB

(301) 415-1031
-2-

(2) determine licensee effectiveness in evaluating and revising requalification programs for

licensed operators based on operational performance, including requalification

examinations; (3) determine licensee effectiveness in ensuring that licensed individuals satisfy

the conditions of their licenses as specified in 10 CFR 55.53; and (4) provide regional

management with the information it needs to assess the performance of licensees’ operator

requalification programs and to determine the need for additional inspections or NRCconducted examinations. The programs were evaluated using the process described in either

NRC Inspection Procedure (IP) 71001, “Licensed Operator Requalification Program

Evaluation,” or its replacement procedure IP 71111.11. IP 71111.11 comports with the revised

reactor oversight program (ROP). Use of IP 71111.11 on a national basis began in April 2000

coincident with implementation of the ROP. Since the last annual status report, the number of

requalification program inspections conducted using the revised inspection guidance of IP

71111.11 increased substantially. IP 71111.11 is currently being revised to incorporate the

newly developed operator requalification significance determination process (OR SDP). By

means of a logic flow chart and matrix, the OR SDP provides guidelines for determining the risk

importance of issues identified during inspections of licensed operator requalification programs.

The staff conducts the requalification program inspection at each facility at least every 24

months which is consistent with each licensee’s requalification examination cycle. The staff

may, as needed, conduct “for cause” requalification examinations when it loses confidence in a

licensee’s ability to conduct its own examinations or when it believes that the inspection process

will not produce the necessary insights into the quality of the licensee’s program. During FY

2000, the staff did not conduct any requalification examinations. Upon incorporation of the OR

SDP into IP71111.11, results of the SDP will be a primary consideration in future "for cause"

decisions.

Inspections of licensed operator requalification programs continue to identify site-specific

strengths and weaknesses. The staff believes that the performance level of licensed operator

requalification training programs, as a whole, at the power reactor facilities inspected is being

sustained. Licensees have continued to demonstrate their ability to develop and administer

licensed operator requalification examinations. Licensee evaluations continued to satisfactorily

identify licensed operator performance deficiencies. Facility licensees constructively use

feedback from training for improving licensed operator requalification training and involve plant

and operations managers in the observation and evaluation of examinations. Resident

inspector quarterly reviews of licensed operator requalification training and examinations did not

reveal any areas of concern that were not being addressed by licensees in their corrective

action programs. Licensees continue to be challenged by the following issues: requalification

written examination quality, use of repetitive test items, and validation of job performance

measures (JPM).

The attachment contains the individual results of the requalification program inspections at

each facility inspected during FY 2000. A SAT (satisfactory) rating for a requalification program

inspection indicates that the licensee’s requalification program exhibited no major program

failures adverse to safety. The following table summarizes the results of requalification

program inspections.
-3-

Requalification Program (Examination and/or Inspection) Results for Fiscal Year 2000

Element

Number

Evaluated SAT/UNSAT

Percent

SAT

NRC Program Examinations (NUREG-1021) None N/A N/A

NRC Program Inspections (IP-71001) 23 23/0 100

NRC Program Inspections (IP-7111.11) 18 18/0 100

Total 41 41/0 100

The next table depicts the continuing success of licensees in the conduct of their requalification

programs as evidenced by no major program failures since FY 1993.

NRC Requalification Program Evaluation Results for Fiscal Years 1993 through 2000

Element

1993 1994 1995 1996 1997 1998 1999 2000

Number of Requalification

Programs Evaluated

43 43 58 41 41 32 40 41

Number of Satisfactory/Number

of Unsatisfactory

43/0 43/0 58/0 41/0 41/0 32/0 40/0 41/0

Percent Satisfactory 100 100 100 100 100 100 100 100

Summary of Initial Examination Results

The staff is continuing to administer initial licensing examinations to applicants for RO and SRO

licenses at power and non-power reactor facilities. The following table gives results of the

power reactor initial operator licensing examinations over a period of 4 years from FY 1997

through FY 2000. During FY 2000, the staff administered approximately 47 site-specific initial

licensing examinations to RO and SRO applicants at power reactor facilities. This number

includes 37 (79% of the total) site-specific licensing examinations that were prepared, in whole

or in part, by facility licensees in accordance with the NRC’s examination guidance. The table

separates NRC-prepared and facility-prepared examination results. In addition, the staff

administered 392 generic fundamentals examinations during FY 2000 to prospective license

applicants at power reactor facilities.
-4-

Power Reactor Initial Examination Results

Examination

Percentage of Applicants Who Passed During the Fiscal Year

1997 1998 1999 2000

NRC

Prepared

Facility

Prepared

NRC

Prepared

Facility

Prepared

NRC

Prepared

Facility

Prepared

NRC

Prepared

Facility

Prepared

RO

Written 96 89 N/A 89 100 89 98 95

Operating 93 94 N/A 99 100 93 100 98

SRO

Written 91 93 100 96 100 94 100 95

Operating 84 92 94 96 100 98 96 99

These results indicate that initial operator training programs at power reactors continue to

produce a large number of applicants who pass the operator licensing examinations,

irrespective of whether the examinations were prepared by the NRC or by the licensees (with

prior NRC review and approval). During FY 2000, approximately 352 applicants were

administered an initial examination. NRC prepared examinations were administered to 29%

(compared to 8% in FY1999) of the applicants and the facility prepared examinations were

administered to 71% (compared to 92% in FY1999) of the applicants. The NRC regional

examiners ensure that initial licensing examinations are consistent with NRC expectations as

provided in NUREG-1021, ˇOperator Licensing Examination Standards for Power Reactors.”

Experience to date with examinations prepared by licensees indicates that, while the quality of

examinations is improving, consistency is a problem due, in part, to turnover of experienced

examination developers. Some facility licensees do not prepare examinations and thus have

little or no experience. In Region II, approximately half of the facility licensees currently prefer

to have the NRC prepare their licensing examinations. This may necessitate the increased

sharing of examiner resources among the regional offices and emphasizes the need for license

examiners to maintain their proficiency. The staff continues to seek ways to improve the initial

licensing examination process and reduce unnecessary burden associated with the

examinations without jeopardizing the integrity of examinations.

The following table indicates the total number of applicants who requested NRC review of their

examination results including the review outcomes from FY 1995 through FY 2000. During

FY 2000, the staff noted a decrease in the percentage of proposed applicant denials being

overturned by the review process. The overturned denials were attributed to new information

provided by the applicant or psychometric test item deficiencies not identified during the

examination review and validation process. The staff believes that regional and national public

workshops over the past two years with examiners and the industry regarding the NUREG-1021

examination development process, coupled with semiannual staff public meetings with the

industry’s focus group on initial operator licensing issues, have brought attention to concerns

with the process and thereby fostered resolution through changes to NUREG-1021 and

generation of responses to frequently asked questions. The staff believes that these efforts as

well as NRC examiner reviews of facility-prepared examinations have contributed to the

decrease in the number of overturned denials. In FY 2001 to date the staff has received only

three applicant-requested reviews.
-5-

Power Reactor Initial Exmination Denial Results

Description 1995 1996 1997 1998 1999 2000

Number of Proposed Applicant Denials 30 47 44 41 60 19

Number of Applicant Requested Reviews 4 12 13 20 16 9

Number of Final Denials 30 44 35 28 47 16*

Number of Licenses Issued Upon Review 0 3 9 13 13 2

* One applicant review is in formal hearing status.

The following table gives the results of the non-power reactor initial operator licensing

examinations over a period of 5 years from FY 1996 through FY 2000. During FY 2000, the

staff administered approximately 20 (compared to 17 in FY 1999) site-specific initial licensing

examinations to RO and SRO applicants at non-power reactor facilities in accordance with the

current examination guidance in NUREG-1478, "Non-Power Reactor Operator Licensing

Examiner Standards."

Non-Power Reactor Initial Examination Results

Examination

Percentage of Applicants Who Passed During the Fiscal Year

1996 1997 1998 1999 2000

RO

Written 74 70 87 63 78

Operating 97 93 100 96 89

SRO

Written 75 100 94 100 82

Operating 96 95 100 100 100

These results indicate that training programs for non-power reactor facility operators generally

produce applicants who pass the NRC's licensing examinations at a lower percentage rate on

the written examination and a higher percentage rate on the operating test. These results are

consistent with those of previous years, except for the reactor operator (RO) applicants’

performance on written examinations administered in FY 1999. The decreased level of

performance exhibited by the reactor operators in FY 1999 is more in line with lower

performance levels of previous years in that the reactor operators historically achieve lower

scores than senior reactor operators. The staff believes that factors such as the under

estimation of the level of commitment and the level of knowledge required by the applicants

(with competing academic issues) to successfully pass the NRC examination may have

contributed to the lower performance.

Operator Licensing Program Initiatives

During FYs 2000 and 2001 to date, the NRC continued its efforts to improve and support the

oversight of the operator licensing program and respond to stakeholder concerns. Examples of

the staff’s initiatives include the following:
-6-

(1) Modified the licensed operator requalification program inspection procedure to comport

with the ROP. The procedure is currently being revised to incorporate the OR SDP to

determine the risk importance of inspection findings. The OR SDP addresses issues that

are identified during a licensed operator requalification program inspection or by a

resident inspector’s observation of requalification activities. The OR SDP assesses the

risk associated with the programmatic aspects of exam quality, security and grading, and

the performance of licensed operators on the biennial written examinations and annual

operating tests. The OR SDP has been discussed during four public meetings with

stakeholders, and has been issued as Appendix I to NRC Inspection Manual Chapter

0609.

(2) Maintained the “Operator Licensing Program,” home page

(www.nrc.gov/NRC/REACTOR/OL/OLhome.html ) on the NRC’s external (public) web

server to provide information concerning the regulations, guidance documents, and

Commission papers associated with the operator licensing program. Additionally it

provides responses to frequently asked questions. Over the past two years this website

has enhanced communication with stakeholders and the public regarding the NRC’s

reactor operator licensing program.

(3) Observed the activities of an INPO Plant Evaluation/Accreditation Team visit at Sequoyah

and observed eight of the eleven monthly National Nuclear Accrediting Board (NNAB)

meetings. The NRC observes accreditation team visits and NNAB meetings, as a means

of monitoring the accreditation process.

(4) Participated in a two-day national workshop sponsored by the Nuclear Energy Institute

(NEI) in February 2000 to focus on the broad-based implementation issues associated

with licensee-prepared license examinations. Also, attended several meetings with

industry training groups, such as the Middle Atlantic Nuclear Training Group (MANTG), to

promote a better understanding of the details of the examination development process

including key changes affecting development of written examinations.

(5) Proceeded with rule-making that would revise the requirements in 10 CFR 55.31(a)(5) and

10 CFR 55.45(b) for control manipulations, licensee certification of simulators, and

simulator testing. The proposed final rule was published in the Federal Register on July 3,

2000, and the public comment period ended on September 18, 2000. The final rule would

allow applicants for operator and senior operator licenses to fulfill a portion of their

required experience prerequisites by manipulating a plant-referenced simulator as an

alternative to use of the actual plant.

(6) Developed and implemented the following measures to reduce unnecessary licensee

burden associated with the initial operator licensing process: (1) added a third generic

fundamentals examination for each fiscal year beginning in October 2000 to provide

enhanced flexibility to licensees in scheduling training programs; (2) extended the public

comment and trial use period on a pilot change to the Examination Standards, NUREG1021, that eliminated burdensome question usage restrictions for facility licensee

developed examinations; and, (3) coordinated with and assisted the Institute of Nuclear
-7-

Power Operations in the development of a national examination question bank by

providing questions from all past and current NRC examinations administered since

October 1995. The national examination bank was made available for licensee and NRC

use in July 2000.

(7) Initiated plans for placing the Generic Fundamentals Examination Section (GFES)

examination bank on the operator licensing home page to include the entire GFE test

bank consisting of validated items and past examinations (effective 2001 and beyond).

Users of the website will be able to view and download publicly available GFE test files for

training and testing purposes.

(8) Continued a program to independently audit and assess the consistency and level of

difficulty of licensing examinations. Also, revised the examination standards to require

regional management to discuss the quality of proposed examination products with

licensee management prior to administration.

CONCLUSION:

The NRC’s licensed operator requalification inspection program has provided an effective

means for monitoring the quality of facility licensed operator requalification training and

examinations. The program indicates that the quality of these training programs at the facilities

inspected is being maintained at a sufficient level to assure operator competence in safely

performing licensed duties. Additionally, over the past year, significant steps have been taken

to reduce unnecessary regulatory burden associated with the initial operator licensing

examination process while maintaining the integrity of that process.

/RA by Carl J. Paperiello Acting For/

William D. Travers

Executive Director

for Operations

Attachment: Status Report on the NRC Requalification

Program - Fiscal Year 2000
-7-

Power Operations in the development of a national examination question bank by

providing questions from all past and current NRC examinations administered since

October 1995. The national examination bank was made available for licensee and NRC

use in July 2000.

(7) Initiated plans for placing the Generic Fundamentals Examination Section (GFES)

examination bank on the operator licensing home page to include the entire GFE test

bank consisting of validated items and past examinations (effective 2001 and beyond).

Users of the website will be able to view and download publicly available GFE test files for

training and testing purposes.

(8) Continued a program to independently audit and assess the consistency and level of

difficulty of licensing examinations. Also, revised the examination standards to require

regional management to discuss the quality of proposed examination products with

licensee management prior to administration.

CONCLUSION:

The NRC’s licensed operator requalification inspection program has provided an effective

means for monitoring the quality of facility licensed operator requalification training and

examinations. The program indicates that the quality of these training programs at the facilities

inspected is being maintained at a sufficient level to assure operator competence in safely

performing licensed duties. Additionally, over the past year, significant steps have been taken

to reduce unnecessary regulatory burden associated with the initial operator licensing

examination process while maintaining the integrity of that process.

/RA by Carl J. Paperiello Acting For/

William D. Travers

Executive Director

for Operations

Attachment: Status Report on the NRC Requalification

Program - Fiscal Year 2000

ACCESSION #ML010540333

*See Previouse Consurrence

To receive a copy of this document, indicate in the box: "C" = Copy without enclosures "E" = Copy with enclosures "N" = No copy

OFFICE IOLB/DIPM IOLB/DIPM IOLB/DIPM TECH ED DIR/DIPM

NAME LVick:lv/rc DTrimble GTracy PKleene BBoger

DATE 02/07/2001 * 02/23/2001 * 02/23/2001 * 02/15 /2001* 02/ 23/2001 *

OFFICE PMAS/NRR ADIP/NRR NRR EDO

NAME JSilber JJohnson SCollins RPZ for WTravers CJP for

DATE 02/ /2001 02/28/2001 03/02/2001 03/09/2001

OFFICIAL RECORD COPY
*See footnote at end of table. ATTACHMENT

STATUS REPORT ON THE NRC REQUALIFICATION PROGRAM

FISCAL YEAR 2000

Facility Program

Evaluated Inspection Procedure Performed SAT*/UNSAT Date

--------------- --------------------------------------------------- ------------------ -------

Ginna IP-71001 - Requal Program Inspection SAT 10-99

Pilgrim IP-71001 - Requal Program Inspection SAT 10-99

Limerick IP-71001 - Requal Program Inspection SAT 10-99

Calvert Cliffs IP-71001 - Requal Program Inspection SAT 10-99

Nine Mile Point IP-71001 - Requal Program Inspection SAT 10-99

Lasalle IP-71001 - Requal Program Inspection SAT 10-99

Braidwood IP-71001 - Requal Program Inspection SAT 10-00

Quad Cities IP-71111.11 - Requal Program Inspection SAT 10-00

Wolf Creek IP-71001 - Requal Program Inspection SAT 10-99

Seabrook IP-71001 - Requal Program Inspection SAT 11-99

Beaver Valley IP-71001 - Requal Program Inspection SAT 11-99

Crystal River IP-71001 - Requal Program Inspection SAT 11-99

Sequoyah IP-71111.11 - Requal Program Inspection SAT 11-99

Clinton IP-71001 - Requal Program Inspection SAT 11-99

Prairie Island IP-71111.11 - Requal Program Inspection SAT 11-99

Fermi IP-71001 - Requal Program Inspection SAT 11-99

Browns Ferry IP-71001 - Requal Program Inspection SAT 12-99

St. Lucie IP-71001 - Requal Program Inspection SAT 12-99

San Onofre IP-71001 - Requal Program Inspection SAT 12-99

Three Mile Island IP-71001 - Requal Program Inspection SAT 1-00

Turkey Point IP-71001 - Requal Program Inspection SAT 1-00

Quad Cities IP-71111.11 - Requal Program Inspection SAT 1-00
Susquehanna IP-71001 - Requal Program Inspection SAT 2-00

North Anna IP-71001 - Requal Program Inspection SAT 2-00

Cook IP-71001 - Requal Program Inspection SAT 2-00

Indian Point 3 IP-71001 - Requal Program Inspection SAT 3-00

Peach Bottom IP-71001 - Requal Program Inspection SAT 3-00

Fitzpatrick IP-71111.11 - Requal Program Inspection SAT 4-00

Kewaunee IP-71111.11 - Requal Program Inspection SAT 4-00

Limerick IP-71111.11 - Requal Program Inspection SAT 5-00

Peach Bottom IP-71111.11 - Requal Program Inspection SAT 5-00

Palisades IP-71111.11 - Requal Program Inspection SAT 5-00

McGuire IP-71111.11 - Requal Program Inspection SAT 6-00

Watts Bar IP-71111.11 - Requal Program Inspection SAT 7-00

Hatch IP-71111.11 - Requal Program Inspection SAT 7-00

Catawba IP-71111.11 - Requal Program Inspection SAT 8-00

Farley IP-71111.11 - Requal Program Inspection SAT 8-00

Byron IP-71111.11 - Requal Program Inspection SAT 8-00

Kawaunee IP-71111.11 - Requal Program Inspection SAT 8-00

Palo Verde IP-71111.11 - Requal Program Inspection SAT 8-00

Vermont Yankee IP-71111.11 - Requal Program Inspection SAT 9-00

* A program rating of SAT (satisfactory) indicates that the licensee’s requalification program

complied with the requirements of 10 CFR 55.53 and 55.59 for the areas inspected and that the

NRC staff did not elect to conduct NRC-administered requalification examinations for cause as a

result of any weaknesses that may have been noted.

2