September 20, 2001

MEMORANDUM TO: Cynthia A. Carpenter, Chief

Risk Informed Initiatives, Environmental, Decommissioning,

 and Rulemaking Branch

Division of Regulatory Improvement Programs, NRR

FROM: Peter C. Wen, Project Manager/RA/

Risk Informed Initiatives, Environmental, Decommissioning,

 and Rulemaking Branch

Division of Regulatory Improvement Programs, NRR

SUBJECT: SUMMARY OF AUGUST 16, 2001, MEETING WITH THE NUCLEAR

ENERGY INSTITUTE REGARDING OPERATOR LICENSING ISSUES

On August 16, 2001, the NRC staff participated in a public meeting with the Nuclear Energy

Institute (NEI) in their offices at 1776 I Street (NW), Washington, DC, to discuss issues related

to the implementation of Revision 8 of NUREG-1021, “Operator Licensing Examination

Standards for Power Reactors.” Attachment 1 lists attendees at the meeting.

This was the latest in a series of public “focus group” meetings intended to promote the

efficient, effective, and consistent preparation and administration of initial operator licensing

examinations now that facility licensees are preparing approximately 75 percent of those

examinations for NRC review and approval in accordance with 10 CFR 55.40. The meeting

focused primarily on the status of outstanding issues that had been raised during prior

meetings, the last of which was held on February 9, 2001. (Refer to ADAMS Accession

Number ML010720244 for a summary of that meeting). The issues discussed during the

meeting are summarized in Attachment 2, and the handouts that were distributed are provided

as Attachments 3 through 6.

Representatives of the NRC and the industry agreed that this meeting had been useful for the

exchange of information on this subject.

Attachments: As stated

cc w/atts: See next page
September 20, 2001

MEMORANDUM TO: Cynthia A. Carpenter, Chief

Risk Informed Initiatives, Environmental, Decommissioning,

 and Rulemaking Branch

Division of Regulatory Improvement Programs, NRR

FROM: Peter C. Wen, Project Manager/RA/

Risk Informed Initiatives, Environmental, Decommissioning,

 and Rulemaking Branch

Division of Regulatory Improvement Programs, NRR

SUBJECT: SUMMARY OF PUBLIC MEETING WITH NEI REGARDING

OPERATOR LICENSING ISSUES

On August 16, 2001, the NRC staff participated in a public meeting with the Nuclear Energy

Institute (NEI) in their offices at 1776 I Street (NW), Washington, DC, to discuss issues related

to the implementation of Revision 8 of NUREG-1021, “Operator Licensing Examination

Standards for Power Reactors.” Attachment 1 lists attendees at the meeting.

This was the latest in a series of public “focus group” meetings intended to promote the

efficient, effective, and consistent preparation and administration of initial operator licensing

examinations now that facility licensees are preparing approximately 75 percent of those

examinations for NRC review and approval in accordance with 10 CFR 55.40. The meeting

focused primarily on the status of outstanding issues that had been raised during prior

meetings, the last of which was held on February 9, 2001. (Refer to ADAMS Accession

Number ML010720244 for a summary of that meeting). The issues discussed during the

meeting are summarized in Attachment 2, and the handouts that were distributed are provided

as Attachments 3 through 6.

Attachments: As stated

cc w/atts: See next page

DISTRIBUTION: See attached page

Document Name: g:\rgeb\pcw\msum0816.wpd

*See Previous Concurrence

OFFICE RGEB SC:IOHS/DIPM BC:IQPB/DIPM SC:RGEB

NAME PWen* DTrimble* TQuay SWest

DATE 09/13/01 09/18/01 09/20/01 09/20/01

OFFICIAL RECORD COPY
List of Attendees - NRC / NEI Meeting August 16, 2001

Name Organization

Richard Conte NRC/RI

Guy Bruner INPO

Jeff Hansen Exelon

Dave Trimble NRC/HQ

Ted Quay NRC/HQ

Bill Dean NRC/HQ

George Usova NRC/HQ

Mike Ernstes NRC/RII

Fred Guenther NRC/HQ

John Munro NRC/HQ

Tony Gody NRC/RIV

David Hills NRC/RIII

Fred Riedel APS.

Hironori Peterson NRC/RIII

Gregg Ludlam CP&L

Jim Davis NEI

Peter Presby Comanche Peak

Ivan Kingsley Sonalysts

Charles Sawyer Duke Energy

Don Jackson PSEG Nuclear

Dale Powers NRC/RIV

Kerry Wright NAESCo

Brian Haagensen PSHA, Inc.

Mike DeFrees STPNOC

George Thullen NMC/Duane Arnold

Chuck Sizemore NMC PBNP/KNPP

Robert Evans NEI

ATTACHMENT 1
ATTACHMENT 2

Operator Licensing Meeting With NEI on August 16, 2001

Agenda Item Discussion Summary

1. NUREG-1021

Clarifications /

Feedback

- The NRC staff explained the additional examination development guidance

that was posted on the operator licensing web site in June 2001. The

guidance clarifies the staff’s expectations regarding the elimination of

randomly selected knowledge and ability (K/A) statements that the

examination author believes to be inappropriate for testing at the facility. The

industry representatives had no specific comments regarding the new

guidance, but they concurred that it should ease the burden of having to

document rejected K/A statements.

- The industry representatives raised a new issue regarding the distribution of

K/As on the written examination. They questioned the extent to which the

importance and number of K/As related to a particular system or evolution

were factored into the design of the standardized written examination sample

plans in Section ES-401 of NUREG-1021, “Operator Licensing Examination

Standards for Power Reactors.” They stated their view that the current

sampling process often over-emphasizes certain topics (e.g., radiological

waste and radiation monitoring) and that the author then has to explain any

modifications to the NRC. The industry representatives agreed to develop a

proposal for a “stratified random sampling process” that could reduce the

necessity for manual intervention.

- The industry representatives questioned whether the NRC was planning to

clarify its expectations regarding peer-checking during the operating test. The

NRC staff briefly summarized its current thinking on this issue (essentially that

an applicant would be held accountable for an imminent error even if it was

detected and prevented by a peer-checker) and indicated that the final policy

would be posted on the guidance page of the operator licensing web site as

soon as possible.

2. Industry

Experience

Since the Last

Meeting

- The NRC staff distributed and briefly discussed a graph (Attachment 3) that

summarizes reactor operator performance on the written licensing examination

from 1994 through 2000. The staff noted that since approximately 85% of the

examinations have no failures, it generally looks pretty closely and objectively

at those exams that do have a high failure rate in an effort to determine if the

examination was at fault. The staff acknowledged that some utilities have

done very good root cause analyses that have often found multiple

contributors to the applicants’ poor performance. In some cases it appears

that the applicants were simply not prepared for the higher cognitive level

(application) questions that make up 50-60% of the examination, and, for

various reasons, facility licensees sometimes allow applicants to take the NRC

licensing examination despite their marginal or unsatisfactory performance on

the facility licensee’s audit exam.

- The industry representatives acknowledged that a high failure rate on the

NRC licensing examination is generally not indicative of a problem with the

examination. They noted that it is getting harder to find highly qualified license

candidates and that, although facility licensees are pretty successful overall at

screening candidates, sometimes their decision-making regarding candidate

screening could be improved.

- The industry representatives’ comments regarding the written examination

were generally positive.
Agenda Item Discussion Summary

3. Proposed

Long-Term

Examination

Options

- The NRC staff indicated that is continuing to evaluate a number of

enhancements for the initial written examination and operating test, with goals

of maintaining exam validity and fairness while reducing unnecessary

regulatory burden. The staff noted that it wanted to share its preliminary

thoughts and gauge the industry’s interest but cautioned that NRC

management has NOT approved any of the possible changes discussed

during the meeting. 

- With regard to the initial written examination: (1) The NRC staff reported that

informal feedback on the possibility of developing a common examination for

operator (RO) and senior operator (SRO) applicants, which was mentioned

during the previous public meeting in February 2001, has not been positive. 

The industry representatives confirmed that such a change would likely be

unacceptable to facilities with operator bargaining units, but they noted that it

might be worth implementing on a voluntary basis. (2) The NRC staff

indicated that it is considering the possibility of decreasing the length of the

RO exam from 100 to 75 questions, which will save resources for both the

NRC and the industry without sacrificing validity or fairness. SRO applicants

would take the RO exam plus a separate 25 question exam focused on the

additional topics required by 10 CFR 55.43. Details regarding the SRO exam

sampling guidelines, bank use, grading, waivers, and retakes have yet to be

determined. The industry representatives expressed tentative interest in the

concept and agreed to discuss and consider it further among themselves

before the next focus group meeting. (3) An industry representative inquired

whether Option 3 (an earlier industry proposal under which utilities would

prepare and administer the exams without prior NRC review) is dead. The

NRC staff responded that it is concentrating on refining the current

examination process as recommended in NEI’s letter dated June 6, 2001

(Accession No. ML 011720017), and that the industry would have to submit a

petition for rulemaking if it is serious about pursuing that option.

- With regard to the operating test: (1) The industry representatives reiterated

their concerns regarding the administrative category of the walk-through (e.g.,

its perceived artificiality and the difficulty of preparing a valid test given

changes in the operators’ job function) and recommended that it be rolled into

the written examination, subsumed in the systems category of the walkthrough, or eliminated altogether. The NRC staff responded that the first and

third options would not be possible given the requirements of 10 CFR 55.45,

but acknowledged that it is giving serious consideration to the possibility of

combining the two walk-through categories. The staff noted that details

regarding the number (10 - 15), distribution (admin vs systems for RO and

SRO applicants), and grading of test items in a combined walk-through have

yet to be resolved. (2) In connection with the industry’s desire to combine the

administrative and systems categories of the walk-through, the NRC staff

indicated that it needs to preserve the overall discriminatory validity of the

operating test and that it plans to do more work on an integrated proposal that

may include enhancements to the simulator operating test (e.g., reassessing

some of the rating factors) as well as the walk-through. (3) Pending resolution

of these issues, the NRC staff sought the industry representatives’ feedback

regarding the following administrative category enhancements that might be

possible in the shorter term: increasing flexibility by eliminating the

requirement to test every applicant on all four administrative topics; improving

reliability by using five job performance measures (JPMs), with no prescripted

questions, to test the administrative topics; and instituting more objective

grading criteria with a straight 80% cut score for the five administrative JPMs. 

The industry representatives agreed to consider the staff’s proposal before the

next focus group meeting. 
Agenda Item Discussion Summary

4. Generic

Fundamentals

Examination

(GFE)

- The NRC staff opened the discussion by summarizing the GFE results over

the last ten years (23 examinations) and noting that the stability of those

results are indicative that the level of difficulty of the examinations has not

increased. The staff acknowledged that the number of higher cognitive level

questions has probably increased over time, but noted that access to the

examination question bank (which is now available on the NRC’s web site)

offers a powerful advantage and that the average grades may actually improve

as a result. The staff further noted that the scores are highly predictable and

that well-trained applicants should have no difficulty passing the exam in light

of the 80/10/10 distribution of bank, modified, and new questions. Copies of

the associated slides and handouts are provided as Attachments 4 and 5.

- The NRC contractor who has prepared the GFEs since the inception of the

program presented an overview of the exam development and validation

process in an effort to address the industry’s concern that an increasing

number of questions require the GFE candidates to have more than a basic

understanding of plant system design and operation. Copies of the associated

slides and handouts are provided as Attachment 6. The NRC staff noted that

the industry’s training standards for non-licensed operators (the primary

source of GFE candidates) require a level of system knowledge that should be

more than adequate for success on the GFE.

- The industry representatives continued to voice their opinion that the GFEs

have become more difficult over time, thereby causing facility licensees to

spend more resources to sustain the level of performance on the exams. 

They suggested that the stable results are misleading because they fail to

account for the fact that facility licensees screen out those candidates who are

unlikely to do well on the examination. However, the industry spokesperson

did not provide any factual data to support their views. The industry

representatives also stated their belief that the GFE is a “moving target”

because the NRC is writing many of the new questions at cognitive levels that

exceed the literal wording of the K/A statements around which facility

licensees have designed their GFE training programs. They indicated that

they would like the NRC to focus more on the knowledge level of the exam

rather than its outcomes, to apply some of the initial exam review criteria (e.g.,

Form ES-409-9) to the GFE, and to give facility licensees the opportunity to

review the exams before they are given. The NRC acknowledged the

industry’s concerns and agreed to continue evaluating this issue and to

provide additional feedback during the next focus group meeting.

5. Reactivity

Manipulation

Rule Change

Update

- The NRC staff indicated that the final rulemaking is before the Commission

and that it has no reason to believe that it will not be approved.

- The NRC staff also noted that, as authorized by the Commission, it has

informed a number of utilities in writing that exemptions would be issued when

the license applications are received. Those exemptions would authorize

applicants to perform the 5 reactivity manipulations required by 10 CFR

55.31(a)(5) on a simulator. However, to date, no actual exemptions have

been requested.

6. National

Examination

Bank Status

- The representative from the Institute of Nuclear Power Operations (INPO)

reported that there are currently about 18,000 questions in the bank. When he

noted that the questions are not always linked to a K/A statement, the NRC

staff indicated that it would continue its efforts to ensure that all the questions

that are made available to INPO include a K/A reference.

- The NRC staff reported that it is trying to resolve some software issues that

have limited the bank’s usefulness to NRC examiners. When the NRC staff

enquired whether facility licensees are finding the bank useful, one facility

representative indicated that they have successfully used the bank as a source

of ideas for developing new site-specific questions. 
GFE DATA

Data from June 1992 - June 2001 

• 23 exam administrations 

• 4000+ applicants

BWR PWR

Overall Mean Score 89.4 90.2

Bank Item Score 93.4 94.4

Modified Item Score 87.0 87.9

New Item Score 76.8 75.8

ATTACHMENT 5
GFE SCORE PERFORMANCE IS HIGH

Data show that well-trained applicants have little or no

difficulty passing the overall exam. 

Some facts:

In the two most recent exams, 

6/01: 47% of BWRs scored 95 or higher 

29% of PWRs scored 95 or higher.

2/01: 33% of BWRs scored 95 or higher 

37% of PWRs scored 95 or higher.
80-10-10 APPROACH

Other consideration regarding difficulty:

• 80% of all items are derived from the published

bank, resulting in high applicant predictability of

examination coverage. 

• Applicants have 100% predictability that 80% 

(80 items) on any given exam will appear verbatim

from the bank.

• Bank study can be good and can result in two

ancillary benefits:

Broader and deeper learning of the content body

of knowledge and 

Reduced applicant stress in exam study and

preparation. 
DATA AND PREDICTABILITY OF SCORES

Data show that the average applicants have little or no

difficulty passing the GFE . 

The average mean bank score is 93%. 

• Thus, the average applicant has a predictable

score of 74 (80 x .93 = 74) going into the exam. 

Of the remaining 20 items, 10 are modified items

drawn from the bank. 

The average mean modified items score is 87%. 

• Thus, the average applicant has an 8 additional

items 

(10 x 8.7 = 8.7) going into the exam. 

Conclusion:

The average applicant has a score of 82% (74+8 = 82) 

going into the exam. 
Nuclear Energy Institute Project No. 689

cc: Mr. Ralph Beedle

Senior Vice President

 and Chief Nuclear Officer

Nuclear Energy Institute

Suite 400

1776 I Street, NW

Washington, DC 20006-3708

Mr. Jim Davis, Director

Operations

Nuclear Energy Institute

Suite 400

1776 I Street, NW

Washington, DC 20006-3708

Mr. Robert Evans

Nuclear Energy Institute

Suite 400

1776 I Street, NW

Washington, DC 20006-3708
Distribution: Mtg. Notice w/ NEI Re Operator Licensing Issues Dated

ADAMS-PUBLIC

IOHB R/F

RGEB R/F

OGC

ACRS

PWen

S Guenther

Email

S. Collins/J. Johnson

W. Borchardt

B. Sheron

D. Matthews/F. Gillespie

C. Carpenter/S. West

B. Boger/W. Dean

T. Quay/D.Trimble

J. Shea, OEDO

R. Conte, RI

M. Ernstes, RII

D. Hills, RIII

D. Powers, RIV

T. Gody, RIV