**NRC INSPECTION MANUAL**IOLB

	INSPECTION PROCEDURE 71111 ATTACHMENT 11

# LICENSED OPERATOR REQUALIFICATION PROGRAM ANDLICENSED OPERATOR PERFORMANCE

# Effective Date:  January 1, 2022

PROGRAM APPLICABILITY:  IMC 2515 Appendix A

CORNERSTONES:	Initiating Events (10 percent) Mitigating Systems (70 percent)Barrier Integrity (10 percent)Emergency Preparedness (10 percent)

INSPECTION BASES:	See IMC 0308 Attachment 2 

# SAMPLE REQUIREMENTS:

*It is preferred that observation samples are performed in each quarter. 

**For Vogtle and sites designated as a Unique Site Budget Model (USBM) as defined in IMC 2515 and 0306, the inspectors shall perform 8 licensed operator control room observations per site annually and 8 licensed operator training/examination observations per site annually, with the samples distributed among the units at the site.  The annual budgeted range per observation section is 8 samples with 32 budgeted hours per site.  A note can be added to RPS regarding the minimum samples for these sites, with justification that the sample size complies with this footnote. 

***For sites that have more than one licensed operator requalification training program, the examination results of each licensed operator requalification training exam shall be collected annually and each requalification program will be inspected biennially.  A note can be added to RPS regarding the minimum samples for these sites, with justification that the sample size complies with this footnote.   

# 71111.11-01	INSPECTION OBJECTIVES

01.01	To evaluate licensed operator[footnoteRef:2] performance during the conduct of requalification examinations, other examinations, training exercises, and in the actual plant/main control room. [2:  Throughout this inspection procedure, “licensed operator” is used to collectively refer to both licensed reactor operators and licensed senior reactor operators.] 

01.02	To assess the facility licensee’s ability to evaluate the performance of their licensed operators during the conduct of requalification examinations, other examinations, and training exercises.

01.03	To assess the facility licensee’s ability to properly develop and administer requalification annual operating tests and biennial written examinations.

01.04 	To evaluate the performance of the control room simulator and the facility licensee’s testing and maintenance of the control room simulator.

01.05	To ensure that individuals who are licensed to operate the facility satisfy the conditions of their licenses as specified in 10 CFR 55.53 and 10 CFR 55.59, and to assess the facility licensee’s effectiveness in ensuring that operator license conditions are satisfied.

# 71111.11-02		GENERAL GUIDANCE

Quarterly Control Room and Training Observations

Observe and assess licensed operator performance in the actual plant/main control room during periods of heightened activity or risk.  Also, observe the facility licensee ability to administer, and licensed operator performance during, requalification training.  At the discretion of Regional Management, a regional operator licensing examiner may perform this observation. 

For each sample, conduct a routine review of problem identification and resolution activities using IP 71152, “Problem Identification and Resolution.”

Although the intent is to perform samples (control room observations and requalification training observations) each calendar quarter, observations may be deferred based upon the plant’s schedule and inspector judgment.  For example, if a plant does not perform requalification training in a given quarter or has few risk-significant control room activities in a quarter, it may be advantageous to defer the samples.

Annual Requalification Examination Results

In accordance with 10 CFR 55.59(a), each licensed operator must pass a comprehensive (biennial) written examination and an annual operating test[footnoteRef:3].  Following the completion of each of these examinations, a Regional operator licensing examiner familiar with the facility licensee’s requalification examination schedule collects the examination results.  This review is expected to be conducted in-office. [3:  The periodicity of these examinations has been interpreted by the NRC as: annual operating tests are to be administered at least once every calendar year, and the comprehensive written examination is to be administered at least once within each 24-month requalification program preferably near the end of each program. ] 

Biennial Licensed Operator Requalification Program Inspection

Conduct a biennial review of:

1. licensed operator performance during requalification examinations, 

2. the ability of the facility licensee to properly develop and administer requalification examinations, 

3. the maintenance of individual operator licenses, 

4. the performance of the control room simulator, and 

5. the ability of the facility licensee to identify and resolve problems related to licensed operator performance.

The inspection team will consist of (1) a team leader who is a qualified operator licensing examiner on the facility licensee’s vendor type, who, at a minimum, is also qualified as basic certified inspector; and (2) one or more additional examiners/inspectors.[footnoteRef:4]  The biennial review is expected to require a one-week on-site visit, plus additional in-office review.  Although most of these activities could be conducted by a single lead inspector during the facility licensee’s administration of the requalification annual operating test, utilize a minimum of two examiners/inspectors.  As an efficiency measure, it is recommended that the NRC request that the facility licensee submit specific examinations and other information prior to the on-site portion of the biennial inspection, such that any issues identified can be discussed with the facility licensee while on-site.  In accordance with 10 CFR 55.59(c), facility licensees are required to make these examination records available for NRC review, and a list of the typical documents reviewed during the biennial inspection is presented in Appendix A. [4:  It is the expectation that the inspection team leader be an examiner who is a fully qualified inspector. In rare cases the inspection team leader may be an examiner qualified as a basic certified inspector, as approved by the Region’s Operations Branch Chief. For the second member of the team, it is preferred that this individual be an examiner qualified on the facility’s vendor type, with or without inspector qualifications. Additional examiners/inspectors may be used at the Region’s discretion.] 

The biennial inspection shall be scheduled and announced to each facility licensee.  Typically, facility licensees divide up each 24-month requalification program into training cycles.  During each requalification training cycle, training and/or examination activities are repeated on subsequent weeks for different groups or operating crews of licensed operators. Typically, facility licensees conduct an annual operating test for licensed operators during a training cycle near the mid-point of each 24-month requalification program.  Similarly, near the end of each 24-month requalification program, the facility licensees conduct both an annual operating test and a biennial written examination.  With such a schedule, it is preferred that the biennial inspection be performed near the end of each 24-month cycle when the facility licensee is administering both the annual requalification operating test and the requalification written examination.  For facility licensees that do not employ this typical schedule, or for facilities at which the NRC has historically completed the biennial inspection in a different way, the overall requirements are for the region to complete the biennial inspection requirements such that (1) an in-progress annual operating test is observed and reviewed, and (2) the most recently administered biennial requalification written examination is reviewed, such that at least one version of the biennial requalification written examination is reviewed from each of the licensee’s successive 24-month requalification programs during each biennial inspection. 

Generally, only the inspection requirements of this procedure will need to be conducted. However, regional managers will consider overall facility performance, allegations related to licensed operator requalification, findings from this inspection procedure (IP), and any traditional enforcement actions taken as a result of this IP in determining whether additional activities will be performed.  Additional activities include observation of additional groups or crews of licensed operators during an annual requalification operating test, the performance of IP 41500, “Training and Qualification Effectiveness,” and the performance of an NRC-conducted licensed operator requalification examination, in accordance with 10 CFR 55.59(a)(2)(iii) and NUREG-1021, “Operator Licensing Examination Standards for Power Reactors,” ES-6 sections.  Additional activities should be considered when any of the following conditions exist:

Observe Additional Crews.  At the discretion of the NRC Region, perform this activity if the inspector observes a dynamic simulator scenario crew failure during the regularly scheduled inspection, or as a possible response to allegations associated with licensed operator performance or requalification training.

Perform IP 41500. This activity should be performed if the NRC is concerned with the quality of licensed operator requalification training at a facility.  Initiators for this activity include:  (1) significant in-plant operator performance issues that have requalification training quality as a root cause, (2) a failure rate on a requalification examination of greater than or equal to 50 percent, (3) indications of a breakdown in the systems approach to training at the facility, or (4) as a possible response to allegations associated with licensed operator performance or requalification. Prior to initiating this activity, consult with the operator licensing program office and obtain concurrence from the affected Region’s Regional Administrator.

Perform an NRC-conducted licensed operator requalification examination. This activity should be performed if the NRC has lost confidence in the facility licensee’s ability to conduct its own examinations.  Initiators for this activity include: (1) white findings in both written examination quality and operating test quality; (2) in response to an actual examination compromise as defined in Manual Chapter 0609, Appendix I, “Licensed Operator Requalification Significance Determination Process;” or (3) as a possible response to allegations associated with licensed operator requalification examinations. Prior to initiating this activity, consult with the operator licensing program office and obtain concurrence from the affected Region’s Regional Administrator.

# 71111.11-03	INSPECTION REQUIREMENTS

Per IMC 0040, inspection requirements are in **bold** and guidance is in normal type.

## 03.01	Licensed Operator Performance in the Actual Plant/Main Control Room

Observe licensed operator performance in the actual plant/main control room during periods of heightened activity or risk.

Specific Guidance****

1. Prior to observing licensed operators, review the facility licensee’s procedures, expectations, and policies regarding licensed operator performance, including:

· Operator compliance and use of plant procedures, including procedure entry and exit, performing procedure steps in the proper sequence, procedure placekeeping, and technical specification entry and exit.

· Control board/in-plant component manipulations

· Communications between crew members

· Use and interpretation of plant instruments, indications, and alarms; diagnosis of plant conditions based on instruments, indications, and alarms

· Use of human error prevention techniques, such as pre-job briefs and peer checking

· Documentation of activities, including initials and sign-offs in procedures, control room logs, technical specification entry and exit, entry into and out of service logs 

· Management and supervision of activities, including risk management and reactivity management

· Pre-job briefs

The licensee’s policies for these areas are typically contained in various operations’ administrative procedures, with titles such as “Conduct of Operations for Shift Personnel,” “Reactivity Management,” “Control Room Conduct and Control Room Shift Activities,” “Risk Management,” etc.; consult with the facility licensee to determine which procedures are applicable at a particular plant.

2. Select an activity (or activities) to observe and review the plant procedures that will be used by the operators during the observed activity.  

Assess licensed operator performance during periods of heightened plant activity or plant risk where the activities could impact plant safety.  Select actual plant/main control room activities to observe by reviewing plant activity and/or work schedules, attending daily briefs, shift turnover briefs, refueling outage planning meetings, maintenance planning meetings, and via discussions with the facility licensee. In particular, consider observing the following activities:

· Plant startups, shutdowns, and mode changes

· Reactor power and turbine load changes, especially when licensee reactivity management policies will be in effect

· Infrequent plant evolutions 

· Unplanned transients and off-normal events, including post-scram response

· Surveillance testing

· Post-maintenance testing of safety-related structures, systems, and components

· Pre-startup equipment line-ups, operational checks, and functional checks

· Changes to the line-ups or modes of operation of safety related systems, structures, and components

· Refueling outage preparations, such as filling the reactor cavity or entering midloop operations

· Reactor refueling activities

3. Observe licensed operators conducting the selected activity, using the generic checklists contained in Appendix H as guidance.  Consider modifying these checklists based on the licensee’s policies. 

For efficiency, perform plant status reviews (IMC 2515 Appendix D), inspection of post maintenance testing (IP 71111.19), refueling and outage activities (IP 71111.20), and surveillance testing (IP 71111.22) can be performed in parallel with this IP.  The activities to be observed involve licensed operators, typically observed from the main control room.  Observe any pre-job briefs held prior to the activity.

During the observation period, refrain from interfering with the performance of the licensed operators being observed unless interference is warranted due to a significant safety concern.  Limit questions and discussions during plant activities to prevent unnecessary distractions to the licensed operators. 

## 03.02	Licensed Operator Requalification Training/Examinations

Observe licensed operator performance during requalification training/examinations, and the ability of the facility licensee to administer requalification training/examinations.

Specific Guidance****

1. Review the facility licensee’s training procedures that govern the training/examination activity to be observed.  

Facility licensees have training procedures that cover topics such as: the administration of annual operating tests, evaluated scenarios, and how training is to be delivered.  In addition, facility licensees may have training observation forms that would be useful as a guide when observing training activities.

2. Review facility licensee’s expectations and policies regarding licensed operator performance.  See Section 03.01.1.

3. Select a licensed operator requalification training/examination activity to observe.  

If available, observe an annual requalification operating test required by 10 CFR 55.59 that is administered to an operating crew.  Scheduling this activity may be determined from the facility licensee and/or the region’s Operations Branch.  

If an annual requalification operating test required by 10 CFR 55.59 will not be observed, consider the following when selecting an activity:

a) Although not required by 10 CFR 55.59, many facility licensees also routinely evaluate their licensed operators using crew-based simulator scenarios.  If so, observe evaluated simulator scenario(s) administered to an operating crew.

b) If the facility licensee does not conduct evaluated scenarios, then observe simulator training for an operating crew.  

c) Other requalification training activities may also be observed (e.g. classroom training and in-plant training.)

If training is to be observed, give priority to training conducted in response to licensee-identified corrective actions, plant modifications, or upcoming plant evolutions.

4. Observe the selected training/examination activity.  

If a portion of an annual requalification operating test required by 10 CFR 55.59 is observed, the basic methodology for performing this inspection activity is presented in Section 03.04.c of this IP, with a focus upon completing the applicable elements of Appendix D of this IP.  Based upon the limited observation time of the resident staff and their variable experience with operator licensing, it is not expected that the resident staff will conduct in-depth reviews of operating test quality (Section 03.04.b) or examination security (Section 03.04.d).  During this inspection activity and upon its conclusion, any annual requalification operating test issue should be discussed with the facility licensee, to assist in confirming the issue.  However, the inspector should not interfere with the facility licensee’s requalification examination process.  If there are any significant concerns with the annual requalification operating test, contact regional management.  In evaluating any annual requalification operating test issues for operating tests required by 10 CFR 55.59, refer to the applicable appendix of this IP.

If observing licensed operator evaluations which are not a part of an annual requalification operating test required by 10 CFR 55.59, the inspector should use a similar methodology.  Without interfering with the facility licensee’s evaluation process, assess:

· Licensed operator performance

· The ability of the facility licensee to administer the evaluations

· The quality of any post-scenario critiques 

· Follow-up actions taken by the facility licensee for any licensed operator who failed an evaluation (e.g., removal from shift duties, remediation, re-examination)

Note that 10 CFR 55.59 requires only an annual requalification operating test.  Facility licensees also evaluate licensed operators in accordance with their accredited systems approach to training program.  The quality of these additional evaluations and how they are administered are determined by facility licensee.  Unless contained in facility licensee procedures, the details contained in Appendices B, C, D, E, and F of this IP are not applicable to additional licensed operator evaluations.  

If observing licensed operator requalification training, assess the quality of the training.  Detailed guidance may be available via facility licensee training observation forms. 

5. Observe and evaluate simulator performance if the selected training/examination activity is conducted in the control room simulator.  

Review simulator physical modeling and simulator performance, especially regarding recent modifications implemented in the control room.  Refer to Section 03.04.g of this IP for additional information regarding this inspection activity.  Based upon the limited observation time of the resident staff and their variable experience with simulators, it is not expected that the resident staff will review any facility licensee simulator corrective action or testing records.  During this inspection activity and upon its conclusion, discuss any simulator performance issue with the facility licensee to assist in confirming the issue.  If there are any significant concerns with simulator performance, contact regional management.  In evaluating any simulator performance issues, refer to Section 03.04.g and Appendix G of this IP. 

## 03.03	Requalification Examination Results

Collect and assess requalification examination results.

Specific Guidance

After the facility licensee has completed and graded the exams of any training cycle which contains a licensed operator requalification annual operating test or biennial written examination required by 10 CFR 55.59, contact the facility licensee and determine the following:

1. Determine the composition of the examinations administered (e.g. written examinations, simulator scenarios, job performance measures (JPMs)).

2. Collect the examination results to complete the following table:

TABLE 03.03-1 EXAMINATION RESULTS

3. If any individual licensed operator failed any portion of a requalification examination (line 3) or any crew failed the simulator scenarios (line 6), determine whether the licensee remediated and the operator or crew passed a re-examination.  If the inspector identifies any concerns with re-examinations which could impact whether licensed operators fulfil a condition of their license, refer to Section 03.04.f of this IP.

4. If the failure rate (individual or crew) for any requalification examination required by 10 CFR 55.59(a)(2) exceeds 20% as identified above (line 4 or line 7), this shall be considered a performance deficiency against the expected knowledge and abilities of licensed operators.  This performance deficiency is typically not considered a violation of regulatory requirements.  

5. Obtain the total number of licensed operators at the site from the region’s Operator Licensing Assistant (RPS OL Report 9).  Discuss any differences in the total number of licensed operators at the site with the total number of licensed operators who took a requalification examination with the facility licensee.  Discuss any licensed operator who did not take a complete examination and any plans the licensee has for makingup any missed examinations.  Determine whether each licensed operator is taking the required requalification examinations as set forth in 10 CFR 55.59(a)(2).

6. Discuss when the next requalification examinations will be administered with the facility licensee.  This will allow the region to plan for future inspections.

## 03.04	Licensed Operator Requalification Program

a. Biennial Requalification Written Examinations 

Review the quality of licensee-developed biennial requalification written examinations.   

Specific Guidance

Typically, facility licensees develop multiple similar versions[footnoteRef:5] of the biennial written examination, with different versions administered on subsequent weeks for different groups or operating crews of licensed operators.  To complete this inspection activity, a minimum of one complete (operator and senior operator) version of the requalification biennial written examination required by 10 CFR 55.59(a)(2) shall be reviewed using the checklists and guidance provided in Appendix B.  If greater than 20 percent of the written examination questions reviewed are determined to be flawed, then a second complete version of the written examination from the same training cycle shall be reviewed.  If additional written examinations are reviewed for reasons other than the percentage of flawed questions, focus on new or modified questions and review overlap between examinations.   [5:  Multiple versions of requalification examinations are necessary to prevent any potential examination compromise due to the different dates on which the examination is administered.  See the inspection guidance contained in Section 03.04.d] 

The biennial requalification written examinations to be reviewed should be from the training cycle that the inspectors are on-site[footnoteRef:6].  However, an alternate review schedule may be used such that the biennial requalification written examinations to be reviewed would be from an examination which occurred since the last NRC biennial review, such that this IP reviews at least one version of the biennial requalification written examination from each of the licensee’s successive 24-month requalification programs.   [6:  When reviewing an exam yet to be administered, inspectors shall be careful to protect the examination from inadvertent disclosure to unauthorized facility personnel and adhere to the facility licensee’s examination security procedures and policies.] 

Upon the completion of this review, discuss any written examination quality issue with the facility licensee to assist in confirming the issue.  However, the inspector should not interfere with the facility licensee’s requalification examination process by suggesting modifications to test items or examination schedules.  If there are significant concerns with the quality of the written examinations reviewed, contact regional management.

b. Annual Requalification Operating Tests 

Review the quality of licensee-developed annual requalification operating tests.

Specific Guidance

Review the quality of a minimum of 10 JPMs and 4 simulator scenarios associated with an annual requalification operating test required by 10 CFR 55.59(a)(2), using the checklists and guidance provided in Appendix C.

Review the JPMs and scenarios that will be observed during the on-site portion of this inspection.  If more JPMs and scenarios are required to be reviewed, select those JPMs and scenarios from other weeks within the same training cycle.  In addition, check for excessive test item repetition between operating tests administered during different weeks within a training cycle (see Section 03.04.d).

Upon the completion of this review, discuss operating test quality issues with the facility licensee to assist in confirming the issue.  However, the inspector should not interfere with the facility licensee’s requalification examination process by suggesting modifications to test items or examination schedules.  If there are significant concerns with the quality of the operating tests reviewed, contact regional management.

c. Administration of an Annual Requalification Operating Test

Observe the administration of simulator scenarios and JPMs during the conduct of an annual requalification operating test required by 10 CFR 55.59(a)(2).

Specific Guidance

This inspection activity should be conducted during a one week on-site visit.  However, the region may re-visit the site and observe the annual requalification operating test for a different group or crew of licensed operators.  Observe simulator scenarios as a team and separately observe different licensed operators and facility evaluators during the conduct of JPMs.  

The basic methodology for performing this inspection activity is to observe licensed operator and facility licensee evaluator and administrator performance during an annual requalification operating test.  Additionally, observe post-simulator scenario critiques and other facility licensee operating test grading activities.  Assess the following items: 

1. Licensed operator performance, including:

· Crew performance in terms of clarity and formality of communication

· Ability to take timely action in the safe direction

· Prioritizing, interpreting, and verifying alarms

· Correct use and implementation of procedures, including the alarm response procedures

· Timely control board operation and manipulation, including high-risk operator actions

· Oversight and direction provided by the shift supervisor, including ability to identify and implement appropriate technical specifications actions such as reporting and emergency plan actions and notifications 

· Group dynamics involved in crew performance

2. The facility licensee’s ability to administer the annual requalification operating test (refer to Appendix D of this IP).

3. The facility licensee’s ability to assess the performance of their licensed operators (refer to Appendix D of this IP).

4. The adequacy of plant procedures.

5. The quality of the annual requalification operating test scenario guides and JPMs (see Section 03.04.b).

6. Examination security (see Section 03.04.d).

7. Simulator performance (see Section 03.04.g).

During this inspection activity and upon its conclusion, discuss any operating test administration issues with the facility licensee to assist in confirming the issue.  However, the inspector should not interfere with the facility licensee’s requalification examination process.  If there are significant concerns with the facility licensee’s administration of the operating test, contact regional management.

d. Requalification Examination Security

Evaluate the effectiveness of the facility licensee’s requalification examination security measures.

Specific Guidance   

Examination security encompasses all practices taken by the facility licensee to ensure compliance with 10 CFR 55.49, which states, in part:

Applicants, licensees, and facility licensees shall not engage in any activity that compromises the integrity of any application, test, or examination required by this part. The integrity of a test or examination is considered compromised if any activity, regardless of intent, affected, or, but for detection, would have affected the equitable and consistent administration of the test or examination.

To perform this inspection activity, complete the checklist contained in Appendix E which includes: (1) checking for excessive test item repetition, (2) observing examination security practices during the administration of an annual operating test, and (3) reviewing examination security incidents.  Discuss any excessive test item repetition with the facility licensee, who should have spreadsheets and/or test outlines which show the written examination questions, JPMs, and scenarios to be used during the entire requalification examination testing cycle.  Discuss any requalification examination security incidents that have occurred since the last biennial inspection with the facility licensee.  

e. Remedial Training and Re-Examinations	

Verify the facility licensee properly implements remedial training and re-examinations.

Specific Guidance

Since the last biennial inspection, determine from the facility licensee if any individual licensed operators or crews failed any portion (written, JPM, or simulator scenario examination) of an NRC requalification examination required by 10 CFR 55.59(a)(2), and determine what type of failure(s) occurred - individual failures or crew failures; written, JPM or simulator scenario examination failures.  For each examination failure, complete the checklist contained in Appendix F.

Upon completion of this review, discuss any remedial training and re-examination issues with the facility licensee to assist in confirming the issue.  If there are significant concerns with the licensee’s performance in this area, contact regional management.  

f. Operator License Conditions

Review a sampling of individual licensed operator records, to verify that the facility licensee has effective processes for ensuring the conformance with operator license conditions.

Specific Guidance

Operator license conditions are contained in 10 CFR 55.53 and 10 CFR 55.59, and include requalification training attendance, maintaining an active license, and medical fitness.  This inspection activity should be performed primarily during the on-site portion of this inspection.  However, prior to the on-site review: (1) obtain a list of all the licensed operators at the site (RPS OL Report 9) and a list of all licensed operator medical restrictions, if any (RPS OL Report 14); (2) preselect licensed operators for review of their medical records, (refer to paragraph 4 below); and (3) review the selected licensed operators’ medical information contained in their 10 CFR Part 55 individual docket files (available from the Operator Licensing Assistant).  Additional guidance for performing this inspection activity is presented in NUREG-1021, “Operator Licensing Examination Standards for Power Reactors,” Section ES-5.3.

1. For one complete operating crew of licensed operators[footnoteRef:7], review the following: [7:  For multi-unit sites, review records for all licensed operators assigned to a shift for all of the units.] 

(i) Records that indicate the participation of licensed operators in the facility licensee’s requalification program (i.e., training attendance records) (10 CFR 55.53(h), 10 CFR 55.59(a)(1), 10 CFR 55.59(c)(5)(i)).  Determine if all requalification training is completed on schedule or made up in accordance with the facility's program.

(ii) Records that indicate the performance of licensed operators on annual requalification operating tests and biennial requalification written examinations (10 CFR 55.59(a)(2), 10 CFR 55.59(c)(5)(i)).

(iii) Records that indicate that licensed operators are properly maintaining an active license (10 CFR 55.53(e)).  Include a determination of which shift crew positions the facility licensee credits towards maintaining an active license.

(iv) NRC Form 398, “Personal Qualification Statement – Licensee,” that is in effect at the time of the inspection and any supporting documentation that indicate operators’ licenses were properly issued or renewed.

2. Determine from the facility licensee if any licensed operator with an inactive license reactivated their license since the last NRC biennial inspection, including reactivation for refueling mode supervision of core alterations.  If license reactivation did occur, select one or more licensed operators who reactivated their license(s) and review records which indicate that their license(s) was (were) properly reactivated in accordance with 10 CFR 55.53(f).

3. Determine from the facility licensee if any licensed operator did not pass a biennial requalification written examination or annual requalification operating test required by 10 CFR 55.59(a)(2) since the last NRC biennial inspection.  If examination failure did occur, select one or more licensed operators who failed an examination, and review records which indicate that the operator(s):  (1) received remedial training, (2) was (were) re-examined, and (3) passed their re-examination(s).

4. Review a sample (approximately 10%) of licensed operator medical records, including any medical records maintained in the regional office’s Part 55 docket system.  Approximately 50% of this sample, should include licensed operators whose licenses contain medical restrictions.  Determine which version of Regulatory Guide 1.134, “Medical Evaluation of Licensed Personnel at Nuclear Power Plants,” and the associated version of ANSI/ANS-3.4, “Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants,” that the facility licensee has committed to.  

In performing this review, check that:

(i) The required medical examinations are conducted biennially (10 CFR 55.53(i)).

(ii) The results of medical examinations agree with any license medical restrictions (10 CFR 55.23(b)).

(iii) NRC Form 396, “Certification of Medical Examination by Facility Licensee,” that is in effect at the time of the inspection including any supporting medical documentation, is accurate and complete.

(iv) Any medical status reports, if required by NRC Form 396, are submitted to the regional office in a timely fashion.

During this inspection activity and upon its conclusion, discuss any issue associated with the conformance with operator license conditions with the facility licensee to assist in confirming the issue.  If it is determined that any licensed operator has not properly conformed with the conditions of their license, discuss with the facility licensee:

1. Any immediate actions to take, such as removing the affected licensed operator(s) from on-shift licensed operator duties and notifying the regional office.

2.  Any plans for restoring compliance with operator license conditions.

g. Control Room Simulator

Evaluate the performance of the control room simulator and review records which indicate that the facility licensee is properly testing and maintaining the control room simulator. 

Specific Guidance

This inspection activity consists of the following elements:

1. Observe simulator modeling and performance during an in-progress annual operating test and note any simulator modeling or performance deficiencies.

2. Review a listing of open simulator deficiencies as maintained by the facility licensee in their simulator corrective action program.  Sample the facility licensee’s proposed corrective actions for any open simulator deficiencies.

3. Review a listing of simulator deficiencies closed by the facility licensee since the last biennial inspection.  Sample the corrective actions taken by the facility licensee in closing simulator deficiencies.

4. Review a sample of records associated with any simulator testing performed by the facility licensee since the last biennial inspection.

5. Review a sample of records associated with any simulator modifications made by the facility licensee since the last biennial inspection.

In performing this inspection activity, determine which version of Regulatory Guide 1.149, “Nuclear Power Plant Simulation Facilities for Use in Operator [Training and] License Examinations,” and the associated version of ANSI/ANS-3.5, “Nuclear Power Plant Simulators for Use in Operator Training [and Examination],” that the facility licensee is committed to.  Determine if the facility licensee is using the simulator to meet the control manipulation eligibility requirements contained in 10 CFR 55.31(a)(5).  In performing this inspection activity, answer the following three questions:

1. Is simulator modeling and performance satisfactory?

2. Does the facility licensee effectively correct identified simulator deficiencies?

3. Does the facility licensee properly perform required simulator testing?  In answering this question, check for correct simulator testing periodicity, proper evaluations of the simulator against the reference plant or best-estimate data, and correct documentation and retention of simulator test records.

10 CFR 55.46 provides additional details towards answering these questions, and 10 CFR 55.46 presents a checklist in Appendix G.  Regulatory Guide 1.149 and ANSI/ANS-3.5 provide additional details.

During this inspection activity and upon its conclusion, discuss any simulator performance issue with the facility licensee to assist in confirming the issue.  If there are significant concerns regarding simulator performance, contact regional management.  In evaluating any simulator performance issues, refer to Appendix G, 10 CFR 55.46, the applicable versions of Regulatory Guide 1.149, and ANSI/ANS-3.5.

# 71111.11-04	REFERENCES

10 CFR 55, “Operators’ Licenses”https://www.nrc.gov/reading-rm/doc-collections/cfr/part055/

NUREG-1021, “Operator Licensing Examination Standards for Power Reactors”https://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1021/

IMC 0609 Appendix I, “Licensed Operator Requalification Significance Determination Process”https://www.nrc.gov/reading-rm/doc-collections/insp-manual/manual-chapter/index.html

IP 71152, “Problem Identification and Resolution”https://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html

Regulatory Guide 1.134, “Medical Assessment of Licensed Operators or Applicants for Operator Licenses at Nuclear Power Plants”https://www.nrc.gov/reading-rm/doc-collections/reg-guides/power-reactors/rg/division-1/division-1-121.html

ANSI/ANS-3.4, “Medical Certification and Monitoring of Personnel Requiring Operator Licenses for Nuclear Power Plants”http://www.internal.nrc.gov/TICS/library/standards/ihs.html

Regulatory Guide 1.149, “Nuclear Power Plant Simulation Facilities for Use in Operator Training and License Examinations”https://www.nrc.gov/reading-rm/doc-collections/reg-guides/power-reactors/rg/division-1/division-1-141.html

ANSI/ANS-3.5, “Nuclear Power Plant Simulators for Use in Operator Training and Examination”http://www.internal.nrc.gov/TICS/library/standards/ihs.html

# END

List of Appendices

A.  Typical Documents Reviewed During Biennial Inspection

B.  Biennial Requalification Written Examination Quality Checklist and Worksheet

C.  Annual Requalification Operating Test Quality Checklist and Worksheets

D.  Operating Test Administration Checklist

E.  Requalification Examination Security Checklist

F.  Remedial Training and Re-Examination Checklist

G.  Checklist for Evaluating Plant-Referenced Simulators Operating Under 10 CFR 55.46(c) and (d)

H.  Generic Licensed Operator Observation Checklists

Attachment 1: Revision History

Issue Date:  12/29/21	10	71111.11

APPENDIX A

TYPICAL DOCUMENTS REVIEWED DURING BIENNIAL INSPECTION

1*.	Biennial requalification written examination(s) administered during the training cycle.  

2*.	JPMs and simulator scenarios associated with an annual requalification operating test required by 10 CFR 55.59(a)(2).  The JPMs and scenarios to be reviewed should include those that will be or have been observed during the onsite portion of this inspection, with any remaining JPMs and scenarios reviewed selected from other weeks within the same training cycle.

3*.	Spreadsheets and/or test outlines which show the usage of written examination questions, JPMs, and simulator scenarios. 

4*.	A list (or lists) of licensee-identified issues associated with licensed operator errors or other licensed operator performance problems which have occurred in the actual plant/main control room since the last biennial requalification program inspection,  their associated corrective actions, and if these issues were incorporated into requalification training. 

5*.	Facility licensee procedures for licensed operator requalification training and examination, including examination security procedures.

6*.	A schedule of the licensee’s examination activities during the on-site inspection week.

7**.	A list of all the licensed operators at the site, a list of all licensed operator medical restrictions at the site, and selected individual licensed operator 10 CFR 55 docket files.

8**.	NRC records which document licensed operator performance issues and the facility licensee’s corrective action program performance since the last biennial requalification inspection.  These records include:  NRC biennial PI&R team inspection reports, NRC Annual Performance reports, NRC inspection findings, Plant Issues Matrix, and NRC operating experience information.

9.	Licensed operator pass/fail statistics for any NRC-required requalification examination that has been completed (see Table 03.03-1, Examination Results). 

10.	An overall schedule of the facility licensee’s requalification program since the last biennial inspection (cycle weeks, training topics, etc.)   

11.	A list and descriptions of any examination security problems since the last biennial inspection (likely documented in training department condition/problem identification reports).  

TYPICAL DOCUMENTS REVIEWED DURING BIENNIAL INSPECTION (Continued)

12.	Records for licensed operator requalification training attendance, licensed operator performance in requalification, licensed operator remedial training, individual licensed operator medical records, and records for maintaining an active license and license reactivation (including reactivations for refueling mode supervision of core alterations).

13.	Simulator testing, maintenance, modification, and performance records.  Simulator corrective action records, including lists of open/closed simulator deficiencies, and corrective actions taken.   

*It is recommended that the NRC request that the facility licensee submit these items prior to the on-site portion of the biennial inspection, such that portions of the biennial inspection can be reviewed and discussed with the facility licensee while on-site, and for inspection planning purposes. 

**These items should be reviewed prior to the on-site portion of the biennial inspection.

Issue Date:  12/29/21	AppA-2	71111.11

APPENDIX B 

BIENNIAL REQUALIFICATION WRITTEN EXAMINATION QUALITY CHECKLIST

Written examinations reviewed or dates administered:  ____****____

If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected quality standards for a licensed operator requalification written examination required by 10 CFR 55.59(a)(2), and as a possible finding.  

Issue Date:  12/29/21	AppB-1	71111.11

Requalification Written Examination Question Review Worksheet 

Written examination reviewed or date administered:  ________

Instructions:

1.	Enter the level of difficulty (LOD) of each question using a 1-5 (easy – difficult) rating scale. 

2.	Check the appropriate block if a question flaw is identified:

· The question’s level of difficulty is inappropriate (LOD = 1 too easy; LOD = 5 too hard). 

· The question is a direct look-up.   For open-reference questions, if the question stem provides the reference title and/or number and the correct answer requires no understanding, application, analysis, or problem solving, but simply requires the ability to find the information in the reference, then the question is likely a direct look-up. See NUREG-1021, ES-6.2 for additional guidance.  

· The question has no correct answer or more than one correct answer.  Short-answer questions must contain objective scoring, with clear guidance on granting partial and full credit.

	In addition, check the following items for all multiple choice questions (see NUREG-1021, Appendix B):

· The stem or answer choices contain cues as to the correct answer (i.e., clues, specific determiners, phrasing, length, etc.)

· The answer choices are a collection of unrelated true/false statements such that the question can be answered correctly without reading the question stem.

· Two or more distractors are not credible.  For open-reference questions, if the question stem provides the reference title and/or number and a distractor can be easily eliminated because it is not located anywhere in the reference, then that distractor should be considered not credible. 

3.	Based on review item 2 above, the question as written is either flawed (F) or not flawed (NF) .

4.	Provide a brief explanation for all questions that are determined to be flawed (F).  

Requalification Written Examination Questions Review Worksheet (Continued)

Written examination reviewed or date administered:  ________

APPENDIX C

ANNUAL REQUALIFICATION OPERATING TEST QUALITY CHECKLIST

Operating test # or date(s) administered: _______ 

ANNUAL REQUALIFICATION OPERATING TEST QUALITY CHECKLIST (continued)

If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected quality standards for a licensed operator requalification operating test required by 10 CFR 55.59(a)(2), and as a possible finding.  

Issue Date:  12/29/21	AppC-2	71111.11

Requalification Job Performance Measure Review Worksheet

Operating test # or date(s) administered: _______

Instructions:

1.	Enter the type of JPM – (S)imulator, (P)lant, or (O)ther.

2.	Enter (Y)es or (N)o whether the JPM is alternate path or not. 

3.	Enter the level of difficulty (LOD) of each JPM using a 1-5 (easy – difficult) rating scale. (LOD > 1 and < 5 are acceptable).  Also see below. 

4.	Check the appropriate block if a JPM flaw is identified:

· The JPM’s level of difficulty is inappropriate.   Simple one-step JPMs, or a JPM that tests solely for recall or memorization, or a JPM which requires directly looking-up a single correct answer is likely LOD = 1 and too easy.  Conversely, a JPM with over 30 steps or that takes in excess of 45 minutes to complete is likely LOD = 5 and too hard. 

· The JPM lacks adequate initial conditions or lacks an adequate initiating cue.  	

· The JPM lacks adequate evaluator cues to allow the examinee to complete the task. 

· The JPM lacks adequate performance standards and/or contains errors in designating critical steps. 

· The JPM lacks an appropriate validation time or lacks a time for completion standard.

5.	Based on the review of item 4 above, the JPM as written is either flawed (F) or not flawed (NF).

6.	Provide a brief explanation for all JPMs that are determined to be flawed (F).

Issue Date:  12/29/21	AppC-3	71111.11

Simulator Scenario Review Worksheet

Instructions:

1.	Enter the scenario identifier and week within the training cycle that the scenario will be (or has been) administered. 

2.	Minimum number of events per scenario are listed.  

3.	Enter the actual number and types of events in the scenario. In order for a scenario event to be counted towards the listed minimums, the scenario event must contain verifiable operator action(s) and an associated performance standard(s) to mitigate or address the event.  

4.	Subtract each column in line 3 from line 2.  Enter zero if the minimum number of events for that event type is equaled or exceeded.

5.	Add up the columns from line 4, and determine the total number of events less than the minimums that the scenario has.  Any deficit will be used in computing line 8.a of the Requalification Operating Test Quality Checklist (pages C-1, C-2), when totaled with the deficits from other scenarios.

6.	Enter the scenario event name and description.      

7.	Review the individual events contained in each scenario, and evaluate for event flaws:

· The scenario guide event description lacks adequate crew/operator performance standards.

· The scenario guide event description incorrectly designates an event as a critical task (i.e., a non-critical task labeled as critical or a critical task labeled as non-critical).

The number of scenario events that are flawed and not flawed will be used in computing line 8.b of the Requalification Operating Test Quality Checklist (page C-2), when totaled with the flawed/not flawed events from other scenarios.

8.	Provide a brief description for any scenario event determined to be flawed (F).

Abbreviations:

Malf. = Malfunction	EOPs = Emergency Operating Procedures	

ABNs = Abnormals	MTs = Major Transients

CTs = Critical Tasks		TS = Technical Specifications	

Issue Date:  12/29/21	AppC-4	71111.11

APPENDIX D

OPERATING TEST ADMINISTRATION CHECKLIST 

Date(s) operating test observed: _______

If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected standards for the administration of a licensed operator requalification operating test required by 10 CFR 55.59(a)(2), and as a possible finding. 

Issue Date:  12/29/21	AppD-1	71111.11

APPENDIX E

REQUALIFICATION EXAMINATION SECURITY CHECKLIST

If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected standards for examination integrity and as a possible finding, typically associated with a regulatory violation of 10 CFR 55.49.  If the equitable and consistent administration of an examination was actually affected, traditional enforcement should also be considered. 

Issue Date:  12/29/21	AppE-1	71111.11

APPENDIX F

REMEDIAL TRAINING AND RE-EXAMINATION CHECKLIST 

If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the expected standards (either quality, integrity, or operator performance) associated with licensed operator requalification examinations required by 10 CFR 55.59(a)(2), and as a possible finding.  

Issue Date:  12/29/21	AppF-1	71111.11

APPENDIX G

CHECKLIST FOR EVALUATING PLANT-REFERENCED SIMULATORS OPERATING UNDER 10 CFR 55.46 (c) AND (d)

CHECKLIST FOR EVALUATING PLANT-REFERENCED SIMULATORSOPERATING UNDER 10 CFR 55.46 (c) AND (d) (Continued)

If any block in this checklist is checked “NO”, these items shall be considered a performance deficiency against the standards associated with 10 CFR 55.46, “Simulation facilities,” and shall be processed as potential findings typically with a regulatory violation against 10 CFR 55.46.  

Issue Date:  12/29/21	AppG-2	71111.11

APPENDIX H 

GENERIC LICENSED OPERATOR OBSERVATION CHECKLISTS

CONTROL ROOM/IN PLANT OBSERVATION CHECKLIST

The following is presented as generic guidance for observing licensed operators in the main control room or in the plant.  The listed items are not regulatory requirements.  For additional plant-specific guidance, refer to the licensee’s conduct of operations policies. 

CONTROL ROOM/IN PLANT OBSERVATION CHECKLIST (Continued)  

The following is presented as generic guidance for observing licensed operators in the main control room or in the plant.  The listed items are not regulatory requirements.  For additional plant-specific guidance, refer to the licensee’s conduct of operations policies.

ACTIVITY BRIEFING CHECKLIST

The following is presented as generic guidance for observing activity briefs, and is primarily suited for assessing pre-job briefs, although aspects of this checklist may also apply to status briefs conducted during an evolution.  The listed items are not regulatory requirements.  For additional plant-specific guidance, refer to the licensee’s briefing policies.   

LICENSED OPERATOR PERFORMANCE:  NOTES AND COMMENTS

DATE(S):  ______________

EVOLUTION(S) OBSERVED:  ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

NOTES AND COMMENTS:

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Issue Date:  12/29/21	AppH-4	71111.11

Attachment 1: - Revision History for IP 71111.11

Issue Date:  12/29/21	Att1-3	71111.11