**ATTACHMENT 71111.11**

INSPECTABLE AREA: Licensed Operator Requalification Program

CORNERSTONES: Initiating Events (10%)

> Mitigating Systems(70%)
>
> Barrier Integrity(10%)
>
> Emergency Preparedness (10%)

INSPECTION BASES: The inspection evaluates licensed operator performance
in mitigating the consequences of events. Poor licensed operator
performance results in increased risk through increased operator
recovery rates and increased personnel-induced common-cause error rates
assumed in the licensees= individual plant examinations (IPEs).

> This inspectable area verifies the following key attributes for which
> there are no performance indicators: (1) human performance (pre- and
> post-event human error) in the Initiating Event cornerstone as well as
> the Mitigating Systems and Barrier Integrity cornerstones; (2)
> procedure quality issues (post-event operating procedures), in the
> event licensed operator performance issues are identified and human
> performance in the Initiating Events and Mitigating Systems
> cornerstone; and (3) emergency response organization performance
> \[self-assessment, severe-accident management guidelines
> implementation and actual response\] regarding initial and subsequent
> interactions by licensed operators \[emergency action levels (EALs)
> and protective action recommendations (PARs)\] in the Emergency
> Preparedness cornerstone.

LEVEL OF EFFORT: [Biennial Review by Regional Specialist]{.underline}

> A biennial review of the licensed operator requalification program
> will be conducted by regional specialist at the facility licensee=s
> site. The level of effort includes in-office review of tests that may
> be performed by the regional specialist. Depending on availability,
> resident staff members may assist the regional specialist during the
> biennial review.
>
> [Requalification Activities Review by Resident Staff]{.underline}
>
> A review of the licensed operator requalification testing and/or
> training activities will be conducted by the resident staff at least
> once each quarter.

71111.11‑01 INSPECTION OBJECTIVES

01.01 To verify that the facility licensee\'s requalification program
for licensed reactor operators (ROs) and senior reactor operators (SROs)
ensures safe power plant operation by adequately evaluating how well the
individual operators and crews have mastered the training objectives,
including training on high-risk operator actions with senior reactor
analyst=s (SRA=s) input.

01.02 To assess the facility licensee\'s effectiveness in evaluating and
revising the requalification program for licensed operators based on
their operational performance, including requalification examinations.

01.03 To assess the facility licensee\'s effectiveness in ensuring that
the individuals who are licensed to operate the facility satisfy the
conditions of their licenses as specified in 10 CFR 55.53.

01.04 To supply regional management with the information necessary to
assess the performance of the facility licensee\'s licensed operator
requalification program and determine the need for additional
inspections or NRC‑conducted examinations.

71111.11-02 INSPECTION REQUIREMENTS

02.01 [Inspection Composition]{.underline}. For biennial reviews, the
inspection should include at least one qualified operator licensing
examiner with expertise relevant to the plant(s) being evaluated.
Normally, an inspection would include individuals with operations
backgrounds and individuals with plant-specific knowledge. For quarterly
reviews, the resident staff will use applicable portions of this
procedure when completing the simulator review each quarter.

02.02 [Sample Selections]{.underline}. When selecting sample areas to
inspect within the licensed operator requalification process, a
risk‑informed, performance‑based regulatory approach should be
considered in which risk insights, engineering analysis and judgment,
including the principle of defense‑in‑depth and the incorporation of
safety margins, and performance history are used to (1) focus attention
on the most important activities, (2) establish objective criteria for
evaluating performance, and (3) develop measurable or calculable
parameters for monitoring system and licensee performance.

No specific number of comprehensive written examination or operating
test samples is recommended. Rather, the inspector should choose as many
examples as warranted to ensure a sufficient basis for evaluating the
effectiveness of the licensee=s requalification program.

02.03 [Facility Operating History]{.underline}. Assess operator
performance since the last requalification program evaluation
(inspection or examination) to determine if performance deficiencies
have been addressed through the requalification training program.

02.04 [Licensee Requalification Examinations]{.underline}. Assess the
adequacy of the facility licensee\'s written examinations and operating
tests for requalification.

02.05 [Licensee Administration of Requalification
Examinations]{.underline}. Observe examinations and tests in progress
and interview personnel to assess the facility licensee\'s effectiveness
in conducting written examinations and operating tests to ensure
operator mastery of the requalification training program content.

02.06 [Licensee Training Feedback System]{.underline}. Assess the
effectiveness of the facility licensee\'s process for revising and
maintaining its licensed operator continuing training program up to
date, including the use of feedback from plant events and industry
experience information.

02.07 [Licensee Remedial Training Program]{.underline}. Assess the
adequacy and verify the effectiveness of the remedial training conducted
since the last requalification examinations and the training planned for
the current examination cycle to ensure that it addresses weaknesses in
licensed operator or crew performance identified during training and
plant operations.

02.08 [Conformance With Operator License Conditions]{.underline}. Review
the facility licensee\'s program for maintaining active operator
licenses and ensuring the medical fitness of its licensed operators.
Assess the facility and operator licensees\' compliance with the
requirements for maintaining license conditions in accordance with 10
CFR 55.53.

02.09 [Written Examination and Operating Test Results]{.underline}. For
each requalification cycle, review the number of applicants and the
pass/fail results of written examinations, individual operating tests
and simulator operating tests.

02.10 [Resident Inspector Quarterly Review of Licensed Operators=
Requalification Testing and/or Training Activities]{.underline}. At
least once each quarter, observe testing and training for SROs and ROs,
identify deficiencies and discrepancies in the training, and assess
licensed operator performance and evaluator=s critique. Emphasis should
be placed on observing training on high-risk licensed operator actions,
operators= activities associated with the emergency plan and previous
lessons learned items or plant experiences. Observations of operating
crew performance conducted in accordance with IP 71114.06, ADrill
Evaluation,@ may be used to satisfy portions of the quarterly inspection
requirements.

02.11 [Conformance with Simulator Requirements Specified in 10 CFR
55.46]{.underline}. Assess the adequacy of the facility licensee\'s
simulation facility for use in operator licensing examinations and for
satisfying experience requirements as prescribed in 10 CFR 55.46. Assess
the effectiveness of the facility licensee\'s process for continued
assurance of simulator fidelity with regard to identifying, reporting,
correcting, and resolving simulator discrepancies via a corrective
action program.

71111‑03 INSPECTION GUIDANCE

[General Guidance]{.underline}

Facility licensees are required by 10 CFR 50.54(i‑1) to have in effect a
Commission‑approved operator requalification program that must, as a
minimum, meet the requirements of 10 CFR 55.59(c). In lieu of paragraphs
(c)(2), (3), and (4) of that section, the Commission may approve a
program developed by using a systems approach to training (SAT), as
defined in 10 CFR 55.4. In accordance with 10 CFR 55.59(a), each
licensed individual must successfully complete the requalification
program developed by the facility licensee and pass an annual operating
test and a comprehensive written examination administered at the end of
each requalification cycle, not to exceed 24 months in duration.

This baseline inspection procedure is intended to determine if a
facility licensee\'s requalification program meets elements (4) and (5)
of a SAT‑based program as defined in 10 CFR 55.4. Inspectors should
prioritize their activities to ensure that inspection requirements
02.03, 02.04, 02.05, 02.09,and 02.11 are completed first. Inspection
requirements 02.06, 02.07, and 02.08 are to be considered and performed
to the extent necessary to conclude that the objectives of the
inspection procedure have been met. In some cases a specific inspection
requirement need not be addressed because the inspector is satisfied
from inspections already conducted or from other information that the
licensee\'s activities are acceptable.

If regional management determines that the facility licensee\'s licensed
operator requalification program is not based on a systems approach to
training as defined in 10 CFR 55.4, consult with the headquarters
program office regarding the appropriate response. Regional management
should submit all proposed enforcement actions related to 10 CFR Part 55
to the Office of Nuclear Reactor Regulation (NRR) staff for review
before issuing them.

The region should announce its intent to conduct requalification
inspection activities at a facility. In order to better coordinate with
the licensee=s requalification examination schedule, it is acceptable to
conduct this inspection in two annual parts. This should be carefully
planned in order to maximize coverage of the inspection procedure while
staying within the allotted resources. Although most of the inspection
activities will be conducted while the facility licensee administers its
annual operating tests, the region may exercise discretion regarding
where and when it completes some of the inspection requirements. For
example, if the region asks the facility licensee to submit specific
examinations to the Nuclear Regulatory Commission (NRC) before the site
visit, the inspectors can complete portions of inspection requirements
02.03 and 02.04 before they travel to the facility. It is anticipated
that two inspectors will then be able to complete the remaining
inspection requirements during a one‑week visit to the site. If the
region does not ask the facility licensee to submit its examinations in
advance, the region may send an inspector to the site to review the
examination materials in preparation for the primary inspection. As a
third option, the region may dispatch three inspectors to complete all
the inspection requirements during a one‑week site visit. When planning
inspection efforts, keep in mind that the regulations only require the
facility licensee to administer a comprehensive written examination
every two years unless its approved requalification program requires
more frequent examinations.

In accordance with 10 CFR 55.59(c), facility licensees are required to
submit to the Commission, upon request, the annual operating tests or
comprehensive written examinations used for operator requalification.
The region may request those tests and examinations in writing by
sending the licensee a corporate notification letter similar to the one
that is used for NRC‑conducted examinations. Usually, the region will
ask the facility licensee to submit only those examinations and tests
that will be administered during the week of the inspection. Other
examination materials, such as previously administered examinations and
tests, question banks, and sample plans, are normally reviewed on site.

Regional managers will consider overall facility performance in the
findings of the NRC\'s inspection programs and initial examinations.
Generally, only the inspection requirements of this procedure will need
to be conducted; however, augmented activities can be initiated in
accordance with10 CFR 55.59(a)(2)(iii) and program office guidance when
necessary to ensure safe plant operation. Those activities could include
a full \"Training and Qualification Effectiveness\" inspection in
accordance with Inspection Procedure (IP) 41500, \"for cause\"
examinations in accordance with NUREG‑1021, \"Operator Licensing
Examination Standards for Power Reactors,\" or operational evaluations
of on‑shift crews.

Since the inspection process relies on sampling a basically sound
facility program, the NRC would conduct examinations at the facility
only when it has lost confidence in the facility licensee\'s ability to
conduct its own examinations or when the staff believes that the
inspection process will not provide the needed insight. Regional
management should consider conducting \"for cause\" requalification
examinations or operational evaluations when any of the following
conditions exist:

> \$ Requalification inspection findings that indicate an ineffective
> licensee requalification program (e.g., one yellow finding or multiple
> white findings based on the Operator Requalification Human Performance
> Significance Determination Process),
>
> \$ Operational problems to which operator error is a major
> contributor, or

-   Allegations regarding significant training program deficiencies.

Implement \"for cause\" examinations through the normal resource
planning process, since an inspection activity would be replaced with
more resource‑intensive examinations. Using the existing inspection
planning process will ensure that the regional office and NRR consider
the need for conducting examinations with alternative expanded
inspection tools available, and will allocate the required resources.
Operational evaluations should be considered as a reactive effort based
on immediate safety concerns.

Most issues that meet the threshold as defined in IMC 0612, Appendix B,
for assessment using the SDP will relate to mitigating activities and
should be assigned to the Mitigating Systems cornerstone. Should the
finding clearly relate to the breech of a barrier, it should be assigned
to the Barrier Integrity cornerstone. Should the finding clearly relate
to an error by the operator that would cause an event had it been on the
actual plant, it should be assigned to the Initiating Events
cornerstone. In all cases, the inspector should provide a rationale for
the cornerstone assignment.

[Specific Guidance]{.underline}

03.01 [Inspection Composition]{.underline}. Refer to paragraph 02.01.

03.02 [Sample Selections]{.underline}. Refer to paragraph 02.02.

03.03 [Facility Operating History]{.underline}. Review the following
documents to determine the effectiveness of the facility\'s licensed
operator requalification training program:

> a\. Most recent plant issue matrix (PIM) report, and plant performance
> review (PPR) report.
>
> b\. Recent examination and inspection reports \[e.g., emergency
> preparedness or emergency operating procedure (EOP) inspections\]
> related to operator training or performance.
>
> c\. Resident inspector observations and reports regarding operator
> performance.
>
> d\. Licensee event reports (LERs).
>
> e\. Other indications of operator performance, such as technical
> specification (TS) violations, internal event reports, human factors
> information system (HFIS) reports, and NRC performance indicators
> \[e.g., engineered safety feature (ESF) actuations and reactor scrams
> or trips\].

In particular, look for patterns of operator performance that create
concern regarding the continued safe operation of the facility. If
safety concerns are identified, consider, in consultation with the
operator licensing program staff, such actions as holding management
meetings, conducting operational evaluations, or taking appropriate
licensing or enforcement actions.

03.04 [Licensee\'s Requalification Examinations]{.underline}. Assess the
facility licensee\'s examination materials \[questions, scenarios, and
job performance measures (JPMs) banks\], sample plans, and proposed and
completed examinations and tests, as described below. NUREG‑1021,
\"Operator Licensing Examination Standards for Power Reactors,\"
contains additional information that may be useful to the inspector in
conducting the evaluations. The inspector should use IMC 0609 Appendix
I, AOperator Requalification Human Performance SDP,@ to determine the
significance associated with the inspection finding as it relates to the
sample size. The inspector should not interfere with the facility
licensee\'s requalification examination process by suggesting
modifications to test items or examination schedules. If there are
significant concerns regarding the quality of the examinations, inform
the facility licensee and refer the concerns to regional management as
soon as possible.

The following activities facilitate this assessment:

> a\. Review a representative sample of the facility licensee\'s
> examination materials:
>
> 1\. The checklists for open reference written test items, JPMs, and
> dynamic simulator scenarios in Appendix A provide guidance.
>
> 2\. Compare plant changes to examination materials to determine
> whether system and procedure changes are being incorporated into the
> appropriate written questions, JPMs, and dynamic simulator scenarios.
> The resident inspectors, other knowledgeable personnel from the
> Division of Reactor Projects, and the NRR project manager may be able
> to provide information regarding substantial procedure or system
> modifications that should have been incorporated into the continuing
> training and testing programs. The 10 CFR 50.59 periodic reports also
> contain information on plant changes.
>
> 3\. For plants at which operators hold multi‑unit licenses, review the
> methodology for incorporating unit differences in the facility
> licensee\'s requalification examinations. Include both written
> examinations and operating tests using the simulator in this review.
> Review the extent to which unit differences are identified in training
> materials and the simulator as they are used in requalification
> training and examinations. Evaluate exceptions to training guidelines
> and simulator fidelity standards taken in the requalification program
> for negative training potential. Verify that operators receive
> specific training on unit differences.
>
> b\. Review the methodology (i.e., sample plan) that the facility
> licensee uses to construct its requalification examinations.
>
> 1\. Assess whether the facility licensee\'s comprehensive written
> examinations and annual operating tests point to areas in which
> retraining is needed \[10 CFR 55.59(c)(4)(i)\].
>
> \(a\) Determine if the facility licensee addressed the operator
> performance deficiencies identified under inspection requirement
> 02.03.
>
> \(b\) Determine if the facility licensee has incorporated current
> industry events applicable to the facility into training and testing,
> as appropriate.
>
> 2\. Assess whether the facility licensee\'s written examinations
> measure the operators\' knowledge of subjects covered in the
> requalification program and provide a basis for evaluating their
> knowledge of abnormal and emergency procedures \[10 CFR
> 55.59(c)(4)(ii)\].
>
> 3\. Determine if the operating tests are consistent with activities
> described in the updated final safety analysis report (UFSAR).
> Determine if operator response times specified in the accident
> analysis are evaluated during the operating test. Be careful about
> determining if the simulator scenario accurately matches the
> assumptions in the accident analysis. Operating test scenarios may
> include equipment malfunctions beyond those assumed in the accident
> analysis. In such a case, the operating test scenario may not be a
> valid measure of UFSAR operator response times.
>
> 4\. Determine if the licensee has incorporated probabilistic risk
> assessment (PRA) insights into the comprehensive written examinations
> and annual operating tests. Refer to NRC Manual Chapter 2515, Appendix
> A, ARisk-Informed Baseline Inspection Program.@ Coordinate with
> regional senior risk analysts (SRAs) to provide risk insights.
>
> c\. Evaluate the quality and content of a sample of the facility
> licensee\'s comprehensive written examinations and annual operating
> tests for the current requalification program cycle. Assess the
> ability of the examinations to identify operators who possesses a
> satisfactory level of safety-significant knowledge, skills, and
> abilities. Ensure the examination items are operationally valid. If
> the facility is not administering a written examination during the
> current inspection and if examinations were not reviewed during the
> previous inspection, review a sample of the examinations that were
> last given. The following activities facilitate this evaluation:
>
> 1\. Determine if the examinations are consistent with the sample plan,
> and verify that test item repeatability issues are addressed in
> accordance with NUREG-1021.
>
> 2\. Analyze and compare the comprehension level tested on selected
> written examinations and operating tests administered during the
> period under review with the comprehension level tested on other
> examinations administered or planned during that requalification
> cycle.
>
> 3\. Determine whether the expected performance standards are clear,
> objective, and relevant.
>
> 4\. Verify that the RO and SRO written examinations adequately sample
> the items stated in 10 CFR 55.41 and 10 CFR 55.43 and that the
> operating tests adequately sample the items listed in 10 CFR 55.45.

03.05 [Licensee Practices in Administering Requalification
Examinations]{.underline}. Observe examinations and test in progress and
interview personnel to assess the facility licensee=s effectiveness in
conducting written examinations and operating tests to ensure operator
mastery of the requalification training program content. The inspector
should use IMC 0609, Appendix I, AOperator Requalification Human
Performance SDP,@ to determine the significance associated with the
identified issues.

> a\. Observe as many examination activities as possible to assess the
> facility licensee\'s effectiveness in conducting written examinations
> and operating tests. Focus on those activities that give the greatest
> insight into the facility licensee\'s ability to evaluate its
> operators\' mastery of the training program content. Resident
> inspectors should periodically observe simulator training for licensed
> operators noting deficiencies and discrepancies in the training and
> assessing operator performance. Coordinate with the resident
> inspector(s) to ensure that all pertinent issues are understood and
> that actions and staffing levels in the simulator are consistent with
> normal control room practices. The following activities facilitate
> this assessment:
>
> 1\. Determine whether the examinations are conducted as planned and
> whether any errors in administration are detected and corrected for
> subsequent examinations.
>
> 2\. Determine whether the facility licensee\'s examination practices
> gave proper consideration to minimizing undue operator stress (e.g.,
> scheduling, timing of segments, security measures) and the potential
> for negative training (e.g., testing crew configuration different from
> operations).
>
> 3\. Assess the facility evaluators\' use of performance standards by
> grading selected written examination questions and operating tests in
> parallel and assessing discussions regarding crew and operator
> performance following the administration of the operating tests. If
> there are concerns regarding the facility licensee\'s grading
> practices, inform the facility licensee of the concerns and refer the
> concerns to regional management as soon as possible. The following
> activities facilitate this assessment:
>
> \(a\) Determine whether the performance standards are applied
> consistently and objectively.
>
> \(b\) Determine whether crew and operator performance errors made
> during simulator evaluations are detected and adequately addressed by
> the facility\'s evaluators.
>
> \(c\) Determine whether any errors made by individual operators during
> the walk‑through examinations are detected and adequately addressed by
> the facility\'s evaluators.
>
> \(d\) Determine whether the facility evaluators effectively identify
> individuals and crews requiring remediation, and appropriately
> indicate when removal from shift activities is warranted.
>
> \(e\) Determine whether post‑examination critiques of operators and
> crews are effective in pointing out strengths and weaknesses and if
> they accurately appraise the observed performance.
>
> 4\. Determine whether plant events are factored into the
> requalification training program based on the review of LERs and plant
> performance indicators completed under inspection requirement 02.03.
>
> 5\. Determine the licensee\'s use of industry experience in the
> requalification training program.
>
> 6\. Assess the facility licensee\'s Operations Department level of
> involvement in the requalification testing.
>
> b\. Interview an operator, an instructor, a training supervisor, and
> an evaluator regarding the facility\'s policies and practices for
> administering examinations. If the interviews result in conflicting
> information, additional interviews may be needed to clarify the
> differences. Refer to the suggested interview topics in Appendix B
> when conducting these and other interviews. These interviews assist
> the inspector in determining whether:
>
> 1\. The training staff understands the operating test performance
> standards and how they are to be implemented.
>
> 2\. Management guidance and expectations parallel the actual conduct
> of testing as it was observed.
>
> 3\. The operators understand the facility licensee\'s policies and
> practices and what is expected of them during the examinations.
>
> 4\. The operators are aware of their plant-referenced simulator
> discrepancies, if there are any, and the potential for negative
> training.
>
> 5\. The interviewees\' perception and knowledge of examination
> security are consistent with administrative procedures.
>
> c\. Review examination security measures to ensure compliance with 10
> CFR 55.49, which prohibits applicants and facility licensees from
> engaging in any activity that compromises the integrity of any
> application, test, or examination required by 10 CFR Part 55. The
> following activities facilitate this review:
>
> 1\. Review the facility licensee\'s process for maintaining
> examination security. Review facility guidelines on allowed overlap
> between examinations in current exam cycle tests and prior year
> examination.
>
> Good examination test practices strongly advise limiting test items
> repetition within and among comprehensive requalification exams that
> are taken by crews undergoing the same requalification training
> program cycle. The reason for limiting the repetition of identical
> test items lies in reducing the potential for examination content test
> leaks (passing of questions or answers to other test takers), possible
> among individuals during high stakes testing. Should any such test
> content leaks occur, the integrity of the exam can be compromised
> because its intended validity to discriminate for subsequent test
> takers would be diminished.
>
> When any utility comprehensive requalification examination repeats
> more than 50 percent of its test items from previously administered
> comprehensive requalification examinations between and among crews
> undergoing the same requalification training program, (i.e., 50
> percent or more of its test items have appeared in and are drawn from
> previous comprehensive requalification examinations during the same
> requalification training program cycle), the inspectors should apply
> the performance-based guidance described in Appendix D.
>
> 2\. Monitor the examination while it is being administered and review
> the results to determine if there is any indication of examination
> compromise.
>
> 3\. If examination security problems were noted in the past, determine
> what corrective action(s) have been taken to preclude recurrence.
>
> d\. Observe the activities of one or more operating crews in the
> control room and compare this performance with performance observed in
> the simulator on requalification examinations. Examples of activities
> to compare are performance of surveillances, supervisory oversight,
> command and control, communication practices, log keeping, crew
> assignments and responsibilities, staffing levels, shift turnover, and
> management presence. Coordinate this observation with the resident
> inspectors observations of control room activities.

03.06 [Licensee Feedback System]{.underline}. Evaluate the effectiveness
of the facility licensee=s process for revising and maintaining its
licensed operator continuing training program up to date, including the
use of feedback from plant events and industry experience information.

> a\. Evaluate whether the facility licensee\'s use of employee feedback
> from operators, instructors, and supervisors is effective. The
> following activities facilitate this evaluation:
>
> 1\. Determine who is responsible for obtaining employee feedback and
> compare that individual\'s understanding of the program goals to the
> management expectations for the program.
>
> 2\. Review and evaluate a representative sample of the employee
> comments to determine if the program\'s consideration of the comments,
> recommendations, and their implementation are appropriate. Determine
> if requalification program changes are backlogged and the cause for
> the backlog. Determine whether program changes are prioritized on the
> basis of safety. Compare these findings with management expectations.
>
> 3\. Interview facility personnel to determine whether they know of,
> use, and are satisfied with the system used to gather and implement
> feedback. Refer to paragraph 03.05.c for related guidance and to
> Appendix B for suggested interview topics.
>
> b\. If warranted by previous facility performance, review the facility
> quality assurance/quality control (QA/QC) oversight activities in
> accordance with 10 CFR Part 50 (Appendix B) and evaluate the
> licensee\'s ability to assess the effectiveness of its requalification
> program and to implement appropriate corrective actions.

03.07 [Licensee\'s Remedial Training Program]{.underline}. Verify the
adequacy and effectiveness of the remedial training conducted since the
last requalification examinations and the training planned for the
current examination cycle to ensure that it addresses weaknesses in
licensed operator or crew performance identified during training and
plant operations. The inspector should use IMC 0609 Appendix I,
AOperator Requalification Human Performance SDP,@ to determine the
significance associated with the identified issues.

> a\. Remedial training includes the additional training provided to
> operators to correct deficiencies that prevent them from successfully
> passing the requalification examination and the training provided to
> operators to correct generic or individual weaknesses observed during
> the previous requalification cycle examination. The following
> activities facilitate this review:
>
> 1\. Review examples of operator and crew performance weaknesses since
> the last inspection and determine whether the facility licensee
> identified their root causes and implemented appropriate corrective
> actions.
>
> 2\. Determine if the facility licensee confirms the effectiveness of
> its corrective actions at the completion of retraining with a suitable
> evaluation method.
>
> 3\. Review the remediation plans (e.g., lesson plans, reference
> materials, and attendance documentation) to assess the effectiveness
> of the remedial training.
>
> 4\. When possible, observe applicable simulator and JPM instruction to
> assess the effectiveness of the remedial training.
>
> 5\. Interview selected facility personnel to verify the effectiveness
> of remedial training. Refer to paragraph 03.05.c for related guidance
> and to Appendix B for suggested interview topics.

03.08 [Conformance With Operator License Conditions]{.underline}. Review
the facility licensee=s program for maintaining active operator licenses
and ensuring the medical fitness of its licensed operators. Sample the
following activities during alternate inspections to verify the facility
and individual licensees\' conformance with the requirements of 10 CFR
Part 55. In order to focus the review, the inspector is encouraged to
solicit observations and insights in this area from resident inspectors.

> a\. Review the facility licensee\'s program for maintaining active
> operator licenses and assess compliance with 10 CFR 55.53(e) and (f).
> The following activities facilitate this review:
>
> 1\. Sample records for at least one operating crew to determine if
> crew members are maintaining active licenses. Review records of
> licensed staff operators (i.e., those not assigned to shift crews) to
> ensure that their licenses have been activated before standing watch.
>
> 2\. Determine if any operator licenses were reactivated since the last
> inspection and verify that the operator\'s qualifications were current
> and the required operator functions were performed \"under
> direction.\"
>
> 3\. Determine if all requalification training is completed on schedule
> or made up in accordance with the facility\'s program. Sample training
> attendance records to include the end of the last 2-year
> requalification cycle.
>
> b\. Review the facility licensee\'s program for ensuring the medical
> fitness of its licensed operators and assess compliance with 10 CFR
> Part 55, Subpart C, \"Medical Requirements,\" and Subpart F,
> \"Licenses,\" item 55.53(i). The following activities facilitate this
> review:
>
> 1\. Review a representative sample (i.e., approximately 10 percent) of
> the licensed operators\' medical records to verify that the required
> physical examinations are being performed and documented.
>
> 2\. Verify that operator licensees are complying with special license
> conditions, as applicable, and that those operators who do not meet
> medical standards are precluded from performing licensed duties.

03.09 [Written Examination and Operating Test Results]{.underline}. At
the end of the annual operating testing cycle and the biennial written
testing cycles, review licensee requalification examination results.
Assess whether operator failure rates are consistent with the guidance
of the most recent version of NUREG 1021. The inspector should use IMC
0609 Appendix I, Operator Requalification Human Performance SDP, to
determine the significance associated with requalification examination
failure rates.

03.10 [Resident Inspector Quarterly Review of Licensed Operators=
Requalification Testing and/or Training Activities]{.underline}. At
least once each quarter, observe testing and/or training for SROs and
ROs, note deficiencies and discrepancies, and assess licensed operator
performance and evaluator=s critique. Emphasis should be placed on
observing testing and/or training on high risk licensed operator
actions, operators= activities associated with the Emergency Plan, and
previous lessons learned items or plant experiences. The inspector
should use IMC 0609 Appendix I, AOperator Requalification Human
Performance SDP,@ to determine the significance associated with the
inspection finding as it relates to the sample size.

> a\. Review simulator evaluations for previously identified weaknesses,
> and observe those areas during control room activities. Suggested
> observation areas are:
>
> ! Crew performance in terms of clarity and formality of communication
>
> ! Ability to take timely action in the safe direction
>
> ! Prioritizing, interpreting, and verifying alarms
>
> ! Correct use and implementation of procedures, including the alarm
> response procedures
>
> ! Timely control board operation and manipulation, including high-risk
> operator actions
>
> ! Oversight and direction provided by the shift supervisor, including
> ability to identify and implement appropriate technical specifications
> actions such as reporting and emergency plan actions and notifications
>
> ! Group dynamics involved in crew performance
>
> The inspector may observe different crews to gain an understanding of
> differences in personality, performance, and group dynamics involved.
> The inspector may factor this experience into daily observation of
> control room operation to draw conclusions on the effectiveness of
> simulator training. The inspector should discuss any concerns,
> findings, or insights with the applicable regional specialist.
>
> b\. Review simulator physical fidelity (i.e., the degree of similarity
> between the simulator and the reference plant control room, such as
> physical location of panels, equipment, instruments, controls, labels,
> and related form and function), especially regarding recent
> modifications implemented in the control room. Per 10 CFR 55.59
> (c)(4)(iv), the arrangement of the instrumentation and controls of the
> simulator must closely parallel that of the facility involved.
> Simulation scope and fidelity must be sufficient to allow conduct of
> evolutions in 10 CFR 55.45 and 55.59.
>
> If any licensed operator requalification issue identified during the
> inspection is related to simulator fidelity, refer to IMC 0609,
> Appendix I, \"Operator Requalification Human Performance SDP\" for
> determining the significance of the issue.

03.11 [Conformance With Simulator Requirements Specified in 10 CFR
55.46]{.underline}. Determine if the facility licensee\'s simulation
facility is acceptable for use in operator licensing examinations and
for satisfying experience requirements prescribed in 10 CFR 55.46.

This assessment is to be carried out on a sampling basis and is not
intended to be an exhaustive inspection of the licensee\'s simulation
facility. Verify that the facility licensee has a process in place to
maintain the fidelity of the plant-referenced simulator. Also assess the
safety impact of any negative training caused by simulator
discrepancies.

> a\. Ensure discrepancies noted during the inspection, particularly
> while observing the dynamic simulator operating tests, have been
> entered into a licensee\'s corrective action program.
>
> b\. Review a sample of significant simulator discrepancy reports,
> including simulator modeling and hardware discrepancies and
> discrepancies identified from scenario validation and performance
> testing, to assess the effectiveness of the licensee\'s process for:
> problem identification and prioritization, reporting, root cause
> evaluation, schedule for implementing timely corrective actions, and
> corrective actions (where simulator discrepancies could result in
> negative training, corrective actions should include training on
> actual plant behavior). In summary, verify that the licensee
> adequately captures simulator problems and that corrective actions are
> performed, tracked, trended and completed in a timely fashion
> commensurate with the safety significance of the item.
>
> c\. Refer to 10 CFR 55.46, \"Simulation facility,\" for specific
> requirements regarding plant-referenced simulators. Use Appendix C of
> this inspection procedure, \"Checklist for Evaluating Plant-referenced
> Simulator Operating Under 10 CFR 55.46(c) and (d),\" to determine, on
> a sampling basis, if the plant-referenced simulator is acceptable for
> use in operator examinations, and to satisfy experience requirements.
> Assess any negative responses to the questions and if necessary, seek
> headquarters guidance on the issue. With regard to Question No. 5 of
> the checklist, a brief inquiry of the licensee on how the
> plant-referenced simulator utilizes models relating to nuclear and
> thermal-hydraulic characteristics that replicate the most recent core
> load (i.e., the current core or, if the plant is in a refueling
> outage, the core just previous to the outage) is sufficient unless an
> issue is identified. With regard to Question No. 10 of the checklist,
> ensure that any discrepancy written against uncorrected performance
> test failures has been entered into a licensee\'s simulator corrective
> action program.

71111.11-04 RESOURCE ESTIMATE

It is estimated that approximately 96 hours, on average, of direct
inspection effort (DIE) will be required to conduct the biennial review.
The effort includes a regional specialist=s in-office review of tests.
It is expected that the actual hours required to complete the inspection
may vary from the estimate. The inspection hours allocated for the
inspection procedure are budget estimates for the typical plant
regardless of the number of units at the site. The hours expended during
an inspection should be tailored for the facility licensee and
accurately recorded. Depending on availability, resident staff members
may assist the regional specialist during the biennial review. An
additional 4 hours a quarter is estimated for the resident staff to
review licensed operator requalification activities.

71111.11.05 REFERENCE

IMC 0609, Appendix I, AOperator Requalification Human Performance
Significance Determination Process@

END

Appendices:

A. CHECKLIST FOR EVALUATING FACILITY TESTING MATERIAL

B. SUGGESTED INTERVIEW TOPICS

C. CHECKLIST FOR EVALUATING PLANT-REFERENCED SIMULATORS OPERATING UNDER
10 CFR 55.46(c) AND (d)

D. GUIDANCE FOR EXCESSIVE TEST ITEM REPETITION AND POTENTIAL EXAMINATION
COMPROMISE

Attachment:

Revision History

APPENDIX A

CHECKLIST FOR EVALUATING FACILITY TESTING MATERIAL

(Circle yes \[Y\] or no \[N\])

[Written Examination Questions Checklist]{.underline}

Y / N 1. Does each question have a documented link to important licensee
tasks, knowledge and abilities (K/As), and/or facility learning
objectives?

Y / N 2. Is each question operationally oriented (i.e., is there a
correlation between job demands and test demands)?

Y / N 3. Is each question written at the appropriate level of knowledge
(fundamental knowledge, comprehension, or application/analysis)? Refer
to Appendix B, AWritten Examination Guidelines,@ of NUREG-1021,
AOperator Licensing Examinations for Power Reactors,@ for guidance.

Y / N 4. Is the context of each question realistic and free of window
dressing and backwards logic?

Y / N 5. Does each question possess a high K/A importance factor (3 or
greater) for the job position?

Y / N 6. Does each question appear to have the ability to discriminate
between an operator who possesses a satisfactory level of safety
significant knowledge and an operator who does not?

Y / N 7. Is each question appropriate for the written examination and
the selected written examination format (e.g., short answer; multiple
choice)?

Y / N 8. Does any question have the potential of being a \"double
jeopardy\" question?

Y / N 9. Is each question clear, precise, and easy to read and
understand?

Y / N 10. Is there only one correct answer to each question?

Y / N 11. Does any question pose situations and problems that differ
from those presented during training?

[Additionally, For Open-Reference Questions]{.underline}

Y / N 1. Does each question require an appropriate use of references
(i.e., use of analysis skills or synthesis of information either to
discern what procedures were applicable or to consult the procedures to
obtain the answer)?

Y / N 2. Is any question a \"direct look-up\" question (i.e., one that
immediately directs an operator to a particular reference where the
answer is readily available)?

Y / N 3. Are there any questions given in a static scenario setup that
takes advantage of the simulator control room setting?

[Job Performance Measure (JPM) Quality Checklist]{.underline}

Y / N 1. Is each task supported by the facility\'s job task analysis?

Y / N 2. Is each task operationally important (i.e., meets threshold
criterion of K/A at 3 or above or as determined by the facility)?

Y / N 3. Is each task designed as either SRO only, RO/SRO, or AO/RO/SRO?

> 4\. Does each JPM include: (Refer to Appendix C, AJob Performance
> Measures Guidelines,@ of NUREG-1021, AOperator Licensing Examinations
> for Power Reactors,@ for guidance.)

Y / N a. Initial conditions

Y / N b. Initiating cues

Y / N c. References, including associated procedures

Y / N d. Performance standards that are specific in that exact control
and indication nomenclature and criteria (switch position, meter
reading) are specified, even if these criteria are not specified in the
procedural step

Y / N e. System response cues in the performance standards that are
complete and correct so that the examiner can properly cue the operator,
if asked

Y / N f. Statements describing important actions or observations that
should be made by the operator

Y / N g. Criteria for successful completion

Y / N h. Identification of the critical steps and their associated
performance standards

Y / N i. Validated time limits (average time allowed for completion)

Y / N j. JPMs identified as time critical or not time critical

Y / N k. Restrictions on the sequence of steps

[Simulator Scenario Review Checklist]{.underline}

[Qualitative Attributes]{.underline}

Y / N 1. Is each scenario of the appropriate scope, depth, and
complexity with clearly stated objectives? (Refer to Appendix D,
ASimulator Testing Guidelines,@ of NUREG-1021, AOperator Licensing
Examinations for Power Reactors,@ for guidance.)

Y / N 2. Are the initial conditions realistic?

Y / N 3. Does each scenario consist mostly of related events?

4\. Does each scenario event description include:

Y / N a. The point in the scenario when it is to be initiated?

Y / N b. The malfunction(s) that are entered to initiate the event?

Y / N c. The symptoms/cues that will be visible to the crew?

Y / N d. The expected operator actions (by shift position)?

Y / N e. The event termination point?

Y / N 5. Is no more than one non-mechanistic failure (e.g., pipe break)
incorporated into each scenario without a credible preceding incident
such as a seismic event?

Y / N 6. Is each event valid with regard to physics and thermodynamics?

Y / N 7. Is the sequencing/timing of each event reasonable, and does it
allow for the examination team to obtain complete evaluation results
commensurate with the scenario objectives?

Y / N 8. Has the simulator modeling been altered?

Y / N 9. Can each rating factor in each crew competency be evaluated?

Y / N 10. Has each scenario been validated?

Y / N 11. If the sampling plan indicates that the scenario was used for
training during the requalification cycle, has the facility determined
whether it should be modified or not used?

The following criteria list scenario traits that are numerical in
nature. A second set of numbers indicates a range to be met for a set of
two scenarios. Therefore, to complete this part of the review, the set
of scenarios must be available.

[Quantitative Attributes]{.underline}

Y / N 1. Total malfunctions inserted: 4 to 8 / 10 to 14

Y / N 2. Malfunctions that occur after EOP entry: 1 to 4 / 3 to 6

Y / N 3. Abnormal events: 1 to 2 / 2 to 3

Y / N 4. Major transients: 1 to 2 / 2 to 3

Y / N 5. EOPs used beyond primary scram response EOP: 1 to 3 / 3 to 5

Y / N 6. EOP contingency procedures used: 0 to 3 / 1 to 3

Y / N 7. Approximate scenario run time: 45 to 60 minutes (one scenario
may approach 90 minutes)

Y / N 8. Crew critical tasks: 2 to 5 / 5 to 8

Y / N 9. Are Technical Specifications exercised during the test?

COMMENTS:

APPENDIX B - SUGGESTED INTERVIEW TOPICS

+-----------+----------------------------------------------------------+
| Activity  | Suggested Interview Topics/Questions                     |
+-----------+----------------------------------------------------------+
| General   | Former positions at the facility: How long? Licensed?    |
+-----------+----------------------------------------------------------+
|           | Current position and duties: How long? Licensed?         |
|           | Requalification program responsibilities?                |
+-----------+----------------------------------------------------------+
| Exams,    | Examinations: How developed? Sampling plan? Appropriate  |
|           | coverage? License level? Practiced/covered in training?  |
| pe        | Duplicate quizzes? Too easy/hard? Too long/short? Were   |
| rformance | references necessary? How compare with NRC exams?        |
| s         |                                                          |
| tandards, |                                                          |
|           |                                                          |
| s         |                                                          |
| imulator, |                                                          |
| and       |                                                          |
| security  |                                                          |
+-----------+----------------------------------------------------------+
|           | Performance standards: How are they formulated?          |
|           | Operations versus training? Are they endorsed by         |
|           | management? Are they objective? How are they             |
|           | communicated to evaluators? Do the operators know what   |
|           | is expected of them? Are they applied consistently?      |
+-----------+----------------------------------------------------------+
|           | Performance feedback: Is it timely? Is it objective?     |
|           | What happens if you fail? How could feedback be          |
|           | improved?                                                |
+-----------+----------------------------------------------------------+
|           | Administration: Operating/training crew = test crew?     |
|           | What happens if you miss an exam? Measures to mitigate   |
|           | undue stress?                                            |
+-----------+----------------------------------------------------------+
|           | Simulator: Does the simulator correctly demonstrate      |
|           | expected plant response to normal, transient, and        |
|           | accident conditions? Any negative training? Are operator |
|           | interfaces in the simulator current with the actual      |
|           | control room?                                            |
+-----------+----------------------------------------------------------+
|           | Security: Are exams common? How is security ensured? Are |
|           | there formal procedures? Responsibility? Do you feel     |
|           | comfortable with process? Do security measures cause     |
|           | undue stress? Are you aware of any incidents? What would |
|           | you change if you could?                                 |
+-----------+----------------------------------------------------------+
| Feedback  | Feedback collection: How is it done? Who collects        |
| system    | comments? Who is solicited? Does the QA/QC Department    |
|           | oversee the program?                                     |
+-----------+----------------------------------------------------------+
|           | Comment resolution: Who does it? Is it timely? Safety    |
|           | basis for changes? How is management involved? How are   |
|           | changes promulgated? Were they resolved to your          |
|           | satisfaction? Feedback to originator? Recent examples?   |
+-----------+----------------------------------------------------------+
|           | Overall, how effective is your training program? The     |
|           | examination program? The feedback system? How would you  |
|           | improve it?                                              |
+-----------+----------------------------------------------------------+
| Remedial  | Program development: How are remedial training needs     |
| training  | identified? Individual/crew exam results? On-the-job     |
| program   | performance/events? Generic weaknesses? Who develops     |
|           | remedial training programs? How is Operations involved?  |
+-----------+----------------------------------------------------------+
|           | Implementation: Is it appropriate? Is it effective? How  |
|           | is remediation verified? How would you improve it?       |
+-----------+----------------------------------------------------------+

APPENDIX C

CHECKLIST FOR EVALUATING PLANT-REFERENCED SIMULATORS

OPERATING UNDER 10 CFR 55.46(c) AND (d)

(Circle yes \[Y\] or no \[N\]. Answer questions based upon sampling
inspection. It is not intended that these questions be answered on the
basis of exhaustive inspection.)

Y / N 1. If the plant-referenced simulator is used for the
administration of NRC reactor operator and senior operator operating
test, does the plant-referenced simulator demonstrate expected plant
response to operator input and to normal, transient, and accident
conditions to which the simulator has been designed to respond?
\[\'55.31(a)(5) and \'55.46(c)(1)\]

Y / N 2. Is the plant-referenced simulator sufficient in scope and
fidelity with the reference plant to allow conduct of the evolutions
listed in 10 CFR 55.45(a)(1) through (13),as applicable to the reference
plant? \[\'55.46(c)(1)(i)\]

Y / N 3. Is the plant-referenced simulator sufficient in scope and
fidelity with the reference plant to allow conduct of the evolutions
listed in 10 CFR 55.59(c)(3)(i)(A) through (AA), as applicable to the
reference plant? \[\'55.46(c)(1)(i)\]

Y / N 4. Is the plant-referenced simulator designed and implemented in a
manner that allows for the completion of control manipulations for
operator license applicants? \[\'55.46(c)(1)(ii)\]

Y / N 5. If the plant-referenced simulator is used to meet experience
requirements for applicants for operator and senior operator licenses,
does the plant-referenced simulator utilize models relating to nuclear
and thermal-hydraulic characteristics that replicate the most recent
core load in the nuclear power reference plant for which a license is
being sought? The phrase \"most recent\" means the current core or if
the reference plant is in a refueling outage, the core just previous to
the outage. \[\'55.31(a)(5)\], \[\'55.46(c)(1), and \'55.46(c)(2)(i)\]

Y / N 6. Has the plant-referenced simulator fidelity been demonstrated
so that significant control manipulations are completed without
procedure exceptions, simulator performance exceptions, or deviation
from the approved training scenario sequence? \[\'55.46(c)(2)(ii)\]

Y / N 7. Has there been any lapse in the facility licensee conducted
simulator performance testing throughout the life of the simulation
facility? \[\'55.46(d)(1)\]

Y / N 8. Are the results of performance testing retained for four years
after the completion of each performance test or until superseded by
updated test results? \[\'55.46(d)(1)\]

Y / N 9. Are modeling and hardware discrepancies and discrepancies
identified from scenario validation and from performance testing being
corrected? \[\'55.46(d)(2)\]

Y / N 10. Are results of any uncorrected performance test failures that
may exist at the time of the operating test or requalification program
inspection available for NRC review? \[\'55.46(d)(3)\]

Y / N 11. Has simulator fidelity been maintained such that license
application, examination, and test integrity are consistent with 10 CFR
55.49 requirements? \[\'55.46(d)(4)\]

**APPENDIX D**

**GUIDANCE FOR EXCESSIVE TEST ITEM REPETITION AND POTENTIAL FOR
EXAMINATION COMPROMISE**

General Guidance

1\. When examinations repeat \> 50 percent of previous requalification
training program examination test items, discrimination validity may be
reduced and examination compromise may have occurred. Inspectors should
examine the requalification examination for each crew mean (average)
scores to determine whether scores show any **pattern of mean score**
**rise** over successive crew examination administrations *OR* show any
unexplained higher-than-expected crew mean score(s). Given the premise
that crews are approximately equivalent in their overall collective
abilities, mean scores on examinations should not vary by more than
several points in either direction between and among crews. Although
some score variations among crews are expected, the variations are
*less* informative than an observed, general trend-rise in scores or any
unexplained higher- than- expected crew(s) score rise. Such score rises
or trend rises may indicate that examination content leaks occurred
among crews.

2\. If there are no observed, discernable trend rises in overall mean
scores, nor any unexplained, higher-than-expected crew scores (exceeding
5 or more points from any previous examination),
[discontinue]{.underline} any further inspection in this area.

3\. If there are observed, discernable trend rises in crew mean scores,
or any unexplained, higher-than-expected crew score (exceeding 5 or more
points from any previous examination), examine the effect of the
*repeated test items* using the following assumptions and methodology:

Assumptions:

1\. Criterion-referenced examinations, such as the requalification exam,
are based upon the operators= recent training program. Thus, individual
exam scores are normally expected to be high, e.g., scores greater than
85 percent and would normally be clustered together within a relatively
narrow upper range.

2 All crews are essentially equivalent in ability.

3\. While some score variation is expected, all crew mean scores should
cluster approximately within a narrow range, e.g., +/- 3-5 points.

4\. A greater than five (5) percent deviation among any crew=s Arepeated
item mean scores@ in light of assumptions of equivalence, comparable
ability, and having undergone the same training program would be an
unexpected occurrence.

5\. The 5 percent threshold is a commonly used statistical threshold
when setting probability levels. Conservatively, it is being applied
here as a percentage of expected score tolerance and variation among
crews whose performance differences should normally be small.

**[Methodology]{.underline}** (to be used only when examinations repeat
more than 50 percent of previous requalification training program
examination test items and a discernable trend or higher than expected
mean scores has been observed).

1\. For all examinations occurring after the first examination
(benchmark exam), identify the repeated items.

2\. For all exams that followed the first exam, calculate each
individual crew mean scores for repeated items only. This is referred to
as Arepeated items mean score.@

3\. Assess each crew=s Arepeated items mean scores@ sequentially over
time. Other than the scores attained by the first test crew in the
sequence, examine any trend rises in Arepeated item mean scores@ OR
examine any higher than expected individual crew rise in Arepeated item
mean scores.@

**Decision Point**

Do any of the individual crew=s Arepeated item mean scores@ exceed 5
points higher than either the benchmark exam or any preceding
examinations in that requalification program cycle?

If not, discontinue any further inspection in this area.

If yes, query the utility about why such results would occur? Explore
the assumption of crew ability equivalence. Explore with utility
personnel the possibility of examination content leaks between
examination groups; discuss any potential examination compromise (10CFR
55.49). Discuss/identify any lax security procedures or need to
establish security statements or other such procedures. See related
Significant Determination Process (SDP) reference below.

Reference: Operator Requalification Human Performance SDP (Step 14):
AKnowledge of an exam integrity compromise can occur through various
means, \... (2) an analysis of operator post exam results, suspected to
have been compromised, reveals that the exam results attained are not
probable or likely given the history of the operator=s past
performance.@

[Retake examinations]{.underline}

In principle and practice, excessive test item repetition also applies
to the retake exam; that is to say, retake exams should not repeat more
than 50 percent of the items from any previously administered
requalification examination during the current requalification program,
but more importantly, should not contain any of the same items (missed
or correct) from the earlier failed examination.

Inspectors should ensure that any test items that appeared on the
original failed examination are not included as a part of the retake
examination. Reusing the same items (missed or correct) from the
original failed test on the retake examination is a flawed practice that
would falsely bias the test results upward, inflating and distorting
true retake performance. Moreover, including any of the same items on
the retake test amounts to little more than a review B not a test as is
operationally defined.

> (Note, for training purposes, it is desirable for applicant and
> utility to **review** specific exam items missed from the failed
> examination so as to remove knowledge deficiencies, but it is never
> good practice to include those same items in a retake exam because the
> same items would have no discriminatory valueBan essential component
> of test validity\-- due to their recent exposure.

END

ATTACHMENT

Revision History - IP 71111.11

+--------+--------+------------------------------+-----+-------+-------+
| Comm   | Issue  | Description of Change        | Tr  | Tra   | Co    |
| itment | Date   |                              | ain | ining | mment |
| Tr     |        |                              | ing | Compl | Resol |
| acking |        |                              | Nee | etion | ution |
| Number |        |                              | ded | Date  | Acce  |
|        |        |                              |     |       | ssion |
|        |        |                              |     |       | N     |
|        |        |                              |     |       | umber |
+--------+--------+------------------------------+-----+-------+-------+
| N/A    | 08/1   | Revised to clarify the       | NO  | N/A   | N/A   |
|        | 6/2001 | original intent of the       |     |       |       |
|        |        | procedure as it relates to   |     |       |       |
|        | CN     | sample size selection.       |     |       |       |
|        | 01-015 |                              |     |       |       |
+--------+--------+------------------------------+-----+-------+-------+
| N/A    | 08/2   | Revised to reflect the       | NO  | N/A   | N/A   |
|        | 0/2002 | amended 10 CFR Part 55,      |     |       |       |
|        |        | \"Operators\' Licenses,\"    |     |       |       |
|        | CN     | regarding operator license   |     |       |       |
|        | 02-031 | eligibility and the use of   |     |       |       |
|        |        | simulation facilities in     |     |       |       |
|        |        | operator licensing (66 FRN   |     |       |       |
|        |        | 52657, dated October 17,     |     |       |       |
|        |        | 2001). This revision         |     |       |       |
|        |        | provides specific guidance   |     |       |       |
|        |        | to inspector when assessing  |     |       |       |
|        |        | conformance with simulator   |     |       |       |
|        |        | requirements specified in 10 |     |       |       |
|        |        | CFR 55.46.                   |     |       |       |
+--------+--------+------------------------------+-----+-------+-------+
| N/A    | 12/1   | Revised to include an        | NO  | N/A   | N/A   |
|        | 6/2003 | additional section that      |     |       |       |
|        |        | inspects excessive test item |     |       |       |
|        | CN     | repetition among             |     |       |       |
|        | 03-041 | comprehensive                |     |       |       |
|        |        | requalification exams that   |     |       |       |
|        |        | are taken by crews           |     |       |       |
|        |        | undergoing the same training |     |       |       |
|        |        | program cycle. Excessive     |     |       |       |
|        |        | item repetition adversely    |     |       |       |
|        |        | affects validity of the      |     |       |       |
|        |        | exam. clarify the original   |     |       |       |
|        |        | intent of the procedure as   |     |       |       |
|        |        | it relates to sample size    |     |       |       |
|        |        | selection.                   |     |       |       |
+--------+--------+------------------------------+-----+-------+-------+
| N/A    | 01/0   | Inspection resource was      | NO  | N/A   | N/A   |
|        | 5/2006 | increased to 4 hrs/quarter   |     |       |       |
|        |        | (net increase of 4           |     |       |       |
|        | CN     | hours/year) to more          |     |       |       |
|        | 06-001 | accurately reflect the time  |     |       |       |
|        |        | spent by resident inspectors |     |       |       |
|        |        | during their quarterly       |     |       |       |
|        |        | observation of operator      |     |       |       |
|        |        | requalification activities.  |     |       |       |
|        |        | Completed historical CN      |     |       |       |
|        |        | search.                      |     |       |       |
+--------+--------+------------------------------+-----+-------+-------+

Issue Date: 01/05/06 E-1 71111.11