NUREG-1220 

Rev. 1 

Training Review Criteria 

and Procedures 

U.S. Nuclear Regulatory Commission 

Office of Nuclear Reactor Regulation 

1 

AVAILABILITY NOTICE 

Availability of Reference Materials Cited in NRC Publications 

Most documents cited in NRC publications will be available from one of the following 

sources: 

1. The NRC Public Document Room. 2120 L Street, NW., Lower Level. Washington. DC 

20555 

The Superintendent of Documents. U.S. Government Printing Office, P.O. Box 37082. 

Washington. DC 20013-7082 

2. 

3. The National Technical Information Service, Springfield. VA 22161 

Although the listing that follows represents the majority of documents cited in NRC publications, it is not intended to be exhaustive. 

Referenced documents available for inspection and copying for a fee from the NRC Public 

Document Room include NRC correspondence and internal NRC memoranda: NRC bulletins, 

circulars, information notices, inspection and investigation notices: licensee event reports; 

vendor reports and correspondence: Commission papers: and applicant and licensee documents and correspondence. 

The following documents in the NUREG series are available for purchase from the GPO Sales 

Program: formal NRC staff and contractor reports, NRC-sponsored conference proceedings. international agreement reports, grant publications, and NRC booklets and brochures. 

Also available are regulatory guides. NRC regulations in the Code of Federal Regulefions. 

and Nuclear Regulatory Commission Issuances. 

Documents available from the National Technical Information Service include NUREG-series 

reports and technical reports prepared by other Federal agencies and reports prepared by 

the Atomic Energy Commission, forerunner agency to the Nuclear Regulatory Commission. 

Documents available from public and special technical libraries include all open literature 

items, such as books, journal articles, and transactions. Federal Register notices, Federal 

and State legislation. and congressional reports can usually be obtained from these 

libraries. 

Documents such as theses. dissertations, foreign reports and translations, and non-NRC 

conference proceedings are available for purchase from the organization sponsoring the 

publication cited. 

Single copies of NRC draft reports are available free, to the extent of supply. upon written 

request to the Office of Administration, Distribution and Mail Services Section. U.S. Nuclear 

RegUlatOry Commission, Washington, DC 20555. 

Copies of industry codes and standards used in a substantive manner in the NRC regulatory 

process are maintained at the NRC Library, 7920 Norfolk Avenue, Bethesda. Maryland, for 

use by the public. Codes and standards are usually copyrighted and may be purchased 

from the originating organization or, if they are American National Standards. from the 

American National Standards Institute. 1430 Broadway, New York. NY 10018. 
NUREG-1220 

Rev. 1 

Training Review Criteria 

and Procedures 

Manuscript Completed January 1993 

Date Published: January 1993 

Division of Licensee Performance and Quality Evaluation 

Ofice of Nuclear Reactor Regulation 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555 . 
ABSTRACT 

This document provides direction to NRC personnel for 

reviewing training programs at nuclear power plants to 

veri@ compliance with the requirements of 10 CFR 

50.120 and 10 CFR 55 as applicable. It describes the 

pr0c.e~~ for evaluating the effectiveness of training 

programs, provides aidsfor collection of information during interviews and obsenations, and provides criteria for 

evaluating the implementation of a systems approach to 

training. This document is not intended to have the effect 

of a regulation, it establishes no binding requirements or 

interpretations of NRC regulations. It is intended as guidance only. 

... 

UL NUREG-1220, Rev. 1 

CONTENTS 

ABmm ................................................................................ 

INlXODUCIION TO THE TRAINJNG EVALUATION PROCESS ............................... 

1 TRAINING INSPECIION TRIGGERS ..................................................... 

2 PLANNING A TRAINING INSPECTION ................................................... 

Form 2.1-Worksheet for the Team Leader Planning a Training Inspection ....................... 

3 ON-SlTE INSPECTION ACTIVITE.5 ...................................................... 

k General Team Protocol ................................................................ 

B . Observation Checklists ................................................................ 

Form 3B-l-CLassroom Checklist ....................................................... Form 3B-2-Simulator Checklist ....................................................... Form 3B-3-On-the-Job Trainiig Checklist .............................................. Form 3B-4-Wo~k-in-Pr0g~~ Checklist ................................................. 

C . InterviewProtocols ................................................................... 

Fom 3C.l-Inst1~ctor ................................................................ Form 3C-2-Job Incumbent ........................................................... Form 3C-3-Plant Supervisor .......................................................... , Form 3C-4--Training Supervisor ....................................................... 

D . SAT Element Evaluation .............................................................. 

SAT Element 1-Analysis ............................................................. Form 3D-1-Element 1-Summary Rating Sheet ......................................... SAT Element 2-Leaming Objectives ................................................... Form 3D-2-Element 2-S~mmary Rating Sheet ......................................... SAT Element 3-Desi gn and Implementation ........................... .................. Form 3D-3-Element 3-Summary Rating Sheet .................. i ...................... SAT Element 4-Trainee Evaluation .................................................... Form 3D-4-Element 4-Summary Rating Sheet ......................................... SAT Element 5-prOgram Evaluation and Revision ........................................ Form 3D-5-Element 5-Summary Rating Sheet ......................................... 

4 REPORT PREPARATION ................................................................ 

APPENDIX . Forms .......................................................................... 

FIGURES 

Training Evaluation Process ................................................................... 

SATPresentationTree ........................................................................ 

page ... 

Ill 

1 

3 

5 

6 

9 

11 

13 

14 

16 

18 

20 

23 

24 

26 

29 

32 

35 

31 

43 

45 

49 

51 

51 

59 

66 

61 

14 

15 

A-1 

2 

16 

V NUREG.1220. Rev . 1 

INTRODUCTION TO TffE TRAINING EVALUATION PROCESS 

This document is designed to be used by NRC personnel 

who review training programs at nuclear power plants in 

order to verify compliance with the requirements of 10 

CFR 50.120 and 10 CFR 55 as applicable. 

Inspections todetermine the effectivenessof Wig and 

qualification programs are designed to focus attention on 

how personnel are performjng. By focusing on performance, the primary role of the inspector changes from 

reviewing documentation to observing work in progress. 

By observing work as it is performed, inspectors can identify areas of weak performance that may indicate that 

training and qualification programs have heen ineffective. 

Since training is not the only factor affecting human performance, inspectors must also take into account such 

information as worker aptitude and experience, as well as 

company policies and procedures that may be influencing 

job performance. 

Observations of actual work in progress in the plant providegenemlinformationontheworkpracticesusedat the 

plant as well as specific information on how tasks are 

assigned and completed. Training observatious (using the 

rating sheets provided) give an up-to-date picture of plant 

activities as interpreted by the training staff. 

Interviews (using the interview protocols provided) are a 

cation effectiveness. Talking with trainees and job incumbents and their supervisors gives the inspector direct information on the quality, quantity, and effectiveness of 

training. Data from interviews can indicate areas toconcentrate on while performing observations and, if appropriate, checking documentation. Interviews are one 

means of determining to what degree plant personnel feel 

they can have an impact on the content and qualily of 

training. Interviews also help the inspector determine 

how satisfied workers are with the traininggiven them. 

Performance problems identired during the interviews, 

training sessions, and work in progress provide direction 

for subsequent reviews of training materials and related 

natural Starting point for inspecting training and qW1- 

documentation and help to diagnose the reasons for reduced training effectiveness. 

Asuccesdully implemented systems approach to training 

(SAT)processwiIl have achieved the following objectives: 

Element 1-Analysis 

Plant and training staff use a systematic analysis to identify the knowledge and skills for presentation in training 

in order to meet job performance requirements. 

Element 2-Learning Objectives 

Learning objectives support succesFful job performance 

by reflecting analysis results and by providing conditions 

and standards for job performance behaviors (actions) 

expected of trainees upon completion of training. 

Element 3-Design and Implementation 

Content of trainiig programs is derived from learning 

objectives and is sequenced and presented for effective 

learning. 

Element 4-Trainee Evaluation 

Trainees are evaluated during all aspects of training to 

determine their progress toward mastery of job performance requirements 

Element S-F’rogram Evaluation and 

Plant and training staff use a systematic process to evaluate the effectiveness of the training and qualification programs and to determine and &ea the needed revisions. 

The flowchart that follows gives an overview of the process detailed in this document and identifies where in this 

document various aspects are addressed. 

Revision 

1 NUREG-1220, Rev. 1 
Training Evaluation Process 

Possible Inspection Triggers 

(section 1) 

Information (Section 2) 

orrsitekctivities 

(Section 3) 

I 

CAdUct condud 1 

(Section 3c) 

Observations Interviews 

(Section 38) I I t 

Review Needed 

Based on Observed 

Mplete Applicable 

SAT Element Evaluation(s) 

t 

Write Report 

(Sedion 4) 

NUREG-1220. Rev. 1 2 
1 TRAINING INSPECTION TRIGGERS 

Training inspeaions are the responsibility of the Office of 

Nuclear Reactor Regulation (NRR) andare conducted by 

regional office-led teams in accordance with Inspection 

Procedure (IF’) 41500. ‘Training and Quali€ication Effectiveness.” Training inspeaions are conducted at the &- 

cretion of the regioual administra tom with assistance 

from NRR training specialists when cause is identified. 

Traiuing inspections can be initiated by the region for 

several reaso~ls including but not limited to (1) an indication of declining human performance that appears to 

have a training-related cause, (2) an unsatisfactory reqWcation propram, (3) withdrawal of training program 

accreditation by the National Nuclear Armediting Board, 

and (4) a requesf from the facility licensee. 

3 NUREG-1220, Rev. 1 

2 PLANNING AND PREPARING FOR A TRAINING INSPECTION 

An inspection plan should be developed as gnidance for 

NRC inspectors when inspecting the effectiveness of 

training and qualifcationprognuns. Aplanningoutlineis 

included here for the Team Leader’s use; it emphasizes 

the specifics associated with a training impextion. 

Team members should prepare for a training inspection 

by reviewing the information provided by the licensee. 

Preparation should include such things as reviewing the 

licensee’s administrative policy on procedure use so that 

-- inaumuhte ._ . oerfonnance can be identified durine obThe worksheet shown in this section is included in the 

insoedion information on a cow of the master of this 

servations. Ad;litionally, inspectors may want to d&op 

of forms pm,,ided ~th document. Record interviewquestionson technid topicstoverifythedepth 

of t&d knowledge pcssesed by individual workers. .. woheet. 

5 NUREG-12U). Rev. 1 
Form 2.1 

Worksheet for the Team Leader 

Planning a Training Inspection 

Six Weeks Before the Inspeetion (or as soon as an inspection has been initiated) 

Working with the Regional Office: 

- 1. 

- 2. 

-3. 

Contact the Human Factors Assessment Branch of NRR to obtain information 

about staff positions on current indushy trends and other issues related to training. 

Determine the specific date(s) of the inspection. 

. Determine which of the foUowing training programs will be inspected 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Senior reactor operator/shift supervisor 

Reactor operator 

Shift technical advisor 

Non-licensed operatoi' 

Instrument and control technician 

Electrical maintenance personnel 

Mechanical maintenance personnel 

Radiological protection technician 

Chemistry technician 

Engineering support personnel 

-4. Determine size and composition of inspection team. 

Five Weeks Before the Inspection 

- Send a letter to the licensee announcing the planned inspection. The licensee should receive this 

letter no later than four weeks preceding the inspection. The letter should contain the following 

information: 

1. Date(s) of the inspection 

2. 

3. 

Training programs to be inspected 

Documents needed by the Inspection Team before it arrives on site: 

0 

0 

Task lists for the position(s) being reviewed 

Administrative instructions and procedures used by the Training Department for the 

development, presentation, and revision of materials and operation of the Training 

Department. 

Audits or evaluations of the training organization 

Organization chart of the Training Department and the facility 

Schedule for requalification and initial training programs for the next three months 

Course outline or lesson plans being presented during the planned inspection 

Shift schedule, including names, for the inspection week 

Training self-assessment, if done within the previous 6 months 

0 

0 

0 

0 

0 Selected station procedures 

0 

- Determine assignments for team members 

-Develop a schedule for the inspection week 

NUREG-1220, Rev. 1 6 
Worksheet for Planning an Inspection (continued) 

Four Weeks Before the Inspection 

- Assign a team member to handle travel arrangements. 

Two Weeks Before the Inspection 

-1. 

-2. 

Ensure licensee documents have been received and distributed to Inspection Team members. 

Contact the licensee to discuss the following: 

Schedule for the entrance and exit meetings Plant access, as necessary 

Size of the inspection team 

Availability of faciliq personnel from the following categories for interviews: 

- Training supervisor 

- Certified instructors 

- Plant supervisors, as appropriate 

- Job incumbents 

The availability of a dedicated plant employee, preferably a person from the Training 

Department, to act as a liaison for the inspection team 

Any special requirements the team may have 

__ 3. Distribute assignments and schedules to the team members 

One Week Before the Inspection 

With the team: 

__ 1. 

- 2. 

- 3. 

- 4. 

- 5. 

Review the inspection schedule. 

Clarify and review any issues raised by Human Factors Assessment Branch. 

Review documentation sent by the licensee. 

Develop technical questions, if needed, for use in interviews 

Review the results of any previous inspection findings involving training. 

NUREG-1220. Rev. 1 

3 ON-SITE INSPECTION ACTIVITIES 

On-site inspection activities consist of an evaluation of 

both observation and interview results. Guidance for performing observations to evaluate training effectiveness is 

inctuded in section 3B of this document. Guidance for 

conducting interviews to determine training effectiveness 

is included in section 3C of this document. As indicated in 

the inspeaion flowcharI shown on page 2 of the Intrcduction, through interviews and observations the inspector 

will determine if additional analysis of the training development process is needed. If the findings indicate that 

training has been effective in supporting rmccessful job 

performance no additional review is required. However, 

observed erratic performance may indicate that some or 

all of the elements of the systems approach to training 

(SAT) process need to he evaluated to identify the 

canse(s) of the poor performance. 

As indicated in the flowchart, begin your activities by 

conducting interviews and performing observations. Each 

of the parts in this seaion contains detailed guidance on 

performing and documenting the activities required in a 

training effectiveness evaluation. 

The SAT element or elements related to a area 

are identifed by the numbers in parenthesis that follow 

behavior statements in the observation checklists and 

after questions in the interview protocols. 

9 NUREG-1220, Rev. 1 

3A GENERAL TEAM PROTOCOL 

During the entrance meeting, dearly explain the purpose 

of the inspection, review the inspection process, introduce the team members and tell what activities each will 

engage in during the inspection. 

The training programs should be evaluated using the 

guidance found in this document and in Inspection Procedure 41W, ‘Training and Qualifiition Effectiveness.” 

The entire team should meet at the end of each day to 

review any issues, discuss questions and concerns, and 

demie problems encountered. New concerns raised by 

team members as a result of their observations and interviews should be investigated as won as possible. 

The exit interview should include all team members and 

appropriaterepresentativesfromthefadtysbff. During 

the exit meeting, all fmdings should be expressed based 

on observed performance problems that are an indication 

that thesystemsapproach to trainingisnot beingsuccessfay implemented. 

11 NUREG-1220, Rev. 1 

3B OBSERVATION CHECKLISTS 

The forms in this section depict the observation checklists 

to be used to document the results of reviewing dassroom, simulator, or on-the-job training and work in progress. The masters for these forms can be found in the 

package of forms incinded with this document. Record 

your rating on copies of the masters of these forms. 

Since performance is the main focus of a training evaluation, the team must perform a sufficient number of observations to ensure that performance trends will be identified. The combination of observations completed is not 

impoaantsincealtoftheobservationscollectdatarelated 

to the various SAT elements. However, observations 

should be completed in each applicable area. 

Each of the observation checklists focuses on behaviors in 

two areas: work practices and technical knowledge. The 

cheddjsts are used to rate selected behaviors related to 

those areas. Each checklist lists behaviors that are characteristic of the setting and, if given a rating of “consistently,’’ are an indication of succesdul job performance. 

Each of the situations bemg observed involves at least two 

people interacting; therefore, both people are rated. 

During observations of training, the trainees and the instructors are rated. During observations of work in progress the worker and the immediate supervisor are rated. 

When conducting observations remember to keep the 

experience of the personnel beii observed in mind. Expected behaviors can differ based on experience. For 

example, the time needed to complete a task is affected 

not only by the amount of knowledge possessed by a 

worker but also by the amount of experience. After completing your observations, review the ratings you have 

given. If performance trends toward “erratically,” further 

review is warranted. Each of the behaviors is related to 

elements of the systems approach to training (SAT). The 

element or elements related to a specilic behavior are 

identified by the numbers in parentheses that follow each 

behavior statement. Identify the SAT element(s) only for 

those areas where performance trends toward “erratic.” 

Use the information in section 3D to evaluate the specific 

SAT elements. Complete the evaluation of all SAT elements related to a particular problem. Detailed information and evaluation criteria for each of the SAT elements 

is provided in &ion 3D. 

13 NUREG-1220, Rev. 1 
Form 3B-1 

Checklist for Observing Behaviors in the Classroom 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistently - if the behavior is always being done and the quality always exceeds 

minimum standards 

Rating Scale: 

Trainee Behaviorsmork Practices 

1. Maintains appropriate classroom demeanor during E G C 

2. Participates in the discussions and activities. (3) E G C 

1. Exhibits understanding of technical information E G C 

training. (3) 

Trainee Behaviors/Technical Knowledge 

received by acknowledging and questioning information 

provided by the instructor. (2x3) 

2. Asks appropriate questions of instructor during 

training. (3) 

Instructor Behaviorsmork Practices 

1. 

2. 

Reviews objectives with trainees. (2x3) 

Uses training time effectively by 

controlling the class and the instructional flow. (3) 

Presents lesson at an appropriate pace for the 

content. (lx2X3) 

3. 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

NUREG-1220, Rev. 1 14 
Classroom Checklist (continued) 

Rating Scale: E - Erratically 

G- Generally 

C - Consistently 

4. 

5. 

6. 

7. 

Adheres to the content and structure of the 

instructor guide or lesson plan. (1)(2)(3) 

Periodically summarizes material to emphasize 

the relationships among topics. (2x3) 

Gives students handouts and yes insbuctional 

materials that are related to the topic and are of good 

Uses questions to determine trainee understanding 

of material being presented. (2)(3)(4) 

quality- (1X2X3) 

Instructor Behaviorsffecbnical Knowledge 

1. 

2. 

3. 

4. 

5. 

6. 

Presents information at a level consistent 

with the abilities of the trainees as evidenced by 

appropriate trainee response. (1)(2)(3) 

Presents information to the appropriate 

depth of knowledge. (1)(2)(3) 

Exhibits familiarity with plant procedures 

and references by referring to them accurately. (2x3) 

Answers questions promptly and accurately. (1)(2)(3) 

Answers questions with an appropriate 

level of detail. (1)(2x3) 

Questions trainees to determine their 

understanding of materials being presented and 

the progress they are making. (2)(3)(4)(5) 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

15 NUREG-1220. Rev. 1 
FOITII 3B-2 

Checklist for Observing Behaviors in the Simulator 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: 

Rating Scale: 

Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistently - if the behavior is always being done and the quality always exceeds 

minimum standards 

Trainee Behaviorsmork Practices 

1. 

2. 

3. 

4. 

5. 

6. 

Demonstrates knowledge and familiarity with plant E G C 

procedures by correctly selecting and using reference 

materials. (2x3) 

provided on procedural adherence. (4) 

during training. (3x4) 

Gives clear and accurate information to the E G C 

shift supervisor and other crew members. (3x4) 

Uses procedures in accordance with the guidance E G C 

Maintains appropriate control room demeanor E G C 

Refers to procedure and procedure steps 

when appropriate. (3x4) 

E G C 

Uses repeat-backs consistently and appropriately. (3x4) E G C 

NUREG-1220, Rev. 1 16 
Simulator Checklist (continued) 

Rating Scale: E - Erratically 

G- Generally 

C - Consistently 

Trainee Behaviors/Technical Knowledge 

1. 

2. 

3. 

4. 

5. 

Notices and acknowledges alarms. (3x4) 

Demonstrates familiarity with physical layout 

of control room by quic!dy locating relevant 

instruments, indicators, and controls. (1)(2)(3x4) 

Exhibits understanding of technical information 

received by acknowledging and questioning information 

provided by the instructor or other crew members. (2)(3)(4) 

Asks appropriate questions of instructors and 

other crew members during training segments. (3x4) 

Demonstrates technical competence by responding 

correctly to indicators, instruments, and alarms. (3x4) 

Instructor Behaviorsmork Practices 

1. 

2. 

Reviews objectives with trainees. (2x3) 

Uses training time effectively by controlling 

the session and instructional flow. (3) 

Directs post-training critiques to maintain 

focus on self-evaluation by individuals and teams. (3x4) 

3. 

Instructor BehaviorslTechnical Knowledge 

1. 

2. 

3. 

4. 

Answers questions promptly and accurately. (3) 

Exhibits good coaching skills. (3) 

Demonstrates adequate knowledge of control 

room personnel responsibilities by providing 

appropriate evaluation feedback during critiques 

on mastery of objectives. (2x3)(4) 

Questions trainees to determine their 

understanding of material being presented and the 

progress they are making. (2X3X4) 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

17 NUREG-1220, Rev. 1 
Form 3B-3 

Checklist for Observing Behaviors 

During On-the-Job Training 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistently - if the behavior is always being done and the quality always exceeds 

minimum standards 

Rating Scale: 

Trainee Behaviorsmork Practices 

1. Demonstrates knowledge and familiarity with plant 

procedures by correctly selecting and using reference 

materials. (2x3) 

Refers to procedure and procedure steps 

when required. (3x4) 

2. 

Trainee Behaviorsfl’echnieal Knowledge 

1. 

2. 

3. 

4. 

5. 

6. 

Demonstrates familiarity with physical layout 

of work area by quickly locating relevant instruments, 

indicators, and controls. (1)(2)(3X4) 

Exhibits understanding of technical information 

received by acknowledging or questioning information 

provided by the instructor. (2x3) 

Asks appropriate questions of instructors during 

training segments. (3x4) 

Demonstrates technical competence by responding 

correctly to indicators, instruments, and alarms. (3x4) 

Questions unusual or unexplained circumstances 

Selects appropriate tools and uses them correctly. 

E G C 

E G C 

E G C 

E G C 

E G C 

E G C 

E G ,c 

E G C 

NUREG-1220, Rev. 1 18 
On-the-Job Cheeklist (continued) 

Rating Scale: E - Erratically 

G- Generally 

C - Consistently 

Instructor Behaviorsmork Practices 

1. Reviews objectives with trainees. (2x3) 

2. Uses training time effectively. (3) 

Instructor BehaviorwTechnieal Knowledge 

1. 

2. 

3. 

Answers questions promptiy and accurately. (3) 

Exhibits good coaching skills. (3) 

Demonstrates adequate knowledge of task 

performance by providing appropriate evaluation 

feedback during training session on mastery of 

objectives. (2X3X4) 

Questions trainees to determine their 

understanding of material being presented and the 

progress they are making. (2X3X4) 

4. 

E G C 

E G C 

E G C 

E G C 

E G C 

E G C 

19 NUREG-1220. Rev. 1 . 
Form 3B-4 

Checklist for Observing Behaviors During Work In Progress 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: Evaluate the overall quality of classroom training by rating each of the behaviors Listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - ErraticaUy - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

Rating Scale: 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistentiy - if the behavior is always being done and the qualify always exceeds 

minimum standards 

Worker Behaviors/Work Practices 

1. Demonstrates knowledge and familiarity with plant 

procedures by correctly selecting and using reference 

materials. (1)(2X3) 

Refers to procedure and procedure steps 

when required, (1X2X3) 

2. 

3. Follows procedure steps. (1)(2)(3) 

4. 

5. Follows appropriate radiation protection 

6. 

7. 

8. 

Questions unusual or unexplained circumstances. 

practices. (lX2X3) 

Works only within the scope of work package. 

Provides clear and accurate information to others. 

Requests assistance or clarification from 

supenisor or manager when procedural or technical 

problems are encountered. 

Uses only “approved/controUed” procedures and 

drawings to perform work. (1)(2)(3) 

9. 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

NUREG-1220, Rev. 1 2.0 
Work-in-Progress Checklist (continued) 

Rating Scale: E - Erratically 

G- Generally 

C - Consistently 

Worker BehaviorslTechnicaI Knowledge 

1. 

2. 

3. 

4. 

5. 

6. 

I. 

8. 

Locates appropriate component correctly. (1)(3)(4) 

Demonstrates familiarity with physical layout of 

work area by quickly locating relevant instruments, 

indicators, and controls. (1)(2)(3) 

Uses appropriate work techniques correctly. (4) 

Records data in appropriate location. (4) 

Selects appropriate tools and equipment. (4) 

Uses appropriate tools correctly. (4) 

Uses appropriate equipment correctly. (4) 

Responds appropriately to unusual or unexplained 

circumstances. (1x4) 

Supervisor Behaviors/Work Practices 

1. 

2 

Gives worker complete work package. 

Selects workers who have been trained and are 

qualified to perform work package tasks. (4) 

Provides a pre-job briefing that stresses safety, 

rad con, work staging, and quality standards. 

Monitors work in progress by visiting work site. (5) 

Answers questions and responds appropriately to 

requests from workers. (5) 

3. 

4. 

5. 

Supervisor BehaviordTechnical Knowledge 

1. Demonstrates familiarity with physical layout of 

work area by quickly locating relwanf instruments, 

indicators, and controls. (3) 

Questions unusual or unexplained circumstances. 

Responds appropriately to unusual or unexplained 

circumstances. 

2 

3. 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

21 NUREG-1220, Rev. 1 

3C INTERVIEW PROTOCOLS 

The formsin this sectiondepict the interview protocols to 

be used to document the results of discussions with trainees, supervisors, and instructors about training. The masters for these forms can be found in the package of forms 

included with this document. Record your rating on cop 

ies of the masters of these forms. 

Since performance is the main focus of a tmining evaluation, a team must perform a sufficient number of interviews to ensure that training concerns will be identified. 

Most of the people interviewed should be job incumbents 

and plant supervisor$ since they are best able to cunclude 

if eaining has adequately prepared workers to perform 

their jobs. 

Each of the interview questions focuses on spdi aspects of training development, presentation, and evaluation. During each interview, the areas of initial, continuing, and on-the-job training trainee evaluation; and 

program evaluation and revision are examined. Each of 

these areas is related to specific elements of the systems 

approach to training (SAT). The SAT element or elements related to a specific area are identaed by the 

numbers in parentheses that follow the questions in the 

interview protocols. 

The purpose of an interview is not to get an answer for 

every question in the protocol but rather to obtain &Ident information to achieve the goals of the inspection. 

The protocols provide a logical frameworkfor obtaining a 

variety of information but can, and should, be modified to 

make them more usefulforaspecificsituation.The interviewer should review the questions ahead of time and 

should delete any questions that do not apply. For example, if a licensee has not conducted an initial training 

program in the recent past, questions related to that aspect of training should be deleted. An interviewer should 

also add questions to address speciftc topics, particularly 

if an inspection is being conducted because of identified 

problems. Specific technical questions may also be asked 

during interviews with job incumbents to determine the 

depth of technical knowledge possessed by a worker. 

Technicai questions should be designed by inspectom 

who are subject matter experts in the areas being mmined. 

After completing the interviews, review the results for 

each of the training program areas listed above. If the 

results suggest that a problem may exist in an area, further evaluation is warranted. Information obtained using 

questions added to the protocols should also be reviewed 

when determining if SAT element evaluation is needed. 

Identify the SAT element(s) related to the areas of concern. Use the information in section 3D to evaluate the 

specific SAT elements. Complete the evaluation of all 

SAT elements related to a particular problem area. Detailed information and evaluation criteria for each of the 

SAT elements is provided in section 3D. 

NUREG-1220, Rev. 1 
Form 3C-1 

Instructor Interview Protocol 

Program: 

Date: 

Interviewer: 

Instructor: 

General Information 

1. What work did you do at this facility before you became an instructor? (3) 

2. 

3. 

4. 

What are your major duties and responsibilities in your present position? (3) 

In which areas are you certified to instruct? (3) 

What skills does your instructor certification address? (3) 

(technical skills, instructional skills, etc.) 

Initial, Continuing, and On-the-Job Training 

5. What materials do trainees find difficult and/or easy? (5) 

6. How accurate and complete are the training materials? (1)(2)(3)(5) 

NUREG-1220, Rev. 1 
Instructor Interview (continued) 

7. In what ways could the training materials, training aids, and instructional settings be 

improved? (3x5) 

8. What area of training do you consider the strongest, and what area of training do you consider the 

weakest? (4x5) 

9. HOW do you know that your training materials are current and correct? (Do the materials reflect 

modifications and procedural changes?) (1X2X5) 

10. Who conducts on-the-job training and evaluations? (3x4) 

11. How are trainers and evaluators qualified to conduct on-the-job training and evaluations? (3x4) 

Trainee and Program Evaluation and Program Revision 

12. Who is responsible for identifying training needs? (1)(4X5) 

U. What method is used to collect and resolve comments and ideas from trainees and 

supervisors? (4x5) 

14. How do you follow up on training effectiveness? (5) 

25 NUREG-1220, Rev. 1 
Form 3C-2 

Job Incumbent Interview Protocol 

Program: 

Date: 

Interviewer: 

Job Incumbent: 

General Information 

1. How long have you worked in the (operations, maintenance, chemistry, etc.) department? (5) 

2. Did you qualify in your present job through training or through exemptions (grandfathering)? 

(NOTE TO INTERVIEWER: IF BY EXEMPTIONS-SKIP QUESTIONS 5-9) (4) 

3. What tasks do you find particularly easy to perform? What makes them easy? (5) 

4. What tasks do you find particularly difficult to perform? What makes them difficult? (5) 

Initial Training 

5. How could initial training have prepared you better for your job? (1)(2)(3)(4)(5) 

6. How relevant is the initial training program to your job? (1x5) 

NUREG-1220, Rev. 1 26 
Job Incumbent Interview (continued) 

7. How well was the initial training timed to meet actuai job demands? (3) 

8. What specific topics or tasks need more emphasis during initial training? (1)(2)(3x5) 

9. What differences have you noticed between the training you received and what is expected 

from you on the job? (1)o 

Continuing Training 

10. What training have you been given for changes in procedures or equipment that occurred in 

your job since you were first assigned to it? (1)(2X3)(4)(5) 

11. What continuing training have you received since being assigned to your job? (4x5) 

12. What additional continuing training topics do you need for your job? (1x5) 

l3. How adequate is the training you have received during continuing training? (5) 

Trainee Evaluation 

14. If you ever failed an initial training exam, what happened? (4) 

21 NUREG-12% Rev. 1 
Job Incumbent Interview (continued) 

15. If you ever failed a continuing training exam, what happened? (4) 

16. How prompt and objective is the feedback about your performance during training? (4) 

17. How do you feel the feedback process could be improved? (4x5) 

Program Evaluation 

18. When you have been asked to evaluate training provided to you, were your comments resolved 

to your satisfaction? (5) 

19. What improvements or increased emphasis have you noticed in the training program over the 

last 2ym? (5) 

20. What suggestions and recommendations have you noticed that were incorporated into the 

training program? (5) 

21. How would you improve training? (5) 

22. Overall, how effective was your training? (4x5) 

NUREG-1220, Rev. 1 28 
Form 3C-3 

Plant Supervisor Interview Protocol 

Program: 

Date: 

Jntewiewer: 

Plant Supervisor: 

General Information 

1. What work did you do at this facility before you became a plant supervisor? 

2. What are your major duties and responsibilities in your present position? 

Initial Training 

3. What tasks are newly trained employees allowed to perform? (4x5) 

What is the basis of this decision? (4x5) 

How is this enforced or controlled? (5) 

4. How do recently trained employees compare to those who received earlier training? (4x5) 

What is the basis of your comparison? (4x5) 

29 NUREG-1220, Rev. 1 
Plant Supervisor Interview (continued) 

5. What additional training do job incumbents receive once they are on the job? (lX3X4) 

What is the basis of that decision? (1x5) 

6. How are changes in job assignments, procedures, and equipment reflected in training? (1x5) 

On-the-Job Training 

7. How do you make assignments to on-the-job training instructors? (3x5) 

8. How do you make assignments to on-the-job evaluators? (3x5) 

9. What are the strengths and weaknesses of the on-the-job training program in your area? (5) 

10. How do you evaluate the training received by your workers? (5) 

11. To what extent has the observed performance of workers met your expectations? (5) 

Program Evaluation and Revision 

12: What new problems has training created? (1X2X5) 

NUREG-1220. Rev. 1 30 
Plant Supervisor Interview (continued) 

U. What current training do you consider unnecessary? Why? (1X2X4x5) 

14. HOW do your comments on training quality and effectiveness get resolved? (5) 

15 HOW do you veri@ that training is jobspecific and plant-specific? (1)(2)(4)(5) 

16. Overall, how effectiviis the training program? (5) 

31 NUREG-1220, Rev. 1 
Form 3C-4 

Training Supervisor Interview Protocol 

Program: 

Date: 

Interviewer: 

Training Supervisor: 

General Information 

1. What work did you do at this facility before you became a training supervisor? (3) 

2. What are your major duties and responsibilities in your present position? (3) 

Initial Training 

3. What involvement do you have in developing and implementing initial training? (3) 

4. What specific areas within the development and implementation of initial training need to be 

improved? (5) 

5. What aspects of the initial training program do you consider the strongest, and what aspects do 

you consider the weakest? (5) 

NUREG-1220. Rev. 1 32 
Training Supervisor Interview (continued) 

Continuing Training 

6. What involvement do you have in developing and implementing continuing training? (3) 

7. What aspects of the continuing training program do you consider the strongest and what 

aspects do you consider the weakest? (5) 

On-the-Job Training 

8. What is the.Training Department’s role in implementation of the on-the-job training and 

evaluation program? (1)(’2)(3)(4)(5) 

Trainee Evaluation 

9. How do you find out how satisfied your line organization supervisors are with trainee 

performance? (5) 

10. How are comments from supervisors and trainees on training content and effectiveness 

resolved? (5) 

Program Evaluation and Revision 

11. What improvements or increased emphasis in the overall area of training have you noticed 

within the last 2 years? (1)(2)(3)(5) 

33 NUREG-1220, Rev. 1 
Training Supervisor Interview (continued) 

12. How do changes in job assignment, procedures, and equipment get reflected in training? 

(WX3X4K5) 

l3. How are training needs identified? (1x5) 

NUREG-1220, Rev. 1 34 
3D SAT ELEMENT EVALUATION 

It is not necessilIy to routinely evaluate all SAT elements 

as part of the training evaluation process. Elements in the 

systems approach to training (SAT) process should be 

evaluated if performance problems are identifed in observations or interviews. me coding menthetic number) on each interview question and each observation 

behavior statement should be used to identify related 

SAT elements. 

Use the tally sheet shown on page 36 to summarize the 

results of observations and interviews. If the results of 

interviews and observations indicate that SAT element 

evaluation is warranted, use the direaions and criteria 

listed with that element to complete the evaluation. The 

results of the SAT.eIement evaluation should be used to 

identify deficiencies in the implementation of the SAT 

process. The results should be detailed in the report of 

inspeaion findings. 

35 NUREG-1220, Rev. 1 
This depicts the Interview and Observation Tally Sheet. The master of this form can be found in the package of forms 

included in this document. Record your summary on a copy of the master of this form. 

Interview And Observation 

Tally Sheet 

SAT SAT 

Element Element 

1 2 

OBSERVATIONS 

C 1 ass r oo m 

Simulator 

On-the-Job Training 

Review the information obtained in interviews and observations. Identify observation items where performance was rated 

"erratic"andinteMkRquestionswhereanswerssuggestthatapmblemmayexist.Usethissheet totallythereferences to 

the related systems approach to training (SAT) elements which are coded at the end of each item. Based on this summary 

identify the SAT elements that require additional review. Use the directions and evaluation criteria Wed with that 

element in Section 3D, "SAT Element Evaluation," to complete the review. 

SAT SAT SAT 

3 4 5 

Element Element Element 

NUREG-1220, Rev. 1 36 
SAT ELEMENT 1 

(Analysis) 

REQUIRED ELASMEW - 10 CFR Parts 50.120 and 55.4 requires a systematic analysis o€ 

the job to be performed. 

Plant and training staff should use a systematic analysis to identify the knowledge and skills for presentation in 

training in order to meet job performance requirements. 

Review the guidance provided for each of the applicable program characteristics listed below. Then rate each 

characteristic using the specific guidance and record the information on the Summary Rating Sheet for this 

element. 

Applicable Program Characteristics: 

1.1 Systematic method used to identify job tasks. 

1.2 Tasks obj&ly and consistently selected for training 

13 Tasks differentiated for initial and continuing training. 

1.4 Analysis results adequate for development of learning objectives. 

1.5 New or changed tasks analyzed to determine traidng needs. 

31 NUREG-1220, Rev. 1 
1.1 Systematic method used to identify job tasks. 

GENERAL GUIDANCE 

A method or procedure is considered systematic if it possesses the following characteristics: 

0 

0 

0 

Consists of a logically ordered set of steps, 

Can be exmted with consistent results on difierent occasions by different people, 

Requires a record of review and/or approval for historical record, and 

Results in a consistently formatted, quality product. 

SPECIFIC GUIDANCE 

Review any documents that describe how tasks are identified. For the technical areas being examined, review any 

evidence related to the identification of job performance requirements. 

Then rate this characteristic as: 

Erratically if there is limited evidence of tasks being identified to determine job performance requirements, 

OR if the implementing procedures or instructions that desaibe the proces for conducting an 

analysis to determine job paformance requirements or training needs is of poor quality beciuse. it 

does not meet the characteristics to be rated "Consistently." 

if there is evidence that a process exists but it is be& used infrqueutly, ORif the process is being 

used on a regular basis but the product is of poor quality. 

if completed analyses have the following characteristics: 

- 

- 

- 

Generally 

Consistently 

involvement of experienced and qualified training professionals and job incumbents, 

review of analyses by job incumbents, 

approval by both plant and training management. 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 1. 

NUREG-1220. Rev. 1 38 
1.2 Tasks objectively selected for training. 

GENERAL GUIDANCE 

There should be a basis for selection of tasksfor formal training. The following are some criteria that are generally used as 

the basis for selecting tasks for training: 

Taskdifficulty, 

On-the-job performance problems, and 

Industry events applicable to the facility. 

Infonuation on the areas noted above should be collected in an objective and reliable way and should be obtained from 

workers who are familiar with the particular job being analyzed. 

SPECIFIC GUIDANCE 

Review the licensee’s criteria for selecting tasks for training. In the technical areas under examination, review any 

evidence of the task selection process. 

Then rate this characteristic as: 

Percent of job incumbents performing the task, 

Percent of time spent performing the task, 

Consequence of inadequate task performance, 

Erratically if there is limited evidence of information being collected about the need for training, OR if the 

implementing procedures or instructions that describe the process for selecting tasks for thing 

does not use the criteria listed in ‘‘General Guidance.” 

if there is evidence that a process exists but it is being used infrequently, OR if the process is being 

used on a regular basis but the product is of poor quality because it does not meet the characteristics 

to be rated “Consistently.” 

if completed analyses have the following characteristics: 

- 

- 

Generally 

Consistently 

involvement of experienced and qualified training professionals and job incumbents, 

a representative sample of job incumbent opinions and performance problems have 

been objectively and reliably identified. 

Record your rating of this characteristic on the Sununary Rating Sheet for SAT Element 1. 

39 NUREG-1220. Rev. 1 
1.3 Tasks differentiated for initial and continuing training. 

GENERAL GUIDANCE 

In general the following three criteria are applied to all tasks in selecting tasks for continuing training: 

Consequence of inadequate perfomance. Those tasks whose performance is so critical that particular attention must 

be paid to ensuring adequate performance. 

Fhquency of performance. Those tasks selected for initial training where proficiency is not maintained through onthe-job performance. 

Decay of tasks performance over Lime. Tasks that require periodic practice to maintain proficiency. 

SPECIFIC GUIDANCE 

Review the licensee’s criteria for determining initial and continuing trainingoontent. For each technical area, review MY 

evidence of how tasks were selected for training. 

Then rate this characteristic as: 

Erraticaliy 

Generally 

Consistently 

if there is no systematic basis for selecting task for initial and continuing Wig, OR if the 

selection process is does not provide adequate guidance to ensure consistent results. 

if there is evidence that a process exists but it is being used infrequently, OR if the process is being 

used on a regular basis but the process does not involve job incumbents. 

if the selection process is unplemented in such a way that a representative sample of job incumbents 

is used to identify appropriate tasks for training and the results are consistently documented. 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 1. 

NUREG-1220. Rev. 1 40 
1.4 Analysis results are adequate to support development of learning objectives. 

GENERAL GUIDANCE 

Task analysis is usually performed to determine the conditions, actions, and standards of performance associated with a 

particular task. Conditions, actions, and standards are used in developing learning objectives to iden@ what the trainees 

are expected to learn from the training. A task analysis will also provide information on the skills and knowledge needed 

for swcedul task performance. Task analysis may not be necessary in all instances. For training programs that are not 

task based, partintlary those designed for engineering support personnel, a needs analysis can be used as a basis for 

developing learning objectives. 

SPECIFIC GUIDANCE 

Review anatysis results. 

Then rate this characteristic as: 

Erratically if there is limited evidence of tasks being analyzed to identify conditions, actions, and standards for 

task performance, OR if the implementing procedures or instructions that describe the process for 

conducting an analysis to determine conditions, actions and standards of learning objectives do not 

provide adequate guidance to ensure consistent results. 

if there is evidence that an analysis pmcess exists but it is being used infrequently, OR if the pmcess 

is being used on a regular basis but the quality of the product varies. 

if analyses that have been done have the following characteristics: 

- 

Generally 

Consistently 

The pm for collecting information concerning conditions, actions, and standards is used 

consistently and 

- Conditions, actions, and standards are specific and measurable, 

Reoord your rating of this characteristic on the Summary Rating Sheet for SAT Element 1. 

41 NUREG-1220. Rev. 1 
1.5 New or changed tasks are analyzed to determine training needs. 

GENERAL GUIDANCE 

Job performance requirements may change over time for a variety of reasons, such as: 

Backfits/modification of plant equipment, 

0 

0 

Iftrainingistoremainperformancebaseathentheimpactof changesmnstbeanalyzedand thetminingpmgramrevised, 

as needed. Task analysis should be performed for new tasks or for tasks that have changed significantly. Needs analysis 

may be more appropriate for training programs that are not task based, such as the program for engineering support 

personnel. The analysis of new or changed tasks should be governed by procedures or instructions to ensure consistent 

results. 

Changes in plant operating or administra tive procedures, 

Lessons learned from operating experience, and 

Reorganization or changes in job responsibilities. 

SPECIFIC GUIDANCE 

For the technical area under examination, review any evidence that new or changed tasks are being analyzed to determine 

training needs. 

Then rate this characteristic as: 

Erratidly if there is limited evidence of changes being analyzed to determine new job pedomce requirements, OR if the implementing procedures or instructions that describe the process for conducting 

an analysis or determining training needs do not provide adequate guidance to ensure consistent 

results. 

if there is evidence that a process exists but it is being used infrequently, OR if the process is being 

used on a regular basis but the quality of the product varies. 

if approved procedures are in place to require that all information is reviewed to ensure that job 

performance requirements remain current, AND if the procedures are being properly implemented 

Generally 

Consistently 

in all aspeas. 

Record your rating of this chmcteristic on the Summary Rating Sheet for SAT Element 1. 

NUREG-1220. Rev. 1 42 
~depictstheSummaryRatingSheetforSATElement 1.Themasterofthisformcanbefoundin thepackageofforms 

included with this document. Record your ratings on a copy of the master of this form. 

Fo~ 3D-1 

Element 1 Summary Rating Sheet 

REQUIRED ELEMENT - 10 CFR Parts 50.120 and 55.4 requires a systematic analysis of the job 

to be performed. 

RatingScale: E- Erratically \ 

G- Generally 

C- Consistently / 

See specific guidance listed with each characteristic 

Applicable Program Characteristics: 

1.1 

1.2 

1.3 

1.4 

1.5 

Systematic method used to identify job tasks? 

Objective and consistent method used to select tasks for training? 

Tasks differentiated for initial and continuing trainiig? 

Analysis adequate for development of learning objectives? 

New or changes tasks analyzed to identify training needs? 

E G C 

E G C 

E G C 

E G C 

E G C 

Not all of the applicable characteristics are nemsaxy for compliance with 10 CFR Parts 50.120 and 55.4. However, a 

trend toward an "emtic" rating may be the basis of a violation since it would indicate that a systems approach to training is 

either not implemented or poorly implemented. 

43 NUREG-1220, Rev. 1 

SAT ELEMENT 2 

(Learning Objectives) 

REQUIRED ELEMENT - 10 CFR Parts 50.120 and 55.4 requires that learning objectives 

be derived from the analysis which describes the desired 

performance after training. 

Learning objectives should support successful job performance by reflecting analysis results and by providing conditions 

and standards for job performance behaviors (actions) expected of trainees upon completion of training. 

Reviewtheguidance providedforeachof theapplicableprogram characteristics Iistedbelow.Thenrate eachcharactetistic using the spedfic guidance and record the information on the Summary Rating Sheet for this element. 

Applicable Program Characleristics: 

2.1. Learning objectives enist for tasks selected for training. 

2.2 Learning objectivesconditions and standards reflect job pedormance requirements. 

2.3 Learning objectives are revised as needed to reflect job changes. 

45 NUREG-1220, Rev. 1 
2.1 Learning objectives reflect analysis results. 

GENERAL GUIDANCE 

Learning objectives describe precisely what is to be learned in terms of the expeaed trainee performance under specified 

conditions to accepted standards. 

Learning objectives should provide the training materials developer with the information needed to organize objectives, 

select instructional settings, and perfom other related actions required for designing and developing a training program. 

Ewmples of good learning objectives follow: 

"Given a handheld calculator, calculate the volume of a cylinder, without error," and 

"When directed by the shift supervisor and given the appropriate procedure, start a main feedwater pump in accordance with the approphte procedure," and 

"Given the disassembled par&s of a service water pump, proper tools and the approved procedure, reassemble the 

pump in accordance with the steps of the approved procedure." 

SPECIFIC GUIDANCE 

Obtain such instructional materials as lesson plans and compare learning objectives to the list of tasks selected for 

Rate this characteristic as: 

training. 

Erratically 

Generally 

Consistently 

if learning objectives related to the tasks being reviewed do not always exist 

if learning objectives that exist do not desmbe what is to be learned, OR if the learning objectives 

that Bdst are not related to the tasks Wig reviewed 

if the learning objectives are related to the tasks being reviewed, AND if the learning objectives 

ddbe precisely what is to be learned 

Record your rating of this characteristic on the Summav Rating Sheet for SAT Element 2. 

NUREG-1220, Rev. 1 46 
2.2 Learning objectives actions, job performance conditions, and job performance standards 

reflect job performance requirements. 

GENERAL GUIDANCE 

An example of an action is underlined in the following learning objective: 

Given the disassembled parts of a service water pump, proper tools and the approved procedure, rearremble the 

pump in accordance with the steps of the approved procedure. 

An example of a job performance wndition is underlined in the following learning objective: 

Giwn tempmature indications and their mociated 

grees per hour. 

calculate the heatup rate with an accuracy of withm + 2 deAn example of a job performance standard is underlined in the following learning objective: 

Given a handheld calculator, calculate the volume of a cylinder, wirhour mor 

SPECIFIC GUIDANCE 

For the technical area under examination, compare the learning objectives to job performance requirements identified in 

analysis documentation. If analysis information is not available, compare learning objectives to the requirements in 

procedures or other documents governing a task’s performance. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

if one or more of the learning objectives are not based ou assonated job performance behavior$ OR 

if learning objectives do not have conditions or standards. 

if the behaviors required are not always appropriate for tlie job requirements, OR if the conditions 

are not specific and/or the standards are not measurable. 

if the behaviors reflect job performance requirements, AND 

if conditions are specifii AND 

if standards are measurable. 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 2. 

47 NUREG-1220, Rev. 1 
2.3 Learning objectives are revised as needed to reflect job changes. 

GENERAL GUIDANCE 

Changes in job performance requirements as a result of equipment modifictions, procedure revisions, and lessons 

learned from plant and industry events must be reviewed to determine their impact on training materials, including 

learning objectives. 

SPECIFIC GUXDANCE 

For the technical area under examination, review plant modification padrages procedure changes, and other revisions to 

identify potential changes for training materials. Review lesson plans to determine if learning objectives reflect job 

performance requirements determined by any recent changes. 

Then rate this characteristic as: 

Erratically if no process exists to evaluate the impact of changes on training materials OR 

if the process does not provide adequate guidance to ensure Consistent results. 

Generally 

Consistently 

if the process exits but the but the quality of the reviews varies. 

if the process is always followed for ali change actions. 

Record your rating of this characteristic on the Summruy Rating Sheet for SAT Element 2. 

NUREG-1220. Rev. 1 48 
7his depicts the Summary Rating Sheet for SAT Element 2. The master of this form can be found in the package of forms 

included with this document. Record your ratings on the copy of the master of this form. 

Form 3D-2 

Element 2 

Summary Rating Sheet 

REQUIRED ELEMENT - 10 CFR Parts 50.1M and 55.4 requires that learning objectives be derived from 

the analysis which describes the desired pedormance der training. 

RatingScale: E- Erratically \ 

G- Generally 

C- consistently / 

See specific guidance listed with each characteristic 

Applicable Program Characteristics: 

2.1 Learning objectives exist for tasks selected for training? E G C 

2.2 Learning objectives conditions and standards reflect job performance E G C 

2.3 Learning objectives are revised as needed to reflect job changes. E G C 

requirements. 

Not all of the applicable characteristics are neceSSary for compliance with.10 CFR Parts 50.120 and 55.4. However, a 

trend toward an “erratic” rating may be the basis of a violation since it would indicate that a systems approach to training is 

either not implemented or poorly implemented. 

49 NUREG-1220, Rev. 1 

SAT ELEMENT 3 

(Design and Implementation) 

REQUIRED ELEMENT - 10 CFR Parts 50.120 and 5 1 A requires that training design and 

implementation be based on the learning objectives. 

Content of trainiig programs should be derived from learning objectives and sequenced and presented for effective 

learning. 

Review the guidance provided for each of the applicable program characteristics listed below. Then rate each characteristic using the specifc guidance and record the information on the Summary Rating Sheet for this element. 

Applicable Program Characteristics: 

3.1 Lesson plans are structured to provide for consistent presentation. 

3.2 Depth of content is adequate to support mastery of leamhg objectives. 

33 Information is presented in a sequence (within a lesson as well as among several lessons) that supports 

effective learning. 

3.4 Training is adequately presented. 

3.5 Personnel providing classroom, on-the-job, laboratory, and simulator training are proficient in the 

methods and techniques for successful presentation in the particular setting. 

51 NUREG-1220. Rev. 1 
3.1 Lesson plans are structured to provide for consistent presentation. 

GENERAL GUIDANCE 

A lesson plan is a structured outline that en 

Lesson plans should include: 

learning objectives 

SPECIFIC GUIDANCE 

In the technical area under examination, review a sample of the lesson plans. 

Then rate this characteristic as: 

es consistency in instructor presentation and evaluation of trainee 

performance regardless of the instructional s &;. ing (classroom, simulator, lab or workshop, or on-the-job training. 

adequate amount and detail of content to ensure consistency, and 

required support materials including equipment, tools, audiovisual and other equipment. 

Erratically 

Generally 

Consistently 

if lesson plans are not all developed and those that are developed do not include the characteristics 

needed to be rated "Consistently." 

if the lesson plans are not all developed but those that are done contain the items listed in ''General 

Guidance." 

if the all the lesson plans are complete AND consistentlydeveloped AND contain the items listed in 

"General Guidance." 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 3. 

NUREG-1220. Rev. 1 52 
3.2 Depth of content is adequate to support mastery of learning objectives. 

GENERAL GUIDANCE 

Learning objectives and lesson content should be consistent. For example, higher level learning objectives that 

require the trainee to “compare and contrast” two situations would not be adequately supported by lesson 

content that does not stress differences and similarities. 

SPECIFIC GUIDANCE 

Review lesson plans and compare the trainee performance requirements with the supporting lesson content. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

if there is no lesson content to support the learning objectives 

if the lesson content is less detailed than the learning required by the objectives 

if the lesson content matches the performance requirements of the learning objectives 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 3. 

53 NUREG-1220, Rev. 1 
3.3 Information is resented in a sequence (within a lesson as well as among several lessons) 

that supports e &. echve learning. 

GENERAL GUIDANCE 

Information on several topics may be related. When there is a dependency among topics, the most basic should be 

presented fit. For example, normal operation of a component mnst be covered before knowledge of abnomaloperation 

can be understood. Complementary topics should be covered together to allow the trainee to understand the interrelationships among them. 

SPECIFIC GUIDANCE 

Review lesson plans to determine if the sequence of presentation within a single lesson and among several lessons 

supports effective learning. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

if the topics are not sequenced from simple to complex 

if the topics are presented in a seqnence within a lesson but not among several lessons 

if the topics are sequenced within lessons as well as among lessons 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 3. 

NUREG-1220. Rev. 1 54 
3.4 Training is adequately presented. 

GENERAL GUIDANCE 

The qualiw of training is directly related to the technical knowledge of the instructor. Adequate technical knowledge is 

needed to emure that the content of the lesson is accurately presented. 

SPECIFIC GUIDANCE 

This characteristic is related to 3.5. Review the technical training received and the work history to determine if the 

instructor has been trained at least to the level of knowledge to be presented in the training. The instructor should also 

have &Kited proficiency in any skills for which training will be presented. Review the technical topics the instructor is 

normally assigned to present. 

Then rate this characteristic as: 

Erratically if the instructor's level of technical knowledge or skill is less than that to be presented in the training 

Generally iftheinstructors'sleveloftechnicaknowledgeorskillisatleastequaltothattobepresentedinthe 

Consistently if the instructor's level of twhnical knowledge or skill exceeds that to be presented in the training 

training 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 3. 

55 NUREG-1220. Rev. 1 
3.5 Personnel providing classroom, on-the-job, laboratory and simulator training are 

proficient in the methods and techniques for successful presentation in the particular 

setting. 

GENERAL GUIDANCE 

In addition to a thorough understanding of the training content, successful execution of a training program depends on 

additional skills and knowledge in the areas of: 

Questioning skius, 

Coachingskillsand 

Poor instructor performance may reflect poor preparation, insufficient preparation time, poor instructional materials, 

lack of technical knowledge in the specific area or poor instructional skill?.. Focus the evaluation of this characteristic on 

instructor preparation for performing the instructional tasks. 

SPECIFIC GUIDANCE 

This is related to 3.4. Review instructional trainiig histov and previous observations/evaluations of the instructor to 

determine if the instructor has received training on instructional skills areas and to identify any prior performance 

deficiencies. 

Then rate this characteristic as: 

Presentation techniques including use of appropriate equipment for a setting. 

Erratically if thereis evidence that instructors are trained but still exhibit performance problems, OR if there is 

no evidence that instructors are being prepared to perform instructional duties 

if the instructor has been trained and previous observations indicate acceptable performance 

if the instructor has been trained and previous observations indicate above average performance 

Generally 

Consistently 

Record your rating of this characteristic on the SummaIy Rating Sheet for SAT Element 3. 

NUREG-1220, Rev. 1 56 
Thisdepicts theSummaryRatingSheetforSATF~ement3.Themasterofthisformonbefoundinthepackageofforms 

included with thjs document. Record your ratings on a copy of the master of this form. 

Form 3D-3 

Element 3 

Summary Rating Sheet 

REQUIRED ELEMENT- 10 CFR Parts 50.120 and 55.4 requires that training design and 

implementation be based on the learning objectives. 

RatingScale: E- Erratically \ 

G- Generally 

C- Consistently / 

See sDenfic mane lied with each characteristic. 

Applicable Program Characteristics: 

1. 

2. 

3. 

4. 

5. 

Lesson plans are structured to provide for 

consistent presentation. 

Depth of content is adequate to support wterj 

of learning objectives. 

Information is presented in a sequence (within 

a lesson as well as among several lessons) that supports effective 

learning. 

Training is adequately presented. 

Personnel providing classroom, on-the-job, laboratory, and 

simulator training are proficient in the methods and techniques 

for succesful presentation in the particular setting. 

E G C 

E G C 

E G C 

E G C 

E G C 

Not aU of the applicable characteristics are neceSSary for compliance with 10 CFR Parts 50.128 and 55.4. However, a 

trendtowardan"erratic"ratingmaybethebasisofaviolationsineitwouldindicate thatasystemsapproachtotdgk 

either not implemented or poorly implemented. 

51 NUREG-1228, Rev. 1 

SAT ELEMENT 4 

(Trainee Evaluation) 

REQUIRED ELEMENT 

Trainees should be. evaluated during all aspects of training to determine their progress toward mastery of job 

performance requirements 

Review the guidance provided for each of the applicable program characteristics listed below. Then rate each 

characteristic using the specific guidance and record the information on the Summary Rating Sheet for this 

element. 

Applicable program characteristics: 

4.1 A relationship exists between job performance requirements or learning objectives and test items. 

4.2 Trainee performance is evaluated regularly. 

4.3 Remediation is provided when appropriate. 

4.4 Continuing training contains performance requirements for difficult, important, or infrequent tasks. 

4.5 Training and task performance exemptions are objectively based. 

4.6 Evaluations of task performance test the trainees’ mastery of job performance requirements. 

- 10 CFR Parts 50.W and 55.4 requires the evaluation of trainee 

mastery of objectives during training. 

59 NUREG-1220. Rev. 1 
4.1 A relationship exists between job performance requirements or learning objectives and test 

items. 

GENERAL GUIDANCE 

The characteristic is related to 4.6. Learning objectives should reflect job performance requirements. Guidance on the 

relationship between learning objectives and the desired job performance is explained in section 2.1. Analysis should be 

used to identify the job performance requirements and learning objectives should be derived from the analysis. A similar 

match should &between learning objectives and test items. Test items should be derived from the learning objectives. 

Test items should require the trainee to demonstrate a level of learning consistent with what is required by the learning 

objectives. Acommon problemisthat testitemsarewEittenataleveIhigherthanthatrequiredbythelearningobjectives. 

For example the objectives would require the trainee to "recall", "Ijst", "identify", etc. However, the test items would 

require the trainee to "compare and Contrast", "interpret", etc. The objectives and the test items could both be of high 

quality but they would not be consistent with each other. 

SPECIFIC GUIDANCE 

Review learning objectives and corresponding test items to evaluate the consistency between them. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

* if the match between learning objectives and test items varies widelyfrom lesson to lesson or test to 

test 

if the match between learning objectives and test items varies to some extent 

if the match between learning objectives and test items is consistent 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 4. 

NUREG-1220. Rev. 1 
4.2 Trainee performance is evaluated regularly. 

GENERAL GUIDANCE 

Learning is enhanced when trainees are given feedback with respect to their performance. The more immediate and 

continuous the feedback the better. This feedback can be self-checks that are mred by the trainee, as in self-study 

workbooks, or by the instructor-admi&tered and scored tests. If instructor-administered test is used, there should be 

provisions for promptly providing the results to the trainees. 

SPFCWIC GUIDANCE 

Review any guidance provided in training procedures. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

if there is no evidence of regular, prompt, and objective feedback of trainee performance 

if trainees are given general feedback concerning overall performance 

if trainees are given specific feedback abou! their Sliills or knowledge that do not meet job performance requirements 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 4. 

61 NUREG-1220. Rev. 1 
4.3 Remediation is provided when appropriate. 

GENERAL GUIDAh'CE 

Remediation can vary based on the individual trainee and situation. A written examination or a performance test may 

indicate weaknesses in some areas rather than outright failure in all areas. The amount and kind of retraining and the 

need for retesting should be determined by the initial performance. Independent review by the trainee may be more 

appropriate that a formal retraining session. In all cases, failures on written or performance tests require retesting. The 

retesting may be on all or part of the original material, but should include all aspects where weak performance was 

originally identified. 

SPECIFIC GUIDANCE 

Review specific examples of situations where minimum standards were not met. Determine the corrective actions taken 

to ensure that the trainee was ultimately suocessful. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

if trainees are not always remediiated, OR if the remediation criteria do not exist 

if remediation does occur but the remediation criteria are not consistently applied 

if remediation occurs in accordance with established criteria 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 4. 

NUREG-1220. Rev. 1 62 
4.4 Continuing training contains performance requirements for difficult, important, or 

infrequent tasks. 

GENERAL. GUIDANCE 

Tasks selected for continuing mining are usually those that have the potential for degraded performance. Retraining in 

these tasks should consist of a review of the knowledge supporting the task and should require trainees to demonstrate 

their ability to sn- perform the task 

SPEclFlC GWANCE 

Review continuing training lesson plans to determine if tmining includes both classroom and perfotmance requirements. 

‘Ilken rate this characteristic as: 

Erratically 

Generally 

Consistently 

if the tasks are identified for continuing training but performance is not required 

if taslrs are identified for continuing training and performance is required for some tasks 

if continuing training consistently repoires task performance 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 4. 

63 NUREG-1220, Rev. 1 
4.5 Training and task performance exemptions are objectively based. 

GENERAL GUIDANCE 

Trainees can meet training and task qualification requirements without completing the training and task qualification 

program. Previous training and experience can be used as the basis for exempting a trainee from same or all aspects of a 

training program. Previous training andlor experience used as the basis of a training exemption should be evaluated 

against a standard set of criteria. The results of any reviews to waive training requirements for an individuat should be. 

documented. 

SPEClFIC GUIDANCE 

Review tasks qualification records to identify individuals that were granted exemptions. Review the documentation 

supporting the exemptions to determine if they were granted in accordance with established criteria. 

Then rate this characteristic as: 

ErraticalIy 

Generally 

Consistently 

I exemptions are granted without review, OR without documentation 

if exemptions are rwiewed but criteria are weak 

if exemp~~beii +asis- d deevmeatettevrrluations 

Remrd your rating of this characleristic on the Summary Rating Sheet for SAT EIement 4. 

NUREG-1220, Rev. 1 64 
4.6 Evaluations of task performance test the trainees’ mastery of job performance 

requirements. 

GENERAL GUIDANCE 

This characteristic is related to 4.1. The evalnation should match the job performance requirements outlined in 

procedures or manuals. Evaluations should require a trainee to actually perform a task. Simulation usually involves gohg 

through the actions without actually carrying them out. Simulating a task performance is only acceptable if actual 

performance might compromise plant status. Task evaluation in the simulator is considered performance rather than 

simulation. 

SPECIFIC GUIDANCE 

Review task qualification manuals to determine the degree to which task simulation is allowed. Compare task performance standards to determine if they reflect actual job standards. 

Then rate this characteristic as: 

Erratidly 

Generally 

Consistently 

if simulation is allowed on tasks that would not compromise plant status, AND if task performance 

standards do not match job performance requirements 

if simulation is allowed on tasks that would not compromise plant status, OR if task performance 

standards do not match job performance requirements 

if simulation is allowed only for those tasks that would compromise plant status AND task performance requirements match job performance requirements 

Record your rating of this characteristic on the Summniy Rating Sheet for SAT Element 4. 

65 ~~ltEG-1220, Rev. 1 
lhisdepicts the SummaryRatingSheetforSATElement 4.Themasterofthisformm befoundin thepackageofforms 

included with this document. Record your ratings on a copy of the master of this form. 

Form 3D-4 

Element 4 

Summary Rating Sheet 

REQUIRED ELEMENT- 10 CFR Parts 50.120 and 55.4 requires the evaluation of trainee mastery of objectives during training. 

RatingScale: E- Erratically \ 

G- Generally 

C- consistently / 

See specific guidance listed with each characteristic. 

Applicable Program CharaiiteriSties: 

4.1 A relationship qkts between job performance 

requirements and learning objectives and test items. 

E G c 

4.2 Trainee pedoxmance is evaluated regularly. E G C 

4.3 Remediation is provided when appropriate. E G C 

4.4 Continuing training contains performance 

4.5 

requirements for difficult, important, or infrequent tasks. 

Training and task performance exemptions arc 

objectively based. 

Evaluations of task performance test the 

trainees’ mastery of job performance requirements. 

4.6 

E G C 

E G 

E G C 

Not all of the applicable characteristics are neceSSary for compliance with 10 CFR Parts 50.120 and 55.4. However, a 

trend toward an “erratic“ rating may be the basis of a violation since it would indicate that a systems approach to traag is 

either not implemented or poorly implemented. 

NUREG-1220, Rev. 1 
SAT ELEMENT 5 

(Program Evaluation and Revision) 

REQUIRED ELEMENT - 10 CFR Parts 50.120 and 55.4 require the evaluation and 

revision of training based on the performance of trained 

personnel in the job setting. 

Plant and training staff should use a systematic process to evaluate the effectiveness of the training and 

qualification programs and to determine and direct the needed revisions. 

Review the guidance provided for each of the applicable program characteristics listed below. Then rate each 

characteristic using the specific guidance and record the information on the Summary Rating Sheet for this 

element. 

Applicable Program Characteristics: 

5.1 

5.2 

Trainee aitiques are reviewed to identG potential improvements to the programs. 

On-the-jobworkerperiencesareso~~tedfrom traineesand jobincumbentstoidentifyjobsltasksfor which lheyfelt 

inadequately prepared to perform. 

Information is solicited from supervisors on performance by new job incumbents to identify tasks for which they 

were inadequately prepared to perform. 

Information on degraded task performance is solicited from job incumbents and supervison. 

External factors (change actions) are evaluated to identify their impacts on job performance requirements. 

Changes in job performance requirements are resulting in changes to training and training materials. 

5.3 

5.4 

5.5 

5.6 

61 NUREG-1220, Rev. 1 
5.1 Trainee critiques are reviewed to identify potential improvements to the programs. 

GENERAL GUIDANCE 

Information on the course effectiveness and ways of improving training should be requested from trainees as they 

complete major segments of training. The information requested should focus on at least the following: 

0 

The information from the critiques should be used during regular program evaluations to identify areas for review and 

improvement. 

SPECIFIC GUIDANCE 

Adequacy of training in providing background knowledge, 

Adequacy of training in developing associated skills, 

Degree to which training is related to job requirements, and 

Degree to which training prepared trainees to fulfiil job requirements. 

Review course critiques to deterrnine if information is being collected and used to identify areas for improvement. 

Then rate this characteristic as: 

Erratically 

Generally 

Consistently 

if critiques are not being collected from trainees 

if critiques are being collected but do not address the training qualities listed in the general 

guidance, OR if the information collected is not being used as paa of program evaluation 

ifcritiquesate beingcoUectedAND thecritiquesaddressthe trainingqualitieslistedabove ANDthe 

critiques are being used as paa of progmm evaluation 

Record your rating of this characteristic on the Sununary Rating Sheet for SAT Element 5. 

NUREG-1220. Rev. 1 68 
5.2 On-the-'ob work experiences are solicited fsom trainees and job incumbents to identi@ 

jobs/tas k s for which they felt inadequately prepared to perform. 

GENERAL GUIDANCE 

hfomtion should be solicited from previous trainees three to six months after they complete training. The information 

solicited can help to identify the strengths and weaknesses of the training. The following types of information should be 

collected: 

SPECIFIC GUIDAh'CE 

Review information solicited from previous trainees to determine if information is being collected and evaluated on the 

areas listed above. 

Then rate this characteristic as: 

Unexpected dfiities in performing tasks on the job, 

Tasks that were particuIarly easy or difficult to perform, 

Additional training needed to do the job, 

Kinds of errors committed on the job, and 

Difference between the way tasks are performed on the job and the way they are taught. 

Erratically 

Generally 

Consistently 

if there is no formal program for soliciting information from job incumbents 

if the information is being collected but does not address the types of information listed in.the 

general guidance OR if the information is not being used as part of program evaluation 

if the information is being collected AND addresses the types of information listed in the general 

guidance AND is being used as part of program evaluation 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 5. 

69 NUREG-1220, Rev. 1 
5.3 Information is solicited from supervisors on performance by new job incumbents to identify 

tasks for which they were inadequately prepared to perform. 

GENERAL GUIDANCE 

information should be solicited periodically to determine how well the initial training program is preparing individuals to 

perform their jobs and to identify what continuiog training is needed for current job incumbents. The following types of 

information: 

0 

0 

0 

0 

SPECIFIC GUIDANCE 

Review information solicited from previous trainees to determine if information is being collected and evaluated on the 

areas listed above. 

Then rate this characteristic as: 

Tasks for which new job incumbents were inadequately prepared, 

Kinds of errors being committed by job incumbents, 

Suggestions for improvements in initial and continuing training, and 

Ewpected changes in job assignments, procedures or equipment. 

ErraticaUy 

Generally 

Consistently 

if there is no formal program for soliciting information from supervisors 

if the information is being collected but does not address the types of information listed in the 

general guidance OR if the information is not being used as part of program evaluation 

if the information is bemg collected AND addresses the types of information listed in the general 

guidance AND is being used as part of program evaluation 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 5. 

NUREG-1220, Rev. 1 70 
5.4 Information on degraded task performance is solicited from job incumbents and 

supervisors. 

GENERAL GUlDANCE 

Thequalityoftaskperfo~cemaydeclineoveraperiodof time.Tbiscanbetrueof bothsimpleanddifficult toperform 

task. Decline may be particularly evident for tasks that are difficult or infrequently performed. 

SPECIFIC GUIDANCE 

Review information solicited from previous trainees to determine if information is being collected to identify tasks where 

performance has degraded. 

Then rate this characteristic as: 

Emtidy 

Generally 

Consistently 

if there is no formal program for soliciting information from snpervisors and job incumbents 

if the information is being collected but is not being used as part of program evaluation 

if the information is being collected AND is being used as part of program evaluation 

Record your rating of this characteristic on the S&mary Rating Sheet for SAT Element 5. 

NUREG-1220, Rev. 1 
5.5 External factors (change actions) are evaluated to identify their impacts on job 

performance requirements. 

GENERAL GUIDANCE 

Extd factom (change acti0ns)incIudeindusy events, the installation of new equipment, and modifications to existing 

equipment. Each should be evaluated to identify their impact on the performance of related job tasks. 

SPECIFIC GUIDANCE 

Review the process used to evaluate the impact of change actions. Review completed evaluations to determine if the 

evaluation was done adequately. 

Then rate this characteristic as: 

Erraticaly 

GeneraUy 

Consistently 

if there is no evidence that change actions are being reviewed for impacts OR if the process is poor 

if the process for reviewing change actions exists but evaluations are not being completed for all 

change actions 

if all change actions are being evaluatedfor impacts on job performance requirements 

Record your rating of this characteristic on the Summary Rating Sheet for SAT Element 5. 

NUREG-1220, Rev. 1 72 
5.6 Changes in job performance requirements are resulting in changes in training and training 

materials. 

GENERAL GUIDANCE 

As changes to job performance requirements are identified, trainjng materials should be evaluated to determine where 

modifications might be needed. Any changes to training materials must be made mnsistently in all affected paas of the 

training program. Training information to support job performance requirements can be found in classroom, lab or 

workshop, simulator, and on-the-job training; therefore, ea& of those aspects should be evaluated and revised if 

appropriate. The most ulmmon weaknesses in this characteristic is that changes are not made to all affected haining 

materials. A method should be in place to track the needed changes until all are made. 

SPECIFIC GUIDANCE 

Review the results of training program evaluations to identi changes to lesson plans that have 

information to determine if it has also been changed. 

Then rate this characteristic as: 

been made within the last year. Compare the lesson materia ? s to supporting on-the-job training 

Erratically 

Generally 

Consistently 

if there is no evidence that changing job performance requirements are resulting in changes to 

training materials 

if changes are being made hut there is no method in place to ensure that changes are made to all 

appropriate training materials 

if changes are being made toreflect changing jobperformance requirements AND there isamethod 

in place to ensure that changes are made to all appropriate training materials 

Record your rating of this Characteristic on the Summary Rating Sheet for SAT Element 5. 

73 NUREG-1220. Rev. 1 
Thisdepicts the Summary RatingSheetforSATElement 5.Themasterofthisformcanbefoundin thepackageofforms 

included with this document. Record your rating on a copy of the master of this form. 

Form 3D-5 

Element 5 

Summary Rating Sheet 

REQUIRED ELEMENT- 10 CFR Parts 50.120 and 55.4 requires the evaluation of trainee mastery 

of objectives during training. 

Ratingscale: E- Erratically \ 

G- Generally 

C- Consistently 1 

See specific guidance listed with each characteristic. 

Applicable Program Characteristics: 

5.1 Trainee critiques are reviewed to identify potential improvements E G c 

5.2 On-the-job work experiences are solicited from trainees and E G C 

to the programs. 

job incumbents to identify jobs/tasks for which they felt 

inadequately prepared to perform. 

5.3 Information is solicited from supervisors on perfonnance by 

new job incumbents to identify tasks for which they were 

inadequately prepared to perform. 

Information on degraded task performance is solicited from 

job incumbents and supervisors. 

5.4 

E G c 

E G c 

5.5 External factors (change actions) are evaluated to identify their E G C 

5.6 Changes in job performance requirements are resulting in changes E G c 

impacts on job performance requirements. 

in training and training materials. 

Not all of the applicable characteristics are necessary for compliance with 10 CFR Parts 50.120 and 55.4. However, a 

trend toward an "erratic" rating may be the basis of a violation since it would indicate that a systems approach to training is 

either not implemented or poorly implemented. 

NUREG-1220, Rev. 1 14 
The results of a training evaluation should be reported A 

usins! the format outlined in Manual Cbauter 0610, "InIf the analvsis defiines uerformance trends that warrant 

s&on Reports." The report should d&ent the assessment of the effectiveness of the training programs 

based on an analysis of the information collected using 

this document. 

Both the interviews and the observations are designed to 

focus on the areas that would provide indications of reduced training effectiveness. It is important to maintain a 

broadperspectivewhenreviewingtheresultsof tbeinterviews and observations. Isolated performance problems 

are usually the result of individual deficiencies and may 

not reflect the effeetivenes of the training program as a 

whole. An analysis of the data collected in interviews and 

observations should focus on defining performance 

trends rather than on identifying individual performance 

deficiencies. 

If the analysis does not identify any trends, no further 

examination is needed. The report should be written 

based on the overall assessment of the information collected in the interviews and observations. 

additional &ation,'then the SATelements should be. 

evaluated. The list of applicable characteristics for each 

SAT element should be viewed broadly. Isolated problemswitha singlecharacteristicarelesofaconcemthan 

an"erratic" performance trendforseveralcharacteristics 

within a SAT element. 

?he "SAT Presentation Tree" shown in this section can 

be used to su-e the results of SAT element evaluations. Using the presentation tree makes trends visible 

and can be useful when presenting ioSpeaion &dings. 

The tree can be used to itlustrate fmdings from the review 

ofa single progmmor it can be used to summarize the 

results of an inspection of several programs. The master 

of this figure can be found in the package of forms included with this document. Record your findings on a 

copy of the master of this figure. 

The report should address the deficiencies in achieving 

the SAT elements and not at a failure to achieve characteristics. The results of the inspection wiU provide the 

necessaxy information to verify compliance with 10 CFR 

Parts 50.120 and 55 as applicable. 

75 

, 

NUREG-1220, Rev. 1 
SAT Presentation Tree 

- 

i? 

s 

Element 4 

Trelnee 

EvaluaUon 

~~] Systematic 

I I 

4.1 

- objeaivesates~~ 

related to job 

- Performance 

evaluallm 

4.2 

4.3 

- AemedlaUon 

I- 1- 

5.5 

- External 

factors 

5.6 

Task Menafled Tade objectbe8 Lesson plans 

1.2 2.3 

Task selected Objeclhres revlsed 

for tralnlng Content depth 

- 

&land I 7 %ndlUonsand I v3.3 I mntlnulng trafnlng standards Content sequence 

Jobperformance 

changes 

3.4 

Presentallon 

adequacy I 

Changes lnstruclors 

LEGEND 

Evaluate Characteristics 

Green - Conslstenlly - Functionlng well 

Red - ErreUcaay - Fundlonlng inadequately 

Yellow - Generally- Functionlng adequately 

1 

Element 5 

Evaluation 

E- Trainee aitlques 

Based on apprelsal Rndlngs aaslgn a 

ratlng for each element 
Form 2.1 

Worksheet for the Team Leader 

Planning a Training Inspection 

Six Weeks Before the Inspection (or as soon as an inspection has been initiated) 

Working with the Regional Office: 

-1. 

-2. 

-3. 

Contact the Human Factors Assessment Branch of NRR to obtain information 

about staff positions on current industry trends and other issues related to training. 

Determine the specific date(s) of the inspection. 

Determine which of the following training programs will be inspected 

a 

a 

a 

a 

a 

a 

a 

a 

a 

0 

Senior reactor operator/shift supeMsor 

Reactor operator 

Shift technical advisor 

Non-licensed operator 

Instrument and control technician 

Electrical maintenance personnel 

Mechanical maintenance personnel 

Radiological protection technician 

Chemistry technician 

Engineering support personnel 

-4. 

Five Weeks Before the Inspection 

- Send a letter tp the licensee announcing the planned inspection. The licensee should receive this 

letter no later than four weeks preceding the inspection. The letter should contain the following 

information: 

1. Date@) of the inspection 

2. 

3. 

Determine size and composition of inspection team. 

Training programs to be inspected 

Documents needed by the Inspection Team before it kives on site: 

0 

0 

Task lists for the position(s) being reviewed 

Administrative instructions and procedures used by the Training Department for the 

development, presentation, and revision of materials and operation of the Training 

Department. 

Audits or evaluations of the training organization 

Organization chart of the Training Department and the facility 

Schedule for requalification and initial training programs for the next three. months 

Course outline or lesson plans being presented during the planned inspection 

Shift schedule, including names, for the inspection week 

Training self-assessment, if done within the previous 6 months 

a 

a 

0 

0 

0 Selected station procedures 

0 

-Determine assignments for team members 

Develop a schedule for the inspection week 

A-1 NUREG-1220, Rev. 1 
Worksheet for Planning an Inspection (continued) 

Four Weeks Before the Inspection 

-Assign a team member to handle travel arrangements. 

Two Weeks Before the Inspection 

1. 

-2. 

Ensure licensee documents have been received and distributed to Inspection Team members. 

Contact the licensee to discuss the following: 

0 

0 

0 

Schedule for the entrance and exit meetings Plant access, as necessary 

Size of the inspection team 

Availability of facility personnel from the following categories for interviews: 

- Training supervisor 

- Certified instructors 

- Plant supervisors, as appropriate 

-Job incumbents 

0 The availability of a dedicated plant employee, preferably a person from the Training 

Department, to act as a liaison for the inspection team 

Any special requirements the team may have 

3. Distribute assignments and schedules to the team members 

One Week Before the Inspection 

With the team: 

- 1. 

2. 

- 3. 

_. 4. 

5. 

Review the inspection schedule. 

Clarify and review any issues raised by Human Factors Assessment Branch. 

Review documentation sent by the licensee. 

Develop technical questions, if needed, for use in interviews 

Review the results of any previous inspection findings involving training. 

NUREG-1220. Rev. 1 A-2 
FOIIU 3B-1 

Checklist for Observing Behaviors in the Classroom 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not obsewed at all or if it is observed but is below 

minimum standards 

G - GeneralJy - if a behavior is observed but is not being done Consistenfly or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistenthr - if the behavior is always being done and the quality always exceeds 

Rating Scale: 

minimum &dads 

Trainee Behaviorsmork Practices 

1. 

2. 

Trainee BehaviordTechnical Knowledge 

1. 

Maintains appropriate classroom demeanor during 

training. (3) 

Participates in the discussions and activities. (3) 

Exhibits understanding of technical information 

received acknowledging and questioning information 

provided by the instructor. (2x3) 

Asks appropriate questions of instructor during 

training. (3) 

2. 

Instructor Behaviors/Work Practices 

1. 

2. 

3. 

Reviews objectives with trainees. (2x3) 

Uses training time effectively by 

controlling the class and the instructional flow. (3) 

Presents lesson at an appropriate. pace for the 

content. (1)(2)(3) 

A-3 

E G C 

E G C 

E G C 

E G C 

E G C 

E G C 

E G- C 

NUREG-1220, Rev. I 
Classroom Checklist (continued) 

Ratingscale: E- Erratically 

G- Generally 

C- Consistently 

4. 

5. 

6. 

7. 

Adheres to the content and structure of the 

instructor guide or lesson plan. (1)(2)(3) 

Periodically summarizes material to emphasize 

the relationships among topics. (2x3) 

Gives students handouts and uses instructional 

materials that are related to the topic and are of good 

Uses questions to determine trainee understanding 

of material being presented. (2)(3)(4) 

quality. (lX2X3) 

Instructor Behaviors/Technical Knowledge 

1. 

2 

3. 

4. 

5. 

6. 

Presents information at a level consistent 

with the abilities of the trainees as evidenced by 

appropriate trainee response. (1)(2X3) 

Presents information to the appropriate 

depth of knowledge. (1)(2)(3) 

Exhibits familiarity with plant procedures 

and references by refemng to them accurately. (2x3) 

Answers questions promptly and accurately. (1)(2)(3) 

Answers questions with an appropriate 

level of detail. (1)(2)(3) 

Questions trainees to determine their 

understanding of materials being presented and 

the progress they are making. (2X3X4x5) 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

NUREG-12u). Rev. 1 A4 
Form 3B-2 

Checklist for Observing Behaviors in the Simulator 

Program: 

Date: 

Observer: 

Instructor. 

Instructions: 

Rating Scale 

Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistently - if the behavior is always being done and the quality always exceeds 

minimum standards 

Trainee Behaviors/Work Practices 

1. 

2 

3. 

4. 

5. 

6. 

Demonstrates knowledge and familiarity with plant E 

procedures by correctly selecting and using reference 

materials. (2x3) 

provided on procedural adherence. (4) 

during training. (3x4) 

Gives clear and accurate information to the 

shift supervisor and other crew members. (3x4) 

Refers to procedure and procedure steps 

when appropriate. (3x4) 

Uses repeat-backs consistently and appropriately. (3x4) 

Uses procedures in accordance with the guidance 

Maintains appropriate control room demeanor 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G 'C 

G C 

A-5 NUREG-1220, Rev. 1 
Simulator Checklist (continued) 

Rating Scale: E - Erratically 

G- Generally 

C - Consistently 

Trainee Behaviors/Technical Knowledge 

1. 

2. 

3. 

4. 

5. 

Notices and acknowledges alarms. (3x4) 

Demonstrates familiarity with physical layout 

of control room by quickly locating relevant 

instruments, indicators, and controls. (1)(2)(3)(4) 

Exhibits understanding of technical information 

received by acknowledging and questioning information 

provided by the instructor or other crew members. (2X3x4) 

Asks appropriate questions of instructors and 

other crew members during training segments. (3x4) 

Demonstrates technical competence by responding 

correctly to indicators, instruments, and alarms. (3x4) 

Instructor Behaviorsmork Practices 

1. 

2 

3. 

Instructor BehaviorslTechnical Knowledge 

Reviews objectives with trainees. (2x3) 

Uses training time effectively by controlling 

the session and instructional flow. (3) 

Directs post-training critiques to maintain 

focus on self-evaluation by individuals and teams. (3x4) 

1. 

2. 

3. 

4. 

Answers questions promptly and accurately. (3) 

Exhibits good coaching skills. (3) 

Demonstrates adequate knowledge of control 

room personnel responsibilities by providing 

appropriate evaluation feedback during critiques 

on mastery of objectives. (2x314) 

Questions trainees to determine their 

understanding of material being presented and the 

progress they are making. (2X3x4) 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

NUREG-1220, Rev. 1 Ad 
Form 3B-3 

Checklist for Observing Behaviors 

During On-the-Job Training 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistently - if the behavior is always being done and the quality always exceeds 

minimum standards 

Rating Scale: 

Trainee Bebaviorsmork Practices 

1. Demonstrates knowledge and familiarity with plant 

procedures by correctly selecting and using reference 

materials. (2x3) 

Refers to procedure and procedure steps 

when required. (3x4) 

2. 

Trainee Behaviors/Technical Knowledge 

1. Demonstrates familiarity with physical layout 

of work area by quickly locating relevant instruments, 

indicators, and controls. (1X2X3)(4) 

Exhibits understanding of technical information 

received by acknowledging or questioning information 

provided by the instructor. (2x3) 

Asks appropriate questions of instructors during 

training segments. (3x4) 

Demonstrates technical competence by responding 

correctly to indicators, instruments, and alarms. (3x4) 

Questions unusual or unexplained circumstances. 

Selects appropriate tools and uses them correctly. 

2. 

3. 

4. 

5. 

6. 

E 

E 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

A-7 NUREG-12M). Rev. 1 
On-the-Job Checklist (continued) 

Rating Scale: E- Erratically 

G- Generally 

C- Consistently 

Instructor Behaviorsmork Practices 

1. Reviews objectives with trainees. (2x3) 

2 Uses training time effectively. (3) 

Instructor BehaviorsRechnid Knowledge 

1. 

2. 

3. 

Answers questions promptly and accurately. (3) 

Exhibits good coaching skills. (3) 

Demonstrates adequate knowledge of task 

performance by providing appropriate evaluation 

feedback during training session on mastery of 

objectives. (2)(3)(4) 

Questions trainees to determine their 

understanding of material being presented and the 

progress they are mhking. (2X3X4) 

4. 

E G C 

E G C 

E G C 

E G C 

E G C 

E. G C 

NUREG-1220, Rev. 1 A-8 
Form 3B-4 

Checklist for Observing Behaviors During Work In Progress 

Program: 

Date: 

Observer: 

Instructor: 

Instructions: Evaluate the overall quality of classroom training by rating each of the behaviors listed 

below. Use the results recorded on this form to determine if evaluation of the SAT 

process is warranted. 

E - Erratically - if a behavior is not observed at all or if it is observed but is below 

minimum standards 

G - Generally - if a behavior is observed but is not being done consistently or if it is 

consistently done but the quality does not always meet minimum standards 

C - Consistently - if the behavior is always being done and the quality always exceeds 

minimum standards 

Rating Scale: 

Worker Behaviorslwork Practices 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

Demonstrates knowledge and familiarity with plant 

procedures by correctly selecting and using reference 

materials. (1)(2)(3) 

Refers to procedure and procedure steps 

when required. (1)(2)(3) 

FoUows procedure steps. (1)(2)(3) 

Questions unusual or unexptained circumstances. 

Follows appropriate radiation protection 

practices. (1)(2)(3) 

Works only within the scope of work package. 

Provides clear and accurate information to others. 

Requests assistance or clarification from 

supexvisor or manager when procedural or technical 

problems are encountered. 

Uses only “approved/controlled” procedures and 

drawings to perform work. (1)(2)(3) 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

A-9 NUREG-1220, Rev. 1 
Work-in-Progress Checklist (continued) 

Rating Scale: E - Erratically 

G- Generally 

C - Consistently 

Worker BehaviorslTechnical Knowledge 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

Locates appropriate component correctly. (1)(3)(4) 

Demonstrates familiarity with physical layout of 

work area by quickly locating relevant instruments, 

indicators, and controls. (1)(2)(3) 

Uses appropriate work techniques correctly. (4) 

Records data in appropriate location. (4) 

Selects appropriate tools and equipment. (4) 

Uses appropriate tools correctIy. (4) 

Uses appropriate equipment correctly. (4) 

Responds appropriately to unusual or unexplained 

circumstances. (1x4) 

Supervisor BehaviorslWork Practices 

1. 

2 

Gives worker complete work package. 

Selects workers who have been trained and are 

qualified to perform work package tasks. (4) 

Provides a pre-job briefing that stresses safety, 

rad con, work staging, and quality standards. 

Monitors work in progress by visiting work site. (5) 

Answers questions and responds appropriately to 

requests from workers. (5) 

3. 

4. 

5. 

Supervisor BehaviorslTechnicai Knowledge 

1. Demonstrates familiarity with physical layout of 

work area by quickly locating relevant instruments, 

indicators, and controls. (3) 

Questions unusual or unexplained circumstances. 

Responds appropriately to unusual or unexplained 

circumstances. 

2 

3. 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

G C 

E G C 

E G C 

E G C 

NUREG-1220, Rev. 1 A-10 
Form 3C-1 

Instructor Interview Protocol 

Propram: 

Date: 

Interviewer: 

Instructor: 

General Information 

1. What work did you do at this facility before you became an instructor? (3) 

2 What are your major duties and responsibilities in your present position? (3) 

3. In which areas are you certified to instruct? (3) 

4. What skills does your instructor certification address? (3) 

(technical skills, instructional skills, etc.) 

Initial, Continuing, and On-theJob Training 

5. What materials do trainees find difficult and/or easy? (5) 

6. How accurate and complete are the training materials? (1)(2)(3)(5) 

A-11 NUREG-12U). Rev. 1 
Instructor Interview (continued) 

7. In what ways could the training materials, training aids, and instructional settings be 

improved? (3x5) 

8. What area of training do you consider the strongest, and what area of training do you consider the 

weakest? (4x5) 

9. How do you know that your training materials are current and correct? (Do the materials reflect 

modifications and procedural changes?) (1)(2)(5) 

10. Who conducts on-the-job training and evaluations? (3x4) 

11. How are trainers and evaluators qualified to conduct on-the-job training and evaluations? (3x4) 

Trainee and Program Evaluation and Program Revision 

12. Who is responsible for identirying training needs? (lX4X5) 

l3. What method is used to collect and resolve comments and ideas from trainees and 

SUpeMSOrS? (4x5) 

14. How do you follow up on training effectiveness? (5) 

NUREG-12M). Rw. 1 A-12 
Form 3C-2 

Job Incumbent Interview Protocol 

Program: 

Date: 

Interviewer: 

Job Incumbent: 

General Information 

1. How long have you worked in the (operations, maintenance, chemistry, etc.) department? (5) 

2. Did you qualify in your present job through training or through exemptions (grandfathering)? 

(NOTE TO INTERVIEWER: IF BY EXEMPTIONS-SKIP QUESTIONS 5-9) (4) 

3. What tasks do you find particularly easy to perform? .What makes them easy? (5) 

4. What tasks do you find particularly difficult to perform? What makes them difficult? (q 

Initial Training 

5. How could initial training have prepared you better for your job? (1)(2)(3)(4)(5) 

6. How relevant is the initial training program to your job? (1x5) 

A-l3 NUREG-1220, Rev. 1 
Job Incumbent Interview (continued) 

7. How well was the initial training timed to meet actual job demands? (3) 

8. What specific topics or tasks need more emphasis during initial training? (1)(2)(3)(5) 

9. What differences have you noticed between the training you received and what is expected 

from you on the job? (lX4X5) 

Continuing Training 

10. What training have you been given for changes in procedures or equipment that occurred in 

your job since you were first assigned to it? (1)(2)(3)(4)(5) 

11. What continuing training have you received since being assigned to your job? (4x5) 

12. What additional continuing training topics do you need for your job? (1x5) 

13. How adequate is the training you have received during continuing training? (5) 

Trainee Evaluation 

14. If you ever failed an initial training exam, what happened? (4) 

NUREG-1220. Rev. 1 A-14 
Job Incumbent Interview (continued) 

15. If you ever failed a continuing training exam, what happened? (4) 

16. How prompt and objective is the feedback about your perfonnanee during training? (4) 

17. How do you feel the feedback process could be improved? (4x5) 

Program Evaluation 

18. When you have been asked to evaluate training provided to you, were your comments resolved 

to your satisfaction? (5) 

19. What improvements or increased emphasis have you noticed in the training program OV~T the 

last 2 years? (5) 

20. What suggestions and recommendations have you noticed that were incorporated into the 

training program? (5) 

21. How would you improve training? (5) 

22. Overall, how effective was your training? (4x5) 

A-15 NUREG-1220, Rev. 1 
Form 3C-3 

Plant Supervisor Interview Protocol 

Program: 

Date: 

Interviewer: 

Plant Supervisor: 

General Information 

1. What work did you do at this facility before you became a plant supervisor? 

2. What are your major duties and responsibilities in your present position? 

Initial Trainiig 

3. What tasks are newly trained employees allowed to perform? (4x5) 

What is the basis of this decision? (4x5) 

How is this enforced or controlled? (5) 

4. How do recently trained employees compare to those who received earlier training? (4x5) 

What is the basis of your comparison? (4x5) 

NUREG-12U). Rev. 1 A-16 
Plant Supervisor Interview (continued) 

5. What additional training do job incumbents receive once they are on the job? (1X3X4) 

What is the basis of that decision? (1x5) 

6. How are changes in job assignments, procedures, and equipment reflected in training? (1x5) 

On-the-Job Training 

7. How do you make assignments to on-the-job training instructors? (3x5) 

8. How do you make assignments to on-the-job evaluators? (3x5) 

9. What are the strengths and weaknesses of the on-the-job training program in your area? (5) 

10. How do you evaluate the training received by your workers? (5) 

11. To what extent has the observed performance of workers met your expectations? (5) 

Program Evaluation and Revision 

12. What new problems has training created? (1x215) 

A-17 NUREG-IW), Rev. 1 
Plant Supervisor Interview (continued) 

U. What current training do you consider unnecessary? Why? (1)(2x4)(5) 

14. How do your comments on training quality and effectiveness get resolved? (5) 

15 How do you veri9 that training is job-specific and plant-specific? (1)(2)(4)(5) 

16. Overall, how effective is the training program? (5) 

NUREG-1220, RW. 1 A-18 
Form 3C-4 

Training Supervisor Interview Protocol 

Program: 

Date: 

Interviewer: 

Training Supervisor: 

General Information 

1. What work did you do at this facility before you became a training supervisor? (3) 

2 What are your major duties and responsibilities in your present position? (3) 

Initial Training 

3. What involvement do you have in developing and implementing initial training? (3) 

4. What specific areas within the development and implementation of initial training need to be 

improved? (5) 

5. What aspects of the initial training program do you consider the strongest, and what aspects do 

you consider the weakest? (5) 

A-19 NUREG-1220, Rev. 1 
Training Supervisor Interview (continued) 

Continuing Training 

6. What involvement do you have in developing and implementing continuing training? (3) 

7. What aspects of the continuing training program do you consider the strongest and what 

aspects do you consider the weakest? (5) 

On-th+Job Training 

8. What is the Training Department’s role in implementation of the on-thejob training and 

evaluation program? (1)(2)(3)(4X5) 

Trainee Evaluation 

9. How do you find out how satisfied you, line organization supervisors are with trainee 

performance? (5) 

10. How are comments from supervisors and trainees on training content and effectiveness 

resolved? (5) 

Program Evaluation and Revision 

11. What improvements or increased emphasis in the overall area of training have you noticed 

within the last 2 years? (lXZX3X5) 

NUREG-UXJ. Rev. 1 A-20 
Training Supervisor Inteniew (continued) 

12. How do changes in job assignment, procedures, and equipment get reflected in training? 

(1X2X3X4X5) 

U. How are training needs identified? (1x5) 

A-21 WG-1220, Rev. 1 
Interview And Observation 

Tally Sheet 

Review the information obtained in interviews and observations. Identify observation items where performance was rated 

“erratic” and intemiew questions where answers suggest that a problem may exist. Use this sheet to taUy the references to 

the related systems approach to training (SAT) elements which are coded at the end of each item. Based on this summary 

identify the SAT elements that require additionat review. Use the directions and evaluation criteria listed with that 

element in Section 3D, “SAT Element Evaluation,” to mmplete the review. 

INTERVIEWS 

Instructors 

Job Incumbents 

Plant Supervisor 

Training Supervisor 

NUREG-1220, Rev. 1 A-22 
Form 3D-1 

Element 1 Summary Rating Sheet 

REQUIRED ELEMENT - 10 CFR Parts 50.120 and 55.4 requires a systematic analysis of the job 

to be performed. 

RatingScale: E- Erratically \ 

G- Generally 

C- Consistently / 

See specific guidance listed with each characteristic. 

Applicable Pmgram Charaderistics: 

1.1 

1.2 

13 

1.4 

1.5 

Systematic method wed to identify job tasks? 

Objective and consistent method used to select tasks for training? 

Tasks differentiated for initial and continuing training? 

Analysis adequate for development of learning objectives? 

New or changes tasks analyzed to identify training needs? 

E 

E 

E 

E 

E 

0 C 

G C 

G C 

G C 

G C 

A-23 WG-1220, Rev. 1 
Form 3D-2 

Element 2 

Summary Rating Sheet 

REQUIRED ELEMENT - 10 CFR Parts 50.120 and 55.4 requires tbat learning objectives be derived from 

(he analysii hi& describes the desired performance after lraining. 

RatingScale: E- Emtially \ 

G- Generally 

C- Consistently / 

See speafic guidance listed with each charaaeristicApplicable Program charpderistici: 

2.1 

2.2 

Learning objectives exist for tasks selected for training? 

Learning objectives conditions and standards reflect job performance 

requirements. 

23 Learning objectives are revised as needed to reflect job changes. 

E G C 

E G C 

E G C 

A-24 
Fo~ 3D-3 

Element 3 

Summary Rating Sheet 

REQUIRED ELEMENT- 10 CFR Parts 50.120 and 55.4 requires that training design and 

implementation be based on the learning objectives. 

RatingScaIe: E- Emtidy \ 

G- GeneC- Consistently / 

See specific guidance listed with each characteristic. 

Applicable Program Characteristics: 

1. 

2. 

3. 

4. 

5. 

Lesson plans are structured to provide for 

consistent presentation. 

Depth of content is adequate to support mastery 

of learning objectives. 

Information is presented in a sequence (withii 

a lesson as well as among several lessons) that supports effective 

learning. 

Trainiig is adequately presented. 

Personnel providing classroom, on-the-job, laboratory, and 

simulator training are proficient in the methods and techniques 

for sudul presentation in the particuiar setting. 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

A-25 NUREG-1220, Rev. 1 
Form 3D-4 

Element 4 

Summary Rating Sheet 

REQUIRED ELEMENT- 10 CFR Parts 50.120 and 55.4 requires the evaluation of trainee mastery of objectives during training. 

RatingScale: E- Erratically \ 

G- Generally 

c - Consistently / 

See sp&c guidance listed with each characteristic. 

Applicable Program Charaderistics: 

4.1 

4:2 

4.3 

4.4 

4.5 

4.6 

A relationship exists between job performance 

requirements and learning objectives and test items. 

Trainee performance is evaluated regularly. 

Remediation is provided when appropriate. 

Continuing training contains performance 

requirements for difficult, important, or infrequent tasks. 

Training and task performance exemptions are 

objectively based. 

Evaluations of task performance test the 

trainees’ mastely of job performance requirements. 

E G 

E G 

E G 

E G 

E G 

E G C 

NUREG-1220, Rev. 1 A-26 
FOIXXI 3D-5 

Element 5 

Summary Rating Sheet 

REQUIRED ELEMENT- 10 CFR Parts 50.120 and 55.4 requires the evaluation of trainee mastery 

of objectives during training. 

Ratingscale: E- Emtically \ 

G- Generally 

C- Consistently /’ 

See s@ic guidance listed with each characteristic. 

Applicable Program Charaderistiu: 

5.1 Trainee critiques are reviewed to identify potential improvements 

to the programs. 

On-the-job work experiences are solicited from trainees and 

job incumbents to identify jobsltasks for which they felt 

inadequately prepared to perform. 

Information is Solicited from supervisors on performance by 

new job incumbents to identify tasks for which they were 

inadequately prepared to perform. 

Information on degraded task performance is soliciwd from 

job incumbents and supervisors. 

External factors (change actions) are evaluat& to identify their 

impacts on job performance requirements. 

Changes in job performance requirements are resulting in changes 

in training and training materials. 

5.2 

5.3 

5.4 

5.5 

5.6 

E 

E 

E 

E 

E 

E 

G C 

G C 

G C 

G C 

G C 

G C 

A-27 NUREG-1220, Rev. 1 
aa 

aa 

I-" 

L 

I I I I I 

NUREG-KZO, Rev. I A-28 
NRC FORM 335 U.5. NUCLEAR REGULATORY COMMISSION 

1. RW&Tli!lEA "~,,~ -,, 1102. nd-m Numhs.nm,y 

3201,3202 BIBLIOGRAPHIC DATA SHEET 

ISSS insrmctiotw M the re-1 NUREG-1220 

1 Rev. 1 ?.TITLE AN0 SUBTITLE 

Training Review Criteria and Procedures 3. DATE REPORTPUBLISHED 

MONTH YEAR 

January ' 1993 

4. FINORGRANTNUMBER 

5. AUMOR(S1 6. TYPE OF REPORT 

7. PERIOD COVEREOl!nr!ulin D.W 

Division of Licensee Performance and Quality Evaluation 

Office of Nuclear Reactor Regulation 

U.S. Nuclear Regulatory Commission 

Washington, DC 20555 

%SPONSORING ORGANIZATION - NAME AND ADDRESS I1INRC.~~5".~.bo~:ilmnrrm~~raWsNRCDnkion. Dffirrwn~.us Nuchrnw&mC=mmW. 

mdMths.dh.lrl 

Same as 8, above. 

+ 

10. WPPLEMENTARY NOTES 

11. ABSTRACT IXQmrmorksl 

This document provides direction to NRC personnel for reviewing training programs at 

nuclear power plants to verify compliance with the requirements of 10 CFR 50.120 and 

10 CFR 55 as applicable. 

training programs, provides aids for collection of information during interviews and 

observations, and provides criteria for evaluating the implementation of a systems 

approach to training. This document is not intended to have the effect of a 

regulation, it establishes no binding requirements or interpretations of NRC 

regulations. It is intended as guidance only. 

It describes the process for evaluating the effectiveness of 

training programs 

training review 

I Unlimited 14. SECURlTV CLASIFICATION 

Unclassified I ITM' Rcponl 

15. NUMBER OF PAGES 

16. PRICE 

I I 

NRC FORM 335 nasi 
Federal Recycling Program