EVALUATION REPORT 

Evaluation of NRC’s Training 

and Development Program 

OIG-08-A-13 July 16, 2008 

All publicly available OIG reports (including this report) are accessible through 

NRC’s Web site at: 

http:/www.nrc.gov/reading-rm/doc-collections/insp-gen/ 
UNITED STATES 

NUCLEAR REGULATORY COMMISSION 

WASHINGTON, D.C. 20555-0001 

OFFICE OF THE 

INSPECTOR GENERAL 

 July 16, 2008 

MEMORANDUM TO: R. William Borchardt 

Executive Director for Operations 

FROM: Stephen D. Dingbaum /RA/ 

 Assistant Inspector General for Audits 

SUBJECT: EVALUATION OF NRC’S TRAINING AND DEVELOPMENT 

PROGRAM (OIG-08-A-13) 

Attached is the Office of the Inspector General’s (OIG) audit report titled, Evaluation of 

NRC’s Training and Development Program. 

This report presents the results of the subject evaluation performed by Booz Allen 

Hamilton, Inc., under contract to the NRC OIG. Agency comments provided at the exit 

conference on June 9, 2008, have been incorporated, as appropriate, into this report. 

Please provide information on actions taken or planned on each of the 

recommendations within 30 days of the date of this memorandum. Actions taken or 

planned are subject to OIG follow up as stated in Management Directive 6.1. 

We appreciate the cooperation extended to us by members of your staff during the 

evaluation. If you have any questions or comments about our report, please contact me 

at 415-5915 or Steven Zane, Team Leader, Financial and Administrative Audit Team, 

at 415-5912. 

Attachment: As stated 
Electronic Distribution

Frank P. Gillespie, Executive Director, Advisory Committee on Reactor 

 Safeguards/Advisory Committee on Nuclear Waste 

E. Roy Hawkens, Chief Administrative Judge, Atomic Safety and 

 Licensing Board Panel 

Karen D. Cyr, General Counsel 

John F. Cordes, Jr., Director, Office of Commission Appellate Adjudication 

Jim E. Dyer, Chief Financial Officer 

Margaret M. Doane, Director, Office of International Programs 

Rebecca L. Schmidt, Director, Office of Congressional Affairs 

Eliot B. Brenner, Director, Office of Public Affairs 

Annette Vietti-Cook, Secretary of the Commission 

Bruce S. Mallett, Deputy Executive Director for Reactor 

 and Preparedness Programs, OEDO 

Martin J. Virgilio, Deputy Executive Director for Materials, Waste, Research, 

 State, Tribal, and Compliance Programs, OEDO 

Darren B. Ash, Deputy Executive Director for Information Services 

 and Chief Information Officer, OEDO 

Vonna L. Ordaz, Assistant for Operations, OEDO 

Timothy F. Hagan, Director, Office of Administration 

Cynthia A. Carpenter, Director, Office of Enforcement 

Charles L. Miller, Director, Office of Federal and State Materials 

 and Environmental Management Programs 

Guy P. Caputo, Director, Office of Investigations 

Thomas M. Boyce, Director, Office of Information Services 

James F. McDermott, Director, Office of Human Resources 

Michael R. Johnson, Director, Office of New Reactors 

Michael F. Weber, Director, Office of Nuclear Material Safety and Safeguards 

Eric J. Leeds, Director, Office of Nuclear Reactor Regulation 

Brian W. Sheron, Director, Office of Nuclear Regulatory Research 

Corenthis B. Kelley, Director, Office of Small Business and Civil Rights 

Roy P. Zimmerman, Director, Office of Nuclear Security and Incident Response 

Samuel J. Collins, Regional Administrator, Region I 

Luis A. Reyes, Regional Administrator, Region II 

James L. Caldwell, Regional Administrator, Region III 

Elmo E. Collins, Jr., Regional Administrator, Region IV 
 

Evaluation of the Nuclear Regulatory Commission’s 

Training and Development Program 

Contract Number: GS-23F-9755H 

June 19, 2008 

The views, opinions, and findings contained in this report are those of the authors and should not be construed 

as an official Nuclear Regulatory Commission position, policy, or decision, unless so designated by other official 

documentation. 
 

[Page intentionally left blank] 
i 

EXECUTIVE SUMMARY 

Background 

The Nuclear Regulatory Commission (NRC) Strategic Plan highlights the need 

for the agency to provide a robust program of training, development, and 

knowledge transfer to ensure that agency employees attain and sustain the skills 

needed to implement the agency’s mission.1 NRC Human Resources Training 

and Development (HRTD) offers a program of training and development for NRC 

employees through the Professional Development Center (PDC) in Bethesda, 

MD, and at the Technical Training Center (TTC) in Chattanooga, TN. Course 

topics range from highly technical nuclear reactor and nuclear materials courses 

to leadership development to desktop computer skills. 

While HRTD has successfully met the majority of NRC’s training needs, the 

agency has identified a number of challenges that may impact the ability of the 

program to sustain the level of support it has provided to the agency thus far. 

Many of these challenges are the result of a planned increase of 600 new 

employees by 20092

 to address the expected growth in licensing and 

construction of new nuclear power plants to meet the Nation’s increased demand 

for energy production. Other challenges that will impact training and 

development includes the need for new skills in threat analysis and emergency 

preparedness, the introduction of new nuclear power technologies, increased 

licensing and regulatory activities to renew existing power plant licenses, 

increased need to train Agreement State personnel, impending attrition due to 

retirements, and a new emphasis on telecommuting. 

 

1

 United States, Nuclear Regulatory Commission, Strategic Plan Fiscal Years 2008-2013 (2003) 19. 

2 Dale E. Klein. Remarks Prepared for GE Nuclear Innovations Conference, Orlando, FL, 6 June 2007. 
ii 

Purpose 

Due to the elevated significance of training, the Office of the Inspector General 

(OIG) identified a need to evaluate NRC’s Training and Development Program to 

determine its effectiveness to meet current and future needs. 

Results in Brief 

The evaluation team reviewed relevant existing reports on the Training and 

Development Program and conducted qualitative research consisting of one-onone interviews with knowledgeable HRTD staff members, agency employees, 

managers, and executives, and found that: 

• Efforts to maintain training documents, track changes, and control 

versions are not consistent across all branches within HRTD. 

• Regional offices have difficulties in scheduling new and current employees 

for training. Additionally, some course schedules increase travel costs 

and time away from job responsibilities. 

• HRTD has limited ability to demonstrate the impact of its Training and 

Development Program on NRC’s mission using current performance 

measures. 

• Current evaluation strategies are limited in their ability to demonstrate the 

effectiveness of training. 

• HRTD’s continued reliance on the traditional classroom as the primary 

delivery method for training is being strained by an increasing number of 

students as well as new training needs. 

• In order to move into other forms of training delivery, HRTD will need to 

enhance or supplement the skill sets of its staff. 
iii 

Efforts to maintain training documents, track changes, and control 

versions are not consistent across all branches within HRTD. 

HRTD has developed operating procedures to define its processes for the 

development and management of training content. The evaluation team 

observed that content management practices varied among branches and 

instructors at both the TTC and PDC. Inconsistent and incomplete content 

management practices and tools have hampered HRTD’s ability to manage and 

control content. The inability to consistently manage and control content impacts 

HRTD’s ability to provide effective and relevant training and development 

programs. 

Regional offices have difficulties in scheduling new and current employees 

for training. Additionally, some course schedules increase travel costs and 

time away from job responsibilities. 

HRTD provides most of the formal training courses required for NRC qualification 

programs and for leadership training. Staff in the regional offices stated that it is 

often difficult for employees to schedule and attend the training that is necessary 

to attain or maintain qualifications or that is required for new supervisors. In 

addition, increasing travel costs have made it difficult to send employees to 

necessary training, especially when the training requires frequent, short-term 

trips to the Washington, DC, area. HRTD’s annual training needs assessment 

has not been highly effective in getting the information needed from the regions 

in order to schedule training. Employees’ inability to schedule training in a timely 

fashion has delayed the acquisition of skills needed to perform their jobs and 

meet qualification requirements. 
iv 

HRTD has limited ability to demonstrate the impact of its Training and 

Development Program on NRC’s mission using current performance 

measures. 

To show its value to the organization, the Training and Development Program 

must show how its training solutions link to strategic outcomes and if the 

solutions justify the cost. HRTD’s current performance metrics fail to provide 

meaningful insight into how its programs align with the agency’s mission and 

strategic goals, or demonstrate training efficiency and effectiveness. HRTD has 

encountered challenges in producing credible, reliable, and consistent 

performance measures. Without effective performance metrics, HRTD is unable 

to demonstrate its contribution to NRC’s mission and risks future support and 

funding. 

Current evaluation strategies are limited in their ability to demonstrate the 

effectiveness of training. 

A systematic evaluation approach and process allows for continuous 

improvement within the training function when aligned with organizational goals. 

NRC does not use a systematic evaluation process for continuously improving its 

training and development program because it has not: 

• Developed a strategy for collecting pertinent evaluation data. 

• Finalized the draft evaluation procedure. 

• Evaluated the current Learning Management System (LMS) capability for 

data collection. 

Without a strategy that standardizes the training evaluation procedure and 

metrics, NRC cannot accurately measure and monitor its achievement of goals. 
v 

HRTD’s continued reliance on the traditional classroom as the primary 

delivery method for training is being strained by an increasing number of 

students as well as new training needs. 

NRC is facing workforce challenges that make e-learning a viable alternative to 

the traditional classroom, which is currently used for the majority of courses. 

HRTD has recognized the benefit that new training methodologies could bring to 

the agency and has deployed a new LMS which expands on-line learning 

capabilities. However, HRTD has not developed an e-learning strategy or 

implementation plan to maximize the benefits of this investment and ensure it is 

helping HRTD meet the agency’s most pressing training needs. HRTD staff is 

strained to meet all of the training needs in the traditional classroom with its 

current staff and facility resources, but additional funding to support e-learning 

initiatives may be difficult to obtain without a defined strategy and implementation 

plan. 

In order to move into other forms of training delivery, HRTD will need to 

enhance or supplement the skill sets of its staff. 

Organizations that have implemented e-learning solutions have found the skills 

needed to develop technology-enabled learning are very different from those 

needed for classroom instruction. Staff in HRTD noted that there are limited 

skills and resources for the development and deployment of e-learning among 

the current staff. If NRC plans to implement more e-learning solutions, HRTD will 

need to find ways to close the e-learning skill gaps—either through training, 

hiring, or outsourcing. Attempting to develop effective e-learning programs 

without the necessary skills can lead to frustration for the staff and ineffective 

learning opportunities for employees. 

Recommendations: 

We recommend that the Executive Director for Operations: 
vi 

1. Review the current course files to identify gaps in necessary documentation. 

2. Develop a plan and timeline for completing any missing course 

documentation. 

3. Complete HRTD Operating Procedure 404 – Training Material Control, to 

include a standard process for version control, tracking changes, and 

assigning accountability for changes. 

4. Develop a plan to centralize course materials in one location, preferably a 

central repository on a shared server. 

5. Develop a lifecycle maintenance procedure for periodically reviewing 

courses to ensure the learning objectives are still valid, the delivery 

mechanism is still the most effective, or the course is still meeting 

stakeholder needs. 

6. Form a working group of select representatives from the program offices, 

regional offices, and HRTD to conduct a thorough review of the training 

calendar including location and frequency of courses, travel costs, and 

opportunities for efficiencies in course sequencing (i.e., course bundling). 

The group should identify problems and develop mutually agreed-upon 

solutions within the constraints of contractual agreements, budgets, and 

resources. 

7. Within contractual limitations, identify and schedule courses at the same 

time each year so employees can anticipate the availability of courses. 

Communicate these courses and dates widely to make sure employees, 

supervisors, training coordinators, and managers are aware of the annual 

schedule. 

8. Before implementing the enforcement of course prerequisites in the 

Learning Management System: 

a. Determine the impact on employees’ ability to take all required training 

within the allotted timeframe. 

b. Communicate the change to NRC personnel in advance and allow 

opportunities for feedback. 
vii 

9. Develop and implement new performance metrics to demonstrate mission 

alignment, effectiveness, and efficiency. 

10. Develop and implement a plan to leverage the capabilities of the LMS for 

collection and reporting of chosen metrics. Specifically, evaluate the 

competency model capabilities to determine if they meet NRC’s needs, 

including identifying competencies, linking courses (or course modules and 

learning objectives) to identified competencies, and closing critical skill 

gaps. 

11. Develop and implement a comprehensive cost tracking capability (including 

cost data for each course) to determine the most economical and efficient 

method to meet NRC’s training needs. 

12. Finalize Operating Procedure 410 including a new standardized student 

feedback form. 

13. Develop an evaluation strategy plan that defines the data HRTD needs to 

collect at varying levels to demonstrate the impact of its programs on the 

agency. 

14. Evaluate the capability for collecting evaluation data via the Learning 

Management System. 

a. If the Learning Management System’s capabilities meet the agency’s 

needs, develop a business case for purchase and deployment of 

additional capabilities. 

b. If the Learning Management System’s capabilities do not meet the 

agency’s needs, develop a plan for using alternative technologies to 

collect and analyze evaluation data. 

15. Develop an e-learning strategy that establishes a broad, fundamental 

connection between e-learning and organizational mission, strategy, 

business objectives and performance improvement. 

16. Develop an implementation plan for e-learning that includes, at a minimum: 

a. An assessment of NRC’s baseline technology. 

b. A plan for roll-out, implementation, maintenance and ongoing evaluation 

of additional Learning Management Systems capabilities. The 
viii 

implementation plan should include a cost/benefit analysis of the 

available LMS features and how they can support NRC’s business 

needs. 

17. Develop a process similar to the Strategic Workforce Planning Process to: 

a. Determine the e-learning skill sets necessary to meet NRC’s business 

needs. 

b. Assess the current staff to determine the availability of the needed 

skills. 

c. Develop a plan to close any identified gaps—either through training, 

hiring, or outsourcing. 

Agency Comments 

OIG provided this report in draft to agency officials on June 3, 2008, and 

discussed its content at an exit conference on June 9, 2008. We modified the 

report as we determined appropriate in response to the discussion. Agency 

officials generally agreed with the report’s findings and recommendations and 

opted not to provide formal comments. 

 

Booz | Allen | Hamilton 

 13200 Woodland Park Road 

 One Dulles 

 Herndon, VA 20171 
 

ABBREVIATIONS AND ACRONYMS 

ASTD American Society for Training and Development 

FY fiscal year 

GAO United States Government Accountability Office 

HRMS Human Resources Management System 

HRTD Human Resources Training and Development 

LMS Learning Management System 

NRC United States Nuclear Regulatory Commission 

NRO Office of New Reactors 

OIG Office of the Inspector General 

PDC Professional Development Center 

ROI Return on Investment 

SAT Systems Approach to Training 

SOW Statement of Work 

SWP Strategic Workforce Planning 

TTC Technical Training Center 
 

[Page intentionally left blank] 
1 of 63 

TABLE OF CONTENTS 

1. Background 3

2. Purpose 6

3. Findings 7

3.1 Content Management 7

3.2 Effectiveness and Efficiency of the Training Calendar 14

3.3 Performance Metrics 20

3.4 Evaluation Strategies 30

3.5 Alternative Delivery Mechanisms for Training 37

3.6 Staff Skill Sets 43

4. Consolidated List of Recommendations 47

5. Appendices 50

A.1 Scope and Methodology 50

A.2 Materials Reviewed 57

A.3 Suggested Components of an e-Learning Implementation Plan 61

A.4 Training Evaluation Levels 63
2 of 63 

[Page intentionally left blank] 
3 of 63 

1. Background 

The Nuclear Regulatory Commission (NRC) Strategic Plan highlights the need for the 

agency to provide a robust program of training, development, and knowledge transfer to 

ensure that agency employees attain and sustain the skills needed to implement the 

agency’s mission.3

 NRC Human Resources Training and Development (HRTD) has 

articulated its plan to help the agency accomplish its mission and goals in the NRC 

Training and Development Strategic Plan, which focuses priorities and resources in an 

effort to achieve measurable results.4

 

HRTD’s plan contains four goals: 

1. Training and development enhance individual performance. 

2. Agency training needs are identified and met. 

3. Training resources are optimized and the agency realizes the intended 

benefits. 

4. Training policies and practices encourage continuous improvement and 

optimal organizational performance.5

HRTD offers a program of training and development for NRC employees through the 

Professional Development Center (PDC) in Bethesda, MD, and at the Technical 

Training Center (TTC) in Chattanooga, TN. Course topics range from highly technical 

nuclear reactor and nuclear materials courses to leadership development to desktop 

computer skills. To make learning opportunities as widely accessible as possible, 

HRTD offers some courses in the regional offices or at other locations across the United 

 

3

 United States, Nuclear Regulatory Commission, Strategic Plan Fiscal Years 2008-2013 (2003) 19. 

4 United States, Nuclear Regulatory Commission, Training and Development Strategic Plan, NUREG/BR-0332 (Feb 

2007) 1. 

5

 Training and Development Strategic Plan 3. 
4 of 63 

States. HRTD also launched a new Learning Management System (LMS) in April 2008 

which expands NRC’s online learning capabilities. 

While HRTD has successfully met the majority of NRC’s training needs, the agency has 

identified a number of challenges that may impact the ability of the program to sustain 

the level of support it has provided to the agency thus far. Many of these challenges 

are the result of a planned increase of 600 new employees by 20096 to address the 

expected growth in licensing and construction of new nuclear power plants to meet the 

Nation’s increased demand for energy production.7 NRC’s ability to effectively review 

and license the new generation of commercial nuclear reactors will depend significantly 

on how well new employees are trained and developed to be effective reviewers and 

regulators at the staff and senior management level.8

 

In addition to a significant increase in staffing levels, NRC faces other challenges that 

will impact training and development. 

• Needs for new critical skills, such as threat analysis and emergency 

preparedness, have emerged because of 9/11. 

• New technologies in the nuclear power industry (e.g., the replacement of analog 

safety systems and control room instrumentation with digital systems) will also 

require new skill sets. 

• Increased licensing and regulatory activities will be associated with the renewal 

of existing power plant licenses and requests for reactor power uprates. NRC is 

also planning for the licensing of Yucca Mountain as a high-level waste 

repository. 

 

6 Dale E. Klein. Remarks Prepared for GE Nuclear Innovations Conference, Orlando, FL, 6 June 2007. 

7

 HRTD is partnering with the Office of New Reactors (NRO) to develop new courses to address the construction and 

licensing of new reactors and their associated technologies. According to an NRO representative, the new 

curriculum needs to be available in two years. HRTD has piloted two courses for construction inspection and is 

beginning development of a construction quality assurance course. In some instances, NRO and HRTD are using 

the skills of rehired annuitants to assist in the development of these courses. 

8 United States, Nuclear Regulatory Commission, Fiscal Year 2006 Performance and Accountability Report (2006) 

132. 
5 of 63 

• The agency has an increased need to train staff on risk-informed and 

performance-based regulatory approaches. 

• As more NRC Agreement States assume the regulatory responsibilities for 

nuclear materials, these States have an increased need to train their personnel. 

• NRC expects a high rate of attrition in the coming years as members of its highly 

technical workforce become eligible for retirement. 

• NRC is hiring a younger generation of employees to replace the retiring 

workforce. As of May 24, 2008, the average age of the workforce had dropped to 

46.9 and 11 percent of the permanent workforce was age 29 or under. NRC is 

continuing its efforts to hire younger employees. 

• In order to attract and retain new employees, NRC has identified the need to 

offer more attractive benefits like telecommuting, which will increase the 

demands on the agency’s technical infrastructure and traditional classroom 

training methods.9

 

9

 United States, Nuclear Regulatory Commission, Fiscal Year (FY) 2004-2009 Strategic Human Capital and 

Workforce Restructuring Plan (March, 2003) 3-4. 
6 of 63 

2. Purpose 

Due to the elevated significance of training, the Office of the Inspector General (OIG) 

identified a need to evaluate NRC’s Training and Development Program to determine its 

effectiveness to meet current and future needs. The OIG requested a thorough analysis 

of the program’s efficiency and effectiveness, combining a review of relevant existing 

reports on the program as well as qualitative research consisting of one-on-one 

interviews with knowledgeable HRTD staff, agency employees, managers, and 

executives. This evaluation did not include a review of specific courses or curricula, 

e.g., new reactor licensing, new reactor construction, or the next generation of reactor 

technologies. 

The evaluation team assessed the program to determine if it was: 

• Scalable: Were current training methods and development activities and tools 

sufficient to accommodate fluctuations in the size of the workforce? 

• Accountable: Were agency training needs systematically identified and were 

they being met by the current curriculum? 

• Trackable: Did the metrics, tools, and information gathering devices in use 

adequately identify whether training and employee development activities 

achieve required individual and organizational performance goals? 

• Aligned with Mission: Were training resources, including funding, aligned so 

that employees receive the training and performance support they need, when 

they need it, so that they are ready to perform as soon as possible after being 

brought into the agency? 

Appendix A.1. includes a detailed description of the methodology used for this 

evaluation. 
7 of 63 

3. Findings 

The evaluation team reviewed background documentation and interviewed 

knowledgeable NRC staff members to develop a good understanding of the current 

Training and Development Program. Our analysis identified the following areas for 

improvement: 

1. Efforts to maintain training documents, track changes, and control versions are 

not consistent across all branches within HRTD. 

2. Regional offices cited difficulties in scheduling new and current employees for 

training. Additionally, some course schedules increase travel costs and time 

away from job responsibilities. 

3. HRTD has limited ability to demonstrate the impact of its Training and 

Development Program on NRC’s mission using current performance measures. 

4. Current evaluation strategies are limited in their ability to demonstrate the 

effectiveness of training. 

5. HRTD’s continued reliance on the traditional classroom as the primary delivery 

method for training is being strained by an increasing number of students as well 

as new training needs. 

6. In order to move into other forms of training delivery, HRTD will need to enhance 

or supplement the skill sets of its staff. 

3.1 Content Management 

HRTD has developed operating procedures to define its processes for the development 

and management of training content. The evaluation team observed that content 

management practices varied among branches and instructors at both the TTC and 

PDC. Inconsistent and incomplete content management practices and tools have 

hampered HRTD’s ability to manage and control content. The inability to consistently 

manage and control content impacts HRTD’s ability to provide effective and relevant 

training and development programs. 
8 of 63 

3.1.1 HRTD has developed operating procedures to define its processes for the 

development and management of training content. 

HRTD has recognized that following an industry-recognized process (the Systems 

Approach to Training10) for the development of training materials is the first step toward 

achieving training effectiveness. HRTD’s Operating Procedure No. 406 states that 

HRTD staff and contractors will use the Systems Approach to Training (SAT) in the 

development of course content.11 As articulated in HRTD’s operating procedure, 

documents generated as part of the SAT process include: 

• Needs analysis. 

• Training plan. 

• Learning objectives. 

• Course materials including course manuals and presentation materials. 

• Lesson plan. 

• Exam bank. 

• Course feedback tools. 

• Training aids. 

 

HRTD has also recognized that maintaining and updating training materials is critical to 

providing an effective and relevant Training and Development Program. In the Training 

and Development Strategic Plan, HRTD states that courses must contain current 

regulatory, technical, and policy information so students have the latest information to 

 

10 The Systems Approach to Training is an industry-recognized process for the analysis, design, development, 

implementation, and evaluation of training programs. 

11 United States, Nuclear Regulatory Commission, Human Resources Training and Development (HRTD) Operating 

Procedure (OP) No. 406: Training Program Development Process, 30 June 2007. 
9 of 63 

assist them in the performance of their duties.12 HRTD is currently developing HRTD 

Operating Procedure 404 – Training Material Control which outlines a process for 

identifying and making content updates.13 

Bersin & Associates, a leader in training and development research, has found many 

organizations are putting greater efforts into content management strategies to increase 

efficiency and effectiveness, ensure content quality and consistency, and meet 

compliance requirements by controlling content versions and tracking changes.14 

Content management strategies usually address the following: 

• Centralization of learning content - A central repository for all learning content 

makes it easier for organizations to ensure all learning content is complete and 

facilitates identification of reusable content.15

• Content maintenance – For training to remain relevant and effective, the 

content must be regularly maintained to ensure it is up-to-date, correct, and 

complete. Maintenance, along with version control and tracking, are easier when 

all content is centrally located and searchable.16

• Course review – Beyond content maintenance, organizations should periodically 

review courses to ensure learning objectives, delivery methods, and evaluation 

strategies are still valid; the learning objectives still support the mission; and 

stakeholder needs are still being met. Organizations need a plan to identify a 

systematic “review, revise and release cycle” which includes triggering 

mechanisms and measurement criteria.17 Such a lifecycle maintenance process 

provides an opportunity for the organization to determine if the course still meets 

 

12 Training & Development Strategic Plan, 6. 

13 United States, Nuclear Regulatory Commission, Human Resources Training and Development (HRTD) Operating 

Procedure (OP) No. 404: Training Material Control, (in draft). 

14 Chris Howard, “Learning Content Management Systems: What Works,” Bersin & Associates, Oct 2005, 9. 

15 Howard, 10. 

16 Bill Brandon, “Handbook of e-Learning Strategy,” The e-Learning Guild, 2007, 7. 

17 Bill Brandon, “Handbook of e-Learning Strategy,” The e-Learning Guild, 2007, 7. 
10 of 63 

the business needs and to ensure the course is still fresh, timely, and engaging 

for learners. 

3.1.2 Content management practices varied among branches and instructors at both 

the TTC and PDC. 

The evaluation team interviewed a total of eleven instructors, eight instructors at the 

TTC and three instructors at the PDC. Of those instructors interviewed, seven 

instructors stated that they adhere to elements of the SAT process and develop most of 

the prescribed documentation. Two instructors for nuclear materials courses stated that 

they have not followed the complete process and, as a result, some of the course 

documents are missing or incomplete. Because of their familiarity with the subject 

matter, some instructors have forgone the development of some training materials (e.g., 

lesson plans and learning objectives) and rely solely on their presentation materials as 

their course documentation. 

The current practice for keeping learning content up-to-date is inconsistent and 

sometimes informal. HRTD instructors are using several different means to identify and 

incorporate necessary changes in course materials. During this evaluation, HRTD was 

in the process of drafting a new procedure for the control of training materials. Although 

HRTD has not yet implemented the procedure, the evaluation team did find that some 

instructors were using the Training Deficiency Materials Report prescribed by the draft 

procedure to identify necessary changes and assign accountability for making 

revisions.18 Other instructors stated that they make necessary updates to course 

materials as they prepare to instruct each session. Some instructors rely on subject 

matter experts within NRC to inform them of necessary changes. Instructors stated that 

students have identified outdated material in their courses. In many cases, needed 

 

18 The Training Materials Deficiency Report is contained in HRTD’s draft Operating Procedure 404 – Training 

Material Control. The operating procedure discusses the use of a paper system using the Training Materials 

Deficiency Report as a tracking mechanism for training material discrepancies. The procedure also allows for the 

use of a paperless system, such as a spreadsheet or database, in lieu of the paper form. 
11 of 63 

changes identified by students or instructors were communicated verbally or informally. 

For most contracted courses, HRTD relied on the contractor to make updates. 

The evaluation team did not note or observe any mechanism or quality assurance 

process to ensure changes are incorporated consistently across multiple affected 

courses. Individual instructors make changes to the courses for which they are 

responsible. The evaluation team noted that there appears to be no mechanism in 

place to determine if a change might impact more than one course. Any efforts to make 

content changes across multiple courses would be difficult because HRTD does not 

have a means to centralize or search all course materials. HRTD has encountered 

technology barriers to centralizing its course materials and is currently conducting a 

baseline assessment of its technical infrastructure. One of the objectives is to 

determine how it can centralize training materials. 

While HRTD performs almost continuous updates to training materials as content 

changes are identified, it does not have a procedure in place to review course materials 

in a more holistic manner. Simply updating course content does not ensure that the 

course still meets the learning objectives, that the learning objectives are still valid, that 

the delivery mechanism is still the most effective, or that the course is still meeting the 

needs of stakeholders. HRTD has conducted such reviews for some courses and 

curricula when they have identified issues. However, there is no procedure for 

conducting course reviews on a more proactive basis. 

3.1.3 Inconsistent and incomplete content management practices and tools have 

hampered HRTD’s ability to manage and control content. 

Although HRTD has a procedure in place for the development of courses according to 

the SAT process, managers have not consistently enforced the procedure across all 

branches of the Training and Development Program. The varying degree to which 

instructors adhere to the principles of the SAT process results in gaps in HRTD’s course 

materials and supporting documentation. 
12 of 63 

HRTD has not completed or implemented the new procedure for the control of training 

materials. Since current practices to update course content are inconsistent, students 

and supervisors have found that course materials are sometimes outdated or incorrect. 

In addition, the lack of a central location (or repository)19 for all course materials 

hampers the ability of instructors to search content and make changes consistently 

across all courses. It also hampers the ability of HRTD management to ensure that all 

prescribed course materials are in place and up-to-date. During this evaluation, HRTD 

management and regional offices identified deficiencies with the agency’s IT 

infrastructure, particularly inadequate bandwidth, as contributing factors in the inability 

to share training materials across the agency. A representative from the Office of 

Information Systems (OIS) is currently working with HRTD to evaluate their current 

technology baseline and the infrastructure needed to support their strategic goals. 

Since HRTD does not have a policy or procedure for the lifecycle maintenance of 

courses, there is no mechanism in place to ensure HRTD periodically reviews courses. 

Consequently, courses may be out of alignment with business needs and learning 

objectives. Also, other delivery methods may prove to be more cost effective or 

efficient. 

3.1.4 The inability to consistently manage and control content impacts HRTD’s ability 

to provide effective and relevant training and development programs. 

Since instructors are not always following the prescribed SAT process, course materials 

are sometimes incomplete. While seasoned instructors may feel comfortable relying 

solely on their presentation materials to teach courses, such an approach does not 

ensure that instructor knowledge is transferred to the next generation of instructors 

hired by the agency. As current instructors retire and HRTD looks to more e-learning 

 

19 “Central location” refers to a shared site or shared server that may be accessed by all HRTD personnel. A 

repository is a more sophisticated software system that facilitates the discovery, sharing, and reuse of learning 

content. Repositories often have well-designed user interfaces and architectures that make them easy to use and 

permit various levels of interactivity including search, submissions, comments/reviews, and creating personal 

collections. 
13 of 63 

and blended strategies for training, it will be critical to have up-to-date and accurate 

materials that new instructors and instructional designers can rely upon for course 

design and delivery. 

Additionally, the potential for course materials to be out-of-date or incomplete is 

increased due to the lack of a defined procedure for the control and maintenance of 

training materials. Staff members stated that they have noted outdated or incorrect 

material in some courses. For example, in the materials and fuel facilities areas, 

students and supervisors stated that materials were not always up-to-date, particularly 

for self-study or where technology changes rapidly. If students are finding outdated 

material in courses, it risks putting the credibility of the course into question. 

Since HRTD lacks a central repository for course materials, master course files and 

materials are stored and maintained in multiple locations. As a result, it is difficult for 

instructors to search across courses to identify information that may need to be updated 

or changed. The possibility exists for students to get conflicting information in different 

courses. 

Because HRTD does not have a procedure to review courses periodically as a whole, 

they may be offering courses, or parts of courses, that are no longer relevant or aligned 

to the mission of the agency or they may be missing opportunities to identify more 

effective or efficient delivery methods. 

RECOMMENDATIONS:

We recommend that the Executive Director for Operations: 

1. Review the current course files to identify gaps in necessary documentation. 

2. Develop a plan and timeline for completing any missing course documentation. 

3. Complete HRTD Operating Procedure 404 – Training Material Control, to include a 

standard process for version control, tracking changes, and assigning 

accountability for changes. 
14 of 63 

4. Develop a plan to centralize course materials in one location, preferably a central 

repository on a shared server. 

5. Develop a lifecycle maintenance procedure for periodically reviewing courses to 

ensure the learning objectives are still valid, the delivery mechanism is still the 

most effective, or the course is still meeting stakeholder needs. 

3.2 Effectiveness and Efficiency of the Training Calendar 

As stated in the Training and Development Strategic Plan, one of HRTD’s goals is to 

identify and meet the agency’s training needs. HRTD provides most of the formal 

training courses required for NRC qualification programs and for leadership training. 

Staff in the regional offices stated that it is often difficult for employees to schedule and 

attend the training that is necessary to attain or maintain qualifications or that is required 

for new supervisors. In addition, increasing travel costs have made it difficult to send 

employees to necessary training, especially when the training requires frequent, shortterm trips to the Washington, DC, area. HRTD’s annual training needs assessment has 

not been highly effective in getting the information needed from the regions in order to 

schedule training. Employees’ inability to schedule training in a timely fashion has 

delayed the acquisition of skills needed to perform their jobs and meet qualification 

requirements. 

3.2.1 The Training and Development Strategic Plan states that HRTD is responsible 

for identifying and meeting the agency’s training needs. 

In order to meet its safety goals, NRC has noted the importance of maintaining trained 

inspectors at nuclear power reactors, fuel cycle sites, regional offices, and the agency’s 

Headquarters.20 To ensure NRC personnel can effectively review and license nuclear 

reactors and regulate the use, transportation, storage, and disposal of radioactive 

materials, the agency has developed highly structured qualification programs that 

include a significant amount of formal training along with self-study assignments and on-

 

20 Strategic Plan Fiscal Years 2004-2009, 10. 
15 of 63 

the-job training. Inspectors, reviewers, and examiners must successfully complete the 

applicable requirements of the qualification programs within 24 months. 

According to NRC Inspection Manual Chapters 1245 and 1246, employees are required 

to take the formal qualification training courses in sequence—completing basic level 

prerequisite courses before enrolling in the more advanced courses. The manual 

chapters state that a deviation is required from the program office to allow an employee 

to take the courses out of sequence. Additionally, inspectors are required to attend 

advanced and refresher training to maintain their inspector qualifications and keep 

current with inspection and licensing programs. The qualification programs include selfstudy, formal training courses, and on-the-job training. 

In addition to the technical skills necessary to support the agency’s licensing and 

regulation activities, NRC supervisors in some regions and program offices are required 

to complete a series of leadership courses within two years of assuming a supervisory 

role.21 These leadership development courses range in length from .5 days to 4 days. 

The TTC and PDC provide most of the formal training courses required for NRC 

qualification programs and for supervisory training. These training courses range in 

length from less than a day to several weeks and are offered mainly at the TTC in 

Chattanooga, TN, or the PDC in Bethesda, MD. HRTD offers some of these courses in 

the four regional offices. 

The Training and Development Strategic Plan contains two goals that address the 

scheduling and delivery of training courses. Goal 2 is to “...ensure agency training 

needs are identified and met.”22 As a part of Goal 2, HRTD has noted that close 

coordination with program offices and regional offices is critical to identifying and 

 

21 United States, Nuclear Regulatory Commission, Region II, Regional Office Instruction, No. 0402, Rev. 5, 20 April 

2005. 

22 Training and Development Strategic Plan, 5. 
16 of 63 

understanding training needs, as well as to coordinate training and development 

activities. 

Goal 3 in the Strategic Plan states that HRTD will, “...ensure training resources are 

optimized and the agency is realizing the intended benefits of training.”23 One of the 

strategies HRTD listed for meeting this goal is to, “...improve the processes for 

planning, budgeting, and scheduling training and development resources and activities.” 

The plan identifies the need for HRTD to improve its methods for anticipating training 

needs and for improving the training needs survey (also called the training needs 

assessment). 

3.2.2 During the course of this evaluation, numerous employees and supervisors noted 

obstacles they or their staff members have in scheduling training that is 

necessary for them to do their jobs as inspectors, reviewers, examiners or 

supervisors. 

In general, regional staff noted the following difficulties in scheduling employees for 

needed training courses: 

• Inability to plan training far enough in advance – Inspection schedules are 

determined 18 months in advance. Once an inspection is planned, any changes 

to the schedule are highly discouraged. Since HRTD publishes some training 

opportunities only six to twelve months in advance, inspectors find it difficult to 

plan their inspection calendars around the training courses they need to take. 

The fact that some courses are offered infrequently (1-2 times per year) 

complicates the matter further. 

HRTD has initiated an effort to schedule some courses (particularly those taught 

by HRTD staff) at the same time each year to allow employees to plan training 

opportunities, but the ability to schedule other courses 18 to 24 months in 

 

23 Training and Development Strategic Plan, 8. 
17 of 63 

advance may be limited by contractual requirements. Some NRC personnel 

were not aware that HRTD schedules some courses at the same time each year. 

• Problems scheduling new employees into the necessary qualification 

training programs – Regional office supervisors noted that they have had 

difficulties getting new employees into requisite training programs immediately 

upon hire. Since employees cannot conduct inspections until they have 

completed the basic training requirements, offices cannot use new employees as 

quickly as they would like in the inspection process. 

Supervisors and employees have developed “work arounds” to help alleviate the 

problem, but these alternatives are not always in the best interest of the 

employee or the agency. For example, some employees enroll in training 

courses out of sequence so they can complete their qualification training during 

the required timeframe. Although the inspection manual states that a deviation is 

required to take the courses out of order, the requirement has not been 

consistently enforced. The new Learning Management System (LMS) has the 

capability to enforce the completion of prerequisite courses before allowing 

enrollment in subsequent courses, which may further extend the time needed to 

complete qualification requirements. However, this capability has not yet been 

implemented. In other instances, supervisors stated that they register for training 

courses in their own name to hold a space for new hires. When the employee is 

hired, the supervisor will request that HRTD switch the registration into the new 

hire’s name. If the employee is not hired in time to take the class, the slot may 

go unfilled, thus preventing others from attending the training and incurring an 

unnecessary cost for HRTD. 

• Excessive travel costs and reduced productivity to send employees, 

especially new supervisors, to requisite training – Several of the new 

supervisor training courses are of very short duration and are offered mainly at 

the PDC. The regional offices have found it burdensome and costly to send their 

employees to the Washington, DC, area multiple times for short courses. 
18 of 63 

Although HRTD has occasionally offered some of these courses in the regional 

offices, the regions still find it difficult to include the resident inspectors who are 

located at multiple sites. In addition, HRTD has responded to regional needs by 

offering some courses consecutively to reduce the number of times an employee 

must travel to complete the training. Two courses in the supervisory series are 

now available online or on DVD. 

3.2.3 HRTD’s annual training needs assessment has not been highly effective in 

providing the information needed to plan an effective training calendar. 

HRTD solicits training requirements from the regional offices and program offices via an 

annual training needs assessment. The assessment asks agency personnel to project 

the number of employees they will send to HRTD training courses in each quarter of the 

fiscal year. In addition, it provides offices an opportunity to identify new training needs. 

HRTD also works through the training coordinators in the regional and program offices 

to identify training needs. 

HRTD has recognized the need to improve their training needs assessment as 

mentioned in the Training and Development Strategic Plan. Regional personnel stated 

that the annual assessment may not be the most effective method for planning annual 

training events; especially since it only projects training needs for 12 months. They also 

stated that their input has not always been accurate and complete. 

The structure of the training needs assessment does not provide opportunities for NRC 

personnel to identify scheduling problems, such as those mentioned above. Engaging 

select supervisors in a working group to conduct a review of the training calendar may 

be a more effective means of gaining meaningful input from the regions and program 

offices. The group could consider the following measures to improve the training 

calendar: 

• Offer courses in more convenient/cost-effective locations. 
19 of 63 

• Combine courses to reduce travel costs. 

• Publish training activities at least 18 months in advance. 

• Use more alternative delivery methods such as Web-based training or video 

teleconferencing. 

All decisions must necessarily take into account contractual agreements, budgets, and 

available resources. The working group format would allow a two-way dialog between 

HRTD and field personnel that is not currently supported by the needs assessment. 

As mentioned above, HRTD has two initiatives underway that may impact the ability of 

employees to plan training in advance. First, HRTD’s effort to schedule courses at the 

same time each year should allow employees to better plan training opportunities in 

advance, but the ability to schedule some courses 18 to 24 months in advance may be 

limited by contractual agreements. Secondly, if HRTD chooses to enforce the 

completion of prerequisites using the LMS, it may increase the length of time necessary 

for employees to complete the required courses. 

3.2.4 Employees’ inability to schedule training in a timely fashion has delayed the 

acquisition of skills needed to perform their jobs and meet qualification 

requirements. 

Delays in obtaining training have affected the ability of supervisors to schedule new 

employees for necessary work assignments. With the increasing inspection workload, 

supervisors need to ensure employees are able to be fully productive as soon as 

possible. In some instances, employees have been on training course wait lists for a 

significant amount of time before they could get into the necessary training.24 A few 

 

24 Several factors influence the length of time an employee must wait to get into a course, including the type of 

course, the frequency with which it is offered, and when the employee requires the training for qualification or 

requalification. For example, HRTD only offers some courses twice a year, so the wait may be 6 to 12 months. The 

wait time is much less for courses that are offered more frequently. All employees interviewed stated that they 

eventually received the training they needed. 
20 of 63 

found it necessary to obtain a waiver for required training in order to appear before their 

qualification boards as scheduled. 

Regional personnel noted that they often spend a significant amount of time 

coordinating and negotiating with HRTD staff to get individual employees enrolled in 

needed training courses. As NRC continues to grow, it may become more difficult to 

use these informal methods to ensure employees get the training they need. 

RECOMMENDATIONS:

We recommend that the Executive Director for Operations: 

6. Form a working group of select representatives from the program offices, regional 

offices, and HRTD to conduct a thorough review of the training calendar including 

location and frequency of courses, travel costs, and opportunities for efficiencies in 

course sequencing (i.e., course bundling). The group should identify problems 

and develop mutually agreed-upon solutions within the constraints of contractual 

agreements, budgets, and resources. 

7. Within contractual limitations, identify and schedule courses at the same time each 

year so employees can anticipate the availability of courses. Communicate these 

courses and dates widely to make sure employees, supervisors, training 

coordinators, and managers are aware of the annual schedule. 

8. Before implementing the enforcement of course prerequisites in the Learning 

Management System: 

a. Determine the impact on employees’ ability to take all required training within 

the allotted timeframe. 

b. Communicate the change to NRC personnel in advance and allow 

opportunities for feedback. 

3.3 Performance Metrics 

To show its value to the organization, the Training and Development Program must 

show how its training solutions link to strategic outcomes and if the solutions justify the 
21 of 63 

cost. HRTD’s current performance metrics fail to provide meaningful insight into how its 

programs align with the agency’s mission and strategic goals, or demonstrate training 

efficiency and effectiveness. HRTD has encountered challenges in producing credible, 

reliable, and consistent performance measures. Without effective performance metrics, 

HRTD is unable to demonstrate its contribution to NRC’s mission and risks future 

support and funding. 

3.3.1 In order to show value to the organization, the Training and Development 

Program must show how its training solutions link to strategic outcomes and if 

the solutions justify the cost. 

Government agencies and the programs they offer are increasingly being called upon to 

prove their impact in order to justify their budgets. The President’s Management 

Agenda places increasing emphasis on accountability and demonstrating performance. 

“Over time, agencies will be expected to identify high quality outcome measures, 

accurately monitor the performance of programs, and begin integrating this presentation 

with associated cost.”25 

NRC has recognized the growing trend in Government accountability and, as a result, 

included an operational excellence strategy for strengthening accountability for setting 

and achieving individual and organizational performance expectations in its 2008-2013 

Strategic Plan.26 The agency identified the need to, “...incorporate effectiveness and 

efficiency measures in the NRC planning and performance measurement process 

throughout the agency.”27 

In this environment of increasing accountability, Government decision makers and 

agency leaders are placing more emphasis on the need to demonstrate results 

achieved through the significant investments devoted to training and developing 

employees. Training and development programs are finding it more important to show 

 

25 United States, Office of the President, President’s Management Agenda, (Washington: 2002) 29. 

26 United States, Nuclear Regulatory Commission, Strategic Plan Fiscal Years 2008-2013 (2007) 20. 27 Strategic Plan Fiscal Years 2004-2009, 18. 
22 of 63 

their value to the organization by demonstrating how their offerings help the agency 

achieve its strategic objectives. The GAO Guide for Strategic Training and 

Development Efforts emphasizes the importance of aligning training investments with 

the agency’s strategic goals. The United States Government Accountability Office 

(GAO) says there needs to be a direct link between the training courses and the skills 

and competencies identified by the agency for mission accomplishment.28 NRC has 

already established this direct link between training and competency development by 

stating in its Strategic Plan that the agency, “...will maintain a dynamic program of 

training, development, and knowledge transfer to ensure that the NRC acquires and 

maintains the competencies needed to implement the strategic plan.”29 

The way to ensure accountability and evaluate progress toward achieving results 

aligned with the agency’s mission and goals is through effective performance measures. 

However, measuring the effectiveness of training and development programs can be 

complicated and challenging. As a result, many organizations have limited their 

measures of effectiveness to factors they can easily quantify, such as number of 

courses offered, number of employees attending training, total training hours, employee 

satisfaction with training, and exam results. While all of these input/output metrics are 

necessary and valuable, they do not show how training and development contributes to 

improved performance, reduced costs, and the ability to accomplish the agency’s 

mission.30

In order to measure the real impact of training, agencies need to move beyond data that 

merely measures inputs and outputs. They need to develop performance metrics that 

show how training and development programs contribute to accomplishing agency 

goals and objectives. Outcome based performance measures focus on: 

 

28 United States, Government Accountability Office, A Guide for Assessing Strategic Training and Development 

Efforts in the Federal Government, March 2004, 13. 

29 Strategic Plan Fiscal Years 2004-2009, 20. 

30 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 60. 
23 of 63 

• How training and development programs contribute to improved organizational 

and programmatic results; often measured through the attainment of 

competencies necessary for the agency to be successful.31

• How the investment in training programs is justified based on the outcomes; 

often measured through the identification of expenses associated with all 

components of the training and development processes.32

• How individual training courses contribute to performance improvement; often 

measured through course evaluation strategies and tools. 

HRTD has identified the need to improve its performance metrics, as stated in the 

Training and Development Strategic Plan. “The agency will explore ways to improve 

performance measures and metrics focusing on the training and development 

programs’ contributions to individual and organizational performance.”33

3.3.2 Current performance metrics fail to provide meaningful insight into how HRTD 

training and development programs align with NRC’s mission and strategic goals 

or demonstrate training efficiency or effectiveness. 

HRTD’s current performance metrics focus primarily on attendance, student 

satisfaction, and test scores. Current measures reported quarterly in HRTD’s Office 

Operating Plan include the following: 

• Average hours of training per employee. 

• Percentage of staff completing a minimum of 24 hours of training. 

• Percentage of staff recording no training activity. 

 

31 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 30. 

32 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 71. 

33 Training & Development Strategic Plan, 9. 
24 of 63 

• Number of training no shows. 

• Number of course per year cancelled or unable to be scheduled when needed 

due to training infrastructure support problems. 

• Percent of professional development programs participants, supervisors, and 

senior advisors that responded positively to program evaluations that program 

objectives were met. 

• Percentage of students that attain a passing score on course exams. 

• Percentage of attendees that rate training satisfactory or better on student 

evaluation forms.34

HRTD has encountered difficulties accurately collecting and reporting the above 

performance data since they have to collect much of it manually. Managers and 

supervisors stated that they found training completion information to be inaccurate or 

not current in the Human Resources Management System (HRMS) and thus rely on 

their own “home-grown” reports to maintain training information. In addition, HRTD 

instructors use various manual methods for tracking student satisfaction and exam 

scores. 

While the metrics listed above provide valuable output measures, they do not effectively 

show the actual business impact of the training interventions. HRTD collects, tracks, 

and reports limited data to show: 

• How training courses are helping employees attain competencies. Through 

its Strategic Workforce Planning (SWP) process, NRC has identified the 

competencies/skills necessary for the agency to accomplish its mission, as well 

as anticipated critical gaps in the skills needed for the agency to continue to meet 

 

34 United States, Nuclear Regulatory Commission, Human Resources Office Operating Plan, Quarterly 

Measurements, 2008 – Quarter 2. 
25 of 63 

its goals. NRC has stated that it relies on two primary methods to fill its critical 

skill gaps: (1) recruiting and hiring and (2) training and development.35 

However, HRTD does not have a method to show it is developing and offering 

training that helps fill critical skills gaps. While HRTD conducts an annual 

training needs assessment to determine which courses should be offered each 

year, the assessment is not linked to the SWP process. HRTD has not 

effectively linked investments in training and development programs to gaps 

identified through SWP. Although employees are required to self report their 

competencies into NRC’s SWP tool, supervisors and managers stated that they 

make limited use of the tool to identify training needs to fill critical skill gaps. 

Further, HRTD does not have the ability to demonstrate how its training 

programs help the agency maintain the identified competencies and fill the critical 

skill gaps. It has not assessed its training courses to determine which, if any, 

identified competencies each course teaches. One advantage associated with 

linking training to competencies is the ability to demonstrate the strategic value of 

training.36

The Training and Development Strategic Plan states that HRTD will take action 

to “...more comprehensively define competencies and training needs for major 

functions or groups of like positions; and better integrate its training and 

development programs with performance elements and standards, position 

descriptions, training needs surveys, and the strategic workforce planning 

system.”37 However, HRTD has not yet taken the steps to achieve these 

objectives. 

 

35 United States, Government Accountability Office, Retirements and Anticipated New Reactor Applications will 

Challenge NRC’s Workforce, January 2007, 11. 36 Davis Dubois and William Rothwell, “Competency-based or a Traditional Approach to Training?” Training & 

Development, April 2004, 44- 56. 

37 Training and Development Strategic Plan, 3.
26 of 63 

• How investments in training programs are justified based on the outcomes. 

Although the Training and Development Strategic Plan states that HRTD, “...will 

use budget planning and execution processes and the time and labor system to 

capture and track agency wide expenditures for conducting and attending 

training,”38 HRTD has not established a means to effectively and consistently 

track training costs beyond the cost of contracted courses or external training 

events. Without a full understanding of the cost of internal training, HRTD is 

unable to effectively compare the costs of internal and external resources, or to 

show its programs are providing a return on the agency’s investments in training. 

• How individual training courses are helping improve performance. While 

HRTD administers an exam at the end of many courses to show learning has 

taken place, it does not conduct other forms of evaluation to show if performance 

is improved. Performance metrics associated with training evaluation strategies 

are addressed in more detail in Section 3.4. 

While the performance measures discussed above are not an exhaustive list of possible 

training and development metrics, they represent areas that GAO has recognized as 

important to showing the effectiveness of training programs. These performance 

measures are also widely used by organizations today to show the impact of training 

and development on results.39 Using metrics similar to those tracked by other 

organizations will allow NRC to more easily compare and benchmark its training and 

development programs against others. 

HRTD is currently reexamining its performance metrics to better measure training 

effectiveness. The implementation of the new LMS, launched in April 2008, has the 

potential to aid in the collection of performance data. In the first phase of the LMS 

deployment, HRTD focused on course registration, tracking, and scheduling 

capabilities. However, the system has extensive capabilities for recording, tracking, and 

 

38 Training and Development Strategic Plan, 8. 

39 Andrew Paradise, “2007 State of the Industry Report,” American Society for Training and Development, 2007, 6-

21. 
27 of 63 

reporting performance data. One of the benefits of the LMS is that it will be NRC’s 

single system of record for training, thus allowing HRTD to leverage its capabilities to 

capture training metrics. 

3.3.3 HRTD faces challenges collecting and reporting credible, reliable, and consistent 

performance data to demonstrate the effectiveness of its training and 

development program. 

In the past, HRTD has encountered obstacles in collecting and reporting accurate 

performance data, as mentioned above. Much of the data has been collected manually 

and in different systems, e.g., students provide course feedback in hardcopy format 

which instructors enter manually into different and independent spreadsheet tracking 

forms. Some of the data has been incomplete and/or inaccurate, e.g., employee 

training records in the HRMS often provide an incomplete picture of individual training 

completions. As a result, data collection, analysis, and reporting have been time 

consuming and labor intensive. HRTD has necessarily focused on collecting 

performance data that is easily quantifiable and accessible such as attendance, 

satisfaction, and exam scores. 

Now that HRTD has a more efficient means to collect, track, and report performance 

data through the LMS, current performance metrics need to be refined to allow HRTD to 

better demonstrate how its programs contribute to agency success. However, the LMS 

is a means to an end, not the entire solution. In order for the LMS to aid HRTD in the 

collection, analysis, and reporting of meaningful performance metrics, the organization 

must: 

• Choose performance metrics that demonstrate mission alignment, track training 

costs, and demonstrate effectiveness and efficiency. 

• Evaluate the LMS’ capabilities for collecting performance data in support of the 

new metrics. 
28 of 63 

The LMS chosen by NRC has extensive capabilities and can assist HRTD in collecting 

data in support of the three outcome-based measures discussed above: demonstrating 

how training programs contribute to results (linking training to competency 

development), justifying training expenditures based on outcomes, and linking training 

to performance improvement. 

First, the LMS has the ability to link training and competencies, but HRTD does not 

currently subscribe to this feature. In order to implement the competency model 

capabilities, HRTD would need to assess all of its courses to identify the associated 

competencies, evaluate the LMS capabilities for determine if they meet the agency’s 

needs, and develop a business case to support the purchase of the capability. 

Secondly, the LMS has extensive cost tracking capabilities that will help HRTD 

demonstrate the efficiency of its operations. HRTD is able to track some costs with the 

currently deployed capabilities, but may have to purchase additional capabilities, 

depending on the measures it chooses to track. Typical training cost measures 

employed by member organizations of the American Society for Training and 

Development (ASTD) include:40

• Direct expenditure per employee. 

• Direct expenditure as a percentage of payroll. 

• Percent of expenditure for tuition reimbursement. 

• Cost per learning hour used. 

• Travel and accommodation costs. 

• Cost of work time lost during training. 

• Percentage of learning expenditure for tuition reimbursement. 

 

40 Paradise, 6-17. 
29 of 63 

• Cost savings realized through outsourcing learning initiatives. 

• Content development costs. 

Finally, the LMS has the ability to collect evaluation data as suggested in Section 3.4 of 

this report. Using the LMS for the collection of evaluations at all levels aids in 

consistency and efficiency while allowing for increased data collection with minimal 

effort. 

3.3.4 Without effective performance metrics, HRTD is limited in its ability to 

demonstrate its contribution to NRC’s mission and risks future funding. 

While NRC’s Training and Development Program has been well-funded to date,41 the 

increasing emphasis on accountability in the Federal Government may make future 

funding less certain. NRC invests a significant amount of money and resources in 

training and developing employees, but may not be able to measure the business 

outcomes and results it is getting for its investment. Without a direct link between 

results and investments, training programs can begin to look like an unjustified cost, 

resulting in a decline in both support and funding.42 The inability to show measurable 

results can result in budget cuts. 

With accurate and effective performance measures, on the other hand, organizations 

can make better decisions about whether to revise or redesign training programs or 

eliminate ineffective programs. Good performance metrics can also be used to make 

decisions about future programs including the most effective delivery mechanisms and 

barriers to improved performance.43 As the President’s Management Agenda directs, 

 

41 HRTD’s Deputy Associate Director stated that the Training and Development Program often has excess funds at 

the end of the fiscal year. 

42 Steve Wexler, et al., “Measuring Success: Aligning Learning Success with Business Success,” The e-Learning 

Guild, (October 2007) 204. 43 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 61. 
30 of 63 

performance measures can be used to reinforce high performing programs and reform 

or end non-performing programs.44

RECOMMENDATIONS:

We recommend that the Executive Director for Operations: 

9. Develop and implement new performance metrics to demonstrate mission 

alignment, effectiveness, and efficiency. 

10. Develop and implement a plan to leverage the capabilities of the LMS for 

collection and reporting of chosen metrics. Specifically, evaluate the competency 

model capabilities to determine if they meet NRC’s needs, including identifying 

competencies, linking courses (or course modules and learning objectives) to 

identified competencies and closing critical skill gaps. 

11. Develop and implement a comprehensive cost tracking capability (including cost 

data for each course) to determine the most economical and efficient method to 

meet NRC’s training needs. 

3.4 Evaluation Strategies

A systematic evaluation approach and process allows for continuous improvement 

within the training function when aligned with organizational goals. NRC does not use a 

systematic evaluation process for continuously improving its training and development 

program because it has not: 

• Developed a strategy for collecting pertinent evaluation data. 

• Finalized the draft evaluation procedure. 

• Evaluated the current LMS capability for data collection. 

 

44 President’s Management Agenda, 29. 
31 of 63 

Without a strategy that standardizes the training evaluation procedure and metrics, NRC 

cannot accurately measure and monitor its achievement of goals. 

3.4.1 A systematic evaluation approach and process allows for continuous 

improvement in the learning and development function by enabling data-driven 

decisionmaking. 

Agencies are finding it increasingly important to evaluate their training and development 

programs and demonstrate how these efforts help develop employees and improve the 

agencies’ performance. HRTD has recognized the value of evaluation and includes a 

strategy to, “...implement meaningful evaluation tools and performance measures to 

assess progress toward achieving results aligned with the agency’s mission and goals” 

in the Training and Development Strategic Plan.45 

The Training and Development Strategic Plan also states, “...the training organization 

will evaluate stakeholder input and feedback mechanisms for training and development 

programs to determine whether additional levels of evaluation are necessary and costeffective to ensure quality and results. Increased use of higher level evaluations would 

require cooperation, input, and resource expenditure from all training stakeholders 

including attendees, supervisors, and managers.”46 

In the statement above, HRTD is alluding to Kirkpatrick’s Four-Level Evaluation Model47

which it references in the evaluation of NRC training programs. This model is probably 

the best-known and most widely used training evaluation methodology. The four levels 

measure: 

• Reaction of students to the training; usually collected through student feedback 

forms (Level 1). 

 

45 Training and Development Strategic Plan, 8. 

46 Training and Development Strategic Plan, 9. 

47 Donald Kirkpatrick, Evaluating Training Programs (San Francisco: Berrett-Koehler Publishers, Inc, 1994). 
32 of 63 

• Amount of learning that took place; often determined by testing students (Level 2). 

• Change in performance; often determined by supervisor input (Level 3). 

• Impact on the organization; determined by various measures such as increase in 

sales or reduction in errors (Level 4). 

Jack Phillips Return on Investment (ROI) methodology has often been combined with 

Kirkpatrick’s model to produce a fifth level of evaluation.48 Level 5 measures if the 

organization is reaping a value from the training program that is equal to or greater than 

their investment costs. Additional information about these levels can be found in 

Appendix A.4. 

In the Federal Government’s Guide for Assessing Strategic Training and Development 

Efforts in the Federal Government, the GAO notes that not all programs necessitate, or 

are appropriate for, higher levels of evaluation. The GAO uses the figure below to 

suggest the percentages of total courses that should be evaluated at each level. Bestpractice organizations usually target 100 percent of all training programs to be 

evaluated at the reaction level (Level 1) and conclude with targets of 5 to 10 percent of 

training programs to be evaluated at the return on investment level (Level 5).49

Evaluation Level

Recommended Percentage of 

Training Programs To be Measured 

at Each Level

Level 1 - reaction 100% of training programs 

Level 2 - learning 60% of training programs 

Level 3 - performance 30% of training programs 

Level 4 - impact 10% of training programs 

Level 5 - return on investment 5% of training programs 

 

48 Jack Phillips, Return on Investment in Training and Performance Improvement Programs. (Houston: Gulf 

Publishing Company, Houston, 1997). 49 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 63. 
33 of 63 

Figure 3.4.1.1 Evaluation Targets for Training Programs 

Nearly 96 percent of the 2004 Training Magazine Top 100 companies measure some 

training effectiveness through Level 4 and 75 percent of the companies measure 

through Level 5.50 A research study by the eLearning Guild found no significant 

relationship between the size of a training budget and usage of Levels 3 and 4 

evaluations. However, the study did find a significant relationship between the need for 

a knowledgeable and skilled work force and usage of Levels 3 and 4. The findings also 

show that as this factor increases in importance for an organization, so too does its 

usage of Kirkpatrick Levels 3 and 4.51 

3.4.2 HRTD has not implemented a systematic evaluation strategy or procedure to 

evaluate the impact of training and development programs. 

Although HRTD uses parts of the Kirkpatrick model, the organization has not fully 

developed and implemented an evaluation strategy or procedure at all levels. NRC 

currently collects Level 1 data with its student feedback form for all courses delivered by 

HRTD or its contractors. However, HRTD uses several different forms to collect student 

feedback.52 Level 2 evaluations, in the form of performance-based tests, are being 

conducted for some courses, most often for the technical courses. HRTD has 

attempted to collect Level 3 evaluation data from supervisors in the past to determine if 

employee performance improved following a course. Concerns about the low level of 

participation from supervisors and managers led to questions about the data’s validity 

and usefulness. Level 4 and 5 evaluations have not been conducted systematically 

although information is gathered from division directors informally about whether 

training has met their needs and provided value for the money. 

 

50 Tammy Galvin, "The 2004 Top 100," Training Magazine 01 April 2004, 32. 

51 Joe Pulichino, “Usage and Value of Kirkpatrick’s Four Levels of Training Evaluation Research Report,” The 

eLearning Guild, August 11, 2006, 27. 

52 The OIG recommended in OIG-07-A-05, “Audit of NRC’s Technical Training Facility,” (January 2007) that HRTD 

develop one consistent form to evaluate all training programs. 
34 of 63 

Most Level 1 evaluation data is collected manually at present. Some instructors track 

student feedback data, but the data is kept in multiple spreadsheet programs. Most 

stated that they read portions of the form and then file them away. In some instances, 

the form is reviewed by a supervisor who identifies any needed changes. The only 

information from the student feedback form that HRTD tracks consistently across the 

PDC and TTC is the overall satisfaction rating that students assign to the courses. 

HRTD has refrained from collecting additional types of data and conducting data 

analysis because it viewed the process as time consuming and resource intensive. 

HRTD has been unsure of the benefits that could be gained and hesitant to dedicate 

resources to a systematic evaluation strategy addressing multiple levels. 

HRTD is in the process of drafting a new procedure, Operating Procedure 410, to 

address its evaluation process. The draft procedure currently outlines the processes 

for: 

• Reviewing, summarizing and reporting student feedback data (Level 1). 

• Evaluating the validity of student examinations (Level 2). 

• Evaluating instructors in the classroom. 

While Operating Procedure 410 addresses the step-by-step procedure for collecting 

Level 1 and Level 2 data, it does not provide an overall strategy for collecting 

meaningful evaluation data that shows how training programs are contributing to 

improved performance and accomplishing the agency’s mission. 

3.4.3 HRTD’s evaluation efforts are hampered by the lack of a defined evaluation 

strategy, evaluation procedure, and automated data collection methods. 

NRC does not use a systematic evaluation procedure partially because it has not 

finalized HRTD Operating Procedure 410. The current draft of the procedure includes 

methods for evaluations at Level 1 (student feedback forms) and Level 2 (course 
35 of 63 

examinations), but does not address any higher levels of evaluation as defined by 

Kirkpatrick’s model. 

Level 1 evaluation data (students’ reactions) can be useful in determining which 

instructors, training methods, aids, and resources are popular and, therefore, likely to 

affect trainee motivation and participation. However, data from student feedback forms 

does not indicate whether the training program contributed to achieving the goals and 

objectives of the organization. It cannot show if participants successfully transferred 

knowledge gained from the course to their on-the-job performance. Involving 

stakeholders, who are in a position to observe behavioral or organizational changes, 

provides a much better indicator of the impact of training. Potential sources of data 

about performance improvement opportunities are the qualification boards and focus 

groups with supervisors and managers. 

HRTD has not defined a strategy for collecting evaluation data at levels beyond 1 and 2. 

NRC would benefit from developing an evaluation strategy that describes the 

organizational goals that must be supported and the types of evaluation that will be 

implemented to ensure that these goals are being met. Aligning the HRTD evaluation 

strategy with organizational goals will allow NRC to determine what activities have the 

most significant business impact and should therefore receive more resources and 

which do not have business impact and should therefore receive fewer resources or be 

adjusted to increase their impact. NRC will also gain important insights about how well 

their new employees are transferring knowledge gained from the courses to their onthe-job performance and whether improvements are necessary. 

While past data collection methods have been fragmented, the implementation of the 

new LMS provides HRTD with the opportunity to automate much of the data collection, 

thus reducing the burden placed on staff to manually collect and analyze data. 

Although the LMS is capable of collecting large amounts and varying kinds of data, 

these additional capabilities can be costly. The selection of capabilities should be 
36 of 63 

guided by an evaluation strategy that defines the type of data HRTD needs to 

demonstrate how its programs contribute to the success of NRC. 

 

3.4.4 Without a strategy that standardizes the training evaluation procedure and 

metrics, NRC can not accurately measure and monitor its achievement of goals. 

The need for evaluation at all levels is to provide a holistic picture of how training 

contributes to organizational objectives. GAO acknowledges that agencies are finding it 

increasingly important to demonstrate how their training and development efforts are 

helping develop employees and improve performance. As the Federal Government 

moves toward linking resources with results, they are focusing on training and 

development efforts along with other agency programs.53 

Without an evaluation strategy and a means to collect meaningful data, HRTD is not 

able to effectively demonstrate how training and development programs contribute to 

NRC’s success. 

RECOMMENDATIONS:

We recommend that the Executive Director for Operations: 

12. Finalize Operating Procedure 410 including a new standardized student feedback 

form. 

13. Develop an evaluation strategy plan that defines the data HRTD needs to collect 

at varying levels to demonstrate the impact of its programs on the agency. 

14. Evaluate the capability for collecting evaluation data via the Learning Management 

System. 

a. If the Learning Management System’s capabilities meet the agency’s needs, 

develop a business case for purchase and deployment of additional 

capabilities. 

 

53 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 8. 
37 of 63 

b. If the Learning Management System’s capabilities do not meet the agency’s 

needs, develop a plan for using alternative technologies to collect and analyze 

evaluation data. 

3.5 Alternative Delivery Mechanisms for Training 

NRC is facing workforce challenges that make e-learning a viable alternative to the 

traditional classroom, which is currently used for the majority of courses. HRTD has 

recognized the benefit that new training methodologies could bring to the agency and 

has deployed a new LMS to enable e-learning. However, HRTD has not developed an 

e-learning strategy or implementation plan to maximize the benefits of this investment 

and ensure it is helping HRTD meet the agency’s most pressing training needs. HRTD 

staff is strained to meet all of the training needs in the traditional classroom with its 

current staff and facility resources, but additional funding to support e-learning initiatives 

may be difficult to obtain without a defined strategy and implementation plan. 

3.5.1 NRC is facing challenges that make e-learning a viable alternative to the 

traditional classroom. 

GAO has estimated that approximately one-third of NRC’s highly skilled employees will 

be eligible to retire by 2010. At the same time, NRC’s workforce is expanding to handle 

the anticipated increase in applications for new nuclear power reactors.54 These factors 

along with the development of new technologies, the need to train Agreement State 

personnel55 and other challenges outlined previously in this report have increased the 

need for training at NRC. At the same time, the agency is placing a greater emphasis 

on telecommuting as a retention strategy, is hiring an increasingly younger population, 

and has a significant number of employees located at offsite locations.56 Organizations 

 

54 Retirements and Anticipated New Reactor Applications will Challenge NRC’s Workforce, 31. 

55 In 2007, NRC made their training programs available at no cost to personnel in the 34 Agreement States. NRC 

pays for all travel and per diem costs for Agreement State personnel to attend NRC training programs. 56 FY 2004-2009 Strategic Human Capital and Workforce Restructuring Plan, 1-7. 
38 of 63 

facing similar challenges have looked to innovative training delivery methods to 

supplement their classroom training. 

In the Training and Development Strategic Plan, HRTD has identified the need to, 

“...evaluate and apply current and emerging learning tools and methodologies, as 

appropriate, to reduce time and travel costs for attending classroom training.”57 They 

have noted that other training delivery mechanisms such as self-study, computer-based 

programs, blended learning, or on-the-job training may meet employee needs more 

conveniently, efficiently, and effectively. The NRC 2004-2009 Strategic Human Capital 

and Workforce Restructuring Plan also addresses the need to “...develop and 

implement e-learning strategies to provide effective training at lower cost.”58 

e-Learning has been defined as, “...the use of internet technologies to deliver a broad 

array of solutions that enhances knowledge and performance.”59 It has become a 

mainstream delivery method as virtual classrooms and online self-study courses now 

account for nearly one-third of all formal training hours consumed by students. While 

approximately 65 percent of formal training is still delivered in the classroom by an 

instructor, this number has decreased significantly over the past ten years.60 ASTD has 

observed a higher ratio of employees to training staff than in previous years because of 

efficiencies gained from technology-based learning solutions.61

Research has shown that successful e-learning implementations are guided by an 

effective e-learning strategy that contributes to the overall vision and mission of the 

organization. The primary component of an e-learning strategy is a set of goals or 

objectives that need to be achieved with e-learning. These goals and objectives need 

 

57 Training and Development Strategic Plan, 5-7. 

58 FY 2004-2009 Strategic Human Capital and Workforce Restructuring Plan, 11. 

59 Marc Rosenberg. e-Learning: Strategies for Delivering Knowledge in the Digital Age. (New York: McGraw-Hill, 

2001), 28. 

60 Karen O’Leonard. “The Corporate Learning Factbook ® 2008,” Bersin & Associates, January 2008, 10. 

61 Paradise, 4. 
39 of 63 

to be aligned with the mission and vision of the organization.62 When e-learning 

initiatives are not closely aligned, they can lead to inappropriate, or even damaging, 

results.63 Alignment ensures e-learning is linked to the attainment of organizational 

objectives and prevents the implementation of solutions for problems that don’t even 

exist. According to the eLearning Guild, a strategy must accompany the tactics of 

implementing e-learning to avoid a lot of wasted activity that yields little value to the 

organization, misses opportunities, and depletes goodwill.64

While the e-learning strategy addresses the goals the organization would like to realize 

in 3-5 years, an implementation plan provides the tactical steps the organization will 

take within the next 12 months to achieve the long-term e-learning goals. In other 

words, the strategy defines “what” and the implementation plan spells out “how.” An 

implementation plan also includes associated milestones and measures of success. eLearning implementation plans focus on the people, process, and technology issues 

that must be addressed for a successful execution. Appendix A.3 includes a list of 

considerations in each of these areas for possible inclusion in an e-learning 

implementation plan. 

3.5.2 Although HRTD has deployed a new Learning Management System and some 

courses have been developed for online delivery, HRTD has not developed a 

complete strategy or implementation plan for e-learning. 

HRTD currently offers approximately 30 online courses, and it recently deployed a new 

LMS. The initial LMS implementation was focused on centralizing and standardizing 

enrollment and tracking of students along with classroom and resource management. 

HRTD has noted in its Strategic Plan that it intends to implement additional functions in 

the LMS including the incorporation of Individual Development Plans (IDPs) and 

qualification templates; increased planning, tracking and reporting tools for employees 

 

62 Jennifer De Vries. “e-Learning Strategy: A Framework for Success,” ASTD’s Learning Circuits, August, 2005. 

http://www.learningcircuits.org/2005/aug2005/devries.htm. 

63 Ryan Watkins and Roger Kaufman, “Strategic Planning for Distance Education,” Handbook of American Distance 

Education, ed. Mark Moore (Mahwah, NJ: Lawrence Erlbaum Associates, 2003), 509. 

64 Brandon, 3. 
40 of 63 

and supervisors; and access to online training opportunities.65 HRTD staff members 

have suggested that the LMS can also be used to accomplish other objectives in the 

Training and Development Strategic Plan, including the linking of training programs to 

job competencies/critical skills and the Strategic Workforce Planning tool. Many of 

HRTD’s plans for the LMS are in line with GAO’s recommendation that training and 

development organizations should have a comprehensive LMS that can “...track the 

delivery of training, as well as, accounting, financial, and performance reporting systems 

that produce credible, reliable and consistent data on training and development 

programs.”66

Like many organizations new to e-learning, HRTD has focused much of its efforts on the 

technology to enable alternative delivery means. HRTD’s e-Learning Plan, developed 

in 2005, focuses mainly on the tactical issues of implementing technology and deploying 

courses. While the HRTD Training and Development Strategic Plan shows linkages to 

the agency’s vision and mission, e-learning needs to be fully integrated in order to show 

the same linkages. An e-learning strategy would help to align learning investments 

directly to business results while providing metrics to measure the value of those 

learning investments. 

While the current e-Learning Plan addresses the initial phases of LMS deployment and 

selection of commercial off-the-shelf (COTS) courses, it does not address the ongoing 

implementation of the e-learning strategy to include the additional capabilities 

mentioned in the Training and Development Strategic Plan. The deployment of the 

LMS provides HRTD with the foundational building block that will allow it to develop a 

robust e-learning program, but it does not have a clearly defined path for a full elearning implementation that helps achieve its strategic learning goals. 

The plan includes some of the elements of an effective implementation plan listed in 

Appendix A.1 (e.g., the need for a change management and communications plan, 

 

65 Training and Development Strategic Plan, 5. 

66 A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, 71. 
41 of 63 

need for user support), but the plan is incomplete and does not contain specific 

strategies and current milestones for implementation. It does not include a governance 

plan although some staff members suggested that input from an advisory board has 

been very beneficial in the past and they would like to see such a board reinstituted. In 

addition, some of the implementation strategies included in the plan have not been very 

effective. For example, several of the communication methods described in the plan 

was used in conjunction with the recent roll-out of the LMS, but staff members stated 

the marketing and communications plan was not completely effective and resulted in 

initial difficulties in using the system. 

In summary, the current NRC e-Learning Plan contains elements of both an e-learning 

strategy and an implementation plan, but neither the strategy nor the plan are complete. 

HRTD staff members have stated that the Plan needs to be updated since the roll-out of 

the LMS because the milestones are three years old and are no longer applicable. 

When the plan is updated, the HRTD staff should take the opportunity to better define 

their e-learning strategy and implementation plan. 

3.5.3 HRTD staff is strained to meet all of NRC’s training needs in the traditional 

classroom with its current staff and facility resources, but additional funding to 

support e-learning initiatives may be difficult to support without a defined strategy 

and implementation plan.

The majority of HRTD courses are currently delivered in the classroom by an instructor. 

According to The Sloan Consortium, approximately 34 percent of all graduate school 

courses were offered online in 2006 representing a 20 percent increase since 2002. In 

addition, almost 70 percent of the postsecondary institutions included in the study 

agreed that student demand for online learning is growing.67 HRTD management, as 

well as instructors at both PDC and TTC, stated that they do not have enough 

 

67 I. Elaine Allen and Jeff Seaman, “Online Nation: Five Years of Growth in Online Learning,” The Sloan Consortium 

and Babson Survey Research Group, (October 2007), 10-15. 
42 of 63 

instructors and classrooms to meet current training needs and are concerned about how 

they are going to meet the increasing needs. 

The increasing need for training may not be a short-term surge, but may continue 

beyond NRC’s current step-up in hiring to handle the new licensing requirements. 

While NRC has benefited from low turnover and a high degree of loyalty in its aging 

workforce, the agency may find a different mindset among the younger employees they 

are now recruiting. Continuing turnover in the employee population will necessitate a 

continuing need for training at higher volumes than HRTD has historically experienced. 

3.5.4 Additional funding to support e-learning initiatives may be difficult to obtain 

without a defined strategy and implementation plan. 

While NRC has an increasing need to explore alternative methods for delivering 

training, HRTD may find it difficult to get and keep support and funding for e-learning 

initiatives if they cannot clearly link these initiatives to the strategic goals of the agency 

and they do not have a well-defined roadmap for implementing the program.68

RECOMMENDATIONS:

We recommend that the Executive Director for Operations: 

15. Develop an e-learning strategy that establishes a broad, fundamental connection 

between e-learning and organizational mission, strategy, business objectives, and 

performance improvement. 

16. Develop an implementation plan for e-learning that includes, at a minimum: 

a. An assessment of NRC’s baseline technology. 

b. A plan for roll-out, implementation, maintenance, and ongoing evaluation of 

additional Learning Management Systems capabilities. The implementation 

plan should include a cost/benefit analysis of the available LMS features and 

how they can support NRC’s business needs. 

 

68 Brandon, 7. 
43 of 63 

3.6 Staff Skill Sets 

Organizations that have implemented e-learning solutions have found the skills needed 

to develop technology-enabled learning are very different from those needed for 

classroom instruction. Staff in HRTD noted that there are limited skills and resources 

for the development and deployment of e-learning among the current staff. If NRC 

plans to implement more e-learning solutions, HRTD will need to find ways to close the 

e-learning skill gaps—either through training, hiring, or outsourcing. Attempting to 

develop effective e-learning programs without the necessary skills can lead to 

frustration for the staff and ineffective learning opportunities for employees. 

3.6.1 The skills and roles needed to implement e-learning are very different from the 

skills and roles used for classroom instruction. 

In 1995, Thach and Murphy identified thirteen roles necessary for the design and 

delivery of e-learning.69 While their work is still relevant, the field has evolved since 

their study thirteen years ago. The table below shows the thirteen roles identified by 

Thach and Murphy along with updates to reflect more typical e-learning project roles of 

today. 

Role Description 

Instructional 

Designer 

Work with instructors and/or subject matter 

experts to design courses, revise existing courses 

to fit distance learning environment. Must be 

familiar with designing for e-learning standards 

and regulations. 

Programmer 

Work closely with instructional designer to select 

best technology solution for learning objectives, 

assess, and implement new technologies as 

needed to support e-learning, and ensure 

compliance with e-learning standards and 

regulations. 

 

69 Elizabeth C.Thach and Karen L. Murphy. “Competencies for Distance Education Professionals,” Educational 

Technology Research and Development March 1995: 57-71. 
44 of 63 

Subject Matter 

Expert 

Provide content matter expertise to design team, 

review content to ensure integrity, and provide 

content updates to design team as necessary to 

keep course content current. 

Graphic Designer Design attractive, clear layout, and ensure 

materials facilitate learning. 

Project Manager Manage staff and operations, monitor timelines, 

and ensure quality. 

Instructor (for 

certain types of 

online courses) 

Facilitates course delivery, monitors and 

evaluates learner performance. In some 

instances, the instructor may also serve as 

Subject Matter Expert. 

Systems 

Administrator 

Advise in selection of distance learning 

technology, ensure reliability of technology, and 

assess future changes in technology. Primary 

interface with IT department to ensure e-learning 

technology operates within the enterprise network.

Technician/Help 

Desk Staff 

Keep equipment in running condition, respond to 

users’ questions and problems, and assist 

instructors with technical delivery of courses. 

Site Facilitator 

Assist students in learning at remote sites, 

distribute/collect materials/assignments, proctor 

tests. 

Support Staff 

Register students, communicate course 

schedule/descriptions, and coordinate support 

services. 

Editor Edit course content for style, clarity, grammar, and 

structure. Must be skilled in web writing styles. 

Librarian/Knowledge 

Management 

Collect and organize the knowledge assets of the 

organization; provide means for students to find 

and use these knowledge assets. May also assist 

with copyright and digital rights issues. 

Evaluation 

Specialist 

Provide tools and evaluation instruments, monitor 

program success/problems, consult instructor 

about evaluation. 

Table 3.6.1.1 Summary of e-learning roles adapted from Thach & Murphy (1995: p. 67-69). 

While some of these roles are primary in an e-learning project, others play a supporting 

role. Using these roles as a guideline, e-learning leaders need to assess the current 
45 of 63 

skills of their training professionals to identify any gaps and then develop a strategy for 

closing those gaps—through training, hiring, or outsourcing.70 

Having the right internal resources with the right skills and experience is vital to a 

successful e-learning implementation.71 A recent Bersin Report notes that 

organizations once heavily staffed with instructors are now hiring higher proportions of 

content developers and technology specialists to facilitate their online learning 

strategies. They have found that these roles come at a lower cost than a senior trainer 

or instructor.72 Bersin and Associates have also noted that highly effective training 

organizations are directing more staff resources toward technology, analytics and 

measurement, and content development for e-learning and fewer resources to 

administration and content development for instructor-led training.73

3.6.2 While the HRTD staff has successfully designed and delivered classroom 

instruction for many years, few instructors have skills and experience in the 

design, development, and delivery of e-learning programs. 

Training programs offered by HRTD are developed and delivered by HRTD staff or by 

contractors. In general, the training staff at the TTC is composed of technical experts in 

the fields of health physics, reactor technologies, or other nuclear related areas. These 

technical experts are responsible for designing and delivering training and may, in some 

instances, oversee the design and delivery of training by contracted staff. The majority 

of the training staff at the PDC generally act as project managers, overseeing the 

design and delivery of courses by contractors.74 Some have received training in the 

 

70 Gustavo Prestera and Leslie Moller. “Organizational Alignment Supporting Distance Education in Post-secondary 

Institutions,” Online Journal of Distance Learning Administration, Winter 2001, 

http://www.westga.edu/~distance/ojdla/winter44/prestera44.html. 

71 Lori Mortimer. “e-Learning 101: The Devil is in the Details,” ASTD Learning Circuits. December, 2001. 

72 O’Leonard, 9. 

73 Bersin & Associates, The High Impact Learning Organization: What Works in the Management, Operations and 

Governance of Corporate Training, June 2005, 85. 

74 These statements are general observations and do not describe the skill sets of all HRTD staff members, for 

example, some PDC staff members, particularly in the new reactor technology area, are also technical experts. 
46 of 63 

Systems Approach to Training (SAT) process in their previous employment.75 A few 

HRTD staff members have degrees in education-related fields such as educational 

psychology and instructional design. Staff members at TTC and PDC stated that few of 

them have skills or experience related to the design, development, and delivery of elearning programs. The majority of the online courses currently available through the 

LMS were developed by contractors or by HRTD staff using automated authoring tools. 

These courses have varying levels of interactivity—ranging from “page turners” to 

limited interactivity in the form of online tests. 

3.6.3 HRTD needs to identify the skills necessary to implement their chosen e-learning 

strategy, assess the skills of the current staff, and develop a plan to close the 

skill gap. 

NRC has an established strategic workforce planning process in place to identify critical 

skills gaps in the employee population. The same process may be useful in helping 

HRTD assess the skills needed to implement e-learning, assess the current availability 

of those skills, and develop a plan for closing the skill gaps. Gap closure tactics may 

include one or more of the following: 

• Train current staff to attain the necessary skills - develop a training plan to close 

the skill gaps and identify appropriate sources for training the staff. 

• Hire the necessary skills - revise or develop appropriate job descriptions. 

• Outsource the development of e-learning solutions - revise current statement of 

work templates to ensure the contracted team includes personnel with the 

requisite skills. 

While many organizations decide to outsource their e-learning development or purchase 

off-the-shelf learning products in the early stages, most transition to building their own 

 

75 By their own assessment, staff members have had varying levels of training and experience in use of the SAT 

process. 
47 of 63 

learning products as their learning strategies evolve and demands can no longer be met 

with packaged solutions.76 

3.6.4 Attempting to develop effective e-learning programs without the necessary skills 

can lead to frustration for the staff and ineffective learning opportunities for 

employees. 

Viewing the implementation of e-learning as merely an extension of existing classroom 

learning can lead to frustration, wasted time and ineffective learning products. Building 

on current strengths and finding ways to supplement weaknesses is the suggested way 

to build a highly functional team that can develop the appropriate learning solutions to 

respond to business needs. 

RECOMMENDATIONS:

We recommend that the Executive Director for Operations: 

17. Develop a process similar to the Strategic Workforce Planning Process to: 

a. Determine the e-learning skill sets necessary to meet NRC’s business needs; 

b. Assess the current staff to determine the availability of the needed skills; and 

c. Develop a plan to close any identified gaps—either through training, hiring or 

outsourcing. 

4. Consolidated List of Recommendations 

We recommend that the Executive Director for Operations: 

1. Review the current course files to identify gaps in necessary documentation. 

2. Develop a plan and timeline for completing any missing course documentation. 

3. Complete HRTD Operating Procedure 404 – Training Material Control to include a 

standard process for version control, tracking changes and assigning 

accountability for changes. 

 

76O’Leonard, 42. 
48 of 63 

4. Develop a plan to centralize course materials in one location, preferably a central 

repository on a shared server. 

5. Develop a lifecycle maintenance procedure for periodically reviewing courses to 

ensure the learning objectives are still valid, the delivery mechanism is still the 

most effective, or the course is still meeting stakeholder needs. 

6. Form a working group of select representatives from the program offices, regional 

offices, and HRTD to conduct a thorough review of the training calendar including 

location and frequency of courses, travel costs, and opportunities for efficiencies in 

course sequencing (i.e., course bundling). The group should identify problems 

and develop mutually agreed-upon solutions within the constraints of contractual 

agreements, budgets and resources. 

7. Within contractual limitations, identify and schedule courses at the same time each 

year so employees can anticipate the availability of courses. Communicate these 

courses and dates widely to make sure employees, supervisors, training 

coordinators, and managers are aware of the annual schedule. 

8. Before implementing the enforcement of course pre-requisites in the Learning 

Management System: 

a. Determine the impact on employees’ ability to take all required training within 

the allotted timeframe. 

b. Communicate the change to NRC personnel in advance and allow 

opportunities for feedback. 

9. Develop and implement new performance metrics to demonstrate mission 

alignment, effectiveness, and efficiency. 

10. Develop and implement a plan to leverage the capabilities of the LMS for 

collection and reporting of chosen metrics. Specifically, evaluate the competency 

model capabilities to determine if they meet NRC’s needs, including identifying 

competencies; linking courses (or course modules and learning objectives) to 

identified competencies and closing critical skill gaps. 

11. Develop and implement a comprehensive cost tracking capability (including cost 

data for each course) to determine the most economical and efficient method to 

meet NRC’s training needs. 
49 of 63 

12. Finalize Operating Procedure 410 including a new standardized student feedback 

form. 

13. Develop an evaluation strategy plan that defines the data HRTD needs to collect 

at varying levels to demonstrate the impact of its programs on the agency. 

14. Evaluate the capability for collecting evaluation data via the Learning Management 

System. 

a. If the Learning Management System’s capabilities meet the agency’s needs, 

develop a business case for purchase and deployment of additional 

capabilities. 

b. If the Learning Management System’s capabilities do not meet the agency’s 

needs, develop a plan for using alternative technologies to collect and analyze 

evaluation data. 

15. Develop an e-learning strategy that establishes a broad, fundamental connection 

between e-learning and organizational mission, strategy, business objectives and 

performance improvement. 

16. Develop an implementation plan for e-learning that includes, at a minimum: 

a. An assessment of NRC’s baseline technology. 

b. A plan for roll-out, implementation, maintenance and ongoing evaluation of 

additional Learning Management Systems capabilities. The implementation 

plan should include a cost/benefit analysis of the available LMS features and 

how they can support NRC’s business needs. 

17. Develop a process similar to the Strategic Workforce Planning Process to: 

a. Determine the e-learning skill sets necessary to meet NRC’s business needs; 

b. Assess the current staff to determine the availability of the needed skills; and 

c. Develop a plan to close any identified gaps—either through training, hiring or 

outsourcing. 
50 of 63 

5. Appendices 

A.1 Scope and Methodology 

The scope was defined by the Office of the Inspector General in the Statement of Work 

and included nine factors for evaluation: 

1. Determine if the agency has a formal process to ensure that changes in Federal 

regulations are promptly incorporated in training and development efforts. 

2. Review how the agency identifies the appropriate level of investment to provide for 

training and development efforts and prioritize funding so that the most important 

training needs are addressed first. 

3. Determine how the agency tracks: 

a. the cost and delivery of its training and development programs, and 

b. the number of students trained. 

4. Review the process the agency uses to determine whether to design training and 

development programs in-house or obtain these services from contractors or other 

external sources. 

5. Determine the effectiveness of agency coordination with designated technical 

experts during reviews of in-house courses to assess and revise both course content 

and training materials. 

6. Determine what performance data the agency uses to evaluate the overall 

effectiveness of training and development activities to determine whether intended 

short and long-range program results are being obtained and whether adjustments 

are necessary. 

7. Evaluate the agency process to ensure that offered training correctly matches staff 

needs and is available when needed. 

8. Determine how the agency incorporates evaluation feedback into the planning, 

design, and implementation of its training and development efforts. Determine 

whether this feedback includes: 

a. the participant’s perception of course value, 
51 of 63 

b. the view of the participant’s supervisor as to whether individual performance 

improved, and 

c. management’s opinion of whether the overall training and development program 

is meeting short and long-range strategic program needs. 

9. Determine whether the agency compares its training methods, or outcomes with 

those of other Federal agencies to identify innovative approaches or lessons 

learned. 

The Booz Allen Hamilton evaluation team did not note any significant findings for factors 5 

and 9 above. Factors 2 and 4 are addressed indirectly in Section 3.3 since improved 

performance metrics including cost data will help HRTD make more informed decisions about 

prioritizing training investments and using internal or external resources to develop and 

deliver training. The other factors are addressed in Section 3. 

While specific training, development, and knowledge management responsibilities are 

shared among the regional offices, program offices and HRTD, this report focuses on 

training programs provided by HRTD. 

The methodology for this evaluation was designed to obtain sufficient, appropriate 

evidence to address the nine evaluation factors, reduce risk to an acceptable level, and 

provide reasonable assurance that the evidence was sufficient and appropriate to 

support the findings and conclusions. 

In defining the methodology, the evaluation team employed a three-phased approach 

(See Figure A.1.1) to ensure the program met the agency’s goals of enhancing 

individual performance, identifying and meeting training needs, optimizing training 

resources, and encouraging continuous improvement and organizational performance. 

Each phase included critical steps and activities to ensure that a complete and thorough 

evaluation of the training program was completed. 
52 of 63 

Figure A.1.1: Training Program Evaluation Methodology 

Phase 1 - Planning Phase: The evaluation team reviewed the recommended NRC 

documents along with other materials to obtain a solid understanding of NRC and 

Federal Government training and development evaluation protocols. Based on the 

initial review, the team developed criteria for measuring each factor listed in the 

Statement of Work (SOW), along with an evaluation protocol, a data collection plan 

and a Standard Operating Procedure (SOP) for the conduct of site visits. During this 

phase of the project, the evaluation team regularly communicated with NRC OIG 

personnel to ensure our evaluation methodology met the needs of the agency and to 

complete scheduling and logistics for the site visits. 

Phase 2 - Data Gathering and Analysis: During this data gathering phase, the 

evaluation team worked in cooperation with the Contracting Officer’s Technical 

Representative (COTR) and NRC representatives to conduct site visits. During the 

course of these visits, the evaluation team interviewed knowledgeable staff 

members at Region I, Region II, the TTC, and the PDC. The interviewees 

represented a broad range of perspectives including employees who have taken 

training at NRC, supervisors/managers of employees who have taken training, 

regional executives, HRTD staff members, and HR management. Each site visit 

began with an entrance conference to introduce our team and its purpose. An exit 

Main Phase Tasks 

Review 

Documentation 

Develop List of 

Criteria 

Develop Data 

Collection Plan 

Complete Scheduling 

and Logistics 

Report 

Writing and 

Response 

Phase Two Phase Three 

Main Phase Tasks 

Review Training 

Program Based on 

SOW Tasks 

Complete Site Visits 

Analyze Captured 

Data 

Create Preliminary 

Draft of Results 

Main Phase Tasks 

Develop and Deliver 

Draft Report 

Receive Comments 

Report 

Update Draft Report 

Based on Review 

 Deliver Final Report 

Attend Results 

Meeting

Data 

Gathering 

and Analysis Planning 

Phase One
53 of 63 

conference at the end of each visit allowed us to present and discuss initial 

observations, as well as allow evaluators to confirm information, ask questions, and 

provide additional data. Following each site visit, the team analyzed and 

synthesized the data to develop findings and trends. The data was presented for 

NRC review after each site in order to address issues and questions early. 

Phase 3 – Report Writing and Response: The final phase was the development, 

review, and delivery of the findings and recommendation report which addressed all 

requirements included in Section 4 of the SOW. A preliminary report was developed 

for NRC review and a final report incorporated appropriate NRC comments. 

Evaluation team members attended the results meeting with NRC to review the 

findings and recommendations. 

Booz Allen Hamilton has developed a tool called the Learning Organization 

Diagnostic to help learning and development leadership build a learning strategy by 

identifying strengths, weaknesses, and areas of emphasis (to improve Current State 

or envision Future State) across four Levers that are essential to successful training 

operations: 

1. Technology Pervasiveness - focuses on the extent to which learning and 

performance technologies are considered and deployed. 

2. Programs and Content Lifecycle - analyzes all of the curricula and programs 

across the organization. (Represents the core of training and development, 

and what most people will recall satisfactorily or unsatisfactorily about training 

operations.) 

3. Operations and Processes - defines the administrative and managerial 

challenges in running training like a business. 

4. Business Alignment - identifies the methods used to actively engage 

leadership in executing strategy and realizing success. (Area where most 

training and development departments fail to realize significant impact.) 
54 of 63 

These four Levers are further broken down into elements and sub-elements that drive to 

the level necessary to inform and action real change. 

Similar to a Balanced Scorecard construct, the levers of the Learning Organization 

Diagnostic represent a holistic view of training operations and address opportunities for 

improvement. The Balanced Scorecard, developed by Robert Kaplan and David 

Norton, is used by many organizations to identify and manage achievement of long-term 

strategic goals beyond solely financial indices.77 In fact, the Learning Organization 

Framework is aligned towards the specific drivers of the business of training and directly 

supports the Learning and Growth quadrant of a typical Balanced Scorecard. Through 

consistently monitoring achievement in each Lever, an organization can strike an 

eventual balance of capabilities. 

The nine factors identified in the NRC Statement of Work were similar to many of the 

elements and sub-elements included in the Learning Organizational Diagnostic. Figure 

A.1.2 below illustrates how the nine evaluation factors align to the four levers in the 

Learning Organization Diagnostic. 

 

77 The Balanced Scorecard identifies goals in the areas of Financial, Customer, Internal Processes, and Learning 

and Growth or Innovation. 
55 of 63 

Figure A.1.2 – Learning Organization Diagnostic with OIG Evaluation Factors 

As a result of the similarities, the evaluation team referenced the tool during the project 

to provide a broader context for the nine identified evaluation factors. However, the 

scope of this evaluation did not go beyond the identified factors. Figure A.1.3 below 

shows how the final recommendations address all four Levers in the diagnostic tool and 

provide a balanced approach to improvement. 

Business/Organization Alignment

4.7 – Determine what performance data the agency 

uses to evaluate the overall effectiveness of training 

and development activities to determine whether 

intended short and long-range program results are 

being obtained and whether adjustments are 

necessary. 

4.8. – Evaluate the agency process to ensure that 

offered training correctly matches staff needs and is 

available when needed. 

4.9 – Determine how the agency incorporates 

evaluation feedback into the planning, design, and 

implementation of its training and development 

efforts.

Content Life Cycle

4.2 – Determine if the agency

has a formal process to ensure that changes in 

Federal regulations are promptly incorporated in 

training and development efforts.

4.6 – Determine the effectiveness of agency 

coordination with designated technical experts

during reviews of in-house courses to assess and 

revise both course content and training materials.

Operations and Processes

4.3 – Review how the 4.3agency identifies the 

appropriate level of investment to provide for training 

and development efforts and prioritize funding so 

that the most important training needs are 

addressed first. 

4.4 – Determine how the agency tracks the cost and 

delivery of its training and development programs, 

and the number of students trained.

4.5 – Review the process the agency uses to 

determine whether to design training and 

development programs in-house or obtain these 

services from contractors or other...

4.10 – Determine whether the agency compares its 

training methods, or outcomes with those of other 

Federal agencies to identify innovative approaches 

or lessons learned. 

Technical Pervasiveness

 None

Business/Organization Alignment

4.7 – Determine what performance data the agency 

uses to evaluate the overall effectiveness of training 

and development activities to determine whether 

intended short and long-range program results are

being obtained and whether adjustments are 

necessary. 

4.8. – Evaluate the agency process to ensure that 

offered training correctly matches staff needs and is 

available when needed. 

4.9 – Determine how the agency incorporates 

evaluation feedback into the planning, design, and 

implementation of its training and development 

efforts.

Content Life Cycle

4.2 – Determine if the agency

has a formal process to ensure that changes in 

Federal regulations are promptly incorporated in 

training and development efforts.

4.6 – Determine the effectiveness of agency 

coordination with designated technical experts

during reviews of in-house courses to assess and 

revise both course content and training materials.

Operations and Processes

4.3 – Review how the agency identifies the 

appropriate level of investment to provide for training 

and development efforts and prioritize funding so 

that the most important training needs are 

addressed first. 

4.4 – Determine how the agency tracks the cost and 

delivery of its training and development programs, 

and the number of students trained.

4.5 – Review the process the agency uses to 

determine whether to design training and 

development programs in- house or obtain these 

services from contractors or others.

4.10 – Determine whether the agency compares its 

training methods, or outcomes with those of other 

Federal agencies to identify innovative approaches 

or lessons learned. 

Technical Pervasiveness

 None
56 of 63 

Figure A.1.3 – Learning Organization Diagnostic with Final Recommendations 

Business/Organization Alignment

4.7 – Determine what performance data the agency 

uses to evaluate the overall effectiveness of training 

and development activities to determine whether 

-range program results are 

4.8. – Evaluate the agency process to ensure that 

offered training correctly matches staff needs and is 

4.9 – Determine how the agency incorporates 

evaluation feedback into the planning, design, and 

implementation of its training and development 

Content Life Cycle

4.2 – Determine if the agency

4.6 – Determine the effectiveness of agency 

coordination with designated technical experts 

during reviews of in-house courses to assess and 

revise both course content and training materials.

Operations and Processes

4.3 – Review how the 4.3agency identifies the 

appropriate level of investment to provide for training 

and development efforts and prioritize funding so 

4.4 –

delivery of its training and development programs, 

and the number of students trained.

4.5 – Review the process the agency uses to 

determine whether to design training and 

development programs in-house or obtain these 

services from contractors or other...

4.10 – Determine whether the agency compares its 

training methods, or outcomes with those of other 

Federal agencies to identify innovative approaches 

or lessons learned. 

Technical Pervasiveness

 None

Content Life Cycle Business/Organization Alignment

Technical Pervasiveness Operations and Processes

• Determine impact of enforcing pre-requisites 

in the LMS 

• Leverage LMS capabilities to collect 

performance metrics 

• Leverage LMS capabilities to collect 

evaluation data 

• Develop an e-learning strategy and 

implementation plan 

• Complete HRTD Operating Procedure 404 

• Complete HRTD Operating Procedure 410 

• Assess skills needed to implement e-learning 

• Review current course files

• Develop plan for completing missing 

course documentation

• Centralize course materials

• Develop lifecycle maintenance procedure

• Review training calendar

• Schedule courses consistently

• Develop new performance metrics

• Develop evaluation strategy
57 of 63 

A.2 Materials Reviewed 

In order to ascertain the current training and development environment at NRC, the 

evaluation team conducted a thorough background review of documents pertinent to 

this evaluation. 

The list of documents reviewed per the SOW included: 

• FY 2004-2009 NRC Strategic Plan 

• FY 2008-2013 NRC Strategic Plan 

• NRC’s Training and Development Strategic Plan 

• NRC Strategic Human Capital and Workforce Restructuring Plan 

• NRC’s Comprehensive Diversity Management Plan 

• President’s Management Agenda Leadership and Knowledge Management 

Standards for Success 

• NRC’s Strategic Workforce Planning process 

• NRC’s Planning, Budgeting, and Performance Management process 

• Agency Performance and Accountability Report for FY 2006 

• Office of the Inspector General report titled Inspector General’s Assessment of 

the Most Serious Management and Performance Challenges Facing NRC, OIG07-A-01, dated October 6, 2006 

• Office of the Inspector General report titled Audit of NRC’s Technical Training 

Center, OIG-07-A-05, dated January 9, 2007 

• GAO report titled Retirements and Anticipated New Reactor Applications Will 

Challenge NRC’s Workforce, GAO-07-105, dated January 17, 2007 
58 of 63 

• GAO report titled NRC Based Its Decision to Move Its Technical Training Center 

on Perceived Benefits – Not Costs, GAO-01-54, dated October 19, 2000 

In addition to the documents outlined in the SOW, the team conducted a thorough 

review of a number of internal NRC materials related to training and development. 

These materials were provided by HRTD staff or by regional personnel and helped 

support the trends identified during the background research and in-person interviews. 

The list of additional documents reviewed included: 

From Region I 

• ROI 0410.4 Revision 3, Region I Training Council

• Training Council Agenda and Minutes, April 1, 2008

• Region I FY 2008 External Training Plan

• Region I Internal Training Schedule

From Region II 

• ROI 0402, Rev. 5, Region II Training Program 

• ROI 0403, Rev. 1, Regional Required Training Program 

• ROI 0451, Rev. 10, Site Access Training for Regional Power Reactors and Fuel 

Facility Inspectors 

• ROI 0452, Rev. 5, Respiratory Protection Qualification 

• ROI 0603, Rev. 2, Designated Individuals to Certify Individual Study Activities 

and On-The-Job Training for Inspector Qualification 

• Region II Training Plan 
59 of 63 

• Prioritizing Training Requests (a description of each priority) 

• Region II Employees in Developmental Programs 

• Qualification Program Sequence Chart 

• Inspector/Examiner Qualification Status 

• Inspector Refresher Training Status 

• Supervisory Training Status 

• Division Training Matrices 

• Training Region II Has Sponsored 

• Announcing I-Learn 

From other sources 

• Training materials used for Licensing Course and Inspection Course 

• HRTD Operating Plan 

• NRC Management Directives (6.8, 7.3, 7.5, 9.25, 9.29, 10.131, 10.137, 10.67, 

10.77) 

• Inspection Manual Chapter 1245 and 1246 

• Office of the Inspector General report titled Review of NRC’s Administration of 

Selected Contracts and Acquisition Workforce Training, OIG-04-A-10, February 

19, 2004 

• Information Digest 2007-2008 

• HRTD Policies (including current and draft) 
60 of 63 

• Copy of sample statements of work 

• Copy of sample acquisition strategy milestones 

• 2007 metrics tracked by TTC 

• Questionnaire for New Training Course Development 

• Report of the HRTD Working Group to Address the NRC Inspector General's 

Recommendation Regarding Student Evaluation of Instructor Performance (Oct. 

5, 2007) 

• Copies of student evaluations for a Site Access Training Course 

• e-Learning Plan 

• NRC Form 368 

• Individual Development Plan Process 

• Secretarial Qualification Program 

• Memo and form for Evaluation of Nuclear Safety Professional Development 

Program (NSPDP) 

• Sample evaluation forms for Acquisition Training and Reactor Technology 

Training 

• List of organizations/groups to which HRTD staff belong 
61 of 63 

A.3 Suggested Components of an e-Learning Implementation Plan 

Companies and agencies that have successfully implemented e-learning have found 

these components to be important. 

Technology considerations 

• An assessment of the baseline technology requirements and capacity. 

• A plan for roll-out of new capabilities and technologies. 

• A maintenance plan for chosen technologies. 

People considerations 

• A change management plan. Successful organizations recognize that e-learning 

is a type of change and must be accompanied by a comprehensive change 

management plan including strategies for training users and developers.78 

• A marketing and strategic communications plan. Studies show that the 

availability of e-learning does not automatically guarantee its use. On-going 

communications are also needed to sustain learner motivation and 

engagement.79 

• Strategies for gaining executive level support. When the e-learning effort is 

aligned with the goals of the organization and has the visible support of senior 

management, most other barriers to implementation and acceptance will be 

minimized or eliminated.80

 

78 Brandon, 46. 

79 Paul Weaver, “Preventing e-Learning Failure: Ten Pitfalls and How to Avoid Them,”. T+D Magazine, May 2002, 

45-48. 

80 Soomyung Cho and Zane Berge, “Overcoming Barriers to Distance Training and Education,” USDLA Journal

August 2002, 16. 
62 of 63 

Policy considerations 

• A governance plan. Bersin suggests a three-level best practices governance 

model: 1) Steering Committee that ensures alignment of training with the mission 

and strategy of the organization; 2) Learning Council who prioritizes program 

allocations; and 3) Training Operations who are responsible for the operation and 

execution of programs, resource allocation, and measurement of satisfaction and 

results.81

• A plan for identification and development of standards, templates, guidelines, 

etc., that need to be in place to support e-learning.82

• Establishment of procedures, processes, and workflows for the development, 

delivery, and maintenance of e-learning.83 e-Learning needs to be incorporated 

into operational procedures for the analysis, design, development, delivery, and 

evaluation of training programs. An alternative is to develop separate operational 

procedures to address the different needs of e-learning. 

 

81 Bersin & Associates, 47. 

82 Karen O’Leonard and Josh Bersin, “Rapid e-Learning: What Works ® 2005,” Bersin & Associates, August 2005, 7. 

83 O’Leonard and Bersin, 7. 
63 of 63 

A.4 Training Evaluation Levels 

Level Description 

Level 1 

Level 1 measures how students in a training program 

react to it. It answers questions regarding the 

students' perceptions - Did they like it? Was the 

material relevant to their work? 

Level 2 

Level 2 measures the extent students have mastered 

the skills, knowledge, or attitude goals of the learning 

product. Evaluating at this level answers the 

question: Did they learn what was intended? 

Level 3 

Level 3 measures the transfer of new skills to the 

workplace by the learners. Evaluating at this level 

answers the question: How well did the training 

impact on-the-job performance? 

Level 4 

Level 4 attempts to assess training in terms of 

business results. Evaluation at this level answers the 

question: How well did the training impact business 

performance? 

Level 5 

Level 5 measures the return on investment (ROI). It 

answers the question: What was the financial return 

on investment from the program? 

Adapted from: Donald Kirkpatrick, Evaluating Training Programs (San Francisco: BerrettKoehler Publishers, Inc, 1994), and Jack Phillips, Return on Investment in Training and 

Performance Improvement Programs. (Houston: Gulf Publishing Company, Houston, 

1997).