AUDIT REPORT

Audit of NRC’s iLearn Learning Management System

OIG-11-A-16 July 27, 2011

All publicly available OIG reports (including this report) are accessible through

NRC’s Web site at: 

http:/www.nrc.gov/reading-rm/doc-collections/insp-gen/
UNITED STATES

NUCLEAR REGULATORY COMMISSION

WASHINGTON, D.C. 20555-0001

OFFICE OF THE

INSPECTOR GENERAL

July 27, 2011

MEMORANDUM TO: R. William Borchardt

Executive Director for Operations

FROM: Stephen D. Dingbaum /RA/

Assistant Inspector General for Audits

SUBJECT: AUDIT OF NRC’S ILEARN LEARNING MANAGEMENT 

SYSTEM (OIG-11-A-16)

Attached is the Office of the Inspector General’s (OIG) audit report titled, Audit of NRC’s 

iLearn Learning Management System.

The report presents the results of the subject audit. During an exit conference held on 

July 20, 2011, the agency agreed with the findings and recommendations in this audit 

report and said that no agency suggestions or comments would be provided in 

connection with the report.

Please provide information on actions taken or planned on each of the 

recommendations within 30 days of the date of this memorandum. Actions taken or 

planned are subject to OIG followup as stated in Management Directive 6.1.

We appreciate the cooperation extended to us by members of your staff during the 

audit. If you have any questions or comments about our report, please contact me at 

415-5915 or Kathleen Stetson, Team Leader, Financial and Administrative Audit Team, 

at 415-8175.

Attachment: As stated 
Electronic Distribution

Edwin M. Hackett, Executive Director, Advisory Committee 

on Reactor Safeguards

E. Roy Hawkens, Chief Administrative Judge, Atomic Safety 

and Licensing Board Panel

Stephen G. Burns, General Counsel

Brooke D. Poole, Director, Office of Commission Appellate Adjudication

James E. Dyer, Chief Financial Officer

Margaret M. Doane, Director, Office of International Programs

Rebecca L. Schmidt, Director, Office of Congressional Affairs

Eliot B. Brenner, Director, Office of Public Affairs

Annette Vietti-Cook, Secretary of the Commission

R. William Borchardt, Executive Director for Operations

Michael F. Weber, Deputy Executive Director for Materials, Waste,

Research, State, Tribal, and Compliance Programs, OEDO

Darren B. Ash, Deputy Executive Director 

for Corporate Management, OEDO

Martin J. Virgilio, Deputy Executive Director for Reactor

and Preparedness Programs, OEDO

Nader Mamish, Assistant for Operations, OEDO

Kathryn O. Greene, Director, Office of Administration

Patrick D. Howard, Director, Computer Security Office

Roy P. Zimmerman, Director, Office of Enforcement

Charles L. Miller, Director, Office of Federal and State Materials

and Environmental Management Programs

Cheryl L. McCrary, Director, Office of Investigations

Thomas M. Boyce, Director, Office of Information Services

Miriam L. Cohen, Director, Office of Human Resources

Michael R. Johnson, Director, Office of New Reactors

Catherine Haney, Director, Office of Nuclear Material Safety

and Safeguards

Eric J. Leeds, Director, Office of Nuclear Reactor Regulation

Brian W. Sheron, Director, Office of Nuclear Regulatory Research

Corenthis B. Kelley, Director, Office of Small Business and Civil Rights

James T. Wiggins, Director, Office of Nuclear Security

and Incident Response

William M. Dean, Regional Administrator, Region I

Victor M. McCree, Regional Administrator, Region II

Mark A. Satorius, Regional Administrator, Region III

Elmo E. Collins, Jr., Regional Administrator, Region IV
Audit of NRC’s iLearn Learning Management System

i

EXECUTIVE SUMMARY

BACKGROUND

The U.S. Nuclear Regulatory Commission (NRC) implemented the iLearn 

Learning Management System (iLearn) in April 2008 in response to the 

E-Government Act of 2002.1 iLearn is a vehicle for providing training 

distribution and tracking services directly to employees. It serves as a 

central point for training activities across the agency and allows 

employees to see all NRC-offered courses, develop a learning plan, 

register for training, track training history, access online training, and 

complete evaluations from their desktop. 

Organizationally, iLearn is the responsibility of the Associate Director of 

Human Resources Training and Development (HRTD) within the Office of 

Human Resources (HR). Approximately 1.5 NRC Full-time Equivalent 

(FTE) and 1 contractor employee manage iLearn. 

NRC procured the system through a Government shared-service provider 

on a pay-as-you-go basis, using an interagency agreement with the Office 

of Personnel Management (OPM). The system, a commercial-off-theshelf product, is hosted by Plateau Systems. As of June 2011, the agency 

had spent approximately $1.5 million on iLearn and plans to spend almost 

$500,000 next year. 

iLearn is intended to be inclusive of all courses offered at NRC. Currently, 

the system houses 599 online courses; 492 are SkillSoft courses, 42 are 

Harvard Business courses, and 65 are NRC-developed courses.

PURPOSE

The audit objective was to determine the effectiveness of the iLearn 

Learning Management System to support the agency's current and future 

training needs.

 

1

E-Government Act of 2002 (Public Law 107-347) approved by the President on December 17, 2002.
Audit of NRC’s iLearn Learning Management System

ii

RESULTS IN BRIEF

While iLearn is capable of effectively supporting NRC’s current and future 

training needs, the Office of the Inspector General (OIG) identified the 

following issues in the management of iLearn:

A. Mandatory fields are not always complete.

B. NRC lacks written policies and procedures describing course manager 

duties.

Addressing these issues will improve the effectiveness of iLearn in 

meeting the agency’s training needs.

RECOMMENDATIONS

This report makes recommendations to improve iLearn. A list of these 

recommendations appears on page 10 of this report.

AGENCY COMMENTS

During an exit conference held on July 20, 2011, the agency agreed with 

the findings and recommendations in this audit report and said that no 

agency suggestions or comments would be provided in connection with 

the report. 
Audit of NRC’s iLearn Learning Management System

iii

ABBREVIATIONS AND ACRONYMS

FTE Full-time Equivalent

HR Office of Human Resources

HRTD Human Resources Training and Development

iLearn iLearn Learning Management System

NRC U.S. Nuclear Regulatory Commission

O&M Operations and Maintenance

OIG Office of the Inspector General

OPM Office of Personnel Management
Audit of NRC’s iLearn Learning Management System

iv

TABLE OF CONTENTS

EXECUTIVE SUMMARY ............................................................................ i

ABBREVIATIONS AND ACRONYMS........................................................iii

I. BACKGROUND .............................................................................. 1

II. PURPOSE ...................................................................................... 2

III. FINDINGS....................................................................................... 3

A. iLEARN MANDATORY FIELDS ARE NOT

ALWAYS COMPLETE............................................................... 4

B. NRC LACKS WRITTEN POLICIES AND

PROCEDURES DESCRIBING COURSE 

MANAGER DUTIES .................................................................. 7

IV. OTHER MATTERS........................................................................ 9

V. CONSOLIDATED LIST OF RECOMMENDATIONS.................... 10

VI. AGENCY COMMENTS................................................................ 10

APPENDIX

SCOPE AND METHODOLOGY ......................................................................... 11

 
Audit of NRC’s iLearn Learning Management System

1

I. BACKGROUND

The NRC implemented iLearn in April 2008 in response to the EGovernment Act of 2002. iLearn is a vehicle for providing training 

distribution and tracking services directly to employees. It serves as a 

central point for training activities across the agency and allows 

employees to see all NRC-offered courses, develop a learning plan, 

register for training, track training history, access online training, and 

complete evaluations from their desktop. 

Organizationally, iLearn is the responsibility of the Associate Director of 

HRTD within HR. Approximately 1.5 NRC FTE and 1 contractor employee 

manage iLearn. In addition, approximately 22 employees serve as course 

managers for the courses in iLearn. According to HRTD staff, course 

manager duties include monitoring the list of students who use the 

“Request a Schedule” feature in iLearn and ensuring the course data in 

iLearn is accurate and complete for the courses they manage.

NRC procured the system through a Government shared-service provider 

on a pay-as-you-go basis, using an interagency agreement with OPM. 

The system, a commercial-off-the-shelf product, is hosted by Plateau

Systems. As of June 2011, the agency had spent approximately 

$1.5 million on iLearn and plans to spend almost $500,000 next year. 

Operation and maintenance (O&M) expense details are shown in the table 

on the following page. 
Audit of NRC’s iLearn Learning Management System

2

Table 1.

Source: HRTD – Note that contract years start on June 15 and end on June 14 of the following year.

iLearn is intended to be inclusive of all courses offered at NRC. Currently, 

the system houses 599 online courses; 492 are SkillSoft courses, 42 are 

Harvard Business courses, and 65 are NRC-developed courses. SkillSoft

and Harvard courses are commercial-off-the-shelf courses for which the 

NRC has purchased licenses (e.g., the Federal Budgeting Process and 

Managing Six Sigma Team Performance). The 65 NRC-developed 

courses are custom courses developed to meet NRC learning objectives

and include titles such as Effective Communications for NRC Inspectors 

and Access Authorization and Fitness for Duty. 

II. PURPOSE

The audit objective was to determine the effectiveness of the iLearn 

Learning Management System to support the agency's current and future 

training needs. The report appendix contains information on the audit 

scope and methodology.
Audit of NRC’s iLearn Learning Management System

3

III. FINDINGS

During this audit, OIG auditors noted several positive items: (1) overall 

iLearn user satisfaction has increased significantly from 59 percent in 

December 2009 to 89 percent in March 2011; (2) the agency developed a 

dashboard2that provides management with information that is simply 

presented, easy to monitor, and indicates where action needs to be taken;

and (3) HR staff developed a No Fear Act online training course that was 

selected by OPM to be used across Government.

While iLearn is capable of effectively supporting NRC’s current and future 

training needs, OIG identified the following issues in the management of 

iLearn:

A. Mandatory fields are not always complete.

B. NRC lacks written policies and procedures describing course manager 

duties.

Addressing these issues will improve the effectiveness of iLearn in

meeting the agency’s training needs.

 

2

A dashboard is defined as a Web site that (1) organizes information so that it is easy to read and 

interpret, (2) provides up-to-date status information on key performance indicators, and (3) presents 

information visually through graphics (charts, gauges, maps, tables).
Audit of NRC’s iLearn Learning Management System

4

A. iLearn Mandatory Fields Are Not Always Complete 

Agency guidance requires use of mandatory fields in iLearn to capture

course prerequisites

3

and general information such as course description, 

course length, and point-of-contact. However, these fields are often left 

blank. This data completeness issue occurs because management 

(1) does not enforce use of the prerequisite feature and (2) lacks a quality 

assurance program to ensure that mandatory fields are used. Without 

course prerequisite information in iLearn, students are able to register for 

courses for which they are not eligible or prepared, thereby using a 

training slot that could be used by another student who may need the 

course for timely completion of his or her qualification curriculum. 

Furthermore, incomplete iLearn data makes it difficult for users to plan 

their training effectively and could affect the accuracy of NRC’s biweekly 

reports to OPM containing training information such as training credits, 

hours, and purpose for each course completed by employees.

Guidance Regarding Mandatory Fields

Agency guidance addresses the use of mandatory fields in iLearn for 

required course prerequisites. In addition, iLearn course input forms detail 

the fields required to be entered when a course is added to the system.

Prerequisite Feature

HRTD Operating Procedure 403 states that required prerequisites must be 

identified in the description section of the iLearn catalog and enforced 

through the iLearn prerequisite feature. For example, students are 

required to complete the Westinghouse Technology course (R-304P) 

before taking the Westinghouse Advanced Technology course (R-504P). 

The iLearn prerequisite feature does not allow a student to sign up for a 

course unless the required prerequisite has been met. (If a student is 

registered for the prerequisite course, and the ending date of that course 

is before the start date of the desired course, iLearn allows the student to 

register.)

 

3

In this context, prerequisites are defined as courses that are required to be completed before another 

course can be taken.
Audit of NRC’s iLearn Learning Management System

5

iLearn Course Input Form

iLearn course input forms detail the fields required to be entered when a 

course is added to the system. There are two different input forms: one 

for instructor-led training and one for online courses. 

Mandatory Fields Left Blank

Despite agency guidance, mandatory information is often not included in 

the iLearn course listing.

Prerequisite Feature

Although agency guidance prescribes mandatory fields, OIG identified

numerous instances where required fields were not completed. OIG 

reviewed 19 technical courses identified by the agency as having required 

prerequisites, and determined that all of them included the required 

prerequisites in the course description; however, only 2 of the course 

offerings enforced the prerequisites through the prerequisite feature. 

iLearn Course Input Form

A comparison of the information in iLearn for a sample of 14 technical and 

non-technical courses against the information required by the appropriate 

iLearn course input forms found that all 14 courses had at least one 

mandatory field (e.g., course description) that was blank and many 

contained more than one blank required field. In addition, 1 of the 14

courses contained inconsistent information related to the course duration. 

Management Does Not Enforce Prerequisite Feature and Lacks a 

Quality Assurance Program

The data completeness issue occurs because HRTD management does 

not enforce use of the prerequisite feature and lacks a quality assurance 

program to ensure the feature is used, if applicable. This results in 

inefficiencies in planning training and potential inaccuracies in reporting 

training data to OPM.
Audit of NRC’s iLearn Learning Management System

6

Prerequisite Feature

Management has not enforced the use of the iLearn prerequisite feature. 

When the prerequisite feature is not used, students are able to register for 

courses for which they are not eligible or prepared, thereby using a slot 

that could be used by another student who may need the course for timely 

completion of his/her qualification curriculum.

iLearn Course Input Form

Course managers in HRTD, as well as course managers in program 

offices, are responsible for ensuring that iLearn course descriptor

information is complete for their respective courses, but they do not 

always fulfill this function. Additionally, there is no quality assurance 

program to ensure that all mandatory fields, such as course description,

are completed. Missing iLearn data makes it difficult for users to plan their 

training effectively and could affect the accuracy of NRC’s reports to OPM 

that contain training information. The information that OPM requires is 

derived from the data contained in the mandatory fields.

Recommendations

OIG recommends that the Executive Director for Operations:

1. Issue guidance to course managers instructing them to complete all

mandatory fields as described on the iLearn course input form.

2. Develop and implement a quality assurance program for all data in 

iLearn to ensure mandatory fields are complete and accurate.

3. Correct incomplete mandatory fields in iLearn.
Audit of NRC’s iLearn Learning Management System

7

B. NRC Lacks Written Policies and Procedures Describing Course Manager 

Duties

Federal guidance directs that agencies have policies and procedures in 

place to help ensure that agency objectives are met. However, NRC lacks 

written policies and procedures describing the responsibilities of 

individuals assigned to serve as course managers for iLearn courses. 

Accordingly, course managers are not consistently fulfilling their duties. 

Without established policies and procedures to clearly communicate 

management expectations, course managers may be unaware of their 

duties or may use inconsistent and less than optimal methods to fulfill their 

duties.

Standards for Internal Control in the Federal Government

The Government Accountability Office’s Standards for Internal Control in 

the Federal Government states that internal control activities help ensure 

that management’s directives are carried out. Control activities are 

described as the policies, procedures, techniques, and mechanisms that 

enforce management’s directives.

Course Managers Do Not Consistently Fulfill Duties Associated with 

iLearn

NRC course managers do not consistently fulfill their duties associated 

with iLearn. There is a large disparity among course managers as to what 

they believe are their iLearn responsibilities. For example, some course 

managers do not monitor the “Request a Schedule” feature in iLearn. This 

feature enables a prospective student to input the date by which a course 

is needed and add a comment. OIG found several instances where

students asked questions in the comment box and expected a response. 

Although iLearn is set up to send an e-mail notification to the course 

manager when the minimum number of students is reached, it does not 

notify the course manager when a student sends a comment or question. 

In addition, as noted previously in this report in Finding A, mandatory 

fields are often left blank because some course managers are not aware 

that they need to be completed. HRTD representatives stated that course 

managers are responsible for monitoring the list of students requesting a 
Audit of NRC’s iLearn Learning Management System

8

schedule before the minimum is reached; however, many course 

managers are unaware of this responsibility. Therefore, potential students 

may not get the requested course in the timeframe needed, and they may 

not get a response to their questions.

Lack of Written Policies and Procedures

Written policies and procedures describing course manager 

responsibilities and management’s expectations do not exist. Without 

established policies and procedures to clearly communicate 

management’s expectations regarding course manager duties and 

responsibilities, course managers may use inconsistent and less than 

optimal methods in carrying out their duties. This could affect students’ 

ability to obtain needed training in a timely manner to support mission 

accomplishment. Periodic training is provided to course managers by 

HRTD staff; however, this training is not mandatory. 

Recommendations

OIG recommends that the Executive Director for Operations:

4. Develop and implement written policies and procedures describing 

course manager duties and responsibilities.

5. Provide mandatory training to course managers specific to their iLearn 

duties and responsibilities.
Audit of NRC’s iLearn Learning Management System

9

IV. Other Matters

Although it was not the focus of our audit, based on a request from HRTD 

management, OIG looked into the matter of training activities that do not 

go through HR and program office procurement authority for these 

activities.

NRC delegations designate HR as having sole delegated authority to 

procure training (other than SF-1824training) in headquarters. Regions 

also have such authority to meet specific regional training needs. 

However, contrary to this authority, there have been instances of 

headquarters program offices procuring training without coordinating with 

HRTD. 

HRTD management has presented this issue to the Human Capital 

Council5as well as to the agency Program Management, Policy 

Development and Analysis staff. Additionally, as these instances surface, 

HRTD is reaching out to the program offices, reminding them of the 

delegation authority, and encouraging them to coordinate with HRTD on 

future acquisitions.

 

4

SF-182 is the form NRC employees use to request training outside of NRC. 

5

The Human Capital Council was established by the NRC Executive Director for Operations in concert 

with HR in July 2006. The purpose of the Human Capital Council is to provide an agency-level forum for 

the formulation of strategies to address human capital challenges, share best practices, and develop an 

integrated approach to address human capital issues.
Audit of NRC’s iLearn Learning Management System

10

V. CONSOLIDATED LIST OF RECOMMENDATIONS

OIG recommends that the Executive Director for Operations:

1. Issue guidance to course managers instructing them to complete all 

mandatory fields as described on the iLearn course input form.

2. Develop and implement a quality assurance program for all data in 

iLearn to ensure mandatory fields are complete and accurate.

3. Correct incomplete mandatory fields in iLearn.

4. Develop and implement written policies and procedures describing 

course manager duties and responsibilities.

5. Provide mandatory training to course managers specific to their iLearn 

duties and responsibilities.

VI. AGENCY COMMENTS

During an exit conference held on July 20, 2011, the agency agreed with the 

findings and recommendations in this audit report and said that no agency 

suggestions or comments would be provided in connection with the report. 
Audit of NRC’s iLearn Learning Management System

11

Appendix

SCOPE AND METHODOLOGY

The audit objective was to determine the effectiveness of the iLearn 

Learning Management System to support the agency's current and future 

training needs. The audit focused on reviewing the internal controls over 

iLearn through an examination of documents and by conducting 

interviews. OIG also performed benchmarking with other agencies. 

OIG reviewed relevant Federal regulations, including Office of 

Management and Budget Circular A-123, Management’s Responsibility for 

Internal Control; the Government Accountability Office’s Standards for 

Internal Control in the Federal Government; and the U.S. Code of Federal 

Regulations, Title 5, Part 410, Training. OIG reviewed agency guidance, 

including management directives, operating procedures, and HRTD’s 

strategic plan. In addition, OIG contacted the Government Accountability 

Office and the Library of Congress to obtain best practices regarding their 

respective learning management systems.

OIG interviewed HR management officials to obtain their insights into the 

implementation and oversight of iLearn. Additional interviews were 

conducted with course managers and administrative staff from 

headquarters, Region IV, and the Technical Training Center. 

OIG reviewed use of the prerequisite feature for 19 technical courses 

identified by the agency as having required prerequisites. OIG also 

reviewed use of the required fields as indicated on the iLearn course input 

form for 14 technical and non-technical courses in iLearn. 

We conducted this performance audit at NRC headquarters from 

September 2010 through April 2011 in accordance with generally 

accepted Government auditing standards. Those standards require that 

the audit is planned and performed with the objective of obtaining 

sufficient, appropriate evidence to provide a reasonable basis for any 

findings and conclusions based on the stated audit objective. OIG 

believes that the evidence obtained provides a reasonable basis for the 

report findings and conclusions based on the audit objective. Internal 

controls related to the audit objective were reviewed and analyzed. 

Throughout the audit, auditors were aware of the possibility or existence of 
Audit of NRC’s iLearn Learning Management System

12

fraud, waste, or misuse in the program. The audit work was conducted by 

Kathleen Stetson, Team Leader; Terri Cooper, Audit Manager; 

Kevin Nietmann, Senior Technical Advisor; and John Tornabane, 

Management Analyst.