# Audit of the Department's Efforts in 

# Identifying IRM KSAs

# 

### 

### FINAL AUDIT REPORT

### ED-OIG/A07-E0002

### August 2004

Our mission is to promote the efficiency, U.S. Department of Education

effectiveness, and integrity of the Office of Inspector General

Department's programs and operations. Kansas City, Missouri Office

##### NOTICE

Statements that managerial practices need improvements, as well as other

conclusions and recommendations in this report

represent the opinions of the Office of Inspector General.
Determinations of

corrective action to be taken will be made by the appropriate Department
of Education officials.

In accordance with Freedom of Information Act (5 U.S. C. § 552) reports

issued by the Office of Inspector General are available to

members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.

[MEMORANDUM]{.underline}

TO**:** William J. Leidinger,

####### Assistant Secretary for Management and Chief Information Officer

######## FROM: Helen Lew

########  Assistant Inspector General for Audit

######## SUBJECT: Final Audit Report - *Audit of the Department's Efforts in Identifying IRM KSAs*

####### *Control No. ED‑OIG/A07‑E0002*

Attached is the subject final audit report that covers the results of
our review of the Department's efforts in identifying Information
Resource Management (IRM) knowledge, skills, and abilities (KSAs) in
accordance with the Clinger-Cohen Act. An electronic copy has been
provided to your Audit Liaison Officer. We received your comments
concurring with the finding and recommendations in our draft report.

Corrective actions proposed (resolution phase) and implemented (closure
phase) by your office will be monitored and tracked through the
Department's Audit Accountability and Resolution Tracking System
(AARTS). ED policy requires that you develop a final corrective action
plan (CAP) for our review in the automated system within 30 days of the
issuance of this report. The CAP should set forth the specific action
items, and targeted completion dates, necessary to implement final
corrective actions on the finding and recommendations contained in this
final audit report.

In accordance with the Inspector General Act of 1978, as amended, the
Office of Inspector General is required to report to Congress twice a
year on the audits that remain unresolved after six months from the date
of issuance.

In accordance with the Freedom of Information Act (5 U.S.C. §552),
reports issued by the Office of Inspector General are available to
members of the press and general public to the extent information
contained therein is not subject to exemptions in the Act.

We appreciate the cooperation given us during this review. If you have
any questions, please call Richard J. Dowd, Regional Inspector General
for Audit, at 312-886-6503.

Enclosure

###### **Audit of the Department's Efforts in Identifying IRM KSAs**

######## 

######## **Table of Contents**

######## Executive Summary ..........................................................................................1

######## Audit Results .......................................................................................\.............2

#### Finding ‑ The Department May not be in Full Compliance With the Clinger-Cohen Act 

#### Requirements for Developing IRM KSAs ................\......................................\....2

######## Background ....................................................................................................5

######## Objectives, Scope, and Methodology.......................................................................6

######## Statement on Management Controls .....................................................................\...7

Appendix I -- Clinger-Cohen Core Competencies

Appendix II -- Auditee Comments

###### 

###### **Audit of the Department's Efforts in Identifying IRM KSAs**

######## 

######## **Table of Contents**

######## Executive Summary ..........................................................................................1

######## Audit Results .......................................................................................\.............2

#### Finding ‑ The Department May not be in Full Compliance With the Clinger-Cohen Act 

#### Requirements for Developing IRM KSAs ................\......................................\....2

######## Background ....................................................................................................5

######## Objectives, Scope, and Methodology.......................................................................6

######## Statement on Management Controls .....................................................................\...7

Appendix I -- Clinger-Cohen Core Competencies

Appendix II -- Auditee Comments

###### 

###### Audit of the Department's Efforts in Identifying IRM KSAs

######## 

######## 

######## Executive Summary

######## 

######## 

The Department of Education (Department) has made progress in complying
with the Clinger-Cohen Act[^1] requirements for obtaining KSAs necessary
to effectively perform IRM functions through limited workforce planning
efforts. However, it did not use a systematic process in evaluating
knowledge, skills, and abilities (KSAs); nor did it address the KSA
requirements for all IRM staff. Without having identified the needed
KSAs for all IRM staff, the Department was not able to develop a
comprehensive strategy to eliminate deficiencies between needed and
actual KSAs. As such, the Department's information resources management
(IRM) may lack the basic KSAs needed to effectively manage information
technology (IT) resources and investments; and to accomplish its goals.
We recommend that the Department 1) use a systematic process such as the
established core competencies in addressing the Clinger-Cohen
requirements related to KSAs for IRM; and 2) ensure that skill
assessments for the Office of the Chief Information Officer (OCIO) are
tied to the IRM goals included in the Department's overall strategic
plan.

We reviewed the Department's efforts to comply with the Clinger-Cohen
Act requirements for obtaining KSAs necessary to effectively perform IRM
functions. The objectives of our review were to determine the
Department's progress in 1) identifying the KSAs needed for IRM; 2)
developing a strategy to eliminate deficiencies between needed and
actual KSAs; and 3) reporting progress made in improving IRM capability.

The Act requires federal agencies to determine the KSAs required for
agency personnel in IRM and identify the current IRM staff
qualifications; develop a strategy for narrowing the gap between the
required KSAs and those of the current IRM staff; and report progress
made in improving IRM capability. The Act also requires the Chief
Information Officer (CIO) to assess the KSA requirements established for
IRM personnel and ensure that those requirements link to IRM performance
goals.

To assist federal agencies in complying with the requirements for
assessing the IRM KSAs, the CIO Council developed the Clinger-Cohen Core
Competencies to serve as a baseline for assessing KSAs. The established
core competencies provide a systematic process and are endorsed by
federal agencies. The Department did not use them in its KSA assessments
for the OCIO workforce; nor has it used them in assessing whether the
current requirements for its IRM workforce will enable it to meet its
IRM performance goals. The Department also has not provided evidence
that it used any specific guidance, criteria, or systematic process in
its workforce planning efforts or that the future requirements for the
IRM area have been coordinated with the Department's overall strategic
plan.

OCIO concurred with our finding and recommendations. In addition, based
on the Department's response that it is no longer considering a merger
of OCIO with the Office of Management (OM), we eliminated the discussion
of our concern about the Department's ability to maintain compliance
with the Act given its plans to merge those two offices.

###### Audit of the Department's Efforts in Identifying IRM KSAs

**Finding -- The Department may not be Effectively Managing** **its IT
Resources** **and Accomplishing** **Department Goals in Compliance With
the Clinger-Cohen Act**

The Department's workforce planning efforts have been limited --
directed at identifying a strategy for replacing staff expected to
retire in the next five years. However, the Department's planning
efforts did not address the KSAs required for the remaining IRM staff.
Further, the Department has not provided evidence that it used any
specific guidance, criteria, or systematic process in its limited
workforce planning efforts or that the future requirements for the IRM
area are consistent with the Department's overall strategic plan.
Without having identified the needed KSAs for all IRM staff, the
Department was not able to develop a comprehensive strategy to eliminate
deficiencies between needed and actual KSAs. Consequently, the
Department may not be effectively managing its IT resources and
accomplishing Department goals and, as a result, may not be in full
compliance with Clinger-Cohen Act requirements.

The Clinger-Cohen Act requires federal agencies to determine the KSAs
required for agency personnel in IRM and identify the current IRM staff
qualifications; develop a gap analysis and strategy for eliminating
differences between the required KSAs and those of the current IRM
staff; and report progress made in improving IRM capability. The Act
also requires the CIO to assess the KSA requirements established for
agency personnel in IRM and the adequacy of these requirements for
meeting IRM performance goals.

Specifically, the Clinger-Cohen Act (§ 5125(c)(3)) states that the CIO
of an agency shall

> annually, as part of the strategic planning and performance evaluation
> process...

(A) assess the requirements established for agency personnel regarding
    knowledge and skill in information resources management and the
    adequacy of such requirements for facilitating the achievement of
    the performance goals established for information resources
    management;

(B) assess the extent to which the positions and personnel at the
    executive level of the agency and the positions and personnel at
    management level of the agency below the executive level meet those
    requirements;

(C) in order to rectify any deficiency in meeting those requirements,
    develop strategies and specific plans for hiring, training, and
    professional development; and

(D) report to the head of the agency on the progress made in improving
    information resources management capability.

To assist federal agencies in complying with the requirements for
assessing the IRM KSAs, the CIO Council developed the Clinger-Cohen Core
Competencies to serve as a baseline for assessing KSAs. Although the
core competencies give agencies a great deal of latitude in KSA
assessments, they provide a systematic process for deliberations in
developing a set of KSAs needed in the IRM area. According to the CIO
Council, using the core competencies allows CIOs to assess KSA
requirements in compliance with the Clinger-Cohen Act. These core
competencies have been endorsed by government agencies as members of the
CIO Council, including the Office of Management and Budget (OMB), the
U.S. General Accounting Office (GAO), and the Office of Personnel
Management (OPM); and are used at the CIO University for training IRM
personnel in federal agencies.

In addition to the Clinger-Cohen requirements, the President's
Management Agenda includes requirements, under the Human Capital
initiative, to assess knowledge and skills for staff. It requires
agencies to assess the KSA requirements for personnel and determine
their adequacy in achieving the performance goals established for
agencies. According to GAO, the most important consideration in
identifying skills and competencies is clearly linking them to an
agency's mission and long-term goals. GAO stated that if an agency
identifies staff needs without linking those needs to strategic goals,
the needs assessment might be incomplete and premature.

The Department completed limited workforce-planning efforts, including
planning for the IT workforce, and reported the results of its efforts
in a Recruitment Plan. The Department's efforts focused on positions
where possible retirements in the next five years could leave vacancies.
The specific analyses performed included retirement eligibility,
succession planning with a focus on supervisory and managerial
positions, and an inventory of the skills and competencies needed by the
workforce to successfully accomplish the Department's mission. Although
the Department's plan identified a strategy for replacing staff expected
to retire in the next five years, it did not evaluate the KSA needs for
all IRM staff. Consequently, the Department's recruitment plan may not
accurately reflect its needs and any actions taken by the Principal
Offices may not meet the needs of both current and future workforce.

The Recruitment Plan stated that each Principal Office within the
Department completed both a skills assessment and a skills gap analysis.
However, without identifying the KSA needs for all IRM staff, the
Department could not develop a comprehensive strategy to eliminate
deficiencies between needed and actual KSAs. In addition, although the
Department's Recruitment Plan identified the most critical positions
within OCIO, OCIO provided no evidence that it performed any kind of
assessment of the actual position requirements, including an assessment
of whether those requirements assisted in meeting the IRM goals within
the Department\'s Strategic Plan.

OCIO's assessment focused on how it would fill positions that might
become vacant over the next five years due to employees retiring. OCIO
developed a plan for closing the gap in KSAs created through expected,
future retirements. The plan provided possible approaches for
backfilling positions, including 1) whether qualified individuals exist
in OCIO who could step into vacated positions; and 2) recruitment
strategies for filling vacated positions through identifying employees
elsewhere in the Department or through recruitment actions. Because the
Recruitment Plan focused only on retirement planning, it did not address
the KSAs required for the remaining IRM staff. As such, the Department's
workforce planning efforts, to date, have been limited and do not fully
comply with the Clinger-Cohen requirements for assessing IRM KSAs.

The Department's E-Government report to OMB provided information from
the Department's Recruitment Plan. The Department also reported that it
had developed specific training curriculum to address identified
deficiencies in the information security area; it would ensure that IT
Project Managers have the skills necessary; and it would be tracking
certifications of all IT Project Managers in the future. In addition,
the report stated that the Department has developed a competency
self-assessment tool that will assist in identifying individual
competency development needs in the current workforce. This tool, known
as the Employee Skills Inventory System (ESIS), is a voluntary,
web-based electronic self-assessment tool that employees can use to
identify competencies related to their jobs and assess their skills
based on the competencies. Its E-Government report indicates the
Department's willingness to address identified deficiencies. The
Department's reported actions are in various stages of implementation,
however, the effective implementation of all or any combination of the
reported actions would not change our report findings.

According to the CIO Council, performing effectively in the established
competency areas and possessing the knowledge, skills, and abilities
under each competency area assists agencies in complying with KSA
requirements in the Clinger-Cohen Act. Failure to use a systematic
approach such as the established core competencies could result in the
Department's failure to comply with the Act's requirements. More
specifically, because it did not assess its entire IRM workforce against
established competencies, the Department may not have effectively
determined where important skill gaps are and how to efficiently and
effectively address those gaps. As a result, the Department's
information resource management may not have the basic core competencies
or KSAs needed to effectively manage IT resources and investments. In
addition, without a workforce plan that delineates the relationship
between KSA requirements and the Department's IRM goals, the Department
could have difficulty identifying current and future KSAs needed to
accomplish its goals.

#### ***Recommendations***

We recommend that the Assistant Secretary, Office of Management and
Chief Information Officer

1.  Use a systematic process such as the established core competencies
    in addressing the Clinger-Cohen Act requirements related to KSAs for
    all IRM staff;

2.  Develop a comprehensive strategy to eliminate deficiencies between
    needed and actual KSAs; and

3.  Ensure that skill assessments for OCIO are tied to the IRM
    performance goals included in the Department's overall strategic
    plan.

**[The Department's Comments and OIG Response]{.underline}**

OCIO concurred with our finding and recommendations and provided a
corrective action plan. Based on the Department's response that it is no
longer considering a merger of OCIO with the Office of Management (OM),
we eliminated the discussion of our concern about the Department's
ability to maintain compliance with the Act given its plans to merge
those two offices.

###### Audit of the Department's Efforts in Identifying IRM KSAs

##### 

######## 

######## Background

The Clinger-Cohen Act was enacted to address longstanding problems
related to federal IT management. Among other things, it requires
federal agencies to

-   Determine the KSAs required for agency personnel in IRM;

-   Determine the extent positions and personnel at executive and
    management level meet those requirements;

-   Develop strategies for narrowing the gap between the required KSAs
    and those of the current IRM staff, including specific plans for
    hiring, training, and professional development for any identified
    deficiency; and

-   Report progress made in improving IRM capability.

OMB, GAO, and OPM provide guidance for implementing the Clinger-Cohen
Act, including requirements for obtaining and retaining the necessary
KSA's for IRM. This guidance defines what an agency would need to
accomplish in order to comply with the Act. In addition, the CIO Council
developed a set of core competencies to assist agencies in complying
with the Clinger-Cohen Act's requirements for assessing KSAs in the IRM
area. The established core competencies are organized into12 areas with
detailed core competencies or KSAs under each area. These areas include
Leadership/Managerial, Project/Program Management, Information Resources
Strategy and Planning, Enterprise Architecture, Capital Planning and
Investment Assessment, and IT security/information assurance. For a
complete list of the 12 areas and the core competencies associated with
each see the Appendix.

###### Audit of the Department's Efforts in Identifying IRM KSAs

######## 

######## Objectives, Scope, and Methodology

The objectives of our audit were to determine the Department's progress
in 1) identifying the KSAs needed for IRM; 2) developing a strategy to
eliminate deficiencies between needed and actual KSAs; and 3) reporting
progress made in improving IRM capability. We did not assess the KSAs
for OCIO organizationally nor did we assess KSAs of individuals within
the Department's IRM area.

To accomplish our objective, we reviewed applicable policies and
procedures, as well as laws, regulations, and agency guidelines. We
interviewed officials in the CIO's office, including the CIO, to obtain
information on the Department's goals, strategies, and staffing plans.
We obtained and reviewed the Department's strategic plan, including the
IRM section on strategic planning and workforce analyses; and strategic
program planning documents, including the plan that guided staffing and
the annual staffing plan. To meet our objectives, we did not use
electronic data from the Department.

To assist in assessing the Department's efforts, we reviewed GAO reports
on human capital and workforce planning at other federal agencies. We
also reviewed human capital literature-including OPM's Human Capital
Assessment and Accountability Framework as well as workforce planning
models at OPM, OMB, and GAO.

We conducted work at the Department's CIO offices in Washington, D.C.
and our OIG office in Kansas City, MO, during the period October 2003 to
April 2004. We held an exit conference with Department officials on June
15, 2004. Our audit was performed in accordance with generally accepted
government auditing standards appropriate to the scope of the review.

###### Audit of the Department's Efforts in Identifying IRM KSAs

######## 

###### Statement on Management Controls

**As part of our review, we gained an understanding of the Department's
management control structure applicable to the scope of the review. For
purposes of this review, we assessed and classified the significant
management controls related to the Department's IT efforts into the
planning and assessment activities over the Department's IRM
capabilities. The assessment also included a determination of whether
the processes used by the Department provided a reasonable level of
assurance of compliance with the Clinger-Cohen Act.**

Because of inherent limitations, and the limited nature of our review, a
study and evaluation made for the limited purpose described above would
not necessarily disclose material weaknesses in the management control
structure. However, our assessment identified a weakness in the
Department's efforts to identify the KSAs needed for its IRM as set out
in the *Findings* section of this report.

###### 

###### **Audit of the Department's Efforts in Identifying IRM KSAs**

######## 

**Appendix I \-- Clinger-Cohen Core Competencies (Revised June 2003)**

**The Clinger-Cohen Core Competencies, developed by the CIO Council,
have been endorsed to serve as a baseline to assist government agencies
in complying with Section 5125(C)(3) of the Clinger-Cohen Act. To
perform effectively in each competency area below, an organization
should possess the knowledge, skills, and abilities in each
competency.**

1.0 Policy and Organizational

  -----------------------------------------------------------------------
  1.1 Department/Agency missions, organization, functions, policies,
  procedures

  1.2 Governing laws and regulations (e.g., the Clinger-Cohen Act,
  E-Government Act, GPRA, PRA, GPEA, OMB Circulars A-11 and A-130, PDD
  63)

  1.3 Federal government decision-making, policy making process and
  budget formulation and execution process

  1.4 Linkages and interrelationships among Agency Heads, COO, CIO, and
  CFO functions

  1.5 Intergovernmental programs, policies, and processes

  1.6 Privacy and security

  1.7 Information management
  -----------------------------------------------------------------------

2.0 Leadership/Managerial

  -----------------------------------------------------------------------
  2.1 Defining roles, skill sets, and responsibilities of Senior
  Officials, CIO staff and stakeholders

  2.2 Methods for building federal IT management and technical staff
  expertise

  2.3 Competency testing - standards, certification, and performance
  assessment

  2.4 Partnership/team-building techniques

  2.5 Personnel performance management techniques

  2.6 Principles and practices of knowledge management

  2.7 Practices which attract and retain qualified IT personnel
  -----------------------------------------------------------------------

3.0 Process/Change Management

  -----------------------------------------------------------------------
  3.1 Techniques/models of organizational development and change

  3.2 Techniques and models of process management and control

  3.3 Modeling and simulation tools and methods

  3.4 Quality improvement models and methods

  3.5 Business process redesign/reengineering models and methods
  -----------------------------------------------------------------------

4.0 Information Resources Strategy and Planning

  -----------------------------------------------------------------------
  4.1 IT baseline assessment analysis

  4.2 Interdepartmental, inter-agency IT functional analysis

  4.3 IT planning methodologies

  4.4 Contingency planning

  4.5 Monitoring and evaluation methods and techniques
  -----------------------------------------------------------------------

5.0 IT Performance Assessment: Models and Methods

  -----------------------------------------------------------------------
  5.1 GPRA and IT: Measuring the business value of IT, and customer
  satisfaction

  5.2 Monitoring and measuring new system development: When and how to
  \"pull the plug\" on systems

  5.3 Measuring IT success: practical and impractical approaches

  5.4 Processes and tools for creating, administering, and analyzing
  survey questionnaires

  5.5 Techniques for defining and selecting effective performance
  measures

  5.6 Examples of, and criteria for, performance evaluation

  5.7 Managing IT reviews and oversight processes
  -----------------------------------------------------------------------

6.0 Project/Program Management

  -----------------------------------------------------------------------
  6.1 Project scope/requirements management

  6.2 Project integration management

  6.3 Project time/cost/performance management

  6.4 Project quality management

  6.5 Project risk management

  6.6 Project procurement management

  6.7 System life cycle management

  6.8 Software development
  -----------------------------------------------------------------------

7.0 Capital Planning and Investment Assessment

  -----------------------------------------------------------------------
  7.1 Best practices

  7.2 Cost benefit, economic, and risk analysis

  7.3 Risk management-models and methods

  7.4 Weighing benefits of alternative IT investments

  7.5 Capital investment analysis-models and methods

  7.6 Business case analysis

  7.7 Integrating performance with mission and budget process

  7.8 Investment review process

  7.9 Intergovernmental, Federal, State, and Local Projects
  -----------------------------------------------------------------------

8.0 Acquisition

  -----------------------------------------------------------------------
  8.1 Alternative functional approaches (necessity, government, IT)
  analysis

  8.2 Alternative acquisition models

  8.3 Streamlined acquisition methodologies

  8.4 Post-award IT contract management models and methods, including
  past performance evaluation

  8.5 IT acquisition best practices
  -----------------------------------------------------------------------

9.0 E-Government/Electronic Business/Electronic Commerce

  -----------------------------------------------------------------------
  9.1 Strategic business issues & changes w/advent of E-Gov/EB/EC

  9.2 Web development strategies

  9.3 Industry standards and practices for communications

  9.4 Channel issues (supply chains)

  9.5 Dynamic pricing

  9.6 Consumer/citizen information services

  9.7 Social issues
  -----------------------------------------------------------------------

10.0 IT security/information assurance

  -----------------------------------------------------------------------
  10.1 Fundamental principles and best practices in IA

  10.2 Threats and vulnerabilities to IT systems

  10.3 Legal and policy issues for management and end users

  10.4 Sources for IT security assistance

  10.5 Standard operating procedures for reacting to intrusions/misuse of
  federal IT systems
  -----------------------------------------------------------------------

11.0 Enterprise Architecture

  -----------------------------------------------------------------------
  11.1 Enterprise architecture functions and governance

  11.2 Key enterprise architecture concepts

  11.3 Enterprise architecture development and maintenance

  11.4 Use of enterprise architecture in IT investment decision making

  11.5 Interpretation of enterprise architecture models and artifacts

  11.6 Data management

  11.7 Performance measurement for enterprise architecture
  -----------------------------------------------------------------------

**12.0 Technical**

  -----------------------------------------------------------------------
  12.1 Emerging/developing technologies

  12.2 Information delivery technology (internet, intranet, kiosks, etc.)

  12.3 Desk Top Technology Tools
  -----------------------------------------------------------------------

#######  

#######  **Source: Chief Information Officers Council**

######## 

######## **Appendix II** -- Auditee Comments on the Draft Report 

[^1]: Previously referred to as the Information Technology Management
    Reform Act of 1996, Division E of Public Law 104-106, 110 Stat. 679
    (1996).