Federal Training Investments: Office of Personnel Management and Agencies Can Do More to Ensure Cost-Effective Decisions
Released: 2012-09-17T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: OPM and agency CHCOs play an important role in ensuring that federal training dollars are invested effectively. GAO was asked to review the extent to which: (1) CHCOs of selected federal agencies have established processes to set and prioritize training investments that are aligned with leading practices; and (2) OPM’s guidance and assistance for developing training investment strategies align with these leading practices. GAO obtained information from 27 CHCOs on their training investment practices through a questionnaire, and selected four agencies—the Departments of Energy (DOE), Homeland Security (DHS), the Interior (DOI) and Veterans Affairs (VA)—to provide illustrative examples. We compared both CHCO and OPM practices to leading practices, identified through past GAO and expert studies.

section_title: What GAO Found
paragraphs: Many Chief Human Capital Officers (CHCOs) reported that they are implementing several leading practices important to making strategic decisions about training delivery, such as determining the best mix of decentralized and centralized training and considering government-wide reform when planning training. However, many CHCOs reported they are not implementing some practices that support making more cost-effective training investment decisions, such as prioritizing training so that the most important needs are met first and evaluating the benefits of training. In addition, many CHCOs do not have information from component or sub-agency leaders regarding their level of investments and priorities. Consequently, some agencies are duplicating internal across their agencies. Federal agencies also need reliable information on how much they spend on training and for what purposes. However, several CHCOs reported they do not completely and reliably track training costs agency-wide.

The Office of Personnel Management (OPM) provides guidance and assistance to agencies on a number of the leading practices, such as evaluating the benefits of training in three of its guides and in workshops. In some practice areas thatare challenges to agencies, such as prioritization of investments and determining whether to design training and development programs in-house or obtain these services from a contractor, guidance is minimal or absent. OPM also requires agencies to submit training investment data and provides guidance on how to do so, but considers this data to be unreliable because it is incomplete. However, OPM officials have not internally assessed improvements in the completeness of the data over the last 3 years or the quality of the data in the six years that agencies have been required to submit it, and have only provided agencies with one summary of their data for correction. Agencies and OPM reported there are also opportunities for OPM to help agencies reduce duplicative investments across agencies. For example, currently, agencies independently purchase or develop training for the same mandated or common occupational training. Agency leaders and OPM recognize that this has led to redundant and inefficient federal training investments. According to OPM officials, HR University—which is a website currently administered by OPM to provide training for the HR community—has already resulted in a cost savings of $14.5 million as a result of sharing the best HR training government-wide. Several agencies and OPM officials reported that HR University could be expanded to provide mandatory mtraining and serve as a model for centralizing training in other occupations or functional areas, which could save millions more and help standardize training.

section_title: What GAO Recommends
paragraphs: GAO recommends, among other things, that OPM improve guidance and assistance to agencies inestablishing a process for setting and prioritizing training investments; improve the reliability of agency training investment information; and identify the best existing courses that fulfill governmentwide training requirements, and offer them to all agencies through the HR University or other appropriate platforms. OPM fully or partially concurred with four recommendations and did not concur with a portion of another. OPM, DOI and VA provided technical comments, which GAO incorporated, as appropriate, into the report. DOE and DHS had no comments.

section_title: Letter
paragraphs: Constrained budgets and the need to address gaps in critical federal skills  and competencies make it is essential that agencies identify the  appropriate level of investment and establish priorities for employee  training and development, so that the most important training needs are  addressed first. The Office of Personnel Management (OPM) and  agency chief human capital officers (CHCOs) play important roles in  ensuring that federal training dollars are invested wisely. OPM provides  leadership and guidance on the establishment and operation of efficient  federal training and development programs government-wide and advises  the President on means for furthering and strengthening federal training  programs. CHCOs statutorily have the primary role in setting the  workforce development strategies of their agencies, and advising and  assisting their agency heads who are responsible for establishing  priorities for needed training and development, and providing for the use  of funds and resources in accordance with these priorities. The decisions  that these key players make in determining how to invest federal training  dollars will ultimately have an important impact on how well the federal  government is equipped to address its current and future performance  and fiscal goals and challenges.

In order to better understand how federal training investment decisions  are made and whether improvements are needed, you asked us to review  the methods that agencies are using to establish their training investment  strategies and OPM’s training investment guidance to agencies.

Accordingly, this review assesses the extent to which (1) CHCOs of  selected federal agencies have established processes to set and prioritize  training investments that are aligned with leading practices and (2) OPM’s  guidance and assistance for developing training investment strategies  align with these leading practices.

For the purposes of this review, we define the key terms “training”,  “development” and “agency-wide” in the following ways:

Training is making available to employees planned and coordinated  educational programs of instruction in professional, technical, or other  fields that are or will be related to the employee’s job responsibilities.  Training can be accomplished through a variety of approaches, such  as classroom training, e-learning, and professional conferences that  are educational or instructional in nature.

Development is generally considered to include training, structured  on-the-job learning experiences, and education. Developmental  programs can include experiences such as coaching, mentoring, or  rotational assignments.

Agency-wide includes all components, sub-agencies or offices within  a cabinet department or independent agency.

For both objectives of the review, we compared OPM and CHCO  practices against eight federal training investment leading practices,  which are based on our prior studies; other expert studies; and statutory,  regulatory and executive order training requirements. (See table 1). OPM  reviewed these criteria and agreed that they are practices that agencies  should be implementing to support effective training investment decisions.  While some of the leading practices we identified are related to statutory,  regulatory and executive order training requirements, the leading practice  questions we examined were not developed to assess whether agencies  are compliant with these requirements.responses to our questions do not indicate whether or not agencies are in  compliance with statutory, regulatory, and executive order training  requirements. Therefore, we do not make determinations regarding  compliance with these requirements in this review.

To obtain government-wide information on agency training investment  practices, we obtained high-level information from a Council made up of  CHCOs representing 27 agencies (known as the CHCO Council) through  a questionnaire on their training investment practices and processes. To  obtain additional perspective and insights on the training investment  practices identified in the questionnaire, we discussed the responses with  the CHCO Council and a Council made up of the Chief Learning Officers  (CLO) from the 27 agencies (known as the CLO Council). We describe  these Councils in greater detail in the background of this report. Based on  the responses to the questionnaire and workforce size, we selected four  agencies—the Department of Homeland Security (DHS), Department of  Veterans Affairs (VA), Department of the Interior (DOI), and Department  of Energy (DOE)—from which to obtain illustrative examples of how they  implemented the training investment practices identified in the  questionnaire. As part of our review of agency practices, we also obtained  information on the steps that agencies are taking to identify and prioritize  investment allocations for training required to implement the GPRA  Modernization Act of 2010 (GPRAMA).

To identify and assess OPM’s guidance and assistance to agencies on  training investment strategies, we reviewed OPM training guidance, and  relevant documentation on forums, workshops or other assistance and  activities. In addition, we interviewed officials from the OPM offices with  primary responsibility for providing training policy guidance and technical  assistance to agencies. We compared this information to the leading  practices identified in table 1. We also identified and described the steps  that OPM has taken to identify the skills and training needed to implement  performance management improvements, such as those required by  GPRAMA, as a foundation for future agency training investments.  However, we did not assess the effectiveness of OPM’s efforts to identify  GPRAMA-related skills and actions to develop related training.

Based on information obtained from agencies and OPM, we assessed  which leading training investment practices were being implemented by  agencies and addressed by OPM guidance and assistance. We also  identified the challenges or limitations reported by agencies to  implementing the practices, and opportunities for improvement in agency  processes and related OPM guidance.

We conducted this performance audit from December 2011 to September  2012, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Further details of our  objectives, scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: The federal government has established a policy to develop its  employees through training programs, to improve public service, increase  efficiency and economy, and build and retain a force of skilled and  efficient employees, among other things. In 1967, President Johnson  signed Executive Order No. 11348, to provide agency heads and OPM  with presidential direction on how training is to be carried out. Under  Executive Order No. 11348, OPM is responsible for planning and  promoting the development, improvement, coordination, and evaluation of  training in accordance with chapter 41 of title 5 of the U.S. Code and the  established policy. Chapter 41 of title 5 sets forth the statutory framework  for federal government training and development. The executive order  further requires OPM to identify functional areas in which new or  expanded interagency training activity is needed and either conduct such  training or arrange for agencies having the substantive competence to do  so; as well as to coordinate interagency training conducted by and for  agencies. It also requires OPM to assist agencies in developing sound  programs and financial plans for training and provide advice, information,  and assistance to agencies on planning, programming, budgeting,  operating, and evaluating training programs. In addition to these activities,  OPM provides advice and assistance to agencies on training and  development programs. OPM’s Training and Executive Development  (TED) group (a subcomponent within the Office of Executive Resources)  is the primary office that provides policy direction and leadership to  agencies in developing plans and strategies to implement training and  development programs. It also provides agency guidance to ensure the  government’s training and development programs support strategic  human capital investments. The TED group provides assistance through  two main mechanisms: guidance documents and technical assistance.  OPM has developed five guides that agencies can use as references for  different aspects of making or reporting training investment decisions in  the planning, designing, implementation, and evaluation phases of their  training and development programs (see table 2).

The TED group also provides technical assistance on agency training  investments through facilitating discussions and forums, and providing  training to agencies’ human resources (HR) staff. For example, the TED  group uses various web-based mechanisms—such as OPM’s website,  OPM LISTSERV, OPM Federal Training and Development web site and  OPM Federal Training and Development Wiki—to facilitate discussions  between agencies on training investments and to share guidance with  agencies. In addition to these facilitated discussions and forums, the TED  group provides training to federal HR professionals in various areas,  including activities that support making training investment decisions. For  example, OPM provides training to HR staff through its partnership with  the CHCO Council to operate HR University, which OPM officials and the  HR University website report is the federal government’s single “one stop”  training resource center for the HR professional throughout the federal  government. HR University is an effort that is intended to achieve  government-wide savings through pooling and sharing training resources  and identifying the best HR training across government.

Agencies have the primary responsibility for establishing, operating,  maintaining, and evaluating their training programs in support of  achieving their mission and goals. OPM regulations specify that agency  employee developmental plans and programs should be designed to build  or support an agency workforce capable of achieving agency mission and  performance goals and facilitating continuous improvement of employee  and organizational performance. Furthermore, Executive Order No.  11348 states that agency heads must undertake several activities in  support of developing employees, including:

Review periodically, but not less often than annually, the agency’s  program to identify training needed to bring about more effective  performance at the least possible cost;

Conduct periodic reviews of individual employee’s training needs as  related to program objectives;

Conduct research related to training objectives required for program

Plan, program, and evaluate training for both short and long-range  program needs by occupations, organizations, or other appropriate  groups;

Establish priorities for needed training, and provide for the use of  funds and man-hours in accordance with these priorities;

Establish training facilities and services as needed;

Extend agency training programs to employees of other agencies and  assign his employees to interagency training whenever this will result  in better training, improved service, or savings to the government.

The CHCO Council, established under the Chief Human Capital Officers  Act of 2002, provides assistance to OPM and agencies in accomplishing  federal human capital goals. The 25-member CHCO Council is  composed of the Director of the OPM, who serves as chairman; the  Deputy Director for Management of the Office of Management and  Budget (OMB), who acts as vice chairman; the CHCOs of the 15  executive departments; and the CHCOs of 8 additional agencies  designated by the OPM Director. Additionally, the CHCO Council has an  Executive Director from OPM who coordinates and oversees the activities  of the council. The CHCO Council supports OPM in leading federal  agencies in the strategic management of human capital, providing a  forum for senior management officials to exchange HR best practices,  and informing the dialogue on civil service reform in order to build and  maintain an outstanding Federal workforce for the Nation. According to  the CHCO Council’s charter, among other purposes, the council is to: advocate and assure a culture of continuous learning and high  performance, developing and implementing effective strategies to  attract, develop, manage, and retain employees with superior abilities;  identify human capital best practices and benchmarks, and apply  those exemplars to their agencies and the federal government as a  whole; and provide leadership in identifying and addressing the needs of the  federal government’s human capital community, including training and  development.

To help CHCOs implement their training goals, many of the 24 Chief  Financial Officers Act agenciesLearning Officers (CLOs). These officers subsequently  formed an  informal Chief Learning Officers (CLO) Council, which is a community of   and smaller agencies established Chief  practice composed of federal CLOs or their equivalents who meet  periodically to share best practices and create learning opportunities for  agencies and organizations. The purpose of the CLO Council is to  provide a regular forum for CLOs to discuss and collaborate on high-level  agency strategic and operational issues affecting the Federal learning  and workforce development community of the federal government. These  two Councils, in partnership with OPM are to play a key role in assisting  agencies in the implementation of federal training and development  efforts.

section_title: CHCOs are Implementing Leading Practices Related to Training Delivery, but Not Practices that Support Making More Cost-Effective Training Investments
paragraphs: Many CHCOs reported that they are implementing leading practices we  identified as being important to making strategic training and development  investment decisions, especially regarding the delivery of training. These  practices include determining the best mix of decentralized and  centralized training, considering government-wide reforms when  identifying their training needs, and measuring employee satisfaction with  training, among other things. However, many CHCOs reported that they  are not implementing the leading practices that would allow them to make  more cost-effective training decisions, such as having an agency-wide  process for prioritizing training investments so that the most important  training needs are addressed first and comparing the merits of different  delivery mechanisms (e.g. classroom or computer-based training) to  determine what mix of mechanisms will be most efficient and cost- effective. All of these practices are important to ensuring that training  investments will be both effective and efficient in equipping federal  employees to accomplish their agencies’ goals.
subsections: section_title: CHCOs Reported Implementing Leading Practices that Support Making Strategic Training Delivery Decisions
paragraphs: Many CHCOs reported that they are implementing six of the leading  practices that we identified as being important to making strategic training  and development investment decisions, especially regarding the delivery  of training, as shown in Table 3. However, regarding the leading practice  related to tracking training investments agency-wide, we found that even  those who reported that they track training agency-wide did not do so  completely or reliably.

All CHCOs reported that their agencies have implemented this practice.  We have previously reported that, while neither approach fits every  situation, agencies need to consciously think about the advantages and  disadvantages of using centralized and decentralized approaches,  particularly for the design of training and development programs.

Centralizing design can enhance consistency of training content and offer  potential cost savings. A decentralized approach to training design can  enable agencies to tailor training programs to better meet local and  organizational unit needs. Agencies with decentralized approaches often  embed training representatives within their business lines and field  structures to assist in coordination of training efforts, including design and  development. Nineteen of the 27 agencies reported that they have both  centralized and decentralized training processes, while eight reported  having completely decentralized training processes. Most of these  agencies reported that their CHCOs or CHCO staff typically make  centralized training decisions, while the leadership within the  components, subagencies or offices make mission-specific training  decisions. In the questionnaire responses, CHCOs identified a range of  officials who are involved in making training investment decisions at the  corporate and sub-agency level, including CHCOs and their staff, chief  management officers, chief executive officers, budget officers, chief  information officers, and others. A number of agencies also reported that  advisory or oversight boards or training universities within their agency  are involved in making training investment decisions.

In the four agencies that we selected for review to obtain illustrative  examples of how they implemented the training investment practices, the  CHCOs or their representatives reported that their agencies made a  decision to have both centralized and decentralized processes because  they believe that the components or sub-agencies are more  knowledgeable about their mission-specific training needs, while the  central human capital staff can add the most value by managing  investment decisions for more general training across the department.  VA—which was one of the four agencies that we selected—established a  corporate university known as the Veterans Affairs Learning University  (VALU) to provide training to all VA employees. VALU provides training  primarily in general areas such as leadership, management and  supervision, as well some career and technical training. VALU offers  training to the administrations and staff offices through a request process  that is based on the training needs that the administrations and staff  offices identify. Those training needs are required to be aligned to VA  critical training areas. An Enterprise Training Advisory Board,  established in April 2012, also advises the Dean of VALU on the impact of  training, potential training development, and methods of delivery.

However, another tier of training is also provided within VA’s three  administrations—the Veterans Health Administration (VHA), Veterans  Benefits Administration (VBA), and National Cemetery Administration.  Each administration independently makes training investment decisions  and provides training to its employees in mission-specific and some  general and mandatory areas. The leadership of each administration  makes decisions about the level and prioritization of these training  investments. For example, at the VHA the Associate Deputy Under  Secretary for Health (or equivalent) and, subsequently, the Deputy Under  Secretary for Health assess the training requested by their offices against  various criteria, including whether training requests are aligned with and  support VA and VHA strategic goals, objectives, strategies, initiatives,  and performance improvement goals. During each review these officials  prioritize the requests through a voting process, and forward selected  training to the next level. Ultimately, the training is sent for approval to the  Under Secretary for Health and the VA Chief of Staff. The other three  agencies that we met with also reported having both centralized and  decentralized processes for making mission-specific training investment  decisions. However, most often, decentralized training decisions were not  required to be vetted with department level leadership for these three  agencies.

Nearly all CHCOs in our review reported that they have a process for  these considerations. We have previously reported that when planning  training and development efforts, agencies should look to the actions of  the administration, Congress, and internal and external auditors by  considering administration priorities, legislative reforms, and major  management challenges that might shape agency priorities and strategies  for training and development. As an administration focuses its efforts on  addressing its priorities, agencies can benefit by having mechanisms or  processes for considering whether and to what extent these initiatives  could be linked to employees’ skills and competencies and the related  training and development approaches that might be needed. Twenty- three of the 27 CHCOs who responded to our questionnaire reported  having such a process in place. For example, 16 of the CHCOs reported  that they are already setting investment allocations or training priorities to  implement GPRAMA.

At DOE, another agency we selected for review to obtain illustrative  examples, officials reported that they have identified the training that the  department currently offers and will need to offer to implement GPRAMA.  The Secretary issued a memo to DOE employees on GRPAMA’s  implementation and is holding town hall meetings on improving  organizational performance. Another effort that DOE expects to support  the implementation of GPRAMA is the Goals-Engagement-Accountability- Results (GEAR) model that OPM and OMB are helping to pilot in DOE  and four other federal agencies, which includes efforts to improve  employee performance, among other things. and related documentation, DOE’s GEAR implementation plan includes  aligning employee performance management with organization  performance management and developing training to support these  goals, which along with initiating knowledge sharing activities, will  promote improvement of DOE’s organizational performance.

Beginning in late May 2011, a workgroup of the National Council on Federal Labor- Management Relations (LMR) partnered with members of the CHCO Council to develop a  new model of employee performance management, referred to as GEAR. GEAR focuses  on articulating a high-performance culture, aligning employee performance engagement  with organizational performance management, implementing accountability at all levels,  and create a culture of engagement. OPM is piloting GEAR at five agencies—the Housing  and Urban Development, the DOE, the Coast Guard, OPM and VA.
subsections: section_title: Have Criteria for Determining Whether to Design Training and Development Programs in- house or to Obtain These Services from a Contractor or Other External Source
paragraphs: Many of the CHCOs in our review reported having criteria for this  purpose. Training can be provided by the agency itself, another  government agency, a school, a manufacturer, a professional association,  or other competent persons or groups in or outside of government. To aid  in making these decisions, agencies should try to develop clear criteria for  determining when to contract for training and development services. We  have previously stated that factors that agencies should consider in these  decisions include the capability of in-house staff to develop and  implement the training; the prior experience, capability, and stability of  possible providers in the marketplace; and agency limitations on cost,  time, and resources Of the 27 CHCOs included in our questionnaire, 15  reported that they have criteria for determining whether to design training  and development programs in-house or obtain these services from a  contractor or other external source.

One agency that we selected for review to obtain illustrative examples  was DOI, which reported implementing this practice, however, the extent  to which this decision-making process is implemented agency-wide is  unclear. In its questionnaire response, DOI’s CHCO reported that DOI’s  Office of Strategic Employee and Organizational Development has  responsibility for offering corporate training through DOI’s university. This  office decides whether to “make or buy” departmentwide training. When  we met with DOI officials in the course of our review, they explained that  although almost all courses are delivered by vendors because DOI has  no internal trainers, they do have a small cadre of instructional designers  who can develop some e-Learning courses. Decisions on whether to  develop the courses internally are based on various criteria, including  whether a course can be developed quickly, does not require a significant  amount of content development, and subject matter experts can be  provided to support the course development. Although this department  level process is useful, DOI officials did not know if the bureaus within the  department consistently use a “make or buy” approach. They reported  that the larger bureaus have some capacity for in-house development  while the smaller bureaus do not have this capability.

Many CHCOs reported that their agencies implement this practice,  although most of the CHCOs who reported that they do not track training  investments agency-wide were leaders of the agencies with the largest  workforces. We have previously reported that to obtain a  comprehensive determination of the costs of these initiatives, agencies  need to find ways around barriers that prevent them from fully and  accurately identifying the expenses associated with all components of  their training and development processes. These costs can include  expenses for instructional development; participant and instructor  attendance; facility, material, and equipment costs, and travel and per  diem expenses. To track the cost and delivery of training and  development programs, agencies need credible and reliable data from  learning management systems as well as accounting, financial, and  performance reporting systems. To the extent possible, agencies also  need to ensure data consistency across the organization (such as having  data elements that are pulled from various systems representing the  same type of information). Variation in the methods used to collect data  can greatly affect the analysis of uniform, quality data on the cost and  delivery of training and development programs. In response to our  questionnaire, 16 CHCOs reported that they track training investments  agency-wide.

A learning management system is a software application that automates the  administration, tracking, and reporting of training events. and stated that they could not provide reliable training data to OPM,  which requests these data to address its government-wide training  responsibilities. Under OPM regulations, agencies are required to  maintain data on training activities and expenditures and submit these  data to OPM.

As an example of challenges tracking training investments, DHS reported  that it is unable to track or better leverage training investments across the  department because of the nine, major incompatible Learning  Management Systems that it uses to track training throughout the agency.  We highlighted these same challenges in a 2005 report on DHS training,  noting that the lack of common management information systems and the  absence of commonly understood training terminology across  components, among other things, may impede the agency’s ability to  achieve its training goals. According to more recent documentation on  the limitations of DHS’ tracking systems, the components’ disparate  systems currently limit them from sharing useful training information  across the department, effectively aggregating training data agency-wide,  and reporting complete training investment information to OPM. As a  result, DHS is seeking to purchase a single learning management  system.

Even when agencies had a single training information system, the  components may not consistently use them to track training investments  because of inconsistent coding schemes for tracking similar training  activities. For example, even though DOI has a single system for tracking  training information, officials reported that their human capital office must  rely on employees or data stewards to input their training data and some cost data may not be included, such as training travel costs, or  certain types of training may not be entered, such as conferences. Such  costs are sometimes paid directly by an employee’s immediate office  using a government credit card and are not tracked as training. In  addition, learning management system training data are often not  reconciled with DOI’s financial expenditure data because, until recently,  their financial systems have not captured education tuition and training  fees, and are still unable to track training travel costs. Therefore DOI’s  cost data are most likely incomplete. Similarly, officials from DOE, DHS,  and VA reported that they are aware of some inconsistencies in whether  some types of training, such as conferences, are entered into their  learning management systems. They also stated that there are  inconsistencies in how agency components capture and code workforce  training into their system because they lack a common definition for what  types of activities should be considered training or have varying coding  schemes or tools for capturing the cost. For example, some organizations  use a procurement request or obtain a contractor to deliver training and  do not document these costs using the standard government form for  tracking data or in their learning management systems. In other cases,  training investment data are captured using different coding in various  learning management systems and financial systems. Some officials  report that reconciling these data would be difficult. For example, DOE’s  chief financial officer reported that it takes a couple of months to gather  training investment data from DOE’s various systems, partly because the  systems have inconsistent coding for these data. DOE’ officials reported  that changing financial codes to reconcile training data would be time  consuming and expensive because their financial systems are 20 years  old. However, after years of highlighting this challenge, they are seeking  approval to make such changes. Officials from the four agencies  generally reported that, as a result of all of these factors, there is no  overarching awareness or oversight of how much is spent on training  investments and for which activities .

Nearly all CHCOs reported having a formal process to evaluate employee  satisfaction with training, but fewer had processes to evaluate the impact  of training on employees or agency performance. We have previously  reported that it is increasingly important for agencies to be able to  evaluate their training and development programs and demonstrate how  these efforts help develop employees and improve the agencies’  performance because it can aid decision makers in managing scarce  resources, and provide credible information on how training and  development programs have affected organizational performance. To  do so agencies need to develop evaluation processes that systematically  track the cost and delivery of training and development efforts and assess  the benefits of these efforts. Training and development programs can be  assessed by measuring (1) participant reaction to the training program,  (2) changes in employee behavior or performance; and (3) the impact of  the training on program or organizational results, which may include a  return on investment assessment that compares training costs to derived  benefits. Some of these methods can help provide better value through  identifying areas for continuous improvement in training programs. We  consider the processes for conducting these evaluations to be formal  when they are systematically conducted throughout the agency, have  established guidelines and criteria that govern how they are implemented  and are documented. However, CHCOs may also have other criteria for  determining what is considered a formal process, based on their  agencies’ environment. We asked CHCOs about their formal processes  for conducting the three levels of evaluation listed earlier, which are the  common types of evaluations. Many CHCOs reported routinely  implementing the first two, but not the third (which we discuss later in this  report). Twenty-five of the 27 CHCOs included in our questionnaire  reported that they measure employee satisfaction; and a little more than  half reported that they measure improvement in employee performance.

Officials from the four agencies that we interviewed reported that they all  assess employee’s reaction to training and sometimes assess changes in  employee performance. For example, officials from DOE reported that  they evaluate all the training that they offer by surveying participants’  reactions to the training—which can include their feedback on the  effectiveness of the instructor, the topics, the presentation style, the  schedule, audiovisuals, and other subjects—and use this information to  make revisions to the program courses. Documents that we reviewed on  training evaluations identified updates or revisions made to course  materials and tests to improve their effectiveness, based on training  feedback and policy updates. As an example of evaluating the impact of  training on employee performance, DOI officials stated that, while they do  not have an agency-wide process, some of their organizations—such as  those within the Bureau of Land Management and National Park Service  use an online evaluation tool to assess the impact of training courses on  employees’ abilities to perform tasks, about 6 weeks after a course has  been completed. According to the official, at this time, the process is not  used department-wide, but the agency is looking into how it may be able  to do so starting in fiscal year 2013. According to DOI’s CLO, establishing  this link between training, employee competencies and mission critical  occupation work is one that DOI is targeting for improvement.

Nearly all CHCOs reported that they implement this practice. We have  previously reported that there are many ways to help improve  performance, so it is important for agencies to continually look to others to  identify innovative approaches that may relate to their training and  development efforts. Within the context of that agency’s unique  environment and situation, an agency can compare its investments,  approaches, and outcomes with those of public and private organizations  that are undertaking notably innovative and effective training and  development efforts. Agencies can uncover weaknesses in their training  and development strategies that need improvement and identify new  ideas, mechanisms, and metrics that they could employ. Twenty-four of  the 27 CHCOs included in our questionnaire reported that they compare  training investments, methods, or outcomes with those of other  organizations to identify innovative approaches or lessons learned.

Officials from the two agencies we asked to provide examples of this  practice described this process as occurring informally through  interactions with other CLO Council members. For example, DHS officials  that we met with reported that they meet with other agencies to share  best practices and recommend vendors during breaks or after the CLO  meetings. The officials said that examples of sharing ideas on new  training programs included recent discussions by OPM and agencies on  the GEAR pilot program lessons learned and new courses for developing  supervisors. While two agencies reported having informal interactions  with other agencies to share and compare training information, none of  the agencies that we met with described efforts to benchmark their  practices with agencies or other relevant entities. We have previously  reported that benchmarking can help agencies determine who is the very  best, who sets the standard, and what that standard is.

section_title: Many CHCOs are Not Implementing the Leading Practices that Support More Cost-Effective Training Investment Decisions
paragraphs: Many CHCOs reported that they are not implementing the leading  practices that would allow them to make more cost-effective training  decisions, as shown in Table 4.

Many CHCOs included in our review reported that they have not  implemented this practice. We have previously stated that, to determine  the best ways to leverage investments and establish priorities, agencies  can develop an annual training plan that targets developmental areas of  greatest need and that outlines the most cost-effective training  approaches to address those needs. When assessing investment  opportunities for its training plan, the agency ought to consider the  competing demands confronting the agency, the limited resources  available, and how those demands can best be met with available  resources. If training is identified as a solution to improve agency  performance, agencies can prioritize training using criteria, such as  expected demand for the investment from internal sources, availability of  resources to support the effort, potential for increased revenue, and risk  of unfavorable consequences if investments are not made. Given current  budget constraints, agencies may also want to prioritize training that has  the potential to improve their efficiency. Developing a business case for  training and development that includes this information sets forth the  expected costs and benefits of the investments and provides decision  makers with essential information they need to allocate necessary  resources. Furthermore, under Executive Order No.11348 and OPM  regulations, agencies are to establish training priorities, although  agencies are not specifically instructed to establish an agency-wide  process to do so. Of the 27 CHCOs questionnaire responses, 16  CHCOs reported that they do not set a level of investment agency-wide  and 15 CHCOs reported that they do not prioritize training agency-wide.

In our meetings with officials from the DOE, DOI, DHS, and VA as well as  the CLO council, agency officials cited several reasons for why they do  not establish a level of training investment agency-wide or prioritize  training agency-wide. Some of the reasons were described as purposeful  decisions not to do so and other reasons were described as limitations in  their ability to do so. First, CHCOs elect to establish and prioritize training  investments for centralized training and are often not involved in the  investment decisions made for specific training within the components or  offices, as we previously described. In addition, large components or sub- agencies often have autonomy over their training budgets because the  budgets are appropriated directly to them from Congress. As a result,  CHCOs and their staff are often unaware of how much these components  spend for training and do not have input into these decisions. Component  and sub-agency heads often act autonomously and are not required to  communicate with the CHCO about these decisions. Further, because of  limitations in internal tracking systems for training (which we discussed  earlier in this report), CHCOs do not have information on all of the training  that is completed in their agency and the related costs.

Officials from various agencies involved in the CLO Council and three of  the four agencies that we individually met with reported having a lack of  visibility into the prioritization and level of training investments throughout  their agencies, which they reported limits their ability to better leverage  and reduce duplication in training investments their agencies. Officials in  the agencies that we met with reported that, although they believe that  their components or organizational elements are more capable of making  training decisions related to their specific missions, the lack of  coordination and communication on training investments and priorities  has led to some duplicative and ineffective training investments in their  departments. For example, senior human capital officials in DOI reported  that the department’s leadership, including the CHCO are not aware of  the department’s overall training investments agency-wide and have no  formalized mechanism for ensuring accountability for how the funds are  used. They are aware that bureaus are buying duplicative training or  offering similar training classes that are of varying effectiveness—which is  resulting in inefficient training investments. For example, one bureau  recently independently contracted with an external provider for mid-level  manager leadership training that was already offered at DOI’s university  and paid $50,000 more than DOI University charges. According to  officials, this is a common problem. In addition to duplicative training  courses, in some cases, bureaus are duplicating the creation of new  training facilities. For example, a regional director of an DOI bureau built a  training classroom with a computer lab, despite having access to existing  computer labs within the complex where he worked and also at DOI  facilities a few miles way. Further, according to the officials, because it is  common practice for each bureau to independently secure training, there  is no consistency, little quality control, and no maximization of  procurement tools (such as blanket purchase agreements) across DOI.  In order to address these challenges, DOI has formed a one-time  departmentwide task force known as the Department Innovation and  Efficiency Team for Training. This task force was expected to identify:  potential duplication in training, funds expended in training delivery, and  the cost of travel and facilities, among other things. In July 2012, the  committee made recommendations to the CLO on opportunities to  generate efficiencies and savings in training operations. DOI’s Office of  Strategic Employee and Organization Development is developing action  plans to address the committee’s recommendations.

Officials from DHS also reported experiencing similar challenges with  duplicative or ineffective training investments in their agencies. Some of  these challenges are long standing. For example, seven years ago we  reported that DHS’s two-tier training process (component and  departmentwide) and lack of communication throughout the department  on the availability of some training programs and resources were  challenges that could impede its ability to achieve departmental training  goals and efficiencies. DHS is still taking steps to address this on-going  challenge. In June 2005, DHS formally chartered a Training Leaders  Council (TLC) and recently revised its charter in June 2011. The TLC is  made up of senior training leaders from each component, and  representatives from headquarters to serves as an advisory and  collaborative community of practice to promote effective and efficient  training, education, and professional development opportunities to DHS  employees. According to DHS’ Human Capital leaders, while this group  does not set or prioritize training investments, it provides a forum for  exchanging useful information about common challenges and training  practices, which helps in making more efficient use of existing agency  resources. DHS also established the Human Resource Information  Technology Executive Steering Committee, made up of management  chiefs and HR and information technology leadership across DHS in  2010, and included TLC leadership as members in July 2011. This group  makes some funding decisions related to some training investments, such  as their recent decision to fund the purchase of a single learning  management system for the entire department. However, according to  DHS officials, because DHS has multiple congressional committees and  subcommittees from which the components receive funding and training  direction, coordinating training investments remains challenging.

Many CHCOs that responded to our questionnaire reported that they do  not compare the merits of different training delivery mechanisms. Our  past research and that of others has shown that agencies should  deliberatively consider the options for delivering training and consider  essential issues, such as the goals and objectives for the training, the  type of audience intended for the training, the nature of the training  content, the availability of technology and resources, and the timing for  delivering the training. Agencies can use a variety of instructional  approaches to achieve learning—in the classroom, through distance  learning, or in the workplace. When warranted, agencies should also  consider blended learning that combines different teaching methods (e.g.  Web-based and instructor-led) within the same training effort and provide  trainees with the flexibilities to choose among different training delivery  methods while leveraging resources in the most efficient way possible.  When assessing delivery options, agencies can try to achieve economies  of scale and avoid duplication of effort by taking advantage of existing  course content or training, such as sharable on-line courses or  multiagency training programs. However, In the responses to our  questionnaire, 16 of the 27 CHCOs reported that they do not compare the  merits of the different training delivery mechanisms in their agency.

In our meetings with DHS and VA to obtain illustrative examples, DHS  officials reported that their current learning management systems do not  allow them to mine information on the different delivery mechanisms used  throughout the department and to assess and compare their  effectiveness. According to the officials, they could obtain this information  manually, but it would be a very labor intensive process. Therefore, it is  not done. In contrast, VA officials informed us that they are assessing  different delivery mechanisms for training and conferences offered by  VALU because they recognize that opportunities exist to offer more  efficient mechanisms (such as e-learning). Moreover, VHA, which has the  largest workforce in the department, builds into its initial investment  decision-making process considerations of which delivery methods will be  most effective and efficient, and subsequently evaluates employee  satisfaction with the various delivery methods to inform future investment  decisions. Without processes such as these, agencies that do not  compare the merits of different training delivery mechanisms have limited  information for determining what mix of methods provides the most  efficient and effective delivery of federal training.

Most CHCOs reported that their agencies do not have a routine formal  process to implement this practice. As we previously mentioned, it is  increasingly important for agencies to be able to evaluate their training  and development programs and demonstrate how these efforts help to  improve the agencies’ performance, and to assist them in making more  effective decisions about how to allocate scarce resources. Agencies are  required by statute and OPM implementing regulations to evaluate how  well training programs contribute to mission accomplishment and meet  organizational performance goals. We have identified having a formal  process for this evaluation as a leading practice. However, there are  some understandable limitations to regularly and formally implementing  this practice. For example, some agency officials that we met with  reported that the cost and time required can be significant for obtaining  results of evaluations of training that measure the impact on agency  performance goals. As a result, they can only conduct this level of review  for training that they identify as highly important to key areas of their  mission. We have previously reported that not all training and  development programs require, or are suitable for, higher levels of  evaluation.For example, it may be ineffective to try to measure the  impact of training in an area that is still undergoing other significant  changes that could affect relevant performance goals, such as changes in  related policy and management structure. We recognize that higher levels  of evaluation (such as evaluating the impact on organizational  performance or return on investment) can be challenging to conduct  because of the difficulty and costs associated with data collection and the  complexity in directly linking training and development programs to  improved individual and organizational performance. Factors to consider  when deciding the appropriate level of evaluation include estimated costs  of the training effort, size of the training audience, management interest,  program visibility, and the anticipated “life span” of the effort. Each  agency will need to consider the feasibility and cost-effectiveness of  conducting these in-depth evaluations, along with budgetary and staffing  circumstances that may limit the agency’s ability to complete such  evaluations. Given the current budget constraints that agencies face,  making thoughtful tradeoffs regarding how to target costly evaluation  reviews is a sensible approach. While it is important to prioritize reviews  of training, 8 of the 27 CHCOs that responded to our questionnaire  reported that they do not have a formal process for evaluating the impact  of their training on their agency’s performance.

For example, the CHCO at DOE reported in our questionnaire that DOE  does not implement this practice. We met with the CLO from DOE who  informed us that DOE does not have a formal process for implementing  this practice because the agency does not have a systematic  documented approach for conducting this level of review. Moreover,  evaluation data are not collected in a way that allows it to be aggregated  into a comprehensive assessment of its impact on the agency’s overall  mission. For example, different organizations within DOE conduct reviews  to assess the impact of training on their goals, but they are not captured  in an automated system and the methodologies that DOE organizations  use to conduct these reviews vary. As an illustration, DOE organizations  that work with nuclear material evaluate the technical training that they  provide to their employees against required certification and mission  goals. However, the organizations conduct these evaluations differently,  and because of these varied methodologies and lack of automated results  data, it is difficult to aggregate the reviews into an assessment of how  training has affected DOE’s overall training and mission goals. Similarly,  the CLO’s office evaluates cross-cutting training for employee satisfaction  and employee performance at DOE, but does not effectively or  consistently evaluate its impact on agency goals. According to the CLO,  to assist them in developing a more systematic formal process, they are  participating in OPM training on developing training evaluations and in the  GEAR pilot program—which is intended to better link employee  performance to organizational goals.

In contrast, VA’s training review processes illustrate that agencies that  have a formal process for assessing the impact of training on their  performance mission and goals can use it to make better training  investment decisions. VA recently assessed the return on investment of  its corporate training and, the department’s Administrations recently  evaluated the impact of mission-specific training on their performance  goals. In January 2012, VA evaluated the monetary and mission-related  benefits of training that was implemented under its Human Capital  Investment Plan. According to the return on investment assessment and  report developed by VALU and VHA’s National Center for Organization  Development, VA’ s two-year $577 million investment in training and  development under VA’s Human Capital Investment Plan has resulted in  $604 million dollars in savings that are tied to reductions in costly VA  turnovers, fewer overdue accounts renewable, and fewer equal  employment opportunity complaints. The report also states that VA has  gained non-financial returns, such as faster benefits processing,  increased veteran hiring programs, and improved patient satisfaction.  According to VALU officials, they have used details in this report along  with other factors to make decisions about future training and  development investments. Similarly, for its mission-specific training, VHA  recently conducted an in-depth review of training provided to Patient  Aligned Care Teams to improve their collaborative delivery of care to  patients. The evaluation assessed the training participants’ satisfaction,  skill acquisition, application on the job, and impact on VHA’s business.  The assessment ultimately determined that the training was successful in  addressing the desired behavior changes in the work place and that key  organizational results were influenced by the training, but it also identified  some improvements that VHA could make.

section_title: OPM Guidance and Assistance to Agencies on Federal Training Investments Addresses Many Leading Practices that We Identified, But Could be Improved in Areas That Are Challenges to Agencies
paragraphs: OPM guidance and assistance to agencies on federal training  investments are in line with five of the eight leading practices, but OPM  lacks guidance and assistance in some areas that are challenges to  agencies, as shown in table 5.

OPM guidance or assistance to agencies on federal training investments  addresses five of the eight leading practices for federal agency training  investment decision-making processes. OPM’s five primary guidance  documents that relate to making training investment decisions include the  Guide to Human Resources Reporting, Training Evaluation Field Guide,  Draft Training Policy Handbook, Guide for Collection and Management of  Training Information, and Guide to Strategically Planning Training and  Measuring Results. (See table 2 for a brief description of these guides). In  addition, OPM provides technical assistance to agencies via facilitated  forums, discussion, and training.

Our review of OPM’s guidance documents and assistance shows that  OPM has provided some technical assistance to agencies on this  practice, although OPM does not have guidance documents that provide  specific advice on this topic. For example, because of requirements in  GPRAMA, OPM is providing assistance to agencies in considering this  government-wide reform when planning their training and development  programs. GPRAMA required OPM to identify the competencies needed  to perform the following three functions: developing goals, evaluating  programs, and analyzing and using performance information for the  purpose of improving government efficiency and effectiveness. OPM,  working with subject matter experts developed a competency model for  the three new roles required by GPRAMA—performance improvement  officer, performance improvement staff, and goal leader. Earlier this year  OPM advised agencies that it would provide guidance on how to  incorporate the skills and competencies into these position descriptions,  as specified in the GPRAMA. The Director of OPM stated that the agency  would work with the CLOs to incorporate the key skills and competencies  into agency training programs.

OPM has begun providing this assistance to agencies by facilitating  sessions for agencies to develop training requirements for implementing  the new positions and roles required by GPRAMA. For example, OPM  worked with OMB to gather information on existing training, provide  learning opportunities, and consolidate new and existing training courses  and materials to support this effort. Using this information, OPM and OMB  led two working group meetings with agencies to discuss GPRAMA  training needs and next steps. In a working group meeting in February  2012, OPM and agencies discussed which competencies identified in  GPRAMA could be improved readily through training. OPM provided  participants with a chart developed from an OPM and the Merit System  Protection Board 2011 Trainability Study on which competencies for the  three new roles required by GPRAMA were highly trainable versus those  that were less trainable. After the discussion, OPM and participants  identified the most critical and manageable next steps, including:  create a common competency assessment tool to assess competency  gaps within agencies;  create a course on writing results-oriented goals and standards— while also gathering existing training;  create a working group to assess the needs and create a solution to  satisfy the training requirement for the Organizational Performance  Analysis, Planning and Evaluating, and Performance Measurement  competencies and to collect relevant case studies, as well as to  identify opportunities to leverage agency resources;  identify existing subject matter experts in the agencies and create  forums, workshops, training sessions, etc. where they can share their  expertise and possibly engage in peer-to peer coaching;  create working groups where necessary; and  consider the development of a career path after OPM’s classification  study.

In addition to this assistance, although not specific to government-wide  reforms, OPM’s Training Policy Handbook advises agencies to conduct a  training needs assessment that includes an evaluation of organization  needs, which should take into consideration changing demographics,  political trends, technology, and the economy.

OPM’s TED group advises agencies in its Training Policy Handbook and  2000 Guide to Strategically Planning Training and Measuring Results to  use multiple delivery methods, or combine them, when providing training  to employees. For example, the Training Policy Handbook maintains that  agencies should decide which delivery option is best to achieve the  instructional goals of the training, highlighting that some methods are  more effective for certain courses. It states that a performance  management course may include role play scenarios which may not be  suited for an e-learning course. Further, the guide states that agencies  need to develop training delivery mechanisms that effectively limit  unnecessary overlap and duplication of efforts. Similarly, we have  previously reported that agencies need to consider essential issues such  as the goals and objectives for the training, the type of audience for which  the training is intended, the nature of the training content, the availability  of technology and resources, and the timing for delivering the training  when identifying the most effective and efficient delivery mechanism.  Agency officials who have implemented this practice reported seeing  positive results. For example, VHA officials that we met with and agencies  that have publically discussed their efforts to assess the different delivery  mechanisms at a March 2012 Partnership for Public Service Forum on:  Going Virtual- Maximizing the Return On Investment of Online Training  reported significant savings and increases in the effectiveness of their  training by assessing and changing their training delivery mechanisms.  Specifically, VHA officials reported achieving several non-financial and  financial benefits as a result of moving from in-person meetings and audio  and video conferencing to providing training on-line for one of its  leadership training programs. According to a VHA assessment report, the  benefits included: consistent curriculum across eight medical centers in  three states; easier accessibility to course materials and job aids;  immediate access to feedback on courses from learners; easier  reproduction of courses for instructors; a return on investment of 140  percent since implementation; and $116,000 saved in travel costs,  facilitation, and facilities; among other things.

OPM guidance informs agencies that they should implement this practice.  However, the guidance does not include methodologies for how to  implement it. Officials from DHS—an agency that reported that it does not  implement this practice—stated that tools provided by OPM could be  strengthened to assist them in comparing training delivery mechanisms.  For example, DHS officials reported that they have difficulty implementing  this practice partly because their components do not track comparative  data on the different delivery mechanisms. According to the DHS officials,  the standard government form for tracking training data (Standard Form- 182) does have a category for tracking training delivery type, but filling out  this block is not mandatory and is often not used. The DHS officials  reported that an OPM requirement to capture these data would improve  their ability to gather the information needed from DHS components to  effectively implement this practice. As noted earlier, 15 of the 27 CHCOs  included in our review, reported that they do not implement this practice,  which indicates that they may also benefit from additional guidance and  tools on ways to do so.

OPM’s TED group provides guidance and assistance to agencies on  tracking and reporting the cost and delivery of training and development  programs in four of its five guides. For example, OPM’s 2000 Guide to  Strategically Planning Training and Measuring Results advises agencies  to calculate the cost of the expenses associated with designing,  developing, implementing, and evaluating their training programs and  provides a list of the most common types of training costs. OPM’s Guide  for Collection and Management of Training Information also outlines  agency requirements to track various types of data training and provides  a list of several data sources (e.g. Standard Form-182, agency personnel  records, procurement documents, financial and performance records,  training evaluation forms, etc.) that agencies could use to collect this  information. Similarly, the Training Policy Handbook also incorporates  guidance on tracking the cost and delivery of agencies’ training and  development. In the more recent 2012 Guide to Human Resources  Reporting, OPM outlines requirements for agencies to track training data  and describes the requirement to use certain standard tracking forms,  such as the standard “Authorization, Agreement, and Certification of  Training” (Standard Form-182) to track data. The guide also instructs  agencies to provide all training information included in this form for  submission to OPM’s Enterprise Human Resources Integration (EHRI)  database systems.

Although OPM provides several guidance documents and assistance on  tracking the cost and delivery of training, we found that this practice  continues to be a challenge for many agencies to implement. Agency  officials that we met with reported that they could benefit from additional  assistance from OPM in developing a common definition of what should  be tracked as training, developing policies to strengthen the utilization of  Standard Form -182 to document and report all training costs, and  encouraging agencies through guidance and technical assistance, to  routinely report training cost data to agency learning management  systems.

In addition, to providing guidance on tracking data, OPM facilitates the  collection of federal training data government-wide. Executive Order No.  11348 requires OPM to develop, install, and maintain a system to provide  the training data needed to carry out its own functions and to provide staff  assistance to the President. OPM’s EHRI is the government-wide  repository for these training data. As noted above, agencies have been  required since 2006 to report training data to OPM monthly via this  system. However, according to OPM officials, they consider the data to  be unreliable because they are incomplete. Therefore, OPM officials have  not used it to inform their training guidance and assistance to agencies, to  counsel heads of agencies and other agency officials regarding federal  training needs or investments or to assist agencies in developing sound  programs and financial plans for training programs.

According to OPM officials and documents, OPM should assess EHRI  training data for technical compliance and data quality validation.  Technical compliance is the testing and approving of agency systems for  data quality (i.e. correct formatting, adherence to edit rules). Once  systems are technically compliant, agencies are required to send monthly  data feeds of completed training events to OPM. Once agencies are  reporting these data for all major components, all employees, all types of  training (e.g. conferences, on-line, classroom), and training cost data,  OPM are to evaluate the data quality to determine if it presents an  accurate picture of all training in the agency. However, OPM officials told  us that they have not assessed the quality of data or developed a report  on its reliability because no agency is sending information on all training  events. According to OPM officials, when agencies request assistance or  when OPM finds that an agency has been grossly delinquent in providing  data, OPM officials will inquire further and offer assistance to the  agencies. However, they typically do not document reliability issues or the  agreed upon action plans to address the problems. The officials agreed  that this is a problem, but stated that they would need more staff  resources to provide this level of assistance and oversight.

We believe that the current reliability of agency training investment data is  unknown because OPM officials have not internally assessed  improvements in the completeness of the data over the last 3 years or the  quality of the data in the six years that agencies have been required to  submit it. The two internal reviews that OPM conducted of training data  were in 2008 and September 2009. In the 2009 review, OPM reported  that there was an increase from fiscal year 2008 in the amount of training  data being reported by agencies, but that the quality of the data was still  less than what was necessary to provide an accurate picture of federal  training investments. According to the 2009 report, over half of all  agencies were reporting data for the entire agency, 86 percent were  reporting on a regular basis, but only 7 percent were reporting cost data.  The report identified several of the same reasons that we previously  described as limitations to agencies reporting training investment data.  Although the report stated that OPM would continue to work with  agencies to assess the quality/validity of training investment data and  determine whether agencies are reporting all training events, as noted  above, OPM officials informed us that they have not assessed the quality  of the data because the data are not 100 percent complete.

While it is important to have complete data, we do not believe that having  incomplete data necessarily prevents OPM from assessing the overall  reliability of the data, if it meets standards for sufficiency. In our guidance  on assessing the reliability of computer based data, we have stated that  agencies can assess data if it is sufficiently complete. Data are  sufficiently reliable when testing and reviews of the existing information  provide assurance that (1) the likelihood of significant errors or  incompleteness is minimal and (2) the use of the data would not lead to  an incorrect or unintentional message. Further, we consider the data not  to be sufficiently reliable when there are significant errors or  incompleteness in some of or all the key data elements and if using the  data would probably lead to an incorrect or unintentional message.  Because OPM has not conducted an assessment of improvements in  agency training data in three years, it is unknown whether it is currently  complete enough to test other aspects of its quality and reliability.  According to the officials, although they have not conducted a formal  review of the data, they are able to visually look at the EHRI data base  and tell that the data are significantly more complete than in past years.

OPM also previously identified several steps that its officials would take to  assist agencies in improving their data, but have not yet implemented all  of them. According to OPM’s 2009 report assessing EHRI data, OPM  planned to assist agencies in improving training investment data, by: (1)  working with agencies to fully report all training investment data— including costs; (2) working with agencies to decrease errors in reporting;  and (3) providing individual agencies with summary reports of the data  that they submitted to OPM for their review and verification. We found that OPM has initiated some related efforts, but has not fully addressed  two of these issues. In order to decrease errors in reporting, OPM officials  and EHRI reports show that OPM has worked with agencies to identify  technical errors in their training data submission. However, to improve  reporting on cost data—which is currently a challenge for agencies— OPM held one focus group with agencies in 2007, which it used to  updated it’s guidance on tracking training data in 2008. OPM also has  not followed through on plans to annually provide agencies with reports of  their training data for verification and correction. According to OPM, the  purpose of the training data report is to (1) inform agencies of the training  data OPM had received (2) offer them the opportunity to work closely with  OPM in correcting any identified deficiencies and (3) to make note of the  progress they have made in addressing OPM’s training reporting  requirement. OPM officials said that they sent one report to agencies (in  fiscal year 2010) summarizing their training data and requesting  verification and this report was provided in response to expectations that  the data would be posted on the government-wide website Data.gov. In  our review of examples of agency responses, we found that agencies  identified important discrepancies in their data, including significant  underestimates of the costs spent on training, and reported that they  would take steps to address incorrect data. However, OPM officials  informed us that they do not have a process for documenting whether  agencies have taken steps to correct their data.

Although OPM has only provided one summary of EHRI data to agencies,  agency officials that we met with stated that they could benefit from using  this type of data summary to improve their training data. Further, OPM  officials stated that using these summaries to improve EHRI data could  help agencies measure the return of investment on their training and  assist agencies’ stakeholders in making more informed decisions on the  best use of training dollars. During our review, OPM officials reported that  they began developing a report to send to agencies with fiscal year 2011  training data.

The TED group provides guidance and assistance to agencies in  evaluating training programs through three of its guides and in  workshops. OPM’s Training Evaluation Field Guide is the primary guide  through which OPM advises agencies on how to evaluate training. The  guide instructs agencies to define the results they want to achieve from  training and to align training with the agency mission and goals. Further,  the Guide discusses useful models for evaluating training and describes  Return on Expectations as the ultimate indicator to demonstrate the  training value to stakeholders. The guide also provides information on  evaluation requirements outlined in laws and regulations, and provides  practical instruction by identifying common challenges and solutions  related to identifying the most cost-effective methods to train and develop  employees. In addition to this guidance, OPM’s Training Policy Handbook  instructs agencies to evaluate all training to determine whether or not it  provides meaningful contributions to agency results. Similarly, in its Guide  for Collection and Management of Training Information, OPM highlights  the importance of collecting accurate, comprehensive training information  and making it available to decision-makers and others who have a vested  interest in the training activities of the federal government. This guide  discusses the two basic types of performance measures for measuring  training and development program effectiveness: process indicators and  outcome indicators.

In addition to its guides, OPM has made evaluation tools available to  agencies on its website and held workshops on training evaluation in  order help agencies identify and share best practices on evaluating  training. For example, OPM’s website contains a Training Evaluation Tool  that describes the levels of training evaluation and provides agencies with  evaluation questions to be answered in each of the four levels and the  types of information typically collected.

As previously mentioned, some agency officials reported that it is difficult  to conduct these reviews because their cost and time demands can be  significant. As a result, some agencies only conduct them for the most  critical training and others reported that they do not have a formal process  for conducting these reviews at all. While we agree that it is appropriate to  target costly evaluations to the most important training, those who do not  implement this practice at all could benefit from using OPM’s  comprehensive guidance and assistance on training evaluations.

OPM does not have a guidance document that advises agencies on how  to compare training investment methods and outcomes with other  agencies, but provides some support to agencies in this area through  technical assistance. For example, the TED group uses various web- based mechanisms such as OPM’s website, OPM LISTSERV, OPM  Federal Training and Development web site and OPM Federal Training  and Development Wiki to facilitate discussions between agencies on  training investments. We observed the exchange and sharing of  information among agencies through OPM’s LISTSERV, which is used by  950 employees from various federal agencies to share training practices  and advice. At times, agencies requested and shared information with  each other on the most effective or efficient ways to implement specific  training programs or requested models for which to compare their  activities. Similarly, OPM’s wiki page contains examples and models of  training programs for others to use when developing their training  programs. According to OPM officials, the TED group also provides best  practice forums on topics when they believe agencies need additional  assistance. For example, OPM officials reported that they have held  forums with agencies on the Training Evaluation Field Guide to share  best practices and tools among agencies. OPM is working on a similar  forum for developing supervisory training.
subsections: section_title: OPM Does Not Provide Guidance and Assistance on Three Leading Training Investment Practices
paragraphs: OPM does not have guidance and assistance for three leading training  investment practices, two of which are areas in which agencies reported  experiencing challenges. We examined the five guidance documents that  OPM provides related to making training investment decisions and  documentation on OPM’s technical assistance and did not find support for  the following three practices. OPM officials confirmed that that the agency  does not provide direct guidance or assistance in some of these areas.

In our review of OPM guidance and documentation on its technical  assistance, we found that OPM provides some guidance to agencies  regarding steps to identify training and development investment needs  and related training strategies, but we did not find guidance on prioritizing  these investments so that the most important training needs are  addressed first. According to TED officials, each agency must assess its  own needs as the primary driver for investment determinations. To that  end, OPM officials provide guidance and tools for conducting training  needs assessments in OPM’s training policy guide and on OPM’s website  and also direct agencies to review benchmarks in American Society for  Training & Development’s State of the Industry reports. OPM officials also  reported that they use the Human Capital Assessment Accountability  Framework and related efforts to emphasize the importance of  considering training as a solution to addressing mission critical  competencies and skill gaps, but acknowledged that OPM does not  provide specific guidance on prioritizing training investments through  these processes.

OPM officials identified guidance that they believe addresses the leading  practice of prioritizing training investments; however, we found that the  guidance does not address prioritization. As we previously mentioned, it  is a leading practice for agencies to prioritize their training investments  using criteria, such as expected demand for the training from internal  sources, availability of resources to support the effort, potential for  increased revenue, and risk of unfavorable consequences if investments  are not made, or their potential to improve their efficiency. TED officials  identified OPM’s 2000 Guide to Strategically Planning Training and  Measuring Results as the source of guidance to agencies on this  practice.guidance to agencies on prioritizing training investments. Instead, the  guide advises agencies to build a business case for their training  strategies. OPM defines a business case as a method for projecting and  documenting the benefits to be gained as a result of investing resources  in a training intervention. The guide encourages agencies to consider  questions that are important to building a business case and provides an  example of how to build a business case using this information (see  Appendix III for diagrams from the guide on building a business case for  training). These steps are consistent with our identified leading practice.  However, the guide does not take the additional step of advising agencies  on how to prioritize training investments selected from their business  case(s) relative to each other. We have previously reported that, when  budgets are constrained, training is often one of first investments that  agencies reduce. Therefore, it is increasingly important for agencies to  prioritize their selected training activities, so that the most important  training is identified. Moreover, they need to communicate those priorities  agency-wide, in order to identify common needs and potential areas for  consolidated investments. As previously noted in our review, this is a  practice that most CHCOs reported that they do not implement, which, as  illustrated by our case example agencies, has resulted in costly  duplicative and inefficient training investments at some agencies.

Identify the most appropriate  mix of centralized and  decentralized approaches for  its training and development  programs.

In our review of OPM guidance and documentation on its technical  assistance, we did not identify specific guidance or assistance to  agencies on this practice. As we previously noted, while neither approach  fits every situation, agencies need to consciously think about the  advantages and disadvantages of using centralized and decentralized  approaches, particularly for the design of training and development  programs. Although OPM officials confirmed that they do not provide  guidance and assistance to agencies in this area, OPM officials agreed  with this leading practice. We found that most agencies included in our  review reported that they already implement the practice, so additional  guidance may not be necessary.

Have criteria for determining  whether to design training and  development programs in- house or obtain these services  from a contractor or other  external source.

TED officials agreed that they do not provide guidance or assistance to  agencies on this practice. TED officials stated that agencies need to  incorporate this leading practice into their training investment decision- making process. As we previously noted, once an agency has identified  its training and development needs, it should make informed decisions  about whether to design and develop training and development programs  in-house or buy these services from a contractor or other external source.  Factors that they should consider include the capability of in-house staff  to develop and implement the training; the prior experience, capability,  and stability of possible providers in the marketplace; and agency  limitations on cost, time, and resources. As previously mentioned, 12 of  the 27 CHCOs reported that they do not implement this leading practice,  and our discussion with DOI officials, one of the four agencies that we  interviewed for this review, illustrates that even those agencies that  reported implementing this practice may not be doing so for all training in  the agency.

section_title: Agency Officials Identified a Need for OPM to Provide More Assistance in Improving the Effectiveness and Efficiency of Training and Development Investments Government-wide
paragraphs: Agency officials reported they could use more OPM assistance in  leveraging federal training investments across the government. Part of  OPM’s role is to identify functional areas in which new or expanded  interagency training activity is needed and either conduct such training or  arrange for agencies having the substantive competence to do so; as well  as to coordinate interagency training conducted by and for agencies.  Members of the CLO council emphasized that they could benefit from  more OPM assistance in achieving greater interagency collaboration on  training to reduce duplicative training investments. All four agencies that  we interviewed reported concerns similar to the Council’s. For example,  DOI officials noted that OPM’s knowledge and expertise could help  agencies identify one basic approach to competency management (e.g.,  establishing levels of proficiency and competency validation processes)  that can be used across government rather than using multiple  approaches at various agencies. At the present, each agency individually  identifies training needed for these competencies, which results in  duplication and variation in the quality of training provided throughout the  government. DOI officials stated it could be more efficient if agencies  would use a standard set of Knowledge, Skills, and Abilities to hire and  identify training and development investment priorities. The officials also  suggested that OPM’s HR University could be used to provide training for  other mission-critical occupations. Further, VA officials stated that it would  assist agencies if OPM established government-wide courses for  mandatory training and cross-cutting areas. As an example, the officials  stated the federal government has 17 different versions of No Fear Act  training. The officials suggested that OPM could establish one  government wide training for such subjects which would help agencies  save federal time and money. Officials from DHS and DOE expressed  similar views.

In contrast, the shared training efforts that are being implemented by the  Federal Healthcare Training Partnership collaborative, which consists of  14 federal agencies that provide clinical health care or related training to  support their mission, illustrate the potential magnitude of savings that  could be achieved by leveraging training across agencies. The Federal  Healthcare Training Partnership was created by its members to share  training programs and resources across the agencies to speed up the  provision of employee learning and reduce training costs. According to  VA officials—who lead the effort, the agencies formed this group because  they saw the unaddressed need and opportunity to save costs in common  training areas. Documentation provided by VA on the collaborative group  states that in fiscal 2011, Federal Healthcare Training Partnership partner  organizations shared more than 2,300 programs, generating a total cost  avoidance of more than $82 million. They did so by utilizing the partner organizations’ existing learning systems to share training that was  originally developed for a single agency’s internal use and making it  available to all federal learners, as well as by coordinating the joint  development or purchase of training needed by two or more partner  agencies. VA officials stated that, while this has been a valuable effort to  specifically improve healthcare-related training investments for the  agencies involved, all federal agencies would benefit from an expansion  of leveraging training investments across the government.

OPM officials agreed that increased coordination of mandatory and  common training across the government could reduce duplication and  improve the efficiency of federal training investments. The officials  reported that OPM has already engaged in some efforts to partner with or  support CLO and CHCO Council efforts to share specific training across  agencies. For example, the officials worked with the Social Security  Administration to share a “Plain-Language” writing course developed by  the Social Security Administration with other agencies, by placing it on  OPM’s Training and Development Wiki page. In addition, OPM officials  stated that in 2010, the CHCO Council and OPM collaborated to establish  HR University, which is aimed at addressing the competency and skill  gaps within the HR community and achieve savings government-wide by  identifying and sharing the best HR training with all agencies. While the  system was initially designed to provide training to the HR community, it  has also been used to provide some mandatory training and HR training  to other supervisors and managers. For example, officials reported that  they recently added a mandated Uniformed Services Employment and  Reemployment Rights Act training to HR University. The CHCO Council  and OPM have also developed a formula to calculate cost savings  resulting from the shared courses, which agencies can use to track their  savings and return on investment. According to the OPM Executive  Director of the CHCO Council and HR University’s website, in its first  year, HR University has saved the government $14.5 million as a result of  the shared training, and OPM officials expect that it could produce  significantly more savings, when other courses are added. According to  OPM officials, while HR University primarily serves the needs of the HR  community, OPM would support using the HR University model to  centralize training in other occupations or functional areas.

section_title: Conclusions
paragraphs: The federal government’s efforts to build and retain a workforce of skilled  and efficient employees are essential to addressing skill gaps in critical  fields and effectively and efficiently deliver services to the public. Training  and development programs play a vital role in fulfilling these goals.  However, agency leaders need to be as strategic about how they invest  resources in this area as they are in other key areas of agency  operations. Training investment decisions should be based on an  assessment of the appropriate level of training investments and the  prioritization of those investments, as well as an evaluation of the most  cost-effective delivery mechanisms, and the known costs and benefits of  their training investments. CHCOs and OPM each play a vital role in  ensuring that these investment decisions are effectively made.

While CHCOs report that they are implementing leading practices that  support the successful delivery of training, they could do more to ensure  that these investments are more cost effective. Because many CHCOs do  not have the information that they need from component or subagency  leaders regarding the level of training investments and mechanisms for  setting priorities agency-wide, their agencies are duplicating some  internal training investments and missing opportunities to leverage  economies of scale and share the most effective training across their  agencies. Many CHCOs are also limiting their opportunities to make  training more cost effective and accessible because they are not  comparing the merits of different training delivery mechanisms. OPM’s  guidance and assistance in these three areas are minimal or absent and  could be strengthened to assist agencies in implementing these leading  practices. In addition to these limitations, some CHCOs do not have a  formal process to evaluate the impact of training on their mission. While  not all training and development programs require, or are suitable for,  higher levels of evaluation, those who do not implement this practice for  any training are missing information that could help them make more  effective investment decisions and could benefit from using OPM’s  existing guidance and assistance on conducting such evaluations.

Federal agencies and OPM also need reliable information on how much  agencies spend on training and for what purposes, in order to make  effective training investment decisions. However, CHCOs do not  completely and reliably track training costs agency-wide and, therefore,  are unable to provide OPM with the reliable information that it needs to  more effectively guide government-wide training policies. OPM has  responsibility for providing regulations for maintenance of agency training  data, assessing the completeness and quality of those data when  agencies submit it, and using it to target its assistance to agencies. But  OPM does not know the extent of the reliability of federal training  investment data because they have not compared improvements in the  completeness of the data over last 3 years and determined if it meets the  standards of sufficiency for data assessment and have not assessed the  quality of the data in the 6 years that agencies have been required to  submit it.

Given the fiscal challenges facing the nation, the federal government  needs to take advantage of every opportunity to better leverage  resources and investments across agencies. However, at present many  agencies independently purchase or develop training for the same  government-wide mandated courses. OPM has an opportunity to reduce  duplicative and inefficient training investments by leveraging existing  training resources government-wide. Agency leaders and OPM recognize  that this has led to redundant and inefficient federal training investments.  HR University—the one-stop-shop training platform administered by OPM  for many courses mostly related to the HR community—provides a model  that can result in cost savings and help standardize some mandatory  training courses across government.

section_title: Recommendations for Executive Action
paragraphs: To improve federal training investment decision-making processes, the  Director of OPM should take the following five actions:  1.  Include in existing or new OPM guidance or technical assistance  additional information in the following areas:

Steps agencies should take and factors they should consider  when prioritizing federal training investments agency-wide,  including developing a process to rank training using criteria, such  as expected demand for the investment from internal sources,  availability of resources to support the effort, potential for  increased revenue, and risk of unfavorable consequences if  investments are not made.

Steps agencies should take and factors they should consider for  comparing the merits of different delivery mechanisms and  determining the mix of mechanisms to use, in order to ensure  efficient and cost-effective delivery of federal training. Such  guidance could include requesting that agencies consistently  utilize Standard Form-182 to document and report training costs  associated with the different delivery mechanisms employed.  2.  In line with statutory and regulatory provisions on maintenance and  reporting of training information, work with the CHCO Council to  improve the reliability of agency training investment information by: ensuring that agencies are familiar with and follow guidance  outlined in OPM’s Guide for the Collection and Management of  Training Information regarding which training events should be  documented as training and reported to OPM; developing policies to strengthen the utilization of Standard Form- 182 to document and report training costs; encouraging agencies through guidance and technical assistance,  to develop policies that require consistent reporting of training  data to their learning management systems; and encouraging each agency to assess its existing training  information system(s) and identify whether it is providing complete  and reliable data and, if not, to develop approaches to improve the  system(s), in order to do so.  3.  Provide regular report summaries to agencies on EHRI training  investment data and its reliability, in order to improve the transparency  and reliability of federal training investment data.  4.  Once federal training data reliability has been sufficiently improved,  consistent with Executive Order No. 11348, use EHRI data to:  a)  counsel heads of agencies and other agency officials on the  improvement of training, and  b)  assist agencies in developing sound programs and financial plans  for training and provide advice, information, and assistance to  agencies on planning and budgeting training programs.

In collaboration with the CHCO and CLO Councils, identify the best  existing courses that fulfill government-wide training requirements, such  as mandatory Equal Employment Opportunity training, or training in  common federal occupations, such as basic training in financial  management, and offer them to all agencies through HR University or  other appropriate platform to reduce costly and duplicative federal training  investments.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the departments of OPM, DHS, DOE,  DOI, and VA for review and comment. OPM commented on our five  recommendations to their agency, concurring with one recommendation,  partially concurring on three recommendations and not concurring with a  portion of one recommendation. OPM’s official comments are reprinted in  appendix IV. OPM, DOI, and VA provided technical comments, which we  incorporated into our report, as appropriate. DOE and DHS had no  comments.

OPM partially concurred with our first recommendation that it should  provide in existing or new guidance information on prioritizing federal  training investments agency-wide and factors agencies should consider  for comparing the merits of different delivery mechanisms. OPM stated  that its publications mentioned in our report already provide guidance on  necessary steps and specific factors agencies should consider when  prioritizing training investments. However, none of the guides that we  obtained or that OPM provided for our review contain a specific  discussion about ranking training investments based on key factors that  should be considered, such as expected demand for the investment from  internal sources, availability of resources to support the effort, potential  for increased revenue, risk of unfavorable consequences if investments  are not made or the potential to improve efficiency. OPM stated that, as  part of its effort to revise the Human Capital Assessment Framework  resources that it provides to agencies, OPM plans to include tools and  guidance on steps agencies can take to prioritize learning investments as  part of its strategic human capital planning. We did not change our  recommendation, which is based on OPM’s current guidance and  assistance. OPM’s reported future plan to provide more specific guidance  on prioritization has the potential to address our recommendation, when  implemented. OPM also agreed to provide further guidance regarding  what steps agencies should take and what factors they should consider in  comparing the merits of different delivery mechanisms and determining  the mix of mechanisms to use to ensure efficient and cost-effective  delivery of federal training.

OPM did not concur with the portion of our second recommendation  regarding working with the CHCO council to improve the reliability of  agency training investments by developing a common definition of what  should be documented as training. OPM stated that the definition of  training is clearly stated in 5 U.S.C Chapter 41 and OPM’s Draft Training  Policy Handbook and Guide for the Collection and Management of  Training Information outlines which training events should be documented  as training and reported to OPM. Consequently, OPM recommended that  we delete this task for OPM. OPM’s Guide for the Collection and  Management of Training Information states that all courses, workshops  and conferences paid for by the government; all federally mandated  training; and, all agency required training should be reported to OPM’s  EHRI system. It further states that agencies do not have to report training  that occurs spontaneously or casually/incidentally (e.g., reading a book,  having a discussion, web casts, briefings, etc.); training that has no  specified training goals; training where there are no ways to evaluate if  the training improved knowledge, skills, abilities or competencies; and,  training that was not paid for by the government. We agree that this  guidance should assist agencies in knowing which training to track and  report, and therefore have removed this task from the recommendation.  However, given the concerns raised by officials in our case example  agencies regarding inconsistencies in whether conferences and other  trainings are actually tracked, and recent events regarding spending at  such training, we modified our recommendation to suggest that OPM  work closely with CHCOs to ensure that this guidance is followed as it  addresses the other actions we recommend to improve reliable reporting.  OPM concurred with the other actions identified in the recommendation  which included working with the CHCO council to: develop policies to  strengthen the utilization of Standard Form-182 to document and report  training costs; encourage agencies through guidance and technical  assistance, to develop policies that require consistent reporting of training  data to their learning management systems; and encourage each agency  to assess its existing training information system(s) and identify whether it  is providing complete and reliable data and, if not, to develop approaches  to improve the system(s), in order to do so.

OPM partially concurred with our third recommendation that it should  provide regular report summaries to agencies on EHRI training  investment data and its reliability, in order to improve the transparency  and reliability of federal training investment data. OPM stated that it will  provide regular summaries to agencies on the training investment data  submitted to OPM to improve transparency. However, OPM stated that  these summaries will not directly lead to improved reliability of the data  because agencies must take action to improve the data in order to have  an effect on data reliability. OPM also noted that agencies currently have  the option of working with OPM to secure a subscription to Business  Objects—a reporting tool that will allow agencies to run reports of the  data they have provided to OPM and determine whether those data  accurately reflect what is occurring in their agencies. OPM recommended  that we revise our recommendation to read, “Provide regular report  summaries to agencies on EHRI training investment data in order to  improve the transparency of federal training investment data.” We agree  that agencies are ultimately responsible for making changes to their data  to improve its reliability. However, OPM plays an important role in the first  step of that process by reporting the current information that it has, so  that agencies can make corrections. We believe that this  recommendation along with our prior recommendation on steps OPM and  CHCOs can take to improve reliability will contribute to improving the  transparency and reliability of agency training data. Therefore, we did not  make changes to this recommendation.

OPM concurred with our fourth recommendation that it should counsel  heads of agencies and other agency officials on the improvement of  training; assist agencies in developing sound programs and financial  plans for training; and provide advice, information, and assistance to  agencies on planning and budgeting training programs using EHRI data,  once federal training data reliability has been sufficiently improved. OPM  stated that it will consult with agencies on possible improvements and  assistance on planning training programs once federal training data are  reliable.

OPM partially concurred with our fifth recommendation that it should, in  collaboration with the CHCO and CLO Councils, identify the best existing  courses that fulfill government-wide training requirements, such as  mandatory Equal Employment Opportunity training, or training in common  federal occupations, such as basic training in financial management, and  offer them to all agencies through HR University or another appropriate  platform to reduce costly and duplicative federal training investments.  OPM stated that it agrees and is already collaborating with the CHCO and  CLO Councils to identify, collect, and share existing mandatory courses  that fulfill government-wide training requirements (e.g., Plain Writing,  Telework. USERRA, Veterans Employment, Constitution Day) through  HR University or on OPM’s Federal Training and Development Wiki.  Therefore, OPM recommended that we revise the recommendation to  recognize that the expansion of mandatory training by HR University  would be a continuation of efforts they have started.  We have revised the  recommendation to reflect this comment.

We are sending copies of this report to the Director of OPM. In addition,  this report will be available at no charge on the GAO website at  www.gao.gov. If you have any questions about this report, please contact  me at 202-512-2717 or jonesy@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix V.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To better understand how federal training investment decisions are made  and whether improvements are needed, you asked us to review the  methods that agencies are using to establish their training investment  strategies and Office of Personnel Management’s (OPM) training  investment guidance to agencies. Accordingly, this review assesses the  extent to which (1) chief human capital officers (CHCOs) of selected  federal agencies have established processes to set and prioritize training  investments that are aligned with leading practices; and (2) OPM’s  guidance and assistance for developing training investment strategies  align with these leading practices.

For the purposes of this review, we define the key terms “training”,  “development” and “agency-wide” in the following ways:

Training is making available to employees planned and coordinated  educational programs of instruction in professional, technical, or other  fields that are or will be related to the employee’s job responsibilities.  Training can be accomplished through a variety of approaches, such  as classroom training, e-learning, and professional conferences that  are educational or instructional in nature.

Development is generally considered to include training, structured  on-the-job learning experiences, and education. Developmental  programs can include experiences such as coaching, mentoring, or  rotational assignments.

Agency-wide includes all components, sub-agencies or offices within  a cabinet department or independent agency.

For both objectives of the review, we compared OPM and CHCO practices  against eight federal training investment leading practices, which are based  on our prior studies, other expert studies, and statutory, regulatory, and  executive order training requirements. (See table 1 at the beginning of this  report). OPM reviewed these criteria and agreed that they are practices  that agencies should be implementing to support effective training  investment decisions. They also informed us that, while some leading  practices are related to training program requirements contained in  statutory, regulatory, or executive order provisions, responses to our  questions about the leading practices are not an indication of whether  agencies are in compliance with these laws and regulations.

To obtain government-wide information on agency training investment  practices, through a questionnaire on their training investment practices  and processes, we obtained high-level information from members of the  27 agencies represented on the CHCO Council. We provided a standard  set of questions to each CHCO to ensure we consistently captured their  responses to our questions on their training investment practices. We  then analyzed the results of the questionnaire to identify the main themes  and develop summary findings. Two of our analysts conducted this  analysis, placed CHCO responses into categories, and tallied the number  of responses in each category. A third analyst traced the responses back  to the original questionnaire and verified the appropriate categorization of  CHCOs’ responses. To characterize CHCOs views throughout this report,  we defined modifiers (e.g., “many”) to quantify users’ views as follows:  “nearly all” users represents 23 to 27 users,  “most” users represents 18 to 22 users,  “many” users represents 13 to 17 users,  “several” users represents 8 to 12 users,  “some” users represents 3 to 7 users, and  “few” users represents 0 to 3 users.

To obtain additional perspective and insights on the training investment  practices identified in the questionnaire, we discussed the responses with  CHCO and chief learning officers (CLO) councils. In addition, based on  the responses to the questionnaire and workforce size, we selected four  agencies (the Department of Homeland Security, Department of Veterans  Affairs, Department of the Interior, and Department of Energy) from which  to obtain illustrative examples of how they implemented the training  investment practices identified in the questionnaire. (See table 6 for  selection traits). As part of our review of agency practices, we also  obtained information on the steps that agencies are taking to identify and  prioritize investment allocations for training required to implement the  GPRA Modernization Act of 2010 (GPRAMA).

To identify and assess OPM’s oversight and guidance to agencies on  training investment strategies, we reviewed OPM training guidance,  relevant documentation on forums, workshop or other assistance, and  oversight activities. In addition, we interviewed officials from OPM offices  with primary responsibility for providing training policy guidance and  technical assistance to agencies. We compared this information to the  leading practices identified in table 6. We also identified and described  the steps, that OPM has taken to identify the skills and training needed to  implement performance management improvements, such those required  by GPRAMA, as a foundation for future agency training investments.  However, we did not assess the effectiveness of OPM’s efforts to identify  GPRAMA-related skills and actions to develop related training  Based on information obtained from agencies and OPM, we assessed  which leading training investment practices were being implemented by  agencies and addressed by OPM guidance and assistance. We also  identified the challenges or limitations reported by agencies to  implementing the practices, and opportunities for improvement in agency  processes and related OPM guidance.

We conducted this performance audit from December 2011 to September  2012, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Table of Statutes, Regulations, and Executive Orders Related to Leading Practices
paragraphs: Exec. Order No. 11348, section 303(e) requires agency heads to  establish priorities for needed training, and provide for the use of funds  and man-hours in accordance with these priorities.  5 C.F.R. § 410.201(c), in implementing the E.O., requires agency heads  (or designee(s)) to establish priorities for training employees and allocate  resources according to those priorities.

There are no statutory, regulatory, or Executive Order requirements  directly related to this practice.

There are no statutory, regulatory, or Executive Order requirements  directly related to this practice.

There are no statutory, regulatory, or Executive Order requirements  directly related to this practice.

There are no statutory, regulatory, or Executive Order requirements  directly related to this practice.

6.  Agencies should track the cost and delivery of its  5 U.S.C. 4118 authorizes OPM to prescribe regulations—and in doing  so—to specifically provide for  the maintenance of necessary information concerning the general  conduct of the training activities of each agency, and such other  information as is necessary “to enable the President and Congress  to discharge effectively their respective duties for supervision,  control and review of these training programs.”  The submission of reports by agencies on results and effects of  training programs and plans and economies resulting therefrom,  including estimates of costs of training.  5 C.F.R. § 410.601(a) requires agencies to maintain records of  training plans, expenditures, and activities in such form and manner  as necessary to submit to OPM. Subsection (b) provides that  beginning December 31, 2006, agencies are to report training data  at such times and in such form as required for OPM’s government- wide Electronic Data Collection System.

Agencies should evaluate the benefits achieved  through training and development programs, including  improvements in individual and agency performance 5 U.S.C . § 4103(c) requires the head of an agency to evaluate, on a  regular basis, each program or plan established, operated, or maintained  under subsection (a)

There are no statutory, regulatory, or Executive Order requirements directly  related to this practice.

These key training investment practices are part of the framework outlined in the GAO’s guide GAO,  Human Capital: A Guide for Assessing Strategic Training and Development Efforts for the Federal  Government, GAO-04-546G (Washington, D.C.: March 2004). This guide summarizes attributes of  effective training and development programs and it is based on the GAO analysis of prior work, other  related expert studies, and federal training requirements.  5 USC 4103(a) requires agency head to establish, operate, maintain, and evaluate a program or  programs, and a plan or plans there under, for the training of employees.

section_title: Appendix III: Illustrations of Building a Business Case for Training from OPM’s Guide to Strategically Planning Training and Measuring Results

section_title: Appendix IV: Comments from the Office of Personnel Management

section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, William Doherty (Assistant  Director), Latesha Love, Angela Leventis, and Karin Fangman made key  contributions to this report. Also contributing to this report were Benjamin  Crawford, Eric Gorman, and Natalie Maddox.


Question-summary annotations: questions: question: What how have CHCOs dealt training delivery?
answer: Many Chief Human Capital Officers (CHCOs) reported that they are implementing several leading practices important to making strategic decisions about training delivery, such as determining the best mix of decentralized and centralized training and considering government-wide reform when planning training.
child_questions: question: Why are some CHCOs not implementing certain practices?
answer: However, many CHCOs reported they are not implementing some practices that support making more cost-effective training investment decisions, such as prioritizing training so that the most important needs are met first and evaluating the benefits of training.

question: What information is lacking, according to the CHCOs?
answer: In addition, many CHCOs do not have information from component or sub-agency leaders regarding their level of investments and priorities.
child_questions: question: How does this affect federal agencies?
answer: Consequently, some agencies are duplicating internal across their agencies. Federal agencies also need reliable information on how much they spend on training and for what purposes.

question: To what extent do CHCOs track training costs?
answer: However, several CHCOs reported they do not completely and reliably track training costs agency-wide.

questions: question: What is included in the role of the OPM?
answer: The Office of Personnel Management (OPM) provides guidance and assistance to agencies on a number of the leading practices, such as evaluating the benefits of training in three of its guides and in workshops.

question: Why might there be limited guidance to some agencies?
answer: In some practice areas thatare challenges to agencies, such as prioritization of investments and determining whether to design training and development programs in-house or obtain these services from a contractor, guidance is minimal or absent.

question: To what extent do agencies track training investment data?
answer: OPM also requires agencies to submit training investment data and provides guidance on how to do so, but considers this data to be unreliable because it is incomplete.
child_questions: question: How well do OPM officials assess this data?
answer: However, OPM officials have not internally assessed improvements in the completeness of the data over the last 3 years or the quality of the data in the six years that agencies have been required to submit it, and have only provided agencies with one summary of their data for correction.

question: How could OPM help agencies improve their investments?
answer: Agencies and OPM reported there are also opportunities for OPM to help agencies reduce duplicative investments across agencies. For example, currently, agencies independently purchase or develop training for the same mandated or common occupational training.
child_questions: question: What opprotunities have emerged regarding redundancy?
answer: Agency leaders and OPM recognize that this has led to redundant and inefficient federal training investments. Several agencies and OPM officials reported that HR University could be expanded to provide mandatory training and serve as a model for centralizing training in other occupations or functional areas, which could save millions more and help standardize training.

question: How does HR University serve as a model of such savings?
answer: According to OPM officials, HR University—which is a website currently administered by OPM to provide training for the HR community—has already resulted in a cost savings of $14.5 million as a result of sharing the best HR training government-wide. Several agencies and OPM officials reported that HR University could be expanded to provide mandatory mtraining and serve as a model for centralizing training in other occupations or functional areas, which could save millions more and help standardize training.

questions: question: What is the relationship between OPM and agency CHCOs?
answer: OPM and agency CHCOs play an important role in ensuring that federal training dollars are invested effectively.

question: Why did GAO examine the CHCOs of selected federal agencies and the OPM?
answer: GAO was asked to review the extent to which: (1) CHCOs of selected federal agencies have established processes to set and prioritize training investments that are aligned with leading practices; and (2) OPM’s guidance and assistance for developing training investment strategies align with these leading practices.

question: What was included in the GAO's report?
answer: GAO obtained information from 27 CHCOs on their training investment practices through a questionnaire, and selected four agencies—the Departments of Energy (DOE), Homeland Security (DHS), the Interior (DOI) and Veterans Affairs (VA)—to provide illustrative examples.

question: How were the CHCO and OPM practices further examined?
answer: We compared both CHCO and OPM practices to leading practices, identified through past GAO and expert studies.

questions: question: What are all the aspects of GAO's recommendation for the OPM?
answer: GAO recommends, among other things, that OPM improve guidance and assistance to agencies in establishing a process for setting and prioritizing training investments; improve the reliability of agency training investment information; and identify the best existing courses that fulfill governmentwide training requirements, and offer them to all agencies through the HR University or other appropriate platforms.

question: How did the OPM respond to the GAO's recommendation?
answer: OPM fully or partially concurred with four recommendations and did not concur with a portion of another.
child_questions: question: How did other groups respond?
answer: OPM, DOI and VA provided technical comments, which GAO incorporated, as appropriate, into the report. DOE and DHS had no comments.