Human Capital: Agencies Should More Fully Evaluate the Costs and Benefits of Executive Training
Published: 2014-01-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government annually spends millions of dollars on executive training programs administered by external training providers. GAO was asked to review the costs and value of this training. This review (1) describes what is known about how much CHCO Council agencies have spent on executive training offered by external providers and assesses the reliability of available cost data; (2) assesses how CHCO Council agencies evaluate the effectiveness of the training; (3) compares executive training efficiency opportunities identified by agency CHCOs to leading practices; and (4) provides views of career SES and SES candidates from selected agencies on the value of the training they received. GAO obtained information from CHCO Council agencies through a questionnaire, and selected three of 26 CHCO Council agencies--the Departments of Energy, Treasury, and Veterans Affairs--to provide illustrative examples. GAO based its selection, in part, on workforce size, number of career SES, and total training costs.
section_title: What GAO Found
paragraphs: Chief Human Capital Officers (CHCO) from 26 agencies reported that, from fiscal years 2008 through 2012, they spent almost $57 million (in constant 2012 dollars) on executive training provided by external providers. CHCOs reported using the Office of Personnel Management (OPM) Federal Executive Institute and the Harvard Kennedy School of Government most often. Agencies are required to maintain records of training plans, expenditures, activities, and (since 2006) to report training data to OPM. However, half of the CHCOs reported data to GAO that they deemed incomplete, or with limitations. For example, two agencies did not include travel related costs; one did not include costs for course materials; another did not include costs from all components. OPM officials agree that training cost data reported by agencies continues to be unreliable, and is probably lower than actual agency expenditures. OPM officials said they are meeting with agencies to address data deficiencies. However, OPM has not set interim milestones for meeting with agencies or established a timeframe to improve reporting. One leading practice is to establish such interim milestones and timeframes, in order to demonstrate progress towards achieving goals. By not establishing interim milestones and timeframes for improving the reliability of executive training cost data, OPM may be missing an opportunity to better position itself to hold agencies accountable for improving their data.
Most CHCOs reported evaluating participant reaction and changing their training programs based on participant input, but fewer reported assessing training impact on agency mission. Agencies are required by statute and OPM implementing regulations to evaluate how well training programs contribute to mission accomplishment and performance goals. OPM is not sharing lessons learned from agencies that have experience assessing executive training impact on agency mission. However, OPM acknowledged that some agencies struggle with these evaluations; in response, OPM has issued guidance on the subject. CHCOs cited time, costs, and difficulty as reasons for not conducting the required evaluations and reported the need for additional OPM assistance. Eight agencies reported conducting these evaluations and may have lessons learned from which other agencies could benefit. For example, VA has a process for assessing the impact of executive training on its mission that it has used to make better training investment decisions.
CHCOs identified opportunities for agencies and OPM to achieve efficiencies in executive training. CHCOs said agencies could (1) increase interagency cooperation by sharing training facilities and expanding eligibility to Senior Executive Service (SES) candidates from other agencies, and (2) implement or expand computer-based training. CHCOs also said OPM could, among other things, (1) help centralize training offerings by creating a centrally funded SES candidate development program, and (2) assist agencies in identifying programs open to other agencies and departments. By not assessing lessons learned and other efficiencies identified by agencies, OPM may be missing an opportunity to better position agencies to achieve these efficiencies in executive training.
Career SES and SES candidates generally said external executive training is useful and valuable, but suggested that it would be more cost-effective for the government to negotiate prices as a large-scale buyer, versus individual agencies purchasing training.
section_title: What GAO Recommends
paragraphs: GAO recommends that OPM (1) establish interim milestones for meeting with agencies to address training data deficiencies and establish time frames for improving the reliability of agency data, (2) share information and examples of how agencies have evaluated the impact of executive training on agency mission and goals, and (3) assess potential efficiencies identified by agencies for possible government-wide implementation. OPM concurred with the recommendations and has taken useful steps. GAO still believes that more effective activities can be taken.

section_title: Letter
paragraphs: Senior leaders of the federal workforce are responsible for directing a  broad range of essential government activities, such as acquiring weapon  systems to safeguard the country or providing accurate and timely  benefits to the elderly. Effective leadership, especially at senior levels, is  essential to providing accountable, committed, consistent, and sustained  attention to these and other organizational issues facing the government.  The Senior Executive Service (SES), the government’s most senior-level  career leaders, is relatively small—about 7,000 members in 2012—and  represents less than one percent of the over two million federal civilian  employees. Underscoring the importance of high-value executive training,  the Partnership for Public Service recently reported on weaknesses in  talent development for the SES pipeline, and the need to ensure effective  leadership in these positions. The federal government annually spends  millions of dollars on executive training programs administered by training  providers external to federal agencies, in order to provide its senior  leaders with the necessary skills to fulfill their agencies’ missions. These  programs can range widely in both duration and costs, from a four-hour  session at a cost of $200 per federal employee to a four-week program  costing over $20,000 or more per federal employee. The cost of these  training sessions has raised questions about the value they provide to the  federal workforce.
The Office of Personnel Management (OPM) and agency chief human  capital officers (CHCOs) play important roles ensuring that federal  training dollars are invested wisely. OPM provides leadership and  guidance on the establishment and operation of efficient federal training  and development programs government-wide; it also advises the  President on the means for furthering federal training programs. The  Chief Human Capital Officers Act of 2002 (CHCO Act) gave CHCOs the  primary role in setting the workforce development strategies of their  agencies, and in advising and assisting their agency heads (who are  responsible for establishing priorities for needed training and  development). The CHCO Act also established a Chief Human Capital  Officers Council (CHCO Council) to advise and coordinate the activities of  members’ agencies on such matters as the modernization of human  resources systems, improved quality of human resources information,  and legislation affecting human resources operations and organizations.  The decisions that these key players make in determining how to best  use federal training dollars for executive education will ultimately have an  important impact on how well the federal government is equipped to  address its current and emerging performance and management  challenges in an era of constrained resources.
You asked us to review the costs of federal executive training programs  provided by external providers and the value they bring to the federal  workforce. This review (1) describes what is known about how much  CHCO Council agencies have spent on executive training offered by  external providers and assesses the reliability of available cost data; (2)  assesses how CHCO Council agencies evaluate the effectiveness of the  training; (3) compares executive training efficiency opportunities identified  by agency CHCOs to leading training practices; and (4) provides views of  career SES and SES candidates from selected agencies on the value of  the training they received.
To answer the first two questions, we developed and administered a set  of standard questions via a structured questionnaire to the 27 agencies  represented on the CHCO Council to collect government-wide information  on executive training programs, and if available, data on the amount  agencies spent on executive training courses administered by external  providers from fiscal years 2008 to 2012. We obtained agency executive  training cost data from 26 of the 27 CHCO Council agencies and  compared these data to training cost data from OPM’s Enterprise Human  Resources Integration (EHRI) training database. We assessed the  reliability of related cost elements in OPM’s training database based on  reviews of our prior work on federal training investments, testing of  agency data collected by OPM, and interviews conducted with OPM  officials. We also assessed the reliability of executive training participant  and cost data provided by agencies by examining the data for obvious  errors and inconsistencies, and by querying 10 agencies to better  understand the data systems each agency used to collect and report such  data, as well as the quality of data entered into their system(s). We  interviewed five external providers used by CHCO Council agencies to  collect information on executive training offered by these providers. To  answer the second question, we selected three agencies—Department of  Energy (Energy), Department of the Treasury (Treasury), and Department  of Veterans Affairs (VA)—to obtain illustrative examples of how they  approached, implemented and evaluated the effectiveness of executive  training. We selected these agencies based on workforce size, number of  career SES, total training costs, types of evaluations conducted to  measure effectiveness of executive training, and whether agencies  identified potential opportunities for achieving efficiencies in executive  training.
To answer the third question, we analyzed responses to our  questionnaire from agencies, analyzed OPM and agency executive  training policies, procedures, and practices, and interviewed agency  human capital officials to identify potential efficiencies. We examined  how, if at all, OPM, the CHCO Council, and the Chief Learning Officers  (CLO) Council (made up of chief learning officers from the 27 agencies)  share information on potential efficiencies by interviewing OPM officials  and members of the CHCO Council and the CLO Council. To answer the  fourth question, we conducted six focus groups consisting of career SES  and SES candidates from our three selected agencies who had taken  executive training to obtain their views on the usefulness and value of  executive training. We then analyzed and summarized responses from  focus group participants.
We conducted this performance audit from December 2012 to December  2013, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Further details of our  objectives, scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: The federal government has established a policy to develop employees,  including its SES, through training and development programs in order to  improve public service, increase efficiency and economy, and build and  retain a force of skilled and efficient employees, among other things.  Under 5 U.S.C. § 3396, OPM is responsible for establishing training and  development programs for SES and SES candidates, or for requiring  agencies to establish such programs that meet OPM criteria. If agencies  establish these training programs, 5 U.S.C. § 3396(b) states that OPM is  required to assist agencies in the establishment of such programs and  monitor their implementation. In accordance with this, OPM has issued  regulations on training and management development. For example, in  2006 OPM issued a regulation requiring agencies to report training  expenditures to OPM through the government-wide electronic reporting  system. Executive Order No. 11348 (issued in 1967 and later amended  by Executive Order No. 12107 in 1978) further requires OPM to assist  agencies in developing sound programs and financial plans for training  and to provide advice, information, and assistance to agencies on  planning, programming, budgeting, operating, and evaluating training  programs. OPM provides federal agencies with assistance through two  primary mechanisms: guidance documents and technical assistance (see  table 1 below).
Executive training courses are one part of a foundation of executive  education programs that include coaching and rotational assignments to  help executives understand leadership roles and responsibilities in their  respective agencies. OPM reviews and certifies agencies’ SES candidate  development programs (CDPs), which are designed to train potential SES  members. OPM also convenes Qualifications Review Boards (QRBs),  staffed by SES members across government, to provide an independent  peer review of candidates proposed for initial career appointment to the  SES.
section_title: Agencies Lack Complete and Reliable Data on the Cost of External Executive Training and Are Likely Underreporting Costs
subsections: section_title: Agencies Reported Spending Millions of Dollars and Most Used Multiple External Providers from Fiscal Years 2008 through 2012
paragraphs: The 26 CHCO Council agencies responding to our questionnaire reported  that, from fiscal years 2008 through 2012, they spent almost $57 million  (in constant 2012 dollars) for about 30,000 participants to attend  executive training courses offered by external providers. On average, the  agencies collectively reported spending between $9 and $15 million  dollars a year (in constant dollars) on external executive training over the  5-year period.
The 26 CHCO agencies reported using at least 410 external providers  from fiscal years 2008 to 2012 for executive training. Among the external  providers most often used, CHCO agencies reported using OPM’s  Federal Executive Institute (FEI) more than other external providers (see  table 2).
OPM requires agencies to maintain records of training plans,  expenditures, and activities, and to submit the recorded data to OPM.  Since 2006, OPM has required agencies to report training data to OPM  monthly via OPM’s EHRI system. OPM’s Guide to Human Resources  Reporting states that these data should include training materials cost,  per diem cost, travel cost, and tuition and fees. Further, federal financial  accounting standards state that reliable information on the costs of  federal programs or activities is crucial for effective management of  government operations; federal financial accounting standards also state  that Congress needs cost information to evaluate program performance,  to make program authorization decisions, and to compare alternative  courses of action.
External providers employed different approaches for executive training,  which had different cost implications. Based on external providers’ views,  the cost of individual executive education training courses can vary  greatly among external providers due to a variety of factors, including  location of training, length of training, and whether courses are tailored to  meet the mission and goals of a particular agency. For example, VA’s  Leading EDGE (Executives Driving Government Excellence) program  offers half-day executive training workshops to all agencies where  employees can participate either in person or online. The workshops can  serve from about 300 to over 700 participants. According to Leading  EDGE’s executive director, the annual cost for Leading EDGE is about  $850 per participant without coaching and $2,100 per participant with  coaching. Other external providers we contacted host residential training  courses or programs which are off-site from participant’s work locations  and last over a period of several days or weeks, including the following:
Leadership for a Democratic Society: FEI hosts this 22-day course  over 28 days (includes weekends), which addresses (among other  subjects) constitutional principles, personal leadership, organizational  theory, and the policy framework in which governmental leadership  occurs. In FY 2012, attending this course cost about $20,000.
Senior Executive Fellows Program: the Harvard Kennedy School of  Government hosts this 22-day program over 27 days (includes  weekends) and addresses (among other subjects) political  management, effective implementation, and performance  management. In FY 2012, attending this course cost about $18,000.
Leadership at the Peak: The Center for Creative Leadership offers  this 5-day course which focuses on key leadership competencies  such as communication, influence, validating and articulating a vision,  and leading outwardly. The course incorporates a five-month learning  experience including five days of classroom training, and other  activities such as assessments and executive coaching sessions. In  FY 2012, attending this course cost about $9,000.
Executive Leadership for America: The Brookings Institution conducts  this 6-day course which addresses (among other subjects) the  historical framework for constitutional government and how to create  an organizational environment that is responsive to change. In FY  2012, attending this course cost about $6,000.
section_title: Agencies Experience Challenges in Collecting and Reporting Data
paragraphs: Agencies track some—but not all—costs of executive training  administered by external providers. All 26 of the agency CHCOs that  responded to our questionnaire provided data on executive training  administered by external providers; however, half of the agencies  reported that the data they provided were incomplete, and in some cases,  had limitations. In addition, during our review, agency officials from 4 of  the 26 agencies asked us to clarify the cost data we requested, noting  that training cost data are not always readily available within their  agencies. They explained that in order to respond to our data request, in  some cases they would have to manually retrieve paper documents and  tabulate the costs. Moreover, the agencies provided data that included  different cost components. For example, one agency reported that it did  not include costs for course materials. Two other agencies reported that  they did not include costs for travel, lodging, or per diem. Other CHCO  agencies cited additional problems with tracking and summarizing costs  paid to external providers of executive education. For example, one  agency reported experiencing challenges in tracking training costs  agency-wide. According to agency officials, they were unable to  completely (or 100 percent) accurately track training costs across the  department because of the multiple, incompatible learning management  systems that the department uses to track training throughout the  agency. In addition, they stated that the agency’s practices for tracking  training data are inconsistent within the agency because agency  components do not always report training completed with external  providers to the agency’s shared training database system. Another  agency said its agency-wide training data were incomplete and unreliable  because its internal training and financial tracking systems were  incompatible with each other. In light of all these factors, agencies are  likely underreporting the amount they spent on executive training.
In our interviews with Energy, Treasury, and VA, the CHCOs (or their  representatives) reported that their agencies track and report training cost  data to OPM in various formats, some automated and some manual.  However, the three agencies reported facing challenges tracking cost  data. Energy reported that executive training reporting is not consistent  across the agency because many Energy components have direct  funding streams from Congress and schedule their own training  programs; thus, they do not always report training completed with  external providers to Energy’s Office of Human Capital. Moreover,  because Energy’s Office of Human Capital has no direct access to  component organization records, the CHCO cannot ensure that all of  Energy’s funded and administered training programs are tracked and  reported to EHRI and OPM; in addition, Energy’s CHCO cannot identify  whether executive training is reported to OPM by a component  organization in a separate report or with other line item reporting of  completed training. However, the cost and delivery of Energy’s SES  candidate development program and its executive training provided  centrally by Energy’s Office of Human Capital is normally tracked and  reported on an annual basis although Energy has not had any recent  centrally funded executive learning activities. Treasury and VA also  reported difficulties in collecting and tracking reliable executive training  data. Treasury reported that some of its individual bureaus were  challenged by our request to provide executive training cost data because  the costs had to be identified by a manual review of financial transaction  listings. They stated that part of the reason the costs were difficult to pull  together is that the training costs were not reported via OPM’s Standard  Form 182—which is used to collect and document training information for  inclusion in participants’ employee records and the EHRI database.  Treasury officials explained that much of its reported executive training  was done by contract and was not procured by either the individual or the  individual offices that would normally complete the training form. As a  result, Treasury enlisted multiple sources to pull the data together at each  of the agency’s bureaus. Treasury officials reported that the agency is  working internally to determine how to track this information at the  department level. VA reported two issues with its electronic reporting of  data to EHRI: inconsistent recording of overall training cost data in its  training management system, and incompatibility between VA’s learning  management system and EHRI.
We found discrepancies between the data for external executive training  costs that agencies submitted to us and the data from OPM’s EHRI. In  most instances where we identified discrepancies, the cost information  that agencies provided to us was higher than the cost total from OPM’s  EHRI. Table 3 provides illustrative examples of discrepancies in cost  information.
OPM officials agreed that agencies have challenges collecting and  reporting costs for executive training through the EHRI database, and as  a result, executive training cost data in EHRI is probably lower than actual  agency expenditures. OPM officials explained that although agencies  have taken steps recently to improve their reporting of training data, some  agencies continue to struggle with the reporting requirements, in part  because these agencies’ learning management systems are not  compatible with OPM’s EHRI. For these agencies, reports on training  costs require manual uploading because some agencies cannot  sufficiently aggregate training information at the sub-agency level. In  September 2012, we reported on the weaknesses of training cost data  that agencies report to OPM and found that several agencies were unable  to collect reliable training cost data from components, due to multiple  incompatible learning management systems and to inconsistent practices  in tracking training cost data, such as the use of inconsistent coding  schemes for tracking similar training activities. We recommended, among  other things, that OPM (1) work with the CHCO Council to improve the  reliability of agency training investment information, and (2) provide  regular report summaries to agencies on EHRI training investment data  and reliability. OPM agreed to work with agency CHCOs to continue to  strengthen the use of the Standard Form 182 and to increase reliability of  their training data; further, OPM stated that it would provide regular  summaries to agencies on the training investment data submitted to OPM  in order to improve transparency. However, OPM stated that these  summaries will not directly lead to improved reliability of the data and  explained that agencies must take action to improve the data to positively  impact data reliability.
OPM officials told us that they provide assistance and guidance for  federal training efforts but because current training data is unreliable,  OPM officials said they have not used it to inform their training guidance  and assistance to agencies. OPM officials stated that they do not believe  they have a viable way to enforce reporting requirements for training data.  However, OPM has avenues, such as interagency councils, that can help  them assist agencies with fulfilling their reporting obligations. For  example, during 2013, OPM officials reported that they, in collaboration  with the Chief Learning Officers Council, created an interagency group to  develop quality standards and desired reporting data elements to help  agencies collect the right information and to make informed training  decisions, but as of September 2012, the interagency group has not  established any timeframes for completing this effort. OPM reported that  as of October 2013, the working group is moving forward on a project to  develop guidance on prioritizing and ranking federal training investments  that will support a broader effort to develop a single standardized process  for prioritizing training investments. In addition, OPM officials told us they  have been meeting separately with CHCO Council agencies to discuss  each agency’s training data reported to OPM and any deficiencies noted  by OPM. OPM believes this will help rectify situations where agencies are  unable to report training data—among other types of data—accurately,  completely, and in a timely fashion. OPM officials said that they also plan  to meet with smaller agencies for these same purposes.
However, OPM has not established interim milestones for the completion  of these planned meetings with agencies to address training data  deficiencies, nor has OPM set timeframes for agencies to have a plan in  place for improving the completeness and reliability of agency data on  executive training costs. One leading transformation practice is to  establish milestones and timeframes to demonstrate progress towards  achieving goals. Without interim milestones and timeframes for these  efforts, OPM may be missing an opportunity to better position itself to  hold agencies accountable for improving the data.
section_title: Most Agencies Reported Evaluating Executive Training but Few Reported Assessing Its Impact on Agency Missions
paragraphs: Twenty-two of twenty-six agency CHCOs reported to us that they  systematically evaluate the effectiveness of executive training by  assessing participant feedback. Most agencies reported they do this by  using end-of-course evaluations. Seventeen of twenty-six agency CHCOs  reported measuring the impact of training on participant performance.  Five agencies reported using post-course surveys given to supervisors or  participants to measure if what was learned affected the participants’  behavior. For example, one agency requested that managers and  mentors of Senior Executive Service Candidate Development Program  (SESCDP) candidates fill out a questionnaire about the impact and value  of the program for their candidate; this questionnaire included to what  degree the candidate applied lessons learned in the program toward their  developmental assignments.
Twenty of the twenty-six agency CHCOs said they changed their  executive training program as the result of evaluations. Six agencies  reported they revised their program’s content and logistics as a result of  feedback. For example, VA reported that it reorganized its training  schedule to add subjects that its senior executives wanted included.  Other agencies reported they expanded the program based on participant  feedback. For example, one agency expanded the number of leadership  development programs offered, while a second agency increased the  number and types of training opportunities available for executives in  order to increase participation in executive training. A third agency now  offers more learning activities (e.g., virtual training events) that do not  require travel, in order to better accommodate the executives’ schedules,  as well as the current budget constraints within the agency.
A majority of agencies do not have a formal process for evaluating the  impact of training on the agency’s performance goals and mission. Ten of  twenty-six agency CHCOs – or less than half – said they measured the  impact of training on the agency’s mission, while eight of twenty-six  agency CHCOs said they measured the impact on agency performance  goals.
Agencies are required by statute and OPM implementing regulations to  evaluate how well training programs contribute to mission  accomplishment and meet organizational performance goals. OPM’s  Training Evaluation Field Guide provides information on methodologies  for conducting training evaluations with the goal of identifying cost  effective training initiatives that maximize mission accomplishments. The  guide is based on the Kirkpatrick model of evaluation, a popular model of  evaluation that includes measuring (1) participant reaction to the training  program; (2) changes in employees’ skills, knowledge, or abilities; (3)  changes in employees’ behavior or performance; and (4) the impact of  the training on program or organizational results, which may be split into a  fifth level that includes a return on investment assessment that compares  training costs to derived benefits.
Similarly, our guidance on assessing training states that to measure the  real impact of training, agencies need to move beyond data on inputs and  outputs by developing additional indicators that help to determine how  training contributes to the accomplishment of agency goals and  objectives. The traditional approach of collecting and reporting data  related to training and development often involves indicators that can be  readily quantified, such as the number of employees trained, training  hours per employee, and total training hours. While necessary, these  kinds of measures do not fully provide agencies with the kind of  information they need to determine how training efforts contributed to  improved performance, to reduced costs, or to a greater capacity to meet  new and emerging transformation challenges.
Two of the agencies we interviewed, Treasury and VA, were among those  who reported in their questionnaires that they measured the impact of  executive training on the agency’s performance goals and mission. VA  uses various assessment measures to evaluate effectiveness against  VA’s overall mission and the VA Secretary’s strategic objectives for  accomplishing transformation: VA reported that these measures have led  to better training investment decisions. For example, VA includes post- course requirements for its Strategic Leadership Course II – Basic, which  is focused on providing its senior executives with tools to support  strategic decision-making, critical thinking, and leading and driving  change. Approximately 30 days after the training, class participants are  asked to provide their “ah ha” moment from the training, with regard to  their leadership style or techniques; approximately 90 days after the  training, participants are asked how they have been able to apply the  concepts from the executive training to their agency’s organization or  mission. Sometimes specific questions of interest are substituted, such as  recommendations on strategic challenge questions for future classes, or  requests for programs or resources that they need to better accomplish  their mission as strategic leaders. A second requirement may be  assigned at the post 90-day training point. This requirement usually asks  the participants to explain how they have applied lessons taken from the  training to their organization or mission. Additionally, the agency provides  assessment data to a team of program analysts at the VA Learning  University who periodically provide the agency with feedback on the  effectiveness of the program. According to VA, the evaluation tends to  focus on the impact of training on agency mission and strategic goals,  and also on return on investment; the evaluation also includes specific  examples where senior executives applied what they gained at the  training and applied it to their mission in the VA. Treasury officials  explained that while it would be hard to isolate when training by itself  contributes to an individual meeting an agency’s goal, by aligning all of its  programs to the department’s strategic plan and to the executive core  qualifications (ECQ), they are ensuring the training support the agency’s  goals.
CHCO agencies recognized the importance of evaluating the impact of  training on their mission but emphasized the difficulty in conducting these  types of evaluations. For example, two agencies reported needing  additional assistance from OPM for these types of activities, such as  centralized evaluation information on the performance of external  providers. In addition, we previously found that some agencies found it  difficult to conduct these evaluations because their cost and time  demands can be significant. OPM acknowledged that some agencies  struggle with evaluating training programs. OPM reported that it has  developed and published regulations related to training evaluation.   According to OPM, these regulations were developed to enhance the  impact of training on agency mission accomplishment. The regulations  generally relate to training planning, execution, and demonstration of  value. However, OPM is not sharing lessons learned from agencies that  reported having assessed the impact of executive training on agency  missions. Given that less than half of the CHCO agencies reported they  are evaluating the impact of executive training on their missions, sharing  lessons learned in its oversight role could help OPM better position the  other CHCO agencies for conducting such evaluations.
We recognize that higher levels of evaluation (such as evaluating the  impact on organization performance or the return on investment) can be  challenging to conduct because of the difficulty and costs associated with  data collection and the complexity of directly linking training and  development programs to improved individual and organizational  performance. However, eight agencies that responded to our  questionnaire (and that are also members of the CHCO Council) reported  that they have conducted these evaluations and may have lessons  learned from which other agencies could benefit. According to GAO  guidance on evaluating training, factors to consider when deciding the  appropriate level of evaluation include estimated costs of the training  effort, size of the training audience, management interest, program  visibility, and the anticipated life span of the effort. The guidance also  states that each agency consider the feasibility and cost-effectiveness of  conducting these in-depth evaluations, along with budgetary and staffing  circumstances that may limit the agency’s ability to complete such  evaluations. Without conducting higher levels of evaluation, agencies  are missing information that could help them make more effective training  investment decisions; in addition, OPM is missing the opportunity to  assist agencies in fully understanding the benefits of executive training on  the continued development of senior executives and on the agency’s  performance.
section_title: CHCOs Identified Opportunities for Agencies and OPM to Achieve Efficiencies in Executive Training
subsections: section_title: Opportunities Include Using Virtual Training and Centralizing Training
paragraphs: Twenty of twenty-six agency CHCOs responding to our questionnaire  said they identified opportunities for achieving efficiencies in delivering  executive training within their respective agencies. For instance, some  agency CHCOs described steps agencies could take to increase  participation in executive training programs within their respective  organizations. For example, one CHCO said his agency could expand the  base of eligible participants to include employees in other senior level  positions outside the SES. Our guide for assessing strategic training and  development efforts states that agencies should select employees for  developmental programs, in part, based on the usefulness of training  content given employees’ current duties and existing skills and  competencies. Employees in other senior level positions could build  their skills and competencies through participation in executive training; at  the same time, agencies could increase participation in executive training  courses by expanding participation eligibility to employees in other senior  level positions outside the SES, thus reducing unit costs of training  delivery. Another CHCO said his agency is considering using executives  and senior leaders to serve as facilitators for leadership events. As we  previously concluded, efforts to identify cost-effective and robust options  for designing training and development programs may include use of in- house staff, such as senior executives, based on their capability to  implement training.
Four CHCOs said they are considering implementing or expanding  computer-based training, such as use of webinars, online books, virtual  “executive summits” with developmental sessions and strategic meetings  for executives, and other online training programs used for executive  development. One CHCO also said his agency is considering use of  blended learning activities—which incorporate classroom-based and  computer-based training—to maximize impact and cost-effectiveness.  This is consistent with leading practices for training investments.  Specifically, one such practice calls for comparing the merits of different  methods of delivery—such as classroom or computer-based training— and determining what mix of mechanisms to use to ensure efficient and  cost-effective delivery. Another leading practice is for agencies to  consider blended learning that combines different teaching methods (e.g.,  Web-based and instructor-led) within the same training effort and  provides trainees with the flexibilities to choose among different training  delivery methods, while leveraging resources in the most efficient way  possible.
In our interviews with selected agencies, Treasury officials reported they  are working to centralize training for all Treasury executives through the  Treasury Executive Institute (TEI) in an attempt to enhance Treasury’s  return on investment for organizational performance and to provide more  consistency for mission-critical leadership training. Treasury officials said  they could use TEI to leverage training resources throughout the  department, with a goal of providing “just-in-time” training to senior  executives, rather than requiring them to set aside their normal job  responsibilities for a number of months to attend training. Treasury  officials told us the agency is in the beginning stages of obtaining OPM  approval for an agency-wide SESCDP, which would allow Treasury to  train executives from all of its components and would help Treasury fill  vacant SES positions agency-wide. Treasury officials said they are likely  to operate the proposed agency-wide SESCDP alongside the Internal  Revenue Service’s SESCDP—the only SESCDP currently in operation  within Treasury. Treasury’s efforts to centralize SES candidate training is  consistent with a leading practice related to identifying the most  appropriate mix of centralized and decentralized approaches for an  agency’s training and development programs. In particular, centralizing  design can enhance consistency of training content and offer potential  cost savings through standardized record keeping and simplified and  more accurate reporting on courses, certifications, educational  attainment, costs, or standards.
VA officials said they performed after action reviews on and made  changes to their Strategic Leadership Course II – Basic course for career  SES members within VA. Based on the reviews and subsequent changes  to its executive training program, VA officials negotiated a new price  structure from the course vendor to deliver the training at a lower cost in  2013 than if they had used the course’s 2012 price structure. We  previously concluded that agencies can use evaluation feedback to  identify problems and improve training and development programs as  needed. Based on modifications to its executive training programs from  after action reviews, VA was also able to reduce program costs, thus  increasing the efficiency of training delivery.
Further, 20 of 26 agency CHCOs said they found opportunities for  achieving efficiencies government-wide. For instance, some CHCOs said  agencies can increase interagency cooperation on executive training by  sharing space in training seminars with other agencies, and allowing  admission of SES candidates from other federal agencies into training  programs such as SESCDPs. Treasury officials we interviewed said they  actively market shared training services to the rest of the government  through TEI. Energy told us the agency produces a catalog on a semi- annual basis with real-time information on leadership development  courses and programs located throughout the country; according to  Energy officials, government officials could use the catalog to reduce  travel and per diem costs for executive seminars conducted around the  nation by identifying equivalent training that is available locally for senior  executives. Additionally, several CHCOs said their agencies used VA’s  Leading EDGE program to deliver executive training. One CHCO offered  to share training facilities with other agencies. However, one CHCO noted  that courses sponsored by OPM and VA raised the possibility of  redundancy in executive training delivery provided by the two agencies.  According to our guide for assessing strategic training and development  efforts, agencies can try to achieve economies of scale and avoid  duplication of effort when assessing training delivery options by taking  advantage of existing course content or training, such as sharable online  courseware or multiagency training programs. Specifically, interagency  training can help address common developmental needs government- wide and promote cost-efficiency while taking advantage of existing  resources rather than creating similar programs in multiple agencies.
section_title: CHCOs Reported OPM Can Help Agencies by Offering More Lower Cost or No Cost Training and by Centralizing Training Offerings
paragraphs: CHCOs from agencies responding to our questionnaire reported that  OPM can take several steps to help both the federal government and  individual agencies deliver executive training more efficiently. First, some  agency CHCOs said that OPM should provide more training to senior  executives government-wide that is free or lower cost to agencies. One  CHCO noted that OPM could provide lower or no cost executive  development opportunities to agencies by more effectively using  technology to deliver virtual training. This would be consistent with  leading training practices comparing the merits of different delivery  vehicles, such as classroom or computer-based training, in order to  assess what vehicles to use for ensuring efficient and cost-effective  delivery. OPM officials said that agencies cannot avoid training costs, but  OPM is trying to reduce agency training costs by bringing courses closer  to agency offices and by using free space for training rather than renting  sites. OPM has also developed free and fee-based training solutions for  use government-wide, and hosts speaker series and meta-leadership  courses for senior executives.
Second, agency CHCOs said that OPM could take steps to help  centralize training offerings for senior executives, such as creating a  centrally funded SESCDP, and could coordinate joint training efforts  among agencies. For example, according to one CHCO, OPM could take  steps to encourage agencies to fill paid-for, but unfilled seats for  executive training programs across agency lines. CHCOs also said that  OPM could assist agencies in locating executive training programs open  to other agencies and departments, and provide a central source for  listing executive development opportunities throughout the federal  government. An example of this type of executive training program is the  Leading EDGE program administered by VA. This program is operated by  a senior executive within VA and offers executive training to agencies  across the federal government. According to the program’s deputy  director, 17 federal agencies are participating in Leading EDGE training.  Treasury’s TEI is another example of a shared service model that  presents an opportunity to leverage resources across the government.  According to Treasury, TEI serves 9 other bureaus and agencies in  addition to Treasury. Our guide for assessing training practices previously  found that centralizing design of training programs can, among other  things, offer potential cost savings.
OPM officials agree they could centralize training offerings for senior  executives. For instance, OPM hosted the Office of the Director of  National Intelligence’s (ODNI) suite of 12 courses detailing the full  performance management cycle; in turn, OPM’s Human Resources  University (HR University) made them available for use by other  agencies, and OPM officials said they incurred savings of about $46  million in 11/2 years. In terms of coordinating joint training efforts among  agencies, OPM officials said they collaborate with the CLO Council, the  CHCO Council, and federal agencies to provide joint training. For  example, OPM collaborates with VA to implement the Leading EDGE  program for use by many agencies. OPM officials also suggested that  federal agencies use additional FEI courses—and to reimburse FEI for  implementing such courses through user fees—so it could become the  federal government’s primary source for executive training. As mentioned  earlier, sharing online courseware and implementing multiagency training  programs can help agencies try to achieve economies of scale while  avoiding duplication of effort in designing executive training courses that  address similar content as existing courses.
section_title: OPM Reports Taking Additional Actions to Improve Training Efficiency
paragraphs: OPM officials noted they are taking additional steps to increase training  efficiencies beyond what agency CHCOs suggested. Attributes of  effective training include planning training that considers government- wide reforms and other targeted initiatives to improve management and  performance, as well as administration priorities, legislative reforms, and  major management challenges. In response to our September 2012  recommendation on federal training investment practices, OPM officials  said the agency is working with the CLO Council to draft a government- wide mandatory training curriculum and finish plans to implement a pilot  course for mandatory training. OPM officials also said the agency is  working with CLO Council representatives to standardize elements of  federal executive training programs by developing a framework for  supervisory and managerial leadership development. The supervisory  framework has been distributed to agencies. According to OPM officials,  senior executives and other managers can use the supervisory training  framework to identify learning objectives they need to achieve, and then  target the right job assignments and training courses to address the  objectives. OPM officials said agencies are not required to use the  framework to develop their executive training programs; however, some  agencies are using the framework as a “roadmap” for aligning their  training programs.
In November 2012, OPM also released its Executive Development Best  Practices guide, which includes key themes and recommendations OPM  identified as hallmarks of successful executive development efforts  across the public and private sectors. OPM has also developed a training  and development webpage where agency officials can find and share  products and courses for all areas of learning and development.  However, the webpage does not describe examples of agency efforts to  achieve efficiencies in delivering executive training.
OPM’s efforts so far to work with agencies to identify efficiencies for  executive training have been relatively broad-based; while OPM has  taken steps to increase efficiencies, it does not have a process for  assessing lessons learned from agencies about potential efficiencies in  executive training that other agencies could implement. Our questionnaire  results suggest that OPM can use its oversight role to take additional  steps to assess and promote ways agencies can achieve efficiencies in  executive training that have government-wide application. By not taking  such steps, OPM may be missing an opportunity to better position  agencies to achieve such efficiencies.
section_title: Career SES and SES Candidates Said External Training is Valuable, but Training Programs Could be Improved
subsections: section_title: Career SES and SES Candidates Found Executive Training Useful in Building Organizational Knowledge, Developing Leadership Skills, and Expanding Networks of Colleagues
paragraphs: Focus group participants said that executive training helped them develop  a better understanding of their roles and responsibilities in their respective  agencies. Following visits to field offices during executive training, one  participant said there is a significant amount of “interconnectivity and  intertwining between different operations” within the participant’s agency.  Participants also said they learned about key organizational issues and  tools that they can use to address them. For instance, a group of SES  candidates in one agency reviewed agency-wide telework implementation  and developed a business case to overcome technical challenges related  to phone systems.
Focus group participants from the three agencies we reviewed also said  they developed or strengthened key leadership skills during executive  training that align with ECQs and related competencies, such as
Leading and communicating with people. Participants we  interviewed said they learned how to better motivate and  communicate with staff. One participant noted that “how you interact  with people is critical” – a lesson that was emphasized when the  participant was implementing difficult changes in the participant’s first  SES position. The participant also learned to be both sensitive and  emphatic when dealing with issues raised by staff members. The  participant said that executive training enabled the participant to help  transform an organization with “primarily a bad reputation” into one of  the top performing organizations within the participant’s agency.  Leading People is one of five ECQs highlighted by OPM.
Problem-solving and decision-making. Executive training,  according to participants, provided an environment where they could  develop skills in problem solving. For instance, one participant  described learning how to make more effective decisions by sharing  best practices from different business units within an agency and  noted that, as a result, the participant and agency colleagues “are  making better decisions than if we were just kind of operating in our  private little bubble.” Another participant noted that executive learning  helps agency officials reach out to other areas in their organization  and leverage others’ experiences in resolving problems, “instead of  trying to reinvent the wheel” to identify solutions already considered  within the agency. Problem-solving and decisiveness are two  competencies identified under the Results Driven ECQ.
Strategic thinking. Executive training courses enabled some  participants to step back from managing day-to-day operations and  reconsider, from a strategic sense, what they wanted to accomplish in  their position. One participant said the use of strategic thinking and  leadership was necessary to create a vision for subordinates in  conducting agency operations. A participant from another agency  added that strategic thinking enables executives to consider the  needs of stakeholders beyond one division and office, and to identify  how any decisions the participant makes could affect an entire  agency. Strategic thinking is a competency identified under the  Leading Change ECQ.
Focus group participants also said that networking with colleagues and  senior agency officials in executive training was beneficial during training  courses and after re-entry into their positions. In particular, participants  noted the following networking opportunities in executive training:
Participation develops personal relationships that boost  confidence and build networks across and within federal  agencies. One participant said relationships built with others helped  make the executive feel more comfortable in an SES position, and  validated that the participant was “on track” when facing difficult  situations on the job. Another participant emphasized that a significant  amount of work in the participant’s position involved relationships  formed in small groups during executive training with people  representing other agencies.
Participation builds accountability among executives within an  organization. One SES candidate said that relationships formed  during executive training extended into executive assignments  following completion of training. The candidate said such relationships  helped training participants hold each other accountable in their  executive assignments, which can be a tremendous benefit to  organizations.
Participants develop better responses to customer and  stakeholder needs. Multiple participants said relationships they  established in executive training helped them collaborate to resolve  customer issues. For example, two participants from different  components in one agency said they developed a relationship during  executive training that enabled them to work together to provide  financial relief to an agency customer without using additional agency  resources. Another participant noted he completed a $400 million  project with another agency before a scheduled deadline, based on  trust and relationships he previously built with fellow participants in an  SESCDP.
Participation promotes interaction and relationship building with  senior leadership to address organizational issues. From  interaction with the Postmaster General, one participant learned  methods to motivate employees following “massive changes” within  organizations, such as those experienced by the U.S. Postal Service.  Another participant (who was addressing equal employment  opportunity issues) established a connection with an agency  undersecretary, who allowed the participant to send staff to the  undersecretary’s office for shadowing assignments to better resolve  equal employment opportunity issues within the participant’s office.
Participation facilitates brainstorming solutions to previously  identified issues faced by career SES and SES candidates. Focus  group participants in one agency discussed two cases where  networking and brainstorming among executives helped solve  problems and generated financial savings. One participant used a  week-long training course to address implementation of a program  involving overseas locations. Course members helped the participant  develop ideas to adjust hiring practices that, according to the  participant, saved the federal government money. Another participant  noted that agency officials used executive training to determine steps  needed to resolve backlogs of work within an agency; according to the  participant, this work may result in cost savings for the federal  government over the next 10 to 20 years.
section_title: Career SES and SES Candidates Said the Usefulness of Executive Training Depends, In Part, on Program Structures Used to Carry Out Training
paragraphs: Focus group participants found the use of cohorts—groups of career SES  and SES candidates participating in the same training courses, and other  types of groups—more conducive for problem-solving and knowledge  sharing than use of large group settings. Participants from one agency,  for example, supported the use of cohorts in executive training to foster  connections and build a knowledge base outside their respective  functions. Participants from Energy, Treasury, and VA said that  participating in small groups and using learning tools—such as case  studies and action learning projects—helped career SES and SES  candidates enhance problem-solving skills and share ideas with each  other. As an example, one participant said that members of small groups  applied lessons learned from case studies towards real-life problems.  Another participant noted that classmates from executive training  regularly met in groups to discuss challenges and share best practices  with each other.
Participants also found executive training to be more useful when  conducted off-site from the participants’ offices and when participants  could step back from day-to-day responsibilities. As described earlier,  participants who were able to withdraw from managing day-to-day  operations said they could incorporate more strategic thinking into  considering organizational issues and accomplishments, and could better  focus on learning needed skills to advance within their organization.  Focus group members specifically praised the use of off-site training for  allowing participants to develop ECQs, learn many of the skills needed to  succeed as an executive, and refresh insights or skills they may not have  used for a period of time, without distractions that may have hindered  training at an on-site location.
In contrast, some participants did not find large gatherings to be useful in  encouraging participation or integrating lessons learned. One focus group  member noted that colleagues took advantage of large gatherings to  complete training requirements without the need to pay attention to  lessons learned from the gathering. Another focus group member who  participated in a large gathering noted that attendees listened to speakers  for several hours without collaborating with the speakers or each other,  despite being informed they would be able to interact with people from  other agencies.
Additionally, focus group participants said they found virtual training with  computers, or with video or phone conferencing, to be less helpful than  face-to-face training. For example, one participant said that virtual training  tools, such as webinars, can provide effective training on technical or  discrete subjects, yet are less effective for developing less tangible  strategic, leadership, and executive skills. Participants said they found it  difficult to build teams or individual relationships through virtual means,  which limited networking and knowledge sharing. Other participants said  they were multitasking and completing other tasks while taking virtual  training; one participant noted that the training could be more interesting if  delivered through other means and in a shorter time period.
section_title: Career SES and SES Candidates Believe Agencies Can Take Steps to Enhance Executive Training Programs While Improving Efficiencies of Training Delivery
paragraphs: Focus group participants also provided suggestions to improve the  usefulness of such programs to career SES and SES candidates, while  also increasing efficiencies in delivering training courses that could  potentially reduce training costs for federal agencies.
As one example, focus group participants we interviewed suggested  agencies could pool resources and consolidate executive training  programs. One participant suggested the creation of a government-wide  SESCDP through OPM or another agency; this step could standardize  program financing and rules for program implementation government- wide. Likewise, focus group participants said executive training courses  could be consolidated through the use of economies of scale which would  acquire such training for multiple government agencies. Participants  noted that the federal government—instead of individual agencies—could  use its influence as a large-scale buyer to leverage taxpayer dollars and  negotiate lower prices for training from external providers. Another  participant suggested that the federal government could achieve cost  savings by requiring external providers to compete against each other to  provide executive training courses.
Focus group participants recommended the use of blended training  courses—involving both face-to-face and virtual training—to increase  efficiencies in training delivery. As mentioned earlier, participants  described limitations in using virtual training such as limited interaction  among participants and reduced attention paid to course material.  However, participants said a blended learning model allows them to  establish face-to-face relationships, with one participant noting that  human interactions were a key part of learning. At the same time, use of  computer-based technology and phone conferencing as part of a blended  learning model enables participants to build upon such relationships and  learn course material that can be effectively delivered through virtual  means.
Additionally, some focus group participants suggested that OPM and  other federal agencies could increase use of workshops and brief training  sessions to deliver training more efficiently to executives. Specifically, use  of one-day workshops and training sessions could increase participation  in training by bringing together executives from multiple agencies to  encourage cross-pollination of ideas and to learn improved leadership  techniques. Finally, participants said that agencies could establish  programs to increase selection of SESCDP graduates for open SES  positions government-wide. These steps, according to participants, would  allow agencies to more efficiently use training acquired by SESCDP  graduates, and to achieve greater cost efficiencies by filling vacant  agency SES positions with SESCDP graduates from other agencies.
section_title: Conclusions
paragraphs: Agencies are using their executive training programs to help develop the  necessary leadership skills for executives to effectively manage the  nation’s most pressing priorities. Agencies recognize that evaluating  executive training activities yields useful information for improving  investment decisions. Yet, agencies can only make decisions that  improve the value of executive training when they have complete and  reliable training cost data and some measure of how the training benefits  the agency beyond just a measure of participant feedback. Over 7 years  have elapsed since agencies were required to report training cost data to  OPM, and many agencies are not yet accurately and completely reporting  this information. For corrective actions OPM has already initiated,  establishing milestones could help ensure that efforts progress as  intended. Moreover, by not establishing a timeframe for improving the  reliability of executive training cost data, OPM may be missing an  opportunity to better position itself to hold agencies accountable for  improving the data. Improving the overall reliability of the data would  better position OPM and the agencies to make sound decisions on  training investments.
Half of the CHCOs reported they do not have complete information on the  cost of executive training provided by external training providers.  Moreover, many agency CHCOs struggle to conduct evaluations that  identify the impact of training on organizational performance and agency  mission, although a few CHCO agencies reported conducting such  evaluations. OPM has not used interagency councils to share lessons  learned from agencies that have experience with assessing executive  training’s impact on agency missions. Although OPM has developed and  published guidance on evaluating training’s impact on agency mission  and goals, over half of the CHCO agencies are not conducting these  evaluations of executive training provided by external providers. Agency  CHCOs reported that more assistance from OPM is needed. Additionally,  many CHCOs believe there are opportunities for an increased role by  OPM for improving efficiency in executive training and providing options  for achieving efficiencies. By not assessing lessons learned and other  potential efficiencies identified by agencies, OPM may be missing an  opportunity to better position agencies to achieve these efficiencies in  executive training. Given the fiscal challenges facing the nation, and the  increasing complexity of policy issues managed by the SES, it is  important for agency leaders and for OPM to take advantage of  reasonable opportunities to share resources and improve the efficiency of  training investments.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that agencies track and report comparable and reliable  cost data and perform evaluations that assess the impact of executive  training on agency performance or missions, we recommend that the  Director of OPM, in coordination with the CHCO Council, take the  following actions:
Establish interim milestones for meeting with agencies in order to  address training data deficiencies and to establish well-defined  timeframes for improving the reliability of the data in its EHRI  database.
Improve assistance to agencies regarding evaluating the impact of  executive training on mission and goals, for example by sharing  information and examples of how agencies could better conduct such  evaluations.
To enhance the efficiency of executive training, we also recommend that  the Director of OPM, in coordination with the CHCO Council
Assess potential efficiencies identified by agencies for possible  government-wide implementation, and then take the steps necessary  to implement these, such as updating the guidance governing  executive training programs.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Director of OPM for review and  comment. OPM provided written comments, which are reprinted in  appendix II. OPM concurred with our recommendations and is engaged in  a number of activities that would help to achieve the objectives of our  recommendations. However, we believe that more steps can be taken.
In response to our first recommendation, OPM discussed actions it has  initiated to address agencies’ training data deficiencies including  launching the Federal Training Investments Work Group, a group of Chief  Learning Officers, to help OPM define data elements that can and should  be captured within agency automated learning management systems.    OPM’s plan is to provide technical assistance to individual agencies  during FY 2014 to help build agency capacity and to allow agencies to set  milestones for improvement after assessing their training data using  forthcoming standards. We believe OPM could work with agencies to set  timeframes for agencies to have a plan in place for improving  completeness and reliability of agency data on executive training costs.  Working with agencies to mutually set timeframes would help OPM hold  agencies accountable for improving training data, thus moving agencies  towards meeting reporting requirements set in 2006.
In response to our second recommendation, OPM stated that it currently  provides assistance to agencies on evaluating the impact of executive  training through use of the OPM Training Evaluation Field Guide, which is  included in our report, but did not indicate whether or how it measures the  effectiveness of that guidance.  We believe OPM could further strengthen  its assistance by leveraging the experiences of agencies that have  conducted these evaluations by sharing techniques and approaches with  CHCO agencies that continue to struggle with evaluating executive  training. OPM could also ask agencies to assess the effectiveness of its  assistance.
In response to our third recommendation, OPM highlighted its Executive  Development Best Practices Guide which contains a compilation of  executive development best practices used by the public and private  sector to support the continued learning of executives, as well as  additional actions it has taken to help educate agencies on achieving  efficiencies in executive training. In its oversight role, OPM, together with  agencies, can develop and use criteria to assess efficiencies put in place  by agencies that have potential government-wide application. For  instance, OPM can assess agency initiatives—such as creating a  centrally-funded SESCDP, or coordinating joint training efforts among  agencies—to determine whether these alternative approaches are  appropriate for wider application.
OPM also provided technical comments on the draft report, which we  incorporated in the report as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the Director  of OPM and other interested parties. In addition, the report will be  available at no charge on the GAO website at www.gao.gov. If you have   any questions about this report, please contact me at 202-512-2717 or  jonesy@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made key contributions to this report are listed in appendix  III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this engagement were to review the costs of federal  executive training programs and the value they provide to the federal  workforce. Specifically, this report (1) describes what is known about how  much Chief Human Capital Officers Council (CHCO Council) agencies  have spent on executive training offered by external providers and  assesses the reliability of available cost data; (2) assesses how CHCO  Council agencies evaluate the effectiveness of the training; (3) compares  opportunities identified by agency CHCOs to achieve efficiencies in  executive training to leading training practices, and (4) provides views of  career Senior Executive Service (SES) and SES candidates from  selected agencies on the value of the training they received.
For the purposes of this review, we define the key terms “executive  training,” “external provider,” “SES candidates,” and “career SES” as  follows:
Executive training is classroom or computer-based instruction in  leadership or development programs with a focus on the Office of  Personnel Management’s (OPM) executive core qualifications (ECQ).  Each individual training episode should be indicated by completion of  Standard Form 182.
External provider is any organization (including educational  institutions, professional societies, think tank organizations, or  consulting firms) that provides training and is independent from the  federal department or agency that employs the trainee. The external  training provider is contracted to develop or provide training courses  or programs. In this case, OPM’s Federal Executive Institute (FEI) and  Management Development Centers are considered external  providers.
SES candidates are GS-15, GS-14, and equivalent employees who  are enrolled in either (1) an OPM-approved SES Candidate  Development Program or (2) any agency-funded program that will  train and develop employees for the SES or an equivalent.
Career SES or equivalents are those members with civil service  status who are appointed competitively to SES positions or  equivalents in positions below the top political appointees in the  executive branch of government.
To obtain government-wide information on agency executive training  programs, we developed a questionnaire on agency executive training  practices and processes and administered it to 27 agencies represented  on the CHCO Council. Through the questionnaire, we obtained high-level  information on executive training programs, evaluation practices, and  opportunities to achieve efficiencies in such programs, as identified by  agencies. To the extent available, we also collected data on the amount  agencies spent on executive training courses administered by external  providers from fiscal years 2008 to 2012. We provided a standard set of  questions to each CHCO to ensure we consistently captured their  responses to our questions on their executive training practices. We  received responses, supporting documentation, or both from 26 of 27  agencies. To analyze the information we received, one analyst inputted  responses to quantitative questions and analyzed the responses for  univariate statistics using Microsoft Excel; a second analyst checked the  input and analysis of responses for accuracy. For the responses to open- ended questions, we conducted a content analysis to identify common  themes among the 26 respondent agencies.
We assessed the reliability of executive training participant and cost data  from OPM’s Enterprise Human Resources Integration (EHRI) system  based on reviews of our prior work on federal training investments,  testing of agency data from an OPM executive training dataset, and  interviews conducted with OPM officials. We determined that the data  provided in EHRI was reliable for the purposes of selecting case  examples and for comparing participant and cost data in EHRI to similar  data provided directly by agencies. We also assessed the reliability of  executive training participant and cost data provided by agencies in  response to our questionnaire. Specifically, we examined the data  provided for obvious errors and inconsistencies and we also queried 10  agencies to better understand the data systems each agency used to  collect and report executive training participant and cost data, as well as  the quality of data entered into their system(s). We determined that  aggregated participant and cost data for fiscal years 2008 to 2012 are  sufficiently reliable for the purposes of the report; however, we qualified  these data throughout the report due to potential underreporting from  agencies.
We selected three agencies – the Department of Energy (Energy),  Department of the Treasury (Treasury), and Department of Veterans  Affairs (VA) – to obtain illustrative examples of how they approached  executive training in their respective agencies and collected data related  to costs; how they implemented and evaluated the results of their  executive training programs identified in the questionnaire; and how they  identified potential efficiencies and steps planned or already taken to  achieve them. We selected these agencies based on workforce size,  number of career SES, total training costs, and types of evaluations  conducted to measure effectiveness of executive training, and whether  agencies identified potential opportunities for achieving efficiencies in  executive training. We interviewed agency officials from the three  selected agencies with responsibility for administering their agency’s  executive training programs. To supplement and support the views  provided by agency officials, we also obtained and examined available  documentation (such as agency executive training plans, evaluation  plans, and other documents associated with executive training) from each  of our selected agencies. The findings are not generalizeable to other  agencies or to the federal government as a whole, because we designed  our selection of agencies to examine the experiences, practices, and  perspectives of a set of agencies that, in recent years, have utilized  external providers to train executives and have reported that they (1)  evaluate the benefits achieved through executive training, and (2) have  identified opportunities to achieve efficiencies in executive training.
Additionally, we conducted focus groups of current or prior executive  training participants from the three selected agencies in order to obtain  their views on the usefulness and value of such training and their ideas  for increasing efficiencies in executive training delivery within agencies  and government-wide. For two of these agencies, we conducted two  focus groups for each: one representing current or recent participants in  SES candidate development programs, and one representing career SES  members. For the third agency, which did not have recent SES  candidates at the time we selected our focus group, we also conducted  two focus groups: one representing early career SES members who  completed a basic agency leadership course, and one representing mid-  to late-career SES members who completed an advanced agency  leadership course. The insights gained from the discussions were not  generalizeable to all career SES and SES candidates within participants’  agencies or to the federal government as a whole.
We also collected documentation from and interviewed OPM officials  involved in the agency’s oversight function for federal training activities in  order to obtain information on their roles and responsibilities in monitoring  agency training programs, on their perspectives on executive training  government-wide, and on steps they have taken to improve training  efficiencies. We examined how, if at all, OPM, the CHCO Council, and the  Chief Learning Officers Council (made up of chief learning officers from  the 27 agencies) share information on potential efficiencies by  interviewing OPM officials and members of the CHCO Council and the  Chief Learning Officers Council. We interviewed five external training  providers who were most frequently used by CHCO Council agencies, to  collect information on executive training course offerings, costs, and  evaluation strategies; however, the collected information is not  generalizeable to all external providers. We also interviewed officials  from the Partnership for Public Service and Booz Allen Hamilton who had  relevant expertise in federal leadership training activities in order to obtain  information on prior studies of executive training, and on their  perspectives about evaluating the effectiveness of executive training.
We conducted this performance audit from December 2012 to December  2013, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Office of Personnel Management
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Signora May (Assistant Director),  Clifton G. Douglas Jr., Janelle House, Andrea Levine, Mark Ryan, Joseph  Santiago, and Cynthia Saunders made key contributions to this report.  Also contributing to this report were Sara Daleski, Deirdre Duffy, Rob  Gebhart, and Rebecca Shea.